Rumors that Apple might be creating its own search engine started doing the rounds last fall, when webmasters started seeing web crawler visits from IP addresses starting with 17 – the IP address block owned entirely by Apple. Apple has now officially confirmed in a support document that it is operating its own web crawler called Applebot.
Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the 18.104.22.168 net block.
While the wording is clearly intended to suggest that this is just business as usual, both the fact that Apple is running its own web crawler at all, and the somewhat vague wording, are interesting … Read more