Apple has finally confirmed that they have a web crawler, Applebot which is chiefly used for Siri and Spotlight Suggestions.
Applebot is the in-house web crawler of Apple that is used to help services like Siri and Spotlight on iOS and OS X platforms. Apple has also posted a help document, confirming about its web crawler, Applebot, “Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots Meta tags. It originates in the 18.104.22.168 net block.
User-agent strings will contain “Applebot” together with additional agent information. For example:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
If robots instructions don’t mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.”
A web crawler is an internet bot which is used by search engines that automatically scans and browse the webpages for indexing of information. You can exclude Applebot from searching your site by updating the robots.txt file on your site. AppleBot works similarly to Google’s Googlebot, and Apple instructed developers that it will follow Googlebot instructions if no AppleBot specific instructions are found.
Website managers can contact Apple’s network operations center if they have query or concerns about the crawler, Applebot.