Skip to main content

Apple confirms its ‘Applebot’ is indexing the web for Siri and Spotlight

applebot

Rumors that Apple might be creating its own search engine started doing the rounds last fall, when webmasters started seeing web crawler visits from IP addresses starting with 17 – the IP address block owned entirely by Apple. Apple has now officially confirmed in a support document that it is operating its own web crawler called Applebot.

Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the 17.0.0.0 net block.

While the wording is clearly intended to suggest that this is just business as usual, both the fact that Apple is running its own web crawler at all, and the somewhat vague wording, are interesting … 

Siri mostly uses Bing and Wolfram Alpha for its web searches, so one could ask the question why Apple would need to do its own indexing of the web? The wording of the support document too is somewhat vague: products including Siri and Spotlight Suggestions. This would certainly leave open the possibility that Apple is working on something bigger.

Is it likely that Apple is developing its own search engine, with plans to take on Google, Bing and Yahoo? I think not, for two reasons. First, Apple remains a hardware company using online services primarily to increase the attractiveness of its physical products. Sure, iTunes would be a Fortune 500 business in its own right – and will grow further when the revamped Beats Music service is launched later this year – but the ecosystem is really there to sell shiny new toys.

Second, Apple has long rejected ad-funded models, Tim Cook stating last year that free online services turned users into a commodity to be sold.

A few years ago, users of Internet services began to realize that when an online service is free, you’re not the customer. You’re the product. But at Apple, we believe a great customer experience shouldn’t come at the expense of your privacy.

Our business model is very straightforward: We sell great products. We don’t build a profile based on your email content or web browsing habits to sell to advertisers.

The likelihood, then, is that Applebot is exactly what it seems: simply a tool to supplement existing third-party search engines to ensure that Siri and Spotlight are delivering the best results. Google, Bing and Yahoo are likely to remain in competition to be Apple’s default search engine for some time yet.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

  1. philboogie - 9 years ago

    I agree with your opinion that Apple probably isn’t creating a search system themselves. I could see the benefit of it but fail to see how Apple could make it work. None of the current search systems work IMO.

    I don’t actually think it’s companies like Google, MS & Yahoo are at fault here for there crap engines (though Google was created in order to generate $10B/y in ad revenue, with a page ranking system, which is totally stupid) but the data itself isn’t exactly presented in a structured database design. I therefore don’t think web search can ever be done ‘right’.

    It’s up to webmasters and the like to tag their images, make correct use of keywords, date-tag everything in order for a search system to work properly. I don’t think we’ll ever see that as it hardly takes any effort for people to put a website, cost-wise or technically fluent.

  2. Paul Douglas - 9 years ago

    I fall in between. I do’t think Apple will launch a search website. But Siri and Spotlight Search ARE Apple’s Search service and I fully expect them to expand the capabilities.

    • rahhbriley - 9 years ago

      This. Spotlight is Apple’s search and they’ve been slowly advancing over the years.

      I speculated else where, in longer winded ramblings, that Apple is trying to kill the browser. I won’t need Safari to browse once spotlight can do all of that. I’ll click on the results, and increasingly those results won’t be to webpages, they’ll be to apps. Other things have to evolve for this, including our idea of apps and how they function, but I fully expect that is where Apple is steadily marching.

      Siri and Watson will only enhance the experience.

  3. Here’s a thought, maybe Apple are going to run an Ad FREE search engine, and give back by not selling the data and having ads everywhere, i mean, the idea itself, would help convert even more people to Apple products, driving there sales even more you would of thought… It is not like they don’t have the resources to run the search facility this way now is it…!

    • Smigit - 9 years ago

      That or the ads are implemented in such a way that they aren’t invasive…without tracking for example. If Tim Cook has a major issue with how search engines are run currently, and believes there is a market for an alternative (they did add DuckDuckGo as an option), then perhaps that’s actually the catalyst for entering the market as opposed to being a reason to stay out of it.

  4. AeronPeryton - 9 years ago

    Apple is probably not going to go the search.apple.com approach, but rather just use the data for exactly what they said; Siri, Spotlight, etc. I can see them using this for a new location-based Maps too. You’re in a new city and you can find out what people in your immediate area are searching for (restaurants, shows, where to park), but also for things like local news (tweets, breaking stories, local history). Hey look, one of my favorite bloggers lives here. Here’s a post of their idea of a great night on the town.

    But if Apple does launch a website with a search bar, that’s cool too. Full encryption against hackers or overzealous governments, no more paid results, no more SEO corruption, letting Apple use my search history to give me better results without the fear that it’s being sold for quick money…

  5. absarokasheriff - 9 years ago

    Another reason for doing this is to get the raw data instead of having it organized by Google, Yahoo, Bing. It will keep the others honest so that the third party providers structure it in the most informative versus most profitable for them.

    It may be a vetting process but it sure is interesting.

    • I think you have hit it spot on there. Equally however it does a similar job by keeping the options open too should other search engines fail to provide the service required by Apple. A third option may be to add filler search based services exclusive to their own products that can give specific information based on its wider use of cloud services very much of the sort that AeronPeryton mentions above.

  6. Tuvatech - 9 years ago

    First of all, Applebot would be an amazing name for a search engine. That alone makes me want to see it happen. :D

    Obviously I have no insight into their products or business models, but I could see several reasons for creating their own search engine. After all, they decided to make Apple Maps (even though Google Maps service was, and still is, a lot better). You have to look at things in a long term perspective. Sure, Apple could use Google Maps and Google Search (or Bing or whatever), but these things are here to stay. I mean unless we kill the planet, we will most likely use those things even 50 years from now. Whether we have a chip inside our brains or something else, that is irrelevant.

    So it might make more sense to just create your own search engine, mapping service, etc – just so you would have more control over how it works. I’m sure there are hundreds of issues that Apple has with Bing, but they can’t really change them.

    And I would disagree that Apple is a hardware company and only uses online services to increase the attractiveness of their physical products. There was a page in Isaacson’s book where he asked Steve whether Apple was a hardware or software company. Although Apple is actually both, Steve ultimately said that software was more important. If my memory serves me right (I read the book right after it came out), Steve even said that Apple was primarily a software company. There was some talk about the iPhone only having one (major) button and how everything else was done via software. I think he also said something about how easy it is to copy physical devices, but not so easy to figure out how to copy the software. You can buy a replica iPhone from China for 100 bucks, but its software is #$@% because they don’t know how to copy that.

    Steve also said that those who are serious about software should make their own hardware. So I think Apple is primarily a software company that just makes their own good hardware. :)

    iTunes, Apple Pay, iCloud, etc have nothing to do with their physical devices. It’s all software. They could leave payment systems to PayPal, cloud-services to Dropbox, etc, but still they do it.

    If they are indeed working on a car as well, they need all the software they can get. Maybe one day you will start ordering pizza or groceries through some kind of an Apple service. Kind of like an iTunes for physical goods. Having their own search engine would definitely NOT hurt because they could get more information about what kind of things people are interested in, etc.

  7. Luis Alejandro Masanti - 9 years ago

    ‘What if’ Apple makes a paid product out of search.
    This will be a truly disruptive change. We have so deep inserted in our minds the search=free=advertising paradigm.

    What if Apple offer a search engine for which you pay an annual fee (first year free with new device…) and that gives you the ‘best answer to what YOU are looking for.’
    Actually, Google et als. give us ‘the best answer to their advertisers.’

    In this service, you’ll have total privacy and be able to review and modify your data, including deleting all of that.

    • Tuvatech - 9 years ago

      Interesting thoughts. It’s certainly possible, but then again unlikely. As someone already commented, Apple is trying to kill the browser. So the future of search will be via Siri, Spotlight and maybe some apps or something. Some things should remain free, so it could be difficult to draw the line.

      It would; however, be truly amazing to be able to review and modify your data, as well as to get the answers that ”I” am looking for. As I said in my original comment, I’m sure there are many-many things that Apple could do differently with their own search.

  8. gshenaut - 9 years ago

    Yeah, I saw many attempts to access my servers from those addresses. Once I realized they were coming from apple-owned IPs and noticed that they weren’t super frequent, I decided not to block them. This article has clarified what was undoubtedly going on, so thanks.

Author

Avatar for Ben Lovejoy Ben Lovejoy

Ben Lovejoy is a British technology writer and EU Editor for 9to5Mac. He’s known for his op-eds and diary pieces, exploring his experience of Apple products over time, for a more rounded review. He also writes fiction, with two technothriller novels, a couple of SF shorts and a rom-com!


Ben Lovejoy's favorite gear