Siri answered 78% of questions correctly in latest test, up from 66% a year ago

Analysts at The Loup Ventures have published the results of the their latest voice assistant comparison test, which involves asking the leading digital assistants 800 questions spanning various domains. Whilst Siri still trails Google Assistant in accuracy, it has improved significantly.

With the same test conducted last year, Siri correctly answered 66% of the questions. The new testing shows Siri is getting smarter, jumping to a 78.5% accuracy rate.

The Loup Ventures report breaks down queries that were answered correctly, and queries that were parsed and understood correctly but the assistant could not deliver an appropriate response. At least on the 800 questions asked in this test, Siri apparently understood 99% of them, with a 78.5% correct answer rate.

Google Assistant had an even better record, understanding 100% of queries and answering 85% of them with an appropriate reply. Both Alexa and Cortana scores poorly on this survey, but to be fair to Amazon and Microsoft, their assistant tests were conducted using the client iOS apps, not the manufacturer’s own hardware.

The test includes a variety of questions including queries for mapping (‘Where is the nearest coffee shop?’), shopping (‘Can you order me more paper towels’), trivia and commands (like setting reminders or changing device settings).

Siri performed exceptionally well with questions in the Command category. It seems a lot of this comes from Siri’s handling of smart home and music questions, which Apple have obviously paid attention to recently because of the HomePod’s debut.

Obviously, these scores do not suggest that Siri can answer 80% of questions successfully first time. Scoring is heavily dependent on the questions asked, and The Loup Ventures do not publish a list of all 800 questions surveyed. They merely claim to represent a sampling of the most important tasks customers expect from voice assistants. It’s not difficult to invent esoteric questions that none of the leading smart assistants could understand, so there’s always a degree of subjectivity here.

The best takeaway is probably the headline that Siri’s recognition is going up, with a 13 percentage point increase year-over-year. That being said, ‘Siri sucks’ is still very much a meme in the community.

Apple has recently hired John Giannandrea as VP Machine Learning and AI Strategy, to oversee Siri and related projects. Giannandrea previously worked at Google on projects including the Assistant, so it’ll be interesting to see how the fresh talent can advance Apple’s efforts in the personal assistant field.