A new study from AAA on the potential safety risks of hands-free systems for vehicles warns Apple’s Siri presents a “relatively high” level of distraction. In its tests, AAA ranked use of Siri as a “category 4 level of mental distraction,” which puts it in last place compared to other hands-free voice controlled systems from the big car makers including Ford, Mercedes, Hyundai, Chevrolet, Chrysler, and Toyota. It was also the only voice controlled system that resulted in two crashes in the simulator study:
To understand the workload rating associated with interacting with Siri, it is first useful to consider what is not causing the effect. The high level of workload is not due to visual/manual interference. Participants never looked at nor touched the iPhone during the session; in fact, the experimenter performed all manual interaction with Siri. As such, this indicates that the impairments were cognitive in nature, associated with the allocation of attention to the task. The high level of workload is also not due to the quality of vocal input or audio output. Participants wore a lapel microphone that allowed them to speak in a normal voice and the audio was played clearly over stereo speakers in the lab, simulator, or car. Our current research also indicates that the quality of the synthetic speech was not a major contributor to the effect. As such, this suggests that the impairment was not attributable to input/output operations. However, as depicted in Figures A1 and A2(Appendix), Siri had the lowest rating
The study tested hands- and eyes-free use of Siri with its own setup, but it didn’t include Apple’s new CarPlay in-dash system that is just now hitting the market through a small selection of new vehicles and aftermarket installations. We found in our review of CarPlay that the system makes some big improvements that make using Siri in the car safer, but car makers also continue to offer eyes-free Siri support in vehicles as CarPlay slowly rolls out. Siri Eyes Free allows users of select vehicles to activate and control Siri using controls on the steering wheel.
You can view the full report from AAA here.
FTC: We use income earning auto affiliate links. More.
Who the hell is AAA to post such a “study”? I am so sick of this assumed authority bullshit. In-Car voice controlled systems are the worst pile of garbage ever developed. Especially those from Ford and Chrysler.
What does AAA have to gain by performing some completely unscientific study and posting results as if they were in anyway meaningful?
You really don’t like any criticism of Apple, huh?
If you took 3 minutes to read even the abstract of the study (or even the excerpt posted in this very article[!]), you would see that they’re actually saying Siri is TOO helpful. This is actually an endorsement of Apple’s product; it’s too helpful to use while operating a motor vehicle. That’s why CarPlay pares some of the functionality away–again, noted in the article.
Btw, this study is actually the opposite of “unscientific”–which you would have seen if you bothered to open the source link. Unfortunately, the validity of information isn’t dependent on the confirmation of your biases.
Any endorsement of the horrible accident-waiting-to-happen systems from the manufacturers is a joke, and reaks of fraud and influence… Not surprising from a company that owes their existence to the automotive industry… Not Apple.
Further to that point, I don’t care what AAA has to say about anything.
Having read their metrics for this test you laud as soooo scientific, I can conclude that their driving simulator test is an embarrassment. “Siri use was the only one that resulted in two accidents” RIGHT!!!!!! IN your completely unaccountable simulator test?? RIGHT. That’s a believable story.
Holy hell, how much stock do you own of Apple? Anything that isn’t a glowing review of an Apple product is immediately met with vitriol from you.
BS. I love using Siri so I can keep my eyes on the road and hands on the wheel. I find most studies way off from the real world.
This mentality is exactly what’s wrong with most of our world leadership. Science and expertise is seen as impractical. Having elite knowledge on a topic is somehow trumped by “real world” experience.
Ha, I wouldn’t call this study any sort of elite knowledge or that this study was even done in a properly scientific way. Don’t assume that. They admitted they have no idea why Siri scored so poorly given that the problem wasn’t with any input/output. So it’s purely cognitive. So what does that mean? People are thinking harder when they talk to Siri then when they talk to Ford? Why? We’d have to see the exact methods used in this study, what were the participants doing for each test, and until we understand why the results are what they are, the study is fairly meaningless.
But I agree with your sentiment in a general sense, the world doesn’t like inconvenient science and leaders often default to their personal gut for answers, lol.
Certainly the research is open to criticism, as much as any research is. The methodology is described in detail the paper though.
I think there’s a disconnect here as to the title of the post, the excerpt given in the post, and the actual subject matter and purpose of the research. From what I understand after skimming the paper, Siri is just one of the many subjects of the research, and it just so happened that it came in as the most distracting.
Many of us commenters are reading this as though the takeaway is that Siri is “bad”, probably thanks–at least in part–to the headline.
Why Siri required the most cognitive function is outside the scope of this research. That kind of finding is probably better left for neuroscientists etc. anyhow.
Ignore people here, Martin. They read a headline and skim the article and scream at the top of their lungs at how stupid it is if isn’t a glowing report about Apple. Just look at jrox16, he asks questions about the “exact methods” when they are clearly spelled out in the source.
I think 2-3 crying kids in the car can be more disturbing than any voice controlled system!
2-3? My 2.5 months old baby is enough by himself :)
I am surprised that 9 to 5 would publish this bullshit.
How is this Bullshit?
I’m not surprised that a person on this website would think that anything that isn’t blowing rainbows up Apple’s ass is labeled as “bullshit”.
Some people lack the skills that are required to multi-task.
I am very comfortable with using Siri to play music, check stock prices, ask for directions, etc. while driving. I simply keep in mind that driving is the primary task.
I agree with the study. Siri is so unreliable that it takes 3 times or more for it to get something right. Each time it gets something wrong, you’re trying to stop it so you can try again. Why after this long it’s still a POS is beyond me. I gave up on Siri a long time ago. It’s simply just a lot quicker to do its tasks manually (while parked or stopped).
But I suppose there’s no problem at all with Google Now or Cortana :(
Given they made a correlation that people seem more distracted with natural language systems than dumber basic command systems, it would follow that Google Now or Cortana probably would be just as distracting. But why is the interesting part, and this study doesn’t solve that. One would think intuitively that being able to speak naturally as you would to a passenger would be easier and safer than remembering a specific command.
I can talk incredibly naturally to Google Now.
Why would you suppose that?
Until they can analyze and understand these results, the study is meaningless. It could be a pile of random correlations. The study itself indicates that it basically has no idea why Siri would rate the worst even though the input/output of the system is not to blame. So what is it specifically about Siri? Is it because it’s natural language? So users have to think too much about how they are going to ask a question? Are the car systems dumbed down so all you say is one or two words?
Is it because people are trying to send text messages using Siri but obviously can’t do that with car systems so the things people are thinking about with Siri are much more complex? How was the study conducted, did people do the exact same operation for each system I’d hope?
There would be much more value to this study if we would understand the reasons for the results.
More specifically, I’d like to see exactly what people were trying to do with each system, the words they were speaking, not just a general synopsis of the content of the experiments. The details can help to understand the results better, until then, it’s interesting but doesn’t tell me much. I did not see any specific details about what was being said when they were doing “text to speech”, just that it was a test to speech exercise for instance. (unless I missed it, I was skimming past all the math parts)
If you had read the study you would know the answers to your questions.
Maybe you should learn to read original sources before you ask questions that are spelled out and answered in straight-forward manners before you make a fool of yourself.
the headline is so misleading, the study DID NOT TEST ANY OTHER SYSTEM, this is not a comparison between SIRI and other voice system, this report is about distraction while using ANY voice system
I think the problem is that Siri is being pitched WAY beyond the tech’s capacity to deliver. It should be reduced to a fun AI to help with minimal tasks, and not be pitched as some HAL 9000 GPS service.
Wow none of you iDiots bothered to read the study. I am quite the iPhone fanatic BTW.
Wow, of all the Apple products I would expect to warrant a knee jerk defense by fans, Siri is not one of them! I know Im just piling on the anecodal evidence, but I know nothing causes me to rage more than trying to talk to Siri while driving.
Guess Cupertino can do no wrong in the eyes of some…
Siri sucks major ass. I have Apple products and I call her Bitch Siri. She is so stupid. She will confuse the word The with They and hang. I asked her to play my new message and she said I didn’t have any. I asked her to make a Genius playlist of the song I’m playing and she said I didn’t have a Genius right now. The other half of the time she gets my words wrong. I’ve lived my whole life in The USA and have a college degree so there’s nothing wrong with my accent. Don’t let me get into autocorrect. That is irritating as fuck. I hope they fix Siri with some IBM Watson and better voice recognition so I don’t have to slap Bitch Siri again with my ring hand.
I hate her stupid remarks like I set a 15 second timer and she says some stupid ass remark all loud ” ohh, you better hurry, the water is gonna boil” or some dumbass shit like that, while I’m trying to listen to something I’m timing. Errr! Google Now is way better, but I like the simplicity of Apple products. I am fucked on my buying decision this Christmas. A Nexus 6 or iPhone 6 Plus. Fuckk!!!
“With regard to Siri, it is also useful to contrast it with the “best case” natural listen + compose condition, which was rated at 3.08, and used Wizard-of-Oz technology to achieve perfect speech recognition. Siri scored more than a full point higher on the workload rating scale (4.15), and this likely reflects the added complexity when the voice-recognition system is less than perfect. Siri can learn about accents and other the characteristics of the user’s voice, so it is possible that with extended practice the workload ratings might improve. Common issues involved inconsistencies in which Siri would produce different responses to seemingly identical commands. In other circumstances, Siri required exact phrases to accomplish specific tasks, and subtle deviations from that phrasing would result in a failure. When there was a failure to properly dictate a message, it required starting over since there was no way to modify/edit a message or command. Siri also made mistakes such as calling someone other than the desired person from the phone contact list. Some participants also reported frustration with Siri’s occasional sarcasm and wit.”
There. For those who couldn’t be bothered to read the study before trashing it. Siri suffered because it is a natural language system. Other natural language systems would likely score poorly as well. The systems that scored better were simpler menu based systems. Nowhere does AAA say that Siri is the lesser of natural based language systems. It was used as a proxy for all of them, of which there are few (e.g. Cortana and Google)
You had better be careful. Actually reading things and making informed comments on a topic might just get you banned from the internet.
No it is much safer to get angry and emotional because of a perceived slight.
Honestly, I am with the others, AI’s reporting on this was poor at best. It is almost like they didn’t actually spend the time to read or even peruse the actual research article.
The other finding of this scientific study, and that is what it is, was that even within non-natural language voice command systems there was wide variance in cognitive workload which correlated with the amount of time required for the interaction.
Put simply, the more words that you have to use the more time you have to spend interacting and the more it distracts you. So, until any natural language system can approach human levels of understanding and interaction natural language will always be more distracting then canned and simple voice commands.
How the F is not safe when all you have to do is “Hey Siri, call this fker’s name”? you don’t even need to look, touch or do anything. Siri will respond with voice “calling fker’s name”
Whoever said Siri is not safe? My god, the world is full of idiots.
Siri is the honestly the worse and the fact Apple has not done anything to innovate that is surprisingly showing their real intentions with improving their iPads/iPhones. Lastly, CarPlay can improve annoying Siri hopefully.
Vince @ http://veloxity.us
In looking at the AAA study, I didn’t see a comparison with other voice systems.
The study was creating some artificial communications in a car simulator.
But the driver (of the simulator) was given the highest workload only with Siri;
From the study;
“using “hands-free” Siri to listen to and send text messages, update Facebook or Twitter status, and modify and review calendar appointments.”
No competing systems from other companies was used in the study.
And only Siri was given the task of updating Facebook or Twitter status.
Conclusion;
THIS STUDY DID NOT TEST SIRI COMPARED WITH OTHER VOICE SYSTEMS.
THE STUDY COMPARED DIFFERENT LEVELS OF DISTRACTION AND HAPPENED TO GIVE SIRI THE HIGHEST WORKLOAD LEVEL.
(Caps because the title of the article was misleading.)
You need to read the study again. I will admit that the modify and review calendar appointments was something that was outside of the range of the other voice command systems, and I would be curious to see how it correlated with the accidents, as making appointments requires a huge mental workload.
That being said, several of the systems were used to listen to text messages and listen to and respond to text messages. Updating Facebook and Twitter should be no more mentally taxing then composing a text message.
Here is the simple take away from this study. Siri’s comprehension is not perfect and its reaction to requested tasks is not perfect. This caused a larger distraction in the test subjects then using a mythical (Wizard-of-Oz tech) perfect voice recognition system or a system designed for non-natural language canned commands.
You should really learn to comprehend what you read. Posting to twitter is not more complicated than listening to texts and responding (which is what they did for other systems).
Had to take my daughter to an arts & crafts store. Hopped in the car, asked Siri, “Where’s the nearest Michaels?” Siri gave me some obscure company name with the name “Michael” in it that’s located hundreds of miles away. Ok, try again. “Where’s the nearest Michaels store?” Another obscure company far away. Open Google app, hit the microphone. “Where’s the nearest Michaels?” Google gives me the store name and number, 3.4 miles away, and hands me directions.
I love Siri for a lot of tasks, but sometimes, I think she’s downright stupid in a lot of other respects.
Am I the only one who lives in a state where it is illegal to wear headphones whilst driving? No wonder they think that it is more dangerous to drive and use Siri! They are having to take a hand off the wheel, hold the button, listen in a small speaker rather than as an all-around sound from their speakers, and then mess with the headphones falling out of their ears.
Apple added a new feature called, “Hey, Siri!” I’d like to see a study that uses this feature. I’d put more real-world stock in that.
Why don’t you learn to read before commenting? From the study article:
“To create a hands-free system, a lapel microphone was clipped to the participant’s collar and the audio output was played through the external speakers. Participants were asked to interact with Siri to perform three tasks: listening to and sending text messages, updating Facebook or Twitter status, and modifying and reviewing calendar appointments. Participants were instructed to activate this version of “Eyes-Free” Siri by saying, “Hello/Hi Siri.” The researcher would then manually activate Siri to allow participants to state their command(i.e., the participant neither looked at nor made physical contact with the iPhone). Before starting the condition, the researcher demonstrated the use of each interaction, and then participants were required to demonstrate proficiency before data collection commenced. Participants were free to alternate between the three tasks in a self-paced order.”
So, just in case you couldn’t understand that, the participants did not have to remove their hands from the wheel nor did they have to hold any button. Also, output was played through the car speakers. The purpose of the mic was just so that there wouldn’t be any issue with Siri not being able to hear the commands.
Though you are right, you are the perfect jerk and reason people avoid commenting. It is okay to be right and not be a complete butt-hole about it.
Kyle, when I try to be nice, people respond with vitriol. The comments section of this website is so filled with circle jerking that it’s insane. I love my Apple products, but people on this site make people think that everyone who enjoys Apple is a psychopath.
“Why don’t you learn to read before commenting?”
“So, just in case you couldn’t understand that,”
You have a twisted sense of “being nice”. All I’m saying is, *most* Apple people are elitists, but they let it carry to far. I own more Apple devices than can fit on a 10×15 wall of box collage, but I do my best to not turn people off by using passive-aggressive language to insinuate that they are less-than-intelligent.
And the people I normally respond to are the same exact commenters who have diarrhea of the mouth. Sorry, your response was the last on the list and I had already waded through the trash that other people have been spewing. I apologize for being rude. Reading the source is usually the best bet though, as most articles (anywhere really) just post clickbait articles and then people never actually read the source, which usually doesn’t even come close to saying what the author on this website or any other blogger has concluded.
I did, in fact, not read the article. I’ve been jerked around by AAA a but too much in my past, so I tend to be skeptical when they come out with an authoritative statement. Clickbait is all the trash media posts now. My mistake for not reading thoroughly and entirely, but the picture of the man driving with headset in-ear and hand trying to find the mic is very, overly misleading.
I agree that it is misleading, and it isn’t anywhere in the source material. To be fair, I wouldn’t be surprised if the author of this “article” here at 9to5 actually read the source material either.