Earlier this month, ChatGPT Health launched with integration for Apple Health and several other data providers. But when The Washington Post’s technology columnist gave ChatGPT access to his Apple Watch data, it didn’t go very well.
ChatGPT Health gave inconsistent, inaccurate interpretations of Apple Health data
Geoffrey A. Fowler writes at The Washington Post:
Like many people who strap on an Apple Watch every day, I’ve long wondered what a decade of that data might reveal about me. So I joined a brief wait list and gave ChatGPT access to the 29 million steps and 6 million heartbeat measurements stored in my Apple Health app. Then I asked the bot to grade my cardiac health.
It gave me an F.
I freaked out and went for a run. Then I sent ChatGPT’s report to my actual doctor.
Am I an F? “No,” my doctor said. In fact, I’m at such low risk for a heart attack that my insurance probably wouldn’t even pay for an extra cardio fitness test to prove the artificial intelligence wrong.
Fowler explains that ChatGPT’s grade seemed tied to several misinterpretations of his Apple Watch health data.
For example, he says that ChatGPT Health “based much of its negative assessment on…VO2 max” despite Apple saying that its VO2 max readings are simply estimates. They’re helpful for tracking trends, but getting truly precise data requires separate equipment.
Additionally, resting heart rate changes that happened when Fowler got a new Apple Watch weren’t true changes, but rather the result of updates to sensors and measurement tools. That didn’t seem factored into ChatGPT’s evaluation, however.
Top comment by Thomas Ross
This is a poor move. ChatGPT has gone the wrong way in evolving their AI. Worse than what it was. I shudder to think of giving my health data to it. Aside from privacy concerns just poor LLM that spends too much time shortcutting answers to be quick and responsive but misses out on nuance, depth, and truth checking.
Inconsistency in responses was another issue:
When I tried asking the same heart longevity-grade question again, suddenly my score went up to a C. I asked again and again, watching the score swing between an F and a B.
Across conversations, ChatGPT kept forgetting important information about me, including my gender, age and some recent vital signs. It had access to my recent blood tests but sometimes didn’t use them in its analysis.
For anyone who has used an AI chatbot, these issues probably won’t be surprising. But for a product intended to be a source of health knowledge, it’s especially concerning.
9to5Mac’s Take
Apple is rumored to be working on an AI-powered ‘Health+’ service for later this year. And this early report on ChatGPT Health makes two things clear:
- It might be very hard to achieve the level of quality Apple will presumably want
- But if Apple can pull it off, Health+ might quickly become a leader in the AI health space
Have you given ChatGPT Health access to your Apple Health data? How has that gone for you? Let us know in the comments.
Best iPhone accessories
- AirPods Pro 3 (now only $199, down from $249)
- 10-year AirTag battery case 2-pack
- MagSafe Car Mount for iPhone
- HomeKit Garage Door Opener
- 100W USB-C fast charging power adapter
FTC: We use income earning auto affiliate links. More.
Comments