AI Is Hallucinating Your Medical Diagnosis

A Carnegie Mellon robotics researcher just demonstrated something that should terrify anyone excited about AI-powered healthcare: when you remove medical images from diagnostic AI systems, they don't admit ignorance. Instead, they confidently invent diagnoses based on whether you're old, young, male, female, or what race you happen to be.

Siddharth Vohra's study of Claude, GPT-5, and Gemini found that these leading large language models fabricate medical diagnoses in 18% of cases when images are withheld. That's not a rounding error. That's nearly one in five interactions producing medically dangerous fiction dressed up as professional assessment.

What makes this particularly insidious is the mechanism. These aren't random hallucinations—they're systematic biases encoded at scale. The models are pattern-matching against demographic stereotypes rather than medical evidence. Age, gender, and race become diagnostic inputs in themselves, producing the kind of algorithmic discrimination that civil rights advocates have warned about for years, now with potentially fatal consequences.

The timing of this research couldn't be more critical. Healthcare AI adoption is accelerating rapidly, with another recent study showing that a new AI blood test can predict heart disease 15 years early. The industry narrative suggests we're on the cusp of a diagnostic revolution, where AI augments or even replaces human judgment in clinical settings. But Vohra's findings suggest we're building castles on algorithmic quicksand.

The problem isn't that AI can't be useful in healthcare—it's that we're deploying systems without understanding their failure modes. An AI that predicts heart disease from blood markers is analyzing objective biochemical data. An AI that diagnoses from images but defaults to demographic stereotyping when confused is doing something far more dangerous: it's automating bias while projecting an aura of scientific objectivity.

This matters because healthcare institutions are under tremendous pressure to adopt AI. The technology promises cost savings, faster diagnoses, and better outcomes. Hospital administrators see competitors deploying AI and fear falling behind. But the Carnegie Mellon study reveals that speed and confidence aren't the same as accuracy.

The solution isn't to abandon healthcare AI—it's to demand radical transparency about when and how these systems fail. If an AI can't make a diagnosis without demographic information serving as a proxy for medical evidence, it should say so explicitly. If removing an image causes the system to fabricate rather than defer, that's a fundamental architectural flaw, not a feature to be optimized away.

We've seen this pattern before. Facial recognition systems that couldn't recognize darker skin tones. Hiring algorithms that discriminated against women. Credit scoring models that penalized minority applicants. Each time, the response has been the same: surprise, investigation, promises to do better. But in healthcare, the stakes aren't getting denied a job interview or a loan. The stakes are getting the wrong diagnosis, the wrong treatment, or no treatment at all.

The robotics and AI community needs to reckon with a uncomfortable truth: sometimes the most important innovation isn't building more capable systems—it's building systems that know their own limitations. An AI that admits uncertainty is far more valuable in a hospital than one that confabulates with confidence. Until we can guarantee that distinction, every healthcare AI deployment should come with a warning label: this system may be making things up based on who you are, not what's wrong with you.