ChatGPT Health sounds promising — but I don’t trust it with my medical data just yet

I can see why ChatGPT Health might entice people, even with all its caveats about what it can and can’t do for you. Healthcare systems are complicated, often overstretched, and can be financially draining. ChatGPT Health can provide clearer explanations of medical jargon, highlight things worth discussing with a doctor or nurse before an appointment, and immediately parse confusing test results.

ChatGPT Health entices with its ability to deliver insights that were previously only available through professional channels. It’s an easy sell, especially if you lack easy access to regular medical care. But valuable, personal healthcare information and AI’s track record for hallucination make for an unhealthy brew.

Misleading or fictional answers are bad enough even without the need for medical nuance. It’s not that I’m against asking ChatGPT questions about my health or fitness, but that’s a far cry from what ChatGPT Health suggests sharing.

Sign up for breaking news, reviews, opinion, top tech deals, and more.

“Exploring an isolated health question is fundamentally different from giving a platform access to your full medical record and lifetime health history,” explained Alexander Tsiaras, founder and CEO of medical data platform StoryMD. “Trust will be the central challenge, not just for individuals but for the healthcare system as a whole. That trust depends on transparency: how data is ingested, how hallucinations are prevented, how longitudinal clinical evidence is incorporated over time, and whether the platform operates within established regulatory frameworks, including HIPAA.”

OpenAI is not a healthcare provider. HIPAA (Health Insurance Portability and Accountability Act) and its strict legal obligations and safeguards for your medical information don’t apply. So, while OpenAI might sincerely promise to treat any health data you freely upload or connect to ChatGPT Health with matching care and security, it’s unclear if they have the same legal motivation to do so.

And when data is outside HIPAA’s reach, you have to trust a company’s own policies and intentions without relying on HIPAA’s enforceable standards. Some people might argue that OpenAI’s own privacy commitments are sufficient (lawyers at OpenAI, for instance).

You may like

Most people would agree that intimate medical information should be locked down under the tightest legal protections possible. But ChatGPT Health users will be in a precarious position, especially because policy language and terms of service can change with little notice and limited recourse.

ChatGPT Health’s isolating of health conversations and allowing users to delete their data are good ideas, but once your data is out of the traditional healthcare system, you have to consider a whole different set of vulnerabilities. Headlines underscoring how difficult it is for major tech platforms to guarantee airtight data protection against leaks and hacks are far too frequent.

And there’s more risk to your data, even from legitimate agencies. Private health data could be subject to subpoenas, legal discovery, or other forms of compelled disclosure. In some cases, companies can be forced to hand over private records to satisfy court orders or government requests. HIPAA has much stronger defenses against such legal pressures than the standard consumer tech privacy laws under which ChatGPT operates.

Affordable, accessible healthcare is a cause worth pursuing. But privacy, trust, and meaningful human oversight cannot be casualties in that pursuit. People are already concerned about how their data is collected and used by AI platforms. Centralizing even more sensitive information with a single commercial entity won’t reduce those concerns.

There’s a bigger conversation to be had about incorporating AI into systems that human beings depend on. AI can be a huge boon for providing healthcare. But it won’t be predicated on surrendering personal data to uncertain security and safety.

“These are foundational requirements,” Tsiaras said. “What people actually need is a clinically sound, longitudinal medical record that supports meaningful patient and patient-provider interaction, not another layer of noise.”