A new study examines how large language models perform in a variety of medical contexts, including real emergency room cases \u2014 where at least one model seemed to be more accurate than human doctors.<\/p>\n
The study was published this week in Science<\/a> and comes from a research team led by physicians and computer scientists at Harvard Medical School and Beth Israel Deaconess Medical Center. The researchers said they conducted a variety of experiments to measure how OpenAI\u2019s models compared to human physicians.<\/p>\n In one experiment, researchers focused on 76 patients who came into the Beth Israel emergency room, comparing the diagnoses offered by two internal medicine attending physicians to those generated by OpenAI\u2019s o1 and 4o models. These diagnoses were assessed by two other attending physicians, who did not know which ones came from humans and which came from AI.<\/p>\n \u201cAt each diagnostic touchpoint, o1 either performed nominally better than or on par with the two attending physicians and 4o,\u201d the study said, adding that the differences \u201cwere especially pronounced at the first diagnostic touchpoint (initial ER triage), where there is the least information available about the patient and the most urgency to make the correct decision.\u201d<\/p>\n In Harvard Medical School\u2019s press release<\/a> about the study, the researchers emphasized that they did not \u201cpre-process the data at all\u201d \u2014 the AI models were presented with the same information that was available in the electronic medical records at the time of each diagnosis.\u00a0<\/p>\n With that information, the o1 model managed to offer \u201cthe exact or very close diagnosis\u201d in 67% of triage cases, compared to one physician who had the exact or close diagnosis 55% of the time, and to the other who hit the mark 50% of the time.<\/p>\n \u201cWe tested the AI model against virtually every benchmark, and it eclipsed both prior models and our physician baselines,\u201d said Arjun Manrai, who heads an AI lab at Harvard Medical School and is one of the study\u2019s lead authors, in the press release.<\/p>\n Techcrunch event<\/p>\n \n\t\t\t\t\t\t\t\t\tSan Francisco, CA To be clear, the study didn\u2019t claim that AI is ready to make real life-or-death decisions in the emergency room. Instead, it said the findings show an \u201curgent need for prospective trials to evaluate these technologies in real-world patient care settings.\u201d<\/p>\n The researchers also noted that they only studied how models performed when provided with text-based information, and that \u201cexisting studies suggest that current foundation models are more limited in reasoning over nontext inputs.\u201d<\/p>\n Adam Rodman, a Beth Israel doctor who\u2019s also one of the study\u2019s lead authors, warned the Guardian<\/a> that there\u2019s \u201cno formal framework right now for accountability\u201d around AI diagnoses, and that patients still \u201cwant humans to guide them through life or death decisions [and] to guide them through challenging treatment decisions.\u201d<\/p>\n
\n\t\t\t\t\t\t\t\t\t\t\t\t\t|
\n\t\t\t\t\t\t\t\t\t\t\t\t\tOctober 13-15, 2026\n\t\t\t\t\t\t\t<\/p>\n