{"id":122532,"date":"2026-07-29T09:05:10","date_gmt":"2026-07-29T09:05:10","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/122532\/"},"modified":"2026-07-29T09:05:10","modified_gmt":"2026-07-29T09:05:10","slug":"can-doctors-trust-clinical-ai-the-complicated-issue-of-llm-benchmarks","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/122532\/","title":{"rendered":"Can doctors trust clinical AI? The complicated issue of LLM benchmarks"},"content":{"rendered":"<p class=\"author-bio\">Katie covers the impact of health technology on patients, clinicians, and businesses. Her stories explore the price tag of clinical AI, digital health at the FDA, and the boom in direct-to-consumer telehealth. Confidential tips can be sent on Signal at palmer.01.<\/p>\n<p class=\"wp-block-paragraph\">Hundreds of thousands of U.S. doctors use clinical large language models, pitched by companies like OpenEvidence, Doximity, and UpToDate as an antidote to the dangers of hallucination-prone generalist models from Big Tech. Yet few studies have pitted them against each other \u2014 and this summer brought a high-profile head-to-head.\u00a0<\/p>\n<p class=\"wp-block-paragraph\">Researchers from NYU Langone Health had tested general and clinical models, including OpenEvidence and UpToDate Expert AI, on three sets of clinical questions. The findings, published in Nature Medicine in June: The clinical AI performed worse than the general models.\u00a0<\/p>\n<p class=\"wp-block-paragraph\">The results rang out like a gunshot. \u201cI\u2019ve never seen a single paper trigger the kind of reactions this one has in the health AI community,\u201d <a href=\"https:\/\/www.linkedin.com\/posts\/danielayang_general-purpose-large-language-models-outperform-share-7472786182888771584-17xY\/?utm_source=share&amp;utm_medium=member_android&amp;rcm=ACoAABwDShQBLcuRrEDaKuPvyY-y-FMOt2B_BHc\" target=\"_blank\" rel=\"noopener nofollow\">wrote<\/a> Kaiser Permanente\u2019s vice president of AI and emerging technologies on LinkedIn. The paper\u2019s findings, like all science, are subject to interpretation and debate \u2014 but many online reactions treated them more like a clear victory for general frontier models.<\/p>\n<p class=\"restricted-text\">STAT+ Exclusive Story<\/p>\n<p class=\"restricted-login\">Already have an account? <a href=\"https:\/\/www.statnews.com\/login\/\" data-stat-login=\"\" data-stat-paywall-cta=\"breaker login cta\" rel=\"nofollow noopener\" target=\"_blank\">Log in<\/a><\/p>\n<p>\t\t\t<img decoding=\"async\" class=\"restricted-content-breaker-bg\" src=\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/04\/stat-plus-breaker-bg.png\" alt=\"STAT+\"\/><\/p>\n<p>\t\t\t\t\t<a href=\"https:\/\/www.statnews.com\/stat-plus\/\" rel=\"nofollow noopener\" target=\"_blank\"><br \/>\n\t\t\t\t\t\t<img decoding=\"async\" class=\"stat-plus-logo\" src=\"https:\/\/www.statnews.com\/wp-content\/themes\/stat\/images\/stat-plus-logo-white.svg\" alt=\"STAT+\"\/><br \/>\n\t\t\t\t\t<\/a><\/p>\n<p>\t\t\t\tThis article is exclusive to STAT+ subscribers<br \/>\n\t\t\t\tUnlock this article \u2014 and get additional analysis of the technologies disrupting health care \u2014 by subscribing to STAT+.<\/p>\n<p class=\"restricted-content-breaker-wrapper-cta mobile\">Already have an account? <a href=\"https:\/\/www.statnews.com\/login\/\" data-stat-login=\"\" data-stat-paywall-cta=\"breaker login cta mobile\" rel=\"nofollow noopener\" target=\"_blank\">Log in<\/a><\/p>\n<p>\t\t\t\t\t\tIndividual plans<\/p>\n<p>\t\t\t\t\t\tGroup plans<\/p>\n<p>\t\t\t\t<a href=\"https:\/\/www.statnews.com\/stat-plus\/\" class=\"stat-button view-all\" data-stat-paywall-cta=\"breaker view all plans\" aria-label=\"View All Plans\" rel=\"nofollow noopener\" target=\"_blank\">View All Plans<\/a><\/p>\n<p class=\"restricted-content-text\">\n\t\t\t\t\t\tTo read the rest of this story subscribe to STAT+.\t\t\t\t\t<\/p>\n<p>\t\t\t\t\t<a href=\"https:\/\/www.statnews.com\/stat-plus\/\" class=\"stat-button restricted-button\" data-stat-paywall-cta=\"breaker subscribe cta 1\" rel=\"nofollow noopener\" target=\"_blank\">Subscribe<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Katie covers the impact of health technology on patients, clinicians, and businesses. Her stories explore the price tag&hellip;\n","protected":false},"author":2,"featured_media":122533,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[24,25,459,460],"class_list":["post-122532","post","type-post","status-publish","format-standard","has-post-thumbnail","category-ai","tag-ai","tag-artificial-intelligence","tag-health-tech","tag-stat"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/122532","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=122532"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/122532\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/122533"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=122532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=122532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=122532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}