{"id":894345,"date":"2026-06-26T14:57:28","date_gmt":"2026-06-26T14:57:28","guid":{"rendered":"https:\/\/www.europesays.com\/us\/894345\/"},"modified":"2026-06-26T14:57:28","modified_gmt":"2026-06-26T14:57:28","slug":"evaluating-the-robustness-and-readiness-of-large-frontier-models-in-health-ai-applications","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/us\/894345\/","title":{"rendered":"Evaluating the robustness and readiness of large frontier models in health AI applications"},"content":{"rendered":"
  • \n

    Singhal, K. et al. Large language models encode clinical knowledge. Nature 620<\/b>, 172\u2013180 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. In ACM Transactions on Computing for Healthcare (HEALTH) (eds Lee, I. & Stankovic, J. A.) 3<\/b>, 1\u221223 (Association for Computing Machinery, 2022).<\/p>\n<\/li>\n

  • \n

    Nori, H. et al. Sequential diagnosis with language models. Preprint at https:\/\/arxiv.org\/abs\/2506.22405<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    OpenAI. Introducing GPT-5. https:\/\/openai.com\/index\/introducing-gpt-5\/<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Saab, K. et al. Capabilities of Gemini models in medicine. Preprint at https:\/\/arxiv.org\/abs\/2404.18416<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Tu, T. et al. Towards conversational diagnostic AI. Preprint at https:\/\/arxiv.org\/abs\/2401.05654<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Wang, S. et al. LINS: a general medical Q&A framework for enhancing the quality and credibility of LLM-generated responses. Nat. Commun. 16<\/b>, 9076 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. Preprint at https:\/\/arxiv.org\/abs\/2505.08775<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Handler, R., Sharma, S. & Hernandez-Boussard, T. The fragile intelligence of GPT-5 in medicine. Nat. Med. 31<\/b>, 3968\u20133970 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630<\/b>, 625\u2013630 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. NPJ Digit. Med. 7<\/b>, 190 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Pfau, J., Merrill, W. & Bowman, S. R. Let\u2019s think dot by dot: hidden computation in transformer language models. In First Conference on Language Modeling (COLM) https:\/\/openreview.net\/forum?id=NikbrdtYvG<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2<\/b>, 665\u2013673 (2020).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical AI. Nat. Med. 28<\/b>, 1773\u20131784 (2022).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Goodfellow, I. J., Shlens, J. & Szegedy, C. Explaining and harnessing adversarial examples. Preprint at https:\/\/arxiv.org\/abs\/1412.6572<\/a> (2015).<\/p>\n<\/li>\n

  • \n

    Szegedy, C. et al. Intriguing properties of neural networks. Preprint at https:\/\/arxiv.org\/abs\/1312.6199<\/a> (2013).<\/p>\n<\/li>\n

  • \n

    The New England Journal of Medicine: Image Challenge. https:\/\/www.nejm.org\/image-challenge<\/a> (2026).<\/p>\n<\/li>\n

  • \n

    JAMA Network Clinical Challenge. https:\/\/jamanetwork.com\/collections\/44038\/clinical-challenge<\/a> (2026).<\/p>\n<\/li>\n

  • \n

    Comanici, G. et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint at https:\/\/arxiv.org\/abs\/2507.06261<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Anthropic. Claude 3.5 Sonnet. https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    OpenAI. GPT-4o system card. https:\/\/openai.com\/index\/gpt-4o-system-card\/<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    OpenAI. OpenAI o3 and o4-mini system card. https:\/\/openai.com\/index\/o3-o4-mini-system-card\/<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In NIPS\u02bc22: Proceedings of the 36th International Conference on Neural Information Processing Systems 24824\u221224837 (eds Koyejo, S. et al.) (Curran Associates, 2022).<\/p>\n<\/li>\n

  • \n

    Lau, J. J., Gayen, S., Ben Abacha, A. & Demner-Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Sci. Data 5<\/b>, 180251 (2018).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Hu, Y. et al. OmniMedVQA: a new large-scale comprehensive evaluation benchmark for medical LVLM. In 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https:\/\/doi.org\/10.1109\/CVPR52733.2024.02093<\/a> (IEEE, 2024).<\/p>\n<\/li>\n

  • \n

    Johnson, A. E. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data 6<\/b>, 317 (2019).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. PathVQA: 30000+ questions for medical visual question answering. Preprint at https:\/\/arxiv.org\/abs\/2003.10286<\/a> (2020).<\/p>\n<\/li>\n

  • \n

    Liu, B. et al. SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. Preprint at https:\/\/arxiv.org\/abs\/2102.09542<\/a> (2021).<\/p>\n<\/li>\n

  • \n

    Zhang, X. et al. PMC-VQA: visual instruction tuning for medical visual question answering. Preprint at https:\/\/arxiv.org\/abs\/2305.10415<\/a> (2023).<\/p>\n<\/li>\n

  • \n

    Yue, X. et al. MMMU: a massive multidiscipline multimodal understanding and reasoning benchmark for expert AGI. In 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https:\/\/doi.org\/10.1109\/CVPR52733.2024.00913<\/a> (IEEE, 2024).<\/p>\n<\/li>\n

  • \n

    Fleiss, J. L. Measuring nominal scale agreement among many raters. Psychol. Bull. 76<\/b>, 378\u2013382 (1971).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Wu, Z. et al. DeepSeek-VL2: mixture-of-experts vision-language models for advanced multimodal understanding. Preprint at https:\/\/arxiv.org\/abs\/2412.10302<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Bai, S. et al. Qwen3-VL technical report. Preprint at https:\/\/arxiv.org\/abs\/2511.21631<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Li, C. et al. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. In NIPS \u02bc23: Proceedings of the 37th International Conference on Neural Information Processing Systems (eds Oh, A. et al.) 28541\u221228564 (Curran Associates, 2023).<\/p>\n<\/li>\n

  • \n

    Sellergren, A. et al. MedGemma technical report. Preprint at https:\/\/arxiv.org\/abs\/2507.05201<\/a> (2025).<\/p>\n<\/li>\n","protected":false},"excerpt":{"rendered":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023). Article\u00a0 CAS\u00a0 PubMed\u00a0 PubMed…\n","protected":false},"author":3,"featured_media":894346,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_share_on_mastodon":"0"},"categories":[35],"tags":[15576,6958,834,210,1141,1142,6454,3740,17781,913,911,67,132,68],"class_list":["post-894345","post","type-post","status-publish","format-standard","has-post-thumbnail","category-health-care","tag-biomedicine","tag-cancer-research","tag-general","tag-health","tag-health-care","tag-healthcare","tag-infectious-diseases","tag-medical-research","tag-metabolic-diseases","tag-molecular-medicine","tag-neurosciences","tag-united-states","tag-unitedstates","tag-us"],"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/116816999044784070","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/894345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=894345"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/894345\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/894346"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=894345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=894345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=894345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}