{"id":294451,"date":"2026-01-20T19:30:11","date_gmt":"2026-01-20T19:30:11","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/294451\/"},"modified":"2026-01-20T19:30:11","modified_gmt":"2026-01-20T19:30:11","slug":"holistic-evaluation-of-large-language-models-for-medical-tasks-with-medhelm","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/294451\/","title":{"rendered":"Holistic evaluation of large language models for medical tasks with MedHELM"},"content":{"rendered":"
  • \n

    Papers with Code. Question answering on MedQA (USMLE). https:\/\/paperswithcode.com\/sota\/question-answering-on-medqa-usmle<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Khosravi, M., Zare, Z., Mojtabaeian, S. M. & Izadi, R. Artificial intelligence and decision-making in healthcare: a thematic analysis of a systematic review of reviews. Health Serv. Res. Manag. Epidemiol. 11<\/b>, 23333928241234863 (2024).<\/p>\n

    PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Nath, D. Artificial intelligence (AI) will transform the clinical workflow with the next-generation technology. HealthTech Magazines https:\/\/www.healthtechmagazines.com\/artificial-intelligence-ai-will-transform-the-clinical-workflow-with-the-next-generation-technology\/<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Carl, N. et al. Evaluating interactions of patients with large language models for medical information. BJU Int. 135<\/b>, 1010\u20131017 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at https:\/\/arxiv.org\/abs\/2303.13375<\/a> (2023).<\/p>\n<\/li>\n

  • \n

    Raji, I. D., Daneshjou, R. & Alsentzer, E. It\u2019s time to bench the medical exam benchmark. NEJM AI 2<\/b>, AIe2401235 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning 174<\/b>, 248\u2013260 (PMLR, 2022).<\/p>\n<\/li>\n

  • \n

    Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333<\/b>, 319\u2013328 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30<\/b>, 2613\u20132622 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. https:\/\/cdn.openai.com\/pdf\/bd7a39d5-9e9f-47b3-903c-8b847ca650c7\/healthbench_paper.pdf<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Liang, P. et al. Holistic evaluation of language models. In Transactions on Machine Learning https:\/\/openreview.net\/pdf?id=iO4LZibEqW<\/a> (2023).<\/p>\n<\/li>\n

  • \n

    Leaderboard overview. LM Arena https:\/\/lmarena.ai\/leaderboard<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Wu, J. et al. BRIDGE: benchmarking large language models for understanding real-world clinical practice text. Preprint at https:\/\/arxiv.org\/abs\/2504.19467<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Fries, J. A. et al. BigBio: a framework for data-centric biomedical natural language processing. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=8lQDn9zTQlW<\/a> (2022).<\/p>\n<\/li>\n

  • \n

    Croxford, E. et al. Automating evaluation of AI text generation in healthcare with a large language model (LLM)-as-a-judge. Preprint at medRxiv https:\/\/doi.org\/10.1101\/2025.04.22.25326219<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11<\/b>, 6421 (2021).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Wornow, M. et al. Context clues: evaluating long context models for clinical prediction tasks on EHRs. In Proc. 13th International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=zg3ec1TdAP<\/a> (ICLR, 2025).<\/p>\n<\/li>\n

  • \n

    Liu, F. et al. Large language models in the clinic: a comprehensive benchmark. Preprint at https:\/\/arxiv.org\/abs\/2405.00716<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Wu, C. et al. Towards evaluating and building versatile large language models for medicine. NPJ Digit. Med. 8<\/b>, 58 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Ouyang, Z. et al. CliMedBench: a large-scale Chinese benchmark for evaluating medical large language models in clinical scenarios. In Proc. 2024 Conference on Empircal Methods in Natural Language Processing 8428\u20138438 (EMNLP, 2024).<\/p>\n<\/li>\n

  • \n

    Singhal, K. et al. Large language models encode clinical knowledge. Nature 620<\/b>, 172\u2013180 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31<\/b>, 2546\u20132549 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Cai, Y. et al. MedBench: a large-scale Chinese benchmark for evaluating medical large language models. In Proc. 38th AAAI Conference on Artificial Intelligence 38<\/b>, 17709\u201317717 (AAAI, 2024).<\/p>\n<\/li>\n

  • \n

    Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. Conference on Computational Natural Language Learning (CoNLL) 314\u2013334 (CoNLL, 2023).<\/p>\n<\/li>\n

  • \n

    Han, T., Kumar, A., Agarwal, C. & Lakkaraju, H. MedSafetyBench: evaluating and improving the medical safety of large language models. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=cFyagd2Yh4<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Liu, F. et al. Application of large language models in medicine. Nat. Rev. Bioeng. 3<\/b>, 85\u2013104 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Magar, I. & Schwartz, R. Data contamination: from memorization to exploitation. In Proc. 60th Annual Meeting of the Association of Computatonal Linguistics (Vol. 2: Short Papers) 157\u2013165 (ACL, 2022).<\/p>\n<\/li>\n

  • \n

    Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 248\u2013255 (IEEE, 2009).<\/p>\n<\/li>\n

  • \n

    Gu, J. et al. A survey on LLM-as-a-judge. Preprint at https:\/\/arxiv.org\/abs\/2411.15594<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Madaan, L. et al. Quantifying variance in evaluation benchmarks. Preprint at https:\/\/arxiv.org\/abs\/2406.10229<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Manakul, P., Liusie, A. & Gales, M. J. F. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proc. 2023 Conference on Empirical Methods Natural Language Processing 9004\u20139017 (EMNLP, 2023).<\/p>\n<\/li>\n

  • \n

    Guha, B. Secret ballots and costly information gathering: the jury size problem revisited. MPRA Paper no. 73048 (2016).<\/p>\n<\/li>\n

  • \n

    Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30<\/b>, 1134\u20131142 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Khandekar, N. et al. MedCalc-Bench: evaluating large language models for medical calculations. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=VXohja0vrQ<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    MT Samples: collection of transcribed medical transcription sample reports and examples. MT Samples https:\/\/www.mtsamples.com\/<\/a> (2023).<\/p>\n<\/li>\n

  • \n

    Ben Abacha, A. et al. MEDEC: a benchmark for medical error detection and correction in clinical notes. in Findings of the Association for Computational Linguistics 22539\u201322550 (ACL, 2025).<\/p>\n<\/li>\n

  • \n

    Vilares, D. & G\u00f3mez-Rodr\u00edguez, C. HEAD-QA: a healthcare dataset for complex reasoning. In Proc. 57th Annual Meeting of the Association of Computational Linguistics 960\u2013966 (ACL, 2019).<\/p>\n<\/li>\n

  • \n

    Chen, H., Fang, Z., Singla, Y. & Dredze, M. Benchmarking large language models on answering and explaining challenging medical questions. In Proc. 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) 3563\u20133599 (NAACL, 2025).<\/p>\n<\/li>\n

  • \n

    Yim, W. -W. et al. ACI-BENCH: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data 10<\/b>, 586 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Ben Abacha, A. et al. Bridging the gap between consumers\u2019 medication questions and trusted answers. In MEDINFO 2019: Health and Wellbeing e-Networks for All 25\u201329 (IOS Press, 2019).<\/p>\n<\/li>\n

  • \n

    Zeng, G. et al. MedDialog: large-scale medical dialogue datasets. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (eds Webber, B. et al.) https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.743<\/a> (Association for Computational Linguistics, 2020).<\/p>\n<\/li>\n

  • \n

    Abacha, A. B., Shivade, C. & Demner-Fushman, D. Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proc. 18th BioNLP Workshop Shared Task 16\u201325 (ACL, 2019).<\/p>\n<\/li>\n

  • \n

    Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2567\u20132577 (EMNLP, 2019).<\/p>\n<\/li>\n

  • \n

    Lee, G. et al. EHRSQL: a practical text-to-SQL benchmark for electronic health records. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=B2W8Vy0rarw<\/a> (NeurIPS, 2022).<\/p>\n<\/li>\n

  • \n

    Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6<\/b>, 195 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Pandit, S. et al. MedHallu: a comprehensive benchmark for detecting medical hallucinations in large language models. Preprint at https:\/\/arxiv.org\/abs\/2502.14302<\/a> (2025).<\/p>\n<\/li>\n

  • \n

    Wornow, M., Thapa, R., Steinberg, E., Fries, J. A. & Shah, N. H. EHRSHOT: an EHR benchmark for few-shot evaluation of foundation models. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=CsXC6IcdwI<\/a> (2023).<\/p>\n<\/li>\n

  • \n

    Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. In Proc. Thirty-Eighth AAAI Conf. Artif. Intell. 38<\/b>, 21545\u201321555 (AAAI, 2024).<\/p>\n<\/li>\n

  • \n

    Xu, J. Discharge me: BioNLP ACL\u201924 shared task on streamlining discharge documentation (version 1.3). PhysioNet https:\/\/doi.org\/10.13026\/0zf5-fx50<\/a> (2024).<\/p>\n<\/li>\n

  • \n

    Chen, Z., Varma, M., Wan, X., Langlotz, C. & Delbrouck, J.-B. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. In Proc. 61st Annual Meeting of the Association of Computational Linguistics (Vol. 2: Short Papers) 469\u2013484 (ACL, 2023).<\/p>\n<\/li>\n

  • \n

    Aali, A. et al. A dataset and benchmark for hospital course summarization with adapted large language models. J. Am. Med. Inform. Assoc. 32<\/b>, 470\u2013479 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Henry, S., Buchan, K., Filannino, M., Stubbs, A. & Uzuner, O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27<\/b>, 3\u201312 (2020).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Edin, J. et al. Automated medical coding on MIMIC-III and MIMIC-IV: a critical review and replicability study. In Proc. 46th Int. ACM SIGIR Conf. Research and Development in Information Retrieval 2572\u20132582 (SIGIR, 2023).<\/p>\n<\/li>\n

  • \n

    Lopez, I. et al. Clinical entity augmented retrieval for clinical information extraction. NPJ Digit. Med. 8<\/b>, 45 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Pillai, M., Posada, J., Gardner, R. M., Hernandez-Boussard, T. & Bannett, Y. Measuring quality-of-care in treatment of young children with attention deficit\/hyperactivity disorder using pre-trained language models. J. Am. Med. Inform. Assoc. 31<\/b>, 949\u2013957 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Bannett, Y. et al. Applying large language models to assess quality of care: monitoring ADHD medication side effects. Pediatrics 155<\/b>, e2024067223 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Rabbani, N. et al. Evaluation of a large language model to identify confidential content in adolescent encounter notes. JAMA Pediatr. 178<\/b>, 308\u2013310 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Tse, G. et al. Large language model responses to adolescent patient and proxy messages. JAMA Pediatr. 179<\/b>, 93\u201394 (2025).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n","protected":false},"excerpt":{"rendered":"Papers with Code. Question answering on MedQA (USMLE). https:\/\/paperswithcode.com\/sota\/question-answering-on-medqa-usmle (2024). Khosravi, M., Zare, Z., Mojtabaeian, S. M. &…\n","protected":false},"author":2,"featured_media":294452,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[275],"tags":[2564,2566,18,910,135,475,474,19,4381,17,610,19880,10047,5801],"class_list":{"0":"post-294451","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-healthcare","8":"tag-biomedicine","9":"tag-cancer-research","10":"tag-eire","11":"tag-general","12":"tag-health","13":"tag-health-care","14":"tag-healthcare","15":"tag-ie","16":"tag-infectious-diseases","17":"tag-ireland","18":"tag-machine-learning","19":"tag-metabolic-diseases","20":"tag-molecular-medicine","21":"tag-neurosciences"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@ie\/115929089476294005","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/294451","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/comments?post=294451"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/294451\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media\/294452"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media?parent=294451"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/categories?post=294451"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/tags?post=294451"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}