{"id":647626,"date":"2026-08-20T17:32:23","date_gmt":"2026-08-20T17:32:23","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/647626\/"},"modified":"2026-08-20T17:32:23","modified_gmt":"2026-08-20T17:32:23","slug":"prospective-evaluation-of-a-large-language-model-clinical-decision-support-system-in-the-emergency-department","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/647626\/","title":{"rendered":"Prospective evaluation of a large language model clinical decision support system in the emergency department"},"content":{"rendered":"
  • \n

    Hoot, N. R. & Aronsky, D. Systematic review of emergency department crowding: causes, effects, and solutions. Ann. Emerg. Med. 52<\/b>, 126\u2013136 (2008).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Morley, C., Unwin, M., Peterson, G. M., Stankovich, J. & Kinsman, L. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS ONE 13<\/b>, e0203316 (2018).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Voaklander, B. et al. Interventions to improve consultations in the emergency department: a systematic review. Acad. Emerg. Med. 29<\/b>, 1475\u20131495 (2022).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Brick, C. et al. The impact of consultation on length of stay in tertiary care emergency departments. Emerg. Med. J. 31<\/b>, 134\u2013138 (2014).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Piliuk, K. & Tomforde, S. Artificial intelligence in emergency medicine: a systematic literature review. Int. J. Med. Inform. 180<\/b>, 105274 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Grant, K., McParland, A., Mehta, S. & Ackery, A. D. Artificial intelligence in emergency medicine: surmountable barriers with revolutionary potential. Ann. Emerg. Med. 75<\/b>, 721\u2013726 (2020).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Nagendran, M. et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 368<\/b>, m689 (2020).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Plana, D. et al. Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw. Open 5<\/b>, e2233946 (2022).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Han, T. et al. Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review. Lancet Digit. Health 6<\/b>, e367\u2013e373 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Oikonomidi, T. et al. Applications of artificial intelligence for real-world evidence generation: a protocol for a living scoping review. BMJ Open 16<\/b>, e109725 (2026).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17<\/b>, 195 (2019).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Wiens, J. et al. Do no harm: a roadmap for responsible machine learning for health care. Nat. Med. 25<\/b>, 1337\u20131340 (2019).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Vasey, B. et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat. Med. 28<\/b>, 924\u2013933 (2022).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29<\/b>, 1930\u20131940 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Singhal, K. et al. Large language models encode clinical knowledge. Nature 620<\/b>, 172\u2013180 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619<\/b>, 357\u2013362 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Williams, C. Y. K., Miao, B. Y., Kornblith, A. E. & Butte, A. J. Evaluating the use of large language models to provide clinical recommendations in the emergency department. Nat. Commun. 15<\/b>, 8236 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30<\/b>, 2613\u20132622 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Preiksaitis, C. & Rose, C. The role of large language models in transforming emergency medicine: scoping review. JMIR Med. Inform. 12<\/b>, e53787 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Gilboy, N., Tanabe, T., Travers, D. & Rosenau, A. M.Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4. AHRQ Publication No. 12-0014 (AHRQ, 2011).<\/p>\n<\/li>\n

  • \n

    Greenhalgh, T. et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J. Med. Internet Res. 19<\/b>, e367 (2017).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    R Core Team R: A Language and Environment for Statistical Computing (R Core Team, 2024).<\/p>\n<\/li>\n

  • \n

    Ancker, J. S. et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med. Inform. Decis. Mak. 17<\/b>, 36 (2017).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Kawamoto, K., Houlihan, C. A., Balas, E. A. & Lobach, D. F. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330<\/b>, 765 (2005).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Roshanov, P. S. et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 346<\/b>, f657 (2013).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Proctor, E. K. et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm. Policy Ment. Health 38<\/b>, 65\u201376 (2011).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Angrist, J. D., Imbens, G. W. & Rubin, D. B. Identification of causal effects using instrumental variables. J. Am. Stat. Assoc. 91<\/b>, 444\u2013455 (1996).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Kwong, J. C. C., Nickel, G. C., Wang, S. C. Y. & Kvedar, J. C. Integrating artificial intelligence into healthcare systems: more than just the algorithm. npj Digit. Med. 7<\/b>, 52 (2024).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    McCambridge, J., Witton, J. & Elbourne, D. R. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J. Clin. Epidemiol. 67<\/b>, 267\u2013277 (2014).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Geskey, J. M., Geeting, G., West, C. & Hollenbeak, C. S. Improved physician consult response times in an academic emergency department after implementation of an institutional guideline. J. Emerg. Med. 44<\/b>, 999\u20131006 (2013).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Soong, C., High, S., Morgan, M. W. & Ovens, H. A novel approach to improving emergency department consultant response times. BMJ Qual. Saf. 22<\/b>, 299\u2013305 (2013).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    VanderWeele, T. J. & Ding, P. Sensitivity analysis in observational research: introducing the E-value. Ann. Intern. Med. 167<\/b>, 268\u2013274 (2017).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Donabedian, A. The quality of care: how can it be assessed?. JAMA 260<\/b>, 1743\u20131748 (1988).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Mant, J. Process versus outcome indicators in the assessment of quality of health care. Int. J. Qual. Health Care 13<\/b>, 475\u2013480 (2001).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Torgerson, D. J. Contamination in trials: is cluster randomisation the answer? BMJ 322<\/b>, 355\u2013357 (2001).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J. & Lilford, R. J. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ 350<\/b>, h391 (2015).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Lewis P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proc. 34th International Conference on Neural Information Processing Systems (NIPS ’20) (eds Larochelle, H. et al.) 9459\u20139474 (Curran, 2020).<\/p>\n<\/li>\n

  • \n

    Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25<\/b>, 44\u201356 (2019).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Ethics and Governance of Artificial Intelligence for Health: WHO Guidance (WHO, 2021).<\/p>\n<\/li>\n

  • \n

    Shin, S., Lee, S. H., Kim, D. H. & Kim, K. H. The impact of the improvement in internal medicine consultation process on emergency department length of stay. Am. J. Emerg. Med. 36<\/b>, 620\u2013624 (2018).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Ravi, A., Shochat, G., Wang, R. C. & Khanna, R. Improvements to emergency department length of stay and user satisfaction after implementation of an integrated consult order. J. Am. Coll. Emerg. Physicians Open 4<\/b>, e12922 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Austin, P. C. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat. Med. 28<\/b>, 3083\u20133107 (2009).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Greenland, S. An introduction to instrumental variables for epidemiologists. Int. J. Epidemiol. 29<\/b>, 722\u2013729 (2000).<\/p>\n

    Article<\/a>\u00a0
    \n     CAS<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Staiger, D. & Stock, J. H. Instrumental variables regression with weak instruments. Econometrica 65<\/b>, 557\u2013586 (1997).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Hodges, J. L. Jr & Lehmann, E. L. Estimates of location based on rank tests. Ann. Math. Stat. 34<\/b>, 598\u2013611 (1963).<\/p>\n

    Article<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Meunier, P. Y., Raynaud, C., Guimaraes, E., Gueyffier, F. & Letrilliart, L. Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review. Ann. Fam. Med. 21<\/b>, 57\u201369 (2023).<\/p>\n

    Article<\/a>\u00a0
    \n     PubMed<\/a>\u00a0
    \n     PubMed Central<\/a>\u00a0
    \n
    \n Google Scholar<\/a>\u00a0\n <\/p>\n<\/li>\n

  • \n

    Python v.3.13 (Python Software Foundation, 2024).<\/p>\n<\/li>\n

  • \n

    llironlibo. llironlibo\/SHAKED-analysis: v1.1.0. Zenodo https:\/\/doi.org\/10.5281\/zenodo.20736931<\/a> (2026).<\/p>\n<\/li>\n","protected":false},"excerpt":{"rendered":"Hoot, N. R. & Aronsky, D. Systematic review of emergency department crowding: causes, effects, and solutions. Ann. Emerg.…\n","protected":false},"author":2,"featured_media":647627,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_share_on_mastodon":"0"},"categories":[275],"tags":[2564,2566,18,910,135,475,474,19,4381,17,19880,10047,5801,152496,3473],"class_list":["post-647626","post","type-post","status-publish","format-standard","has-post-thumbnail","category-healthcare","tag-biomedicine","tag-cancer-research","tag-eire","tag-general","tag-health","tag-health-care","tag-healthcare","tag-ie","tag-infectious-diseases","tag-ireland","tag-metabolic-diseases","tag-molecular-medicine","tag-neurosciences","tag-preclinical-research","tag-translational-research"],"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@ie\/117129035357456911","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/647626","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/comments?post=647626"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/647626\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media\/647627"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media?parent=647626"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/categories?post=647626"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/tags?post=647626"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}