![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/figure1_AImodel.png\" "\\"figure1_AImodel\\"")
<\/p>\nBy Daniel Rongo, MD; Scott Ryals, MD; Mattina A. Davenport, PhD; and Trung Le, PhD, on behalf of the AASM Artificial Intelligence in Sleep Medicine Committee <\/p>\n
As artificial intelligence (AI) becomes integrated into the clinical workflow of sleep medicine, the clinician\u2019s role is evolving. Clinicians should understand how AI models function and decide whether they perform as intended. Advanced expertise in computer science is not required to advocate for our patients. A working knowledge of the AI lifecycle, performance metrics, and common statistical pitfalls is sufficient to prevent being misled by impressive-looking numbers. These skills help to ensure AI improves care as the field of sleep medicine enters a new era.<\/p>\n
The AI model lifecycle<\/p>\n
Every AI model begins with defining a problem or objective. Relevant data are collected and preprocessed so that the model can learn patterns and predict outcomes (i.e., supervised learning). The model is then evaluated on separate test data not used during training (i.e., for testing\/validating data). Demonstrating generalizability across diverse patient populations, institutions, and clinical settings is essential before deployment. Deployment is not the endpoint. Continuous monitoring is required to detect performance drift, bias, and unintended clinical consequences. One loop of this iterative life cycle is demonstrated in Figure 1. Ongoing oversight by scientists, administrators, and clinicians is necessary to maintain AI model safety and usefulness.<\/p>\n
<\/p>\n
Figure 1: The AI model lifecycle<\/p>\n
Managing metrics of performance<\/p>\n
AI models optimize predictions based on a chosen objective. The challenge is deciding which performance metrics matter most clinically. Consider predicting a rare disease (e.g., narcolepsy) with a prevalence of 1%. Accuracy alone is meaningless in rare conditions. In this setting, minimizing false negatives is critical. The preferred metric becomes recall (sensitivity): the ability to detect true cases. Model training involves optimizing toward specific performance targets. However, optimizing one metric introduces tradeoffs. A model tuned for recall may increase false positives. Selecting the correct metric requires aligning performance goals with clinical risk.<\/p>\n
A major concern is overfitting, when a model memorizes noise in the training patient data and fails on new patients. True validation requires testing on unseen, real-world data. Strong performance during development does not guarantee safe clinical performance.<\/p>\n
Different metrics for different models<\/p>\n
No single metric applies to every task. The appropriate metric depends on the clinical question. Table 1 summarizes common sleep medicine applications and the metrics most relevant to each. These are not exhaustive but provide a framework for evaluating claims about AI model performance.<\/p>\n
Table 1:\u00a0Examples of tasks\u00a0for sleep medicine and the metrics that matter\u00a0<\/p>\n
Type of Model
\nExample (Sleep Medicine)
\nMetrics<\/p>\n
Classification
\nPredicting epoch sleep stage or CPAP noncompliance
\nAccuracy, precision, recall, F1 score, area under the curve of a receiver operating characteristic (AUC ROC), Cohen\u2019s kappa, Matthews correlation coefficient (MCC), calibration<\/p>\n
Regression
\nPredicting apnea\u2013hypopnea index (AHI) or total sleep time from physiologic or wearable data
\nMean absolute error (MAE), root mean square error (RMSE), R2, Pearson r, Bland\u2013Altman bias<\/p>\n
Time-series \/ signal processing
\nDetecting respiratory events or sleep stages (K-complex, sleep spindle, frequencies) from raw EEG\/airflow signals using Fourier transform, wavelet transform, recurrent neural networks (RNNs), or transformers
\nEvent-level sensitivity\/specificity, AUC ROC, Cohen\u2019s kappa vs. expert scoring, correlation coefficients, per-epoch accuracy<\/p>\n
Causal inference models
\nEstimating the effect of PAP therapy on daytime sleepiness after controlling confounders
\nAverage treatment effect (ATE), conditional ATE (CATE), overlap diagnostics, balance metrics<\/p>\n
Generative AI \/ large language models
\nAI scribe systems for note writing
\nHallucination rate, omission rate, attribution error rate, critical error rate, note acceptance rate, median time saved per note<\/p>\n
Classification tasks, such as sleep staging or predicting CPAP adherence, are especially common in clinical decision-making. Table 2 describes metrics that appear frequently and deserve careful interpretation.<\/p>\n
Table 2: Classification metrics, definitions, and considerations\/pitfalls<\/p>\n
Metric
\nDefinitions
\nConsiderations\/Pitfalls<\/p>\n
Accuracy
\nProportion of all predictions that are correct
\nCan appear high when the model predicts the most common class (e.g., N2 sleep). Misleading in imbalanced datasets and should never be interpreted alone.<\/p>\n
Sensitivity \/ Recall (R)
\nTrue positive rate; proportion of actual cases correctly identified
\nCritical for patient safety when missing disease has high consequences (e.g., OSA). A model can have high accuracy but dangerously low recall.<\/p>\n
Precision (P) \/ PPV
\nProportion of predicted positives that are truly positive
\nOptimizing precision alone can underdiagnose disease. Low precision increases false positives and strains patient and clinician resources.<\/p>\n
F1-score
\nHarmonic mean of precision and recall
\nSummary metric balancing false positives and false negatives. Useful as a secondary score but can mask poor recall in rare conditions.<\/p>\n
Matthews Correlation Coefficient (MCC)
\nBalanced measure using all four confusion matrix elements (TP, TN, FP, FN*)
\nStrong metric for imbalanced classes and event detection. Often more informative than accuracy but less familiar to clinicians.<\/p>\n
Cohen\u2019s kappa (\u03ba)<\/p>\n
(Po \u2212 Pe) \/ (1 \u2212 Pe)<\/p>\n
Agreement between model and human scoring beyond chance**<\/p>\n
Useful for comparing AI with expert PSG scoring. Reflects clinical agreement, not overall model optimization. Should be interpreted alongside other metrics.<\/p>\n
Area under the curve of a receiver operating characteristic (AUC ROC)
\nAbility to discriminate across decision thresholds
\nCan appear high while still missing clinically important cases. Must be interpreted with recall and class balance.<\/p>\n
Area under the precision-recall curve (AUC-PR)
\nPrecision-recall performance emphasizing the positive class
\nMore sensitive to rare events than AUC ROC. Drops sharply with poor recall or precision; better reflects performance in imbalanced sleep datasets.<\/p>\n
*TP = true positive, TN = true negative, FP = false positive, FN = false negative<\/p>\n
**Po = observed agreement = accuracy (for the same unit), Pe = expected agreement by chance based on label marginals<\/p>\n
Conclusion<\/p>\n
AI models will increasingly shape how sleep clinicians diagnose, prognosticate, and document care. AI model performance depends on the objectives and the data used to train it. Metrics must be interpreted in the context of clinical risk to avoid bias, overdiagnosis, and missed treatment. Healthy skepticism entails aligning the highlighted metric with the clinical stakes. Clinicians do not need to become data scientists, but they must ask practical questions:<\/p>\n
What problem is the AI model solving?
\nWhich errors are most harmful?
\nAre development and deployment metrics transparent?<\/p>\n
In sleep medicine, clinicians act as informed gatekeepers rather than passive adopters.<\/p>\n
###<\/p>\n
Further thoughts<\/p>\n
Future\u00a0priorities\u00a0in\u00a0sleep\u00a0medicine\u00a0should include\u00a0promoting\u00a0the use of\u00a0interpretable AI model systems,\u00a0maintaining\u00a0oversight\u00a0throughout\u00a0the\u00a0AI\u00a0model lifecycle, and\u00a0adopting standardized\u00a0reporting\u00a0checklists.\u00a0These practices support fairness, reduce bias, and improve trust in clinical AI models.\u00a0<\/p>\n
This article appeared in volume 11, issue\u00a02\u00a0of\u202fMontage magazine<\/a>.\u202f\u00a0<\/p>\n References<\/p>\n Kocak B, Klontzas ME, Stanzione A, Meddeb A, Demircio\u011flu A, Bluethgen C, Bressem KK, Ugga L, Mercaldo N, D\u00edaz O, Cuocolo R. Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations. Eur J Radiol Artif Intell. 2025;3:100030. https:\/\/doi.org\/10.1016\/j.ejrai.2025.100030<\/a><\/p>\n Further Reading<\/p>\n Bandyopadhyay A, Bae C, Cheng H, et al. Smart sleep: what to consider when adopting AI-enabled solutions in clinical practice of sleep medicine.\u202fJ Clin Sleep Med. 2023;19(10):1823-1833. https:\/\/doi.org\/10.5664\/jcsm.10702<\/a>
\nBandyopadhyay A, Oks M, Sun H, et al. Strengths, weaknesses, opportunities, and threats of using AI-enabled technology in sleep medicine: a commentary.\u202fJ Clin Sleep Med. 2024;20(7):1183-1191. https:\/\/doi.org\/10.5664\/jcsm.11132<\/a>
\nGoldstein CA, Berry RB, Kent DT, et al. Artificial intelligence in sleep medicine: background and implications for clinicians.\u202fJ Clin Sleep Med. 2020;16(4):609-618. https:\/\/doi.org\/10.5664\/jcsm.8388<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"By Daniel Rongo, MD; Scott Ryals, MD; Mattina A. Davenport, PhD; and Trung Le, PhD, on behalf of…\n","protected":false},"author":2,"featured_media":61523,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[35214,35217,35218,24,35225,35219,25,18206,35228,35229,2537,35221,35215,35227,4360,35216,35220,35226,35224,35212,35222,35211,35223,35213],"class_list":["post-61522","post","type-post","status-publish","format-standard","has-post-thumbnail","category-ai","tag-aasm","tag-accreditation","tag-accredited","tag-ai","tag-american-academy","tag-apnea","tag-artificial-intelligence","tag-association","tag-circadian-rhythym","tag-disorder","tag-doctor","tag-insomnia","tag-membership","tag-mslt","tag-sleep","tag-sleep-apnea","tag-sleep-association","tag-sleep-deprivation","tag-sleep-disorders","tag-sleep-doctor","tag-sleep-lab","tag-sleep-medicine","tag-sleep-society","tag-technologist"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/61522","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=61522"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/61522\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/61523"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=61522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=61522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=61522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}