Ethics statement
This research complies with all relevant ethical regulations. The study protocol received ethical approval from the Institutional Review Boards of Mass General Brigham (MGB) and Dana-Farber (DF)/Harvard Cancer Center (HCC; MGB 2022P002195, DF/HCC 24-735 and DF/HCC 20-328). A waiver of informed consent was granted by MGB and DF/HCC for the secondary use components, as the research involved minimal risk and did not alter any clinical care. Electronic informed consent was obtained for the HCI studies.
Note-level irAE system development and evaluation (phase 1)
For the data source and study population, we assembled a retrospective cohort of patients treated with ICIs at an academic medical center in Boston, Massachusetts, between 2015 and 2024. The inclusion criteria were: (1) age ≥18 years; (2) treatment with one or more of the following Food and Drug Administration-approved ICIs: tremelimumab, relatlimab, retifanlimab, pembrolizumab, nivolumab, atezolizumab, durvalumab, avelumab, cemiplimab, toripalimab and dostarlimab; and (3) the presence of at least one progress note or visit note in the clinical system. The number, age and sex of the patients whose notes were included in each phase are presented in Supplementary Table 5; sex was recorded as documented in the EHR.
To establish the gold-standard annotation procedure, annotation guidelines were developed to support manual ground-truth labeling of the presence of current and past irAEs and their key clinical attributes (see annotation guidelines in the study protocol)31. We focused on capturing six irAEs, representing the six most common and/or lethal irAEs and selected to capture 70–80% of clinically-relevant irAEs: pneumonitis, hepatitis, myocarditis, dermatitis, thyroiditis and colitis30. Each irAE was mapped to a set of National Cancer Institute CTCAE terms following the definitions used in the Alliance for Clinical Trial in Oncology irAE biobank (NCT04242095; see annotation guidelines in the study protocol)31. For each irAE type and temporality (current versus past) combination, we labeled its binary presence or absence, CTCAE grade, attribution and certainty as described in a note. The clinical attributes are defined as follows. (1) Severity: severity grading followed the National Cancer Institute CTCAE version 5.0, a standardized system commonly used for regulatory reporting. The values comprise grade 1, 2, 3, 4 and 5. (2) Attribution: attribution captured the clinician’s explicitly documented assessment of causality between the adverse event and ICI therapy, recognizing that many symptoms in patients with cancer have multiple potential etiologies. The attributes comprise positive and negative. (3) Certainty: certainty reflected the degree of diagnostic confidence expressed in the documentation, guided in part by the Naranjo scale but adjusted for note-level, text-based determination39. The attributes comprise none, unlikely, possible, probable and definite.
Pilot rounds of 10–20 notes were dual-annotated by a physician and CRC with 4 years of oncology data abstraction experience, and annotation guidelines were iteratively updated for clarity and consistency until interannotator agreement exceeded 0.70 for each irAE type. All discrepancies were tracked and resolved through consensus discussion. Cases requiring adjudication were reviewed by an attending oncologist. The open-source Label Studio Community Platform v1.15 (HumanSignal), locally hosted behind our institution’s firewall, was used as our annotation platform.
A sample of notes from 100 patient encounters, forming a total of 289 retrospective patient notes, was annotated for retrospective evaluation (Supplementary Table 5). These notes were divided into a development set of 26 notes, all from pneumonitis-positive patients, and a test set of 263 notes, used to evaluate the final developed system.
For the development of the agentic LLM architecture, OpenAI API-based models were accessed using our institutional HIPAA-compliant application programming inference access points. Open-source models were downloaded locally, behind our institutional firewall, to a dedicated server equipped with two NVIDIA RTX 6000 Ada GPUs.
The architecture used an agentic design pattern, decomposing the complex task of irAE extraction into specialized subtasks handled by distinct model instances (Fig. 2). A preprocessing agent first extracted and formatted relevant clinical text, handling the varied structures and formats present in EHR documentation. Subsequently, specialized extraction agents focused on specific attributes: one agent determined temporal status by analyzing verb tenses and temporal markers, another mapped clinical descriptions to CTCAE grades, a third assessed attribution by identifying causal language and clinical reasoning, and a fourth evaluated the certainty of diagnosis on the basis of hedging language and diagnostic workup discussions. We refer to this as the ‘default’ agentic architecture hereafter. Prompt engineering was optimized on the development set, and only for pneumonitis, to assess the generalizability of our approach to other irAE types.
Given the potential variability in LLM outputs and the risk of hallucination, we also evaluated adding a self-consistency mechanism33,34,38,40. Here, each note was processed through three parallel inference runs with controlled randomness. A final judge agent then synthesized these parallel outputs, implementing a majority-vote mechanism for discrete categories and selecting the most coherent extraction when unanimity was absent.
Finally, on the basis of preparatory field study work, the models were designed to extract ‘string-matched’ snippets directly from clinical notes, as well as a brief rationale for the determinations, to support human interaction in real-world downstream tasks. The snippets serve as supporting evidence for the determinations made at each stage of the agentic workflow. This functionality was then integrated into the user interface as features to aid human reviewers in their annotation tasks.
As comparative baseline methods, to contextualize the performance of our agentic system, we implemented several baseline approaches representing different points on the complexity–performance spectrum. These included rule-based extraction using regular expressions expanded from a set of terms previously used in our institution to identify patients admitted for irAEs41 and zero-shot prompting. We compared open-source (Qwen-3-4B-Instruct, Qwen3-30B-A3B-Instruct-2507, Gemma-3-27B and GPT-OSS-20B) and proprietary OpenAI models (GPT-4.1-mini, GPT-4.1-nano and gpt-4o-mini) with medium reasoning effort on our institutional HIPAA-secure OpenAI API end point. Model selection was guided by the goal of representing a range of model sizes deployable on single- and dual-GPU institutional hardware for open-source candidates, alongside cost-constrained proprietary models available under Azure HIPAA-compliant agreements.
Precision, recall and macro-averaged F1 scores were calculated overall and stratified by irAE type across all available classes for a given label. The inference cost per note was also calculated for all variants.
Prospective silent validation (phase 2)
Following retrospective analysis, we implemented the best setting of our system in a silent, prospective, real-time clinical environment to assess performance under operational conditions and to assess generalizability to evolving clinical language and documentation practices. The system was integrated with our institution’s clinical data warehouse. Details of the integration infrastructure have been previously reported38. New physician, nursing, and physician assistant notes documented on patients treated with an ICI within the preceding year were automatically identified each morning, including progress notes, radiology reports, consultation reports, emergency department records and discharge summaries. Notes meeting criteria were immediately processed through the agentic system. The first 102 notes were dual-annotated by a board-certified oncologist and a trained research assistant with experience as a medical scribe and clinical research, achieving an interannotator agreement across irAE type and temporality (κ = 0.91 for binary detection, κ = 0.88 for grade). The remaining notes were single-annotated by the research assistant. This prospective phase lasted for 3 months (May–July 2025), during which we processed 884 real-time clinical notes. Agreement between prospective model predictions and human determinations was quantified using agreement percentages and override rates for each label and across all labels.
Preparatory field study
Before the randomized user effect study (phase 3), we conducted a preparatory field study to refine the annotation interface (Fig. 4), establish baseline efficiency and agreement metrics for power calculations and assess the temporal stability of human–AI collaboration. We developed a web application (React 18.2 frontend and FastAPI backend) integrating the agentic system into the annotation workflow and iteratively refined it through think-aloud sessions with three CRCs; key features included evidence highlighting within the note viewer, keyboard shortcuts and prepopulation of structured fields with model suggestions, with audit trails capturing interactions, timings and modifications. Two CRCs then used the application over 3 months to assist chart curation for a separate retrospective study, with annotation time, interannotator agreement and human–model agreement tracked over time and reported descriptively.
Across 299 dual-annotated notes over six sequential (approximately weekly) periods, inter-rater agreement on grade was substantial overall (Cohen’s κ = 0.80) and largely stable over time (a transient dip to κ = 0.55 for notes 101–150 rebounded to κ = 0.71–0.93), and agreement with the model on grade was high and stable (mean κ = 0.89). Time per note averaged 161 s versus an estimated 600-s unaided baseline (~3.7× faster), with efficiency improving over time and between-annotator variability narrowing, consistent with a learning curve. These findings informed the phase 3 design: annotation time was selected as the primary efficiency end point; the observed effect sizes and variance informed the power calculation; evidence highlighting and keyboard navigation were retained; and a 1-h training session was established as sufficient. Field study summary statistics are presented in Supplementary Table 6.
Randomized user effect study (phase 3)
To quantify the clinical impact of agentic assistance on irAE curation, we conducted a 2 × 2 randomized crossover study comparing AI-assisted annotation to standard manual review (Extended Data Fig. 2). This design was informed by a preparatory field study that informed the interface design, characterized learning curves, established baseline inter-rater agreement and identified annotation efficiency as the primary driver of real-world workflow burden. This study was approved by the institutional review board (DF/HCC 20-328).
We enrolled CRCs and CRNs from our department; there were no exclusion criteria. Following informed consent, participants completed a 1-h standardized training session with annotated examples31 and an unobserved practice period using the annotation interface (Fig. 4). Participants were then randomized to one of four sequences in a crossover design that counterbalanced both intervention order (AI-assisted versus manual spreadsheet) and note set (set A versus set B). This structure was chosen to mitigate carryover effects from either the annotation method or note content, and enabling difference-in-differences estimation of treatment effects. A total of 17 participants completed the study. Participant characteristics, including professional role, years of oncology data-abstraction experience, attitudes toward AI and experience with AI are presented in Supplementary Table 3.
Fig. 4: Annotation interface for irAE chart curation.
a–c, Enlarged views from the same interface show a synthetic demonstration note with highlighted evidence (a), model predictions and supporting evidence (b) and structured annotation controls (c).
Clinical notes were drawn from the phase 1 retrospective corpus. Sets A and B were matched on note length and clinical complexity and number of irAE descriptions through manual review to ensure comparability across arms.
Sample-size planning was informed by the preparatory field study. After inflating the sample size to account for the within-participant correlation of 0.05, 15 participants completing 20 notes each (300 total notes) would provide 84.7% power to detect a standardized effect size of −0.43 (comparing AI assistance to manual) with two-sided α = 0.10. We targeted the enrollment of 17–20 participants to account for potential dropouts and missing observations.
The primary end point was annotation efficiency, measured as time in seconds from note opening to final submission, captured via screen recording. The secondary end points were: (1) accuracy, defined as an exact match with the gold standard for all current irAE grades within a note—a stringent metric requiring perfect concordance across all six conditions; and (2) interannotator agreement, assessed using Krippendorff’s α for ordinal grade labels.
For the primary efficiency end point, we used GEE with a Gaussian distribution, identity link, exchangeable working correlation and participant-clustered robust variance to estimate the difference in untransformed annotation time between conditions. Log-transformed time was analyzed as a sensitivity analysis; separate models adjusted for note set and sequence.
For complete-note accuracy (340 annotations from 17 participants; 170 per condition), we used a GEE with binomial outcomes to estimate the odds ratio of the exact gold-standard match comparing AI-assisted with manual annotation, with robust sandwich variance estimators. Exploratory label-level accuracy analyses used paired data from 16 participants: 312 of 320 note annotations could be linked to gold-standard records; 8 annotations without matching records were excluded. Point estimates pooled labels by condition. Normal-approximation 95% intervals summarized participant means and paired differences. Comparisons used exact two-sided paired sign tests, excluding zero differences. Further, interannotator reliability was assessed separately within each note set using Krippendorff’s α for current grade labels. Bootstrap resampling (10,000 iterations, resampling participants with replacement) generated 95% confidence intervals (CIs) for each method. Questionnaires assessed usability, perceived efficiency and AI attitudes31. Survey analyses included 17 complete poststudy respondents and excluded five records with missing poststudy responses. Paired System Usability Scale42 scores were compared with a two-sided paired t-test; efficiency and label-confidence ratings used two-sided Wilcoxon signed-rank tests with a normal approximation, average ranks for ties, zero differences excluded and no continuity correction. Differently worded pre-/post-AI-capability questions were summarized descriptively. Participants were compensated US$50 h−1.
Statistics and reproducibility
No statistical method was used to predetermine sample size for the retrospective evaluation (phase 1) or the prospective silent deployment (phase 2). The phase 1 sample size was informed by sample sizes in previous studies and the capacity for manual double annotation. For the randomized user effect study (phase 3), sample size was determined a priori by power analysis using pilot estimates from the preparatory field study, as described above.
Robustness was instead assessed in three ways: through the self-consistency mechanism, in which the detection and grading stages were run three times independently per note and reconciled by a judge agent; through evaluation of the same fixed system in three nonoverlapping settings (retrospective, prospective and randomized user effect); and through bootstrap resampling (10,000 iterations, resampling participants with replacement) for interval estimation of agreement statistics as described above.
Notes were not randomized in phases 1 and 2. In phase 3, participants were randomized by computer-generated allocation to one of four sequences in a 2 × 2 crossover design counterbalancing intervention order and note set. Data collection and analysis were not performed blinded to the conditions of the experiments. Collectively, 24 note-level observations across participants were excluded owing to missing video recordings.
Data distribution was assumed to be normal but this was not formally tested; the primary efficiency end point was additionally analyzed after log transformation as a sensitivity analysis, and nonparametric summaries (medians and IQRs) are reported alongside model-based estimates. All statistical tests were two-sided, and no adjustments were made for multiple comparisons. Statistical analyses used R 4.1.0, and the computational pipeline used Python 3.10. The annotation interface was built with React 18.2 and FastAPI. Final statistical reproduction used R 4.5.3 with geepack 1.3.13, irr 0.85 and boot 1.3-32. The accompanying analysis code records the full reproduction environment; Python dependency requirements are provided in the AEGIS repository (see the ‘Code availability’ section)43.
The age and sex of the patients whose notes were analyzed in phases 1 and 2 are presented in Supplementary Table 5. These characteristics were recorded as documented in the EHR. Age and sex were not considered in the study design. Age and sex were not collected for phase 3 clinical research staff participants.
Further information on the research design is available in the Nature Portfolio Reporting Summary linked to this Article.
Study protocols
The annotation guidelines developed and used for gold standard labeling of irAEs in clinical notes can be used to replicate labeling procedures used in this work. These guidelines, together with the training procedures and surveys used in the randomized user effects study, are available in the cited protocols.io collection31.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.