{"id":484744,"date":"2026-05-14T18:41:20","date_gmt":"2026-05-14T18:41:20","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/484744\/"},"modified":"2026-05-14T18:41:20","modified_gmt":"2026-05-14T18:41:20","slug":"advancing-conversational-diagnostic-ai-with-multimodal-reasoning","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/484744\/","title":{"rendered":"Advancing conversational diagnostic AI with multimodal reasoning"},"content":{"rendered":"

This work advances the diagnostic capabilities of the multimodal AMIE system4<\/a>,5<\/a> by integrating multimodal perception, enabling it to conduct diagnostic conversations that are more clinically realistic and that incorporate various forms of medical data beyond text.<\/p>\n

Our approach centers on a state-aware dialogue phase transition framework, which leverages the multimodal reasoning capabilities of Gemini 2.0 Flash29<\/a>. This framework guides multimodal AMIE through structured phases of history-taking, diagnosis and management and follow-up, dynamically adapting the conversation based on intermediate model outputs that reflect the evolving patient state and diagnostic hypotheses. To rigorously evaluate the system\u2019s performance during model development and in comparison to human clinicians, we employed a two-pronged evaluation strategy. First, we developed an automated evaluation pipeline involving perception tests on isolated medical artifacts and simulated dialogues assessed by an auto-rater across key clinical dimensions, such as diagnostic accuracy and information gathering. Second, we conducted an expert evaluation using an OSCE-style methodology to assess the capabilities of multimodal AMIE in realistic, simulated patient encounters involving multimodal data.<\/p>\n

Multimodal state-aware reasoning<\/p>\n

Real clinical diagnostic dialogues follow a structured yet flexible path. Clinicians methodically gather information, form potential diagnoses, strategically request and interpret further details (including multimodal data such as skin photographs or ECGs), continually update their assessment based on new evidence and eventually formulate a management plan. This process requires a clinician to adapt the line of questioning based on evolving hypotheses and uncertainty while ensuring that all critical information is considered30<\/a>.<\/p>\n

Given the rapid advancements in LLM capabilities and their increasing proficiency in following complex instructions, one might achieve considerable progress toward emulating this process using a sophisticated system prompt alone. However, we hypothesize that, for a safety-critical and highly dynamic task like multimodal diagnosis, building an explicit state-aware reasoning system layered on top of the LLM offers critical advantages. Such a system provides greater control over the dialogue flow, enables more reliable tracking of the diagnostic state and uncertainty, facilitates more deliberate integration of multimodal inputs and ultimately leads to higher-quality, more dependable clinical reasoning rather than relying solely on complex prompting (validated in the \u2018Automated evaluations of multimodal AMIE configurations\u2019 section).<\/p>\n

Therefore, the multimodal AMIE system implements this state-aware dialogue phase transition framework to manage the diagnostic process. This framework dynamically controls the progression of multimodal AMIE through three distinct phases: (1) history-taking, (2) diagnosis and management and (3) follow-up. Transitions between phases, and actions within each phase, are driven by intermediate model outputs representing the evolving patient state and diagnostic hypotheses (Fig. 6<\/a>). Crucially, each phase builds upon the accumulated dialogue history, which contextually incorporates various forms of patient data, including text, images (such as skin photographs or ECG tracings) and clinical documents (such as laboratory reports or prior consultation notes). This state-aware approach allows multimodal AMIE to emulate the structured yet adaptive reasoning process of clinicians (see examples in Supplementary Figs. 5<\/a> and 6<\/a>).<\/p>\n

Phase 1: history-taking\u2014building a comprehensive picture<\/p>\n

    \n
  1. \n 1.<\/p>\n

    Patient profile initialization:<\/b> A structured patient profile is initialized and dynamically updated throughout the interaction. This profile acts as a condensed, evolving record of known patient information, encompassing the chief complaint; history of present illness; demographics (age, sex and race); positive and negative symptoms; past medical, family and social\/travel histories; medications; other relevant details; and a prioritized list of knowledge gaps. Initially, this profile may contain minimal information.<\/p>\n<\/li>\n

  2. \n 2.<\/p>\n

    Evolving DDx generation:<\/b> An internal, evolving DDx is generated. This DDx is not initially presented to the patient. DDx generation begins after an initial interaction period, allowing baseline information collection. The frequency of DDx updates is configurable.<\/p>\n<\/li>\n

  3. \n 3.<\/p>\n

    Continuation decision:<\/b> A key decision point is whether to continue gathering history or transition to presenting a diagnosis. This decision uses a set of criteria, including assessing if sufficient information exists to formulate a reasonable differential diagnosis. A decision module, querying Gemini 2.0 Flash, determines if current information is sufficient to proceed or if more targeted questions are needed. This module considers the dialogue history and preliminary DDx.<\/p>\n<\/li>\n

  4. \n 4.<\/p>\n

    Targeted question and multimodal data request generation:<\/b> If history-taking continues, the system generates focused questions to address information gaps identified in the patient profile and DDx uncertainty. It is important to note that this internal measure of uncertainty, derived from the model\u2019s iterative DDx generation, is used as a pragmatic heuristic to guide the history-taking dialogue toward more relevant lines of questioning. It is not presented to the user nor is it a formally calibrated diagnostic probability intended for direct clinical interpretation. Crucially, the system is designed to recognize when multimodal data are necessary and to strategically request them. For instance, based on reported symptoms such as a rash, multimodal AMIE will prompt the user to upload skin photographs. Its reasoning extends to requesting additional views if needed (for example, \u2018Could you please provide a photo from a different angle or in better lighting?\u2019 or \u2018Do you have photos showing how the rash looked previously?\u2019). Similarly, for reported cardiac symptoms, it might request an ECG tracing if available. Upon receiving an artifact, multimodal AMIE elicits detailed descriptions adhering to instructions for interpreting the artifacts and their key aspects for determining salient findings (for example, for skin: lesion morphology, distribution and color; for ECGs: heart rate, rhythm, key waveforms and intervals). This explicit prompting for descriptive details ensures that the model extracts salient features from the artifact to inform the ongoing conversation and diagnostic reasoning. The generated question or request is presented to the patient.<\/p>\n<\/li>\n

  5. \n 5.<\/p>\n

    Iterative refinement:<\/b> The process is iterative. New information from patient responses and data uploads is incorporated into the dialogue history, updating the patient profile and internal DDx. The patient summary is periodically refreshed to reflect the current understanding.<\/p>\n<\/li>\n<\/ol>\n

    Once the continuation decision indicates readiness, the system transitions to phase 2.<\/p>\n

    Phase 2: diagnosis and management\u2014from data to actionable plan<\/p>\n

      \n
    1. \n 1.<\/p>\n

      Patient profile (optional update):<\/b> The patient profile may be further refined.<\/p>\n<\/li>\n

    2. \n 2.<\/p>\n

      DDx validation (subphase):<\/b> The system enters a DDx validation phase. Focused questions are generated to gather specific evidence that supports or refutes potential diagnoses within the internal DDx. A decision module determines when the DDx is sufficiently validated to present to the patient.<\/p>\n<\/li>\n

    3. \n 3.<\/p>\n

      DDx presentation:<\/b> The system presents a ranked DDx (5\u221210 conditions). Crucially, the explanation for each diagnosis is grounded in evidence from the entire interaction, explicitly referencing and explaining findings from the provided multimodal data (for example, \u2018Based on the photo you sent, the circular shape and central clearing of the rash are characteristic of\u2026,\u2019 or \u2018The ECG shows specific changes in the ST segment which support the possibility of\u2026\u2019).<\/p>\n<\/li>\n

    4. \n 4.<\/p>\n

      Management plan formulation:<\/b> After presenting the DDx, the system formulates a management plan based on dialogue history, patient profile and the presented DDx. The plan includes recommendations for investigations, testing and\/or treatment.<\/p>\n<\/li>\n

    5. \n 5.<\/p>\n

      Management plan delivery:<\/b> The system delivers the management plan, potentially iteratively across several turns, allowing for clarification and addressing patient concerns.<\/p>\n<\/li>\n<\/ol>\n

      The presentation of the refined DDx and management plan signals the transition to phase 3.<\/p>\n

      Phase 3: answer follow-up questions\u2014 ensuring patient understanding<\/p>\n

        \n
      1. \n 1.<\/p>\n

        Patient profile (optional update):<\/b> The patient profile may continue to be updated with information from follow-up questions.<\/p>\n<\/li>\n

      2. \n 2.<\/p>\n

        Plan communication and question answering:<\/b> The system addresses remaining patient questions, ensuring that the patient understands the proposed management plan, potentially referencing the multimodal artifacts again to clarify points. Responses are guided by the dialogue history, patient profile, presented DDx and management plan.<\/p>\n<\/li>\n

      3. \n 3.<\/p>\n

        Dialogue termination:<\/b> Dialogue continues until patient questions are addressed, the management plan is communicated and a natural conclusion is reached.<\/p>\n<\/li>\n<\/ol>\n

        Upon reaching this termination state, the system proceeds to synthesize the interaction into a structured summary.<\/p>\n

        After dialogue conclusion: structured post-questionnaire generation<\/p>\n

        Once the dialogue reaches a natural conclusion (phase 3 completion), the system automatically generates a structured post-questionnaire. This process leverages the full multimodal dialogue history and the final internal state (patient profile, final DDx and management plan) to produce a comprehensive summary suitable for clinical review. Specifically, multimodal AMIE is prompted to:<\/p>\n

          \n
        1. \n 1.<\/p>\n

          Finalize DDx:<\/b> Based on the entire interaction, including all textual exchanges and interpretations of multimodal artifacts, generate a final, ranked DDx listing the most probable condition and several plausible alternatives.<\/p>\n<\/li>\n

        2. \n 2.<\/p>\n

          Formulate the management plan:<\/b> Leveraging both the established DDx and a constrained web search process for grounding in current medical knowledge and guidelines, detail the recommended management plan. This includes proposed in-visit and ordered investigations, specific actions or recommendations for the patient and necessary escalation level (for example, video call or in-person visit) with justification and follow-up requirements (necessity, timeframe and reason). This retrieval-based approach is used exclusively during this offline step to ensure that recommendations are informed by up-to-date information. The specific methodology is detailed in Supplementary Section 5<\/a>.<\/p>\n<\/li>\n

        3. \n 3.<\/p>\n

          Extract salient artifact findings:<\/b> Identify and list the key clinical findings observed in any provided images (skin photographs, ECGs and documents) that were relevant to the diagnostic and management reasoning.<\/p>\n<\/li>\n<\/ol>\n

          The structured post-questionnaire serves as a standardized record of multimodal AMIE\u2019s clinical assessment, reasoning and recommendations based on the completed multimodal consultation; it is used both as an artifact rated by clinicians in the OSCE evaluation and as the standardized output scored by the auto-rater in simulated dialogues.<\/p>\n

          Automatic evaluations and system validation<\/p>\n

          We established an automated evaluation framework to support rapid iteration and rigorous assessment of the multimodal AMIE system. This included evaluating perception capabilities on isolated medical artifacts and using a simulation environment with auto-raters to assess complete diagnostic multimodal dialogues and perform component ablations and determine optimal model selection.<\/p>\n

          Validation of base model perception of medical artifacts<\/p>\n

          A critical precursor to effective multimodal diagnostic conversation is ensuring that our underlying models possess a fundamental capability: accurate perception of diverse medical artifacts. Although LLMs have demonstrated remarkable progress in understanding and generating text, their ability to reliably \u2018see\u2019 and interpret medical images and documents\u2014 akin to a clinician\u2019s initial visual assessment\u2014remains less explored in the context of conversational DDx. Without robust perceptual grounding, even the most sophisticated conversational framework would be limited in its ability to meaningfully integrate multimodal data into history-taking and diagnostic reasoning.<\/p>\n

          Therefore, we first conducted a suite of perception tests designed to isolate and evaluate the visual understanding of our base models when presented with common medical artifacts: smartphone-captured skin images, ECG tracings and clinical documents. The objective was not to achieve state-of-the-art performance on complex diagnostic tasks per se but, rather, to establish a baseline confidence in whether our models could reliably discern key visual features and clinical information from these modalities in isolation.<\/p>\n

          For skin images, we used the SCIN dataset31<\/a> and PAD-UFES-20 (ref. 32<\/a>), assessing the ability of the model to describe lesion morphology, color and distribution. For ECGs, we employed PTB-XL33<\/a> and ECG-QA benchmark34<\/a>, prompting the model to provide probable diagnoses or answer expert-validated questions based on ECG images generated from raw signals. Finally, for clinical documents, we created the ClinicalDoc-QA dataset that consists of question-answering tasks based on a large collection of deidentified clinical notes and patient records generated by physicians to evaluate the model\u2019s comprehension of medical information from clinical documents. Additional dataset details can be found in Supplementary Section 3.1<\/a>.<\/p>\n

          To ensure accurate processing of artifacts, we verified the perceptual capabilities of our base model, Gemini 2.0 Flash. Our objective was not to achieve state-of-the-art performance on isolated benchmarks but, rather, to confirm a sufficient perceptual foundation for our state-aware reasoning framework. Overall, this verification (detailed in Extended Data Fig. 5<\/a> and Supplementary Section 3.1.2<\/a>) demonstrated that Gemini 2.0 Flash possessed the necessary foundational perceptual grounding across our target modalities. This provided the confidence required to build multimodal AMIE\u2019s advanced reasoning and dialogue functions upon this base model. This is quantitatively validated by our ablation studies (Fig. 5<\/a>) and ultimately demonstrated by the superior diagnostic accuracy of multimodal AMIE in the human-evaluated OSCE study (Fig. 2<\/a>). The outcomes of these perception tests, presented in the \u2018Automated evaluations of multimodal AMIE configurations\u2019 section and Supplementary Section 3.1.2<\/a>, provide an essential confidence check on our base LLM, Gemini 2.0 Flash, and also enable us to explore critical questions such as the impact of history-taking in addition to multimodal perception.<\/p>\n

          Auto-rating simulated diagnostic conversations<\/p>\n

          The following subsections detail our three-step approach illustrated in Fig. 4<\/a> to creating a simulation environment for auto-evaluation. Step (1) encompasses patient profile and scenario generation, which is then used in step (2) for the turn-by-turn multimodal dialogue generation process used to mimic real-world telemedicine interactions, and, finally, the simulated dialogue is used in step (3) by the auto-rater for automated assessment of multimodal AMIE\u2019s conversational abilities.<\/p>\n

          Step 1: Patient profile and scenario generation<\/p>\n

          Generating realistic synthetic dialogues requires detailed patient profiles and corresponding clinical scenarios. Our methodology involves compiling comprehensive patient metadata, including symptoms, demographics and medical history, tailored to specific medical domains.<\/p>\n