PaperlayerView exact revision on arXiv

Source-linked AI summary

MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz, George Doumat, Georg Groh

arXiv:2608.25085v1cs.CL

TL;DR

Static medical QA benchmarks do not adequately evaluate the interactive, incremental information gathering required for diagnosis, and interactive settings show degraded accuracy and reliability. MTDiag addresses this gap with a multi-turn dataset built from heterogeneous clinical sources, normalized into an ontology-grounded schema, and paired with dialogue-level metrics beyond diagnostic accuracy. The dataset and pipeline support evaluation of how LLMs ask questions, reason over clinical evidence, and fail during diagnostic conversations.

  • Problem

    Static QA benchmarks provide complete, pre-structured information, while diagnosis requires interactive and incremental evidence gathering; interactive evaluations report degraded accuracy and reliability.

  • Method

    MTDiag combines DDXPlus, MIMIC-IV, and AJCR cases in a canonical PatientVector schema with UMLS CUI symptom anchors, ICD-10 diagnoses, generated utterances, and structured dialogue logs.

  • Results

    MTDiag enables clinical knowledge-grounded evaluation of differential-diagnosis dialogues beyond diagnostic accuracy, including question behavior, reasoning failures, and interaction patterns.

  • Takeaways & Limitations

    Ontology-grounded dialogue logs provide a reusable substrate for transparent, reproducible evaluation of LLMs as diagnostic agents.

  • Takeaways & Limitations

    Realistic assessment remains bounded by carefully prompted simulated encounters and would require large-scale ontological mapping of highly unstructured real user chatlogs.

Abstract

from arXiv · show

Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.

1 Introduction

MTDiag addresses the gap between strong static medical QA performance and weaker interactive diagnosis by providing a multi-turn dataset grounded in heterogeneous clinical sources and UMLS concepts.

  • Motivation: Medical LLMs can approach 90% accuracy on knowledge-based questions but achieve roughly 45% on practice-based diagnostic tasks.The gap reflects the interactive, incremental nature of diagnosis compared with complete, pre-structured benchmark information.
  • Motivation: Interactive diagnostic studies report clear accuracy and reliability degradation, while qualitative analyses find unsafe answers in patient-posed medical conversations.These findings motivate evaluation in realistic diagnostic encounters rather than static question answering alone.
  • Dataset: MTDiag is a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous clinical sources and anchored in UMLS.The dataset is designed to support clinical reasoning evaluation beyond diagnostic accuracy.
  • Dataset: The dataset normalizes DDXPlus, MIMIC-IV, and AJCR cases into a canonical PatientVector schema, mapping diagnoses to ICD-10 and symptoms to UMLS CUI identifiers.This schema provides a common representation across the source datasets.
  • Pipeline: A UserLM-8B framework converts structured clinical evidence into natural patient-like utterances, while a MedGemma-based Patient Orchestrator selects runtime responses from the generated pool.The pipeline separates utterance generation from dialogue execution.

2 Background and Related Work

MTDiag frames diagnosis as an interactive encounter in which examiners must elicit and synthesize uncertain patient information, addressing limitations of static benchmarks and prior conversational evaluations.

  • Diagnostic Encounter: The diagnostic encounter is defined as a patient seeking diagnosis for a chief complaint through examination dialogue with a medical practitioner.The examiner performs differential diagnosis by generating, ranking, and narrowing candidate diagnoses across turns.
  • Diagnostic Encounter: MTDiag’s examiner-side evaluation uses ground-truth diagnoses, symptom and history anchors, and dialogue logs to assess whether models ask the right questions in the right order.This extends evaluation beyond whether the final diagnosis matches.
  • Diagnostic Encounter: Patients may be vague, forgetful, resistant, or unaware of clinically relevant information, reducing the discriminative power of symptoms during history-taking.The history of present illness is negotiated across conversation rather than handed to the physician in complete form.
  • Diagnostic Encounter: Models exhibit inconsistent risk stratification, guideline-following failures, and sensitivity to patient linguistic framing during incremental information gathering.One cited analysis reports up to 52% emergency undertriage in a direct-to-consumer medical offering.
  • Related Work: Static medical QA benchmarks use self-contained questions from licensing exams and literature, whereas conversational systems must elicit information across multiple turns.Prior interactive systems show substantial degradation relative to their static benchmark performance.
  • Related Work: Existing medical dialogue datasets capture multi-turn dynamics, but their suitability for MTDiag is assessed alongside the broader dataset landscape surveyed in prior work.The paper positions MTDiag as a response to limitations in current conversational diagnostic evaluation.

3 Ontological Anchoring and Resource Choices

MTDiag uses ontology-grounded anchors, dialogue traces, and heterogeneous source datasets to evaluate differential diagnosis beyond simple ICD-10 classification accuracy.

  • Design Principles: The construction design was informed by practicing medical professionals and clinical behavioral literature to represent logical and practical errors in multi-turn differential diagnosis.The paper introduces minimum anchors and ontology-enabled metrics as design consequences.
  • Minimum Anchors: The minimum anchors are the chief complaint, symptoms, and primary diagnosis, with an ICD-10 match serving as the minimum target for diagnosis accuracy.The chief complaint opens the dialogue, while the diagnosis is the case’s ground-truth principal finding.
  • Dialogue Trace: Two dialogues with the same headache opening show that probing for GCA symptoms can reach the correct diagnosis, while migraine-focused questioning can produce a tension-type headache diagnosis and miss GCA.The contrasting traces motivate evaluating questioning behavior rather than final classification alone.
  • Dialogue Metrics: CUI-based dialogue metrics detect anchoring bias in addition to ICD-10 classification errors by analyzing the examiner’s line of questioning.The paper extends this metric framework with a broader formal treatment in Section 5.
  • Resource Choices: User-facing assessment excludes patients unable to communicate through chat and therefore requires preprocessing choices that support realistic conversational symptom expression.The same realism requirement motivates using UserLM-8B to simulate user interactions rather than physician-facing responses.
  • Source Datasets: The three selected sources cover a structured synthetic head, a real high-frequency ED cohort, and a long tail of rare and atypical case reports.DDXPlus contributes 1.3 million cases across 49 pathologies and 223 evidences; MIMIC-IV contributes over 200,000 ED visits and 65,000 ICU stays; AJCR contributes 392 adult case reports.

4 Methodology

MTDiag constructs a multi-source diagnostic dataset by preprocessing heterogeneous clinical records, resolving evidence to canonical identifiers, and normalizing cases into a unified PatientVector before offline utterance generation and dialogue evaluation.

  • Anchor Resolution and Dialogue Runtime: Clinical anchors are mapped to UMLS CUIs and diagnoses to ICD-10 codes before UserLM-8B converts PatientVector evidence into stored natural-language utterances.The pipeline uses a two-stage NER-to-UMLS process, while pre-generated utterances remain decoupled from live dialogue; the examiner receives the chief complaint and asks follow-up questions across turns.
  • Data Collection & Preprocessing: MTDiag combines DDXPlus, MIMIC-IV, and AJCR sources selected for verified chief complaints, symptoms, and primary diagnoses, while preserving additional heterogeneous clinical information.The sources were chosen to satisfy minimum anchor requirements; secondary diagnoses, histories, HPI, medications, laboratory tests, and demographic features are retained with varying sparsity.
  • MIMIC-IV: MIMIC-IV cases are filtered from 425,087 raw ED visits and deduplicated into 68,346 unique patients across ED-discharge and hospital-admission tracks.The two tracks differ in available evidence: ED-discharge cases use ED data, whereas hospital-admission cases add HPI and selected structured clinical data.
  • AJCR: AJCR processing extracts structured article content and yields 415 curated adult diagnostic cases from 392 reports with verified final diagnoses.The general-purpose pipeline retrieves article HTML and extracts metadata, abstracts, sectioned text, figures, and tables into per-case JSON.
  • PatientVector Unified Case Schema: Each case is normalized into a PatientVector containing Provenance, GroundTruth, PatientFacts, and CasePresentation blocks.These blocks capture source identity, masked diagnoses and provenance, patient information, and the opening symptom presentation with temporality, locality, and characterization.

5 Discussion

MTDiag extends diagnostic evaluation beyond accuracy by combining ontology-grounded multi-source cases with metrics for evidence elicitation, reliability, bias, and harm. Its structured design supports controlled assessment across demographic, linguistic, and diagnostic-frequency variations.

  • Controlled evaluation: MTDiag’s structured, generative substrate supports masking, noise injection, and profile generation, enabling robustness studies across demographics, languages, and common-to-rare diagnoses.UMLS’s language-agnostic identifiers support 31 languages, while source stratification spans routine MIMIC-IV cases and rare AJCR cases.
  • Knowledge-grounded evaluation: UMLS and ICD-10 anchoring enables dialogue-level metrics that assess diagnostic semantics and clinical reasoning beyond surface-level string matching.These representations support automatic scoring from CUI-annotated dialogue logs.
  • Knowledge-grounded evaluation: The Symptom Elicitation Score measures whether a model asks about disease-relevant evidence using frequency-weighted HPO symptom associations.Weights range from 1.0 for obligate symptoms to 0.1 for very rare symptoms.
  • Reliability and failure modes: Reliability, anchoring-bias, and premature-closure measures distinguish correct diagnoses supported by adequate evidence from diagnostic hallucinations and flawed questioning strategies.Reliability requires both the correct diagnosis and a symptom-elicitation score above a minimum threshold.
  • Clinical risk: The Harm Index categorizes errors as contraindicated treatment, delay of care, or omission of critical advice, while remaining applicable to any system producing CUI-annotated logs.The three tiers cover direct harm, low-acuity misdiagnosis of emergencies, and instructional failure.

Limitations

MTDiag’s limitations concern ontology versioning, data-access restrictions, possible pretraining contamination, limited utterance diversity, incomplete multi-model evaluation, and imperfect realism relative to unconstrained user interactions.

  • UMLS as a Living Resource: The 2025AB UMLS release may produce different anchor resolutions or CUI assignments when rerun with another installation.UMLS updates regularly, although bespoke installations can improve compatibility with specific health systems.
  • Data Governance and Release Constraints: MIMIC-derived data cannot be shared freely and requires credentialed PhysioNet access plus Zero Data Retention-compliant processing.PhysioNet treats the formally de-identified data as potentially re-identifiable.
  • AJCR Pretraining Contamination: AJCR cases may overlap with evaluated models’ pretraining corpora because the reports are widely indexed and freely available online.The structured PatientVector produces dialogue substantially different from the source prose, but subset comparisons remain future work.
  • Utterance Pool Coverage: The release offers canonical utterances per symptom, while large-scale variation across personas, masking, health literacy, and noise remains future work.Real patients vary substantially in how they describe the same symptom.
  • Multi-Model Evaluation: Systematic multi-model evaluation as an examiner is not included and is left to future work.The current work introduces the pipeline and evaluation framework and provides a validated dataset.
  • Patient-Chatbot Realism Revisited: The benchmark’s prompted medical-professional setting differs from real users’ unconstrained conversations and would require large-scale ontological mapping of real chatlogs.Real users may provide unrelated context or use settings that alter outputs.

Ethical Considerations

The paper questions whether LLMs are suitable for long-range diagnostic tasks because context limits, sycophantic behavior, and imperfect patient reporting complicate clinical interaction.

  • Ethical Considerations: LLM context-window limits, sycophantic behavior, and patients’ hedging, minimizing, or reframing can undermine long-range diagnostic tasks.Physicians instead rely on extensive training, mentorship, clinical experience, and input from qualified colleagues.

A Datasets Considered but Excluded

The authors excluded alternative dialogue corpora because they lacked multi-turn diagnostic structure, naturalistic generation, or sufficient scale and language coverage.

  • Structural limitations: MedDialog and MIE were excluded because they lack multi-turn diagnostic structure, serving primarily forum-style QA or information extraction.These corpora do not provide a diagnostic trajectory suitable for the target task.
  • Generation limitations: MDDial was excluded because its template-driven generation lacks naturalistic variation and sufficient multi-turn depth.
  • Scale and coverage limitations: ReMeDi, MedDG, and MDD… were excluded because they are heavily restricted in scale and cover only five pediatric conditions.

B Source Datasets

MTDiag combines heterogeneous clinical sources with typed evidence, ontology-aware processing, and two MIMIC-IV cohorts that support distinct diagnostic tasks.

  • DDXPlus: DDXPlus evidence is typed as binary, categorical, or multi-choice, with antecedent annotations distinguishing presenting symptoms from medical history.Multi-choice evidence can represent multiple sub-features of one symptom.
  • DDXPlus: Under the closed-world assumption, evidence presence means affirmation and absence means a negative state, rather than relying on binary default values.This reconstructs evidence states from list membership and avoids misrepresenting negative findings.
  • DDXPlus: DDXPlus retains a randomly selected INITIAL_EVIDENCE item as the opening chief complaint to preserve the original diagnostic progression.Its primary benchmark value lies in differential-diagnosis trajectories and structured evidence instantiation.
  • MIMIC-IV: MIMIC-IV provides de-identified ED and ICU data from BIDMC, with randomized identifiers and shifted dates preserving intra-patient temporal consistency.The database covers emergency and intensive-care encounters from 2008 to 2019.
  • MIMIC-IV: MTDiag repurposes MIMIC modules and extensions to construct coherent narrative presentations from demographics, admissions, diagnoses, and clinical data.The selected sources include hosp, ed, bhc, and cdn modules.
  • Population selection: The MIMIC cohort targets self-presenting ED patients who could plausibly engage in multi-turn chatbot dialogue.Selection filters operationalize coherence, communicability, and realistic chatbot engagement.
  • Population selection: Cases without diagnosis codes or interpretable pain scores are removed because they lack reliable ground truth or indicate inability to communicate pain.
  • Two-Tracks: After filtering, 222,071 visits were discharged and 203,016 resulted in hospital admission, forming two clinically distinct cohorts.

B.2.4 Data Extraction and Linkage

MTDiag extracts and links heterogeneous clinical data into structured, patient-level cases and natural-language dialogue inputs. The pipeline combines cohort filtering, deduplication, case-report parsing, ontology mapping, and utterance generation.

  • MIMIC-IV extraction: 424,995 MIMIC-IV candidate rows cover 40 sparse features per visit after merging emergency-department, admissions, and patient tables.The ED module initializes the cohort, with joins keyed by stay_id, subject_id, and hadm_id.
  • MIMIC-IV extraction: Hospital-course and assessment sections are masked to prevent ground-truth leakage, while curated clinical records take precedence over note-derived HPI.The pipeline merges HPI, examination, laboratory, radiology, and microbiology data when available.
  • MIMIC-IV extraction: MIMIC-IV visits undergo sequential filtering and hierarchical deduplication, retaining each patient’s highest-priority data-rich visit.Priority levels favor CDM and parsed hospital-course records before falling back to the earliest visit.
  • Ontology mapping: Clinical strings are mapped locally through medspaCy and QuickUMLS after LLM and REST-API approaches showed hallucination, recall, or precision problems.The mapping targets UMLS CUIs and ICD-10 codes within the canonical PatientVector schema.
  • Utterance generation: UserLM-8B generates varied utterance candidates for each symptom or evidence entry, spanning precise, colloquial, direct, and third-party formulations.Persona parameters condition the generated utterance variants.

E Clinical Reasoning Metrics

MTDiag evaluates clinical reasoning beyond whether the final diagnosis is correct. Ontology-linked symptoms and diagnoses support graded, clinically interpretable scoring at both dialogue and utterance levels.

  • Scope: MTDiag extends evaluation beyond diagnostic accuracy to assess clinically grounded reasoning in multi-turn encounters.The framework is organized from diagnostic outcome to reasoning fidelity and harm.
  • Ontology-grounded evaluation: UMLS CUI and ICD-10 anchors enable graded scoring over ontological identifiers rather than surface-string similarity.Scores can be computed at both dialogue and utterance levels.

E.1 Diagnostic Outcome Metrics

Diagnostic outcome metrics distinguish exact diagnosis matching from clinically meaningful near misses. They use ICD-10 structure and secondary diagnoses to avoid treating every mismatch as equally wrong.

  • Diagnosis Accuracy: Diagnosis Accuracy is a binary baseline that scores one when the predicted ICD-10 code matches the ground truth and zero otherwise.The metric tests exact code agreement.
  • Semantic Diagnostic Distance: Semantic Diagnostic Distance treats hierarchical ICD-10 under-specification as less severe than categorical error.For example, I21.0 versus I21.02 differs in code specificity, while matching a secondary diagnosis can count as clinically consistent.

E.2 Symptom Elicitation Metrics

Symptom-elicitation metrics evaluate whether models ask relevant, sufficient, efficient, and logically necessary questions before committing to diagnoses. The framework also flags anchoring, premature closure, and pathological dialogue lengths.

  • Symptom relevance: Disease-wise Symptom Score measures relevant symptom queries while penalizing inefficient turn counts.It compares queried CUI-resolved symptoms with symptoms associated with the diagnosed disease.
  • Symptom relevance: Frequency-weighted relevance rewards prioritizing high-yield symptoms early, using weights from 1.0 for obligate to 0.1 for very rare findings.Intermediate suggested weights are 0.75, 0.5, and 0.25.
  • Symptom coverage: Precision penalizes irrelevant questioning, whereas recall measures coverage of the relevant symptom space.Both metrics operate on the overlap between queried and disease-associated symptoms.
  • Reliability: Reliability Score credits a model only when it reaches the correct diagnosis with sufficiently relevant symptom inquiries.A correct guess paired with clinically incoherent questioning receives Rscore = 0.
  • Logical fidelity: Pathognomonic and sine qua non errors identify failures to elicit uniquely identifying or necessary evidence, with sine qua non omissions eligible as hard vetoes.A diagnosis unsupported by necessary evidence is treated as invalid regardless of string match.
  • Reasoning failures: Anchoring detects restriction to one candidate’s symptom associations, while premature closure flags commitment before sufficient weighted symptom coverage.Premature-closure thresholds are configurable because no established clinical threshold exists.
  • Efficiency: Turn Efficiency separately flags dialogues shorter than the number of sine qua non symptoms or longer than the maximum turn limit.Turn count is also represented through the cost penalty in Sdise.
Loading 2608.25085v1…