Source-linked AI summary
MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models
Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, Jinjie Gu, Junwei Liu
TL;DR
Existing medical LLM benchmarks inadequately evaluate information gathering and diagnostic reasoning in realistic multi-turn consultations. MedDialogRubrics combines synthetic cases, controlled patient simulation, and EBM-grounded expert-refined rubrics, and finds substantial model challenges centered on inquiry planning. The framework therefore supports fine-grained assessment of the diagnostic process rather than final answers alone.
Problem
Existing benchmarks do not rigorously evaluate information gathering and diagnostic reasoning in realistic multi-turn medical consultations.
Method
MedDialogRubrics combines 5,200 synthetic patient cases, a Patient Agent grounded in atomic facts, and EBM-derived expert-refined must-ask rubrics.
Results
Current models face substantial challenges, with context length not guaranteeing better diagnostic reasoning and inquiry planning emerging as a bottleneck.
Takeaways & Limitations
Fine-grained multi-turn evaluation exposes diagnostic-process gaps that aggregate metrics and static benchmarks can miss.
Takeaways & Limitations
Rubric generation assumes access to an evidence-based knowledge graph encoding clinically relevant entities and guideline-derived diagnostic relations.
Abstract
from arXiv · showhide
Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.
1 Introduction
MedDialogRubrics addresses the difficulty of evaluating multi-turn medical reasoning, where realistic information gathering, reliable patient simulation, and objective scoring remain challenging. It combines controlled synthetic cases, hallucination-resistant patient agents, and EBM-grounded rubrics to assess diagnostic dialogue more rigorously.
- Motivation: Clinical diagnosis is a dynamic, multi-turn process, but existing medical LLM benchmarks predominantly assess static question answering or summarization.These benchmarks evaluate domain knowledge retention but do not assess structured information gathering during iterative diagnosis.
- Challenges: Privacy constraints limit scalable multi-turn medical dialogue datasets, while standard LLM patient agents may hallucinate symptoms or contradict medical logic.These limitations threaten reproducibility and evaluation validity.
- Challenges: Dialogue evaluation is subjective because assessing whether must-ask questions occur at the right time often requires costly human annotation or vague heuristics.Unlike multiple-choice tasks, consultation trajectories lack a single gold-standard answer.
- Framework: The framework introduces a clinically grounded multi-agent pipeline with a Patient Agent anchored to atomic medical facts and dynamically corrected hallucinations.The design aims to maintain internal consistency and clinical plausibility throughout multi-turn interactions.
- Contributions: 5,200 synthetically constructed patient cases are released without accessing private real-world records.The dataset covers diverse diseases while addressing privacy and data-governance concerns.
- Contributions: Over 60,000 fine-grained rubrics are generated from EBM guidelines using LLM retrieval, expert refinement, and a must-ask criterion for objective scoring.The rubric pipeline targets clinically relevant inquiry items rather than only final diagnostic accuracy.
- Contributions: Evaluation of state-of-the-art LLMs reveals gaps in dialogue management architectures that incremental instruction tuning alone may not resolve.The framework is intended to expose failures in multi-turn information gathering and diagnostic reasoning.
2 Related Work
Medical evaluation has progressed from static question answering toward multi-turn clinical interaction, patient simulation, and action-oriented assessment. However, existing work still lacks scalable, clinically grounded datasets, structured inquiry-strategy benchmarks, and sufficiently rigorous LLM-as-judge evaluation for medical reasoning.
- Medical QA and Clinical Reasoning Benchmarks: Earlier medical benchmarks such as MedMCQA and MedQA primarily evaluate single-turn knowledge and reasoning through multiple-choice questions.Later work expanded toward long-form answers and multi-turn clinical conversations.
- Medical QA and Clinical Reasoning Benchmarks: Interactive patient simulators and diagnostic-reasoning studies introduce dynamic tasks, but existing simulators may be proprietary, manually constructed, or unsuitable for large-scale evaluation.These limitations constrain reproducibility and broad model comparison.
- Dialogue Simulation and Patient Agents: Medical patient agents require stronger factual consistency, symptom logic, safety controls, and reproducibility than general task-oriented dialogue simulators.Existing LLM-driven patient agents often lack sufficient controls for standardized evaluation across models.
- Dialogue Simulation and Patient Agents: Patient-agent research has advanced from persona prompting toward frameworks using behavioral modeling, preference optimization, retrieval augmentation, and multi-agent clinical environments.PatientSim models personality, literacy, and cognitive confusion, while other systems connect agents to simulated records or diagnostic tools.
- Evaluation Methods: Action-oriented benchmarks and longitudinal-memory systems extend evaluation beyond linguistic quality toward multi-step clinical tasks and persistent patient models.These efforts highlight reliability gaps between action-based and query-based tasks and address agent statelessness.
- Evaluation Methods: LLM-as-a-judge methods provide scalable rubric-based and ensemble evaluation, but medical reasoning requires greater rigor because judging errors can have safety implications.HealthBench uses granular criteria, while MedHELM uses judge ensembles to reduce individual-judge weaknesses.
- Research Gaps: Three gaps remain: clinically grounded scalable datasets with controlled patient behavior, structured inquiry-strategy benchmarks, and rigorous medical LLM-as-judge methods.MedDialogRubrics addresses them through synthetic cases, structured clinical annotations, reproducible patient agents, and calibrated judging.
3 Methods
MedDialogRubrics builds a privacy-preserving, multi-agent framework for generating clinically coherent patient cases, simulating deterministic consultations, producing evidence-grounded inquiry rubrics, and evaluating dialogue performance. Its pipeline combines validation, hallucination correction, expert refinement, and LLM-based judging.
- Multi-Agent Patient Record Generation: A hierarchical multi-agent system constructs records from disease outlines, basic information, and detailed clinical data before synthesizing the chief complaint.Generated variables include demographics, severity, onset, comorbidities, lifestyle, symptom duration, and pertinent negative findings.
- Multi-Agent Patient Record Generation: The framework selects diseases from a curated taxonomy and retrieves disease knowledge to define core, auxiliary, and red-flag symptoms.The taxonomy covers primary-care presentations, chronic diseases, mental-health conditions, and acute emergencies.
- Multi-Agent Patient Record Generation: Automated validators check missing symptoms, contradictory time courses, demographic or risk-factor mismatches, and overlap with distractor illnesses; failing cases are revised or discarded.This validation process produces a dataset of high-quality diagnostic scenarios.
- Patient Agent Design: The Patient Agent uses atomic medical facts as ground-truth memory, answers consistently with those facts, and corrects conflicting responses through a persistent guidance loop.The agent is designed to remain realistic enough to challenge models while deterministic enough for fair comparisons.
- Rubric Generation: Key inquiry rubrics are generated from case descriptions and evidence-based knowledge subgraphs, scored for clinical quality, verified against hard principles, and refined through reject sampling.The pipeline evaluates relevance, completeness, non-redundancy, clarity, actionability, evidence consistency, red-flag coverage, and absence of unsafe inquiries.
- LLM-as-Judge Evaluation: The evaluation framework uses an LLM-based grader to assign binary rubric judgments and computes weighted case scores across doctor–patient conversations.The framework is designed for clinically grounded, reproducible, and bias-minimized assessment and was informed by three independent datasets comprising 7,166 matched samples.
4 Experiments Results and Analysis
Experiments evaluate four LLM doctor agents through multi-turn consultations, rubric-based scoring, and ensemble validation against clinical experts. Results reveal substantial differences in information-seeking dynamics, persistent rubric-coverage gaps, and strong but strategy-dependent agreement between automated and human evaluation.
- Experimental Setup: Over 60,000 expert-annotated rubric criteria across more than 4,700 cases support atomic-level assessment of consultation quality.The benchmark evaluates four representative LLMs as doctor agents within MedDialogRubrics.
- Experimental Setup: The evaluation pipeline generates consultations, extracts inquiry and reasoning actions, applies structured LLM judging, verifies consistency, aggregates scores, and computes Precision, Recall, Accuracy, and F1-score.
- Information Gathering and Temporal Dynamics: Gemini-2.5-pro reaches approximately 52% rubric-match precision around turns 9–10, then stabilizes, indicating efficient early information seeking.Its trajectory prioritizes high-value medical facts early in the conversation.
- Information Gathering and Temporal Dynamics: GPT-5 begins below 30% at turn 4, surpasses DeepSeek by turn 9, and reaches its highest performance at 13+ turns, consistent with a more exhaustive strategy.
- Information Gathering and Temporal Dynamics: DeepSeek-R1 plateaus near 40%, while Qwen3-235B-A22B-Instruct-2507 remains around a 30–35% match rate regardless of dialogue length.
- Information Gathering and Temporal Dynamics: The strongest model reaches only approximately 52% against a 100% theoretical maximum, indicating that advanced models miss nearly half of expert-defined diagnostic criteria.
- Automated Evaluation Reliability: Majority Voting maintains 75–79% F1-scores across doctor agents, while Liberal Strategy reaches approximately 79.6% F1 for GPT-5 and Unanimous Voting lowers recall and F1.Majority Voting is adopted for the main leaderboard to balance sensitivity and specificity.
- Discussion: Temporal analysis exposes up to a 20% rubric-coverage difference between models, while increased context length alone does not guarantee better diagnostic reasoning.Only Gemini-2.5-pro closes the inquiry loop effectively within 8–12 turns.
5 Conclusions
MedDialogRubrics evaluates multi-turn inquiry using synthetic cases and expert-refined rubrics that assess diagnostic-process quality beyond final-answer correctness. Its experiments show substantial weaknesses in strategic information seeking and long-context management, motivating advances in dialogue-management architectures.
- Conclusions: MedDialogRubrics uses 5,200 synthetic patient cases and more than 60,000 expert-refined rubric criteria to assess completeness, logic, effectiveness, and diagnostic correctness.
- Conclusions: Evidence-Based Medicine guidelines define prioritized “must-ask” questions, while dynamic guidance reduces hallucinations and preserves clinically plausible, coherent evaluations.
- Conclusions: State-of-the-art LLMs struggle particularly with strategic information seeking and long-context management under the benchmark’s standards.
- Conclusions: The results indicate that improving medical conversational AI requires advances in dialogue-management architectures rather than only incremental base-model tuning.
A Supplementary Figures and Tables
The supplementary material includes a rubric-matching example comparing Gemini-2.5-pro with GPT-5 and a table describing key-rubric categories, their definitions, and examples.
- Supplementary Figures: Figure 8 presents an example of key-rubric matching between Gemini-2.5-pro and GPT-5.
- Supplementary Tables: Table 4 lists key-rubric categories together with their descriptions and examples.