Source-linked AI summary
Sequential Diagnosis with Language Models
Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Viknesh Sounderajah, Jonathan Carlson, Matthew P Lungren, Bay Gross, Peter Hames, Mustafa Suleyman, Dominic King, Eric Horvitz
TL;DR
The paper addresses the gap between static diagnostic benchmarks and the iterative, cost-sensitive reasoning of clinical practice. It introduces SDBench for sequential evaluation and MAI-DxO for orchestrated diagnosis, reporting improved accuracy and cost-accuracy trade-offs. The authors caution that curated NEJM CPC cases do not establish performance on common everyday conditions or false-positive rates.
Problem
Most language-model diagnostic evaluations use static, one-turn vignettes that do not reflect iterative clinical reasoning, evidence gathering, or cost-sensitive decisions.
Method
SDBench transforms 304 NEJM CPC cases into interactive encounters, while MAI-DxO orchestrates physician-like agents that request evidence and tests before diagnosis.
Results
MAI-DxO improved diagnostic accuracy while reducing cumulative costs, establishing a new cost-accuracy Pareto frontier beyond strong off-the-shelf models.
Takeaways & Limitations
Sequential evaluation and physician-inspired orchestration provide an empirically grounded way to assess and advance AI-assisted diagnosis under realistic constraints.
Takeaways & Limitations
Because SDBench uses curated complex NEJM CPC cases without healthy or benign cases, the reported gains may not generalize to everyday conditions and false-positive rates could not be measured.
Abstract
from arXiv · showhide
Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In clinical practice, physicians iteratively formulate and revise diagnostic hypotheses, adapting each subsequent question and test to what they've just learned, and weigh the evolving evidence before committing to a final diagnosis. To emulate this iterative process, we introduce the Sequential Diagnosis Benchmark, which transforms 304 diagnostically challenging New England Journal of Medicine clinicopathological conference (NEJM-CPC) cases into stepwise diagnostic encounters. A physician or AI begins with a short case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only when explicitly queried. Performance is assessed not just by diagnostic accuracy but also by the cost of physician visits and tests performed. We also present the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrator that simulates a panel of physicians, proposes likely differential diagnoses and strategically selects high-value, cost-effective tests. When paired with OpenAI's o3 model, MAI-DxO achieves 80% diagnostic accuracy--four times higher than the 20% average of generalist physicians. MAI-DxO also reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. These performance gains with MAI-DxO generalize across models from the OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families. We highlight how AI systems, when guided to think iteratively and act judiciously, can advance diagnostic precision and cost-effectiveness in clinical care.
1 Introduction
The paper argues that static, one-turn diagnostic evaluations inadequately capture iterative clinical reasoning. It introduces SDBench and MAI-DxO to evaluate and improve diagnosis under sequential information and cost constraints.
- Motivation: Static vignette benchmarks may overstate language-model competence by omitting iterative questioning, testing, and diagnostic revision.They can also obscure premature closure, indiscriminate test ordering, and anchoring on early hypotheses.
- SDBench: SDBench converts 304 NEJM clinicopathological conference cases into stepwise encounters where agents choose questions, tests, and when to diagnose.A Gatekeeper reveals case-consistent information only when explicitly queried.
- MAI-DxO: MAI-DxO combines physician-inspired orchestration, virtual physician roles, marginal-cost estimates, and model ensembling to improve the cost-accuracy trade-off.The system is designed to select high-value tests while simulating a panel of physicians.
- MAI-DxO: 79.9% accuracy at $2,397 was achieved by MAI-DxO with o3, compared with 78.6% at $7,850 for off-the-shelf o3.At higher cost, MAI-DxO reached 85.5% accuracy at $7,184.
- MAI-DxO: MAI-DxO boosted off-the-shelf models from multiple providers by an average of 11 percentage points.The authors characterize these orchestration techniques as general-purpose across model providers.
- SDBench: SDBench assesses both diagnostic accuracy and cost-conscious evidence seeking under uncertainty.The benchmark evaluates whether agents request informative evidence and recognize when diagnostic certainty warrants commitment.
2 Sequential Diagnosis Benchmark
SDBench operationalizes sequential diagnosis as a multi-turn interaction among diagnostic, Gatekeeper, and Judge agents. It evaluates agents against clinical ground truth while assigning standardized monetary costs to visits and tests.
- Benchmark construction: 304 consecutive NEJM CPC cases published between 2017 and 2025 were converted into interactive simulations of sequential diagnostic reasoning.Cases span common conditions and rare disorders and begin with a brief patient summary and chief complaint.
- Agent actions: Diagnostic agents can ask history or examination questions, request diagnostic tests, and make a one-time final diagnosis.Each action is part of the benchmark’s explicit interaction protocol.
- Multiagent orchestration: The Gatekeeper provides requested patient information or test results and refuses vague or nonspecific queries.It consults the full case file and mediates information access during the encounter.
- Gatekeeper: Synthetic findings for tests absent from the original cases were introduced to avoid signaling that queries were off-path.The change followed pilot observations that “Not Available” responses discouraged valid alternative reasoning pathways.
- Diagnosis evaluation: The Judge evaluates diagnoses by clinical substance against ground truth rather than surface-form wording.Its physician-authored rubric compares disease entity, cause, anatomy, specificity, and completeness, while accepting medical synonyms.
- Diagnosis evaluation: Cohen’s κ was 0.70 for MAI-Dx diagnoses and 0.87 for human diagnoses when validating Judge scores against physician ratings.Physicians considered the automated Judge overly strict in four of five disagreements.
- Cost estimation: Monetary cost is used as a secondary metric to discourage indiscriminate imaging or biopsy ordering and proxy practical constraints.The framework assigns $300 per physician visit and derives test costs through CPT-code matching and pricing data.
- Cost estimation: Cost estimates are standardized for comparison rather than exact representations of actual clinical expenses.Over 98% of requested tests were matched to relevant CPT codes; remaining edge cases used language-model price estimates.
3 Experimental Setup
SDBench evaluates physicians and language models in interactive diagnostic encounters where agents choose questions, tests, and when to commit under cumulative costs. MAI-DxO extends this setup with a physician-informed, model-agnostic virtual panel and configurable accuracy-cost tradeoffs.
- Benchmark and evaluation: At each turn, agents may ask about history or examination findings, order a diagnostic test, or commit to a final diagnosis.The process begins with a brief clinical vignette and continues through Gatekeeper interactions.
- Benchmark and evaluation: Diagnostic accuracy was evaluated alongside cumulative medical costs, with agents required to specify tests explicitly and pay for each ordered test.The protocol was designed to measure evidence seeking under realistic clinical constraints.
- MAI-DxO: MAI-DxO uses a five-role virtual physician panel that debates hypotheses, selects discriminative tests, challenges anchoring, enforces stewardship, and checks consistency.After structured deliberation, the panel asks questions, orders tests, or commits when certainty exceeds a threshold.
- MAI-DxO: Five MAI-DxO variants navigate the accuracy-cost frontier, ranging from instant answers and question-only operation to budgeted, unconstrained, and ensemble modes.The system was primarily developed with GPT-4.1 but uses the same model-agnostic orchestration structure across variants.
- Physician comparison: Human physicians used the same Gatekeeper-based interface as AI agents, with 21 US or UK physicians recruited for the held-out cases.Participants had a median of 12 years of experience, and external medical resources were prohibited.
4 Results
MAI-DxO establishes a higher diagnostic-accuracy and lower-cost frontier than off-the-shelf models and physicians, with improvements across model families and held-out cases. Its gains are associated with structured hypothesis tracking, discriminative evidence seeking, and cost-aware orchestration.
- Physician comparison: Physicians averaged 19.9% diagnostic accuracy at an average cost of $2,963 per case.The 21 physicians completed 764 diagnoses across the 56-case test set, averaging 11.8 minutes, 6.6 questions, and 7.2 tests per case.
- Overall performance: MAI-DxO establishes a Pareto frontier that outperforms off-the-shelf models and physicians at every cost or accuracy level.Figure 7 plots diagnostic accuracy against average cumulative medical cost, with each ordered test adding to expenditure.
- Overall performance: 81.9% accuracy and $4,735 average test cost were achieved by standard MAI-DxO on o3, versus 78.6% and $7,850 for off-the-shelf o3.The comparison shows simultaneous improvement in accuracy and test cost for the standard no-budget configuration.
- Overall performance: 79.9% accuracy was achieved at $2,396, while ensembling reached 85.5% accuracy at $7,184.Budget constraints progressively reduced costs with more accuracy loss as the budget decreased.
- Mechanisms: Manual case inspection found MAI-DxO often sought disconfirming evidence and changed diagnostic paths, whereas baseline o3 appeared to anchor on initial impressions.The panel’s explicit hypothesis tracking and adversarial Challenger role supported targeted questioning in inspected cases.
- Cross-model results: MAI-DxO improved diagnostic accuracy across all sufficiently capable foundation models, with gains for no-budget variants statistically significant at p < 0.005 except for o3 and o4-mini.For o3 and o4-mini, the reported significant result was very large cost reduction; o3 also showed significant accuracy gains with the ensemble variant.
- Generalization: Relative improvements in diagnostic accuracy and cost efficiency were preserved on the 56-case held-out test set.The authors report comparable absolute MAI-DxO performance across validation and test data, arguing that gains were not driven by memorization.
5 Discussion
The paper argues that sequential diagnosis benchmarks better reflect clinical reasoning than static vignette evaluations, while MAI-DxO improves the accuracy-cost trade-off on challenging cases. It also identifies important limits to real-world generalization, including case selection, cost modeling, clinician baselines, and unmeasured clinical constraints.
- Sequential Diagnosis Benchmark: SDBench transforms 304 NEJM-CPC cases into interactive, multi-turn diagnostic challenges requiring agents to choose questions, tests, and stopping points under cost constraints.The benchmark uses a Gatekeeper to reveal case-consistent findings only in response to explicit queries and a Judge to assess clinical diagnostic accuracy.
- MAI Diagnostic Orchestrator: MAI-DxO simulates panels of clinical personas to select diagnostic questions and tests, establishing a new accuracy-cost Pareto frontier in SDBench.The system improved accuracy beyond strong off-the-shelf models while reducing cumulative test costs.
- Relation to prior evaluations: Static evaluations present packaged clinical information, whereas SDBench requires agents to seek evidence sequentially and decide when to commit under cost constraints.Earlier NEJM-CPC assessments asked for top-k diagnoses with the full case presented upfront.
- Limitations: SDBench’s curated NEJM-CPC distribution excludes healthy and benign cases, so MAI-DxO’s performance on hard cases may not generalize to everyday conditions or reveal false-positive rates.The authors call for diagnostic corpora that mirror real-world prevalence patterns.
- Limitations: The cost estimates are first-order approximations based on U.S. test costs and omit expenses such as reporting time, maintenance, and patient travel.They remain consistent across agents, allowing relative accuracy-resource trade-offs to be compared.
- Limitations: The physician comparison is also a first-order approximation because it uses generalists, restricts search-engine use, and excludes tools or specialist collaboration available in practice.The authors state that these limitations affect interpretation of comparisons between clinicians and AI systems.
- Implications: The results demonstrate promise for modeling working differentials and test informational value, but do not establish MAI-DxO’s clinical efficacy in real-world decision support.Practical agents must also consider patient-specific risk, invasiveness, discomfort, delays, test availability, authorization, and reimbursement.
Code and data availability
The authors are pursuing external peer review and exploring release of SDBench as a public benchmark.
- The work is being submitted for external peer review.
- The authors are working with partners to explore releasing SDBench as a public benchmark.
A Batched vs Single Testing
The benchmark reports batched testing to better reflect practical clinical strategies, although single and batched agents achieved identical accuracy while single-mode testing cost less.
- MAI-DxO results are reported in batch mode because batching was judged more realistic for reducing discomfort, result waits, and practical testing burdens.MAI-DxO agents were limited to 5 questions and 3 tests per batch.
- 83.9% accuracy was identical for single and batched mode agents.Single-mode testing cost $3,991 versus $5,084 for batched testing.
B Visualizing Model and Clinician Performance
Figure 10 displays case-level scores for MAI-DxO variants and clinicians across a 56-case test set, ordered by diagnostic accuracy and perceived case difficulty.
- Figure 10 shows case-level scores for MAI-DxO variants and clinicians across the 56-case test set.Clinicians assessing at least 10 cases are included.
- Model variants and clinicians are ordered by diagnostic accuracy, while cases are ordered by perceived difficulty according to the MAI-DxO ensemble.
C Recent CPC Case
A 63-year-old woman developed rapidly progressive dyspnea, orthopnea, and edema, with imaging showing extensive perivascular and perirenal soft-tissue thickening. Biopsy and immunohistochemistry supported Erdheim-Chester disease.
- Presentation: A 63-year-old woman presented with two months of progressive exertional dyspnea, orthopnea, lower-extremity edema, and abdominal distention.Symptoms progressed from exertional dyspnea to limitation with minimal activity and sleep; edema partially improved after increased diuretic dosing.
- Imaging: Imaging demonstrated extensive soft-tissue thickening around both kidneys, the abdominal and thoracic aorta, the pericardium, and the mediastinum, with pleural effusions.Abdominal MRI showed retroperitoneal involvement and a bulky pancreatic head; chest CT also showed pulmonary edema and a small pericardial effusion.
- Pathology: Core biopsy showed foamy histiocytoid cells in fibrous stroma, with CD68 positivity and CD1a and S100 negativity.No granulomas or giant cells were seen, and fewer than five IgG4-positive plasma cells per high-power field were reported.
- Diagnosis: The case diagnosis was Erdheim-Chester disease.The supplied case record also reports BRAF V600E immunohistochemical staining.