Source-linked AI summary
From Uncertainty to Clinical Risk: Severity-Aware Conformal Planning for Interactive Medical Diagnosis
Yue Zhou, Haiyang Zhou, Jin Zhang, Kong Wang, Yongxin Ni, Youhua Li, Hanwen Du
TL;DR
Interactive diagnosis must acquire missing evidence while deciding whether to continue questioning or commit, but predictive uncertainty alone does not capture the asymmetric risk of missing severe diseases. The paper proposes Severity-Aware Conformal Clinical Planning, which calibrates severity-weighted clinical risk and uses it in joint Monte Carlo Tree Search over Ask and Commit trajectories. Across DDXPlus and MediQ, the method reports more accurate diagnoses with fewer questions while balancing differential-diagnosis quality and severe-case safety.
Problem
Interactive diagnosis requires sequential evidence acquisition and commitment decisions, while existing uncertainty-based methods overlook the asymmetric clinical risk of prematurely excluding severe diseases.
Method
The framework models diagnostic, safety, and masked-evidence beliefs, calibrates turn-specific prediction sets and severity-weighted risk, and evaluates Ask and Commit jointly with Monte Carlo Tree Search.
Results
The method achieves more accurate diagnoses with fewer questions across DDXPlus and MediQ while improving differential-diagnosis quality and severe-case safety.
Takeaways & Limitations
Calibrated clinical risk can serve as an actionable planning signal for interactive diagnostic decision making rather than only a descriptive uncertainty measure.
Takeaways & Limitations
Evaluation is limited to two English structured benchmarks and a limited set of LLM backbones, and does not establish performance in real clinical environments.
Abstract
from arXiv · showhide
Interactive medical diagnosis dynamically acquires patient information through multiple rounds of questioning, supporting accurate, efficient, and safe clinical decisions under incomplete evidence. Existing methods commonly guide information acquisition with predictive uncertainty or label ambiguity, but overlook the asymmetric clinical risk of missing severe diseases and lack unified long-horizon planning over whether to continue asking questions or commit to a diagnosis. To address these limitations, we propose Severity-Aware Conformal Clinical Planning, which formulates interactive diagnosis as a risk-sensitive sequential decision problem. The framework maintains complementary diagnostic, safety, and masked-evidence beliefs; calibrates turn-specific diagnostic prediction sets and severity-weighted differential-diagnosis risk on held-out diagnostic trajectories; and introduces the calibrated clinical risk into Monte Carlo Tree Search to jointly evaluate long-horizon Ask and Commit trajectories. Experiments on DDXPlus and MediQ show that our method achieves more accurate diagnoses with fewer questions across multiple large language models, while improving differential-diagnosis quality and reducing high-risk errors in severe cases. These findings validate the value of using clinical risk, rather than predictive uncertainty alone, as a planning signal and demonstrate the effectiveness of the proposed framework for information acquisition and risk-aware diagnostic decision making. They also motivate future work on clinical-risk-oriented interactive diagnosis and information-acquisition methods.
1 Introduction
Interactive diagnosis must acquire evidence over multiple rounds and decide both what to ask and when to commit under incomplete information. The paper addresses this with severity-aware calibrated risk and joint long-horizon planning, reporting improved accuracy, efficiency, and safety.
- Motivation: Interactive diagnostic agents must choose what to ask next, whether more information is necessary, and when evidence supports a final diagnosis.This setting differs from conventional medical question answering, which typically assumes a relatively complete case description.
- Motivation: Predictive uncertainty can overlook the asymmetric clinical consequences of excluding a severe disease.Similar statistical uncertainty may permit stopping for benign alternatives but warrant further investigation when severe disease remains plausible.
- Approach: The paper formulates interactive diagnosis as a risk-sensitive sequential decision problem distinguishing predictive ambiguity from clinical consequence.The formulation provides a unified basis for information acquisition and diagnostic commitment.
- Approach: Severity-Aware Conformal Clinical Planning combines diagnostic, safety, and masked-evidence beliefs with turn-specific conformal prediction sets and severity-weighted clinical risk.Risk is calibrated on held-out interaction trajectories rather than taken directly from uncalibrated confidence.
- Approach: Joint Monte Carlo Tree Search evaluates future Ask and Commit trajectories with the same calibrated risk signal.This allows delayed information value to influence planning rather than restricting uncertainty to local question selection or post-hoc stopping.
- Results: 98.57% Top-1 accuracy on DDXPlus was achieved while asking approximately 43–46% fewer questions than the strongest evaluated information-gathering baseline.On MediQ, accuracy improved over C-IP by 3.09 percentage points, with a stronger overall trade-off across accuracy, efficiency, differential diagnosis, and severe-case safety.
2 Related Work
Prior work advances interactive information acquisition, conformal uncertainty, and trajectory planning, but commonly centers on ambiguity rather than the asymmetric consequences of severe-disease omission. This paper positions calibrated clinical risk as the missing link between conformal prediction and sequential diagnostic planning.
- Interactive Diagnosis: Interactive diagnostic research has progressed from static question answering toward multi-turn symptom collection, proactive consultation, and iterative differential diagnosis.DDXPlus and MediQ are cited as benchmarks for this setting.
- Calibrated Clinical Risk: Conformal prediction provides distribution-free uncertainty sets, with later work extending coverage guarantees toward task-specific risk control.Recent extensions address LLM validity, class-conditional coverage, sequential risk control, distribution shift, and self-calibration.
- Calibrated Clinical Risk: The paper differs by introducing a severity-aware calibrated clinical-risk state that explicitly accounts for clinically consequential omission.This targets a limitation of confidence, entropy, and prediction-set size as measures of diagnostic ambiguity alone.
- Sequential Information Acquisition: Active information-acquisition methods use uncertainty or expected information gain, while newer approaches apply simulation, Bayesian design, trajectory learning, and tree-search planning.These approaches extend interaction beyond single-step question selection.
- Sequential Information Acquisition: Future-trajectory evaluation can outperform purely myopic question selection, while complementary work develops statistically controlled stopping rules.The related work therefore combines planning over interactions with calibrated stopping approaches.
3 Method
The method treats interactive diagnosis as a partially observable, sequential decision problem in which the agent maintains complementary beliefs, calibrates severity-aware clinical risk, and plans jointly over asking and committing. Its objective balances diagnostic decisiveness with clinical safety and accounts for delayed information value and action cost.
- 3.1 Problem Formulation: Interactive diagnosis progressively acquires evidence from an initially incomplete case while choosing between additional questions and final commitment.Unqueried findings remain unknown rather than being treated as negative evidence, making the process partially observable.
- 3.2 Belief State Modeling: The framework maintains diagnostic, safety, and masked-evidence beliefs to separate leading-diagnosis preference, differential protection, and missing-information semantics.The resulting diagnostic belief supports reasoning while the safety belief remains available for clinical-risk calibration.
- 3.4 Severity-Aware Conformal Clinical Risk: Turn-specific conformal calibration combines diagnostic ambiguity with disease-dependent severity weights to control clinically meaningful omission risk.This calibration distinguishes omission of low-consequence alternatives from exclusion of clinically critical diseases and supports a controlled safety set.
- 3.5 Joint Risk-Guided Planning: The planner uses observation-branching MCTS to evaluate future interaction trajectories with a joint objective combining residual ambiguity and severity-aware clinical risk.The objective favors states that are both diagnostically decisive and clinically safe rather than merely high-confidence.
- 3.5 Joint Risk-Guided Planning: Action values include interaction cost and delayed information value, allowing questions with limited immediate effects to be preferred when they lead to safer future states.Ask and Commit compete within the same long-horizon search, so commitment is selected explicitly rather than by a confidence-only stopping heuristic.
4 DDXPlus: Differential Diagnosis and Severe-Case Safety
On DDXPlus, the study evaluates interactive diagnosis under a fixed question budget using diagnostic, efficiency, differential-quality, and severe-case safety metrics. Severity-aware calibrated-risk planning improves accuracy and efficiency while preserving differentials, reducing severe-case errors, and benefiting from conformal calibration and multi-step search.
- Experimental Setup: The DDXPlus evaluation uses 490 pathology-balanced cases, including 170 severe-disease cases, with identical initial observations, query spaces, responses, and interaction budgets across methods.Episodes begin with age, sex, and one positive finding; agents may ask up to 10 non-repeated questions before committing.
- Experimental Setup: The evaluation reports diagnostic accuracy, Average Queries, differential-diagnosis quality, severe-disease rule-in and rule-out performance, and Severe Error.DDF1 measures agreement with reference differentials, DSF1 measures severe-disease performance, and Severe Error is the Top-1 error rate over severe cases.
- Main Results: 97.76% Top-1 accuracy and 4.80 average questions replace BED-LLM’s 92.24% and 8.46 with Qwen3-4B, while similar accuracy-efficiency trends hold for the other backbones.The method achieves the best Top-1 accuracy across all three evaluated backbones and gains primarily through selectively acquired evidence rather than more questioning.
- Main Results: Approximately 0.60 DDF1 and 0.59 DSF1 exceed best-baseline values of 0.385 and 0.279, while Llama-3.1-8B-Instruct’s Severe Error falls from 6.47% to 0.00%.These results indicate improved leading-diagnosis discrimination alongside preservation of clinically meaningful differentials and fewer high-risk errors.
- Main Results: Top-1 accuracy varies by only 1.02 percentage points across the three backbones, compared with 8.78 points for C-IP, while average questions vary by 0.19.The reported consistency is associated with the shared calibrated-risk-driven Ask/Commit process rather than dependence on a particular backbone.
- Ablation and Sensitivity: The full model achieves the highest Top-1 accuracy and DSF1, maintains low interaction cost, and records zero observed Top-1 errors among 170 severe cases.Removing conformal calibration or severity weighting leaves severe-case errors, while one-step planning reduces Top-1 accuracy; α = 0.1 reaches 98.57% accuracy with 4.77 questions and no observed severe-case errors.
5 MediQ: Information Efficiency and Adaptive Stopping
On MediQ, calibrated decision-risk planning improves diagnostic accuracy and trajectory performance while asking fewer questions and maintaining reliable stopping coverage.
- Interactive Setting: The MediQ setting exposes only intake information initially, allows Ask or Commit decisions, and limits interaction to a maximum horizon of 10 turns.Patient facts are revealed only when queried, while All Information serves as an oracle with access to the complete record.
- Evaluation: MediQ evaluates Accuracy, Average Queries, Stop-Time Coverage, Accuracy-Turn AUC@10, and Full-Info Gap to assess diagnostic performance, efficiency, calibration, and trajectory quality.Table 4 treats higher Accuracy, Coverage, and Acc.-Turn AUC@10 as better, and lower Avg. Queries and Full-Info Gap as better.
- Main Results: 66.18% accuracy is achieved, 3.09 percentage points above C-IP, while stop-time coverage remains 91.54%, above the 90% target.The method also reduces the gap to the All Information oracle to 1.30 points.
- Information Efficiency: 6.57 average questions reduce information acquisition by 33.4% versus All Information’s 9.87 questions while approaching its diagnostic accuracy.This indicates that the gain comes from allocating interaction more effectively rather than simply asking more questions.
- Trajectory Performance: 60.76% Accuracy-Turn AUC@10 is the highest among non-oracle methods, showing that the advantage persists across the interaction trajectory.Entropy-Greedy and UoT-adapted stop slightly earlier but achieve lower final accuracy and lower trajectory-level performance.
6 Conclusion
The paper proposes Severity-Aware Conformal Clinical Planning to jointly plan questioning and diagnostic commitment under incomplete information. Experiments on DDXPlus and MediQ show fewer questions while balancing accuracy, differential-diagnosis quality, and severe-case safety.
- Conclusion: Severity-Aware Conformal Clinical Planning calibrates clinical risk to jointly plan long-horizon Ask and Commit actions.The framework targets interactive diagnosis under incomplete patient information.
- Conclusion: Experiments on DDXPlus and MediQ show reduced question counts while balancing diagnostic accuracy, differential-diagnosis quality, and severe-case safety.The conclusion presents these findings as evidence supporting the proposed planning framework.
7 Limitations
The evaluation is limited to structured simulated benchmarks and does not establish performance in real clinical environments or broad generalization across settings, languages, specialties, institutions, or model families.
- Scope and Generalization: The study does not establish real-clinical performance because its evaluation primarily uses structured diagnostic benchmarks with potentially noisy, incomplete, or shifted patient responses in practice.The experiments do not assess clinician workflow integration, prospective patient outcomes, or robustness to adversarial and misleading responses.
- Scope and Generalization: Findings may not generalize beyond two English benchmarks and a limited set of LLM backbones, languages, specialties, institutions, or model families.Severity labels are predefined disease grades and may not capture patient-specific consequences.
- Calibration Boundaries: Turn-wise conformal calibration provides marginal risk control under the fixed evaluation protocol, not patient-conditional coverage or validity at arbitrary stopping times.This bounds the validity claim to the evaluated protocol and stopping conditions.
8 Ethical Considerations
The system is a research prototype for assistive clinical decision support, not medical advice or a replacement for qualified clinicians. Real-world use requires prospective validation, oversight, governance, auditing, and mechanisms to contest recommendations.
- Ethical Considerations: Model errors, benchmark bias, imperfect severity labels, and distribution shift could produce harmful or unequal recommendations, especially for underrepresented patient populations.The paper frames these risks as relevant to any potential deployment of the prototype.
- Ethical Considerations: Real-world deployment would require prospective clinical validation, human oversight, privacy-preserving data governance, transparent auditing, and mechanisms for contesting recommendations.Until safety, fairness, and reliability are established in the intended setting, the framework should remain assistive research technology.