PaperlayerView exact revision on arXiv

Source-linked AI summary

EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents

Lihang Zeng, Shaoting Zhang, Xiaofan Zhang

arXiv:2608.24570v1cs.AI

TL;DR

Many LLM diagnosis systems treat clinical diagnosis as static prediction rather than iterative evidence seeking and termination. EviDx builds patient-specific interactive environments, scaffolds evidence acquisition, and regulates termination with uncertainty and coverage signals. Experiments report improved diagnostic performance and process stability, while open-ended diagnosis remains constrained by base-model medical knowledge.

  • Problem

    Many LLM diagnosis systems use static case-to-answer prediction with limited support for iterative evidence acquisition and termination decisions.

  • Method

    EviDx combines patient-specific environments, role-specialized evidence scaffolds, an observer-guided harness, and a three-level evaluation pyramid.

  • Results

    EviDx improves multiple-choice diagnostic performance and process stability across its evaluation pyramid, while open-ended diagnosis remains constrained by base-model medical knowledge.

  • Takeaways & Limitations

    Clinical LLM diagnosis should be assessed not only by final answers but also by evidence acquisition, diagnostic trajectories, and termination decisions.

  • Takeaways & Limitations

    The harness relies on model-reported entropy and tool-trace-based EHR coverage rather than clinically validated termination criteria.

Abstract

from arXiv · show

Clinical diagnosis is an active evidence-seeking process in which clinicians acquire evidence, update competing hypotheses, and decide when the available evidence is sufficient for diagnosis. Yet many medical diagnosis systems built around large language models (LLMs) still formulate diagnosis as static case-to-answer prediction, with limited support for evidence acquisition. Agentic LLMs offer a dynamic alternative through tool use and intermediate diagnostic trajectories, but existing systems often under-specify how patient evidence should be exposed, scaffolded, and controlled at runtime. We introduce EviDx, an evidence-aware active diagnosis framework that pairs patient-specific diagnostic environments with a clinical diagnostic scaffold and an observer-guided runtime harness. In EviDx, $\mathcal{E}$-Synthesis constructs interactive environments from raw clinical cases; the scaffold organizes role-specialized agents, evidence tools, and evolving evidence states; and the harness regulates diagnostic termination by tracking uncertainty and evidence coverage. A 3-level evaluation pyramid assesses execution robustness, reasoning dynamics, and diagnostic outcomes. Experiments show that EviDx improves diagnostic performance and process stability while revealing model-dependent capability boundaries.

1 Introduction

Clinical diagnosis requires iterative evidence acquisition, hypothesis updating, and decisions about evidence sufficiency, but many LLM systems reduce it to static case-to-answer prediction. EviDx addresses this gap with interactive environments, clinical scaffolding, runtime termination control, and process-level evaluation.

  • Clinical diagnosis is an active process of acquiring evidence, refining competing hypotheses, and deciding whether further information is needed.
  • Static LLM diagnosis systems collapse evidence acquisition, integration, and termination into one prompt, under-specifying premature-commitment failures.
  • Tool access alone does not produce dynamic diagnosis because environments, scaffolds, and runtime harnesses shape evidence-seeking behavior.
  • EviDx converts raw clinical text into patient-specific interactive environments and organizes execution with roles, evidence tools, and evolving evidence states.
  • The Observer-Guided Harness tracks uncertainty and evidence coverage to support explicit diagnostic termination decisions.
  • EviDx evaluates execution robustness, reasoning dynamics, and diagnostic outcomes, with experiments reporting improved diagnostic performance and process stability.

2 Related Work

Prior clinical LLM evaluations often emphasize fixed-input answer generation, while agentic systems enable evidence retrieval and role coordination. Related work motivates active evidence seeking but leaves control over missing evidence and termination as a remaining problem.

  • Medical LLM benchmarks commonly expose fixed cases and request diagnoses in one pass, eliciting final-answer behavior rather than iterative evidence seeking.
  • Tool-augmented agents retrieve patient data, external medical knowledge, or role-specialized reasoning within interactive clinical contexts.
  • Existing approaches motivate shifting from answer generation to evidence seeking but leave decisions about missing evidence and termination insufficiently controlled.
  • Table 1 compares systems and evaluation environments by interactive evidence acquisition, clinical scaffolding, runtime control, and process-level evaluation.

3 Problem Formulation

EviDx formulates clinical LLM diagnosis as active differential diagnosis over an evolving evidence trajectory. The system acquires observations, updates competing hypotheses, and determines whether accumulated evidence supports termination.

  • Active differential diagnosis iteratively acquires evidence, updates a differential diagnosis, and decides when the trajectory is sufficient for a final answer.
  • At each step, the system takes an evidence-seeking action, receives an observation, appends it to the trajectory, and maintains a differential diagnosis set.
  • The model reports a normalized belief over competing diagnostic hypotheses while integrating evidence and refining its diagnosis.
  • Termination requires deciding whether the current trajectory is sufficient or whether additional evidence should be acquired.
  • Lower entropy indicates a more concentrated belief state, but confidence does not establish that sufficient patient-specific evidence has been acquired.

4 Proposed Methodology

EviDx combines environment synthesis, a role-structured diagnostic scaffold, an observer-guided termination harness, and a three-level trajectory evaluation. The method separates evidence acquisition from runtime termination and assesses both process and outcomes.

  • Proposed Methodology: EviDx converts static cases into active diagnostic environments, structures evidence acquisition, regulates termination, and evaluates diagnostic trajectories.
  • E-Synthesis: E-Synthesis transforms raw clinical text into structured patient data while preserving ambiguity, qualifiers, and negative findings for interactive evidence acquisition.
  • Clinical Dx Scaffold: The Clinical Dx Scaffold coordinates role-specialized components, evidence states, and workflows around a Diagnostician that maintains hypotheses, beliefs, and trajectories.
  • Clinical Dx Scaffold: The scaffold routes patient-record requests to the EHR Executor and external-evidence requests to the Clinical Consultant through standardized tool interactions.
  • Observer-Guided Harness: The Observer-Guided Harness intercepts proposed completion and requires trajectories to satisfy both diagnostic resolution and runtime evidence coverage.
  • Observer-Guided Harness: Diagnostic uncertainty uses entropy H(t), while runtime coverage V(Tt) measures the fraction of available EHR sections queried through tool calls.
  • 3-Level Evaluation Pyramid: The three-level evaluation pyramid measures execution robustness, reasoning dynamics, and diagnostic outcomes, including tool errors, uncertainty convergence, observer intervention, and evidence recall.

5 Experimental Setup

EviDx is evaluated on diverse diagnostic datasets and language models, with a curated extractive pipeline for diagnosis-critical reference evidence.

  • Datasets: The evaluation uses JAMA, MedXpertQA-Text, and a diagnostic-focused MedXpertQA-Diag subset spanning real-world cases and expert-level questions.JAMA contains 1,524 cases, MedXpertQA-Text contains 2,455 questions, and MedXpertQA-Diag contains 234 diagnostic-focused questions.
  • Models: The study evaluates frontier and smaller open-weight language models, using Qwen3-embedding-4B for dense retrieval in the Clinical Consultant.Evaluated models include GPT-5.2, Claude Sonnet 4.6, GLM-5, DeepSeek-V3.2, Ministral3-8B, Qwen3-8B, and Llama3.1-8B.
  • Reference Evidence Curation: The reference-evidence pipeline uses GPT-5.2 to plan targeted queries, retrieve supporting knowledge, and extract bottleneck evidence from each case.The extracted set is intended to capture minimal clinically relevant clues supporting the ground-truth diagnosis and distinguishing competing differentials.
  • Reference Evidence Curation: Final annotations are constrained to exact substrings of the source case, then a subset undergoes physician review for clinical plausibility and evidence relevance.The research team also checks adherence and validity after automated extraction.

6 Results & Analysis

EviDx is assessed across execution robustness, diagnostic outcomes, and cognitive dynamics. Results show model-dependent tool-use reliability, improved multiple-choice performance and stability, and harness-mediated prevention of unsupported termination.

  • 6.1 Execution Robustness: GPT-5.2 and DeepSeek-V3.2 complete evaluated runs without syntax or schema failures, whereas smaller open-weight models accumulate more schema errors.Llama3.1-8B exceeds 500 failure events, while Claude Sonnet 4.6 and GLM-5 occasionally fall back to natural language instead of native tool calling.
  • 6.2 Main Outcomes: EviDx improves multiple-choice diagnostic performance and process stability, with gains clearest for 8B-parameter open-weight models.On JAMA, Qwen3-8B rises from 18% to 44%, while Llama3.1-8B rises from 17% to 31%.
  • 6.2 Main Outcomes: Open-ended diagnosis remains limited for small models, with scores near 0% in several settings despite improved evidence acquisition and hypothesis elimination.The results indicate that the scaffold and harness cannot fully compensate for limitations in parametric medical knowledge.
  • 6.2 Main Outcomes: Reference Evidence Recall exposes a tension between answer accuracy and trace-level evidence matching, including a Cognition score drop for Llama3.1-8B on JAMA.The score decreases from 63.8% with the Single Agent to 15.1% with EviDx.
  • 6.3 Cognitive Dynamics and Process Control: The Observer-Guided Harness checks diagnostic uncertainty and evidence coverage before approving termination, while leaving next-action selection to the scaffolded Diagnostician.Without termination control, agents can show spurious entropy drops and reach premature conclusions without adequate evidence.
  • 6.3 Cognitive Dynamics and Process Control: For Ministral3-8B, the harness blocks an incorrect exit at H = 1.16 and continued evidence acquisition leads to the correct diagnosis at H = 0.88.The case trajectory was initially misled by a negative scan before additional evidence acquisition.

7 Conclusion

EviDx frames clinical LLM diagnosis as an evidence-aware process shaped by environments, scaffolds, and runtime harnesses. Across its evaluation pyramid, it improves multiple-choice performance and process stability, while open-ended diagnosis remains constrained by base-model medical knowledge.

  • Conclusion: EviDx converts static cases into interactive environments, organizes evidence acquisition through the Clinical Dx Scaffold, and regulates termination with the Observer-Guided Harness.The harness tracks uncertainty and runtime evidence coverage.
  • Conclusion: Across execution robustness, reasoning dynamics, and outcomes, EviDx improves multiple-choice diagnostic performance and process stability.The framework evaluates not only final answers but also evidence acquisition, trajectory maintenance, and termination decisions.
  • Conclusion: Open-ended diagnosis remains constrained by base-model medical knowledge, indicating that evidence-aware scaffolding does not remove all capability boundaries.The conclusion motivates evaluating how agents acquire evidence and decide when evidence is sufficient, rather than relying only on final answers.

Limitations

EviDx is evaluated in controlled patient-specific environments synthesized from established clinical benchmarks, enabling trajectory-level analysis. Its current runtime controls and benchmark curation leave clear scope for richer settings and broader expert validation.

  • Scope: EviDx studies active diagnosis in patient-specific environments synthesized from established clinical benchmarks.This controlled setting supports analysis of evidence acquisition, termination behavior, and diagnostic outcomes.
  • Future scope: The environment-construction process can be extended to multimodal evidence, longitudinal EHR trajectories, patient-clinician interaction, and deployment-specific workflows.
  • Dataset curation: Med-Evidence-2.6k uses LLM-assisted curation, programmatic validation, and physician plausibility review on a stratified subset.Larger-scale blinded expert annotation is identified as a natural extension.
  • Runtime control: The Observer-Guided Harness currently relies on operational process signals designed for controlled research environments.Its entropy measure uses model-reported belief distributions, while runtime evidence coverage uses EHR sections accessed through tool-call traces.

Ethical Considerations

EviDx is presented as a research framework rather than a deployable replacement for professional medical judgment. The paper highlights human oversight, potential demographic and geographic disparities, and privacy protections for its benchmark data.

  • Safety and oversight: EviDx is strictly intended for research and is not ready to replace professional medical judgment or deploy in real-world healthcare without human oversight.The authors state that the harness does not guarantee clinical safety or factual correctness.
  • Safety and oversight: Autonomous diagnostic agents must operate under a strict human-in-the-loop paradigm.
  • Data governance: The experimental datasets are derived from publicly available clinical benchmarks and are described as not containing sensitive Protected Health Information.Med-Evidence-2.6k is slated for release under non-commercial academic licenses.
  • Fairness: Diagnostic agents might perform differently across demographics, genders, or socioeconomic groups because LLMs encode biases from their pre-training corpora.The paper calls for rigorous fairness auditing before real-world deployment.
  • Generalizability: Clinical knowledge retrieved by the Clinical Consultant may primarily reflect Western clinical guidelines, limiting generalizability to global healthcare contexts.

A.2 Failure Analysis: Execution Failure

The Llama3.1-8B failure case exposes both execution fragility in strict tool-calling and a capability boundary: evidence gathering succeeded, but the final diagnosis remained incorrect.

  • Execution Failure: Llama3.1-8B repeatedly produced malformed or conversational tool calls, causing a schema-collapse failure loop that system feedback eventually recovered.The trajectory records malformed JSON, missing valid tool calls, and two of five retry attempts failing.
  • Reasoning Control: The Observer-Guided Harness denied premature termination because runtime evidence coverage was low and prompted retrieval of family history.The observer reported V(t) = 0.12 ≤ 0.40 and requested family-history and xanthoma checks.
  • Outcome Failure: Despite acquiring additional clinical clues, the agent diagnosed Familial Hypercholesterolemia instead of the ground-truth Dysbetalipoproteinemia.The recorded final diagnosis was Familial Hypercholesterolemia, while the ground truth was option D, Dysbetalipoproteinemia.
  • Capability Boundary: The case indicates that process control can enforce evidence gathering but cannot supply nuanced medical knowledge absent from the foundation model.The authors attribute the error to the 8B model’s inability to distinguish the two diagnoses.

B Finite-Sample Stability

Stratified paired bootstrap analysis finds positive average gains for EviDx across all four comparisons, with the strongest and most consistent improvements in multiple-choice accuracy over the Single Agent baseline.

  • Aggregate Stability: Positive average gains occurred in all four comparison groups, with the largest and most consistent improvements in MC accuracy against the Single Agent baseline.Points represent paired accuracy gains in percentage points, with intervals showing 95% confidence intervals.
  • Aggregate Stability: Observer-specific gains were smaller and more model- and dataset-dependent than the broader EviDx improvements.The comparison is EviDx versus EviDx without the Observer-Guided Harness; the harness targets premature termination rather than replacing the evidence-acquisition scaffold.
  • Aggregate Stability: Open-ended diagnosis showed positive average shifts but wider uncertainty and stronger dependence on the base model’s medical reasoning capability.This dependence is consistent with the reported limitation of open-ended diagnosis for smaller models.
  • Per-Dataset Stability: MC gains over the Single Agent baseline recur across datasets, whereas observer-specific and open-ended gains are more heterogeneous.Per-dataset confidence intervals provide the corresponding uncertainty estimates, with intervals excluding zero indicating more stable positive differences.
  • Per-Dataset Stability: On JAMA, MC gains over the Single Agent baseline appear across models, while observer-specific gains are selective.The JAMA table identifies confidence intervals excluding zero for Qwen3-8B in MC and GLM-5 in open-ended diagnosis.

D.5 Quality Assurance and Limitations

The evidence-annotation pipeline combines extractive constraints, source-text verification, and physician review to keep curated evidence grounded and clinically useful.

  • Quality Assurance: The pipeline restricts annotations to exact substrings from the original case text, reducing unsupported evidence synthesis and preserving verifiability.The intermediate retrieval step grounds the reasoning bridge in medical references.
  • Quality Assurance: Programmatic checks verify schema adherence, substring validity, and consistency between extracted spans and the source case text.Audit packages highlight each span against the original case for review.
  • Physician Review: Physicians audited 515 unique evidence spans across 100 stratified cases, with 1,419/1,545 judgments rating spans clinically relevant and diagnostically useful.Three physicians independently reviewed the same audit package.

E Implementation Details

EviDx uses deterministic retrieval and controlled agent settings, while the Observer applies dynamic uncertainty and evidence-coverage thresholds to regulate termination.

  • Serving Environment: Local 8B models and embeddings run through vLLM, while frontier models are accessed through external serving infrastructure.The passage names the local deployment framework and distinguishes it from frontier-model access.
  • Agent Configuration: Generation temperature is 0.0 for the Diagnostician, Context Initializer, and LLM-Judge, while the Clinical Consultant uses 0.1 and 0.3 for query decomposition and generation.The EHR Executor is deterministic and not LLM-based.
  • Observer Configuration: The Observer permits at most 50 interaction steps and uses step-aware dynamic thresholds rather than rigid static boundaries.The thresholds balance exhaustive exploration with practical termination.
  • Observer Configuration: After the fifth step, the uncertainty threshold relaxes, and after the tenth step, the evidence-coverage requirement decreases to a minimum of 0.1.These changes are intended to prevent deadlocking when clinical evidence is sparse.
  • Clinical Consultant: The Clinical Consultant uses MedRAG medical corpora with hybrid retrieval tailored to the underlying corpus structure.The setup includes StatPearls and medical textbooks, with dense semantic retrieval for the structured StatPearls database.
Loading 2608.24570v1…