PaperlayerView exact revision on arXiv

Source-linked AI summary

CDEG: Learning Decision-Critical Evidence for Long-Horizon Diagnostic Agents

Xiwei Dai, Zijie Meng, Zhiting Fan, Yixuan Tang, Ziru Niu, Zuozhu Liu

arXiv:2608.22899v1cs.AI

TL;DR

Long-horizon diagnostic agents must acquire and incorporate decision-critical evidence across multiple interactions, but trajectory reuse may retain incidental information without identifying what drives diagnostic decisions. CDEG validates such evidence through controlled counterfactual interventions, organizes it into a structured graph, and uses evolving evidence states to guide acquisition or reappraisal. Across in-domain and out-of-distribution benchmarks with multiple backbones, it consistently improves performance, with gains of up to 11.5% over vanilla agents.

  • Problem

    Long-horizon diagnostic agents may fail to acquire critical evidence or adequately incorporate available observations, while reused trajectories mix decision-relevant and incidental information.

  • Method

    CDEG contrasts successful and failed trajectories from the same case, counterfactually validates candidate evidence, and organizes diagnosis–evidence–action relations into a graph for state-guided intervention.

  • Results

    CDEG improves average accuracy by 7.63% on MIMIC-IDx and 8.88% on AgentClinic, with gains of up to 11.5% over vanilla agents.

  • Takeaways & Limitations

    Evidence-level learning provides a more reliable basis for reusing diagnostic experience than relying on trajectory-level experience alone.

Abstract

from arXiv · show

Unlike static medical question answering, long-horizon diagnosis captures the sequential nature of clinical practice: evidence is progressively acquired, integrated, and evaluated over multiple rounds of interaction before reaching a final diagnosis. However, existing doctor agents often fail when critical evidence is either not acquired or not adequately incorporated into diagnostic reasoning. Recent agentic approaches attempt to address these failures by reusing historical trajectories or distilled memories. But their diagnostic gains remain constrained because such experience may contain noisy or incidental information and is typically reused without validating which evidence actually drives diagnostic decisions. To address this limitation, we introduce CDEG, a graph-based framework that learns reusable decision-critical evidence from historical diagnostic trajectories. CDEG contrasts successful and failed trajectories from the same case to identify candidate evidence, validates their diagnostic impact through controlled counterfactual interventions, and organizes the resulting diagnosis--evidence--action relations into a structured graph. During inference, CDEG tracks the evolving patient evidence state to retrieve relevant diagnostic relations and selectively guide missing evidence acquisition or overlooked evidence reappraisal. Across in-domain and out-of-distribution benchmarks with multiple doctor agent backbones, CDEG consistently improves diagnostic performance, achieving up to an 11.5% accuracy gain over vanilla agents. These results demonstrate that reliable long-horizon diagnosis requires moving beyond trajectory-level experience reuse toward evidence-level learning of the factors that truly shape clinical decisions.

Introduction

Long-horizon diagnosis requires agents to acquire and integrate evidence across interactions, yet they may miss relevant evidence or overlook observations already available. CDEG addresses this by learning decision-critical evidence from trajectories and using it to guide evidence acquisition or reappraisal.

  • Long-horizon diagnosis requires acquiring and integrating clinical evidence under partial observability before making a final diagnosis.
  • A fundamental challenge is identifying and using evidence that truly distinguishes among diagnoses.
  • Historical trajectories mix decision-relevant evidence with incidental observations, while outcomes do not identify which evidence shaped the diagnosis.
  • CDEG contrasts failed and successful trajectories from the same case, validates candidate evidence through controlled counterfactual tests, and stores diagnosis–evidence–action relations in a graph.
  • During inference, CDEG matches the evolving evidence state to the graph to guide acquisition of missing evidence or reappraisal of overlooked observations.
  • CDEG improves average accuracy by 7.63% on MIMIC-IDx and 8.88% on AgentClinic, with gains of up to 11.5%.

Related Work

Related work studies interactive diagnosis, trajectory-based experience reuse, and counterfactual reasoning for clinical evidence. These lines of work motivate CDEG’s focus on evidence acquisition, diagnostic reasoning, and controlled evidence interventions.

  • Long-Horizon Interactive Diagnosis: Long-horizon interactive diagnosis uses multi-turn patient interaction and diagnostic tools to acquire and integrate evidence before final diagnosis.
  • Long-Horizon Interactive Diagnosis: Recent methods improve diagnostic reasoning through targeted questioning, test selection, and iterative diagnosis.
  • Learning from Diagnostic Trajectories: Agent methods reuse prior trajectories as exemplars or distill them into reflections, lessons, workflows, skills, or reasoning memories.
  • Counterfactual Reasoning in Medical Diagnosis: Counterfactual medical-diagnosis methods modify clinical evidence or diagnostic contexts to study changes in diagnostic preference, prediction, or explanation.

Methods

CDEG models interactive diagnosis as sequential evidence acquisition under partial observability, then learns and retrieves validated diagnosis–evidence–action relations from paired successful and failed trajectories.

  • Problem Setup: The doctor agent progressively acquires observations through patient interaction and diagnostic tools before executing a final diagnostic action.The environment initially reveals limited information, while non-final actions append observations to the interaction history.
  • Candidate Evidence Mining: CDEG mines candidate evidence by comparing successful and failed trajectories from the same patient case, pairing similar action–observation histories.Observations are normalized into UMLS-grounded evidence atoms, while candidate sets distinguish missing, ignored, and blocking evidence.
  • Candidate Evidence Mining: Three candidate sets separate evidence acquired only in successful trajectories, shared evidence, and evidence appearing only in failed trajectories.These correspond respectively to missing-evidence, ignored-evidence, and blocking-evidence candidates.
  • Counterfactual Validation: Counterfactual validation retains candidates only when a controlled intervention changes diagnostic preference while all other evidence remains unchanged and diagnosis order is reversed.Completion, reappraisal, and blocking tests respectively add missing evidence, expose shared evidence, or add failed-trajectory evidence to successful trajectories.
  • Graph Consolidation: The graph organizes diagnoses, validated evidence, and acquisition actions into nodes and relations, adding diagnosis-revision edges only when counterfactual validation supports the shift.The graph includes action-to-evidence, evidence-to-diagnosis, and directed diagnosis-revision relations.
  • Inference-Time Retrieval: At inference, CDEG maps new observations into an evolving evidence state and retrieves high-coverage diagnostic edges to guide further evidence acquisition or reappraisal.Retrieval uses the current evidence state and applies a predefined coverage threshold before guidance is issued.

Experimental Setup

CDEG is evaluated on in-domain and out-of-distribution interactive diagnosis benchmarks using multiple doctor-agent backbones, baselines, and complementary diagnostic metrics.

  • Benchmarks: MIMIC-IDx provides the in-domain benchmark, while AgentClinic’s MedQA subset provides the out-of-distribution evaluation.The CDEG graph is constructed only from the MIMIC-IDx graph-construction split; held-out MIMIC-IDx and AgentClinic are used for evaluation.
  • Baselines: The comparison includes Vanilla Agent, Self-Reflection, Trajectory RAG, Flat Experience, and CDEG under matched patient cases.Self-Reflection revisits current reasoning, Trajectory RAG retrieves successful historical trajectories, and Flat Experience uses unstructured distilled experience.
  • Metrics: Accuracy and Similarity measure diagnostic quality, while Correction and Regression measure how enhanced agents alter vanilla-agent decisions.Accuracy assesses clinical consistency with reference diagnoses, whereas Similarity measures semantic proximity; Correction and Regression track fixes and harmful changes.
  • Implementation: Graph construction samples K = 4 trajectories per MIMIC-IDx case and pairs each failed trajectory with its most similar successful trajectory.Gemini-3.1-Pro and Gemini-3-Flash-Preview generate trajectories, while Qwen3-Embedding-0.6B performs trajectory encoding and retrieval.
  • Implementation: Inference evaluates Gemini-3-Flash-Preview, Gemini-3.1-Pro, GPT-5.6-Luna, and Claude-Sonnet-5 as doctor-agent backbones.New observations and proposed diagnoses are mapped to graph nodes using hybrid exact and SapBERT-based semantic matching.

Main Results

CDEG consistently improves diagnostic performance across all evaluated doctor-agent backbones and both benchmarks. Its gains transfer to an out-of-distribution benchmark and backbones not used for graph construction.

  • Accuracy: 7.63% average accuracy gain occurs on MIMIC-IDx and 8.88% on AgentClinic across doctor-agent backbones.Trajectory RAG and Flat Experience correct some vanilla-agent errors but do not produce consistent accuracy gains.
  • Transfer: CDEG improves GPT-5.6-Luna and Claude-Sonnet-5 despite graph construction using only Gemini-generated trajectories.The gains also transfer to AgentClinic, the out-of-distribution evaluation set.
  • Mechanism: Counterfactual validation and evidence-state organization support selective acquisition and reappraisal rather than unvalidated retrieval of historical experience.CDEG retains evidence only when intervention changes diagnostic preference and stores validated diagnosis–evidence–action relations.

Ablation Studies

Ablations show that counterfactual validation is central to CDEG’s gains, while broader evidence coverage and adaptive intervention further improve diagnostic correction. More graph-construction data also strengthens correction and related metrics.

  • Counterfactual Validation for Structured Experience: Removing counterfactual validation lowers Accuracy from 53.00% to 43.50%, below Flat Experience at 48.50%.The result indicates that graph structuring alone does not account for CDEG’s gains.
  • Counterfactual Validation for Structured Experience: Retaining evidence that changes diagnostic preference excludes noisy or incidental trajectory differences from the graph.Counterfactual validation concentrates diagnostically useful information in reusable diagnostic relations.
  • Different Candidate Evidence Sets: Removing either missing or ignored evidence reduces Correction from 33.33% to 25.64%, whereas removing blocking evidence produces a smaller decline to 30.77%.Missing and ignored evidence broaden coverage across evidence states; blocking evidence helps identify boundary conditions for revisions.
  • Evidence Selective Intervention: Reappraisal Only improves Correction by 10.25% over Acquisition Only, but both underperform complete CDEG.Selecting acquisition or reappraisal according to the current evidence state captures opportunities missed by either intervention alone.
  • Diagnostic Experience Scaling: Increasing graph-construction data from 10% to 100% raises Correction from 19.66% to 33.33%, with Accuracy and Similarity improving in parallel.Additional trajectories broaden coverage of counterfactually validated diagnostic revision relations.

Analysis

An audit finds that most vanilla-agent errors involve incomplete acquisition or insufficient use of decision-critical evidence. CDEG improves correction and stability while adding only limited interaction overhead through evidence-state-guided interventions.

  • Evidence States at Diagnostic Failure: 88.03% of 117 vanilla-agent errors arise from incomplete evidence acquisition or insufficient incorporation of already available evidence.Only 11.97% occur under conflicting evidence, and CDEG targets both dominant failure modes.
  • Evidence-State Intervention and Diagnostic Stability: CDEG records the lowest average Regression at 17.70% on MIMIC-IDx and 14.62% on AgentClinic while maintaining higher Correction.Evidence-state intervention checks whether a retrieved revision applies before triggering acquisition or reappraisal.
  • Diagnostic Gains Beyond Longer Interaction: CDEG improves diagnostic performance with only 1.41 additional turns on MIMIC-IDx and 1.27 on AgentClinic.The same pattern appears in tool-call counts, reflecting targeted rather than indiscriminate interaction.
  • Diagnostic Gains Beyond Longer Interaction: On AgentClinic with GPT-5.6-Luna, CDEG improves Accuracy by 11.22% while reducing average interaction length by 0.71 turns.This suggests less ineffective exploration and faster arrival at an evidence state sufficient for diagnosis.

Conclusion

CDEG learns reusable decision-critical evidence by validating it through successful–failed trajectory contrasts and controlled counterfactuals. Its structured graph guides evidence acquisition or reappraisal as the patient state evolves, improving diagnostic performance across ID and OOD benchmarks.

  • CDEG contrasts successful and failed diagnostic trajectories, validates candidate evidence through controlled counterfactual interventions, and organizes validated relations into a structured graph.
  • The graph guides acquisition of missing evidence or reappraisal of existing evidence according to the evolving patient evidence state.
  • Across ID and OOD benchmarks with four doctor-agent backbones, CDEG consistently improves vanilla-agent diagnostic performance and achieves the best performance.
  • Future work will explore online graph evolution for continual experience validation and extension to broader clinical scenarios.
Loading 2608.22899v1…