PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriDx: Earning the Right to Diagnose with Disease-Centric Verification

Zhong Cao, Shuying Chen

arXiv:2609.14018v1cs.AIcs.CL

TL;DR

Medical LLM evaluations often emphasize final answers or local reasoning steps, while disease hypotheses create further clinical obligations that need checking. VeriDx links free-form reasoning to structured disease profiles and tracks whether those obligations are satisfied, unresolved, or violated. The authors report that profiles and verification mechanisms contribute to more stable hypothesis refinement and reduce the risk of premature closure.

  • Problem

    Medical LLM evaluations often emphasize final answers, local steps, or isolated facts rather than whether each disease hypothesis’s clinical obligations are fulfilled.

    Source: §1 · p. 1

  • Method

    VeriDx links free-form diagnostic trajectories to structured disease profiles and labels each hypothesis’s obligations satisfied, unresolved, or violated.

    Source: §1 · p. 2

  • Results

    Ablation results indicate that disease profiles and verification mechanisms contribute to more stable hypothesis refinement and reduce the risk of premature closure.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The framework shifts clinical reasoning evaluation beyond final-answer accuracy toward auditable, revisable, disease-grounded reasoning.

    Source: §1 · p. 3

    VeriDx has been validated only in respiratory medicine, so its empirical generalizability to other specialties remains unestablished.

    Source: §7 · p. 9

Abstract

from arXiv · show

A correct diagnosis can still be reached for the wrong reasons. In clinical reasoning, every disease hypothesis creates obligations: key evidence must be checked, alternatives must be ruled out, contradictions must be resolved, useful tests must be considered, and closure must be justified. Current evaluations of medical LLMs mostly focus on final answers, local steps, or isolated facts, and therefore miss these hypothesis-induced commitments. We introduce \textbf{VeriDx}, a disease-centric verification framework that links free-form diagnostic reasoning to structured disease profiles. VeriDx tracks whether each hypothesis is satisfied, unresolved, or violated its clinical obligations, exposing failures such as missing critical tests, unresolved differentials, ignored contradictions, unsupported claims, and premature closure. We instantiate VeriDx for complex respiratory diagnosis using guideline-derived disease profiles and expert-annotated longitudinal cases. Our results show that many diagnostic errors are not isolated mistakes, but broken commitments made earlier in the reasoning process.

1 Introduction

VeriDx reframes diagnostic evaluation around whether each disease hypothesis fulfills its clinical obligations, not merely whether the final label is correct. It links free-form reasoning to disease profiles to make these obligations auditable.

1 Introduction

  • Medical LLMs can miss critical evidence, choose inappropriate examinations, fixate on early hypotheses, overlook contradictions, and close cases prematurely.As answer generation improves, the authors argue that reasoning errors become harder to detect.Source: §1 · p. 1
  • Existing benchmarks assess multi-step diagnosis and process failures, but mostly score cases, stages, or next actions rather than each hypothesis’s commitments.Prior methods also leave a gap between rigid clinical rules and general verification that does not specify what a particular disease hypothesis requires.Source: §1 · pp. 1–2
  • A correct diagnosis may still skip a discriminative test, leave a dangerous alternative unresolved, or ignore contradictory evidence.The paper argues that final-answer correctness alone does not establish that a diagnostic process was clinically responsible.Source: §1 · pp. 1–2
  • VeriDx maps diagnostic hypotheses and reasoning claims to disease-specific obligations, labeling each satisfied, unresolved, or violated.Profiles encode required evidence, exclusion criteria, differentials, discriminative examinations, contradictions, treatment constraints, and closure conditions.Source: §1 · p. 2
  • The authors construct a respiratory resource from 2,563 guidelines and reference documents and 188 expert-annotated complex cases.The cases include 1,455 medical images and three-stage diagnostic trajectories averaging 20k–30k tokens per case.Source: §1 · p. 3
  • The contributions include disease-centric verifiable reasoning, the VeriDx framework, and a guideline-derived respiratory disease-profile library evaluated on longitudinal pulmonary cases.The evaluation aims to extend beyond final-answer accuracy toward auditable, revisable, disease-grounded reasoning.Source: §1 · pp. 1, 3

2 Related Work

Prior work advances process-level evaluation, knowledge-grounded reasoning, and clinical verification. VeriDx differentiates itself by checking the obligations induced by each disease hypothesis.

Medical LLMs and clinical reasoning evaluation

  • Benchmarks evaluate multi-step diagnosis, examination recommendations, hypothesis refinement, and longitudinal interaction, but remain mostly case- or step-oriented.They expose process failures such as missed evidence, weak differentials, overlooked contradictions, and premature closure (Zhu et al., 2026; Chiu et al., 2025).Source: §2 · p. 3

Retrieval-augmented and knowledge-grounded reasoning

  • Retrieval and knowledge-organization methods improve access to or representation of clinical knowledge, but do not ensure that disease-specific tests, alternatives, and contradictions are addressed.VeriDx complements these approaches by tracking each disease hypothesis’s obligations as satisfied, unresolved, or violated.Source: §2 · p. 3

Structured clinical knowledge and LLM verification

  • Structured clinical systems and general LLM verifiers improve auditability or statement-level reliability, yet do not specify what a particular disease hypothesis requires checking.VeriDx combines flexible LLM reasoning with disease-specific profiles to assess those clinical commitments.Source: §2 · pp. 3–4

3 Data

VeriDx uses a guideline-derived profile library to define diagnostic obligations and a separate longitudinal pulmonary benchmark to evaluate reasoning as evidence evolves.

Guideline-derived disease profiles

  • The disease-profile library derives from 2,563 respiratory guidelines and reference documents totaling approximately 3.93 GB, covering nearly all pulmonary diseases.Profiles encode required evidence, important mimics, discriminative tests, and closure conditions; physicians reviewed them for clinical validity and discriminative value.Source: §3 · p. 4

Evaluation benchmark

  • The benchmark contains 188 complex, de-identified real-world cases organized into three-stage trajectories with progressive evidence disclosure.It includes 1,128 task instances across six scored sections and 1,455 stage-disclosed medical images, supporting text-only and multimodal evaluation.Source: §3 · p. 4
  • The benchmark preserves incomplete early evidence, examination results, and later diagnostic revisions rather than exposing only a completed diagnostic trajectory.Each case has nine structured sections, and models access only the evidence available at the current stage.Source: §3 · p. 4

4 Method

VeriDx links staged diagnostic trajectories to disease profiles and checks whether each hypothesis fulfills its clinical obligations. Its evaluation spans the diagnostic workflow, from hypothesis generation through treatment planning.

4.1 Overview

  • VeriDx links each model-generated disease hypothesis in a staged trajectory to a structured profile and checks whether its obligations are satisfied, unresolved, or violated.This shifts evaluation beyond final-diagnosis correctness to whether the reasoning process supports making that diagnosis.Source: §4.1 · p. 4

4.2 Longitudinal Diagnostic Task

  • The longitudinal task progressively reveals an initial presentation, new examination results, and later confirmatory evidence across three stages.The trajectory records initial and updated differentials and examination plans, the final diagnosis, its basis, and the treatment plan; reference annotations distinguish required from optional items.Source: §4.2 · pp. 4–5

4.3 Disease Profiles

  • Each disease profile represents hypothesis-activated obligations, including phenotype, supporting and opposing findings, mimics, missing information, critical examinations, and closure conditions.Profiles encode default obligations rather than deterministic decisions; patient-specific evidence at each stage can make an obligation satisfied, unresolved, or violated.Source: §4.3 · pp. 5–6

4.4 Disease-Centric Verification

  • For each matched hypothesis-profile pair, VeriDx checks evidence, contradictions, mimics, critical tests, and closure, treating unobserved relevant information as unresolved rather than negative.It labels obligations satisfied, unresolved, or violated and identifies six violation patterns: missing critical tests, unresolved mimics, ignored contradictions, unsupported claims, premature closure, and unsafe treatment implications.Source: §4.4 · p. 6

4.5 Evaluation

  • The D1–D6 evaluation covers hypothesis generation and revision, investigation planning, diagnostic precision, evidence grounding, and treatment safety.The experiments compare complete-information prediction with stepwise reasoning and the full framework with variants lacking selected verification components.Source: §4.5 · p. 6

5 Results

Frontier models perform strongly with complete information but struggle during sequential reasoning. VeriDx’s combined verification performs best overall, and ablations reveal distinct contributions from its profiles and checking modules.

5.1 Frontier LLMs Struggle with Sequential Clinical Reasoning

  • Models perform strongly with complete clinical information but deteriorate in sequential reasoning, including unstable differential refinement despite sufficient information for judgment at each stage.Intermediate reasoning can remain poor even when endpoint performance is relatively acceptable.Source: §5.1 · pp. 6–7

5.2 VeriDx Improves Clinical Reasoning

  • The full VeriDx framework achieves the strongest performance across nearly all reasoning dimensions, combining DDx verification’s gains in dynamic revision and precision with Exam verification’s gains in investigation planning.The two modules provide complementary benefits; reasoning transparency changes relatively little across the four settings.Source: §5.2 · p. 7, Figure 3 · p. 8
  • The results provide evidence that VeriDx generalizes across model families.Source: §5.2 · p. 8

5.3 Ablation Study

  • Removing DDx profiles leads to the broadest degradation, while removing phenotype, DDx, or closure checks affects distinct reasoning dimensions.Phenotype checking mainly affects precision; DDx checking affects transparency and treatment planning; closure checking weakens dynamic reasoning and investigation planning.Source: §5.3 · p. 8
  • Specialist-built profiles modestly improve most dimensions over automatically extracted profiles, while initial hypothesis generation remains unchanged.The automatically extracted profiles already preserve most clinical structure needed for downstream verification.Source: §5.3 · pp. 8–9

6 Discussion

VeriDx reframes clinical reasoning evaluation around the obligations created by disease hypotheses, rather than answer accuracy alone. Its discussion presents structured disease profiles as a bridge between symbolic clinical knowledge and flexible LLM reasoning, while noting that broader specialty generalizability remains untested.

Redefining Diagnostic Evaluation

  • Diagnostic errors may arise from unresolved or violated commitments that propagate through the diagnostic trajectory, not only from isolated hallucinations.Without structural guidance, models may fail to gather discriminative evidence, leave important mimics unresolved, or close before sufficient evidence is obtained.Source: §6 · p. 9
  • VeriDx shifts evaluation from answer-centric accuracy to disease-centric verification of phenotype support, important mimics, and justified diagnostic closure.Ablations show disease profiles and verification mechanisms contribute to more stable hypothesis refinement and reduce the risk of premature closure.Source: §6 · p. 9

A Neuro-Symbolic Bridge for Clinical AI

  • Structured disease profiles anchor LLM interpretation of longitudinal trajectories and patient-specific evidence, while preserving language understanding for complex narratives.The design provides an explicit audit trail for disease-specific obligations.Source: §6 · p. 9

Beyond Respiratory Medicine

  • VeriDx is not tied to a particular disease, but its empirical generalizability beyond respiratory medicine remains unestablished.The profile structure may serve as a template for other domains, but that possibility requires further empirical investigation.Source: §6 · p. 9

7 Limitations

VeriDx has limitations in its LLM-dependent verification pipeline, expert-curated profiles, respiratory-only validation, and stepwise evaluation design. Its outputs are decision-support signals rather than autonomous diagnoses, and real-world use requires further safeguards and validation.

7 Limitations

  • LLM-based claim extraction and obligation evaluation can cascade early errors when complex clinical contradictions are parsed incorrectly.Disease profiles also require physician oversight to calibrate the clinical weight of mimics and required tests.Source: §7 · p. 9
  • VeriDx has been validated only in respiratory medicine; extension requires specialty-specific guidelines, longitudinal benchmarks, and expert annotation.Its empirical generalizability to other specialties remains untested.Source: §7 · p. 9
  • Progressive disclosure divides diagnosis into predefined steps, whereas real workflows are more continuous and asynchronous.Future work is intended to support asynchronous evidence integration as lab or imaging results arrive.Source: §7 · p. 9

A Ethical Considerations

  • VeriDx should not be used autonomously, and biases in guidelines, case selection, annotation, or LLM judgments may affect its verification results.Outputs should be interpreted as clinical decision-support signals rather than definitive medical conclusions.Source: A Ethical Considerations · p. 11
  • Real-world use would require institutional review, privacy protection, bias auditing, prospective validation, and continuous monitoring with clinical experts.In this study, clinicians reviewed the disease profiles and benchmark annotations.Source: A Ethical Considerations · pp. 11–12

B.1 Data Processing

The appendices describe how longitudinal cases are represented and exposed to models in stages, then detail disease-profile verification and how its changes carry into later stages. They also specify the prompts and safeguards used to constrain generation and evidence extraction.

B.1 Data Processing

  • Cases use sections S1–S9 to organize presentation, reference differentials and workups, staged evidence, final diagnosis, and treatment response.Reference diagnoses and plans are extracted from their corresponding sections as lists for semantic evaluation.Source: §B.1 · p. 12

B.2 Closed-Loop Generation Protocol

  • The closed-loop protocol starts from the initial presentation, updates the differential and follow-up plan after ordered-test results, then uses later evidence for final diagnosis and treatment outputs.Generated early histories can affect later evidence and decisions; generated treatment outcomes are reference-only and not evaluated.Source: §B.2 · p. 12

B.3 Leakage Control

  • Before Stage 1, explicit final-diagnosis leakage in the initial presentation is checked and any leaked phrase is removed.The cleaned presentation becomes R1, preventing the model from solving by copying an explicit final label.Source: §B.3 · p. 12

B.4 Generation Prompts

  • Prompts constrain evidence extraction to results matching ordered tests and provide prior reasoning and findings as inputs to later stages.Stage 1 generates the initial assessment, Stage 2 updates the differential and plan, and Stage 3 outputs final diagnosis, basis, and treatment fields.Source: §B.4 · pp. 12–13

B.5 Disease-Profile Verification

  • Disease profiles specify phenotype, supportive and opposing findings, mimics, missing information, critical tests, and verifier notes for each candidate diagnosis.The verifier records supported, contradicted, or unresolved core phenotype status; unresolved evidence is not treated as negative evidence.Source: §B.5 · p. 13

B.6 Verification Hints Across Stages

  • Verifier changes are passed to later prompts as concise hints, so excluded or added diagnoses and tests can influence the downstream trajectory.Removed diagnoses are marked as excluded by strong counter-evidence, while added diagnoses and tests are marked as verifier additions.Source: §B.6 · p. 13

B.7 Ablation Conditions

The appendix isolates diagnosis-space changes from examination-plan changes, then describes the structured judging, workflow dimensions, safety events, and aggregation used for comparisons. These procedures define what each ablation retains and how outputs are scored and reported.

B.7 Ablation Conditions

  • The full ablation modifies both differential diagnoses and examinations, while DDx-only and Exam-only retain one modification type and restore the other baseline component.These conditions isolate hypothesis-space filtering from workup modification.Source: §B.7 · pp. 13–14

B.8 LLM-as-Judge Evaluation

  • Three structured judge calls use clinical semantic matching to assess differential quality, examination planning, and final diagnosis, basis, and treatment.The judgments allow equivalent medical expressions while penalizing vague, broad, unsupported, or hallucinated outputs.Source: §B.8 · p. 14

B.9 Evaluation Dimensions

  • The evaluator aggregates judge outputs into six normalized workflow dimensions, from initial hypothesis generation through treatment planning quality and safety.The dimensions cover dynamic reasoning, investigation planning, final precision, and transparency and evidence grounding.Source: §B.9 · p. 14

B.10 Safety Metrics

  • Safety reporting includes binary events such as dangerous diagnostic misses, over-investigation, hallucinations, and harmful treatment; lower event rates indicate safer behavior.The event rate is calculated using whether event e occurs in each of N evaluated cases.Source: §B.10 · p. 14

B.11 Aggregation and Reporting

  • Case-level score logs and aggregate reports record mean D1–D6 scores and safety-event rates, comparing baseline, full verification, DDx-only, and Exam-only runs.Cases still failing judge evaluation after repeated attempts are marked missing and excluded from scored aggregates.Source: §B.11 · p. 14

B.12 Cross-Model Generalization

Across GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.6, VeriDx’s modules show complementary gains, and the combined framework performs best across most dimensions. This pattern supports generalization beyond GPT-5.5.

B.12 Cross-Model Generalization

  • DDx-only verification improves dynamic diagnostic revision (D2), while Exam-only verification provides its largest gains in investigation planning (D3).Source: §B.12 · p. 14
  • The complete VeriDx framework achieves the best performance on D1, D2, and D4 for all three models, and strongest treatment-planning performance for GPT-5.5 and Claude-Opus-4.6.Source: §B.12 · p. 14
  • VeriDx achieves the strongest overall performance across most dimensions, although Exam-only verification slightly outperforms it on D3 and D5.The table reports mean scores and standard deviations across evaluation cases, with bold indicating the best result within each model family.Source: §B.12 · pp. 14–15, Table 3 · p. 15
  • The complementary module pattern appears across GPT, Gemini, and Claude, so the observed improvements are not restricted to GPT-5.5.Source: §B.12 · pp. 14–15

C Additional Details for Responsible Research

The responsible-research materials document the respiratory resources, their intended uses, and safeguards for evaluation. They also identify risks and conditions for any future real-world deployment.

C Additional Details for Responsible Research

  • The disease-profile library supports research, auditing, and decision-support evaluation, while the benchmark evaluates longitudinal reasoning; neither is intended for autonomous diagnosis or replacing clinical judgment.Clinical or educational use should have institutional oversight and qualified medical supervision; clinical cases are de-identified and not unrestrictedly released unless data governance permits.Source: C Additional Details for Responsible… · p. 15
  • The respiratory corpus contains 2,563 guidelines and reference documents; the evaluation benchmark has 188 cases, three diagnostic stages, six scored outputs, and 1,455 progressively disclosed images in multimodal evaluation.The scored outputs yield 1,128 task instances per model setting.Source: C Additional Details for Responsible… · p. 15
  • Cases were de-identified by removing direct identifiers and normalizing dates while preserving clinically relevant temporal relationships; automatic screening and manual clinical review address ongoing re-identification risk.Source: C Additional Details for Responsible… · p. 16
  • Progressive disclosure withholds later-stage evidence until available, and textual summaries were checked for leaked diagnoses or future evidence; cases with unavoidable leakage were revised or excluded.Source: C Additional Details for Responsible… · p. 16
  • Respiratory clinicians reviewed profiles and benchmark annotations, distinguishing required from optional items and resolving disagreements through discussion and revision.No crowdworkers or non-expert public annotators performed clinical judgment tasks.Source: C Additional Details for Responsible… · p. 16
  • Risks include source-guideline bias or omissions, benchmark-specific case mix and documentation, and LLM extraction or judgment errors on complex narratives.Profiles and annotations received qualified-clinician review, and outputs are decision-support signals rather than definitive medical conclusions.Source: C Additional Details for Responsible… · p. 17
  • Future real-world deployment requires prospective validation, institutional review, privacy protection, bias auditing, continuous monitoring, and clinician-in-the-loop workflows.Source: C Additional Details for Responsible… · p. 17
  • AI assistants supported language polishing, drafting, and code development, while authors reviewed and verified scientific claims, experimental choices, clinical content, annotations, results, and manuscript text.Source: C Additional Details for Responsible… · p. 17
Loading 2609.14018v1…