PaperlayerView exact revision on arXiv

Source-linked AI summary

Ontology-guided multi-agent reasoning for explainable differential diagnosis

Roxana Ramona Szomiu, George Ursu, Adrian Groza

arXiv:2610.36325v1cs.AIcs.MA

TL;DR

Incomplete patient descriptions make reliable differential diagnosis difficult, and single-step LLM reasoning may lack grounded evidence and transparent rationale. The paper proposes ontology-guided multi-agent diagnosis with targeted follow-up questions; interactive refinement improves global accuracy, although Top-3 accuracy falls at 50% completeness.

  • Problem

    Initial patient notes are often incomplete, while LLM diagnostic reasoning may rely on learned patterns rather than grounded clinical evidence and remain opaque to clinicians.

    Source: §I · p. 1

  • Method

    The framework separates language understanding from clinical reasoning, grounds diagnostic scoring in curated ontologies, and asks targeted questions about missing discriminative symptoms.

    Source: §I · p. 1

  • Results

    Interactive refinement improved global Top-1 accuracy from 18.7% to 21.2% and Top-3 accuracy from 25.8% to 27.3%.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Deterministic control supports predictable execution and validation of intermediate decisions, suiting the architecture to applications where reproducibility and auditability are essential.

    Source: §V · p. 7

    At 50% completeness, Top-3 accuracy decreases from 26.3% to 24.2%; unresolved errors also include knowledge-graph gaps and difficulty identifying exact subtypes.

    Source: §V · p. 7

Abstract

from arXiv · show

Effective differential diagnosis depends on well structured and unambiguous patient case descriptions, since incomplete or ambiguous clinical information readily leads to diagnostic errors and inconsistent reasoning. Large Language Models (LLMs) have recently shown promising capabilities in medical question answering and clinical reasoning; however, even when achieving competitive performance on medical benchmarks, they often rely on pattern recognition learned from training data rather than grounded clinical evidence, and can generate plausible yet unsupported diagnostic conclusions. Their reasoning remains largely opaque, making it difficult for clinicians to assess the evidence and rationale behind a given recommendation. To address this gap, we propose an ontology-guided, multiagent differential diagnosis framework that replicates the iterative reasoning process of physicians. The framework grounds diagnostic inference in a structured, expert-curated medical knowledge base, enabling interpretation of symptoms, clinical findings, or disease relationships, while decomposing diagnosis into explicit, auditable reasoning steps. This ontology-based grounding constrains inference to evidence-backed associations, reducing hallucination and improving diagnostic transparency, traceability, and consistency. The system then identifies the most discriminative missing symptoms and generates targeted follow-up questions to reduce uncertainty through a process that mirrors to clinical inquiry. Together, these components improve the reliability, explainability, and diagnostic accuracy of AI-assisted differential diagnosis. The source code is publicly available.

I. INTRODUCTION

The paper frames differential diagnosis as iterative because initial patient notes are often incomplete. It proposes ontology-grounded multi-agent reasoning with targeted questioning and traceable scoring.

I. INTRODUCTION

  • Initial patient notes are often incomplete, while early LLM evaluations assumed all diagnostic information was available in a single-turn prediction.The paper presents differential diagnosis as an iterative process for distinguishing diseases with similar symptoms.Source: §I · p. 1
  • The proposed neuro-symbolic framework separates language understanding from clinical reasoning and grounds diagnostic scoring in expert-curated ontologies.It is presented as a way to reduce unsupported LLM conclusions.Source: §I · p. 1
  • The system identifies missing discriminative symptoms and generates targeted follow-up questions, addressing up to 30% loss in initial symptom disclosure.It incrementally refines diagnosis as new clinical evidence becomes available.Source: §I · p. 1
  • Mathematical scoring functions rank diagnoses at each stage and link refinements to contributing symptoms and scores for clinician interpretation.The authors describe this as enabling traceable and transparent reasoning.Source: §I · pp. 1–2

II. RELATED WORK

Prior work motivates moving from single-turn medical question answering toward interactive, grounded diagnosis. MedMAS-DDX implements that shift as a staged pipeline coordinated by deterministic control.

II. RELATED WORK

  • Prior medical QA systems often use single-turn exchanges, limiting their ability to clarify ambiguities or refine diagnoses through dialogue.Med-PaLM2 reported 86.5% accuracy on MedQA, while the cited ophthalmology patient-education interaction remained single-turn.Source: §II · p. 2
  • Q4Dx evaluates adaptive questioning with incomplete synthetic notes; its baseline accuracy falls from 51% to 44% and 37% as symptom disclosure drops.The review also describes knowledge-graph question selection in MedKGI and dialogue weaknesses identified by MedPI and MINT.Source: §II · p. 2
  • MedDialogRubrics reports that strong models miss nearly half of must-ask questions and that performance peaks at approximately 10–11 turns [15].MedAgents reaches 86.7% average accuracy across nine multiple-choice datasets with GPT-4.Source: §II · p. 2

III. METHODOLOGY

  • MedMAS-DDX turns free-text complaints into structured symptoms, candidate diseases, and an initial ranking, then updates the differential using answers to follow-up questions.Its design assigns language understanding and generation to language models, but ranking computations to deterministic code and medical knowledge to an external curated source.Source: §III · p. 2
  • Three agents participate in live diagnosis, while a patient-simulator agent is reserved for automated evaluation.A deterministic control layer mediates agent communication and validates or transforms intermediate representations rather than allowing unconstrained agent-to-agent dialogue.Source: §III · pp. 2–3

A. Symptom extractor agent

The pipeline converts informal clinical notes into standardized symptoms, links them to ontology concepts and candidate diseases, then ranks candidates using symptom evidence and coverage.

A. Symptom extractor agent

  • The symptom extractor converts an incomplete natural-language note into canonical symptom terms, alternative phrasings, and optional context for ontology matching.Descriptive information remains separate from standardized terminology.Source: A. Symptom extractor agent · p. 3
  • Symptoms are resolved through ordered term matching, and unresolved symptoms are retained for traceability but excluded from inference.Resolved concepts retrieve diseases explicitly linked to at least one patient symptom in the knowledge graph.Source: A. Symptom extractor agent · p. 3
  • Candidate diseases are ranked using symptom matches, with rarer, more informative symptoms receiving greater weight.The weighting reflects the idea that symptoms associated with fewer diseases are more specific.Source: A. Symptom extractor agent · p. 3
  • The composite score combines weighted diagnostic evidence with the fraction of mapped patient symptoms explained by each disease.The coverage factor adjusts rather than dominates the raw score, scaling it between 0.5 and 1.Source: A. Symptom extractor agent · p. 4
  • An inclusive threshold retains candidates that reach a small fraction of the leading score or match at least two symptoms.The threshold is deliberately low because discarded candidates cannot be recovered later.Source: A. Symptom extractor agent · p. 4

B. Follow-up interviewer agent

When multiple candidate diseases remain, the interviewer identifies missing discriminative information and asks targeted questions, then pauses for structured responses before diagnostic refinement.

B. Follow-up interviewer agent

  • The interviewer uses candidate symptom profiles, the current ranking, and reported symptoms to identify missing information and resolve ambiguities.It filters out non-discriminative or previously reported symptoms and selects patient-observable ones for targeted Yes/No questions.Source: B. Follow-up interviewer agent · p. 4
  • After generating questions, the system pauses for patient or user responses in a constrained yes, no, or unsure format.Separating answer collection from diagnostic logic lets the same pipeline use different response sources.Source: B. Follow-up interviewer agent · p. 4
  • The structured answers supply the missing facts before conclusion, turning one-shot classification into an iterative consultation.The process is described as narrowing possibilities through targeted questioning.Source: B. Follow-up interviewer agent · p. 4

C. Patient simulator agent

A testing-only patient agent simulates answers, while the diagnosis refiner uses iteratively collected findings to update candidate rankings and produce evidence-based explanations.

C. Patient simulator agent

  • The patient agent is used only for automated validation, answering interviewer questions from a fully specified case description.It is outside the production environment and has no external tool access.Source: C. Patient simulator agent · p. 4

D. Diagnosis refiner agent

  • The refiner synthesizes collected findings and retrieved disease-symptom relations into a final diagnosis with an evidence-based explanation.It assigns confidence from score margins, using cautious confidence when the margin is narrow.Source: D. Diagnosis refiner agent · pp. 4–5
  • Queried symptom responses update the differential through deterministic scoring, with greater ranking influence for symptoms that distinguish among the remaining candidates.Confirmed and denied relevant symptoms contribute to normalized support and conflict terms.Source: D. Diagnosis refiner agent · pp. 4–5
  • Higher conflict penalties of β = 0.4 and β = 0.6 overly penalized plausible candidates with temporarily unobserved symptoms, causing premature elimination from the differential.The selected α = 0.6 and β = 0.2 values were reported as optimal for diagnostic accuracy and rank stability on a validation subset.Source: D. Diagnosis refiner agent · p. 5
  • Limiting β relative to α bounds the adjustment so follow-up responses refine, rather than replace, the initial ranking.The candidates are then re-ranked and the most influential answer is identified.Source: D. Diagnosis refiner agent · p. 5

IV. EXPERIMENTS AND RESULTS

The evaluation focuses on diagnostic trajectories rather than static classification, with active agents using LLMs while deterministic scoring and ranking run locally.

IV. EXPERIMENTS AND RESULTS

  • The evaluation assesses MedMAS-DDx as a diagnostic trajectory rather than as a static classifier.Its stated focus is acquiring missing information through follow-up questions and iteratively refining the differential.Source: §IV · p. 5
  • The trajectory framing centers on follow-up information acquisition and iterative refinement of the differential diagnosis.Source: §IV · p. 5
  • Active agents access LLMs through remote APIs or self-hosted vLLM, while deterministic scoring and ranking run locally, independent of the LLM host.Source: §IV · p. 5

A. Evaluation of framework and experimental setup

The framework was evaluated on incomplete Q4DX cases, then assessed for symptom mapping and the effect of interactive refinement. Refinement improved aggregate metrics, although Top-3 accuracy dipped slightly at 50% completeness.

A. Evaluation of framework and experimental setup

  • The evaluation used 99 cases and 29 diagnoses at 80% and 50% symptom completeness, with 198 samples total; 100% completeness was excluded because no symptoms remained for follow-up.Follow-up answers came from a simulated patient with access to the full case profile, yielding an optimistic estimate rather than real-patient performance.Source: A. Evaluation of framework and… · p. 5

B. Diagnostic accuracy and interactive refinement

C. Qualitative case study: Recovering the diagnosis of Hepatitis A (Case #96)

Case #96 shows how a targeted question about an omitted discriminative symptom changed a competitive differential. Confirming jaundice moved Hepatitis A to the top prediction.

C. Qualitative case study: Recovering the diagnosis of Hepatitis A (Case #96)

  • The 80%-complete narrative omitted jaundice, the most discriminative hepatitis sign, although all eight symptoms present in the text mapped to SYMP concepts.Source: C. Qualitative case study: Recovering… · p. 6
  • Without jaundice, anthrax candidates led at 0.922, while four hepatitis subtypes tied at 0.861; the top prediction was incorrect at Top-1 but correct at Top-3.Source: C. Qualitative case study: Recovering… · p. 6
  • The interviewer asked eleven targeted Yes/No questions; confirming jaundice raised Hepatitis A, B, and C to 0.872 despite three denials, while Hepatitis E reached 0.838.The two anthrax candidates fell to 0.781 after receiving no support and four denials.Source: C. Qualitative case study: Recovering… · p. 6
  • Hepatitis A became the top prediction, with SapBERT similarity rising from 0.340 to 1.0 after jaundice was confirmed.A single confirmed symptom resolved ambiguity among six equally scored Phase 1 candidates.Source: C. Qualitative case study: Recovering… · p. 6

D. Comparative analysis against the MedQDx baseline

The comparison with MedQDx depends on both accuracy metric and completeness level. MedQDx had higher zero-shot Top-1 accuracy, while the Top-3 gap varied across completeness levels.

D. Comparative analysis against the MedQDx baseline

V. DISCUSSIONS AND LIMITATIONS

Interactive refinement improves Top-1 accuracy, but its Top-3 effect varies with symptom completeness and whether a case has actionable ambiguity. Remaining errors and limits reflect ontology coverage, structured reasoning constraints, and restricted yes/no interactions.

V. DISCUSSIONS AND LIMITATIONS

  • Top-3 accuracy rises from 25.3% to 30.3% at 80% completeness but falls from 26.3% to 24.2% at 50%, despite Top-1 gains at both levels.At 50%, incomplete symptoms admit weak candidates that follow-up evidence can replace; aggregated scores can understate per-case impact.Source: §V · p. 7
  • A single targeted confirmation raises Case #96's Top-1 similarity from 0.340 to 1.0 by resolving ambiguity among six equally scored candidates.Refinement has no actionable uncertainty when one diagnosis dominates or no candidate is close to the reference label.Source: §V · p. 7
  • Ontology quality and terminology alignment limit what the system can use, while yes/no/unsure replies omit clinical nuance such as symptom severity and temporal progression.Missing, incomplete, or outdated knowledge cannot enter reasoning, and terminology mismatch limits alignment between patient data and ontology concepts.Source: §V · pp. 7–8
  • Ontology coverage bounds candidate diagnoses: because SYMP treats Jaundice and Vertigo as symptoms, the system cannot output them as diagnoses, regardless of reasoning quality.For Jaundice, accuracy remains zero until the ontology is extended; the system instead retrieves diseases featuring jaundice and asks about it as follow-up evidence.Source: §V · p. 7
  • Unresolved errors primarily reflect structured reasoning constraints, including symptom–diagnosis label mismatches, sparse symptom–disease links, and subtype ambiguity without laboratory results.Symptom mapping reached 88.4%, while broader disease families could be identified more robustly than exact subtypes.Source: §V · p. 7
  • Constraining inference to a curated knowledge graph prevents unsupported diagnoses but caps benchmark performance; a more advanced LLM would not resolve errors rooted in these constraints.The LLM's role is deliberately restricted to semantic translation.Source: §V · pp. 7–8

VI. CONCLUSION AND FUTURE WORK

The method uses a directed pipeline with a feedback loop to turn free text into a ranked differential and incorporate follow-up answers. Evaluation reports improved global Top-1 and Top-3 accuracy, while future work targets real-world validation, broader ontology coverage, richer interactions, and ranking improvements.

VI. CONCLUSION AND FUTURE WORK

  • The directed pipeline converts free text into structured symptoms, ranks candidate diseases, and reintegrates follow-up answers into scoring, with sparse input shifting more work to questioning.Language models interpret and generate; the knowledge source supplies symptom–disease associations; deterministic code handles ranking, evaluation, and control flow.Source: §VI · p. 8
  • Interactive refinement improved global Top-1 accuracy from 18.7% to 21.2% and Top-3 accuracy from 25.8% to 27.3%.Targeted follow-up questions recovered diagnostic confidence to levels comparable to the zero-shot semantic-similarity baseline using substantially richer initial narratives.Source: §VI · p. 8
  • Future work includes large-scale real-world validation, ontology expansion, richer deterministic interactions, prospective clinical evaluation, and adaptive weighting or probabilistic evidence.The current benchmark is limited in size.Source: §VI · p. 8
Loading 2610.36325v1…