PaperlayerView exact revision on arXiv

Source-linked AI summary

DDXPlus: A New Dataset For Automatic Medical Diagnosis

Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, Joumana Ghosn

arXiv:2205.09148v3cs.CLcs.AIcs.LG

TL;DR

AD and ASD systems collect symptoms and antecedents and may predict disease, but do not model the differential diagnosis doctors use to represent uncertainty and guide questioning. DDXPlus is a large-scale synthetic dataset that combines differential diagnoses with ground-truth pathologies and richer evidence, and the authors extend two existing systems to use differentials. They report that differential training improves system performance or helps doctors understand system reasoning.

  • Problem

    AD and ASD datasets commonly support prediction of a patient's pathology but lack differential-diagnosis data, although doctors use differentials to represent uncertainty and guide questions.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The authors build a large-scale synthetic dataset with each patient's differential, ground-truth pathology, and binary, categorical, and multi-choice symptoms and antecedents, then extend two AD and ASD systems to use differentials.

    Source: §1 · p. 2

  • Results

    Using differential diagnoses as training signals improves system performance or helps doctors better understand the systems' reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Differential diagnoses can represent diagnostic uncertainty and help guide which questions systems ask while collecting patient evidence.

    Source: §1 · p. 2

    The paper identifies a shared scope gap in existing public datasets: they lack differential-diagnosis data.

    Source: §2 · p. 3

Abstract

from arXiv · show

There has been a rapidly growing interest in Automatic Symptom Detection (ASD) and Automatic Diagnosis (AD) systems in the machine learning research literature, aiming to assist doctors in telemedicine services. These systems are designed to interact with patients, collect evidence about their symptoms and relevant antecedents, and possibly make predictions about the underlying diseases. Doctors would review the interactions, including the evidence and the predictions, collect if necessary additional information from patients, before deciding on next steps. Despite recent progress in this area, an important piece of doctors' interactions with patients is missing in the design of these systems, namely the differential diagnosis. Its absence is largely due to the lack of datasets that include such information for models to train on. In this work, we present a large-scale synthetic dataset of roughly 1.3 million patients that includes a differential diagnosis, along with the ground truth pathology, symptoms and antecedents for each patient. Unlike existing datasets which only contain binary symptoms and antecedents, this dataset also contains categorical and multi-choice symptoms and antecedents useful for efficient data collection. Moreover, some symptoms are organized in a hierarchy, making it possible to design systems able to interact with patients in a logical way. As a proof-of-concept, we extend two existing AD and ASD systems to incorporate the differential diagnosis, and provide empirical evidence that using differentials as training signals is essential for the efficiency of such systems or for helping doctors better understand the reasoning of those systems.

1 Introduction

AD and ASD systems collect patient evidence and predict disease, but omit the differential diagnosis that doctors use to represent uncertainty and guide further questioning. DDXPlus addresses the dataset gap with synthetic patients carrying both a differential and ground-truth pathology, alongside richer evidence types, and the paper tests differential-aware extensions to two systems.

1 Introduction

  • Doctors refine a short list of plausible diseases as they gather symptoms and antecedents, then may use exams to rule out candidates or confirm pathology.The differential diagnosis is considered throughout the interaction and updated based on patient responses.Source: §1 · p. 1
  • Existing AD and ASD systems predict a single underlying disease but lack differential diagnoses, which can represent diagnostic uncertainty and guide evidence collection.The introduction attributes the missing differential data in part to the lack of datasets that include it.Source: §1 · p. 2
  • DDXPlus adds categorical and multi-choice symptoms and antecedents, hierarchical symptoms, and a distinction between symptoms and antecedents alongside each patient's differential and ground-truth pathology.The authors describe it as the first large-scale dataset to combine ground-truth pathologies and differentials with non-binary symptoms and antecedents.Source: §1 · p. 2
  • DDXPlus provides a synthetic benchmark of roughly 1.3 million patients and the paper extends two AD and ASD systems, reporting that differential training improves performance or helps doctors understand system reasoning.The dataset includes each patient's differential diagnosis and underlying pathology.Source: §1 · p. 2

2 Existing datasets and limitations

Existing medical datasets leave important gaps for training AD and ASD systems: some lack symptom data, while others are small or incomplete and mostly encode symptoms as binary. The cited datasets also lack differential-diagnosis data.

2 Existing datasets and limitations

  • MIMIC-III often lacks symptom-related data, while DX and Muzhi are small and may omit unmentioned symptoms because they derive from medical conversations.When a symptom is not mentioned in a conversation, its presence in the patient cannot be determined from that source.Source: §2 · p. 2
  • SymCAT and newer HPO- and MedlinePlus-based synthetic datasets contain only binary symptoms, while SymCAT's incomplete symptom information limits realism.Binary-only symptom queries can require several turns where a categorical or multi-choice question could gather the same information at once.Source: §2 · p. 3
  • Existing public datasets share a critical gap: they do not contain differential-diagnosis data.The paper introduces DDXPlus as addressing this limitation.Source: §2 · p. 3

3 DDXPlus dataset

DDXPlus is a roughly 1.3-million-patient synthetic dataset generated from a medical knowledge base and a rule-based diagnosis system. It includes patient demographics, pathology, symptoms, antecedents, and differential diagnoses.

3.1 Medical knowledge base

  • A proprietary, doctor-reviewed knowledge base compiled from over 20,000 medical papers supplies disease, symptom, and antecedent statistics for the selected pathologies.The focus is 49 cough-, sore-throat-, or breathing-related pathologies, covering 110 symptoms and 113 antecedents; evidence can be binary, categorical, or multi-choice.Source: §3.1 · pp. 3–4

3.2 Patient generation

  • Patient generation factorizes age, sex, and geography, samples disease from incidence rates or prevalence when incidence is absent, and draws evidence conditioned on disease, age, and sex.Evidence is assumed conditionally independent unless the knowledge base specifies otherwise; the incidence/prevalence limitation affects 8 of 49 diseases.Source: §3.2 · pp. 4–5

3.3 Differential diagnosis generation

  • A rule-based AD system deployed in a real-world telemedicine platform generated ranked differentials after all patient symptoms and antecedents were provided at once; cases without an output or with the simulated disease absent were discarded.Pathologies outside the 49-disease scope were removed and the distribution renormalized; on average, 1.78 (±1.68) pathologies were removed, representing 0.10 (±0.11) cumulative probability mass.Source: §3.3 · pp. 5–6

3.4 Dataset characteristics

  • The released benchmark includes differential diagnoses, separates symptoms from antecedents, and adds hierarchical categorical and multi-choice evidence plus pathology severity labels.The dataset contains roughly 1.3 million patients and uses an 80%-10%-10% train-validation-test split.Source: §3.4 · p. 6

3.5 Data analysis

  • Patients average roughly 10 symptoms and 3 antecedents, while 6.7% of evidence types are categorical or multi-choice.Table 1 summarizes evidence types and Table 2 summarizes patient evidence counts.Source: §3.5 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • URTI, Viral pharyngitis, and Anemia dominate the pathology histogram, while the other pathologies are also represented; synthesized demographics match the 2015 New York census data used.The pathology histogram is shown in Figure 2 and the socio-demographic statistics in Figure 3.Source: §3.5 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7
  • The simulated pathology ranks first in the differential for more than 70% of patients, and differentials can contain multiple pathologies.The distribution of differential lengths and the rank of the simulated pathology are shown in Figure 4.Source: §3.5 · p. 7, Figure 4 · p. 8

3.6 Dataset usage

  • The dataset is intended for research; models trained on it should not be used directly in real-world systems without rigorous evaluation and checks of population coverage and representation.Source: §3.6 · p. 8

4 Experiments

The experiments adapt two existing AD/ASD systems to predict differentials and compare differential-trained models with disease-trained versions. Differential training improves several differential-related outcomes, with trade-offs in interaction length and evidence collection.

4.1 Models

  • AARLC combines RL evidence acquisition with supervised disease classification, while BASD adds a classifier to an ASD module; both were trained to predict disease or differential diagnoses.Each version's hyperparameters were tuned independently, and its end-of-interaction probability distribution was used as the predicted differential.Source: §4.1 · pp. 8–9

4.2 Experimental setup

  • Patients provide age, sex, and initial evidence, then the model asks about symptoms or antecedents until evidence is collected or the 30-turn limit is reached.A differential is predicted at the end of the interaction.Source: §4.2 · p. 9

4.3 Results

  • AARLC differential training significantly improves differential metrics and positive-evidence recall, but increases interaction length; BASD shows no positive-evidence improvement.For BASD, the differential improves explainability by letting doctors assess alignment between collected evidence and the predicted differential.Source: §4.3 · p. 9, Table 3 · p. 10
  • Disease-trained models score better on GTPA@1, while all approaches perform well on GTPA, including models trained to predict the full differential.The ground-truth pathology need not rank first in a differential.Source: §4.3 · pp. 9–10, Table 3 · p. 10
  • The reported results are initial baselines; the authors position DDXPlus as a basis for improving these models and developing other ASD/AD systems.Appendix K also presents a patient interaction and a doctor's comments on evidence–differential alignment.Source: §4.3 · p. 10

5 Conclusion

The conclusion presents DDXPlus as a large synthetic benchmark with patient evidence, pathology, and differential diagnoses. Experiments suggest differential training can improve system performance or help doctors assess model reasoning, while the dataset and results have important scope limits.

5 Conclusion

  • DDXPlus contains roughly 1.3 million synthetic patients with symptoms, antecedents, pathology, and related differentials, including binary, categorical, and multi-choice evidence.The dataset includes a severity flag for each pathology.Source: §5 · p. 10
  • The experiments provide evidence that differential training enhances system performance or helps doctors assess reasoning by evaluating collected evidence, predicted differentials, and their alignment.Source: §5 · p. 10
  • Differentials are informed by relevant positive evidence, but the generating rule-based system is not guaranteed to consider every relevant negative evidence; the experiments also treat pathologies as equally important.The authors identify attention to severe pathologies as an area for further work.Source: §5 · p. 10
  • The authors caution against deploying a model trained on DDXPlus before rigorous performance evaluation and checks of population coverage and representation.Source: §5 · p. 10

Checklist

The checklist reports that the work's limitations, assets, and reproducibility details are addressed in the paper or supplementary materials.

Checklist

  • The checklist marks the limitations and potential negative societal impacts as discussed in the paper.Source: Checklist · p. 13
  • The checklist reports that code and instructions for reproducing the main results are included, training details are specified, and Table 3 reports error bars.It also points to the paper and supplementary material for asset licensing and code-related citations.Source: Checklist · p. 13
  • The dataset is synthetic, and the checklist marks consent and participant-risk questions as not applicable.Source: Checklist · p. 13

A Additional information on the KB and rule-based AD system used to generate DDXPlus

This appendix describes clinician-informed knowledge-base construction and the rule-based system used in dataset generation. It also outlines how the system prioritizes questions and how it was evaluated with real patients.

A Additional information on the KB and rule-based AD system used to generate DDXPlus

  • Doctors reviewed more than two years of disease-related papers and independently validated disease presentations and evidence distributions in the knowledge base.The review included usual and atypical presentations.Source: A Additional information on the KB and… · p. 14
  • Categorical and multi-choice symptom details include extensive descriptions of pain and rash, shaped by domain experts and medical literature.Pain descriptions cover location, radiation, intensity, character, precision, and onset.Source: A Additional information on the KB and… · p. 14
  • The rule-based system first asks questions likely to rule out many diseases, then explores high-risk diseases and risk factors for the top five differential diagnoses.Source: A Additional information on the KB and… · p. 14
  • The engine was built for primary and acute care, then tested with real patients; doctors rated the collected history and compared its differential with their own.The clinicians evaluating the system's differential were blinded to its initially collected evidence.Source: A Additional information on the KB and… · p. 14

B Demographic statistics from census data

The dataset uses New York census age-and-sex statistics to support patient synthesis, while documenting geographic coverage, missing incidence inputs, and differential post-processing.

B Demographic statistics from census data

C Geographical regions

  • The dataset covers 11 regions, including North and South America, Europe, Asia, Africa, Oceania, and the Caribbean.The listed regions also include Central America, South East Asia, and West Africa.Source: C Geographical regions · p. 15

D Missing incidence rates

  • Incidence rates are missing for eight pathologies across some age, sex, and geographic combinations, including rare epidemic diseases and mild diseases not tracked for public-health purposes.The named conditions include anemia, inguinal hernia, anaphylaxis, allergic sinusitis, Chagas, tuberculosis, Ebola, and chronic rhinosinusitis.Source: D Missing incidence rates · p. 15

E Differential diagnosis post-processing

  • Pathologies outside the 49 considered conditions are excluded from the rule-based platform’s differential diagnosis.Table 5 reports the proportion of patients with exclusions at each differential rank.Source: E Differential diagnosis post-processing · p. 15

F Subset data analysis

The analysis describes evidence, pathology, demographic, and differential-diagnosis distributions across dataset splits, then details how two agents are adapted to train with differentials.

F.1 Evidence statistics

F.2 Pathology statistics

F.3 Socio-demographic statistics

  • Figures 8–10 present the socio-demographic statistics of synthesized patients in the train, validation, and test subsets.The passage identifies the figures and subsets but does not report specific distribution values.Source: §F.3 · p. 15, Figure 8 · p. 17, Figure 9 · p. 18

F.4 Differential diagnosis statistics

  • Figures 11–13 show differential lengths and the simulated pathology’s rank within the differential across the three subsets.Table 5 reports the proportion of patients with pathologies excluded at each differential rank.Source: Table 5 · p. 16, §F.4 · p. 17, Figure 11 · p. 18

G.1 AARLC

  • AARLC is trained with ground-truth differential probabilities, and its stopping threshold and positive reward use an exact match between predicted and ground-truth disease sets.The differential-based version replaces per-disease thresholds with one global threshold.Source: §G.1 · p. 19

G.2 BASD approach

  • BASD has policy and classifier branches; simulated dialogue states set stop-or-inquire and classifier targets, with the classifier updated only when the policy target is stop.The classifier target can be either the ground-truth pathology or the ground-truth differential.Source: §G.2 · p. 19

H Evaluation metrics

The evaluation defines measures for interaction length, differential quality, pathology recovery, and positive-evidence recall. It also notes that negative-evidence evaluation remains undefined.

Interaction length

  • Interaction length is the average number of evidences collected from a patient by an agent.The differential distributions are post-processed to remove pathologies with mass at or below 0.01, a threshold approved by the collaborating physician.Source: H Evaluation metrics · pp. 19–20

Differential diagnosis metrics

  • Differential diagnosis recall, precision, and F1 measure overlap between the agent’s predicted differential and the ground-truth differential.The differential distributions used for evaluation are post-processed at the 0.01 mass threshold.Source: H Evaluation metrics · pp. 19–20

Ground truth pathology accuracy

  • GTPA@k checks whether the simulated pathology appears among the top k predicted diseases, while GTPA checks whether it appears in the predicted differential.The measures evaluate recovery of the pathology from which the patient was simulated.Source: H Evaluation metrics · p. 20

Positive evidence recall

  • Positive evidence recall measures how much of the patient’s experienced symptoms and antecedents the agent inquires about.The section does not report evidence precision because collected positive evidence alone gives clinicians an incomplete picture.Source: H Evaluation metrics · p. 20

Evaluation scope

  • Metrics for questions about negative evidence are left for future work because clinicians also check that other pathologies were considered and their evidence is absent.The authors therefore do not report evidence precision.Source: H Evaluation metrics · p. 20

I Dataset release

DDXPlus is released under CC-BY as a synthetic dataset combining patient characteristics, symptoms, antecedents, ground-truth pathology, and differential diagnosis. Its documentation supports varied evidence formats, while its differentials reflect an acute-care setting and a deliberately limited disease subset.

I Dataset release

  • The dataset contains synthetic patients with socio-demographic data, symptoms, antecedents, a ground-truth pathology, and a differential diagnosis; evidence can be binary, categorical, or multi-choice.Some symptoms are organized hierarchically, and each disease has a severity level.Source: I Dataset release · p. 21

I.1 Dataset documentation

  • Evidence documentation distinguishes symptoms from antecedents and records evidence type, possible values, and the meanings of categorical or multi-choice codes.Related evidence entries can be grouped through code_question, which identifies evidence that may need to be activated before other group members are simulated.Source: §I.1 · p. 21
  • The patient records include age, sex, pathology, experienced evidence, initial evidence, and a ground-truth differential represented as pathology-probability pairs.Initial evidence is selected from the patient’s evidence list to give systems a common starting point during evaluation.Source: §I.1 · p. 22

I.2 General notes about the dataset’s differentials

  • The dataset targets acute care and is biased toward high-mortality and high-morbidity pathologies, where negative evidence may be important to physicians.The authors caution that the differential uses a small disease subset, reducing the potential evidence to collect.Source: §I.2 · p. 22

J Dataset samples

The samples show synthetic patients with varied symptoms, antecedents, and ranked differentials. Doctor feedback on two cases highlights that a differential may require additional questions and refinement in light of clinical context.

J Dataset samples

  • The samples pair each synthetic patient’s characteristics and clinical evidence with a differential diagnosis; the first two also include a doctor’s feedback.The examples include categorical and multi-choice findings such as pain qualities and locations, alongside binary symptoms and antecedents.Source: J Dataset samples · p. 22, Sample 1 · p. 23

Sample 1

  • For the PSVT example, the doctor says the differential warrants questions about palpitation causes and further discrimination because the diagnosis space is smaller than a physician’s frame of reference.The doctor notes that risk factors support PSVT but that other possibilities could change the care episode.Source: Sample 1 · p. 23

Sample 2

  • For the GERD example, the doctor notes that acute myocardial infarction can present atypically in women and recommends refining the differential to better estimate disease probabilities and tailor tests.Source: Sample 2 · p. 24

K Model Analysis

The model analysis presents question-answer sequences and predicted differentials for two systems trained either on differential diagnoses or ground-truth pathology. A doctor judges the resulting evidence collection and differential quality differently across these training choices.

AARLC trained to predict the differential diagnosis

  • AARLC trained on the differential predicted a broad set including unstable and stable angina, possible NSTEMI / STEMI, spontaneous pneumothorax, GERD, pericarditis, pulmonary embolism, and atrial fibrillation.In the doctor’s assessment, this variant searched a wider array of diseases, including infectious causes, though further questions could still be needed.Source: AARLC trained to predict the… · p. 30, BASD trained to predict the ground… · p. 32

AARLC trained to predict the ground truth pathology

BASD trained to predict the differential diagnosis

BASD trained to predict the ground truth pathology

  • BASD trained on pathology predicted spontaneous pneumothorax; the doctor judged it very good at seeking pathology confirmation and asking specific questions.The doctor also noted that a clinician would need to complete the evaluation with more questions and that pertinent questions depend on clinical context.Source: BASD trained to predict the ground… · p. 32

K Model Analysis

  • The doctor judged AARLC trained on pathology the worst, BASD trained on differentials insufficient in evidence collection, and the other two variants best for gathering a sensible amount of evidence.AARLC trained on differentials searched a wider range of diseases; BASD trained on pathology focused on confirmation and asked more specific questions.Source: BASD trained to predict the ground… · p. 32
Loading 2205.09148v3…