PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations

Yusuf Kesmen, Aniruddha Mukherjee, Yena Chang, David Sasu, Trevor Brokowski, Alexandra V. Kulinkina, Kristina Keitel, Akhil Arora, Lars Henning Klein, Mary-Anne Hartley

arXiv:2609.35462v1cs.LGcs.AIcs.CL

TL;DR

Clinical diagnosis must identify co-occurring conditions from evidence acquired during a consultation, but common evaluation settings do not capture both requirements. CLIMB benchmarks multi-turn diagnosis of synthetic multimorbid patients, and models often name one condition but rarely recover the complete set, even with the number of conditions or full record supplied.

  • Problem

    Common clinical benchmarks generally evaluate one-shot, single-diagnosis prediction, while consultation requires sequential evidence acquisition and identification of multiple co-occurring conditions.

    Source: §1 · pp. 1–2

  • Method

    CLIMB generates synthetic multimorbid patients grounded in clinical sources and evaluates doctor models recovering hidden diagnosis sets through multi-turn interaction.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Models often name one of a patient’s conditions but rarely recover the complete set, and supplying the condition count or full record does not close the gap.

    Source: §5 · p. 6, §6.1 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    Single-most-likely-diagnosis evaluations can hide this multimorbidity failure profile, and future work should improve evidence gathering and tracking of several hypotheses at once.

    Source: §7 · p. 9

    CLIMB is intended for evaluating language models, not for clinical use or as evidence of clinical safety or deployment readiness.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Patients often have several co-occurring clinical conditions, and the findings needed to identify and disambiguate them emerge over the course of a consultation. Evaluating clinical reasoning in this setting requires both multi-turn interaction and multi-label diagnosis. We introduce CLIMB, a benchmark in which a doctor model interviews a simulated patient to recover a ground truth set of co-occurring clinical conditions. Cases are synthesized from clinical decision algorithms and diagnostic datasets, grounding multimorbid presentations in structured clinical knowledge. Across six frontier and open models, none recovers the exact set of conditions in more than 10% of interactive cases. Diagnostic performance declines when conditions co-occur, even when models receive the full clinical record and the true number of conditions. Interaction reduces performance further. In controlled experiments, models behave like single-hypothesis trackers: they anchor on the diagnosis suggested by the opening findings, keep questioning around it, and recover a second condition mainly when a finding in view points to it. Questioning them further does not complete the set but adds mostly wrong diagnoses. We formalise this pattern with a theoretical reference model of single-hypothesis tracking. The benchmark, generator, and evaluation code are available at https://anonymous.4open.science/r/CLIMB-8340.

1 INTRODUCTION

Clinical diagnosis requires acquiring evidence over time and identifying sets of co-occurring conditions, whereas common benchmarks test static evidence or one diagnosis at a time. CLIMB targets that gap with a controlled benchmark and analyses of diagnostic performance and failure behavior.

1 INTRODUCTION

  • Static benchmarks test diagnosis with evidence upfront, but consultations require sequential evidence acquisition and distinguishing recognition from acquiring and integrating evidence for multiple conditions.The introduction frames these as the multiturn and multi-label dimensions of clinical consultation.Source: §1 · pp. 1–2
  • The multi-label dimension has received comparatively less attention, and prior benchmarks do not provide controlled analysis of interactive diagnosis for multimorbid patients.The authors identify the intersection of interactive evidence elicitation and co-occurring-condition diagnosis as their focus.Source: §1 · p. 2
  • CLIMB generates synthetic multimorbid patients from established clinical sources and evaluates multi-turn, multi-label diagnostic reasoning.The introduction describes this as a controlled benchmark for recovering co-occurring diagnoses.Source: §1 · p. 2
  • The paper contributes a comparison across numbers of co-occurring conditions and information regimes, plus controlled failure analysis and a formal description of single-hypothesis tracking.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates either diagnosis from information presented upfront or information gathering in interactive consultations, while multimorbidity research has less often combined co-occurrence with interactive diagnosis. CLIMB brings these dimensions together and scores recovery of the full predicted set.

Static and interactive benchmarks

  • Medical exam benchmarks provide relevant information upfront and accept one answer, primarily measuring stored-knowledge retrieval rather than how to gather missing evidence [Jin et al., 2021; Pal et al., 2022; Jin et al., 2019; Singhal et al., 2023].Source: §2 · p. 3
  • Interactive benchmarks pair doctor models with simulated patients and score final diagnoses [Nori et al., 2025], with reported under-asking, nonspecific questions, and early consultation closure.Source: §2 · p. 3

Multimorbidity and diagnostic error

  • Multimorbidity is associated with higher mortality and polypharmacy, while recent work addresses multi-disease consultations or comorbid psychiatric presentations without studying co-occurrence itself.The paper combines evidence gathering during consultation, controlled condition counts, and scoring of the full predicted set.Source: §2 · p. 3

3 METHOD

CLIMB defines diagnosis as recovering a hidden set of co-occurring conditions from a patient’s findings. Its cases combine source-grounded diagnostic signals with background complaints, using separate construction procedures for ePOCT+ and DDXPlus.

3 METHOD

  • The doctor observes candidate conditions and an opening presentation, while the target diagnoses, their number, and remaining findings are hidden; the objective is to recover the diagnosis set.Source: §3 · p. 3
  • Cases sample a target set using source-specific co-occurrence information, merge its associated findings, and resample the set when consistency checks fail.The ePOCT+ sampler uses empirical prevalence and pairwise co-occurrence factors, with the first draw based on prevalence alone.Source: §3 · p. 3, Figure 2 · p. 4

3.1 CASE GENERATION FROM EPOCT+

  • For ePOCT+, diagnoses are represented by retained paths through a clinical decision graph; merged paths are rejected for conflicts, infeasible pairs, incompatible age windows, or insufficient visible findings.Accepted cases receive demographics from the compatible age window and an opening presentation.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 CASE GENERATION FROM DDXPLUS

  • For DDXPlus, the generator selects compatible source records, merges findings by type, and retries or resamples when values conflict or a diagnosis lacks its own findings.The source records are synthetic and contain a single target condition; infeasible diagnosis pairs and non-overlapping age windows are masked during sampling.Source: §3.2 · p. 4

3.3 BACKGROUND COMPLAINTS AND FINAL RECORDS

  • Both generators add one or two background complaints sampled according to sex-specific symptom prevalences, independently of the number of diagnoses.The ePOCT+ generator restricts these complaints to those applicable to children.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments use fixed, balanced case sets and compare full-information diagnosis with interactive consultation. Controlled conditions vary what models know and which evidence they encounter to help localize diagnostic failures.

4.1 DATA AND MODELS

  • Each source contributes 800 stored cases, balanced at 200 per condition count from k = 1 to 4, and cases are held fixed across models and conditions.The main evaluation uses the complete record in both full-information prompts and patient-simulator answers.Source: §4.1 · p. 5
  • Six doctor models are evaluated, with one fixed patient model answering from the complete finding record without target labels.GPT-5.6 is used for the failure analyses.Source: §4.1 · p. 5

4.2 EVALUATION PROTOCOL

  • Full-information evaluation supplies all retained findings upfront, while interactive evaluation uses a doctor–patient dialogue with a budget of 20 question turns.Both conditions request all supported diagnoses from the same candidate set and use the same scorer.Source: §4.2 · p. 5
  • Controlled comparisons disclose the true condition count, request a single diagnosis, or use matched and designed cohorts to examine information acquisition, anchoring, and symptom overlap.One designed cohort varies only whether symptoms after the first complaint belong to the first diagnosis alone or to both diagnoses.Source: §4.2 · p. 5
  • The protocol reports mean per-case Jaccard and exact-set recovery.Source: §4.2 · p. 5

5 RESULTS

Across the evaluated models, recovering the complete diagnosis set is difficult even with full information, and interaction lowers performance. Models return more diagnoses as multiplicity rises, but this does not translate into a complete or more accurate search.

5 RESULTS

  • Exact-set recovery remains low even with full information, while interaction lowers Jaccard point estimates for every model.The comparison pools exact-set recovery and Jaccard under full-information and interactive conditions.Source: Table 1 · p. 6, §5 · p. 6

5.1 DOES THE MODEL REGISTER MULTIPLICITY?

  • As k grows, models return more diagnoses, but correct diagnoses rise far less than k and consultation length hardly changes.The figure summarizes diagnosis counts and questions across six models on DDXPlus.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 THE SINGLE-ANSWER GAP

  • Single-answer success increasingly overstates complete-set recovery: the full set approaches zero at k = 4, while interaction lowers scores at every k.The gap is already present at k = 1, when models often find the condition but add wrong ones.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

6 FAILURE ANALYSIS

The failure analysis separates evidence acquisition from decoding, then examines anchoring, shared symptoms, and continued questioning. Across these tests, models often fail to recover a second diagnosis even when more evidence or questions are available.

6 FAILURE ANALYSIS

  • The authors separate failures to collect evidence from failures to report a diagnosis despite sufficient evidence, using matched patients and a fixed decision-tree query policy.The tree selects questions that most reduce uncertainty about the diagnosis set.Source: §6.1 · p. 7, §6 · p. 7

6.1 CAN MODELS ACQUIRE AND USE DIAGNOSTIC EVIDENCE?

  • GPT-5.6 asks more questions than the decision tree but gains less information per question and achieves lower exact-set recovery; the gap persists with tree evidence or the full record.The comparison uses a matched respiratory cohort and a fixed decoder.Source: Figure 5 · p. 8, §6.1 · p. 8

6.2.1 ANCHORING ON THE OPENING FINDINGS

  • GPT-5.6 recovers the opening-supported diagnosis A about twice as often as B, and its most common final set contains A without B.The result is shown for the designed cohort.Source: Figure 6 · p. 8, §6.2.1 · p. 8

6.2 WHY IS THE SECOND DIAGNOSIS LOST?

  • Anchoring and satisfaction of search are proposed as biases that may follow from treating diagnoses as mutually exclusive.The experiments examine where the second diagnosis is lost when both conditions are equally severe.Source: §6.2 · p. 8

6.2.2 SHARED SYMPTOMS

  • Shared symptoms raise recall of B from 37% to 51%, but naming both diagnoses rises only from 27% to 32%, not reliably.A’s recall barely changes between the two opening conditions.Source: §6.2.2 · p. 9

6.3 DO MORE QUESTIONS HELP?

  • After twenty questions, probes retain 1.3, 1.4, and 1.9 correct diagnoses at k = 2, 3, and 4; more questions mostly add wrong diagnoses, so Jaccard does not rise after the natural stop.The other open models show the same pattern.Source: Figure 7 · p. 9, §6.3 · p. 9

7 DISCUSSION AND CONCLUSION

CLIMB evaluates interactive diagnosis of patients with co-occurring conditions; models often identify one condition but rarely recover the complete set. Its findings point to single-hypothesis tracking, while the benchmark is intended for evaluation rather than clinical use.

7 DISCUSSION AND CONCLUSION

  • Models often name one condition but rarely recover the complete set, even when given the number of conditions or the full record.In interaction, models pursue the diagnosis suggested by the opening, keep questioning around it, and further questions add mostly wrong diagnoses.Source: §7 · p. 9
  • The synthetic patients use DDXPlus and ePOCT+ sources, without patient-level records or identifiable or sensitive information.ePOCT+ contributes a clinician-authored decision tree and aggregate statistics; DDXPlus is fully synthetic.Source: ETHICS STATEMENT · p. 10
  • Benchmark performance should not be interpreted as evidence of clinical safety or deployment readiness.The benchmark is intended for evaluating language models, not clinical use.Source: ETHICS STATEMENT · p. 10
  • CLIMB benchmarks interactive diagnosis of patients with several co-occurring conditions.The benchmark, generator, and evaluation code are available to support reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

Prior work addresses static diagnosis, interactive consultation, or selected forms of multi-disease prediction. CLIMB focuses on recovering an unordered set of concurrent diagnoses while evidence is elicited in conversation.

A EXTENDED RELATED WORK

  • Static medical question benchmarks assess use of evidence supplied upfront, not information acquisition or maintaining a diagnostic state across conversation.CRAFT-MD reports behavior that differs from vignette-based assessment, while AgentClinic adds simulated patients, tools, and environmental biases.Source: A EXTENDED RELATED WORK · p. 15
  • CLIMB fixes the case, candidate vocabulary, and scorer to compare full-record diagnosis with eliciting the same retained findings interactively.Its free-text patient interface follows prior interactive work, while deterministic record-based answers support analyses needing an unambiguous evidence trace.Source: A EXTENDED RELATED WORK · p. 15
  • Existing multi-turn benchmarks include multi-disease strata or a fixed mixed psychiatric category, rather than arbitrary concurrent diagnosis sets.Dr. Assistant reports ICD-code recall and precision; LingxiDiagBench tests a predefined mixed depression–anxiety pattern.Source: A EXTENDED RELATED WORK · p. 15
  • CLIMB scores the complete unordered set of diagnoses present, including cases with one condition, rather than a differential or a predefined comorbidity pattern.It varies diagnosis count to locate where a second diagnosis is lost and separates evidence acquisition from evidence use.Source: A EXTENDED RELATED WORK · p. 15
  • CLIMB reports exact match alongside Jaccard, precision, recall, and predicted cardinality to distinguish full recovery from partial recovery and unsupported labels.Its contribution is evaluating set-level outcomes when evidence must first be acquired, not proposing a new multi-label objective.Source: A EXTENDED RELATED WORK · p. 16
  • CLIMB applies longstanding ideas of joint disease representation and interactive diagnosis to contemporary language-model consultations with controlled synthetic cases.Its generator preserves case components and supports matched comparisons of evidence availability, diagnosis trajectories, and stopping.Source: A EXTENDED RELATED WORK · p. 16

B.1 SOURCE DATA AND RECORD REPRESENTATION

CLIMB builds multimorbid cases from ePOCT+ clinical decision rules and DDXPlus synthetic records. The sources contribute different diagnosis and finding structures that the generator combines into target sets.

B.1 SOURCE DATA AND RECORD REPRESENTATION

  • The two source datasets provide distinct building blocks for constructing concurrent diagnosis sets.ePOCT+ supplies decision rules and cohort-derived diagnosis frequencies; DDXPlus supplies synthetic records with one target pathology.Source: §B.1 · p. 17

B.1.1 EPOCT+: PEDIATRIC DECISION RULES AND COHORT STATISTICS

  • ePOCT+ contributes pediatric clinical decision paths for findings and cohort diagnoses for sampling frequencies and pairwise associations.The local algorithm export has 617 nodes, 132 diagnosis branches, and 235 final-diagnosis IDs representing 216 normalized labels; treatment and referral conditions are removed from emitted findings.Source: §B.1.1 · p. 17

B.1.2 DDXPLUS: SYNTHETIC RECORDS FOR DIFFERENTIAL DIAGNOSIS

  • DDXPlus contributes records for 49 pathologies, each with one target pathology and a differential that is not treated as a concurrent-condition set.The local copy contains 1,292,579 records across training, validation, and test splits; its evidence schema has 223 variables covering symptoms and antecedents.Source: §B.1.2 · p. 17

B.2 GENERATED POPULATIONS AND EVALUATION SUBSETS

The appendix documents CLIMB’s generated populations, source-specific case construction, and safeguards. A learned reader recovers most DDXPlus sets and nearly all ePOCT+ sets from the full record, showing that the records largely determine their targets.

B.2 GENERATED POPULATIONS AND EVALUATION SUBSETS

  • The full-record reference reader recovers most DDXPlus sets at every diagnosis count, far above every model, and recovers 99 to 100% of ePOCT+ sets at every count.Each stored population has 800 cases, balanced across k = 1 to 4; every target diagnosis retains visible supporting evidence in the generated record.Source: §B.2 · pp. 17–19, Table 6 · p. 19

B.3 EPOCT+ STATISTICS, PATHS, AND REJECTION

  • ePOCT+ proposals merge retained diagnosis paths and are rejected for conflicts, exclusions, empty age intersections, missing diagnosis-specific findings, or fewer than two visible findings.Rejected proposals are resampled at the same diagnosis count, and all 800 stored cases retain visible findings supporting each diagnosis.Source: §B.3 · p. 20

B.4 DDXPLUS ASSOCIATIONS, MASKS, AND RECORD MERGING

  • DDXPlus sampling weights diagnosis prevalence and elicited pair associations, excludes infeasible pairs, matches age and sex, and rejects incompatible record merges.Pairwise bounds do not validate a higher-order joint distribution, and the current sequential generator uses stored multipliers rather than the clipped matrix.Source: §B.4 · pp. 20–21

B.5 OFFLINE MASKING PROMPTS

  • Clinician-verified masks restrict sampled labels and infeasible diagnosis pairs but do not replace clinical review of every generated case.Only infeasible DDXPlus pairs are hard exclusions; unlikely pairs are not forbidden.Source: §B.5 · pp. 21–22

B.6 BACKGROUND SAMPLING

  • Background complaints are sampled without replacement from sex-specific vocabularies, with pediatric filtering for ePOCT+; added complaints are not checked again for diagnosis preservation.The source survey concerns adults, so pediatric filtering is a construction choice rather than an estimate of pediatric prevalence.Source: §B.6 · p. 22

B.7 CONSTRUCTION ALGORITHMS

  • The generation algorithms resample rejected diagnosis sets while preserving their target cardinality and enforce source-specific consistency rules before accepting cases.Neither algorithm has a global attempt limit, and the implemented checks are not proof of clinical validity or unique identifiability.Source: §B.7 · pp. 23–24

B.8 ASSOCIATION AND COMPLAINT-PHRASE ELICITATION

  • Offline elicitation prepares co-occurrence multipliers and a complaint-phrase dictionary that the current generator reads from saved tables.The stored multipliers reflect two elicitation rollouts aggregated in log space; the phrase dictionary maps questionnaire text to an opening complaint phrase.Source: §B.8 · pp. 24–25

C COMORBIDITY VERSUS MULTIMORBIDITY

  • CLIMB’s target is an unordered set of concurrent diagnoses, with no clinically privileged first diagnosis or index condition.The presentation opens the consultation but does not assign a diagnosis priority in the target set or scoring.Source: C COMORBIDITY VERSUS MULTIMORBIDITY · p. 26

D EXPERIMENTAL PROTOCOLS, PROMPTS, AND ANALYSIS

This section documents the protocols for collecting and analyzing CLIMB consultations, predictions, and diagnostic trajectories, along with prompt adaptations. It examines diagnostic errors and recovery across analyses, including a matched respiratory experiment, uncertainty, condition-count effects, and over-calling.

D.1 DATA, MODELS, AND DECODING

  • The doctor roster spans frontier closed and strong open models; a stratified patient screen selected DeepSeek-V4.1-Flash as the cheapest candidate within 0.05 of the best score.The selected patient model also serves as a doctor, and the same patient model is used across doctors, so residual patient error is shared across models.Source: §D.1 · p. 27

D.2 CONSULTATION PROTOCOL AND TASK MAPPING

  • Interactive doctors are instructed to investigate after identifying one likely condition, while oracle-count and single-diagnosis controls change the requested diagnosis count.The protocol records natural stopping and final predictions, then separately probes supported diagnosis sets at observed prefixes without feeding probe outputs back into the consultation.Source: §D.2 · pp. 27–29

D.3 SCORING AND DIAGNOSTIC TRAJECTORIES

  • Predictions are scored per case using exact-set, Jaccard, and exhaustive correctness categories; trajectory measures distinguish labels recovered by probes from those retained in the committed prediction.Candidate names are matched case-insensitively, duplicates are removed, and unmatched strings count as false positives; probe outputs measure readout retention, not an unobserved internal state.Source: §D.3 · p. 30

D.4 QUESTION SELECTION AND EMPIRICAL INFORMATION GAIN

  • Empirical information gain compares the selected question with the best available question at the same prefix, while the reference tree uses multilabel entropy and a 14-depth cap.Gains are unavailable, not zero, for prefixes with fewer than 30 matching records; the reference tree is fit on 8,000 generated training cases with minimum leaf size 20.Source: §D.4 · p. 30

D.5 REALIZED FINDING OVERLAP AND PRESENTING-DIAGNOSIS RECOVERY

  • Finding overlap counts shared retained variable–answer pairs across disease components, and presenting-diagnosis contrasts are observational because the generator selects the opening diagnosis using severity.Recovery and omission associations are assessed separately within each cardinality stratum; bootstrap intervals preserve condition pairs in the integration analyses.Source: §D.5 · pp. 30–31

D.6 MATCHED RESPIRATORY COHORT AND THE EVIDENCE CROSS-OVER

  • The learned reference tree reaches a leaf before its 20-question budget on every test path, and its unrestricted fit is no more accurate than a depth-10 tree.The fixed reader’s validation accuracy on k = 2 cases rises from 27% with nine random answers to 50% with twenty and 71% with the full record; its absolute accuracy is source-relative.Source: §D.6 · p. 32, Table 10 · p. 32
  • After eight questions, the fixed reader recovers the exact set from GPT-5.6’s evidence in 50% of k = 1 cases and 17% of k = 2 cases, versus 92% and 72% from the tree’s evidence.The matched respiratory cohort pairs two-diagnosis patients with single-diagnosis patients built from the same source component and opening; questions come from a fixed menu with deterministic answers.Source: §D.6 · pp. 31, 33, Figure 10 · p. 34

D.7 INFORMATION GAINED OVER THE CONSULTATION

  • Over the tree’s question count, tree questions gain 0.42 bits at k = 1 and 0.53 at k = 2, while GPT-5.6 questions gain 0.27 and 0.26 bits.After 20 questions, belief on the true set is 0.86 and 0.57 for GPT-5.6, versus 0.97 and 0.76 for the tree and 0.97 and 0.81 for the full record; GPT-5.6 asks 21% and 12% of the tree’s questions at k = 1 and k = 2.Source: §D.7 · p. 34
  • The information-gain values describe the observer, not GPT-5.6’s internal belief, and Naive Bayes overstates expected information from multi-select answers by treating their values as conditionally independent.The analysis therefore reports information actually gained after each answer, rather than expected gain for each question.Source: §D.7 · p. 35

D.8 PROMPT PROVENANCE AND ADAPTATIONS

  • The free-text doctor and patient prompts adapt AgentClinic, adding comorbidity instructions and a candidate list for the doctor; the patient receives rendered case findings, while menu controls supply separate absent/unknown-finding handling.The static full-information prompt adapts MediQ, and reading a consultation’s own findings with it changes exact recovery by at most four points.Source: §D.8 · p. 35

D.9 IMPLEMENTED PROMPT TEMPLATES

  • Implemented dialogue prompts cap questioning at 20, require continued probing after a likely diagnosis, and ask patients to answer in dialogue without explicitly revealing disease names.The menu doctor uses one ASK <id> line per turn, while static diagnosis, replay, and tree-guided tasks share a diagnosis template with different finding lists.Source: §D.9 · pp. 36–37

E.1 CONFIDENCE INTERVALS FOR THE MAIN TABLES

  • At k ≥3, every model remains at or below 7% exact-set recovery in interaction, with upper confidence bounds of at most 10%; pooled recovery is resolved to ±0.01 to ±0.03.The 95% intervals use 4,000 case-bootstrap draws stratified by k, with the same resampled cases for each model and condition; close Table 1 models are not separated.Source: Table 14 · p. 39, Table 13 · p. 39, §E.1 · p. 39

E.2 PER-COUNT SCORES AND OUTCOMES

  • Most failures are partially incorrect at every condition count: models add wrong diagnoses rather than return a correct subset, and predicted set size grows with k, exceeding the truth up to k = 3.A correct count rarely comes with the correct set.Source: §E.2 · p. 40, Figure 12 · p. 40, Table 15 · p. 40

E.3 ORACLE COUNT

  • With the true k supplied, exact-set recovery still collapses as k increases; oracle count helps at k = 1 but has mixed effects at k ≥2 versus standard interaction.Oracle-count runs use the first 50 cases per k of the main evaluation.Source: §E.3 · p. 40, Figure 14 · p. 41

E.4 OVER-CALLING AND THE INTERACTION BETWEEN COUNT AND CONDITION

  • Across six models, precision stays near 55% with full information and 35% in interaction, while recall falls from 89% to 55% and from 70% to 32%, respectively.At k = 1, the loss comes from extra labels—models return about two diagnoses per case—whereas with co-occurring diagnoses it shifts to missed diagnoses; Jaccard falls similarly with k in both conditions, with pooled interaction contrast near zero.Source: §E.4 · p. 41, Figure 15 · p. 42, Figure 16 · p. 42

E.5 SINGLE-DIAGNOSIS CONTROL

  • When asked for one diagnosis, Gemma names a true condition in roughly half of cases at every k but recovers the exact set in few cases with or without the true k.Among true hits, it names the presenting-complaint disease in 68% at k = 2, 82% at k = 3, and 43% at k = 4, above equal-likelihood rates of 50%, 33%, and 25%.Source: §E.5 · p. 42

E.6 EVIDENCE ACQUISITION AND USE: SUPPORTING ANALYSES

Across the supporting analyses, GPT-5.6 loses more focal-diagnosis recovery than a fixed reader when a second disease is added, while a designed cohort finds opening-diagnosis preference in both models.

E.6 EVIDENCE ACQUISITION AND USE: SUPPORTING ANALYSES

  • GPT-5.6’s excess focal-diagnosis loss over the fixed reader is resolved with the tree’s findings (−8.9 points, 95% interval [−16.1, −1.8]) but not the full record (−4.8, [−11.3, 1.2]).The paired comparison scores the same focal diagnosis alone versus with a second disease.Source: §E.6 · pp. 42–43, Table 18 · p. 43
  • GPT-5.6’s growing question-related cost holds across other metrics and without the two rhinosinusitis labels, while the direct tree-versus-GPT-5.6 gap does not clearly widen with k.The direct gap mixes the effects of questioning and diagnosis.Source: §E.6 · p. 43, Table 19 · p. 44

E.7 DESIGNED k = 2 COHORT

  • The designed cohort uses equally severe DDXPlus pairs with low or high symptom-profile overlap and 195 patients covering 47 diagnoses, each consulted twice.The opening names one diagnosis and two findings exclusive to it within the pair; consultations use a 20-question budget.Source: §E.7 · p. 43
  • Both models recover the opening diagnosis more often in the designed k = 2 cohort, with twice as large an effect for GPT-5.6; both more often return A alone than both diagnoses.The cohort holds diagnosis severity equal and varies symptom-profile overlap across diagnosis pairs.Source: §E.7 · p. 43

E.8 SYMPTOM-SHARING COHORT

The controlled symptom-sharing probe finds that shared symptoms can raise recognition of a second diagnosis without reliably improving exact-set recovery. In main consultations, the opening diagnosis pulls ahead as questioning proceeds, and the cohort probe is limited in scope.

E.8 SYMPTOM-SHARING COHORT

  • The probe compares the same patient records opened with shared symptoms or symptoms specific to A, with a B-symptom opening as a control where possible.It analyzes 97 patients with diagnoses of severity 3 to 5; 65 have all three openings.Source: §E.8 · p. 44
  • Shared symptoms behave more like symptoms of B than symptoms of A, but naming both rises only from 27% to 32%, not reliably.Among 24 patients whose B opening matches the shared opening in kind, the difference is +8 points [−14, 25].Source: §E.8 · p. 45, Table 21 · p. 45, Figure 19 · p. 46
  • In main consultations, 31% of second diagnoses have the chief complaint in B’s profile; within the same B diagnosis, naming rises by 5 to 13 points across all six models.Without matching on diagnosis, the difference is reliable in only 2 of 12 comparisons; the pattern also appears with the full record.Source: §E.8 · p. 45
  • The controlled cohort covers 21 disease pairs and one doctor model, so it is a probe rather than a population estimate.Its openings are built for the test, and the increase in consultations naming both diagnoses is not reliable at this sample size.Source: §E.8 · p. 46

E.9 ANCHORING IN THE MAIN CONSULTATIONS

  • The opening diagnosis’s lead over an equally severe second diagnosis widens from 8 points at the opening to 16 points when the doctor stops.The other diagnosis never appears before the stop in 46% of consultations; the median stop is twelve of twenty questions.Source: §E.9 · p. 46

E.10 POST-STOP BEHAVIOUR ACROSS MODELS

Post-stop probing across models does not show that further questioning completes the diagnosis set. The accompanying framework formalizes the task and contrasts set tracking with single-hypothesis tracking.

E.10 POST-STOP BEHAVIOUR ACROSS MODELS

  • Across four open models, F1 at the question cap is no higher than at the natural stop, and models differ mainly in whether they refuse to keep asking.In these probe runs the open models stop at a median of 8 to 14 questions, compared with 17 to 20 in the main evaluation.Source: §E.10 · p. 46
  • At k = 1, exact-set recovery falls from 25% after three questions to 15% at the cap; at k ≥3, probes almost never hold the complete set.Aligned at the natural stop, probe Jaccard and exact-set recovery do not change over six forced questions.Source: §E.10 · p. 47, Figure 22 · p. 48

F FORMAL FRAMEWORK

  • The framework contrasts a set tracker that treats diagnoses as separate hypotheses with a single-hypothesis tracker that carries one working diagnosis at a time.It states what each idealized reference doctor implies and which implications the data can test, not how a language model computes its answer.Source: F FORMAL FRAMEWORK · p. 50

F.1 SETUP

  • The formal task hides the target set, its size, and remaining findings while the doctor sees candidate labels and an opening presentation.A case has k co-occurring conditions; under full information, the whole record is disclosed and the consultation stops at τ = 0.Source: §F.1 · p. 50

F.2 TWO REFERENCE DOCTORS

  • The single-hypothesis tracker chooses its working diagnosis from the opening and asks only for findings associated with that hypothesis.The contrasting set tracker assumes a label is named as often alongside other conditions as when it occurs alone.Source: §F.2 · p. 50

F.3 RESULTS

The formal results show why single-answer and count-based scores can obscure errors in recovering a full diagnosis set. They also derive testable distinctions between separate-hypothesis and single-hypothesis tracking.

F.3 RESULTS

  • Exact-set recovery cannot exceed the lowest per-label recall among the true diagnoses, while single-answer success can conceal extra labels at k = 1.At k = 1, the single-answer and exact-set rates differ when the true diagnosis is named alongside additional labels.Source: §F.3 · p. 50
  • With full information, the six models return the right count in 41%, 28%, 19%, and 14% of cases at k = 1 to 4; among those, the set is right in 84%, 47%, 22%, and 7%.Exact-set recovery requires both the right count and the right label identities.Source: §F.3 · pp. 50–51
  • Separate-hypothesis tracking predicts no interference: a diagnosis should be named equally often alone and with other diagnoses when no other component meets its informative findings.The claim concerns matched records built from the same component and without background.Source: §F.3 · p. 51
  • In the matched respiratory cohort, GPT-5.6 loses 8.3 points with the full record and 11.3 with the tree’s findings when diagnoses co-occur; the fixed per-label reader loses 3.6 with the full record.The model’s excess loss over the reader is resolved with the tree’s findings, but not with the full record.Source: §F.3 · p. 51
  • Under exclusive belief, a second diagnosis gains support only at the expense of the working diagnosis, and once that diagnosis is settled the tracker has little reason to seek a second one.The result follows from the tracker’s limited information gain for findings after its working diagnosis becomes strongly supported.Source: §F.3 · p. 51
  • More questions raise the correct count only if they reach a missing diagnosis’s informative findings; under concentration, that requires the working hypothesis to move.Without a question reaching those findings, an ungrounded true diagnosis remains ungrounded at the stopping point.Source: §F.3 · pp. 51–52

Related papers

Loading 2609.35462v1…