Source-linked AI summary
A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo
TL;DR
Exam-style accuracy may not reflect clinical reasoning over records, and existing instruments do not yet cover the defined construct. This review maps relevant assessment and evaluation literatures and finds that tools must be combined, with further work needed for longitudinal free-text reasoning.
Problem
Exam-style accuracy may poorly predict performance on clinical cases, and it remains unclear which instruments measure reasoning over serial records.
Source: §1 · p. 2Method
Results
No reviewed instrument covers all six dimensions; existing literatures offer partial coverage for problem representation, temporal reasoning, and sequential updating, while faithfulness remains to be tested in free-text settings.
Source: §5 · pp. 4–5, §8 · p. 8, §14 · pp. 9–10Takeaways & Limitations
The review recommends combining existing tools and developing a carefully specified instrument that tests uncertainty, counterfactual robustness, and faithfulness over serial records.
Source: §12 · p. 9, §14 · p. 10This structured narrative review used single-reviewer screening without dual screening or a formal flow diagram, and is not a PRISMA-compliant scoping review.
Source: §13 · p. 9
Abstract
from arXiv · showhide
Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.
1 Introduction
Exam-style accuracy is reproducible but may poorly reflect clinical performance, motivating evaluation of reasoning over records. This review defines that construct and maps its scope and approach.
1 Introduction
- Exam-item correctness only weakly predicts performance on real clinical cases, and model rankings can reverse on health-record cases.The review therefore asks what reasoning instruments actually measure, rather than treating exam accuracy as sufficient.Source: §1 · p. 2
- Clinical reasoning is operationalised as integrating and updating evidence across time and sources to form, revise, and justify patient representations and defensible investigation or management decisions.The definition targets longitudinal, multi-document records where later entries may supersede earlier ones and spans six dimensions, including temporal synthesis, uncertainty, counterfactual reasoning, and faithfulness [Turpin et al., 2023].Source: §1 · p. 2
- The review asks which instruments measure this construct and what must be added or combined to evaluate reasoning over serial clinical records.The question matters to evaluation and fine-tuning because tools built for other purposes may not measure what their labels suggest.Source: §1 · p. 2
2 Scope and approach
- This is a structured narrative review using purposive, iterative source selection, citation chasing, and single-reviewer screening, without quantitative synthesis.It covers LLM evaluation work from 2023 onwards, includes flagged preprints as provisional, and retained no complete search log.Source: §2 · p. 2
- Decision-support studies assess effects on clinical behaviour or outcomes, think-aloud methods collect protocols rather than score static text, and error taxonomies focus on outcomes [Wright and Sittig, 2008; Fonteyn et al., 1993, Pinnock et al., 2015; Agency for Healthcare Research and Quality, n.d., Singh and Sittig, 2015].These adjacent literatures are excluded because they do not directly score reasoning processes in static model outputs.Source: §2 · pp. 2–3
3 The landscape in outline
The reviewed literature has no single instrument for the defined construct. Medical education tools offer useful anchors for problem representation and differential reasoning, but their settings and assumptions limit coverage.
3 The landscape in outline
- No single reviewed rubric or benchmark captures the defined form of clinical reasoning; the literature divides among medical education tools, clinical LLM benchmarks, and general evaluation methods.The strands overlap but serve different purposes.Source: §3 · p. 3
4 The medical education strand
- IDEA scores interpretive summary, differential diagnosis, reasoning explanation, and alternatives; its original studies reported ICCs of 0.53 for reporting, 0.64 for diagnostic reasoning, and 0.63 for decision-making.Revised-IDEA reported an ICC of 0.84 (95% CI 0.74–0.90).Source: §4 · p. 3
- R-IDEA assesses how well reasoning is documented in admission notes, supporting problem representation and parts of differential reasoning, but not temporal, uncertainty, counterfactual, or faithfulness dimensions.Its reliability for LLM outputs is not established in this passage.Source: §4 · p. 3
- ART structures human diagnostic reasoning assessment around problem representation, prioritised differentials, and metacognition, but its presentation-based setting assumes sincere self-report.Source: §4 · p. 3
- Scaled SCT uses 750 validated items from ten datasets and finds lower performance than on conventional multiple-choice exams, alongside overconfidence and underuse of neutral belief ratings.It provides a constrained proxy for belief updating, not a score for free-text uncertainty handling.Source: §4 · p. 3
- Medical education tools are strongest for human problem framing, differential prioritisation, and plan justification, but generally treat written or spoken reasoning as a meaningful account of process [Baker et al., 2015, Schaye et al., 2022, Thammasitboon et al., 2018, Lubarsky et al., 2013].IDEA validity evidence already strains that assumption for human authors, and it is shakier for LLM outputs.Source: §4 · pp. 3–4
5 The clinical LLM benchmark strand
Clinical LLM benchmarks add measures for temporal synthesis, sequential updating, uncertainty, counterfactuals, and safety-weighted completeness, but structured tasks and text-only inputs leave important limits.
5 The clinical LLM benchmark strand
- Many benchmarks judge answer quality rather than reasoning mechanisms; DR.BENCH's ROUGE-L and F1 scores likewise measure task performance, not broad reasoning quality [Gao et al., 2023].HealthBench offers weighted physician-authored criteria, while OLAPH is a clinical entry point to long-form factuality evaluation [Jeong et al., 2024].Source: §5 · pp. 4–5
- MedR-Bench compares generated reasoning with reference cases across 1,453 diagnosis and treatment-planning cases, but does not establish whether stated reasoning caused the answer.It is a partial proxy for reasoning quality rather than a solution to faithfulness.Source: §5 · p. 4
- TIMER-Eval tests temporal boundaries, trends, and chronological order, but does not test whether later records supersede earlier ones [Cui et al., 2025].It addresses recency imbalance with Edge and Uniform distributions; 55.3% of MedAlign instructions refer to the final timeline quarter.Source: §5 · p. 4
- ER-Reason tests sequential belief updating in emergency-department notes, while clinician-cohort comparisons found strong text-only triage and uncertainty performance bounded by absent bedside cues [Mehandru et al., 2025; Brodeur et al., 2026].ER-Reason's structured tasks do not establish faithfulness of unrestricted free-text reasoning or a general multi-document rubric.Source: §5 · p. 4
- Across 21 frontier models and 29 vignettes, performance was strongest at final diagnosis and management but weakest in early reasoning and differential generation, exposing premature diagnostic closure [Rao et al., 2026].The review argues this supports scoring reasoning trajectories rather than endpoints.Source: §5 · p. 5
- Safety-weighted scenarios expose a priority gap: models scored 80%–90% on simple checklist items but met weight-5 critical criteria at only 32.4%–41.7% [Ismail et al., 2026].Over half of the critical criteria were met by no model.Source: §5 · p. 5
- MedThink-Bench scores intermediate reasoning points across 500 question-answer pairs and reports strong correlation with clinician judgment, but correlation is not chance-corrected agreement.Source: §5 · p. 5
- Uncertainty-preservation, counterfactual-pair, omission-detection, and confidence-conditioned simulation studies address emerging gaps, though their designs are narrow or findings provisional [Du et al., 2026].MamaBench uses paired vignettes, and the Meincke working paper is not yet peer-reviewed.Source: §5 · p. 5
6 Evaluation methodology as a separate literature
General-domain evaluation offers checklist, factual-completeness, and judge-reliability methods relevant to clinical rubrics, but does not yet address longitudinal records or whether stated reasoning caused an answer.
Evaluation methodology as a separate literature
- General-domain judging moved from holistic scores to fine-grained criteria, while factuality methods verify atomic-claim precision without checking whether required claims are missing [Liu et al., 2023] [Zheng et al., 2023] [Kim et al., 2024] [Ye et al., 2024] [Min et al., 2023] [Song et al., 2024] [Wei et al., 2024].These two methodological lineages inform tools for clinical reasoning rubrics.Source: §6 · pp. 5–6
- Binary checklist questions raised average inter-evaluator agreement by 0.45 across twelve judge models, but a separate audit found raw agreement overstated chance-corrected kappa by 33 to 41 percentage points [Lee et al., 2025].The audit also found judge rankings shifted across benchmarks and high test-retest consistency could coexist with severe position bias.Source: §6 · p. 6
- ExpertLongBench found models covered over two-thirds of checklist items on average, yet the best model’s overall task F1 was 33.4 and coverage correlated only weakly with accuracy [Ruan et al., 2025].The authors warn that high coverage can make incorrect output appear expert-aligned.Source: §6 · p. 6
- General-domain findings favor scoring completeness separately from precision: omission dominated GAMUT failures, while clinical studies also report important omissions [Jafari et al., 2026] [Chen et al., 2026] [Wanner et al., 2025].GAMUT is a general-domain multimodal benchmark with roughly one percent medical content, so its omission result is extrapolated rather than clinical evidence.Source: §6 · p. 6
- The proposed design uses human-facing anchors compiled into binary items, separate completeness and correctness scores, case-specific importance weights, and chance-corrected reliability and position-bias checks [Lee et al., 2025] [Chen et al., 2026] [Ruan et al., 2025] [Jafari et al., 2026] [Rao and Callison-Burch, 2026].These are design recommendations, not implemented or validated features.Source: §6 · p. 6
- Existing general-domain frameworks neither test whether stated reasoning caused an output nor accommodate longitudinal records in which later documents supersede earlier ones.A temporal component should check event order and whether later documentation supersedes earlier information.Source: §6 · p. 7
7 What is covered, what is partially covered, and what is missing
Coverage across six clinical-reasoning dimensions is uneven: some reasoning skills are relatively well covered, while temporal synthesis, uncertainty, counterfactual reasoning, and faithfulness retain important gaps.
What is covered, what is partially covered, and what is missing
- Problem representation and differential or management reasoning are relatively well covered, while temporal reasoning is directly measured mainly by TIMER-Eval and its supersession problem remains open [Cui et al., 2025].Table 1 summarizes coverage of the six clinical-reasoning dimensions by existing instruments and benchmarks.Source: §7 · p. 7, Table 1 · p. 7
- Calibrated uncertainty and counterfactual reasoning have dedicated clinical operationalisations, but neither has a reusable rubric for longitudinal free-text outputs [Du et al., 2026].The reviewed tools use static or paired vignettes rather than serial records.Source: §7 · p. 7
- Factual completeness is well theorised in general-domain work and has early clinical instantiations [Jafari et al., 2026] [Chen et al., 2026] [Wanner et al., 2025].Source: §7 · p. 7
- Reasoning faithfulness remains the least-covered dimension, although the review identifies an initial clinical test.Source: §7 · p. 7
8 Reasoning faithfulness
Coherent model explanations may not reflect the processes that produced answers, leaving faithfulness distinct from conventional reasoning assessment. A clinical causal-ablation study offers a partial test, but not a longitudinal free-text rubric.
Reasoning faithfulness
- Chain-of-thought studies show answers can be steered by unmentioned biasing features, and realistic prompts also elicit implicit post-hoc rationalisation at varying rates across production models [Turpin et al., 2023] [Arcuschin et al., 2025].Medical education tools generally treat explanations as part of reasoning performance, an assumption that needs testing for LLMs.Source: §8 · pp. 7–8
- In a clinical causal-ablation probe, only around ten percent of chain-of-thought steps were individually causal, and removing steps improved predictions about as often as it harmed them.The workshop-track study used three closed-source models and multiple-choice questions; single-step redaction cannot distinguish unnecessary steps from those redundant with retained content.Source: §8 · p. 8
- The identified gap is a rubric-integrated faithfulness test for free-text longitudinal reasoning; the review finds no clinical work measuring latent-versus-verbalised reasoning divergence.Source: §8 · p. 8
- Faithfulness is treated as graded rather than binary and requires intervention or counterfactual probing, because checklists assess what was stated, not why it was stated.The proposed direction extends causal ablation from multiple-choice answers to free-text differential generation.Source: §8 · p. 8
9 Counterfactual reasoning
Counterfactual clinical reasoning has a transferable measurement design, but no reusable rubric for longitudinal free-text outputs.
Counterfactual reasoning
- MamaBench shows counterfactual clinical robustness can be scored with paired perturbations and a Bias Trap Rate metric, but is limited to one specialty and short paired vignettes.Source: §9 · p. 8
- The reviewed sources offer no reusable tool for testing counterfactual reasoning in longitudinal free text, where a relevant change may occur several documents earlier.Source: §9 · p. 8
10 Calibrated uncertainty
Evidence suggests that models can struggle to preserve or express clinically appropriate uncertainty, while early sequential-diagnosis data show confidence may carry useful signal. Evaluation of uncertainty as it changes across longitudinal records remains missing.
10 Calibrated uncertainty
- Models preserve source diagnostic uncertainty in under half of cases, while early simulated sequential-diagnosis data suggest stated confidence carries clinical signal.The reviewed evidence also reports premature diagnostic closure and a bias toward extreme SCT responses with near-avoidance of neutral ratings.Source: §10 · p. 8
- Models may collapse complex differentials into premature closure and show limited subtle probabilistic adjustment when new information arrives.These patterns are reported in separate studies of competing possibilities and expert-panel SCT distributions.Source: §10 · p. 8
- Longitudinal evaluation still needs to test calibration as the appropriate uncertainty level changes while records accumulate.Source: §10 · p. 8
11 Factual completeness
Clinical completeness evaluations show that low-stakes coverage can conceal omissions of safety-critical actions, so completeness should not be treated as a simple proportion. The review recommends combining weighted, safety-aware scoring with separate correctness measures and reliability checks.
11 Factual completeness
- Omission is described as more common and clinically more dangerous than incorrect stated claims, and flat or unweighted checklists can mask critical omissions.The cited general-domain evaluations and clinical weighted-rubric evidence motivate scoring completeness separately from correctness.Source: §11 · pp. 8–9
- Weighted evaluation found high coverage of low-stakes criteria alongside frequent omissions of safety-critical actions, supporting non-compensable safety caps.Examples include emergency escalation and contraindicated-treatment warnings; step-level checks can also detect bypassed logical steps.Source: §11 · pp. 8–9
12 What needs to be combined
- The proposed combination retains existing tools while pairing reasoning-specific measures with separated completeness and correctness, temporal order checks, and chance-corrected reliability.It also calls for binary rubric items and case-specific importance weighting with safety caps.Source: §12 · p. 9
- Several integration details remain planned rather than implemented, and automated grading evidence must be re-established using chance-corrected statistics on a new instrument.The stated planned steps include binary compilation, importance weighting, judge specifications, repeated runs, option-order randomisation, and acceptance thresholds.Source: §12 · p. 9
13 Limitations of this review
The review is a structured narrative map with methodological, evidence-maturity, construct-scope, and synthesis limitations. Its conclusion is that existing tools cover separate dimensions, but a small, reliable instrument for longitudinal reasoning remains to be built.
13 Limitations of this review
- Single-reviewer screening without dual screening or a formal flow diagram makes this a structured narrative map rather than a PRISMA-compliant scoping review.Source: §13 · p. 9
- Preprint findings are provisional, coverage judgements are relative to the review’s operational definition, and no quantitative synthesis was attempted because outcomes differ.Reported figures were taken from primary sources without re-analysis.Source: §13 · p. 9
14 Conclusion
- No reviewed instrument covers all six dimensions, while the literatures supply distinct anchors and methods for assessing longitudinal clinical reasoning.The conclusion distinguishes clinically meaningful problem-representation and differential-reasoning anchors from setting-specific benchmark designs and general evaluation methods.Source: §14 · pp. 9–10
- The next step is a small instrument for serial records that tests uncertainty, counterfactual robustness, and reasoning faithfulness, with reliability established using chance-corrected statistics.Source: §14 · p. 10
Declarations
The declarations report funding and support, no competing interests, no new data, and the use of AI tools in manuscript preparation.
Declarations
- The author reports support from the Engineering and Physical Sciences Research Council through DRIVE-Health at King’s College London, plus Cleveland Clinic London support.Source: Declarations · p. 10
- The authors declare no competing interests.Source: Declarations · p. 10
- No new data were generated; sources are cited in the reference list.Source: Declarations · p. 10
- AI-based writing and research-assistance tools supported synthesis and drafting, and the authors reviewed and edited the manuscript and accept responsibility for it.Source: Declarations · p. 10