Benchmarking Clinical Decision Pathway Adherence in Large Language Models

• Clinical decision-making requires multi-step reasoning over heterogeneous patient evidence, so evaluation should assess decision processes alongside final answers.Source: §1 · p. 1
• Omitting or misordering intermediate CDP steps can produce clinically unsafe, non-explainable, or difficult-to-review decisions despite correct outcomes.Source: §1 · p. 1
• Existing studies largely overlook guideline variation and updates, motivating evaluation of models using provided guidelines as references for faithful CDP generation.Source: §1 · p. 1
• MEGA-CDP transforms 2,274 English and Chinese guidelines into over 42,353 pathway-aligned cases and reference CDPs through an automated guideline-to-case pipeline.Source: §1 · p. 1
• The benchmark evaluates models in single-turn vignette and multi-turn interactive settings, with a CDP-oriented framework for measuring pathway consistency.Source: §1 · pp. 1–2

• MedQA, MedMCQA, and PubMedQA evaluate medical question answering primarily through exam-style tasks, while emrQA and MedDialog add richer clinical contexts.Source: §2 · p. 2
• DR.BENCH, MedJourney, and LLMEval-Med extend evaluation toward diagnostic reasoning, progressive-information decision-making, and judged clinical reasoning.Source: §2 · p. 2
• MIMIC-CDM, CPGBench, and MedGUIDE evaluate guideline-consistent or CDP-adherent recommendations, including multi-turn and flowchart-based settings.Source: §2 · p. 2
• MEGA-CDP represents guidelines as CDPs linked to structured case vignettes and canonical reasoning sequences, enabling process assessment of guideline adherence.Source: §2 · p. 2

Data Curation Pipeline

• The data pipeline extracts guideline-defined CDPs and synthesizes corresponding clinical case vignettes for downstream evaluation.Source: §3.1 · pp. 2–3

Framework

• Figure 2 organizes MEGA-CDP into guideline collection, guideline-to-case construction, task settings, and CDP-oriented evaluation.Source: Figure 2 · p. 3

Task Settings

• The single-turn task provides a guideline and case vignette, requiring a step-by-step predicted CDP and final decision outcome.Source: §3.2 · p. 3
• The multi-turn task has the model ask one question per turn, receive patient information from an environment, and continue until a final decision.Source: §3.2 · p. 4

CDP-Oriented Evaluation

• The evaluation first scores consistency between predicted and reference steps, then aggregates scores into pathway-level consistency using dynamic time warping.Source: §3.3 · p. 4
• DTW converts step-consistency scores into alignment costs and preserves decision order while aligning sequences of potentially different lengths.Source: §3.3 · p. 4
• The pathway consistency metric reflects order alignment, with lower PCC values indicating better alignment, and is compared with alternative metrics and human judgments.Source: §3.3 · p. 4

Evaluation

• PCC measures pathway consistency cost alongside outcome accuracy and success rate across the two evaluation settings.Lower PCC and higher Acc and SR are preferred.Source: Table 1 · p. 5

Experimental setup

• 7,458 cases from 440 English and Chinese guidelines are evaluated in both single-turn vignette and multi-turn interactive settings.The test set contains 210 Chinese and 230 English guidelines, with each instance including a guideline, vignette, outcome, and reference CDP.Source: §4.1 · p. 5

Main results

• Proprietary models generally perform strongest, while specialized medical models do not consistently outperform general models in guideline adherence.Baichuan-M3-235B’s PCC degrades under both settings despite clinical decision-making fine-tuning.Source: §4.2 · p. 5

Multi-turn interactive setting

• 26.98 percentage points higher PCC and 40.51 percentage points lower outcome accuracy occur on average in multi-turn versus single-turn evaluation.Claude-Sonnet-4.6 performs best in the multi-turn setting, with PCC of 55.92% and outcome accuracy of 45.40%.Source: §4.2 · p. 6

Pathway deviations

• Correct final decisions can coexist with process deviations such as omission, redundancy, skipping, and reversal.Figure 3 illustrates this pathway-level inconsistency in a model-generated CDP.Source: §4.2 · p. 6, Figure 3 · p. 6

Human alignment

• 88.5% and 91.5% agreement rates across two human-preference groups yield 90.0% overall agreement between PCC and human judgments.Agreement between the two human annotators is also 90.0%.Source: §4.3 · p. 6

Metric and judge analysis

• GPT-5.4 Mini and Qwen3-8B correlate strongly across metrics, while DTW and OT show higher pairwise consistency when the judge model is fixed.The analysis selects GPT-5.4 Mini as the default judge and DTW as the primary metric.Source: §4.5 · p. 7

• MEGA-CDP is a large-scale benchmark for evaluating guideline-adherent clinical decision pathways generated by medical LLMs.Source: §5 · p. 7
• The benchmark uses an automated guideline-to-case pipeline to extract guideline-defined pathways and synthesize pathway-aligned clinical cases.Source: §5 · p. 7
• MEGA-CDP supports both single-turn vignette and multi-turn interactive evaluation settings.Source: §5 · p. 7
• Outcome accuracy and pathway consistency capture different aspects of clinical decision-making.Source: §5 · p. 7
• Experiments on 16 representative LLMs show that both task settings remain substantially challenging and specialized medical models do not consistently outperform general-purpose models.Source: §5 · p. 7

• Closed-source models were accessed through official APIs, while open-source models were deployed locally using GPU-based inference.Table 3 summarizes the hardware and software configurations for local inference and evaluation.Source: A Reproducibility Details · p. 10
• Random seeds were fixed to 42 for local experiments involving stochastic components.This configuration was used to ensure reproducibility.Source: A Reproducibility Details · p. 10

• OT establishes global correspondences between reference and predicted pathway steps without enforcing strict sequential alignment.The method uses overall matching cost rather than strict sequence-by-sequence alignment.Source: §B.1 · p. 10
• The consistency-based distance is defined as 1 − C_i,j, converting step-level consistency scores into transport costs.Source: §B.1 · p. 10
• A relative positional penalty encourages order-aware alignment by comparing normalized positions of reference and predicted steps.Source: §B.1 · p. 10
• The unified transport cost combines consistency-based distance and positional alignment, with α = 1.0 and β set to 0.2 for GPT-5.4 Mini and Qwen3-8B and 0.1 for BERTScore.Source: §B.1 · p. 10
• The transport plan assigns mass between reference and predicted steps, using uniform marginals within an unbalanced KL-regularized OT formulation.Unbalanced OT permits partial mass variation while softly penalizing deviations from the marginal distributions.Source: §B.1 · p. 10
• PCCOT computes normalized total transport cost from the optimal plan, where lower values indicate better reference–prediction alignment.Source: §B.1 · p. 10

LNDS-based Measurement

• LNDS identifies the largest subset of matched steps that preserves decision order, unlike DTW and OT's more flexible matching.Source: §B.2 · p. 10
• Candidate matches are selected by highest step-level consistency and retained only when C_i,j_i ≥ γ.Matches below the threshold are treated as unmatched before sequence analysis.Source: §B.2 · p. 10
• The longest non-decreasing subsequence of matched predicted-step indices represents the largest order-preserving alignment with the reference pathway.Source: §B.2 · p. 10
• PCSLNDS normalizes the subsequence length m′ by the number of reference steps m.Source: §B.2 · p. 11
• Higher PCSLNDS indicates better adherence to the guideline-defined decision order.Source: §B.2 · p. 11

• CDP length counts the non-leaf nodes along a root-to-leaf pathway, representing clinical decision conditions before the final outcome.Source: C Statistics of CDP Lengths · p. 11
• Figure 5 shows reference CDP-length distributions for the full CDP set and its test and non-test subsets, with the long-tail region magnified.Source: C Statistics of CDP Lengths · p. 11
• The test and non-test subsets exhibit similar CDP length statistics, suggesting that the held-out test set preserves the overall CDP set’s structural characteristics.Source: C Statistics of CDP Lengths · p. 11

• Similar model rankings and relative performance trends are generally preserved across different judge models.The reported 95% confidence intervals are estimated from 10,000 bootstrap resampling iterations.Source: D Additional Evaluation Results · p. 11
• DTW-PCC, OT-PCC, and LNDS-PCS show consistent relative trends across models in both single-turn and multi-turn settings.These metrics provide broadly consistent assessments despite using different mathematical formulations.Source: D Additional Evaluation Results · p. 11
• Narrow confidence intervals indicate that the estimated pathway-consistency scores are stable.Source: D Additional Evaluation Results · p. 11

Additional Evaluation Outputs

• Tables 5–7 define DTW-, OT-, and LNDS-based pathway metrics, with lower PCC values or higher PCS values indicating better pathway consistency.Source: Table 5 · p. 12, Table 6 · p. 13, Table 7 · p. 14

Reasoning Tree Extraction

• The reasoning-tree prompt converts guideline content into a JSON decision tree with yes/no branches and patient-specific clinical criteria.It separately records the guideline’s primary clinical focus without representing that focus as a decision node.Source: E Prompt Templates · p. 15
• Tree construction requires explicit, objective criteria, one independent variable or threshold per node, sequential representation of multiple criteria, and no external medical knowledge.Ambiguous guideline text should be preserved without speculation.Source: E Prompt Templates · p. 15
• Each decision-tree node receives a unique numeric index and exact supporting guideline sentences drawn from clinically meaningful recommendations or statements.Source: E Prompt Templates · p. 15

Multi-Turn Step-Level Consistency Scoring

• The multi-turn templates represent clinician questioning pathways as ordered lists of questions for comparing reference and candidate information-gathering sequences.Source: E Prompt Templates · pp. 20, 23