Source-linked AI summary
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Nuo Chen, Xinyang Jiang, Zilong Wang, Zhifei Zhang, Xiaoye Qu, Jiajun Deng, Yulan Guo, Cairong Zhao
TL;DR
Medical LLM benchmarks have largely emphasized final-answer accuracy, leaving guideline adherence and decision-process fidelity insufficiently evaluated. MEGA-CDP addresses this gap with a large guideline-derived benchmark, dual task settings, and CDP-oriented pathway evaluation. Across 16 LLMs, the experiments show that outcome accuracy and pathway consistency differ, both settings remain challenging, and medical specialization does not consistently improve adherence.
Problem
Existing medical LLM benchmarks largely focus on final outcomes or general rationale quality rather than faithful adherence to guideline-defined clinical decision pathways.
Method
MEGA-CDP uses an automated pipeline to derive CDPs and case vignettes from clinical guidelines, then evaluates generated pathways in single-turn and multi-turn settings using step-level scoring and DTW-based consistency.
Results
Experiments on 16 LLMs show that outcome accuracy and pathway consistency capture different aspects of decision-making, both task settings remain substantially challenging, and medical models do not consistently outperform general-purpose models.
Takeaways & Limitations
MEGA-CDP supports guideline-adherence evaluation that distinguishes correct outcomes from the consistency of the clinical decision pathway.
Takeaways & Limitations
Evaluation was restricted to models supporting English and Chinese inputs with context windows of at least 40K tokens, using default configurations.
Abstract
from arXiv · showhide
Following clinical decision pathways (CDPs) defined by clinical practice guidelines is essential for safe and reliable medical decision-making. However, existing medical large language model (LLM) benchmarks mainly evaluate final-answer accuracy, providing limited evaluation of models' ability to adhere to guidelines. To address this gap, we introduce MEGA-CDP, a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs using provided guidelines as references. MEGA-CDP is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline, yielding 42,353 clinical cases with explicit reference CDPs. It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework for measuring pathway consistency. Experiments on 16 representative LLMs show that reliable clinical decision support remains challenging for current models, demonstrating the need for CDP-oriented evaluation and the value of MEGA-CDP for advancing guideline adherence in medical LLMs.
1 Introduction
MEGA-CDP addresses the need to evaluate whether medical LLMs follow guideline-defined clinical decision pathways, not merely reach correct final decisions. It introduces a scalable benchmark and evaluates pathway adherence in single-turn and interactive settings.
- Clinical decision-making requires multi-step reasoning over heterogeneous patient evidence, so evaluation should assess decision processes alongside final answers.
- Omitting or misordering intermediate CDP steps can produce clinically unsafe, non-explainable, or difficult-to-review decisions despite correct outcomes.
- Existing studies largely overlook guideline variation and updates, motivating evaluation of models using provided guidelines as references for faithful CDP generation.
- MEGA-CDP transforms 2,274 English and Chinese guidelines into over 42,353 pathway-aligned cases and reference CDPs through an automated guideline-to-case pipeline.
- The benchmark evaluates models in single-turn vignette and multi-turn interactive settings, with a CDP-oriented framework for measuring pathway consistency.
2 Related Work
Medical benchmarks have expanded from exam-style question answering toward richer clinical reasoning and decision-making tasks. However, most existing frameworks do not explicitly encode guideline-based decision structures or systematically assess CDP adherence.
- MedQA, MedMCQA, and PubMedQA evaluate medical question answering primarily through exam-style tasks, while emrQA and MedDialog add richer clinical contexts.
- DR.BENCH, MedJourney, and LLMEval-Med extend evaluation toward diagnostic reasoning, progressive-information decision-making, and judged clinical reasoning.
- MIMIC-CDM, CPGBench, and MedGUIDE evaluate guideline-consistent or CDP-adherent recommendations, including multi-turn and flowchart-based settings.
- MEGA-CDP represents guidelines as CDPs linked to structured case vignettes and canonical reasoning sequences, enabling process assessment of guideline adherence.
3 MEGA-CDP
MEGA-CDP constructs guideline-derived cases and evaluates model-generated CDPs by aligning predicted and reference decision sequences. Its framework supports both vignette-based and interactive tasks while accounting for variable pathway length and ordering.
- Data Curation Pipeline: The data pipeline extracts guideline-defined CDPs and synthesizes corresponding clinical case vignettes for downstream evaluation.
- Framework: Figure 2 organizes MEGA-CDP into guideline collection, guideline-to-case construction, task settings, and CDP-oriented evaluation.
- Task Settings: The single-turn task provides a guideline and case vignette, requiring a step-by-step predicted CDP and final decision outcome.
- Task Settings: The multi-turn task has the model ask one question per turn, receive patient information from an environment, and continue until a final decision.
- CDP-Oriented Evaluation: The evaluation first scores consistency between predicted and reference steps, then aggregates scores into pathway-level consistency using dynamic time warping.
- CDP-Oriented Evaluation: DTW converts step-consistency scores into alignment costs and preserves decision order while aligning sequences of potentially different lengths.
- CDP-Oriented Evaluation: The pathway consistency metric reflects order alignment, with lower PCC values indicating better alignment, and is compared with alternative metrics and human judgments.
4 Experiments
Experiments evaluate 16 LLMs on guideline-adherent clinical decision-making in single-turn and multi-turn settings, using outcome and pathway-level measures. Results show substantial variation across models and settings, with multi-turn interaction and pathway adherence remaining especially challenging.
- Evaluation: PCC measures pathway consistency cost alongside outcome accuracy and success rate across the two evaluation settings.Lower PCC and higher Acc and SR are preferred.
- Experimental setup: 7,458 cases from 440 English and Chinese guidelines are evaluated in both single-turn vignette and multi-turn interactive settings.The test set contains 210 Chinese and 230 English guidelines, with each instance including a guideline, vignette, outcome, and reference CDP.
- Main results: Proprietary models generally perform strongest, while specialized medical models do not consistently outperform general models in guideline adherence.Baichuan-M3-235B’s PCC degrades under both settings despite clinical decision-making fine-tuning.
- Multi-turn interactive setting: 26.98 percentage points higher PCC and 40.51 percentage points lower outcome accuracy occur on average in multi-turn versus single-turn evaluation.Claude-Sonnet-4.6 performs best in the multi-turn setting, with PCC of 55.92% and outcome accuracy of 45.40%.
- Pathway deviations: Correct final decisions can coexist with process deviations such as omission, redundancy, skipping, and reversal.Figure 3 illustrates this pathway-level inconsistency in a model-generated CDP.
- Human alignment: 88.5% and 91.5% agreement rates across two human-preference groups yield 90.0% overall agreement between PCC and human judgments.Agreement between the two human annotators is also 90.0%.
- Metric and judge analysis: GPT-5.4 Mini and Qwen3-8B correlate strongly across metrics, while DTW and OT show higher pairwise consistency when the judge model is fixed.The analysis selects GPT-5.4 Mini as the default judge and DTW as the primary metric.
5 Conclusion
MEGA-CDP evaluates whether medical LLMs generate clinical decision pathways that adhere to guidelines across single-turn and multi-turn settings. Experiments show that pathway consistency and outcome accuracy capture different aspects of decision-making, while both settings remain challenging.
- MEGA-CDP is a large-scale benchmark for evaluating guideline-adherent clinical decision pathways generated by medical LLMs.
- The benchmark uses an automated guideline-to-case pipeline to extract guideline-defined pathways and synthesize pathway-aligned clinical cases.
- MEGA-CDP supports both single-turn vignette and multi-turn interactive evaluation settings.
- Outcome accuracy and pathway consistency capture different aspects of clinical decision-making.
- Experiments on 16 representative LLMs show that both task settings remain substantially challenging and specialized medical models do not consistently outperform general-purpose models.
A Reproducibility Details
The experiments combine API-based inference for closed-source models with local GPU-based inference for open-source models. Random seeds were fixed to 42 for stochastic local experiments to support reproducibility.
- Closed-source models were accessed through official APIs, while open-source models were deployed locally using GPU-based inference.Table 3 summarizes the hardware and software configurations for local inference and evaluation.
- Random seeds were fixed to 42 for local experiments involving stochastic components.This configuration was used to ensure reproducibility.
B.1 OT-based Measurement
The OT-based measurement models pathway alignment as a global matching problem using step consistency, positional penalties, and flexible transport. PCCOT normalizes the optimal transport cost, with lower values indicating better alignment.
- OT establishes global correspondences between reference and predicted pathway steps without enforcing strict sequential alignment.The method uses overall matching cost rather than strict sequence-by-sequence alignment.
- The consistency-based distance is defined as 1 − C_i,j, converting step-level consistency scores into transport costs.
- A relative positional penalty encourages order-aware alignment by comparing normalized positions of reference and predicted steps.
- The unified transport cost combines consistency-based distance and positional alignment, with α = 1.0 and β set to 0.2 for GPT-5.4 Mini and Qwen3-8B and 0.1 for BERTScore.
- The transport plan assigns mass between reference and predicted steps, using uniform marginals within an unbalanced KL-regularized OT formulation.Unbalanced OT permits partial mass variation while softly penalizing deviations from the marginal distributions.
- PCCOT computes normalized total transport cost from the optimal plan, where lower values indicate better reference–prediction alignment.
B.2 LNDS-based Measurement
The LNDS-based measurement evaluates whether matched pathway steps preserve decision order. It normalizes the longest non-decreasing subsequence length by the number of reference steps, with higher PCSLNDS indicating better adherence.
- LNDS-based Measurement: LNDS identifies the largest subset of matched steps that preserves decision order, unlike DTW and OT's more flexible matching.
- LNDS-based Measurement: Candidate matches are selected by highest step-level consistency and retained only when C_i,j_i ≥ γ.Matches below the threshold are treated as unmatched before sequence analysis.
- LNDS-based Measurement: The longest non-decreasing subsequence of matched predicted-step indices represents the largest order-preserving alignment with the reference pathway.
- LNDS-based Measurement: PCSLNDS normalizes the subsequence length m′ by the number of reference steps m.
- LNDS-based Measurement: Higher PCSLNDS indicates better adherence to the guideline-defined decision order.
C Statistics of CDP Lengths
Reference CDP lengths are characterized by pathway-level descriptive statistics and distributions across the full, test, and non-test sets. The test and non-test subsets have similar length statistics, indicating comparable structural characteristics.
- CDP length counts the non-leaf nodes along a root-to-leaf pathway, representing clinical decision conditions before the final outcome.
- Figure 5 shows reference CDP-length distributions for the full CDP set and its test and non-test subsets, with the long-tail region magnified.
- The test and non-test subsets exhibit similar CDP length statistics, suggesting that the held-out test set preserves the overall CDP set’s structural characteristics.
D Additional Evaluation Results
Additional evaluations show that pathway-consistency conclusions are broadly stable across judge models and pathway-level metrics. Narrow bootstrap confidence intervals also indicate stable estimated scores.
- Similar model rankings and relative performance trends are generally preserved across different judge models.The reported 95% confidence intervals are estimated from 10,000 bootstrap resampling iterations.
- DTW-PCC, OT-PCC, and LNDS-PCS show consistent relative trends across models in both single-turn and multi-turn settings.These metrics provide broadly consistent assessments despite using different mathematical formulations.
- Narrow confidence intervals indicate that the estimated pathway-consistency scores are stable.
E Prompt Templates
The prompt templates define MEGA-CDP’s guideline extraction, case-oriented inference, and pathway-consistency scoring procedures. They operationalize clinical pathways as structured decision or questioning sequences and require standardized outputs.
- Additional Evaluation Outputs: Tables 5–7 define DTW-, OT-, and LNDS-based pathway metrics, with lower PCC values or higher PCS values indicating better pathway consistency.
- Reasoning Tree Extraction: The reasoning-tree prompt converts guideline content into a JSON decision tree with yes/no branches and patient-specific clinical criteria.It separately records the guideline’s primary clinical focus without representing that focus as a decision node.
- Reasoning Tree Extraction: Tree construction requires explicit, objective criteria, one independent variable or threshold per node, sequential representation of multiple criteria, and no external medical knowledge.Ambiguous guideline text should be preserved without speculation.
- Reasoning Tree Extraction: Each decision-tree node receives a unique numeric index and exact supporting guideline sentences drawn from clinically meaningful recommendations or statements.
- Multi-Turn Step-Level Consistency Scoring: The multi-turn templates represent clinician questioning pathways as ordered lists of questions for comparing reference and candidate information-gathering sequences.