PaperlayerView exact revision on arXiv

Source-linked AI summary

Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought

Mengzhu Xu, Jifan Gao, Xia Jiang, Yaoxin Wu, Xi Long

arXiv:2608.24790v1cs.AI

TL;DR

Medical LLM evaluations rarely test whether visible CoT uses the clinical evidence it claims to use. This paper audits that gap with paired chain and question perturbations plus joint chain-update and answer-flip analysis, finding widespread decoupling and clinician-confirmed defensible gold answers.

  • Problem

    Medical evaluations treat reasoning chains as black boxes, while general CoT-faithfulness probes do not incorporate clinical content or cost.

  • Method

    A 30-operator medical perturbation audit edits chains and questions, then classifies joint chain-update and answer-flip outcomes using CDR and related diagnostics.

  • Results

    72.9% CDR across clinically meaningful destructive edits, alongside unchanged accuracy after chain corruption and no CoT-prompting advantage, indicates widespread chain–answer decoupling.

  • Takeaways & Limitations

    Visible CoT functions as a documentation surface rather than a reliable load-bearing mediator under this medical audit.

  • Takeaways & Limitations

    The benchmark suite is multiple-choice, so the framework should next be extended to short-answer clinical vignettes where different failure classes may be more active.

Abstract

from arXiv · show

Clinicians read chain-of-thought (CoT) rationales as evidence of medical reasoning, but whether the visible chain plays that role is rarely tested. General-domain CoT-faithfulness probes ignore clinical cost, and medical LLM evaluations treat the chain as a black box. We close this gap with a medical perturbation audit: a 30-operator battery edits both the chain and the question with clinically motivated operators (severity reversal, negation flip, demographic swap, evidence ablation), paired with a chain-update times answer-flip joint analysis that classifies each model by its failure mode. Applied to 14 LLMs on four medical QA benchmarks, three independent tests converge: the Chain-Decoupling Rate (CDR; chain does not register the edit and the answer does not flip) is 72.9% panel-wide on clinically meaningful destructive edits, chain corruption leaves accuracy unchanged, and removing CoT prompting does not reduce accuracy. Two board-certified clinicians re-annotate N=197 perturbed questions; 98.5% leave the gold defensible. The pattern holds across medical and reasoning fine-tuning and scale; on the closed-source tier, where the chain text is unavailable, the answer-side signals are consistent with the same decoupling. Our framework and CDR provide a reusable yardstick for auditing whether medical CoT is faithful or merely documentation.

1 Introduction

The paper argues that medical CoT should be audited as a potentially causal reasoning record, not treated as an opaque explanation. It introduces a clinically grounded perturbation framework and reports widespread chain–answer decoupling across medical LLMs.

  • The introduction previews that chains often ignore clinical rewrites and remain unchanged when answers also do not flip.
  • Medical evaluations usually score correctness, safety, or bias at the answer level without testing whether the rationale uses its stated evidence.
  • The audit combines clinical perturbations, cost-sensitive flip weighting, demographic faithfulness, and clinician validation to distinguish faithful reasoning from decorative narration.
  • 30 operators edit either chains or medical questions, while joint chain-update and answer-flip outcomes classify model failure modes.
  • 14 LLMs across four medical benchmarks and approximately 364K perturbed generations provide the empirical audit scale.
  • 98.5% of 197 clinician-reviewed edits leave the gold answer defensible, while 17.3%–33.3% of destructive flips are clinically harmful.

2 Related Work

Prior CoT-faithfulness research probes chains in general domains, while medical work evaluates final answers, safety, or demographic effects without explicitly auditing reasoning-chain faithfulness.

  • General-domain studies test chain dependence, hidden decision cues, and agreement between stated reasoning and decision-driving features.
  • Medical benchmarks and safety or bias studies focus on answer outcomes, leaving medical reasoning chains outside the evaluation target.

3 Medical Perturbation Framework

The framework perturbs either the generated chain or the medical question, labels edits by intent and locus, and analyzes chain and answer changes jointly.

  • Each perturbation edits a baseline question, chain, or answer-generation path, producing post-edit chain c′ and answer ŷ′.
  • Destructive edits change clinical evidence a faithful chain should react to, whereas preserving edits leave clinical content intact; structural and surface tags identify the edit locus.
  • The audit uses separate F-block and M-block paths whose outputs feed joint chain-update × answer-flip analysis and multiple metrics.
  • The chain battery contains 17 variants across F1–F7, including truncation, deletion, token substitution, filler insertion, reordering, paraphrase, and operand commutation.
  • The medical question battery contains 13 M1–M6 variants, including demographic edits that are paired with a demographic fairness signal.
  • Two clinicians independently verified intent and locus labels, with Cohen’s κ values of 1.00 and 0.81.

4 Medical Faithfulness Metrics

The paper makes CDR its primary medical faithfulness measure and supplements it with sensitivity, hazard, demographic-disparity, and composite diagnostics.

  • CDR measures the joint no-update/no-flip outcome: the chain misses a destructive clinical edit and the final answer remains unchanged.
  • CDR is preferred to FCS for medical multiple-choice QA because answer flips are not necessarily faithful when clinicians judge the original gold defensible.
  • The answer-flip rate, early-commitment rate, and Faithfulness Composite provide general sensitivity baselines, with FCS chance at 0.5 and its corridor at 0.45–0.55.
  • CHS weights clinically harmful and other acuity-direction flips differently, producing a lower-is-safer relative cross-model ranking.
  • DFG measures accuracy spread across baseline, age, and sex counterfactuals as a disparity screen rather than a group-parity metric.
  • MFC combines FCS, CHS, and DFG for convenience and failure-mode characterization, not as the basis of the main conclusion.

5 Experimental Setup

The study evaluates 14 LLMs across four medical benchmarks using clinically motivated chain and question perturbations, with open, reasoning-tuned, and closed-source tiers. Its protocol combines large-scale generation, greedy decoding, clinician-inaccessible API evaluations, and separate chain/question continuation procedures.

  • Models and datasets: 14 LLMs span general, medical, reasoning-distilled, and closed-source model tiers.The open and reasoning-tuned models are evaluated locally; three closed-source models are accessed through vendor APIs.
  • Models and datasets: Four medical benchmarks—MedQA, MedMCQA, PubMedQA, and medical MMLU—provide the primary evaluation set.Cross-domain checks additionally use GSM8K and StrategyQA with medical models.
  • Procedure and budget: 30 operators generate the main perturbation matrix, with 216K generations before extensions and baselines.The full run contains approximately 364K CoT samples, including reasoning-tuned, cross-domain, closed-source, and baseline evaluations.
  • Procedure and budget: F-block prompts modify an existing chain for continuation, whereas M-block prompts present only a rewritten medical question.Answers are regex-extracted from response tails, with up to 3% unparseable outputs dropped.

6 Results

Across 14 models and four medical benchmarks, three independent tests converge on substantial chain decoupling: visible chains usually neither register clinically meaningful edits nor affect accuracy. Clinician re-annotation supports interpreting these edits as largely gold-preserving, while model tier, fine-tuning, and scale do not reliably improve faithfulness.

  • 6.2 Chain decoupling: three lines of evidence: 72.9% panel-wide CDR indicates that clinically meaningful edits usually produce neither chain updates nor answer flips.The rate is 68–96% for 9/11 open-panel and reasoning-tuned models, with Meditron-7B and DeepSeek-R1-D as outliers.
  • 6.2 Chain decoupling: three lines of evidence: 10.5% of cases mention the new token in the chain, while only 2.9% both register the change and flip the answer.The token-matching rule has recall 0.97 and precision 0.49 in a 150-row spot check, making the reported CDR conservative under its operational definition.
  • 6.2 Chain decoupling: three lines of evidence: Median accuracy change after F-block chain perturbation is approximately 0 pp across all 14 models, with 11/14 within ±1.6 pp.Seven models improve under perturbation, but only two gains have paired-bootstrap 95% confidence intervals excluding zero.
  • 6.2 Chain decoupling: three lines of evidence: CoT prompting does not outperform direct answering: the median CoT−direct gap is −0.4 pp across nine open-panel models.Some high-ecr chain emitters lose substantially under CoT, while several strong emitters remain within ±1 pp.
  • 6.3 Model and tier comparisons: Medical fine-tuning improves accuracy without a corresponding faithfulness gain, while reasoning fine-tuning does not transfer to faithfulness.Med42-8B reaches 0.66 accuracy versus a 0.56 general-model mean, but its fcs and chs remain within sampling noise; DeepSeek-R1 has cdr 0.51 and fcs 0.53.
  • 6.6 Closed-source models: Closed-source answer-side signals are consistent with decoupling, but CDR and CoT−direct accuracy gaps are unavailable because chain text and no-CoT baselines were not captured.PubMedQA accuracy is also excluded from closed-source aggregates because its yes/no/maybe labels are incompatible with the letter extractor.
  • 6.7 Clinician validation: 98.5% of 197 perturbed questions leave the original gold answer defensible, and 13.3% of destructive flips are unanimously judged clinically harmful.The clinician harmful-flip average is 25%, spanning 17.3% for rater A and 33.3% for rater B.

7 Discussion

The audit distinguishes several forms of medical CoT decoupling and shows that longer or more detailed chains do not necessarily make reasoning causally load-bearing. It also separates chain faithfulness from hazard and demographic disparity, while defining “decorative” more narrowly than “unfaithful.”

  • Failure modes: Three failure modes dominate: chain compression, inert chains, and verbose early-commit narration; Meditron-7B shows a separate template-derailment mode.Chain compression reflects chains too short to register edits; inert chains are written but not read; verbose early-commit chains are consistent with post-hoc narration.
  • Failure modes: DeepSeek-R1-D reaches the panel-high cell (b) at 7.5%, suggesting reasoning fine-tuning adds some chain causality without making the chain load-bearing.The result is presented as insufficient to establish a causally upstream, load-bearing chain.
  • Cross-axis interpretation: fcs alone conflates faithful-responsive and externally driven flips, while remaining silent on hazard and disparity.On MedMCQA, Qwen2.5-14B and Gemma-2-9B share fcs = 0.51 but differ 2× on chs; Med42-8B’s accuracy varies by 0.24 across demographic variants.
  • Clinical grounding: The framework extends general-domain perturbation audits to medical MCQA by pairing clinical-content operators with clinician anchoring and a 13.3% unanimous-harmful flip reference.This reference complements, rather than replaces, the relative decoupling and hazard signals.
  • Implications and scope: “Decorative” means the chain does not causally drive the final answer letter, but the audit does not establish that such chains mislead readers.The paper distinguishes this narrower finding from “unfaithful” reasoning and leaves calibration and downstream interpretation out of scope.
  • Implications and scope: Chain-content routing requires question-keyed verification, because reasoning fine-tuning can lengthen chains without making them load-bearing.The paper recommends reporting token-level cdr alongside the clinician harmful-flip rate as complementary audits.

8 Conclusion

Across 14 models and four benchmarks, the audit finds visible CoT largely decoupled from answers, while validation and robustness checks constrain how broadly the result should be interpreted.

  • Conclusion: 72.9% panel-wide CDR indicates that destructive edits usually leave both the chain and answer unchanged.Semantic rescoring yields 74.3%, confirming the decoupling pattern under an alternative registration rule.
  • Conclusion: 98.5% of 197 clinician-reviewed perturbations leave the gold answer defensible, while 17.3%–33.3% of destructive flips are clinically harmful.Thirteen-point-three percent of harmful flips were unanimous.
  • Scope and validation: The result is stable across medical fine-tuning, reasoning fine-tuning, model scale, and cost tier, but closed-source analyses lack chain text and no-CoT baselines.Closed-source models are characterized only through answer-side signals such as fcs and afrD.
  • Limitations: The multiple-choice benchmark suite limits the chs miss class, making short-answer clinical vignettes a natural next evaluation setting.The authors state that the framework itself is benchmark-agnostic.
  • Limitations: Demographic perturbations cover only binary gender and age, and all datasets are English and US/Indian-sourced.Race and ethnicity introduce additional ethical and methodological questions.
  • Measurement caveats: The chain-update rule over-counts updates, making 72.9% a conservative lower bound, while semantic validation confirms the decoupling conclusion.The token rule has recall 0.97 and precision 0.49; semantic rescoring reaches κ = 0.75 and gives 74.3% CDR.

D Option-shuffle robustness check

The option-shuffle audit tests whether GPT-4o-mini depends on answer position rather than option content, while prompt and perturbation checks assess robustness of the broader audit design.

  • Option-shuffle audit: GPT-4o-mini accuracy drops by 27 to 45 percentage points after answer-choice permutation, with same-content selection only modestly above the 25% random baseline.The result suggests substantial option-position sensitivity on the tested multiple-choice benchmarks.
  • Interpretation: The option-shuffle result supports the claim that visible CoT is not load-bearing when surface-level reordering changes answers without changing question content.The open-panel 72.9% CDR audit used the original option order and is not driven by this confound.
  • Scope: The diagnostic covers one closed-source model on three MCQA datasets, so held-out clinical vignettes remain necessary for deployment validation.PubMedQA is omitted because it uses yes/no/maybe labels rather than four answer choices.
  • Prompt robustness: Rationale-conditioning prompts were tested on GPT-4o-mini across four benchmarks and Qwen2.5-14B on MedQA and PubMedQA as an open-weight cross-check.These checks vary the prompting context rather than the underlying benchmark questions.
  • Perturbation design: The audit’s 30 operators include chain truncation, deletion, substitution, insertion, reordering, paraphrase, demographic, distractor, negation, and severity edits.The operators are seed-driven and grouped into chain-level F1–F7 and question-level M1–M6 families.

G Robustness analyses

Robustness analyses show that the reported rankings and faithfulness dimensions are stable across weighting choices, operator blocks, and domains, while chain length complicates hazard interpretation.

  • Metric sensitivity: τ = 1.00 across six alternative hazard-weight schemes, preserving every model ranking.The alternative schemes span ±50% of the default weights.
  • Metric sensitivity: The default ordering remains highly stable under mfc reweighting, with τ = 0.91 in the least stable scheme.Only the equal-weight scheme drops below 1.00.
  • Operator robustness: r = +0.17 between F-block and M-block fcs, indicating that the blocks capture distinct faithfulness dimensions.The correlation is below the stated |r| < 0.3 orthogonality threshold.
  • Operator robustness: Most operators fire on ≥99% of samples, but M4, M5, and especially M6 fire infrequently; M2 interpretation is restricted to MedQA.M6 fires on only 1% of samples, while M2 fires on 86–87% of MedQA samples and 3–30% elsewhere.
  • Metric robustness: r = +0.80 between chain length and chs, while correlations with fcs and ecr remain small.The strong coupling implies that chs can understate hazard for early-committing models.
  • Leave-one-out checks: Removing F4 shifts the panel-wide median ∆Acc by about 0.1 pp without changing model sign flips or relative rankings.F4 is the preserving operator; the corresponding M6 contribution is bounded by its 1.25% average firing rate.
  • Cross-domain robustness: Medical models retain different fcs patterns on GSM8K and StrategyQA, while prompt swaps move fcs by at most 0.025.Meditron-7B loses six points on StrategyQA because of accuracy, whereas Med42-8B largely retains its medical-tier fcs.
  • Cross-domain robustness: Chain-compressed models show ecr = 0 alongside high fcs, separating faithfulness failures by chain format rather than domain.BioMistral-7B reaches fcs = 0.84 on GSM8K despite ecr = 0, and OpenBioLLM-8B shows the same pattern.

I Reasoning-tuned models (full details)

Reasoning-distilled models produce long explicit chains but do not show stronger medical faithfulness. Medical CoT training is associated with lower demographic disparity for HuatuoGPT-o1-8B, whereas general reasoning distillation does not transfer medical safety calibration.

  • Headline results: 0.55 accuracy and fcs = 0.53 characterize HuatuoGPT-o1-8B across the four medical datasets.Its other metrics are ecr = 0.93, chs = 0.10, dfg = 0.09, and mfc = 0.71.
  • Headline results: 0.60 accuracy and fcs = 0.53 characterize DeepSeek-R1-Distill-Llama-8B across the four medical datasets.Its other metrics are ecr = 0.93, chs = 0.17, dfg = 0.26, and mfc = 0.66.
  • Faithfulness: Both models place every cell-level fcs inside the iso-fcs chance corridor.Their longer explicit chains therefore do not correspond to higher measured faithfulness.
  • Medical versus general reasoning: HuatuoGPT-o1-8B matches Med42-8B at fcs = 0.53 but has dfg = 0.09 versus 0.16 and 11 fewer accuracy points.The comparison links medical CoT training with demographic fairness rather than improved faithfulness.
  • Medical versus general reasoning: DeepSeek-R1-Distill-Llama-8B matches medical fcs with chs = 0.17 and dfg = 0.26, so general reasoning distillation does not transfer medical safety calibration.Both models reach ecr = 0.93, consistent with longer traces functioning as post-hoc narration rather than load-bearing rationales.

J fcs against parameter count

Across evaluated scales, medical-specialised models do not separate from size-matched general baselines on fcs, and the broader flip-rate comparison shows no medical-specialised low-flip corner.

  • J fcs against parameter count: Every medical-specialised cell falls inside the size-matched general-baseline fcs band across all four medical benchmarks.The band does not lift off the chance line at any evaluated scale.
  • K Per-family flip rates: Table 11 pools per-family flip rates across four medical datasets and within-family variants, separating F-block from M-block.Lower flip rates are better, and non-firing families are marked with “–”.
  • K Per-family flip rates: M1 fact ablation is the panel-wide hotspot, with every model flipping on at least one in three samples.Meditron-7B is uniformly brittle across both blocks, while Med42-8B remains among the cooler open-source baselines.
  • K Per-family flip rates: Qwen2.5-14B and Gemma-2-9B form a low-flip corner that no medical-specialised checkpoint reaches.This is the empirical counterpart of the reported medical/general fcs parity.
  • Cross-tier interpretation: Cross-tier numerical comparisons assess family-level destructive-perturbation robustness rather than identical-operator robustness.The local and API runtimes share conceptual families but use non-byte-identical M-block definitions; clinician validation preserves this split.
  • Clinician validation: N=197 M-block perturbations were clinician-validated, with 13 of 14 panel models represented among additional destructive-flip rows.Two board-certified clinicians annotated the sample while blinded to model identity.
  • Clinician validation: No row was unanimously marked gold-shifted by both raters, making the binary shifted-versus-not conclusion invariant to the strict/lenient split.Among 61 disagreements, 55 were lenient no_shift versus strict ambiguous.
Loading 2608.24790v1…