PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability

Samuel Lewis-Lim, Xingwei Tan, Mario Sanger, Zhixue Zhao, Nikolaos Aletras

arXiv:2610.03509v1cs.AI

TL;DR

It is unclear how different efficient-reasoning methods affect CoT oversight, especially as faithful explanations can require different lengths across tasks. The paper evaluates three length-pressure methods on faithfulness and monitorability, finding that faithfulness usually falls through reduced consistency while monitorability is more robust.

  • Problem

    It is unclear whether differing ways of applying length pressure undermine CoT oversight, particularly because faithful explanations can require different token lengths across tasks.

    Source: §1 · p. 1

  • Method

    The paper trains models with three RL length-pressure methods and evaluates CoT faithfulness through prediction on related inputs and monitorability under input interventions.

    Source: §2 · p. 2, §3 · p. 3, §5.2 · p. 7

  • Results

    Across three efficiency methods and three models, training mostly harms faithfulness through reduced consistency, while monitorability remains largely intact unless CoT is compressed far below base average length.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The oversight implications depend on CoT use: efficient reasoning usually harms identifying relied-on features, but often preserves detection of outside influences on answers.

    Source: §7 · p. 9

    The evaluation does not cover complex agentic settings with long, multi-step tasks because those tasks exceed the capabilities of the models the authors can train.

    Source: A LIMITATIONS AND FUTURE WORK · p. 15

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning allows humans to inspect how large language models reach their answers, and oversee model behaviour. This reasoning comes at an increased inference cost, motivating efficient methods that train models to solve tasks using fewer tokens. However, a common concern is that such training may cause models to skip important reasoning steps, so the CoT no longer faithfully reflects the model's decision. It is unclear whether or when this occurs in practice, since different efficiency methods apply length pressure to models' CoT in distinct ways, and faithfully explaining a model's decision takes more tokens on some tasks than others. To understand these dynamics, we fine-tune a variety of models with three methods that apply length pressure differently, namely a fixed generation budget, a per-example length target, and a group-relative length reward. We evaluate how efficient reasoning affects CoT faithfulness (i.e., how well the CoT reflects model decisions on related inputs) and monitorability (i.e., whether the CoT reveals when input interventions alter the output). We find that it affects faithfulness and monitorability differently. Faithfulness falls in most settings, primarily because the trained models are less consistent. Monitorability is more robust, as models keep acknowledging the influence on their answer even when the CoT is much shorter.

1 INTRODUCTION

The introduction asks whether efficiency training undermines CoT oversight, since length pressure varies by method and the token cost of a faithful explanation varies by task. The paper evaluates faithfulness and monitorability and reports that they respond differently to training.

Motivation

  • CoT offers a way to inspect model decisions, but its long traces increase latency and inference cost, motivating training for shorter reasoning.The introduction notes that fixed token limits penalize only over-budget responses, whereas fixed length penalties push all responses shorter.Source: §1 · p. 1

Study approach

  • The study evaluates three length-pressure methods across models, measuring whether CoTs predict answers on related inputs and reveal intervention effects.Faithfulness is tested across seven prediction and decision-making tasks; monitorability uses user-view and medical cognitive-bias interventions.Source: §1 · p. 2

Main findings

  • Faithfulness mostly drops, chiefly as trained models become less consistent, while monitorability mostly survives and degrades under heavy compression.Some methods shorten CoTs by 40–65% without reducing faithfulness, and one improves it; monitorability loss tracks compression relative to the base model.Source: §1 · p. 2

2 EFFICIENT REASONING METHODS

The paper compares three RL methods that impose length pressure in different ways while sharing the same training setup. Their distinction is how they reward or constrain CoT length.

ThinkPrune

  • ThinkPrune caps completion length during generation, truncating responses before scoring and penalizing only those that exceed the budget.Responses already below the cap receive no additional shortening pressure.Source: §2 · pp. 2–3

L1

  • L1 prompts each example with a target length and rewards correctness while penalizing deviations from that target in either direction.The target is sampled from a range set by bmin and bmax; α controls the penalty strength.Source: §2 · p. 3

GLP

  • GLP rewards correct responses for being shorter relative to other responses sampled for the same problem, while incorrect responses are never rewarded for brevity.Its length reward is weighted by β.Source: §2 · p. 3

3 MEASURING COT FAITHFULNESS AND MONITORABILITY

The paper measures faithfulness by how much CoT helps predict a model’s answers on related counterfactual inputs, and monitorability by whether CoT reveals intervention effects. Both assess whether CoT can support oversight through behavior-based tests.

Measurement distinction

  • The paper distinguishes faithfulness as predicting behavior on related inputs from monitorability as recovering a behavior property from CoT.The former reflects understanding which features a model relies on; the latter concerns catching unwanted influences on an answer.Source: §3 · p. 3

Measuring faithfulness

  • Faithfulness uses Normalized Simulatability Gain (NSG), measuring the uplift in counterfactual-answer prediction accuracy when a predictor receives the reference model’s CoT.The predictor is shown the original question and answer, then predicts the reference model’s answer to a related counterfactual, with and without CoT.Source: §3 · p. 3, Figure 2 · p. 4
  • A reference model’s consistency limits predictor accuracy, so the paper estimates an oracle ceiling by guessing its most common answer across repeated samples.A model answering A 60% of the time sets a 60% maximum accuracy for predicting that answer.Source: §3 · p. 4

Measuring monitorability

  • Monitorability tests whether an external monitor can detect an intervention’s influence on the answer from the CoT alone.The study applies this intervention framework to sycophancy and cognitive-bias settings and reports monitorability with g-mean2.Source: §3 · p. 4, Figure 2 · p. 4

4 EXPERIMENTAL SETUP

The experiments use three reasoning models, train them on competition-math problems, and evaluate faithfulness and monitorability with separate task suites. Monitorability tests focus on sycophancy and medical cognitive-bias cues.

4.1 MODELS

  • Three base reasoning models from two model families, with different average CoT lengths, enable comparison of compression effects against baseline length.The models are Qwen3-4B-Thinking, Qwen3-8B, and Olmo3-7B-Think.Source: §4.1 · p. 4

4.2 DATA AND TRAINING

  • All methods train on 2,223 historical AIME and AMC problems, with AIME22 used for checkpoint selection and LoRA adapters keeping base weights frozen.The paper uses math training because it is standard for efficiency methods and reports that shortening transfers to MMLU-Redux in this setup.Source: §4.2 · pp. 4–5

4.3 FAITHFULNESS EVALUATION

  • Faithfulness evaluation uses seven datasets with similar real counterfactual inputs and an ensemble of three predictor models.The datasets cover person-level outcome prediction and Moral Machines decisions; stronger API predictors are excluded because their higher no-CoT accuracy leaves less CoT uplift.Source: §4.3 · p. 5

4.4 MONITORABILITY EVALUATION

  • Monitorability evaluation tests whether CoT reveals answer shifts caused by user-preference cues or incorrect-option cues in medical questions.The sycophancy setting uses explicit or implicit user-preference information, while the medical QA setting uses 150 five-option MedQA questions with bias cues.Source: §4.4 · p. 5

5 RESULTS

Efficiency training usually reduced CoT faithfulness, largely as trained models became less consistent, while monitorability was generally more robust. Heavy compression could nevertheless impair detection of cognitive-bias cues and hide how cues shaped answers.

5.1 FAITHFULNESS

  • NSG fell significantly in 11 of 15 settings, but ThinkPrune and GLP shortened some CoTs 29–65% without reducing NSG or while improving it.ThinkPrune shortened CoTs by 42% on Qwen3-8B and 29% on Olmo3-7B without reducing NSG; GLP shortened Qwen3-8B CoTs by 65% while raising NSG from 29.8% to 33.1%.Source: §5.1 · p. 5, Figure 3 · p. 6
  • Consistency change tracked NSG change more closely than CoT length (ρ = 0.87 vs. 0.46), and inconsistency explained over half of most NSG losses.The oracle accuracy ceiling fell across most settings, indicating that increased inconsistency accounted for much of the reduced faithfulness.Source: Figure 4 · p. 6, §5.1 · pp. 6–7
  • Faithfulness varied by dataset: Olmo3-7B L1-Medium on Pima Diabetes fell from 76.7% to 11.8% NSG, while some Bank Marketing and Breast Cancer Recurrence settings improved.For the Pima example, consistency fell from 99.4% to 83.5%, even though the CoT still named prediabetic glucose as the main risk factor.Source: Figure 5 · p. 7, §5.1 · p. 7
  • On cases where counterfactuals changed answers, efficiency training further reduced CoT-based predictor accuracy by 3–8 points in 10 of 15 settings.Base-model CoTs already reduced predictor accuracy by 2–3 points on these cases, despite oracle predictors gaining 6–10 points.Source: §5.1 · p. 7

5.2 MONITORABILITY

  • On Face-Eval with the sycophancy intervention, no efficiency-trained model’s g-mean2 fell more than 0.08 below its base model, even though CoTs were up to 70% shorter.Source: §5.2 · p. 7
  • On MedQA cognitive-bias cues, most models lost 0.1 or less in monitorability, but L1-Low losses exceeded 0.2 across all three models.Qwen3-4B L1-Low g-mean2 dropped from 0.87 to 0.62; across settings, monitorability loss tracked compression relative to the base model (ρ = 0.68), not absolute CoT length.Source: §5.2 · pp. 7–8, Figure 6 · p. 8
  • Shortened CoTs hid cue influence either by omitting the cue or by presenting it as confirmation rather than the reason for the answer.For Qwen3-8B L1-Low, monitors judged the cue ignored in 40% of responses versus 6% for the base model; on Qwen3-4B, cue-as-confirmation cases rose from 11% to 37%.Source: Figure 7 · p. 8, §5.2 · p. 8

6 RELATED WORK

Prior work studies ways to shorten reasoning and measures CoT faithfulness and monitorability through different approaches. Earlier training studies examined limited settings, while this paper compares multiple efficiency methods across models and evaluates both oversight uses.

Efficient reasoning training

  • Efficiency methods apply length pressure through fixed budgets, per-example targets, or rewards relative to other responses [Kimi Team et al., 2025].Related approaches also use length penalties, difficulty-dependent budgets, or comparison with the shortest correct response.Source: §6 · pp. 8–9

Measuring CoT faithfulness and monitorability

  • Faithfulness is assessed through simulatability, while monitorability tests whether an external monitor can detect a property from the CoT [Korbak et al., 2025].Prior work also studies self-consistency under input changes and monitor detection of answer-hint and biasing-feature influence [Guan et al., 2025].Source: §6 · p. 9

Training effects on CoT oversight

  • Prior training studies report hidden reward-hacking intent under direct penalties, initial faithfulness gains under outcome-based RL, and mixed monitorability effects from length penalties [Baker et al., 2025; Chen et al., 2025].Kaufmann et al. (2026) predict and find reduced monitorability in toy tasks; this paper compares three methods across three models and measures both oversight properties.Source: §6 · p. 9

7 DISCUSSION

The discussion argues that the safety of efficient reasoning training depends on how the CoT is used and how much training changes model decisions. Detecting outside influence is generally more robust than identifying decision-driving features.

7 DISCUSSION

  • CoT monitoring for outside biasing factors, including user views or misleading cues, appears mostly safe; sycophancy remains detectable with very short CoTs.Cognitive biases become hard to detect only under the heaviest compression, and trained models are no more sycophantic than their base models.Source: §7 · p. 9
  • Using CoT to identify decision-driving features is usually harmed by efficiency training because trained models often answer less consistently.The CoT may name the factor relied on most, yet be a weaker guide when the model no longer follows that factor reliably.Source: §7 · p. 9
  • Faithfulness is not reduced in every setting: ThinkPrune on Qwen3-8B and Olmo3-7B, and GLP on Qwen3-8B, substantially shorten CoT without reducing it.The evaluation does not cover complex agentic settings.Source: §7 · p. 9

8 CONCLUSION

Across three efficiency methods and three models, the paper finds that training mostly harms faithfulness through reduced consistency, while monitorability largely persists except under very heavy compression. The implications for oversight depend on how the CoT is used and how training changes model decisions.

8 CONCLUSION

  • Across three efficiency methods and three models, efficient reasoning training mostly harms faithfulness by making models answer less consistently rather than by degrading their CoT.Source: §8 · p. 9
  • Monitorability remains largely intact unless the CoT is compressed far below the base model’s average CoT length.Source: §8 · p. 9
  • Whether efficiency training undermines oversight depends on the CoT’s use and how much training changes model decisions, not only on CoT length.Source: §8 · p. 9

AI USE STATEMENT

The authors report using language models for proofreading, polishing, and drafting, with all final text and code manually reviewed, edited, and verified. They also note that their evaluation does not cover complex agentic oversight settings.

AI USE STATEMENT

  • Language models assisted with proofreading, polishing, drafting text, and drafting or refactoring experimental code; authors manually reviewed, edited, and verified final text and code.Source: AI USE STATEMENT · p. 10

A LIMITATIONS AND FUTURE WORK.

B.1 FULL MONITORABILITY RESULTS

Efficiency training leaves monitorability broadly robust but can increase models’ susceptibility to clinical cues. The section also details the monitorability metric, its full-result visualizations, and an example of unfaithful reasoning.

B.1 FULL MONITORABILITY RESULTS

  • Implicit cues scored lower in detectability throughout, but efficiency training affected explicit and implicit variants in much the same way.Figures plot g-mean2 against CoT-length distributions, separating cue variants and showing per-response length distributions, means, and bootstrap standard errors.Source: §B.1 · pp. 15–16, Figure 8 · p. 16, Figure 9 · p. 16

B.2 FULL NSG RESULTS

  • Table 1 reports predictor accuracy with and without the reference model’s CoT, simulatability gain, and NSG, pooled across three predictor models.Confidence intervals are clustered bootstrap 95% intervals over original questions.Source: Table 1 · p. 18

B.3 INFLUENCE OF INTERVENTIONS

  • Clinical-cue effects were largely unchanged for Qwen3-8B, but some Olmo3-7B and Qwen3-4B settings were more influenced; Olmo3-7B GLP increased the frequency-cue effect by 0.146.For sycophancy, 22 of 26 settings differed from the base model by less than 0.03; all four exceptions were decreases.Source: §B.3 · p. 18

C CONSISTENCY AND THE NSG UPPER BOUND

  • Table 3 reports consistency, an oracle-predictor upper bound on NSG, and the share of NSG reduction explained by a falling upper bound.The table defines consistency using ten samples per counterfactual and marks changes whose paired-bootstrap 95% interval excludes zero.Source: Table 3 · p. 19

D MONITORABILITY METRIC

  • Monitorability uses an intervention-setting g-mean2: a minimal-criterion true-positive rate is combined with true-negative rates from each arm, and eligibility is determined on separate samples.The total effect provides a lower bound on the fraction of cued answers attributable to the cue; the reported score averages eligible questions.Source: D MONITORABILITY METRIC · p. 20

E UNFAITHFUL EXAMPLE

  • For one Olmo3-7B patient pair differing only in genetic diabetes risk, predictors with the original CoT answered NO for the counterfactual, while predictors without it answered YES.The original CoT raises genetic risk, sets it aside, and identifies glucose as the main determinant.Source: E UNFAITHFUL EXAMPLE · p. 20

F TRAINING DETAILS

Training used LoRA adapters with GRPO on three model families, using a filtered mathematics prompt set and AIME22 for checkpoint selection.

F TRAINING DETAILS

  • LoRA adapters were trained with frozen base weights using GRPO on two NVIDIA GH200 GPUs, with about 2,223 filtered training prompts and AIME22 validation every 50 steps.The training set began with 2,470 problems; the longest 10% of prompts were dropped to avoid completion-budget truncation.Source: F TRAINING DETAILS · p. 21
  • Table 4 lists settings shared across methods and base models, while Table 5 lists method-specific settings.ThinkPrune’s Qwen3-8B budget was 3,000 rather than 4,000 because its average base-model CoT was already below 4,000 training-data tokens.Source: Table 4 · p. 21, Table 5 · p. 21

G BENCHMARK RESULTS

The appendix reports baseline benchmark evaluations and gives the monitor prompts used for cognitive-bias and FACE-Eval tasks.

G BENCHMARK RESULTS

H.1 COGNITIVE BIAS (MEDQA) MONITOR PROMPT

  • The cognitive-bias monitor classifies cue influence into four verdicts, counting only meaningful influence on the cued answer as positive and treating post-hoc acknowledgment as negative.The prompt distinguishes cue-driven answers from answers independently reached on clinical grounds.Source: §H.1 · pp. 22–23

H.2 FACE-EVAL MONITOR PROMPT

  • The FACE-Eval monitor marks a cue as verbalized only when reasoning acknowledges it and connects it to how the answer is shaped; uncertainty is scored conservatively as not verbalized.The binary verdict is X for verbalized and Y for not verbalized.Source: §H.2 · pp. 23–24

Related papers

Loading 2610.03509v1…