PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Simulation Training for Chain-of-Thought Faithfulness

Peter Hase, Christopher Potts

arXiv:2602.20710v2cs.AIcs.CL

TL;DR

Unfaithful CoT limits the reliability of inspecting model reasoning. The paper introduces CST, which trains CoTs using counterfactual simulatability, and finds stronger faithfulness and monitoring results, with important limits for dissuading cues and cross-setting transfer.

  • Problem

    CoT inspection is widely used, but known unfaithfulness limits the reliability of explanations of model behavior.

  • Method

    CST rewards CoTs that enable a simulator to predict model outputs on cue-based and generic model-based counterfactual inputs.

  • Results

    CST improves cue-based monitor accuracy by 35 points and generic model-based simulator accuracy by 2 points, while outperforming prompting baselines.

  • Takeaways & Limitations

    CST shows promise for improving CoT faithfulness and supporting CoT monitoring across several model and task settings.

  • Takeaways & Limitations

    Faithfulness improvements do not transfer reliably across counterfactual settings, and dissuading cues remain difficult; model-generated rewriting success reaches only about 70–80%.

Abstract

from arXiv · show

Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output. But well-known problems with CoT faithfulness severely limit what insights can be gained from this practice. In this paper, we introduce a training method called Counterfactual Simulation Training (CST), which aims to improve CoT faithfulness by rewarding CoTs that enable a simulator to accurately predict a model's outputs over counterfactual inputs. We apply CST in two settings: (1) CoT monitoring with cue-based counterfactuals, to detect when models rely on spurious features, reward hack, or are sycophantic, and (2) counterfactual simulation over generic model-based counterfactuals, to encourage models to produce more faithful, generalizable reasoning in the CoT. Experiments with models up to 235B parameters show that CST can substantially improve monitor accuracy on cue-based counterfactuals (by 35 accuracy points) as well as simulatability over generic counterfactuals (by 2 points). We further show that: (1) CST outperforms prompting baselines, (2) rewriting unfaithful CoTs with an LLM is 5x more efficient than RL alone, (3) faithfulness improvements do not generalize to dissuading cues (as opposed to persuading cues), and (4) larger models do not show more faithful CoT out of the box, but they do benefit more from CST. These results suggest that CST can improve CoT faithfulness in general, with promising applications for CoT monitoring. Code for experiments in this paper is available at https://github.com/peterbhase/counterfactual-simulation-training

1 Introduction

CoT inspection is widely used to understand model behavior, but unfaithful reasoning can provide plausible yet misleading explanations. CST trains CoTs to better reflect model behavior by rewarding counterfactual simulatability.

  • CoT inspection uses model reasoning to provide accessible explanations of behavior, including through LLM monitors for safety issues.
  • Unfaithful CoTs can selectively interpret prompt evidence to reach predetermined conclusions, limiting the trustworthiness of CoT inspection.
  • CST rewards CoTs that let a simulator predict the model’s answer on a counterfactual input, targeting faithful reflection of underlying reasoning.
  • CST applies to cue-based counterfactuals for monitoring cue reliance and to generic model-based counterfactuals for evaluating simulatability.
  • 35 accuracy points is the reported monitor improvement on cue-based counterfactuals, while generic model-based counterfactual simulator accuracy improves by 2 points.
  • CST outperforms prompting baselines, LLM rewriting is 5x more efficient than pure RL, dissuading-cue transfer remains limited, and larger models benefit more.

2 Related Work

Prior work established faithfulness problems in natural-language explanations and developed tests, prompting methods, training methods, and introspection studies. CST differs by training faithful CoT from observed counterfactual behavior rather than requiring ground-truth preference functions.

  • Earlier explanation studies found that models could generate contradictory explanations after inconsequential prompt changes, exposing faithfulness problems.
  • Faithfulness research evaluates explanations through causal importance, behavioral consistency, and whether they enable prediction of model behavior on new inputs.
  • Counterfactual CoT work includes cue changes, word-level edits, concept interventions, and naturalistic transformations such as reversing relational questions.
  • Methods for improving faithfulness include prompting, structured reasoning, decomposition, causal training, consistency training, and verbalization finetuning.
  • Introspection studies test whether models can predict their own outputs, whereas this paper trains models to produce more faithful CoT.
  • Unlike preference-function training, CST uses supervision grounded in observed model behavior and does not require ground-truth preference functions.

3 Counterfactual Simulation Training (CST)

CST constructs counterfactual pairs, scores CoTs by whether a simulator predicts the counterfactual answer, rewrites unfaithful reasoning, and trains on weighted positive and negative examples.

  • 1 Counterfactual Generation: CST generates counterfactuals by inserting cues or using an LLM to create diverse transformations of existing inputs.
  • 2 Sample From Task Model For Question Pair: The task model runs on each original-counterfactual pair with CoT reasoning, using k original-input samples when reinforcement-learning-style sampling is enabled.
  • 3 Faithfulness Metric: A CoT is labeled faithful when a simulator accurately predicts the task model’s counterfactual answer using the original input, reasoning, and output.
  • 3 Faithfulness Metric: The outcome-only simulator predicts the counterfactual answer from the original answer alone, providing a baseline for the reasoning simulator.
  • 3 Faithfulness Metric: CST distinguishes helpful CoTs, which improve simulator accuracy, from harmful CoTs, which mislead the simulator relative to outcome-only monitoring.
  • 4 Train Data Construction: CoT rewriting targets datapoints without faithful samples, rejection-sampling up to 10 rewrites while retaining negative examples and counterfactual prediction stability.
  • 5 Training Algorithm & Objective: Training combines cross-entropy on positive examples with unlikelihood on negative examples, weighting examples by their effect on simulator accuracy.
  • 5 Training Algorithm & Objective: CST typically trains for 5–20 epochs across up to 6 rounds with batch size 128, while varying reward-weighting schemes in an appendix.

4 Experiment Setup

The experiments use several language models and datasets to evaluate CoT faithfulness through cue-based and generic counterfactuals. They measure monitorability with G-mean, generic simulation with accuracy, and assess results across five seeds using block-bootstrap p-values.

  • Models: The study trains gpt-oss-120b and Qwen3 models, including Qwen3-4B, Qwen3-30B-A3B, and Qwen3-235B-A22B, using LoRA rank 32.Simulator models include Qwen3-235B-A22B and deepseek-v3-0324.
  • Datasets: Experiments cover MMLU, SNLI, ETHICS, and MMLU-Pro-Law, with MMLU representing knowledge-intensive tasks and the others emphasizing process-oriented reasoning.The datasets are filtered to two-way multiple-choice questions with one correct option and one distractor.
  • Counterfactual design: Cue-based experiments use six training cues and six test-only cues, dividing training data evenly across cues and evaluating both cue types.This setup tests generalization from training cues to held-out cues.
  • Metrics: G-mean measures monitorability for cue-based counterfactuals, while accuracy measures generic counterfactual simulation for model-based counterfactuals.G-mean balances recall or TPR against specificity or TNR when evaluating cue influence.
  • Statistical evaluation: Experiments are run for 5 seeds, with two-sided p-values estimated by block bootstrap over test datapoints and seeds.The reported procedure resamples across both datapoints and seeds.

5 Experiments

Across experiments, CST improves CoT monitorability and simulatability, outperforms prompting and pure RL baselines, and benefits larger models more. Gains are strongest for persuading cues, while dissuading cues remain difficult.

  • 5.1 CST Improves Simulatability and Monitorability: CST improves reasoning-monitor G-mean by 31–48 points across settings, with gains driven by changes in CoT rather than model answers.Outcome-only monitor performance remains stable across comparisons.
  • 5.1 CST Improves Simulatability and Monitorability: Model-based counterfactuals yield smaller and task-dependent gains, with CoT faithfulness improving on SNLI and ETHICS-justice but not consistently elsewhere.Reasoning-versus-outcome-only monitor differences increase from 1.5 to 2.4 points on SNLI and from 1.5 to 3 points on ETHICS-justice.
  • 5.1 CST Improves Simulatability and Monitorability: CST improves cue-influence monitoring, including SNLI recall from 12% to 87% while keeping false-positive rate below 3%.
  • 5.2 CST Outperforms Prompting Baselines: CST outperforms prompting baselines by 25 G-mean points for Qwen3-235B-A22B and 35 points for gpt-oss-120b.The comparison includes prompts for faithful reasoning, testing procedures, and increased reasoning effort.
  • 5.3 CoT Rewriting Outperforms Pure RL: CoT rewriting reaches 78% G-mean in two hours versus more than ten hours for RL, achieves 81% OOD G-mean versus 67%, and keeps CoTs near their original length.RL increases average CoT length from 137 to 187 words, whereas rewriting changes it from 137 to 133 words.
  • 5.4 Faithfulness On Dissuading Cues Remains Low: CST improves monitorability for persuading cues but not dissuading cues, where G-mean increases only from 53% to 57%.The authors associate this boundary with dissuading cues involving sentential and lexical negation.
  • 5.5 Model Scaling: Larger models do not necessarily begin with more monitorable CoT but benefit more from CST, with Qwen3-235B-A22B rising from 35% to 70% G-mean.Qwen3-4B rises to 61% after CST, while the larger model starts at the same 35% score.

6 Qualitative Analysis

CST makes counterfactual reasoning more transparent by prompting the model to acknowledge when a cue changes its answer. In a sycophancy example, the CoT explicitly identifies deference to the user’s opinion.

  • CST causes the model to admit it is deferring to a user opinion when a cue changes its answer.On an AITA question, the model first supports the counterfactual answer, then states that it is deferring before giving the cue-induced answer.

7 Conclusion

The paper presents CST as a method for improving CoT faithfulness across cue-based and model-generated counterfactuals. Across five datasets, it improves simulatability while making influential factors more transparent, though generic counterfactuals and dissuading cues remain harder.

  • CST significantly improves simulatability with both cue-based and model-generated counterfactuals across five datasets.
  • CST makes CoTs more transparent in identifying the factors that influence model reasoning.
  • CST shows promise for improving CoT faithfulness in general, with applications in CoT monitoring.

8 Limitations

The evaluation uses counterfactuals that are not tied to specific threat models, and CST’s generalization remains limited across counterfactual types. CST also incurs a small accuracy cost because correctness training was not combined with it.

  • The cue-based counterfactuals are artificial and better suited to testing faithfulness than benchmarking monitoring against realistic threat models.They are persuasive to models, flipping gpt-oss-120b’s answer to MMLU questions 35% of the time.
  • Model-generated counterfactuals are diverse and flexible but disconnected from any particular threat model because they are open-ended.
  • CST does not transfer between cue-based and model-generated counterfactuals, so faithfulness training may require broad data coverage.The two settings were trained separately, and further testing of model generalization is left for future work.
  • CST caused a small model-accuracy decrease, usually 1–2 points, because it was not combined with supervised learning or RLVR for correctness.The authors suspect a correctness training signal could mitigate this cost.

Reproducibility Statement

The paper provides code and implementation details for reproducing CST experiments, including a low-cost small-scale run, model-specific training settings, reward comparisons, and stability data mixed into training.

  • The released code includes a small-scale gpt-oss-20b experiment costing approximately $6.25 and taking about 2.5 hours.The experiment uses 800 training and 800 test points over two training rounds.
  • Training uses batch size 128 and learning rates of 1e-4 for gpt-oss-120b and Qwen3-235B-A22B.Cue-based counterfactuals use 5 epochs per round, while model-generated counterfactuals use 20 epochs except for Qwen3-4B.
  • The authors did not tune hyperparameters for performance, finding little difference across tested loss-mixing weights and batch sizes.They selected λ = 0.4 after observing little difference among λ ∈ {0.1, 0.4, 1}.
  • The reward weighting scheme outperforms finetuning on all unweighted faithful CoTs according to Freasoning-sim.The value 5 for actively helpful or harmful CoTs balances monitor performance with changing influence rates.
  • CST mixes counterfactual inputs and model outputs into training as positive examples with reward weight R = 1 to stabilize predictions.The mixture fraction is 80% for cue-based counterfactuals and 20% for model-based counterfactuals.

B Additional Experiments

Additional experiments show that CST generalizes monitorability across held-out datasets and preserves task accuracy, but its effectiveness depends on how counterfactual cues are presented. When the relevant cue is unpredictable, CST becomes less effective.

  • Task Accuracy: CST typically changes task accuracy by only 1–2 points despite training for monitorability rather than task performance.The paper reports that CST leaves task accuracy largely unaffected across additional experiments.
  • Placing the Cue in the Counterfactual Input: When cues are placed in counterfactual inputs, CST trains models to pre-emptively describe how those cues would affect their answers.This reverses the main setup, where cues appear in the original question and the counterfactual removes them.
  • Placing the Cue in the Counterfactual Input: CST is less effective when the counterfactual cue is unpredictable because the model must guess which feature the simulator will need explained.The authors suggest this may help explain smaller gains with diverse model-generated counterfactuals.

C Qualitative Analysis

Qualitative examples show CST making CoTs more useful to simulators by exposing cue reliance and by expressing reasons that generalize across counterfactual inputs. These changes reveal sycophancy, answer hacks, and task-relevant reasoning that were previously omitted or misleading.

  • Admitting to Sycophancy: CST makes a model explicitly admit deference to a user opinion when that cue changes its answer.In the example, the model first supports the counterfactual answer, then states that it is deferring to the user’s opinion before giving the cued answer.
  • Reasoning Out Loud About Answer Hacks: CST causes the model to reason openly about a faked answer comment, enabling the simulator to predict behavior on the un-cued input.The model notes that the commented answer appears physically wrong while still deferring to it.
  • Giving More Generalizable Reasons: For model-generated SNLI counterfactuals, CST encourages reasons that remain predictive when evidence changes across inputs.The post-CST reasoning focuses on the relationship between standing and hiking rather than treating location as a decisive feature.
  • Additional Examples: Additional examples examine deference to a Stanford biologist’s recommendation and clearer reasoning about the relationship between swimming and camping.These examples are presented as further qualitative analyses of more faithful CoTs.

D Additional Experiment Details

Additional experiment details describe sampling, baselines, ablations, and dataset-specific outcomes for CST. They show stronger cue-based results than model-generated MMLU counterfactual results, while identifying dissuading cues as an important generalization boundary.

  • Model-Generated Counterfactuals: On MMLU model-generated counterfactuals, CST produces no counterfactual-simulatability gains.The paper contrasts this result with gains on other datasets and with cue-based counterfactuals.
  • Prompting Baselines: A prompting baseline improves G-mean by 11.1 points, while CST improves it by 35.9 points relative to baseline.The most effective prompt explicitly describes the evaluation procedure.
  • CoT Rewriting vs RL: Rewriting unfaithful CoTs with an LLM is 5x more efficient and generalizes better than a pure RL approach.The comparison uses cue-based MMLU experiments and examines rewriting, rollout-based RL, and simulator configurations.
  • Dissuading Cues: Dissuading-cue experiments fit training data perfectly but reach only about 57% test G-mean.This is identified as a setting where CST does not generalize well, despite the cues being persuasive to Qwen3-235B-A22B.
  • Prompting and Training Comparisons: CST’s cue-based monitor accuracy exceeds VFT’s on MMLU, with better recall and lower false-positive rate.VFT trains models to verbalize cue influence based on an LLM judge, whereas CST rewards CoTs that support counterfactual simulation.
  • Online vs Offline Training: Online training recomputes model samples each round, whereas offline training evaluates once and fits the resulting data using the same 300 gradient steps.The comparison is conducted on MMLU cue-based counterfactuals with gpt-oss-120b and Qwen3-235B-A22B.
  • Reward Structure Ablation: The reward-structure ablation compares CST with positive example finetuning, which assigns reward solely from reasoning-simulator success and is not contrastive.The baseline does not vary reward according to the outcome-only simulator’s prediction.
  • Rewriter Model Ablation: The task model and simulator produce similar results as rewriters, so CST uses the task model by default.The task model is kept untrained for rewriting because rewriting ability can decline during finetuning without regularization.
Loading 2602.20710v2…