PaperlayerView exact revision on arXiv

Source-linked AI summary

Are DeepSeek R1 And Other Reasoning Models More Faithful?

James Chua, Owain Evans

arXiv:2501.08156v5cs.LG

TL;DR

The paper examines whether reasoning models’ Chains of Thought more faithfully reflect the factors influencing their answers. It tests cue articulation on MMLU questions across reasoning and non-reasoning models, finding a large and consistent faithfulness advantage for reasoning models; reward-model preferences may help explain the difference.

  • Problem

    It is unclear whether reasoning models’ more effective Chains of Thought better reflect their actual reasoning process, an issue with implications for AI safety.

  • Method

    The study inserts answer-influencing cues into MMLU prompts and evaluates whether models explicitly describe each cue’s effect, including tests addressing false positives.

  • Results

    Reasoning models articulate influential cues significantly better than non-reasoning models across cue types and after accounting for false positives.

  • Takeaways & Limitations

    The results suggest reasoning models have more faithful reasoning, while the authors hypothesize that rule-based reward systems may incentivize articulation.

  • Takeaways & Limitations

    The study measures only whether models articulate one influential cue and does not test broader faithfulness forms, so improved articulation does not establish high absolute faithfulness.

Abstract

from arXiv · show

Language models trained to solve reasoning tasks via reinforcement learning have achieved striking results. We refer to these models as reasoning models. Are the Chains of Thought (CoTs) of reasoning models more faithful than traditional models? We evaluate three reasoning models (based on Qwen-2.5, Gemini-2, and DeepSeek-V3-Base) on an existing test of faithful CoT. To measure faithfulness, we test whether models can describe how a cue in their prompt influences their answer to MMLU questions. For example, when the cue "A Stanford Professor thinks the answer is D" is added to the prompt, models sometimes switch their answer to D. In such cases, the DeepSeek-R1 reasoning model describes the cue's influence 59% of the time, compared to 7% for the non-reasoning DeepSeek model. We evaluate seven types of cue, such as misleading few-shot examples and suggestive follow-up questions from the user. Reasoning models describe cues that influence them much more reliably than all the non-reasoning models tested (including Claude-3.5-Sonnet and GPT-4o). In an additional experiment, we provide evidence suggesting that the use of reward models causes less faithful responses -- which may help explain why non-reasoning models are less faithful. Our study has two main limitations. First, we test faithfulness using a set of artificial tasks, which may not reflect realistic use-cases. Second, we only measure one specific aspect of faithfulness -- whether models can describe the influence of cues. Future research should investigate whether the advantage of reasoning models in faithfulness holds for a broader set of tests. Still, we think this increase in faithfulness is promising for the explainability of language models.

1 INTRODUCTION

The paper asks whether reasoning models’ more effective Chains of Thought are also more faithful, focusing on whether models articulate prompt cues that influence their answers. Across cue-based tests, reasoning models articulate influential cues more reliably than non-reasoning models, while reward-model preferences may help explain the gap.

  • Motivation: Reasoning models use reinforcement learning and Chain of Thought, prompting the question of whether their explicit reasoning better reflects their actual reasoning process.The paper distinguishes these models from the previous non-reasoning paradigm, whose Chains of Thought are less effective, particularly over many reasoning steps.
  • Evaluation: The faithfulness test inserts behavior-influencing cues into MMLU prompts and checks whether models describe the cues’ influence in their Chains of Thought.The study focuses on this specific form of faithfulness because the ground truth of internal reasoning may be difficult to determine.
  • Evaluation: Three reasoning models are compared with equivalent non-reasoning models, alongside additional non-reasoning models including Claude-3.5-Sonnet, GPT-4o, Grok-2-Preview, and Llama-3.3-70b-Instruct.The direct comparisons use QwQ-32b-Preview, Gemini-2.0-flash-thinking-exp, and DeepSeek-R1 against corresponding non-reasoning models.
  • Results: Across tested cues, reasoning models outperform Claude-3.5-Sonnet and show superior recall and F1 scores for articulating influential cues.The paper also reports similar results when reasoning models are compared with the best non-reasoning model for each cue, although articulation varies across cue types.
  • Interpretation: Reward models strongly prefer unfaithful responses across all tested models and cues, which may help explain the lower faithfulness of non-reasoning models.The authors present this as indirect evidence related to differences in training techniques.

2 SETUP AND RESULTS OF CUES

The study tests whether models articulate cues that cause them to switch answers on MMLU questions. Reasoning models generally articulate influential cues more often than non-reasoning models, although performance varies substantially by cue type.

  • Setup: Reasoning models were evaluated only on switched examples, where a cue changes the model’s answer to the cued response.Models’ answers were evaluated after Chain of Thought reasoning; spontaneous switching occurred around 1%–5%, compared with roughly 10%–30% for tested cues on reasoning models.
  • Professor’s Opinion: 47% Qwen, 68% Gemini, and 59% DeepSeek reasoning-model articulation rates exceeded the best non-reasoning model’s 13% for the Stanford professor cue.The cue adds a professor’s opinion about which answer choice is correct; reasoning models were significantly better at articulating its influence.
  • Few-Shot with Black Square: 17% Qwen, 35% Gemini, and 25% DeepSeek reasoning-model articulation rates exceeded Claude-3.5-Sonnet’s 3% for the black-square cue.The cue uses marked answer choices in few-shot examples and tests whether models explain switching to the marked option; a white-square variant produced similar results.
  • Other Cues: 16% Qwen, 47% Gemini, and 34% DeepSeek R1 articulated the argument cue, compared with 2% for GPT-4o.The argument cue inserts a long argument supporting a particular option.
  • Other Cues: 49% Gemini reasoning-model articulation for wrong few-shot examples exceeded the best non-reasoning model’s 2%, while post-hoc and follow-up-question cues produced mostly poor performance.For post-hoc cues, Qwen and DeepSeek R1 articulated 10% ± 4% and 6% ± 5%, Gemini 0%, and all non-reasoning models 0%; all models were near 0% for “Are you sure?”.

3 WHY ARE REASONING MODELS MORE FAITHFUL?

The paper investigates why reasoning models articulate influential cues more faithfully, focusing on differences between outcome-based rewards, reward modeling, and response-length incentives.

  • Reward-model incentives: Reasoning models are trained primarily with outcome-based rewards, whereas traditional non-reasoning models commonly use reward models evaluating the full chain of thought and answer.Outcome-based rewards depend on answer correctness rather than CoT steps.
  • Reward-model incentives: Reward models may prefer unfaithful responses because they can mistake genuine cue articulation for hallucination and favor simpler responses over backtracking.The paper presents this as a proposed explanation for reward-model preferences.
  • Reward-model incentives: 63%–71%: GPT-4o used as a reward model highly preferred unfaithful responses from reasoning models across the tested models.The range runs from Gemini reasoning to Qwen reasoning.
  • Reward-model incentives: Strong preferences for unfaithful responses remained for Professor and Black Squares cues, which lacked instructions that might otherwise explain the preference.This weakens the objection that instruction wording alone caused the reward-model preference.
  • Length penalties: 69%, 82%, and 73%: selecting the shortest response favored unfaithful responses for Gemini, Qwen, and DeepSeek reasoning models, respectively.The unfaithful response won a majority of the time across different cues.
  • Conclusion: The authors conclude that reward models and length penalties provide evidence of mechanisms reinforcing unfaithful responses.The reward-model evidence is indirect, and DeepSeek-R1 also used some CoT reward modeling for harmlessness training.

4 DISCUSSION

The discussion tests objections to the faithfulness results, examines robustness across cues and training data, and considers whether distillation can transfer cue-articulation capabilities.

  • Robustness checks: Reasoning models often produce CoTs 2–5x longer than non-reasoning models, raising concern that they may mention cues merely because they mention more factors.The paper addresses this concern with false-positive analyses.
  • Robustness checks: The study uses a strict articulation criterion and F1, the harmonic mean of precision and recall, to address false positives from cue mentions unrelated to answer switching.Simple repetition of a cue does not count as articulation under this criterion.
  • Robustness checks: Reasoning models articulate significantly above a random-articulation F1 baseline for all cues except Post-hoc and Are-You-Sure?.The baseline accounts for the rate at which models switch to the cue.
  • Non-reasoning models: Adding instructions to include all relevant factors in reasoning did not significantly improve articulation rates for non-reasoning models.This tests whether poor articulation mainly reflects insufficient prompting.
  • Cue variation: Articulation rates differ across cues, but cue-specific acceptability does not explain why reasoning models outperform non-reasoning models overall.The paper gives Stanford-professor opinions and “Are you sure?” as contrasting examples.
  • Distillation: The distilled Llama-70B model articulated the Stanford professor cue 48% of the time versus near 0% for the original model, while its Black Squares articulation was 9% versus 25% for DeepSeek R1.The authors interpret the overall pattern as evidence that distillation can transfer capabilities related to faithfulness, though imperfectly.

5 RELATED WORK

Related work frames faithfulness as a broader interpretability and robustness problem involving counterfactual tests, training incentives, causal reasoning, and adversarial behavior.

  • Faithfulness methods: Prior faithfulness tests include counterfactual simulation and input reconstruction, alongside the cue-influence method used here.These approaches examine whether explanations support predictions or reflect relevant inputs.
  • Improving faithfulness: Existing approaches to improve faithfulness include incentivizing causal links, decomposing CoT into verifiable sub-steps, using self-consistency, and training models on realistic decisions.These methods target different sources or forms of unfaithfulness.
  • Causes of unfaithfulness: Prior work suggests RLHF can incentivize explanations that convince humans while remaining detached from internal reasoning, and that training paradigms may not directly teach counterfactual self-explanation.Other work identifies encoded reasoning and steganography as additional challenges.
  • Robustness: Cue articulation can help monitor undesirable behavior, while robustness research also studies resistance to adversarial cues, many-shot attacks, and jailbreaking.Recent findings link greater inference-time compute with improved robustness across attack strategies.

6 LIMITATIONS

The study’s conclusions are limited by its narrow articulation-based faithfulness measure, subjective judging, and synthetic cue settings that may not represent realistic applications.

  • Scope of faithfulness: The study measures only whether models articulate one influential cue, not comprehensive articulation or other unfaithfulness forms such as encoded reasoning and steganography.Improved articulation therefore does not imply that reasoning models are highly faithful in absolute terms.
  • Evaluation: The articulation criterion is subjective because it relies on a judge model, and only part of the evaluation was manually checked.The authors recommend further validation with human labelers.
  • Experimental setting: The experiments use synthetic scenarios created by editing questions to insert cues, leaving realism in domains such as housing eligibility and medical decisions for future work.The practical boundary is that these results may not transfer directly to realistic settings.

7 CONCLUSION

Reasoning models articulate influential cues more faithfully than non-reasoning models across cue types and after accounting for false positives. The authors hypothesize that rule-based rewards may incentivize this faithfulness, while noting that further investigation is needed.

  • Reasoning models articulate influential cues significantly better than non-reasoning models.
  • The improved articulation holds across different cue types and after accounting for false positives.
  • Rule-based reward systems may incentivize faithfulness in reasoning models, although the authors lack technical documentation for all models.

A.1 COMPARING TO THE BEST NON-REASONING MODEL

Reasoning models perform better overall than the best non-reasoning model for articulating cues, while showing no distinctive answer-switching pattern. Their Chains of Thought are also substantially longer, and reward-model evaluations favor unfaithful responses.

  • A.1 COMPARING TO THE BEST NON-REASONING MODEL: Reasoning models perform better overall than the best non-reasoning model for articulating cues.The comparison covers seven non-reasoning models, including Claude-3.5-Sonnet and GPT-4o.
  • A.1 COMPARING TO THE BEST NON-REASONING MODEL: Reasoning models do not show a significantly different switching rate from non-reasoning models.Switching is measured on questions where the cue suggests an answer different from the model’s original answer.
  • A.1 COMPARING TO THE BEST NON-REASONING MODEL: 1% to 4% is the baseline switching rate caused by prepending “Please answer this question:”, substantially below cue-induced switching.This baseline measures prompt sensitivity without a substantive cue.
  • A.1 COMPARING TO THE BEST NON-REASONING MODEL: Reasoning models’ CoTs are often 2-5 times longer than those of non-reasoning models.The comparison uses median character lengths across different cue prompts.
  • A.1 COMPARING TO THE BEST NON-REASONING MODEL: GPT-4o strongly prefers unfaithful responses from reasoning assistants when used as a reward model.The evaluation compares faithful and unfaithful responses for instruction following across cues and models.

A.5 DISTILLED MODEL RESULTS

Distillation on DeepSeek R1 Chains of Thought improves cue articulation, but the improvement depends on cue type and model size.

  • A.5 DISTILLED MODEL RESULTS: Models distilled on DeepSeek R1 CoTs show improved articulation rates.The figure compares articulation rates for distilled models.
  • A.5 DISTILLED MODEL RESULTS: Articulation varies across cue types in distilled models.The gains are not uniform across the evaluated cues.
  • A.5 DISTILLED MODEL RESULTS: Smaller distilled models show reduced articulation.Model size is associated with lower articulation in the reported results.

A.6 RECALL, PRECISION AND F1

The paper evaluates cue articulation using recall, precision, and F1, finding that reasoning models articulate influential cues more reliably and with better overall balance than non-reasoning models.

  • A.6 RECALL, PRECISION AND F1: Reasoning models achieve the highest F1 scores, balancing precision and recall for cue articulation.Non-reasoning models frequently receive 0.0 scores because they do not articulate switched responses.
  • A.6 RECALL, PRECISION AND F1: Precision measures false positives where models claim a cue caused an answer change even though the cue was unnecessary.This test addresses whether articulation reflects genuine cue influence rather than generic cue mentions.
  • A.6 RECALL, PRECISION AND F1: Reasoning models show notably higher F1 scores across scenarios than non-reasoning models.The F1 analysis focuses on articulation in switched answers.
  • A.6 RECALL, PRECISION AND F1: The evaluation prompt supplies true and false examples and asks the model to classify whether a response articulates the cue.The prompt also specifies structured JSON output for the evaluator.
  • A.6 RECALL, PRECISION AND F1: The post-hoc cue experiment tests whether a model explains an initial answer that anchors its later response.The model is instructed not to anchor, and analysis focuses on samples where anchoring persists.

C SAMPLE CUES AND MODEL ARTICULATIONS

The section presents examples of cues inserted into MMLU prompts and the resulting model articulations, including professor opinions, arguments, few-shot examples, and follow-up questions.

  • Professor cues append an asserted answer from a Stanford professor to an MMLU question, prompting models to consider that answer.
  • Wrong few-shot and black-squares cues are shown as examples of misleading or spurious information embedded in prompts.
  • Argument cues provide another person’s purported reasoning before the question, which models may evaluate explicitly in their responses.
  • Examples also include an argument cue and an “Are You Sure” follow-up, with the latter described as poorly articulated by all models.
  • The post-hoc cue inserts “The best answer is (B)” into the assistant’s first response and tests whether a resulting answer switch is articulated.

D EXAMPLES WHERE REASONING MODELS ARTICULATE THAT THE CUE DOES NOT DECIDE THEIR FINAL ANSWER

The examples show reasoning models acknowledging cues while explaining that those cues do not determine their final answers.

  • The GPT-4o judge marks these cases as non-articulations because the cues are acknowledged but not treated as deciding the final answers.
  • Reasoning models can mention a professor cue while reasoning independently to select a different answer.
  • A DeepSeek-R1 example acknowledges an argument suggesting answer C but independently chooses answer D.
Loading 2501.08156v5…