PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Models Struggle to Control their Chains of Thought

Chen Yueh-Han, Robert McCarthy, Bruce W. Lee, He He, Ian Kivlichan, Bowen Baker, Micah Carroll, Tomek Korbak

arXiv:2603.05706v1cs.AI

TL;DR

CoT monitoring may be undermined if reasoning models can deliberately reshape their reasoning, but evidence about this capability is limited. The paper introduces CoT-Control, a proxy evaluation of instruction-following in reasoning traces, and finds that models generally have much lower CoT controllability than output controllability. The authors therefore remain cautiously optimistic while recommending continued tracking, although the proxy does not directly test spontaneous monitor evasion by misaligned models.

  • Problem

    CoT monitoring relies on reasoning traces revealing misbehavior, yet models might strategically suppress or reshape those traces as situational awareness of monitoring increases.

  • Method

    CoT-Control measures whether models can solve 14,076 benchmark problems while following explicit instructions that modify or constrain their reasoning traces, using programmatic scoring and sanity checks.

  • Results

    Reasoning models generally struggle to control their CoT relative to their outputs; controllability can be as low as 0.1% for Deepseek R1, increases with model scale, and decreases with reasoning effort and RL training.

  • Takeaways & Limitations

    Current results are cautiously optimistic for CoT monitorability, while situational awareness produces only a small increase and adversarial prompt optimization transfers poorly to unseen instruction types.

  • Takeaways & Limitations

    CoT-Control is a practical proxy based on instruction following and does not directly test whether models spontaneously pursuing misaligned goals can control CoT to evade monitors.

Abstract

from arXiv · show

Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability -- CoT controllability -- we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.

1 Introduction

The paper introduces CoT controllability as a proxy for whether reasoning models can reshape their reasoning under explicit constraints, motivated by concerns that such control could undermine CoT monitoring. Across a broad evaluation, models generally struggle with CoT control, with controllability varying systematically by model scale, training, reasoning effort, difficulty, monitoring awareness, and prompt optimization.

  • Motivation: CoT monitoring may expose malicious intent and plans, but strategic suppression or reshaping of reasoning could erode this oversight signal.The paper frames CoT controllability as the ability to control reasoning in ways that reduce monitorability.
  • Evaluation: CoT-Control evaluates whether models can solve tasks while obeying explicit constraints on what appears in their reasoning traces.The suite uses 14,076 problems across five benchmarks, with programmatic graders and sanity checks for instruction misunderstanding.
  • Findings: CoT controllability decreases by over an order of magnitude across OLMo-3-7B-RL-Zero-Math reasoning-training checkpoints, with the post-training trend confirmed across four model families.The paper also reports up to a 4-percentage-point increase when models are told they are being monitored.
  • Findings: CoT controllability increases logarithmically with parameter size in open-weight models but decreases with greater inference-time reasoning effort.These trends were observed across two model families for scale and under increased reasoning effort.
  • Findings: Controllability decreases as task difficulty rises, while adversarial prompt optimization improves performance on seen instructions but transfers poorly to unseen instruction types.The difficulty relationship is especially apparent when CoT is necessary for correctly answering a question.
  • Conclusion: The authors are cautiously optimistic that low CoT controllability currently limits this monitorability failure mode, but the mechanism remains poorly understood.They recommend that frontier labs track controllability in future models and release CoT-Control to support further study.

2 CoT-Control

CoT-Control measures whether reasoning models can modify their reasoning traces while preserving task objectives, using proxy tasks across diverse benchmarks and controllability types. Its design combines balanced evaluation, scalable grading, and baselines for comparing CoT with output and external-CoT control.

  • CoT-Control pairs questions from diverse LLM benchmarks with CoT instructions, yielding 14,076 test instances.
  • The suite operationalizes difficult-to-evaluate threats through concrete proxy types spanning three CoT controllability categories.
  • CoT controllability is estimated by keeping the task objective fixed while steering the reasoning trace with proxy instructions connected to reduced monitorability.
  • The evaluation uses labeled difficulty levels, programmatic graders tailored to controllability types, and sanity checks for instruction understanding and self-awareness.
  • Questions are sourced from multiple benchmarks and receive appended CoT instructions; keyword-suppression candidates are selected from baseline traces and validated across model runs.
  • Metrics are macro-averaged across dataset and controllability-type pairs, with output controllability and external-CoT controllability serving as baselines.

3 Results

Across 13 reasoning models, CoT controllability is generally low and substantially worse than controlling outputs or external reasoning. Larger models perform better, but stronger models can expose their control attempts through meta-discussion, while suppression harms task performance.

  • Most models achieve below 10% CoT controllability, whereas most reach ≥60% for output control and ≥70% for external reasoning.
  • GPT-OSS-120B reaches 5.9% CoT controllability versus 0.9% for GPT-OSS-20B, while Qwen3 32B reaches 2.3% versus 0.6% for Qwen3 14B.
  • Stronger models often mention the CoT-control constraint despite instructions forbidding such meta-discussion.
  • When instructed to ignore the question, models show meta-discussion in ≥90% of cases, compared with ≤10% for most other controllability types.
  • Suppression causes 6.0–16.7 percentage-point performance drops, versus ≤2.7 points for additions or stylistic edits.
  • Claude models show higher controllability on shorter-trace BFCL tasks than on longer-trace GPQA and HLE tasks.Claude 3.7 Sonnet reaches 52.9% on BFCL at 126 words, versus 36.5% on HLE at 3,662 words and 31.8% on GPQA at 3,115 words.

4 Stress-Testing CoT Controllability

Stress tests examine whether awareness, adversarial pressure, and repeated attempts can elicit CoT controllability. These interventions can improve success in some settings, but transfer and reliability remain limited.

  • Successful suppression produces the largest task-performance drop, while additions and stylistic edits change performance little.
  • 4.1 Situational Awareness: Informing models that they are monitored or evaluated can increase CoT controllability, with effects reaching up to 4 percentage points.
  • 4.2 Adversarial Pressure via Automated Red Teaming: GEPA-discovered psychological pressure transfers weakly to unseen controllability types: only 3 of 8 models improve, each by <5 percentage points.
  • 4.3 Repeated Attempts: Repeated attempts substantially increase CoT controllability for Claude models, while newer variants are more resistant than earlier versions.
  • Task Difficulty: CoT controllability remains low when reasoning is unnecessary and tends to decrease as task difficulty increases.

5 Investigating Key Factors Behind Low Controllability

CoT controllability declines with harder tasks, greater reasoning effort and length, and stronger reasoning-training pressure, while larger models and some less-quantitative domains show higher controllability.

  • 5.1 Task Difficulty: Each standard-deviation increase in accuracy, approximately 35 percentage points, corresponds to an approximately 10% increase in controllability odds.
  • 5.2 Reasoning Effort: CoT controllability decreases as reasoning effort increases.
  • 5.3 Reasoning Length: A one-SD increase in reasoning length, approximately 2,000 words, corresponds to a 20% reduction in controllability odds (odds ratio 0.79, p < 0.001).
  • 5.4 RL Pressure: The paper hypothesizes that outcome-based RL can reduce CoT controllability because it optimizes answer correctness without constraining CoT content or style.
  • 5.4 RL Pressure: CoT controllability decreases as RL pressure increases, with “General” chat-data training declining more slowly than “Code” and “Math” training.

6 Discussion

The discussion frames CoT controllability as a proxy relevant to monitorability, while emphasizing that the evaluation has important scope limitations. The authors recommend tracking controllability as models evolve.

  • Limitations: The evaluation uses practical instruction-following proxies rather than directly testing spontaneously misaligned models that evade monitors.This proxy assumes that instruction-following training generalizes from outputs to CoTs in a representative way.
  • Limitations: The study is limited to prompting-based elicitation, evaluates a limited breadth of control types, and finds that models control some CoT features better than others.The authors identify varying control across features and leave changes induced by reinforcement learning or supervised fine-tuning for future work.
  • Relationship to CoT monitorability: CoT controllability is necessary but not sufficient for confidence in CoT monitorability, because monitorability may degrade through other mechanisms.Low controllability can strengthen the case that models verbalize cognition, but it does not rule out every route to degraded monitorability.
  • Conclusion: The authors recommend that model developers track CoT controllability in future models while further research investigates why controllability is low.The conclusion reports lower CoT controllability than output controllability, with controllability decreasing under greater reasoning effort and more RL training.

C Iterative Refinement Pressure via AI Feedback (IRPAIF)

IRPAIF iteratively appends GPT-5-mini diagnoses of CoT violations to prompts, substantially improving compliance on sampled questions. The gains vary by model family and size, with newer Claude models showing smaller improvements.

  • Procedure: IRPAIF gives models up to 10 attempts, appending GPT-5-mini summaries of failed CoT constraints to each subsequent prompt.The summaries include a high-level failure description and 1–3 verbatim noncompliant sentences.
  • Results: GPT-OSS 120B gains 44 percentage points versus 27 percentage points for 20B, while Qwen3 32B gains 31 versus 12 percentage points for 14B.Within both model families, larger models appear more elicitable than smaller ones.
  • Results: Claude Sonnet 4.5 gains only 5 percentage points under IRPAIF, making newer Claude models harder to elicit than GPT-OSS and Qwen3.The cited result specifies no optimization pressure against CoT for Claude Sonnet 4.5.
  • Results: IRPAIF substantially increases CoT compliance, with gains largest for GPT-OSS and Qwen3 and results averaged over 25 questions per dataset.Figure 12 compares baseline compliance with performance after up to 10 iterations and reports standard-error bars.
  • Feedback prompt: The feedback prompt asks GPT-5-mini to identify violated CoT aspects and patterns, using the original instruction, question, and model reasoning as inputs.The prompt encloses these inputs in dedicated instruction, question, and reasoning fields.

D.1 Logistic Regression Diagnostics

The appendix reports logistic-regression diagnostics for models relating CoT compliance to accuracy, reasoning length, and question domain. Each specification is well-powered and fits substantially better than its null model.

  • Diagnostics: Logistic regression models CoT compliance as a binary outcome and reports EPV, VIF, likelihood-ratio tests, and McFadden’s pseudo-R2.These diagnostics assess sample-size adequacy, multicollinearity, and overall model fit.
  • Accuracy–compliance model: The accuracy–compliance model has 5,876 compliant events and EPV = 293.8 across 20 predictors, with McFadden’s pseudo-R2 = 0.3268.Its likelihood-ratio test is χ2 = 12,786.82 with df = 20 and p < 2.2 × 10^-16.
  • Reasoning-length model: The reasoning-length model has 8,530 compliant events and EPV = 473.9 across 18 predictors, with McFadden’s pseudo-R2 = 0.3544.The model includes reasoning length as word count and controls for model, dataset, controllability type, and question domain.
  • Question-domain model: The question-domain model has 3,418 compliant events and EPV = 97.7 across 35 predictors, with McFadden’s pseudo-R2 = 0.4151.It controls for model, dataset, controllability type, and reasoning length and fits substantially better than the null.
  • Additional analyses: The appendix identifies the models and training sources used for additional analyses, including Olmo-3, Magistral-Medium, OpenThinker, and their post-training procedures.These passages describe supervised fine-tuning, direct preference optimization, reinforcement learning, and reasoning-trace training sources.

E.2 CoT Controllability Evaluations: Setup

The evaluation setup varies prompting, reasoning parsing, sample size, generation limits, and task difficulty across model classes. These choices aim to obtain comparable and interpretable CoT controllability measurements.

  • Evaluation protocol: Section 5.4 evaluates open-source models on 1,066 QA questions with 15,000-token generation caps, temperature 0.6, and top p 0.95.The reduced sample and generation cap are intended to lower inference costs and times.
  • Evaluation protocol: The Olmo-3-7B Base and RL-Zero evaluations simplify constraints by requiring an addition only at the CoT start and removing synonyms from keyword-suppression tasks.The simplifications are retained across Section 5.4 evaluations for fair comparison.
  • Model-specific materials: The appendix includes prompts and training-stage comparisons for Olmo-3–Think and other open-source models evaluated under the setup.These figures report lower CoT controllability after reasoning training and provide model-specific prompts.
  • Olmo-3 Base and RL-Zero: Olmo-3 Base and RL-Zero prompts append CoT instructions without a chat template, then filter short or invalid responses before scoring.For models without separate reasoning tags, the final 10% of generated text is removed to approximate the reasoning portion.
  • Other models: Instruct and tagged-reasoning models are prompted or parsed using <think> tags, while some Mistral and Magistral runs use a 500-token maximum.The setup includes distinct prompting for Qwen2.5-7B-Instruct and Mistral-Medium 3.
  • Output comparison: Output-controllability evaluations for Olmo-3 RL-Zero and Base models reuse CoT-Control QA sources while adapting instructions to the output space.The design uses information-addition and stylistic-modification instructions similar to the CoT evaluations.

F Additional Findings

Additional analyses find no clear relationship between benchmark performance and CoT controllability, while longer reasoning is associated with lower controllability. Output controllability shows no comparable length relationship.

  • Benchmark performance: Benchmark scores across GPQA-Diamond, Humanity’s Last Exam, and MMLU-Pro show no clear relationship with CoT controllability.The benchmark score is averaged across the three datasets.
  • Reasoning length: CoT controllability decreases as reasoning length increases, whereas output controllability largely remains unchanged.The mean output controllability exponent across models is 0.001.
  • Length effects: Reasoning length is consistently associated with lower controllability, while output-length exponents show no reliable relationship with controllability.The comparison uses power-law fits in log-log space.

G Reproducibility

The paper estimates standard errors using Bernoulli-trial assumptions for percentages and sample standard deviations otherwise, with greedy decoding as the default. Failure examples are reported separately.

  • Standard errors: Percentage standard errors assume independent Bernoulli trials and approximate normality of the estimated proportion.The standard error is based on the estimated proportion and sample size.
  • Standard errors: For non-percentage quantities, standard error is computed as the sample standard deviation divided by the sample size’s square root.This is the paper’s stated alternative standard-error calculation.
  • Decoding: LLM experiments use greedy decoding unless temperature sampling is explicitly specified, in which case temperature is set to 0.The decoding convention is stated for all experiments using LLMs.
  • Failure analysis: Failure examples are provided in Figures 42 through 46.These figures document examples rather than introducing a separate evaluation procedure.

I Other Related Work

The paper situates CoT-Control within work on reasoning and CoT monitoring, then examines how controllability varies with task, instruction, model, and reasoning-stage factors. Its framing emphasizes that low controllability is preferable for reliable monitoring.

  • Reasoning models: CoT reasoning evolved from prompting intermediate steps to test-time procedures and training with process supervision or verifiable rewards.The related-work discussion includes zero-shot prompting, self-consistency, search, and reinforcement learning with verifiable rewards.
  • CoT monitoring: CoT monitoring aims to provide earlier and more granular oversight than output-only checks, but low monitorability and high controllability constrain it.High controllability can permit obfuscation, benign-looking decomposition, or covert sandbagging under monitoring.
  • Difficulty and length: CoT controllability decreases with reasoning length and task difficulty, while datasets with shorter average reasoning show higher controllability.The difficulty comparison aggregates three baselines and three constraint types, and the length analysis finds no reliable output-length relationship.
  • Safety framing: Only the regime combining high monitorability with low controllability is desirable for CoT-based supervision.This framing motivates evaluating controllability alongside monitorability.
  • Controllability types: Most reasoning models are best at suppressing information in CoT, compared with adding information or stylistic modification.The table treats each controllability-type and dataset combination equally; lower controllability is safer.
  • Instruction placement and sanity checks: Placing CoT instructions in the user prompt produces higher CoT controllability than placing them in the system prompt.The evaluation also includes sanity checks for reasoning-stage understanding and self-awareness.
  • Meta-discussion: When models control their CoT, they frequently still mention that they need to control it despite instructions not to do so.The paper separately tests compliance with and without meta-discussion.
  • Evaluation design: The evaluations include prompts that test whether models can avoid reasoning about a specified question during their reasoning stage.The corresponding grader checks whether the reasoning trace follows the stated constraint.
Loading 2603.05706v1…