PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, Noah D. Goodman

arXiv:2503.01307v2cs.CLcs.LG

TL;DR

The paper investigates why some language models improve through reinforcement learning while others plateau, focusing on whether initial reasoning behaviors explain this difference. It analyzes four behaviors and tests behavioral priming and targeted pretraining, finding that behavior patterns—not solution correctness—support improved self-improvement. The authors conclude that initial reasoning behaviors are strongly related to models’ capacity to use additional computation.

  • Problem

    Reinforcement learning improves some language models on verifiable tasks but leaves others unable to improve substantially, raising the question of which initial properties enable self-improvement.

  • Method

    The paper analyzes verification, backtracking, subgoal setting, and backward chaining, then tests controlled behavioral priming and behavior-focused OpenWebMath pretraining.

  • Results

    Models primed with reasoning behaviors improve during reinforcement learning, and incorrect solutions with proper behaviors achieve identical performance to correct-solution training.

  • Takeaways & Limitations

    Initial reasoning behaviors are strongly related to a model’s capacity for improvement and its ability to use additional test-time computation.

  • Takeaways & Limitations

    The priming intervention is domain-specific because it relies on the Countdown game, potentially limiting generalization.

Abstract

from arXiv · show

Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

1 Introduction

The paper asks why otherwise similar language models differ in their ability to improve through reinforcement learning. It proposes that initial cognitive reasoning behaviors help models use additional test-time computation effectively.

  • Motivation: Qwen-2.5-3B substantially improves under reinforcement learning on Countdown, whereas Llama-3.2-3B shows limited gains under the same process.This contrast motivates the question of which initial model properties enable self-improvement.
  • Framework: The framework analyzes verification, backtracking, subgoal setting, and backward chaining as behaviors representing search beyond linear reasoning.Verification checks errors, backtracking abandons failing approaches, subgoal setting decomposes problems, and backward chaining reasons from outcomes to inputs.
  • Findings: Qwen naturally exhibits these reasoning behaviors, particularly verification and backtracking, while Llama initially lacks them.The paper treats these initial behavioral differences as a possible explanation for divergent improvement trajectories.
  • Interventions: Priming Llama with synthetic traces containing cognitive behaviors enables substantial reinforcement-learning improvements, matching Qwen’s performance trajectory.The intervention tests whether changing the initial model’s behaviors affects later self-improvement.
  • Interventions: Priming with incorrect solutions still produces comparable gains when the solutions contain the desired reasoning patterns.This separates the contribution of behavioral structure from access to correct answers.
  • Interventions: Targeted modification of the pretraining distribution using behavior-focused OpenWebMath data can induce patterns needed for efficient use of test-time compute.The paper reports this as a second route for improving models that initially lack the behaviors.

2 Related Work

Prior work improves reasoning through external search, in-context search, and reinforcement learning. These approaches differ in whether search is performed across sampled trajectories, within language-model outputs, or through autonomous strategy discovery.

  • Research directions: Recent reasoning methods fall into external search, in-context search, and reinforcement learning approaches.The categories cover complementary ways of using additional computation or learning to improve reasoning.
  • External Search for Reasoning: External search methods improve complex-task performance by searching through multiple reasoning trajectories.Methods range from parallel sampling to approaches using verifiers or process reward models.
  • In-Context Search and Self-Improvement: In-context search enables sequential reasoning over model outputs through examples, linearized search traces, and self-correction training.This line of work focuses on search performed within language rather than through external trajectory selection.
  • Reinforcement Learning for Reasoning: Reinforcement-learning approaches aim to let models autonomously discover effective reasoning strategies from verifiable outcomes.Prior work includes off-policy, batch, and on-policy methods, with later work reporting in-context search behavior.

3 Identifying and Engineering Self-Improving Behavior

The paper investigates why comparable language models differ in reinforcement-learning self-improvement, linking successful learning to identifiable reasoning behaviors and testing behavioral interventions. Across Countdown experiments and pretraining-data curation, these behaviors distinguish trajectories, improve RL adaptation, and can be induced through targeted data selection.

  • 3.1 Initial Investigation: A tale of two models: Countdown provides a tractable testbed for mathematical reasoning, planning, and search while emphasizing problem-solving ability over domain knowledge.Its restricted search space enables analysis of reasoning capabilities without as much mathematical-knowledge confounding.
  • 3.1 Initial Investigation: A tale of two models: Qwen reaches approximately 60% accuracy after RL, substantially outperforming Llama’s 30% after both begin at similarly low performance.Qwen’s improvement includes longer responses and a qualitative behavioral shift during training.
  • 3.2 Identifying reasoning behaviors: The framework identifies verification, backtracking, subgoal setting, and backward chaining as recognizable, non-linear reasoning behaviors in model outputs.These behaviors respectively check results, revise failing approaches, decompose problems, and construct solution paths from goals toward inputs.
  • 3.3 The Role of Initial Behaviors in Self-Improvement: Qwen naturally exhibits substantially more of all four behaviors than Llama-3.2-3B and Llama-3.1-70B, while larger Llama shows uneven increases and limited backtracking.The analysis links Qwen’s performance improvements particularly to verification and backtracking, whereas Llama shows minimal evidence throughout training.
  • 3.4 Intervening on initial behaviors: Priming with backtracking or other reasoning behaviors enables substantial RL improvement, while empty chain-of-thought controls remain comparable to base Llama at approximately 30–35%.RL selectively amplifies empirically useful behaviors, retaining backtracking and verification while diminishing backward chaining and subgoal setting in the all-strategies condition.
  • 3.4 Intervening on initial behaviors: Incorrect priming examples containing the desired behaviors achieve identical performance to correct examples, indicating that behavioral patterns rather than answer correctness drive the intervention.The result suggests reasoning patterns from weaker models can bootstrap more capable models through RL.
  • 3.4 Intervening on initial behaviors: The Countdown-based priming method is domain-specific, which may adversely impact generalization beyond that setting.This limitation motivates the complementary use of curated pretraining data to induce behaviors before RL.
  • 3.5 Selectively amplifying behaviors in pretraining data: Cognitive behaviors appear infrequently in 200,000 sampled OpenWebMath and FineMath documents, motivating behavior-enriched pretraining data.After pretraining and RL, the behavior-enriched Llama model reaches performance comparable to Qwen, while the control shows limited improvement and base-Llama-like behaviors.

4 Discussion

The findings link initial reasoning behaviors to language models’ ability to improve through reinforcement learning and additional test-time computation. The discussion also bounds this conclusion to task-dependent behaviors and calls for broader investigation.

  • 4 Discussion: Cognitive behaviors, especially verification and backtracking, are associated with stronger self-improvement through reinforcement learning.Priming models with these behaviors also produced gains, including when the solutions were incorrect.
  • 4 Discussion: Behavior-enriched OpenWebMath pretraining enabled Llama to achieve self-improvement comparable to Qwen.The result suggests that improvement capacity can be engineered through curated pretraining data.
  • 4 Discussion: In Countdown, backtracking and verification were the most critical behaviors studied.The authors note that behaviors amplified or suppressed during RL may depend strongly on the task and optimization environment.
  • 4 Discussion: The four specified behaviors are not exhaustive, and their applicability beyond Countdown remains an open question.Future work should examine task-specific constraints and additional behaviors such as analogy-making and knowledge-state identification.
  • 4 Discussion: Cognitive behaviors enable models to use increased test-time compute to solve increasingly challenging problems.The conclusion frames these behaviors as a basis for effective reasoning and possible future discovery of new reasoning strategies.

A Data Generation

The Countdown evaluation uses randomly generated problems with an equal split between three-digit and four-digit instances, maintaining consistent difficulty parameters while varying trials.

  • B Data Generation: Countdown problems are evenly divided between 3-digit and 4-digit instances.Starting numbers and target numbers are randomly sampled for variability across trials.

B Priming

The priming data consists of five supervised fine-tuning datasets generated with Claude and designed to isolate or combine the target reasoning behaviors. Customized prompts enforce the intended behavior composition.

  • B Priming: Claude 3.5 Sonnet generates reasoning trajectories for five supervised fine-tuning datasets.The datasets target different combinations of backtracking, verification, subgoal setting, and backward chaining.
  • B Priming: The Backtracking Only dataset restricts trajectories to exploring paths and retreating from dead ends.It isolates backtracking as the intended reasoning strategy.
  • B Priming: The remaining combination datasets pair backtracking with verification, subgoal setting, or backward chaining.Each pairing emphasizes checking intermediate solutions, decomposing problems, or reasoning backward from the goal.
  • B Priming: The All Strategies dataset incorporates all four reasoning strategies.It provides the comprehensive behavioral condition.
  • B Priming: Customized system prompts elicit only the specified behaviors while prohibiting the others.The prompts were designed to maintain consistency across datasets and support behavioral comparisons.

D Metrics

The evaluation tracks average occurrences of four reasoning behaviors in sampled model trajectories using a classifier-based counting pipeline.

  • D Metrics: The metrics measure average backtracking, verification, backward-chaining, and subgoal-setting counts.Each metric tracks the average number of distinct occurrences of its corresponding behavior.
  • D Metrics: Model samples use temperature 1.0 and a maximum length of 1024 tokens.A GPT-4o-mini classifier evaluates the resulting reasoning trajectories.
  • D Metrics: The classifier receives four behavior-specific questions for each reasoning trajectory.Prompts include examples, such as a sequence that produces a value different from the target for verification.
  • D Metrics: The classifier counts distinct occurrences of each behavior using temperature 0 for reproducibility.It samples 512 tokens while reporting the four behavior counts.

E Pretraining Data Interventions.

The paper measures cognitive-behavior labels and uses behavior-focused data curation to study reasoning patterns in training data. GPT-4o-mini shows substantial agreement with Claude and human raters across these labels.

  • Analyzing Frequency in Pretrained Data: The analysis samples 200,000 documents from OpenWebMath and FineMath to estimate the natural frequency of reasoning behaviors.Qwen2.5-32B classifies the documents for verification, backtracking, subgoal setting, and backward chaining.
  • Inter-rater Reliability: The reliability analysis covers verification, backtracking, subgoal setting, and backward chaining across GPT-4o-mini, Claude, and human raters.These labels correspond to the four panels in Figure 11.
  • Curating Pretraining Data: The curation pipeline identifies passages exhibiting specific cognitive behaviors and separates them into behavior-curated and behavior-minimized control datasets.Binary behavior evaluations populate the two training datasets.
  • Inter-rater Reliability: The reliability study compares GPT-4o-mini, Claude, and two human annotators on 100 reasoning trajectories across four problem-solving behaviors.Agreement is measured with the Intraclass Correlation Coefficient (ICC3).
  • Inter-rater Reliability: GPT-4o-mini reaches ICC3 values of 0.70 with Claude and 0.66 and 0.65 with human raters for verification.For backtracking, agreement is 0.74 with Claude and 0.67 with human raters.

G Test-time Scaling.

The study evaluates how inference token budgets affect performance across Llama RL checkpoints. Performance improves with larger budgets, while later checkpoints perform better even under restricted budgets.

  • Test-time Scaling: Figure 12 compares training checkpoints using maximum token lengths of 128, 256, 512, and 1024 tokens.The figure examines how available inference computation relates to model performance.
  • Test-time Scaling: Performance consistently improves as the maximum inference token length increases from 128 to 1024 tokens, with diminishing returns after 512 tokens.The analysis uses all-strategies-primed Llama checkpoints from 0 to 200 PPO training steps.
  • Test-time Scaling: Later Llama checkpoints achieve scores of ≈0.4 with just 128 tokens, whereas earlier performance is near zero under the same restriction.The stronger restricted-budget results occur after 150–200 PPO steps.

H Transfer of Behaviors to Other Domains.

Reasoning behaviors learned in mathematical settings transfer to GPQA and MATH. Both curated pretraining and Countdown RL increase observed behavior frequencies beyond the original training context.

  • Transfer to Question Answering: The curated-pretraining model shows higher frequencies of all four cognitive behaviors than the behavior-minimized control on GPQA.Subgoal setting rises from 0.7 to 6.5 instances per question.
  • Transfer to Question Answering: Figure 14 compares behavior frequencies between base and Countdown-RL-finetuned Qwen models on GPQA and MATH.Figure 13 compares curated-pretraining and behavior-minimized-control models on GPQA.
  • Transfer to Question Answering: Countdown RL increases all four cognitive behaviors when Qwen solves GPQA, including verification from 0.008 to 0.03 and backtracking from 0.019 to 0.114.Subgoal setting increases from 0.478 to 0.616, while backward chaining increases from 0.128 to 0.313.
  • Transfer to Question Answering: On MATH, the RL-finetuned Qwen model increases performance from 38% to 50% with the paper’s prompt format.With Qwen’s 4-shot prompt format, performance increases from 44% to 50% in the reported comparison.
  • Transfer to Question Answering: Subgoal setting and backward chaining become accentuated after RL despite not appearing in the Countdown RL-finetuned model’s Countdown behavior profile.The reported increases therefore include behaviors not demonstrated on the training task.

I Results for Instruction-tuned Models

Instruction tuning affects both task behavior and subsequent RL dynamics. Instruction-tuned Llama and Qwen models nearly saturate task score, use more tokens, and progressively strengthen verification and backtracking during RL.

  • Results for Instruction-tuned Models: Instruction-tuned Llama and Qwen models nearly saturate task score and use more of their token budgets than their base-model counterparts.The comparison is conducted during Countdown RL training.
  • Results for Instruction-tuned Models: Verification and backtracking emerge and progressively strengthen during RL for the instruction-tuned models.The study applies behavioral probes to instruction-tuned Llama-3.2-3B and Qwen-2.5-3B variants.
  • Results for Instruction-tuned Models: Figure 15 compares score, response length, and four cognitive behaviors between base and instruction-tuned models across Countdown RL.The behavior trajectories are shown separately for Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct.
Loading 2503.01307v2…