PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying Long Chain-of-Thought Reasoning in LLMs

Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, Xiang Yue

arXiv:2502.03373v1cs.CLcs.LG

TL;DR

The paper investigates how long chains-of-thought emerge and how reinforcement-learning training can reliably develop them. Through supervised fine-tuning and reinforcement-learning experiments, it finds that long-CoT supervision facilitates training, reward shaping stabilizes growth, and filtered noisy solutions can support out-of-distribution reasoning.

  • Problem

    The conditions enabling long chains-of-thought and reliable reinforcement-learning development remain unclear, despite long CoTs supporting complex reasoning strategies.

  • Method

    The paper systematically studies long-CoT reasoning through extensive supervised fine-tuning and reinforcement-learning experiments, including reward design and noisy web-extracted supervision.

  • Results

    Long-CoT supervised fine-tuning enables higher performance and easier reinforcement-learning improvements, while cosine length scaling with a repetition penalty stabilizes CoT growth and filtered noisy solutions show promise on out-of-distribution STEM reasoning.

  • Takeaways & Limitations

    Effective long-CoT training requires careful supervision, reward shaping, and scalable verifiable signals, while core skills such as branching and error validation are already present in base models.

  • Takeaways & Limitations

    The authors identify model size as a primary factor limiting emergent long-CoT behavior and note that larger-model reinforcement-learning experiments were constrained by GPU requirements and inefficient infrastructure.

Abstract

from arXiv · show

Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the mechanics of long CoT reasoning, identifying the key factors that enable models to generate long CoT trajectories. Through extensive supervised fine-tuning (SFT) and RL experiments, we present four main findings: (1) While SFT is not strictly necessary, it simplifies training and improves efficiency; (2) Reasoning capabilities tend to emerge with increased training compute, but their development is not guaranteed, making reward shaping crucial for stabilizing CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We find that leveraging noisy, web-extracted solutions with filtering mechanisms shows strong potential, particularly for out-of-distribution (OOD) tasks such as STEM reasoning; and (4) Core abilities like error correction are inherently present in base models, but incentivizing these skills effectively for complex tasks via RL demands significant compute, and measuring their emergence requires a nuanced approach. These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Our code is available at: https://github.com/eddycmu/demystify-long-cot.

1. Introduction

The paper investigates how long chain-of-thought reasoning emerges and how SFT, RL, reward design, and verifiable data affect its development. It finds that long-CoT SFT supports higher performance and easier RL improvement, while reward shaping and filtered noisy supervision help stabilize and extend reasoning.

  • Motivation: Long CoTs support recognizing and correcting mistakes, decomposing difficult steps, and iterating over alternative approaches for complex reasoning.These behaviors produce longer and more structured reasoning processes.
  • SFT for long CoTs: Long-CoT SFT enables higher performance and easier RL improvement than short-CoT SFT.The paper studies SFT scaling and its impact on subsequent RL.
  • RL-driven CoT scaling: RL does not always stably extend CoT length and complexity, motivating a cosine length-scaling reward with a repetition penalty.The proposed reward stabilizes CoT growth while encouraging branching and backtracking.
  • Verifiable reward signals: Filtered noisy web-extracted solutions show promise for scaling verifiable rewards, especially on out-of-distribution STEM reasoning tasks.The approach combines an appropriate SFT mixture with filtration during RL.
  • Origins of long-CoT abilities: Branching and error validation are present in base models, but effectively incentivizing them through RL requires careful designs and nuanced measurement.The paper traces reasoning patterns in pre-training data and examines RL incentives for long-CoT generation.

2. Problem Formulation

This section formulates long-CoT reasoning and outlines the SFT, RL, reward, model, and evaluation setup used to study it. Long CoT combines extended reasoning-token sequences with behaviors such as branching, backtracking, and error correction.

  • 2. Problem Formulation: The paper’s goal is to demystify long-CoT reasoning through systematic analysis and ablations that yield practical strategies.The stated aim is to enhance and stabilize long-CoT performance.
  • 2.1. Notation: A language model πθ defines a conditional distribution over each output token given the query and preceding tokens.The notation is πθ(yt | x, y1:t−1).
  • 2.1. Notation: CoT(y) denotes the subset of generated output tokens forming a reasoning trace or explanatory sequence.The final answer may be separate or occupy the last part of y.
  • 2.1. Notation: Long CoT means an unusually long reasoning-token sequence that also exhibits sophisticated reasoning behaviors.The paper specifically highlights branching, backtracking, error validation, and correction.
  • 2.2. SFT and 2.3. RL: SFT initializes the policy on normal or long-CoT reasoning tokens, after which RL can further optimize long-CoT generation.SFT is optional before RL in the formulation.
  • 2.3. Reinforcement Learning (RL): The RL reward is outcome-based: it evaluates correctness of the final answer rather than intermediate reasoning steps.The term ranswer(y) captures final-solution correctness.
  • 2.3. Reinforcement Learning (RL): PPO is the default policy optimizer, using a rule-based verifier that directly compares predicted and ground-truth answers.The resulting updates favor tokens that produce higher reward.
  • 2.4. Training Setup and 2.5. Evaluation Setup: Experiments use Llama-3.1-8B and Qwen2.5-7B-Math on MATH training data and evaluate MATH-500, AIME 2024, TheoremQA, and MMLU-Pro-1k.The evaluation spans in-domain and out-of-domain reasoning benchmarks.

3. Impact of SFT on Long CoT

Long-CoT SFT reaches a higher performance ceiling than short-CoT SFT and provides a stronger initialization for subsequent RL. Among long-CoT data sources, emergent patterns generalize better and support further RL gains than constructed patterns.

  • SFT Scaling: Long-CoT SFT continues improving with more tokens, while short-CoT SFT saturates earlier at a lower accuracy.On MATH-500, long-CoT SFT exceeds 70% accuracy without plateauing at 3.5B tokens, whereas short-CoT SFT converges below 55%.
  • SFT Initialization for RL: Long-CoT SFT models can gain over 3% absolute accuracy from RL on MATH-500, while short-CoT SFT models show almost no improvement.The comparison uses long- versus short-CoT SFT checkpoints as RL initialization.
  • Sources of Long CoT SFT Data: RL with the Classic Reward can drive CoT length beyond the context window, causing MATH-500 accuracy to decline toward zero.Figure 2 reports this behavior for both Llama3.1-8B and Qwen2.5-Math-7B.
  • Sources of Long CoT SFT Data: Emergent long-CoT patterns generalize better than constructed patterns and remain amenable to further RL improvement.Constructed-pattern models could not be further improved with RL in the reported comparison.
  • Sources of Long CoT SFT Data: Emergent long-CoT-pattern models improve relatively by 15–50% on the OOD benchmarks AIME 2024 and MMLU-Pro-1k.The models use the same Llama-3.1-8B base model, approximately 200k SFT samples, and the same RL setup.

4. Impact of Reward Design on Long CoT

Reward design strongly influences whether long CoT scaling remains stable and useful. Cosine length shaping with repetition penalties stabilizes growth, improves efficiency, and controls how models allocate inference compute.

  • CoT Length Stability: Under the Classic Reward, both models lengthened their CoTs until reaching the context limit, causing training accuracy to decline.The weaker Llama-3.1-8B showed larger fluctuations than Qwen2.5-Math-7B.
  • CoT Length Stability: Context-window overflow implicitly penalized trajectories, creating downward pressure on the CoT length distribution even without an explicit exceed-length penalty.Reward or advantage normalization can produce this effect in standard RL frameworks.
  • Active Scaling of CoT Length: Cosine reward shaping prioritizes correct CoTs, shorter correct CoTs, and longer wrong CoTs to encourage efficient computation while preserving additional thinking when needed.The reward is sparse and awarded once at the end of the CoT based on answer correctness.
  • Active Scaling of CoT Length: The Cosine Reward significantly stabilized CoT length scaling, training accuracy, and RL efficiency while improving downstream performance.It uses generation length alongside correctness and applies a repetition penalty in the reported experiments.
  • Cosine Reward Hyperparameters: Increasing the correct-answer reward with CoT length caused explosive length growth, while a lower correct-to-wrong reward ratio produced longer CoTs.The authors interpret the ratio as influencing how confident the model must be before terminating with an answer.
  • Reward Hacking and Penalties: With enough compute, models exploited length rewards through repetition, but repetition penalties reduced CoT length and improved downstream performance.When training accuracy was low, stronger upward length pressure increased repetition-based reward hacking.
  • Optimal Discount Factors: Different reward types require different discount factors: lower factors strengthen repetition penalties, whereas higher factors strengthen correctness and exceed-length penalties.A higher correctness-reward discount factor better rewards selecting a correct answer earlier in the CoT.

5. Scaling up Verifiable Reward

Scaling verifiable rewards is constrained by limited high-quality annotated data, motivating experiments with noisy web-extracted supervision. Filtering and mixing strategies make these signals useful, especially for out-of-distribution reasoning benchmarks.

  • Motivation: Noisy web-extracted reasoning data offers a larger but less reliable alternative to scarce high-quality human-annotated verifiable data.The paper studies WebInstruct as a source of silver supervision signals.
  • SFT with Noisy Verifiable Data: Adding WebInstruct to long-CoT SFT produced a 510% absolute accuracy gain on MMLU-Pro-1k over using MATH alone.Mixing MATH and WebInstruct achieved the best average accuracy across benchmarks.
  • SFT with Noisy Verifiable Data: Adding noisy but diverse data to SFT led to more balanced performance across different tasks.The experiments varied the proportion of data without gold supervision signals between 0%, 100%, and approximately 50%.
  • RL with Noisy Verifiable Data: The study compares rule-based verification of extracted short-form answers with model-based verification of free-form responses.It also evaluates whether filtering the dataset to retain short-form-answer samples improves reward quality.
  • RL with Noisy Verifiable Data: Under the same number of RL samples, filtered prompts with short-form answers and a rule-based verifier achieved the best performance across most benchmarks.Compared with human-annotated MATH data, noisy verifiable data produced OOD gains up to 2.9% on TheoremQA and 6.8% on MMLU-Pro-1k; unfiltered rule-based verification performed worst.

6. Exploration on RL from the Base Model

RL from Qwen2.5-Math-7B improves accuracy but does not reliably produce reflection or longer-than-base CoT behavior. The analysis suggests that long-CoT SFT initialization is substantially more effective, while model size and training-data exposure may constrain base-model RL.

  • 6.1. Nuances in Analysis Based on Emergent Behaviors: RL improved accuracy without substantially increasing reflection patterns such as recheck, retry, or alternatively.The authors caution that behaviors sometimes present in the base model should not automatically be treated as emergent.
  • 6.2. Nuances in Analysis Based on Length Scaling: Output length rose after an initial drop but never exceeded the base model’s initial length.The coding rate later increased, while the initial length drop occurred for both coding and natural-language outputs.
  • 6.2. Nuances in Analysis Based on Length Scaling: Length scaling coincided with decreasing KL divergence, suggesting that KL regularization may influence length rather than exploration alone.If the KL constraint limits exploration, policy output length may have little potential to exceed the base model’s.
  • 6.3. Potential Reasons Why Emergent Behavior is Not Observed with Qwen2.5-Math-7B: The authors identify the 7B base model’s capacity and possible overexposure to MATH-like short-instruction data as potential causes of incomplete DeepSeek-R1-like training behavior.These are presented as potential causes rather than established explanations.
  • 6.4. Comparison between RL from the Base Model and RL from Long CoT SFT: RL initialized from long-CoT SFT outperformed RL from the base model by 8.7% on average and improved over SFT initialization by 2.6%.Long-CoT SFT itself already yielded strong performance when distilled from QwQ-32B-Preview.

7. Discussions and Future Work

The paper identifies model scale, RL infrastructure, verification design, and data origins as important directions for advancing long-CoT reasoning. It also reports instability in an alternative RL algorithm and frames latent reasoning as potentially traceable to pre-training data.

  • 7.1. Scaling up Model Size: Model size is proposed as a primary factor limiting emergence of the behavior observed with Qwen2.5-Math-7B.The authors suggest investigating RL with a larger base model.
  • 7.2. RL Infrastructure Is Still in Its Infancy: Scaling to 32B was blocked by GPU requirements, while parameter copies, synchronous PPO workloads, and variable CoT lengths reduce hardware utilization.Long-CoT variance creates inference stragglers that further worsen efficiency.
  • 7.3. Scaling up RL Training: REINFORCE++ was less stable than PPO in the tested setup and produced lower training accuracies.The authors avoid generalizing because the setup may not have been tuned.
  • 7.4. Scaling up Verification: Rule-based verification and prompt filtering were effective but labor-intensive across domains and dependent on human-designed heuristics.The paper motivates future work on scalable silver-supervision and self-supervised verification signals.
  • 7.5. Tracing Reasoning Behaviors to Data Origins: Human dialogue on Internet discussion forums is proposed as one possible source of latent reasoning behaviors in pre-training data.The authors call for more detailed tracing of model behaviors to their data origins.

Impact Statement

The paper examines training strategies for long chain-of-thought reasoning and emphasizes their potential for interpretable reasoning and problem-solving across domains. It also highlights the importance of stable, verifiable training and careful reward design.

  • Impact Statement: The paper studies how scaling inference compute and training strategies can enable long chain-of-thought reasoning in LLMs.Long CoTs are associated with breaking difficult problems into finer steps and reflecting during problem solving.
  • Impact Statement: The paper recommends robust data preparation, training stability, verifiable ground truths, and safeguards for responsible deployment.These considerations are presented as part of minimizing unintended consequences while developing reasoning capabilities.
  • Impact Statement: Long CoTs support reasoning behaviors such as recognizing mistakes, breaking down difficult steps, and iterating over alternatives.The paper connects these behaviors with inference-time scaling through longer reasoning processes.
  • Impact Statement: The paper identifies reward hacking and training instability as concerns when increasing training compute or using reinforcement learning.The supplied passages associate increased repetition with reward hacking and report instability under classic reward.
  • Impact Statement: Reward design choices affect CoT branching and response length, with lower discount factors increasing branching and repetition penalties reducing length.Different correctness-reward and repetition-penalty settings also have different optimal values.

E.1. Evaluation Setup

The evaluation uses in-domain and out-of-domain reasoning benchmarks, multiple random seeds, and task-specific sampling and grading procedures. The setup also records implementation choices and an earlier grader version in some RL experiments.

  • E.1. Evaluation Setup: The evaluation covers MATH-500, AIME 2024, TheoremQA, and MMLU-Pro-1k as in-domain and out-of-domain reasoning benchmarks.MATH-500 is in-domain, while AIME 2024, TheoremQA, and MMLU-Pro-1k are out-of-domain evaluations.
  • E.1. Evaluation Setup: MATH-500 uses an i.i.d. test subset, AIME 2024 contains 30 problems, TheoremQA contains 800 samples, and MMLU-Pro-1k contains 1,000 test samples.TheoremQA spans more than 350 theorems across mathematics, engineering, computer science, physics, and finance.
  • E.1. Evaluation Setup: Average accuracy is computed with at least 4 random seeds, while AIME 2024 uses 16 responses per prompt to reduce small-sample variance.The evaluation therefore uses additional response sampling for the smaller AIME benchmark.
  • E.1. Evaluation Setup: The implementation uses vLLM for accelerated inference and SymEval2 for grading complex mathematical objects.A few RL experiments used an earlier grader version, producing nuanced performance differences.

E.5. Experiment Hyperparameters

The experiment hyperparameter section specifies shared batching conventions and the datasets used for SFT, including long-CoT data distilled from QwQ-32B-Preview and WebInstruct.

  • E.5. Experiment Hyperparameters: The BS column denotes both rollout and training batch size in most RL setups, while Samples denotes samples per prompt.The two batch-size quantities use the same number in most reported RL configurations.
  • E.5. Experiment Hyperparameters: SFT data consists of CoT responses distilled from QwQ-32B-Preview or Qwen2.5-Math-72B-Instruct using the MATH training split.The data varies by the number of candidate responses sampled per prompt.

E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA)

The section describes noisy-data training and the prompting machinery used to construct long CoTs. It combines filtered WebInstruct data with action-based prompting for clarification, decomposition, solution steps, reflection, and final answers.

  • E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA): The noisy long-CoT dataset contains 115k instances filtered from 462k WebInstruct examples distilled from QwQ-32B-Preview.The filtering step is part of the SFT data used for noisy verifiable-data experiments.
  • E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA): A model-based verifier receives the reference answer and the last 20 lines of a long CoT, then judges correctness from the final answer.The verifier is Qwen2.5-7B-Instruct, and the long CoT is truncated to avoid confusing it.
  • E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA): The action-prompting framework sequences clarify, decompose, solution step, reflection, and answer actions to construct long CoTs.The LLM controls branching or looping decisions while the framework reacts as a state machine.
  • E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA): The state machine initializes a chain with the prompt, begins in the clarify state, parses each action output, and appends the resulting thought.It continues through action states while maintaining the chain-of-thought sequence.
  • E.5.8. DETAILS OF SECTION 5.2 (RL WITH NOISY VERIFIABLE DATA): Prompt templates require subgoal generation, alternative proposals, concise solution steps, and verification of whether current and parent goals were achieved.The reflection prompt can propose a new alternative goal when the current solution is incorrect or incomplete.

F. Long CoT Patterns in Pre-training Data

Pre-training data contains examples of explicit verification, alternative solution paths, self-correction, branching, uncertainty, and feasibility assessment. These patterns appear across mathematical discussions and probabilistic-programming exchanges.

  • Verification: Web examples show explicit verification by substituting a proposed solution back into the original equation.A correct substitution yields 10 = 10, while an incorrect one yields 9 ≠ 10 and prompts another solution attempt.
  • Alternative approaches: The “double check game” solves the same arithmetic problem through multiple decomposition and subtraction strategies.The passage gives three approaches to 78 − 57, including breaking apart 57 and subtracting 60 before adding 3.
  • Corpus identification: GPT-4o-generated pivot keywords were used to locate OpenWebMath documents with properties characteristic of long CoT trajectories.The keywords include stepwise reasoning, checking correctness, detecting missing elements, and reconsidering whether an approach works.
  • Discussion patterns: A message-board discussion exhibits branching, self-correction, and feasibility assessment as characteristic long-CoT patterns.The discussion is framed as evidence that such patterns occur in naturally occurring web text.
  • Discussion patterns: A physics-forum exchange contains self-reflection, uncertainty, and hypothesis formulation while working through a cylinder cross-section problem.The user revisits the setup, asks whether the geometric interpretation is correct, and reassesses the hint.
  • Discussion patterns: A probability discussion combines rapid consideration of alternative solutions with an explicit realization that supports self-verification.The exchange distinguishes a stopping-time calculation from the probability of exactly three heads and identifies a switched probability assignment.
Loading 2502.03373v1…