Source-linked AI summary
LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, Ion Stoica
TL;DR
The paper addresses how to elicit Long CoT reasoning without relying on poorly understood, closed, or expensive training procedures. It fine-tunes an LLM on distilled reasoning traces and studies whether structure or step content drives learning. The results show strong gains from limited data and parameter updates, with logical disruption more damaging than content corruption.
Problem
How to elicit Long CoT reasoning efficiently remains poorly understood because existing methods are closed-source or expensive to replicate.
Method
The paper distills Long CoT traces into supervised fine-tuning and LoRA experiments, then perturbs reasoning content and global structure to compare their effects.
Results
17k Long CoT samples substantially improve Qwen2.5-32B-Instruct across math and coding benchmarks, while structural perturbations degrade accuracy more than content perturbations.
Takeaways & Limitations
Logical structure is the key characteristic for learning Long CoT reasoning, rather than the specific content of individual reasoning steps.
Takeaways & Limitations
Best-of-N performance continues improving beyond 128 samples, leaving further gains possible through context scaling or broader reasoning paths.
Abstract
from arXiv · showhide
Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT) that incorporate reflection, backtracking, and self-validation. However, the training techniques and data requirements to elicit Long CoT remain poorly understood. In this work, we find that a Large Language model (LLM) can effectively learn Long CoT reasoning through data-efficient supervised fine-tuning (SFT) and parameter-efficient low-rank adaptation (LoRA). With just 17k long CoT training samples, the Qwen2.5-32B-Instruct model achieves significant improvements on a wide range of math and coding benchmarks, including 56.7% (+40.0%) on AIME 2024 and 57.0% (+8.1%) on LiveCodeBench, competitive to the proprietary o1-preview model's score of 44.6% and 59.1%. More importantly, we find that the structure of Long CoT is critical to the learning process, whereas the content of individual reasoning steps has minimal impact. Perturbations affecting content, such as training on incorrect samples or removing reasoning keywords, have little impact on performance. In contrast, structural modifications that disrupt logical consistency in the Long CoT, such as shuffling or deleting reasoning steps, significantly degrade accuracy. For example, a model trained on Long CoT samples with incorrect answers still achieves only 3.2% lower accuracy compared to training with fully correct samples. These insights deepen our understanding of how to elicit reasoning capabilities in LLMs and highlight key considerations for efficiently training the next generation of reasoning models. This is the academic paper of our previous released Sky-T1-32B-Preview model. Codes are available at https://github.com/NovaSky-AI/SkyThought.
1. Introduction
The paper asks how to elicit Long CoT reasoning efficiently and finds that both content perturbations and structural coherence matter differently. Small-scale fine-tuning substantially improves math and coding performance, while disrupting reasoning structure causes larger degradation than altering step content.
- Long CoT elicitation remains poorly understood because existing methods are closed-source or expensive to replicate.
- 17k Long CoT samples enable Qwen2.5-32B-Instruct to improve across five math and coding benchmarks through SFT or LoRA.The base model learns reflection, backtracking, and self-validation patterns.
- The controlled studies compare perturbations to individual-step content with modifications to overall reasoning structure.
- 50% randomized number changes reduce AIME 2024 accuracy by only 3.3% relative to training with correct samples.
- 67% reasoning-step shuffling reduces AIME 2024 accuracy by 13.3% relative to training with correctly ordered samples.Structural perturbations break logical coherence across the Long CoT.
1. We demonstrate that an LLM can learn Long CoT
The paper demonstrates that LLMs can learn Long CoT reasoning efficiently through limited data and parameter updates. It identifies logical structure, rather than individual-step content, as central and evaluates this finding broadly across models, datasets, and benchmarks.
- Fewer than 17k samples and LoRA updates enable Qwen2.5-32B-Instruct to become competitive with o1-preview.
- Long CoT structure is critical to learning, whereas the content of individual reasoning steps is less important.
- The study conducts ablations across model sizes, architectures, dataset sizes, generation models, and five math and coding benchmarks.
2. Related work
Related work frames reasoning improvement through test-time scaling, iterative self-improvement, reinforcement learning, and distillation. These approaches include both generating longer individual reasoning traces and sampling multiple reasoning paths.
- Test Time Scaling for Large Language Models: Test-time scaling improves reasoning through single long CoT responses or repeated CoT sampling methods such as Best-of-N and search-guided generation.
- Training to improve reasoning capabilities of LLMs: LLM reasoning can be improved through iterative self-improvement and reinforcement-learning methods, including RLVR and process-label-free approaches.
- Distillation: Distillation commonly uses outputs or logits from stronger models to supervise smaller models and enhance their capabilities.
3. Simple distillation is effective
The paper uses curated reasoning traces from capable open models to test whether small, efficient fine-tuning can improve reasoning. Results support data and parameter efficiency, while comparisons indicate that Long CoT fine-tuning can match modest test-time sampling gains.
- Simple distillation is effective: A small amount of curated data and LoRA can improve reasoning capabilities in a large language model.
- Experiments Setup: Distillation data combines DeepSeek-R1 and QwQ-32B-Preview traces from difficult math and coding prompts, with correctness verification.
- Experiments Setup: The evaluation covers Math500, OlympiadBench, AIME 2024, AMC 2023, and LiveCodeBench.
- Simple distillation is effective: 17k training samples improve both SFT and LoRA models significantly across all evaluated benchmarks.
- LoRA fine-tuning without performance degradation: LoRA models achieve similar or superior reasoning performance to full-parameter SFT, including with only 7k QwQ samples.The result supports parameter-efficient and data-efficient reasoning distillation.
4. Long CoT: Structure Is The Key
The studies compare local content perturbations with global structural perturbations in Long CoT demonstrations. Local inaccuracies are tolerated, whereas breaking logical continuity across reasoning steps substantially harms accuracy.
- Local Content Perturbations: 63.1% average accuracy from samples with wrong final answers was only 3.2% below training with entirely correct samples.This tests whether correct final answers are necessary for learning reasoning patterns.
- Local Content Perturbations: 62% average performance remained after corrupting 70% of digits, only 4.3% below the correct-sample baseline.Complete digit corruption instead caused average performance to plunge to 2.7%.
- Local Content Perturbations: Removing 100% of reasoning keywords still produced 63% average accuracy, within 3.3% of training with correct samples.The keywords tested included phrases associated with reflection and backtracking.
- Comparing Local Content and Global Structure: Structural perturbations are far more detrimental to model accuracy than changes to reasoning-step content.The study compares wrong answers, corrupted digits, removed keywords, deleted steps, inserted steps, and shuffled steps.
- Global Structural Perturbations: Deleting, inserting, or shuffling reasoning steps breaks logical consistency and degrades accuracy.Shuffled traces can retain valid domain-specific steps while producing incoherent overall solutions.
- Global Structural Perturbations: At 67% deletion, keyword usage increased 45× and output tokens 9× relative to the base model, but accuracy did not improve accordingly.The result indicates that long responses and keyword imitation alone are insufficient.
5. Ablation Study
The ablations examine whether Long CoT fine-tuning preserves general capabilities, transfers across student models, compares favorably with Best-of-N, and outperforms Short CoT training. Across these settings, Long CoT generally improves reasoning while retaining instruction-following and multilingual performance.
- 5.1. Performance on Non-Reasoning Benchmarks: Long CoT fine-tuning is evaluated for catastrophic forgetting and general language understanding using MMLU, ARC-C, IEval, and MGSM.These benchmarks cover multitask understanding, science, instruction-following, and multilingual mathematics.
- 5.1. Performance on Non-Reasoning Benchmarks: QwQ reaches 42.5% on IEval and 19.1% on MGSM, whereas SFT and LoRA retain most of the base instruction model’s capabilities.The distilled models avoid the drastic degradation observed in QwQ on instruction-following and multilingual tasks.
- 5.2. Effect on Different Student Models: Seven out of eight tested student models improve noticeably across multiple benchmarks after training on 17k Long CoT samples.The models span different sizes and architectures, including Qwen, Llama, and Gemma variants.
- 5.3. Comparison to Best-of-N: Long CoT fine-tuning performs comparably to Best-of-N sampling with 2 to 16 instances across student models.The comparison uses an oracle verifier and 128 generated samples per prompt for Best-of-N evaluation.
- 5.3. Comparison to Best-of-N: Best-of-N performance continues improving beyond 128 samples, indicating that the reported comparison does not exhaust test-time scaling gains.The authors suggest context scaling or broader reasoning paths as possible ways to enhance Long CoT models.
- 5.4. Comparison to Short CoT Fine-tuning: Compared with Short CoT training, Long CoT produces longer responses, uses more reasoning keywords, and improves performance on harder problems.The comparison uses 4.6k QwQ-32B-Preview Long CoT samples and Short CoT data from NuminaMath-CoT.
6. Conclusion
The paper concludes that strong reasoning capabilities can be fine-tuned with few thousand examples and low-rank adapters, and that Long CoT structure matters more than individual step content.
- 6. Conclusion: Long CoT reasoning can be fine-tuned using a few thousand examples and low-rank adapters.The conclusion frames this as an accessible route to instilling stronger reasoning capabilities in language models.
- 6. Conclusion: The logical structure of Long CoT samples is more important to learning than the content of individual reasoning steps.The paper supports this conclusion through controlled structural and content perturbation studies.
- 6. Conclusion: The paper also reports ablations across teacher-student models and comparisons with Best-of-N, identifying potential directions for future work.These experiments contribute to the paper’s broader analysis of what is needed to instill reasoning capabilities.
Impact Statement
The paper states that it aims to advance machine learning and identifies no societal implications requiring specific emphasis. The supplied passages also include a logarithmic integer-counting example and its attempted solution.
- Impact Statement: The authors state that no societal implications currently require specific emphasis.The statement is framed within the paper’s broader aim of contributing to machine-learning advancement.
- Example: The accompanying example asks how many integer values make a logarithmic square-root expression real.The logarithm is specified as base 10.
- Example: The attempted solution imposes positivity and nonnegativity conditions, then reports contradictory ranges during combination.It first derives n > 1000 and later reports 1 ≤ n ≤ 100 before acknowledging a misinterpretation.
6. Conclusion The integers n that satisfy the given conditions are those for which
The expression is real for n = 1 and for integers 100 ≤ n < 1000, giving 901 acceptable integers.
- n = 1000 is excluded because the denominator is zero and the expression is undefined.
- The expression is acceptable at n = 1 and n = 100 because its value is zero.
- The expression is negative for 1 < n < 100 and n > 1000, so those integers are excluded.
- 901 integers satisfy the condition: n = 1 contributes one value, while 100 ≤ n < 1000 contributes 900 values.The interval contains integers 100 through 999 inclusive.
- The analysis uses x = log n, identifies critical points at n = 1, 100, and 1000, and tests the resulting intervals.
B. Reasoning keywords
The reasoning-keyword procedure identifies verbal markers associated with reflection, backtracking, and self-validation, then incorporates them into a structured long-thinking prompt.
- The keyword bank was created by manually inspecting reasoning responses from QwQ and R1.
- The identified markers include “Alternatively,” “Wait,” “Just to be thorough,” and “Let me verify.”The list also includes phrases expressing checking, reconsideration, and alternative approaches.
- The adopted system prompt requires a systematic long thinking process before presenting precise solutions.
- The prompt explicitly requests analysis, summarization, exploration, reassessment, reflection, backtracking, and iteration.
- Responses are structured into Thought and Solution sections, with thought steps separated by blank lines.
D. Average response lengths and keyword counts
Three tables report average keyword counts and output-token lengths for samples with deleted, inserted, or shuffled reasoning steps.
- Table 5 reports average keyword counts and output tokens for deleted reasoning steps.
- Table 6 reports average keyword counts and output tokens for inserted reasoning steps.
- Table 7 reports average keyword counts and output tokens for shuffled reasoning steps.