Source-linked AI summary
Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Ji-Rong Wen
TL;DR
The paper addresses the challenge of reproducing broadly capable o1-like reasoning systems whose technical details are largely undisclosed. It introduces STILL-2, which imitates distilled long-form thought, explores difficult problems, and self-improves through iterative dataset refinement, achieving competitive results across three benchmarks. The authors also identify scarcity of challenging problems with ground-truth answers as a limitation.
Problem
Existing reproductions are often domain-limited or built on weak base models, while key industry-system details remain undisclosed.
Method
STILL-2 trains reasoning through imitation of distilled long-form thought, exploration of challenging problems, and self-improvement via iteratively refined training data.
Results
STILL-2 demonstrates strong performance across MATH-OAI, AIME2024, and GPQA, with distillation-based variants approaching industry counterparts.
Takeaways & Limitations
Long-form thinking can be elicited with a small amount of high-quality data, transfer across domains, and improve through exploration and self-improvement.
Takeaways & Limitations
The approach is constrained by the scarcity of challenging problems paired with ground-truth answers, which can quickly exhaust the training pool.
Abstract
from arXiv · showhide
Recently, slow-thinking reasoning systems, such as o1, have demonstrated remarkable capabilities in solving complex reasoning tasks. These systems typically engage in an extended thinking process before responding to a query, allowing them to generate more thorough, accurate, and well-reasoned solutions. These systems are primarily developed and maintained by industry, with their core techniques not publicly disclosed. In response, an increasing number of studies from the research community aim to explore the technical foundations underlying these powerful reasoning systems. Building on these prior efforts, this paper presents a reproduction report on implementing o1-like reasoning systems. We introduce an ``imitate, explore, and self-improve'' framework, denoted as \textbf{STILL-2}, as our primary technical approach to train the reasoning model. In the initial phase, we use distilled long-form thought data to fine-tune the reasoning model, enabling it to invoke a slow-thinking mode. The model is then encouraged to explore challenging problems by generating multiple rollouts, which can result in increasingly more high-quality trajectories that lead to correct answers. Furthermore, the model undergoes self-improvement by iteratively refining its training dataset. To verify the effectiveness of this approach, we conduct extensive experiments on three challenging benchmarks. The experimental results demonstrate that our approach achieves competitive performance compared to industry-level reasoning systems on these benchmarks.
1 Introduction
The paper targets the gap between publicly explored o1-like systems and industry-level reasoning systems by introducing STILL-2, an imitate, explore, and self-improve framework. It reports competitive benchmark performance using distilled thought data and iterative refinement.
- Open o1-like systems remain limited by domain specificity, weak base models, and undisclosed implementation details, hindering broadly capable reproduction.
- The prior framework faced poor cross-domain reward-model generalization, costly inference-time tree search, and no train-time scaling.
- STILL-2 imitates long-form thought, explores difficult problems through multiple rollouts, and self-improves by refining its training data.
- The approach uses distilled responses as demonstrations to elicit slow-thinking behavior and align outputs with thought-and-solution formatting.
- Using 3,900 demonstration instances, the distillation variant approaches some industry-level systems, while 1,100 seed instances support promising exploration and self-improvement results.
2 Method
The method section introduces STILL-2 as the paper’s implemented o1-like reasoning system and presents an illustrative training-pipeline overview.
- STILL-2 is the implemented reasoning system introduced in the paper.
- Figure 1 provides an illustrative overview of the STILL-2 training pipeline.
2.1 Overview
STILL-2 trains reasoning through three phases: imitate long-form thought, explore difficult tasks, and self-improve through refined training data. The framework remains partly conceptual in implementation.
- Overview: STILL-2 uses a three-phase training approach—imitate, explore, and self-improve—to develop o1-like reasoning systems.
- Imitate: Imitation teaches the model to generate internal thought and a final solution in one response, using formatting support for slow-thinking outputs.
- Overview: The paper defines “o1-like” systems as those that conduct extensive reasoning before producing the final solution.
- Explore: Exploration is intended to expand the model’s ability to use long-form thought on complex tasks beyond imitation alone.
- Overview: The framework is conceptual, and the reported implementation does not fully realize its potential.
2.2 Imitation Learning for Slow-Thinking Reasoning
The imitation stage constructs and filters distilled long-form thought demonstrations, unifies thought-and-solution formatting, and fine-tunes a base model to reproduce slow-thinking behavior across domains.
- Dataset and training: Imitation learning trains the model to produce extended thought before a solution by using high-quality long-form thought demonstrations.
- Data collection: The authors distill long-form thought data from R1 and QwQ because of simplicity and budget constraints.
- Format unification: R1 separates thought and solution components, whereas QwQ outputs the answer after thought; the authors unify these formats with explicit boundary tokens.
- Dataset construction: Demonstrations mix mathematics, coding, science, and puzzles, emphasizing challenging problems to support cross-domain reasoning.
- Preprocessing and fine-tuning: Rule-based preprocessing removes repetitions, gibberish, and language mixtures before supervised fine-tuning.
- Observed effects: Long-form thinking appears transferable across domains, and challenging problems are especially useful for improving model performance.
2.3 Exploration and Self-Improvement
The framework generates and filters correct trajectories from challenging problems, then iteratively adds them to the training data for self-improvement. Experiments use supervised fine-tuning and direct preference optimization, while reinforcement learning is left for future work because of computational constraints.
- Exploration and Self-Improvement: The approach addresses the cost of producing long-form thought data by having models explore challenging problems and generate additional trajectories for training.The process generates rollouts, identifies suitable trajectories, and uses them to enhance reasoning abilities.
- Exploration and Self-Improvement: Multiple rollouts are generated for each problem until a trajectory containing the correct answer is produced, with ground-truth answers replacing a trainable reward model.Scaling rollouts increases opportunities to collect correct thought-and-solution trajectories.
- Exploration on Hard Problems: Challenging problems improve performance more reliably than simple problems, although the scarcity of hard problems keeps the training set relatively small.The paper also reports that slow-thinking behavior can transfer across domains, even when training uses only mathematical problems.
- Iteratively Refined Training Data: Iterative refinement alternates between training a stronger model, generating new trajectories, and adding them to the dataset, while filtering short, noisy, or otherwise low-quality data.Perplexity is used to retain more challenging trajectories recognized by the current model.
- Optimization for Self-improvement: Self-improvement uses supervised fine-tuning and direct preference optimization on refined datasets; reinforcement learning is deferred because of computational resource constraints.The DPO procedure pairs correct and incorrect responses for contrastive learning.
3 Experiments
Experiments evaluate STILL-2 on three challenging benchmarks and compare distillation, exploration, self-improvement, data mixtures, and DPO variants. Distillation with 3.9k instances performs best among the authors’ attempts, while exploration and self-improvement improve over the backbone but show limited, fluctuating gains.
- Experimental Setup: STILL-2 is evaluated on MATH-OAI, AIME2024, and GPQA using Qwen2.5-32B-Instruct as the backbone and comparisons with o1-like and general-purpose models.The evaluation reports accuracy and gain relative to the backbone’s performance.
- Main Results: 46.7% AIME accuracy and 90.2% MATH-OAI accuracy are achieved by the 3.9k-instance distillation variant, approaching industry counterparts.Careful cleaning, selection, and mixing of demonstration instances are reported as useful for this variant.
- Main Results: Increasing high-quality demonstration data from 1.1k to 3.9k instances improves model performance in the distillation-based variants.The comparison is explicitly made between models trained with 1.1k and 3.9k instances.
- Main Results: 33.3% to 40.0%, 46.7%, and 40.0% AIME accuracy is reported for variants incorporating exploration and self-improvement relative to w/ SFT 1.1k.The explored instances are drawn from a global pool of correct trajectories collected across multiple runs, and more explored instances improve performance to some extent.
- Limitations: Iterative-training improvements are often limited to initial iterations and may fluctuate across benchmarks, while DPO results also fluctuate across runs.The authors recommend more thorough experiments on alignment strategies and plan to extend exploration time.
- Further Analysis of Data Mixture: Excluding hard problems causes a significant performance drop, while mathematical data alone performs strongly across all three benchmarks.The authors relate hard problems to longer thought processes and report that math-only data is not limited to the math domain.
- Further Analysis of DPO Training: Aligning only the thought process yields promising DPO results, while adding SFT loss has little positive influence when thought and solution are both aligned.The authors attribute the latter observation to the solution already being incorporated into DPO training.
4 Conclusion
STILL-2 reproduces an o1-like reasoning system through distilled long-form thought training followed by exploration and self-improvement. The system performs strongly on three challenging benchmarks, but the authors describe the work as preliminary and note a substantial gap from industry-level systems.
- STILL-2 uses a three-phase development approach: training on distilled long-form thought data, exploring difficult problems, and self-improving through the resulting data.
- The slow-thinking mode transfers across domains and is particularly effective for hard, complex problems.
- Long-form thinking can be elicited with a small amount of high-quality demonstration data and appears to generalize across domains.
- Math demonstrations and longer thought processes are especially effective for developing long-form thinking and tackling challenging problems.
- Exploration and self-improvement enhance slow-thinking capacity, while offline-learning gains occur mainly in initial iterations, especially on challenging tasks.
- The authors characterize the exploration as preliminary, acknowledge a substantial capacity gap from industry-level systems, and plan to scale training for more complex tasks.