PaperlayerView exact revision on arXiv

Source-linked AI summary

O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?

Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, Pengfei Liu

arXiv:2411.16489v1cs.CLcs.AI

TL;DR

The paper examines concerns that undisclosed O1 distillation complicates assessment of replication claims and may discourage fundamental innovation. It tests API-based distillation with supervised fine-tuning and reports gains on AIME and open-domain QA, while identifying limits in safety alignment and mathematical performance.

  • Problem

    The authors identify undisclosed distillation in O1 replication as a transparency concern that makes it harder for researchers to assess and build on claimed advances.

    Source: §1 · p. 2

  • Method

    The paper prompts O1’s API to generate long-thought chains, uses them for supervised fine-tuning, and introduces a framework to evaluate replication transparency and reproducibility.

    Source: §1 · p. 2

  • Results

    The distilled model surpassed O1-preview on AIME, while fine-tuning improved open-domain QA scores and a reported case showed more systematic safety-focused reasoning.

    Source: §1 · p. 2, §4.1 · p. 6, §4.3 · p. 9

  • Takeaways & Limitations

    The authors advocate transparent reporting and balancing performance gains with fundamental technical innovation.

    Source: §1 · p. 2

    WildSafety performance fell from 92% to 86.5%, suggesting systematic thinking alone is insufficient for comprehensive safety alignment.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities, with particular focus on the widespread but often undisclosed use of knowledge distillation techniques. While our previous work explored the fundamental technical path to O1 replication, this study reveals how simple distillation from O1's API, combined with supervised fine-tuning, can achieve superior performance on complex mathematical reasoning tasks. Through extensive experiments, we show that a base model fine-tuned on simply tens of thousands of samples O1-distilled long-thought chains outperforms O1-preview on the American Invitational Mathematics Examination (AIME) with minimal technical complexity. Moreover, our investigation extends beyond mathematical reasoning to explore the generalization capabilities of O1-distilled models across diverse tasks: hallucination, safety and open-domain QA. Notably, despite training only on mathematical problem-solving data, our models demonstrated strong generalization to open-ended QA tasks and became significantly less susceptible to sycophancy after fine-tuning. We deliberately make this finding public to promote transparency in AI research and to challenge the current trend of obscured technical claims in the field. Our work includes: (1) A detailed technical exposition of the distillation process and its effectiveness, (2) A comprehensive benchmark framework for evaluating and categorizing O1 replication attempts based on their technical transparency and reproducibility, (3) A critical discussion of the limitations and potential risks of over-relying on distillation approaches, our analysis culminates in a crucial bitter lesson: while the pursuit of more capable AI systems is important, the development of researchers grounded in first-principles thinking is paramount.

1 Introduction

The paper argues that undisclosed O1 distillation makes replication claims harder to assess and risks shifting research away from fundamental innovation. It presents API-based distillation and a framework for evaluating replication transparency and reproducibility.

1 Introduction

  • With tens of thousands of O1-API-distilled samples and standard supervised fine-tuning, a base model surpasses O1-preview on AIME.The approach prompts O1 with complex problems to generate long-thought chains for training other models.Source: §1 · p. 2
  • Undisclosed distillation makes it harder to assess and build on claimed advances, risks reliance on existing models over new techniques, and limits opportunities to develop researchers’ problem-solving skills.The authors also argue that distilled models are bounded by their teacher’s capabilities.Source: §1 · p. 2
  • The paper introduces a benchmark framework for categorizing and evaluating O1 replication attempts by technical transparency and reproducibility.The authors argue for balancing performance gains with genuine technical innovation and transparent reporting.Source: §1 · p. 2

2 The “Shortcut” Path to O1 Replication

The paper contrasts journey learning and other ways to produce long-thought data, then describes distillation and fine-tuning as an alternative replication path. Its account covers data preparation and a two-phase supervised fine-tuning process.

2.1 Core Technical Stack for O1 Replication

  • Journey learning uses tree search to explore solution paths, then LLM reflection to correct errors and produce trajectories for fine-tuning.The tuned models can be used in later training iterations.Source: §2.1 · p. 2

2.2 Alternative Methods for Long-thought Synthesis

  • Long-thought synthesis alternatives include human thought-process annotation, multi-agent exploration, and advanced-model distillation, with distillation described as cost-effective and reliable.Tree search can be effective but computationally expensive and time-consuming; multi-agent debate pairs continuous reasoning with critique.Source: §2.2 · p. 3

2.3 Distillation-based Long Thought Synthesis

  • The distillation preparation filters out image-dependent, unlabeled-answer, and proof-based problems, then rewrites solutions as detailed, step-by-step responses with boxed answers.The authors motivate high-quality data and select Qwen2.5-Math-72B as the base model.Source: §2.3 · pp. 3–4

2.3.1 Supervised fine-tuning approach

  • Supervised fine-tuning first adapts the model to the long-thought format using refined data, then trains it on curated O1-distilled reasoning data.The phases target detailed reasoning and standardized output first, then precise and coherent long-thought responses.Source: §2.3.1 · p. 4

3 Experiment

The experiments evaluate mathematical reasoning benchmarks and performance across inference-time compute costs. Under comparable average output-token costs, the distilled model surpasses O1-preview on AIME2024, though a gap to O1-mini and solution imperfections remain.

3.1 Benchmark Usage

  • Evaluation uses MATH500, AIME2024, and 30 problems from the 2024 China National High School Mathematics Competition.The newly released AIME problems were chosen to minimize data-leakage risk.Source: §3.1 · p. 4

3.2 Evaluation Metric for Inference-Time Scaling

  • Average output-token count serves as a proxy for inference-time computational cost, with longer outputs indicating more extensive reasoning.Maj@k approximates performance at higher average token counts when a single output does not reach the target, without an extra reward model.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 Performance Analysis

  • Under similar reasoning computational costs, the distilled model surpasses O1-preview on AIME2024.Table 1 compares the distilled O1-mini model with O1-series models on AIME2024 and MATH500 under inference-cost constraints.Source: Table 1 · p. 5, §3.3 · p. 5
  • The model still trails O1-mini in mathematical reasoning, and its generated long-thought solutions remain imperfect.The authors identify closing the performance gap and improving solution clarity and correctness as remaining needs.Source: §3.3 · p. 5

4 Application Beyond Math Reasoning

The section tests whether math-focused long-thought SFT transfers beyond mathematical reasoning, reporting gains on general QA and sycophancy but mixed safety results and no significant factuality improvement.

4.3 General Scenario

  • Scores rose from 81.6% to 88% on Auto-J queries and from 77.2% to 87.2% on LIMA queries after SFT on math problem-solving data.The authors report improvements in open-domain QA, particularly for long-term planning and structured thinking.Source: Table 2 · p. 6, §4.3 · p. 9

4.1 Safety

  • Overall safety fell from 94.3% to 93.0% after fine-tuning; Flames rose from 91% to 92.5%, DiaSafety stayed at 100%, and WildSafety fell from 92% to 86.5%.The authors caution that reflective thinking without explicit safety alignment may not sustain safety across diverse scenarios.Source: Table 2 · p. 6, §4.1 · p. 6, §4.2 · p. 7
  • In the electric-bicycle corridor case, the fine-tuned model considered fire hazards, regulations, and residents' safety alongside the user's theft concern.The baseline focused on anti-theft measures and overlooked corridor-charging fire risks.Source: §4.1 · p. 6

4.2 Hallucination

  • SFT did not significantly improve factuality, as longer reasoning chains sometimes included fabricated search results.The model's search-engine interactions were simulated rather than connected to external databases.Source: §4.2 · p. 7
  • Sycophancy evaluation improved from 89.70% to 92.65% after SFT, which the authors attribute to reflection on unreasonable prompt assumptions.The case studies describe the model correcting a false premise through self-reflection.Source: §4.2 · p. 7

4.3 General Scenario

  • On a Python asyncio query, the fine-tuned response added hierarchical structure, systematic debugging, practical troubleshooting, and documentation references.The baseline response was technically accurate but more basic and omitted several of these elements.Source: §4.3 · p. 9

5 A Framework for Evaluating O1 Replication Claims: The Technical Transparency Index

The Technical Transparency Index (TTI) provides metrics for comparing O1 replication claims by their transparency and reproducibility. Its data-transparency dimension considers both the stated origins of datasets and how data is selected and prepared.

5 A Framework for Evaluating O1 Replication Claims: The Technical Transparency Index

  • The TTI quantifies the transparency and reproducibility of claimed O1 implementations.It is proposed as a framework for systematic evaluation and comparison.Source: §5 · p. 10

5.1 Evaluation Dimensions of Transparency

  • Transparency spans data, methodology, and evaluation reporting, as well as openness of resources for independent verification.The framework considers dataset accessibility, methodological clarity, evaluation reproducibility, and availability of code, datasets, and models.Source: §5.1 · p. 10

Index 1: Data Transparency

  • Data transparency checks whether dataset origins are identified through dataset names, providers, or source publications.This applies to data used in SFT, RL, search, and synthesis of long-thought data.Source: Index 1: Data Transparency · p. 10
  • Data transparency also assesses whether filtering, cleaning, and preprocessing criteria are clearly described.The assessment covers preparation for SFT, search, or RL.Source: Index 1: Data Transparency · p. 10

Index 2: Methodology Transparency

Methodology transparency assesses whether methods are described well enough for independent reproduction and validation. It covers model and training details, the algorithms used, and evidence that individual methods contribute to performance.

Index 2: Methodology Transparency

  • Methodology reporting should provide enough detail to reproduce and validate the approach, and quantify individual techniques’ contributions rather than only final results.The framework covers foundational model descriptions, training, and data curation.Source: Index 2: Methodology Transparency · p. 10
  • Foundation-model reporting covers architecture and parameter size, while training reporting covers procedures, hyperparameters, and configuration.Source: Index 2: Methodology Transparency · p. 10
  • The checklist assesses documentation of search algorithms, RL or preference-learning methods, and long-thought data synthesis procedures.Relevant details include search parameters and steps, reward functions and training dynamics, and synthesis algorithms or heuristics.Source: Index 2: Methodology Transparency · p. 10

5.2 Checklist for O1-style Technique

The checklist evaluates whether O1-style techniques are adequately validated and whether performance assessments are transparent. It considers the evidence for method effectiveness, benchmark suitability, and metric clarity and comparability.

5.2 Checklist for O1-style Technique

  • Effectiveness validation calls for ablations, comparisons, or incremental analyses that quantify each method’s contribution to system performance.Source: §5.2 · p. 11

Index 3: Evaluation Transparency

Index 4: Open-Source Resources

The open-source resources dimension evaluates the availability and accessibility of materials needed for independent validation and further experimentation. It covers data, model weights, code, and documentation.

Index 4: Open-Source Resources

  • Open-source resources are assessed for whether they support independent validation and further experimentation.The framework considers datasets, models, code, and documentation.Source: Index 4: Open-Source Resources · p. 11
  • The data criterion checks whether raw post-training data and synthesized O1-like datasets are publicly available.The passage links dataset availability to reproducibility and use on additional tasks.Source: Index 4: Open-Source Resources · p. 11
  • The model-weights criterion checks whether trained weights are publicly released for replication and further optimization.Source: Index 4: Open-Source Resources · p. 11
  • The code and documentation criteria assess release of training and evaluation scripts and clear materials explaining methods and results.The codebase should be complete and well documented; supplemental materials include papers, reports, or blog posts.Source: Index 4: Open-Source Resources · p. 11

5.2 Checklist for O1-style Technique

The checklist assigns transparency indicators point values totaling 100, using binary Yes/No assessments and weighting key technical pipeline components more heavily.

5.2 Checklist for O1-style Technique

  • The framework scores O1 replication transparency out of 100 points using a systematic checklist.Table 3 presents the scoring framework.Source: §5.2 · p. 11, Table 3 · p. 12
  • Each indicator is scored Yes or No, and openly documenting that a technique is not used still earns transparency credit.The authors use this binary approach to prioritize transparency over implementation scope.Source: §5.2 · p. 11
  • Indicators for the search algorithm and long-thought data synthesis receive higher scores because of their importance to success and reproducibility.Source: §5.2 · p. 11

5.3 Compared Works

The authors compare O1 replication efforts across transparency and performance; their work earns a perfect transparency score in the leaderboard.

5.3 Compared Works

  • The comparison covers Open O1, O1-Journey Part 1, LLaMA-O1, k0Math, Skywork O1, Deepseek-R1-Lite, and O1-Journey Part 2.The works are assessed across transparency and performance dimensions.Source: §5.3 · p. 13

5.4 Leaderboard

  • The O1-Journey series earns a perfect transparency score, covering data accessibility, methodology clarity, and open-source resource availability.Table 4 presents transparency scores for multiple O1 replication efforts.Source: §5.4 · p. 13, Table 4 · p. 13

6 The Bitter Lesson of Simple Distillation

The paper argues that easy gains from O1 distillation can divert effort from fundamental innovation and first-principles training. It recommends balancing distillation with investment in core research and researcher education.

6 The Bitter Lesson of Simple Distillation

  • Distillation offers straightforward access to performance gains, but the authors warn that relying on this shortcut may impede long-term progress.They characterize distillation as accessible and relatively straightforward to implement, while arguing that convenience comes at a cost.Source: §6 · p. 13
  • Distilled models are bounded by their teacher’s capabilities, creating a ceiling for extending performance to new domains or unseen challenges.Source: §6 · p. 13
  • The authors argue that relying on distillation bypasses work on inference-time scaling and search optimization, risking a widening gap in core technical capabilities.They identify these mechanisms as likely central to O1’s breakthrough.Source: §6 · p. 13

6.1 Suggestions

  • Easy distillation gains may shift focus away from fundamental challenges, reducing investment in computing infrastructure and advanced search and reasoning algorithms.The paper describes a reinforcing cycle in which limited infrastructure further encourages reliance on distillation.Source: §6.1 · p. 14
  • The authors warn that shortcut-focused training risks weakening researchers’ first-principles problem-solving skills and practical understanding of how systems work.They contrast deep technical problem-solving with an emphasis on optimization and prompt engineering.Source: §6.1 · p. 14
  • Pressure for quick results may displace deeper technical investigations and discourage students from pursuing challenging fundamental research.Source: §6.1 · p. 14
  • The paper warns that the divide between organizations with core search and inference technologies and those relying mainly on distillation may become increasingly difficult to bridge.Source: §6.1 · p. 14
  • The authors recommend balancing distillation with fundamental research, continued infrastructure investment, core search and inference expertise, and education that pairs practical work with theory.They also call for curricula and research projects that support deep understanding and long-term innovation.Source: §6.1 · p. 14

Related papers

Loading 2411.16489v1…