Source-linked AI summary
O1 Replication Journey: A Strategic Progress Report -- Part 1
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, Pengfei Liu
TL;DR
The paper addresses limited transparency around O1 and shortcomings in conventional AI research reporting. It proposes a transparent, real-time replication journey and journey learning, which trains models on complete exploration rather than shortcuts. With 327 training samples and no additional tricks, journey learning outperformed conventional supervised learning by over 8% on MATH.
Problem
O1’s implementation, training data, and complete outputs remained undisclosed, while prolonged team-based AI research faced information isolation and delayed feedback.
Method
The paper combines a transparent, real-time replication report with journey learning that records and trains on trial, reflection, and backtracking.
Results
327 training samples and no additional tricks yielded over 8% improvement over conventional supervised learning on the MATH dataset.
Takeaways & Limitations
The authors identify journey learning as the most crucial component of O1 technology decoded so far and share resources to support open, collaborative research.
Takeaways & Limitations
The authors describe the findings as initial exploratory results and acknowledge that their trial-search constraints limit exploration of incorrect paths.
Abstract
from arXiv · showhide
This paper introduces a pioneering approach to artificial intelligence research, embodied in our O1 Replication Journey. In response to the announcement of OpenAI's groundbreaking O1 model, we embark on a transparent, real-time exploration to replicate its capabilities while reimagining the process of conducting and communicating AI research. Our methodology addresses critical challenges in modern AI research, including the insularity of prolonged team-based projects, delayed information sharing, and the lack of recognition for diverse contributions. By providing comprehensive, real-time documentation of our replication efforts, including both successes and failures, we aim to foster open science, accelerate collective advancement, and lay the groundwork for AI-driven scientific discovery. Our research progress report diverges significantly from traditional research papers, offering continuous updates, full process transparency, and active community engagement throughout the research journey. Technologically, we proposed the journey learning paradigm, which encourages models to learn not just shortcuts, but the complete exploration process, including trial and error, reflection, and backtracking. With only 327 training samples and without any additional tricks, journey learning outperformed conventional supervised learning by over 8\% on the MATH dataset, demonstrating its extremely powerful potential. We believe this to be the most crucial component of O1 technology that we have successfully decoded. We share valuable resources including technical hypotheses and insights, cognitive exploration maps, custom-developed tools, etc at https://github.com/GAIR-NLP/O1-Journey.
1 Chronological Overview of the O1 Exploration Journey
The O1 exploration journey progressed chronologically from initial evaluation and knowledge acquisition through team analysis, long-thought construction, and iterative model improvement.
- The journey began after O1’s release with initial evaluation, knowledge acquisition, team assembly, and analysis.
- The exploration then pursued four long thought construction attempts.
- The core second attempt followed three tracks: Process-Level Reward Model, Construction of Reasoning Tree, and Integrating Reasoning Tree into Long Thought.
- These tracks converged in an iterative model-improvement cycle using quantitative and qualitative evaluation.
- The training pipeline included pre-training, iterative training, and optimization techniques, while dashed paths marked planned future exploration.
2 Introduction
The paper presents the O1 Replication Journey as a transparent, real-time research process responding to limited public knowledge of O1’s implementation and outputs. Its central technical proposal, journey learning, trains models on complete problem-solving exploration and showed over 8% improvement on MATH with 327 samples.
- OpenAI’s O1 capabilities were publicly announced while its implementation, training data, and complete outputs remained undisclosed.
- The report reimagines AI research through continuous documentation, real-time feedback, transparency, and active resource sharing rather than a traditional publication format.
- Journey learning trains models on complete exploration, including trial and error, reflection, and backtracking, instead of only solution shortcuts.
- 327 training samples produced over 8% improvement over conventional supervised learning on the MATH dataset without additional tricks.
- The project documents hypotheses, cognitive exploration paths, trial-and-error insights, and cognitive processes as resources for the research community.
3 Why We Created Progress Report?
The progress report addresses information isolation, delayed feedback, and limited recognition in prolonged team-based AI research. It uses transparent documentation to support open collaboration, scientific discovery, and responsible AI development.
- The project aims to provide a comprehensive, real-time account that fosters open science and redefines scientific communication.
- Prolonged team-based AI projects can restrict information flow, delay researcher feedback, and complicate recognition of diverse contributions.
- The report shares insights, resources, lessons, thought processes, and innovative approaches to help researchers navigate pitfalls and generate new ideas.
- Documenting successes, failures, decision rationales, inspirations, and thought processes creates material intended to train AI systems for scientific discovery.
- The report’s transparency is presented as a basis for public trust and responsible AI development.
4 Journey Learning: A New Paradigm Shift from “Shortcut Learning”
The paper contrasts shortcut learning, which trains direct solution paths, with journey learning, which trains models on complete exploration trajectories involving trial, error, reflection, and backtracking.
- Shortcut Learning: Shortcut learning emphasizes direct root-to-leaf solution paths and can be limited on complex, dynamic, open-ended problems.The paper characterizes conventional supervised learning as shortcut learning and contrasts it with the demands of broader reasoning.
- Comparison: Table 2 compares shortcut learning and journey learning as distinct training paradigms.The supplied table caption identifies the comparison but does not provide its individual entries.
- Journey Learning: Journey learning trains models on complete decision trajectories rather than only successful shortcuts.Its intended process includes learning, reflection, backtracking, and adaptation.
- Implications: Journey learning is presented as a paradigm for building adaptable reasoning systems suited to complex, dynamic environments.The paper contrasts these systems with narrow, task-specific tools.
5 Background
The background situates journey learning alongside process-level rewards, chain-of-thought reasoning, internal thought, inference-time scaling, search-to-thought methods, and self-improvement in language models.
- Process-Level Rewards: Process reward models provide fine-grained evaluations of individual reasoning steps, especially for mathematical reasoning.They can support post-training and inference-time search by assessing step correctness.
- Chain-of-Thought: Chain-of-thought prompting improves reasoning by exposing intermediate steps and supports serial computation in decoder-only transformers.The cited background connects intermediate reasoning with performance on complex tasks.
- Internal Thought: Internal-thought research trains models to generate rationales, reflect on decisions, and revise reasoning through iterative introspection.The cited examples include STaR, Quiet-STaR, reflection training, and recursive introspection.
- Inference-Time Scaling: Inference-time scaling seeks performance gains by allocating more computation during inference rather than relying only on larger models or training datasets.The background notes that parameter scaling can produce diminishing returns and substantial costs.
- Search-to-Thought: Search-to-thought research traces a shift from explicit search algorithms toward implicit reasoning approaches based on generated intermediate reasoning.The background contrasts classic systems such as Deep Blue with chain-of-thought methods.
- Self-Improvement: Self-improvement methods let language models learn from their own outputs through supervised fine-tuning or preference optimization.Mathematical reasoning can use final-answer or step-level rewards, including human annotations and process reward models.
6 Exploration Journey
The exploration journey documents a question-driven investigation of O1’s thought structure and long-thought construction, combining analysis of examples with iterative technical exploration.
- Exploration Framework: The chapter organizes the replication effort around sequential questions about O1 evaluation, thought structure, long-thought construction, rewards, and reasoning trees.This structure is intended to expose both the technical progression and the reasoning behind research decisions.
- Exploration Framework: The question-driven format emphasizes documenting the complete learning and exploration process rather than only final outcomes.The authors connect this format directly to the journey learning paradigm.
- O1 Thought Analysis: The researchers analyze eight OpenAI-provided O1 reasoning examples across task types and difficulty levels using token, line, and keyword features.The analysis categorizes examples from simple reading comprehension to complex multistep mathematical reasoning.
- O1 Thought Analysis: Keywords such as “consider,” “if,” and “possible” are associated with branching, while “wait” and “Alternatively” indicate reflection and self-correction.The paper reports that branching-related keywords were more frequent in higher-complexity problems.
- Long-Thought Characteristics: The extracted long-thought characteristics include iterative problem solving, explicit reasoning-stage indicators, recursive validation, hypothesis exploration, and conclusion verification.These characteristics describe how the model breaks down equations, explores alternatives, reassesses results, and checks conclusions.
6.2 Q2: How does Long Thought Work?
The paper treats long thought as exploration through correct and incorrect reasoning paths, with reflection and backtracking supported by tree search, model-selected actions, multi-agent methods, and human annotations.
- Hypothesis: The authors hypothesize that journey learning may explain O1’s long-thought success, but explicitly state that empirical evidence is currently insufficient to verify this claim.Their hypothesis links full decision trajectories with learning from errors and self-correction.
- Journey Learning: Journey learning exposes models to correct and incorrect paths so they can develop error-handling and self-correction capabilities.The proposed process includes trial-and-error, reflection, and adjustment rather than only learning final answers.
- Construction Attempts: Tree search constructs a reasoning tree whose nodes are reasoning steps and uses process-level rewards to identify erroneous nodes for reflection and backtracking.Search paths from the problem root to conclusions incorporate incorrect steps into longer thoughts.
- Construction Attempts: A Propose-Critique Loop lets the model choose among predefined actions such as continue, backtracking, reflection, and terminate.This expands action selection beyond fixed tree-search rules.
- Construction Attempts: The multi-agent approach assigns continuous reasoning and critique roles to separate agents to address ineffective nodes and logical inconsistency.The cited passage identifies these as challenges in building long thought from reasoning trees.
- Construction Attempts: Complete human thought-process annotation records reflection, backtracking, and rewriting as people solve reasoning problems.The authors present this as a way to produce high-quality long-thought data.
6.4 Q4: How to Construct Reward Models?
The reward-model pipeline evaluates reasoning at the step level, uses reward-guided beam-search pruning, and compares reward models on PRM800K and MR-GSM8K subsets.
- Reward-model design: Step-level evaluation separates solution lines so reward models can assess reflection, backtracking, and other cognitive processes.The approach uses fine-tuning data from Chern et al. (2023) to distinguish solution steps by line number.
- Results: O1-mini performs best across the evaluated subsets of PRM800K and MR-GSM8K.The comparisons are reported in Table 4 and Table 5.
- Reward-model design: Reasoning-tree construction uses a policy model to generate multiple candidate reasoning steps recursively from the problem root.The policy model generates w possible children at each stage of forward reasoning.
- Pruning: Beam-search pruning retains only a small number of candidates at each iteration, reducing the total number of generated reasoning steps.The method uses a reward model to remove erroneous steps before continuing tree expansion.
- Pruning: O1-mini provides step-level correctness rewards because Math-Shepherd struggles to evaluate difficult reasoning steps effectively.At each iteration, the method retains at most K steps judged correct by O1-mini.
6.6 Q6: How to Derive a Long Thought from a Reasoning Tree?
The method derives long thoughts by traversing a reward-annotated reasoning tree, preserving incorrect trials and backtracking while controlling exploration and refining the resulting draft.
- Tree-to-thought construction: A reasoning tree is annotated with reward-model judgments indicating whether each step is correct or incorrect.These ratings include reasoning that justifies each judgment.
- Tree-to-thought construction: The shortCut is the path containing only valid intermediate steps and the correct answer.Multiple correct answer nodes produce multiple correct paths.
- Traversal: Depth First Search records the path from the root to a correct answer while including reasoning for incorrect nodes.This creates trial-and-error paths rather than only a direct shortcut.
- Traversal: Each correct-path node permits at most K trials, limiting exploration of incorrect branches while retaining meaningful trial and error.The constraint allows one incorrect-path trial and one correct-path trial per node.
- Draft refinement: GPT-4o revises the concatenated traversal draft to improve coherence and smoothness while preserving its reasoning content.Raw drafts produced suboptimal training performance in initial experiments.
- Evaluation support: A Streamlit platform supports manual inspection of synthetic trees, long thoughts, and model outputs with conditional filtering and iteration comparison.Filters include correctness and keywords associated with reflection or hesitation, such as “wait”.
6.9 Q9: What Would be an Effective Annotation Strategy for Human-AI Collaboration?
The annotation strategy records complete exploratory reasoning and uses human-AI augmentation to generate fine-grained, reflective long-form data. Journey Learning improved MATH performance over Shortcut Learning in the reported experiments.
- Reported outcome: Journey Learning improved results over Shortcut Learning by +8.4 and +8.0 on two MATH training configurations.The results use a 500-example MATH test subset re-divided from PRM800K.
- Human-AI pipeline: The human-AI pipeline expands human-annotated multi-line solutions into thousands of tokens following the Journey Learning paradigm.The pipeline is designed to generate high-quality, long-form reasoning data for MATH-derived problems.
- Human annotation: High-quality annotations document trials, reflections, associations, and corrections rather than every word of thought.The strategy treats cognitive transitions as important training information for language models.
- Human annotation: Annotations add explicit explanations for common-sense knowledge because omitted context can lead language models to hallucinate.The goal is to prevent misinterpretation of human annotations.
- AI augmentation: LLM augmentation breaks reasoning into finer steps, inserts pauses for reflection, and adopts a student-explorer perspective.These prompts are intended to make reasoning more granular, gradual, and discovery-oriented.
7 Detailed Event Explanation of Our Research Exploration
Table 7 provides detailed explanations of the research exploration by linking nodes and short descriptions to Figure 2 and associating them with purposes and required resources.
- Table 7: Table 7 maps Figure 2 exploration nodes to explanations of their purpose and the resources required for each node.The table also includes node-level short descriptions.
8 Future Plan
The future plan expands the replication effort through scaling, evaluation, training, collaboration, and continued resource sharing. It emphasizes continuous improvement, transparent exploration, and collaborative advancement while remaining adaptable to new discoveries and challenges.
- Overall direction: These avenues are intended to advance O1 replication and AI research methodologies through continuous improvement, transparent exploration, and collaborative advancement.The plans remain adaptable to new discoveries and challenges and seek to contribute to more capable, interpretable, and ethically aligned AI systems.
- Scaling and evaluation: The team plans to scale long-thought integration and investigate how performance changes with data, model size, and computational resources.These experiments target more complex thought patterns and fundamental scaling principles.
- Scaling and evaluation: More fine-grained, thought-centric evaluations will assess the quality and coherence of generated long thoughts and provide deeper insight into reasoning capabilities.
- Reasoning and collaboration: The plan combines human-AI collaboration, refined reward and critique models, and advanced reasoning-tree integration to improve high-quality thought generation.These efforts leverage human and AI strengths, capture reasoning nuances, and synthesize information from complex structures.
- Training methodology: The training pipeline will be expanded and refined across pre-training, iterative training, reinforcement learning, preference learning, and DPO stages.
- Open science: Continued transparency and resource sharing will provide the wider AI research community with insights, tools, and other materials developed during the journey.The stated aim is to foster collaboration and accelerate broader research progress.