Source-linked AI summary
Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang
TL;DR
Marco-o1 addresses whether o1-like reasoning can extend to open-ended domains where standards and rewards are difficult to specify. It combines CoT fine-tuning, MCTS, reflection, and reasoning strategies, and reports improved MGSM accuracy alongside success on complex slang translation. The paper presents these results as preliminary evidence of o1-like reasoning characteristics, while acknowledging that Marco-o1 falls short of a fully realized o1 model.
Problem
Marco-o1 investigates how large reasoning models can generalize to open-ended, complex real-world tasks where clear standards and quantifiable rewards are absent.
Method
Marco-o1 combines CoT fine-tuning, MCTS-guided solution-space expansion, variable-granularity reasoning actions, and reflection.
Results
+6.17% MGSM (English) and +5.60% MGSM (Chinese) accuracy improvements were reported, alongside successful complex slang translation.
Takeaways & Limitations
Marco-o1 shows preliminary o1-like reasoning characteristics across reasoning and translation tasks, including colloquial and slang expressions.
Abstract
from arXiv · showhide
Currently OpenAI o1 sparks a surge of interest in the study of large reasoning models (LRM). Building on this momentum, Marco-o1 not only focuses on disciplines with standard answers, such as mathematics, physics, and coding -- which are well-suited for reinforcement learning (RL) -- but also places greater emphasis on open-ended resolutions. We aim to address the question: ''Can the o1 model effectively generalize to broader domains where clear standards are absent and rewards are challenging to quantify?'' Marco-o1 is powered by Chain-of-Thought (CoT) fine-tuning, Monte Carlo Tree Search (MCTS), reflection mechanisms, and innovative reasoning strategies -- optimized for complex real-world problem-solving tasks.
1. Introduction
Marco-o1 explores large reasoning models for complex real-world and open-ended tasks, combining CoT fine-tuning, MCTS, and reasoning-action strategies. Preliminary results show improved MGSM accuracy and stronger handling of colloquial translation.
- 1. Introduction: Marco-o1 targets complex reasoning beyond standard-answer disciplines by exploring approaches for the unclear technical roadmap of large reasoning models.The work is inspired by OpenAI’s o1 and emphasizes open-ended questions.
- 1. Introduction: Marco-o1 combines CoT fine-tuning, MCTS, and reasoning action strategies to enhance reasoning power.It fine-tunes Qwen2-7B-Instruct with filtered Open-O1 CoT, Marco-o1 CoT, and Marco-o1 Instruction datasets.
- 1. Introduction: +6.17% MGSM (English) and +5.60% MGSM (Chinese) accuracy improvements were achieved by Marco-o1.The authors describe these preliminary results as evidence of enhanced reasoning capabilities.
- 1. Introduction: Marco-o1 expands solution search with MCTS, adds self-reflective reasoning strategies, and investigates inference-time scaling for multilingual translation.The contributions include CoT fine-tuning, MCTS-guided solution-space expansion, reasoning-action strategies, and translation applications.
2. Marco Reasoning Datasets
Marco-o1’s reasoning datasets combine filtered open-source CoT data, synthetic MCTS-generated reasoning paths, and instruction-following data to support broad task competence.
- 2. Marco Reasoning Datasets: Supervised fine-tuning uses a variety of datasets to enhance Marco-o1’s reasoning capabilities.The dataset strategy is presented as the foundation for model training.
- 2. Marco Reasoning Datasets: The filtered Open-O1 CoT dataset is refined through heuristic and quality filtering to provide structured reasoning patterns.The dataset is part of the Marco reasoning-data mixture.
- 2. Marco Reasoning Datasets: The synthetic Marco-o1 CoT dataset is generated using MCTS to formulate complex reasoning pathways.MCTS-generated data is intended to bolster reasoning capabilities.
- 2. Marco Reasoning Datasets: The Marco Instruction Dataset supports instruction-following capabilities for executing complex tasks.The combined data is described as maintaining broad task effectiveness while boosting reasoning.
3. Solution Space Expansion via MCTS
Marco-o1 integrates LLM outputs into MCTS, using token-level confidence to score rollout paths and guide exploration toward more probable reasoning chains.
- 3. Solution Space Expansion via MCTS: MCTS represents reasoning states as nodes, LLM outputs as actions, and completed rollouts as paths receiving reward scores.The reward evaluates and selects promising reasoning paths within the search tree.
- 3. Solution Space Expansion via MCTS: Each generated token receives a confidence score from a softmax over its log probability and the top 5 alternative-token log probabilities.This compares the selected token with competing predictions at the same step.
- 3. Solution Space Expansion via MCTS: The confidence score normalizes the chosen token’s relative probability against alternatives between 0 and 1.The normalization makes token confidence usable for evaluating rollout paths.
- 3. Solution Space Expansion via MCTS: The average confidence across rollout tokens becomes the overall reward score for the reasoning path.A higher reward indicates a more confident and likely accurate path.
- 3. Solution Space Expansion via MCTS: Confidence-guided MCTS expands the solution space and selects among many reasoning paths.The search favors paths with higher calculated confidence scores.
4. Reasoning Action Strategy
Marco-o1 refines MCTS action granularity from complete steps to 32- or 64-token mini-steps and adds reflection to revisit potentially incorrect reasoning. These strategies expand search detail and improve difficult-problem correction, though the paper does not identify one universally superior action strategy.
- 4.1. Action Selection: Complete-step actions are efficient but may overlook nuanced reasoning paths needed for complex problems.The authors therefore examine finer-grained search units.
- 4.1. Action Selection: Mini-step actions use 32 or 64 tokens to expand the solution space and navigate complex reasoning tasks more finely.Finer search can expose correct answers missed by larger action units.
- 4.2. Reflection after Thinking: Reflection prompts the model to rethink its reasoning from scratch after each thought process.The mechanism encourages self-critique and reevaluation of reasoning steps.
- 4.2. Reflection after Thinking: Approximately half of challenging problems initially solved incorrectly are answered correctly after adding reflection.The reported improvement is strongest on difficult problems.
5. Experiments
Experiments evaluate Marco-o1 on multilingual MGSM reasoning and colloquial translation, comparing CoT fine-tuning and MCTS configurations. Results show gains from MCTS and stronger handling of slang, while performance varies across languages and search strategies.
- MGSM: Marco-o1-CoT improves English MGSM performance but decreases on Chinese MGSM, attributed to English-only CoT fine-tuning.The authors suggest the English training data may not transfer effectively to Chinese.
- MGSM: The three MCTS-enhanced models outperform Marco-o1-CoT, but the best action granularity differs between English and Chinese MGSM.Step-level actions perform best on MGSM-en, whereas 32-token mini-steps perform best on MGSM-zh.
- MGSM: MCTS shows an advantage at Test@1, measuring whether at least one of N guesses is correct, with evaluations at Test@1, Test@8, and Test@32.The authors interpret the lower-guess advantage as evidence of MCTS’s potential.
- Reasoning behavior: Marco-o1’s reasoning paths can omit an obvious final step while still reaching the correct answer, as illustrated by the “strawberry” question.Most MCTS paths omitted the final “y,” although some included it.
- Translation: In translation case studies, Marco-o1 provides more accurate and natural translations of colloquial and slang expressions than standard translation tools such as Google Translate.The reported examples emphasize contextual and nuanced understanding in complex translation tasks.
6. Conclusions and Future Work
Marco-o1 combines CoT fine-tuning, MCTS, and reasoning action strategies to strengthen reasoning. The authors report improvements on reasoning tasks and success with complex slang translation, while proposing reward-model and reinforcement-learning extensions.
- Contributions: Marco-o1 integrates CoT fine-tuning, MCTS, and novel reasoning action strategies to enhance reasoning ability.MCTS expands solution spaces, while different action granularities explore steps and mini-steps.
- Conclusions: The approach demonstrates improvements in reasoning tasks and success translating complex slang expressions.The conclusion summarizes both reasoning-task gains and translation performance.
- Future work: Future work will refine MCTS rewards through Outcome Reward Modeling and Process Reward Modeling to reduce randomness and improve performance.The paper also discusses reinforcement learning to fine-tune Marco-o1’s decision-making processes.