Source-linked AI summary
Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang
TL;DR
Marco-o1 addresses whether o1-like reasoning can extend to open-ended domains where standards and rewards are difficult to specify. It combines CoT fine-tuning, MCTS, reflection, and reasoning strategies, and reports improved MGSM accuracy alongside success on complex slang translation. The paper presents these results as preliminary evidence of o1-like reasoning characteristics, while acknowledging that Marco-o1 falls short of a fully realized o1 model.
Problem
Marco-o1 investigates how large reasoning models can generalize to open-ended, complex real-world tasks where clear standards and quantifiable rewards are absent.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
Marco-o1 shows preliminary o1-like reasoning characteristics across reasoning and translation tasks, including colloquial and slang expressions.
Source: §1 · p. 2, MGSM-Zh · p. 7, §6 · p. 7
Abstract
from arXiv · showhide
Currently OpenAI o1 sparks a surge of interest in the study of large reasoning models (LRM). Building on this momentum, Marco-o1 not only focuses on disciplines with standard answers, such as mathematics, physics, and coding -- which are well-suited for reinforcement learning (RL) -- but also places greater emphasis on open-ended resolutions. We aim to address the question: ''Can the o1 model effectively generalize to broader domains where clear standards are absent and rewards are challenging to quantify?'' Marco-o1 is powered by Chain-of-Thought (CoT) fine-tuning, Monte Carlo Tree Search (MCTS), reflection mechanisms, and innovative reasoning strategies -- optimized for complex real-world problem-solving tasks.
1. Introduction
Marco-o1 explores large reasoning models for complex real-world and open-ended tasks, combining CoT fine-tuning, MCTS, and reasoning-action strategies. Preliminary results show improved MGSM accuracy and stronger handling of colloquial translation.
1. Introduction
- Marco-o1 targets complex reasoning beyond standard-answer disciplines by exploring approaches for the unclear technical roadmap of large reasoning models.The work is inspired by OpenAI’s o1 and emphasizes open-ended questions.Source: §1 · p. 2
- Marco-o1 combines CoT fine-tuning, MCTS, and reasoning action strategies to enhance reasoning power.It fine-tunes Qwen2-7B-Instruct with filtered Open-O1 CoT, Marco-o1 CoT, and Marco-o1 Instruction datasets.Source: §1 · p. 2
- +6.17% MGSM (English) and +5.60% MGSM (Chinese) accuracy improvements were achieved by Marco-o1.The authors describe these preliminary results as evidence of enhanced reasoning capabilities.Source: §1 · p. 2
- Marco-o1 expands solution search with MCTS, adds self-reflective reasoning strategies, and investigates inference-time scaling for multilingual translation.The contributions include CoT fine-tuning, MCTS-guided solution-space expansion, reasoning-action strategies, and translation applications.Source: §1 · p. 2
2. Marco Reasoning Datasets
Marco-o1’s reasoning datasets combine filtered open-source CoT data, synthetic MCTS-generated reasoning paths, and instruction-following data to support broad task competence.
2. Marco Reasoning Datasets
- Supervised fine-tuning uses a variety of datasets to enhance Marco-o1’s reasoning capabilities.The dataset strategy is presented as the foundation for model training.Source: §2 · p. 2
- The filtered Open-O1 CoT dataset is refined through heuristic and quality filtering to provide structured reasoning patterns.The dataset is part of the Marco reasoning-data mixture.Source: §2 · p. 2
- The synthetic Marco-o1 CoT dataset is generated using MCTS to formulate complex reasoning pathways.MCTS-generated data is intended to bolster reasoning capabilities.Source: §2 · p. 2
- The Marco Instruction Dataset supports instruction-following capabilities for executing complex tasks.The combined data is described as maintaining broad task effectiveness while boosting reasoning.Source: §2 · pp. 2–3
3. Solution Space Expansion via MCTS
Marco-o1 integrates LLM outputs into MCTS, using token-level confidence to score rollout paths and guide exploration toward more probable reasoning chains.
3. Solution Space Expansion via MCTS
- MCTS represents reasoning states as nodes, LLM outputs as actions, and completed rollouts as paths receiving reward scores.The reward evaluates and selects promising reasoning paths within the search tree.Source: §3 · pp. 3–4
- Each generated token receives a confidence score from a softmax over its log probability and the top 5 alternative-token log probabilities.This compares the selected token with competing predictions at the same step.Source: §3 · p. 4
- The confidence score normalizes the chosen token’s relative probability against alternatives between 0 and 1.The normalization makes token confidence usable for evaluating rollout paths.Source: §3 · p. 4
- The average confidence across rollout tokens becomes the overall reward score for the reasoning path.A higher reward indicates a more confident and likely accurate path.Source: §3 · p. 4
- Confidence-guided MCTS expands the solution space and selects among many reasoning paths.The search favors paths with higher calculated confidence scores.Source: §3 · p. 4
4. Reasoning Action Strategy
Marco-o1 refines MCTS action granularity from complete steps to 32- or 64-token mini-steps and adds reflection to revisit potentially incorrect reasoning. These strategies expand search detail and improve difficult-problem correction, though the paper does not identify one universally superior action strategy.
4.1. Action Selection
- Complete-step actions are efficient but may overlook nuanced reasoning paths needed for complex problems.The authors therefore examine finer-grained search units.Source: §4.1 · p. 4
- Mini-step actions use 32 or 64 tokens to expand the solution space and navigate complex reasoning tasks more finely.Finer search can expose correct answers missed by larger action units.Source: §4.1 · pp. 4–5
4.2. Reflection after Thinking
- Reflection prompts the model to rethink its reasoning from scratch after each thought process.The mechanism encourages self-critique and reevaluation of reasoning steps.Source: §4.2 · p. 5
- Approximately half of challenging problems initially solved incorrectly are answered correctly after adding reflection.The reported improvement is strongest on difficult problems.Source: §4.2 · p. 5
5. Experiments
Experiments evaluate Marco-o1 on multilingual MGSM reasoning and colloquial translation, comparing CoT fine-tuning and MCTS configurations. Results show gains from MCTS and stronger handling of slang, while performance varies across languages and search strategies.
MGSM
- Marco-o1-CoT improves English MGSM performance but decreases on Chinese MGSM, attributed to English-only CoT fine-tuning.The authors suggest the English training data may not transfer effectively to Chinese.Source: §5.2 · pp. 5–6
- The three MCTS-enhanced models outperform Marco-o1-CoT, but the best action granularity differs between English and Chinese MGSM.Step-level actions perform best on MGSM-en, whereas 32-token mini-steps perform best on MGSM-zh.Source: §5.2 · p. 6
- MCTS shows an advantage at Test@1, measuring whether at least one of N guesses is correct, with evaluations at Test@1, Test@8, and Test@32.The authors interpret the lower-guess advantage as evidence of MCTS’s potential.Source: §5.2 · p. 6, Table 3 · p. 7
Reasoning behavior
- Marco-o1’s reasoning paths can omit an obvious final step while still reaching the correct answer, as illustrated by the “strawberry” question.Most MCTS paths omitted the final “y,” although some included it.Source: §5.2 · p. 6
Translation
- In translation case studies, Marco-o1 provides more accurate and natural translations of colloquial and slang expressions than standard translation tools such as Google Translate.The reported examples emphasize contextual and nuanced understanding in complex translation tasks.Source: §5.3 · p. 6, MGSM-Zh · p. 7
6. Conclusions and Future Work
Marco-o1 combines CoT fine-tuning, MCTS, and reasoning action strategies to strengthen reasoning. The authors report improvements on reasoning tasks and success with complex slang translation, while proposing reward-model and reinforcement-learning extensions.
Contributions
- Marco-o1 integrates CoT fine-tuning, MCTS, and novel reasoning action strategies to enhance reasoning ability.MCTS expands solution spaces, while different action granularities explore steps and mini-steps.Source: §6 · p. 7
Conclusions
- The approach demonstrates improvements in reasoning tasks and success translating complex slang expressions.The conclusion summarizes both reasoning-task gains and translation performance.Source: §6 · p. 7
Future work
- Future work will refine MCTS rewards through Outcome Reward Modeling and Process Reward Modeling to reduce randomness and improve performance.The paper also discusses reinforcement learning to fine-tune Marco-o1’s decision-making processes.Source: §6 · p. 7