Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning

1 Introduction

• Mathematical problems offer a well-defined AI research paradigm with precise binary correctness feedback, while proprietary leading models leave no clear pathway to state-of-the-art reasoning.The introduction identifies chain-of-thought methods as part of recent progress.Source: §1 · pp. 1–2
• 95.0 pass@1 accuracy on MATH-500 lets OREAL-32B outperform all previously reported distilled and RL-based models.Figure 1 compares OREAL-32B with competitive baselines.Source: Figure 1 · p. 2, §1 · p. 3
• Intermediate reasoning steps are labor-intensive to evaluate, leaving binary final-answer rewards sparse across trajectories exceeding 2,000 tokens.Search- and value-based credit assignment approaches are described as remaining unsatisfactory compared with distilled models.Source: §1 · p. 2
• OREAL shows that cloning positive trajectories sampled by best-of-N is sufficient for KL-regularized optimality and that negative-sample rewards need shaping for consistent gradients.The shaping compensates for BoN’s under-sampling of negative gradients and supports difficulty-adaptive optimization over successful and failed trajectories.Source: §1 · p. 2
• A token-level reward model trained with outcome rewards estimates step-wise importance weights to focus learning on critical reasoning steps or errors.The approach decomposes trajectory advantages to address partial correctness in long reasoning chains.Source: §1 · p. 2
• 91.0 pass@1 accuracy on MATH-500 makes OREAL-7B, to the authors’ knowledge, the first 7B model to reach that level using RL instead of distillation.The paper says this result exceeds QwQ-32B-Preview and o1-mini; applying OREAL to DeepSeek-R1-Distilled-Qwen-7B raises its MATH-500 score from 92.8 to 94.0.Source: §1 · p. 2

2.1 Preliminary

• In the RL setup, a math problem prompts a multi-token reasoning trajectory, while correctness is evaluated from the final answer and returned as a binary outcome reward.This leaves thousands of tokens without intermediate progress or correctness feedback.Source: §2.1 · p. 3

2.2 Learning from Positive Samples

• Behavior cloning on BoN-selected positive trajectories learns the KL-regularized optimal policy, and increasing the sample budget can approximate any positive KL constraint.The BoN-selected positive trajectory distribution converges to the policy conditioned on correctness, independently of the sampling budget.Source: §2.2 · p. 4

2.3 Learning from Negative Samples

• Because BoN suppresses negative samples, reward reshaping is needed for gradient consistency between the BoN objective and training on positive and negative samples.At 50% empirical accuracy, the negative-sample probability under Best-of-4 is 6.25%.Source: §2.3 · p. 5

2.4 Dealing with Long Reasoning Chains

• A token-level reward model trained from outcome feedback scores reasoning prefixes, directing credit toward steps that distinguish correct from incorrect responses.The model uses preference pairs and its scores reweight the training loss to emphasize critical reasoning steps or errors.Source: §2.4 · pp. 6–7

3.1 Policy Initialization

• OREAL initializes its policy from rejection-sampling fine-tuned Qwen2.5-7B or Qwen2.5-32B models and also tests DeepSeek-R1-Distill-Qwen-7B as an alternative.The RFT data includes in-house data, Numina, and the MATH training set.Source: §3.1 · p. 7

3.2 Reinforcement Learning

• Training samples 16 trajectories per question and filters by correctness rate, retaining questions with rates between 0 and 0.8 for further training.During RL, questions with pass rates of 0 or 1 are discarded, and one correct and one incorrect response are retained per remaining question.Source: §3.2 · pp. 7–8
• A generative and rule-based verifier supplies binary outcomes, which train a token-level reward model whose scores provide token-wise importance weights.The reward model is trained with cross-entropy on sampled trajectories and verifier outcomes.Source: §3.2 · p. 7

3.3 Skill-based Enhancement

• For repeatedly missed skills, OREAL augments training with similar MATH questions sharing the skill and adds them during the RFT stage.The approach uses the MATH dataset to reduce the cost of skill annotation.Source: §3.3 · p. 8

4.1 Evaluation Setup

• Evaluations use pass@1 with zero-shot chain-of-thought prompting and greedy decoding across MATH-500, AIME2024, AIME2025 Part1, LiveMathBench, and OlympiadBench.The evaluation compares against reported and paper-run baselines.Source: §4.1 · pp. 8–9

4.2 Overall Results

• 91.0 pass@1 on MATH-500 and 59.9 on OlympiadBench for OREAL-7B; OREAL-DSR1-Distill-Qwen-7B reaches 94.0 and 66.1, respectively.The paper reports the 7B result as achieved using RL rather than distillation, and says it surpasses larger QwQ-32B-Preview and OpenAI-o1-mini models.Source: §4.2 · p. 9
• 95.0 pass@1 on MATH-500, 46.7 on AIME2025-I, 74.8 on LiveMathBench, and 72.4 on OlympiadBench make OREAL-32B state-of-the-art among previously reported models.The authors also describe the method as scalable across model sizes.Source: §4.2 · p. 9

4.3 Ablation Study

• Adding reward shaping (L2), behavior cloning (L1), and importance shaping (Ltotal) step by step steadily increases 7B MATH-500 Pass@1, raising the policy model from an initial 84.8 to 91.0.Source: §4.3 · p. 10
• REINFORCE training is unstable, reward shaping mitigates this, positive-sample behavior cloning speeds early convergence, and importance sampling grows slowly before ultimately performing best.The comparison reports average pass@1 accuracy across benchmarks during training.Source: §4.3 · p. 10, Figure 2 · p. 10

4.4 Analysis of Initial Policy Models

• Using skill-based enhancement raises MATH-500 scores for both initial and RL-trained models, while stronger initial policies correspond to higher post-RL performance.The analysis reports that OREAL improves several starting policies, with slight AIME fluctuations when initial performance is already high.Source: §4.4 · p. 10, Table 3 · p. 11

• Prompting methods such as chain-of-thought [7] [9], self-consistency [8], Tree-of-Thought [51], and Graph-of-Thought [52] stimulate reasoning at inference time but do not fundamentally improve underlying model ability.Source: §5 · p. 11
• Supervised fine-tuning depends on high-quality data and capable teachers [15], while distillation [11] [12] inherits teacher limitations and SFT has been criticized for limited generalization [58].Some studies argue SFT turns models into knowledge retrievers rather than actual reasoners.Source: §5 · p. 11
• Earlier RL work targets preference alignment [27] or reasoning [5] [6] [15] [60]; ReSTEM [61] and RFT [23] clone positive samples, while GRPO [6], RLOO [62], and REINFORCE use both outcomes.The paper distinguishes OREAL as a unified framework grounded in mathematical reasoning’s characteristics.Source: §5 · p. 11

6 Conclusion and Future Work

• OREAL's three insights are positive-trajectory behavior cloning under binary feedback, reward shaping, and token-level credit assignment without additional value networks.The paper describes positive-trajectory cloning from Best-of-n sampling as necessary and sufficient for optimal policy learning.Source: §6 · pp. 11–12
• 94 pass@1 accuracy on MATH-500 is reported for OREAL initialized from DeepSeek-R1-Distill-Qwen-7B, on par with previous best 32B models.OREAL-32B also obtains new state-of-the-art results among 32B models on MATH-500, LiveMathBench, and OlympiadBench.Source: §6 · p. 12
• Scalable mathematical-reasoning RL is identified as relying on strong initial policies and diverse, sufficient training data across difficulty, quantity, and scope.The authors identify these factors as primary focuses of future work.Source: §6 · p. 12

A Token Level Reward Model Score Visualization

• Correct responses have high overall reward scores, especially near the end, despite some lower-scoring middle sections.Scores are normalized to [0, 1]; cooler colors indicate higher scores and warmer colors lower scores.Source: A Token Level Reward Model Score… · p. 17
• Incorrect responses show a reversed reward distribution, with scores falling closer to the end.The passage contrasts these patterns with the high overall rewards for correct responses.Source: A Token Level Reward Model Score… · p. 17
• The differing token scores indicate that tokens contribute unequally, motivating token-level credit assignment.Source: A Token Level Reward Model Score… · p. 17

B Prompt

• Figure A3 contains the model-based verifier prompt, which is used during RL training to provide binary outcome rewards.Source: B Prompt · p. 17, Figure A3 · p. 19
• The verifier prompt asks whether candidates' final answers are correct by comparing them with standard answers.Source: B Prompt · p. 19
• The long-CoT system prompt casts the model as an expert mathematician following systematic thinking and rigorous reasoning.Figure A4's prompt is used for fine-tuning, RL training, and evaluation.Source: B Prompt · pp. 17, 20, Figure A4 · p. 20