Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths

• Mistakes in long reasoning paths can compound and derail the process, motivating methods that favor correct paths and avoid unfavorable branches.Source: §1 · p. 1
• Prompting approaches that sample multiple paths and integrate feedback can require extensive token use, computational cost, and execution latency.Source: §1 · p. 1
• The authors hypothesize that preference optimization may target entire reasoning paths even though errors often occur at specific steps and affect subsequent branches.Source: §1 · p. 2
• RPO generates a correct reference path, explores branches from its steps, and uses reference and contrastive branch-pair losses to favor correct reasoning.Source: §1 · p. 2
• The reported improvements over the high-performing baseline are up to 3.1% on GSM8K and 4.3% on MMLU (STEM).Source: §1 · p. 2

2.1 Task Formulation

• A task maps a natural-language question to an answer through sentence-level reasoning steps, with the final step containing the answer; punctuation-based NLTK splitting separates steps.Source: §2.1 · pp. 2–3

2.2 Framework Overview

• RPO has generation, exploration, and optimization stages that use reference paths and explored branches to favor favorable branches and penalize unfavorable ones.The final stage aggregates and optimizes according to the reference paths and explored branches.Source: §2.2 · p. 3

2.3 Reasoning Generation

• RPO generates reference paths with chain-of-thought prompting, verifies the final answer, and removes problems without a correct path after at most ten sampling attempts.The demonstrations pair example questions with their reasoning paths.Source: §2.3 · p. 4

2.4 Reasoning Exploration

• At each reference step, RPO samples branches and verifies their final answers, retaining one correct-ending and one incorrect-ending branch or dropping problems without a pair after ten samples.Source: §2.4 · p. 4

2.5 Reasoning Optimization

• RPO combines a loss that raises reference-path likelihood with contrastive feedback over explored branch pairs; λ balances the two objectives.The branch-pair objective uses log odds-ratios across reasoning steps.Source: §2.5 · pp. 4–5
• The branch-pair loss excludes shared correct prefixes and is computed only on output tokens, the incorrect last part.The shared prefix serves as input rather than part of the loss calculation.Source: §2.5 · p. 5

3.1 Datasets

• The evaluation covers GSM8K math word problems, competition-level MATH, and 3,375 MMLU-STEM questions; experiments use Mistral-7B and LLaMA-3-8B with LoRA fine-tuning.Four randomly selected training questions provide chain-of-thought demonstrations; the method does not require large-scale annotated reasoning paths.Source: §3.1 · p. 5, §3.2 · pp. 5–6

3.4 Main Results

• Up to 3.1% and 4.3% improvement over the highest-performing baseline on GSM8K and MMLU-STEM, respectively, with consistent gains across datasets and models.The main-results figure reports evaluation accuracy on GSM8K, MATH, and MMLU-STEM.Source: Figure 3 · p. 6, §3.4 · p. 7

3.3 Comparison Methods

• Comparisons include SFT, RFT, DPO, and ORPO, with each method trained on samples meeting its respective answer or reasoning-path requirements.For RFT and preference-based methods, questions without a suitable generated path after at most ten attempts are excluded.Source: §3.3 · p. 6

3.5 Effect of Exploration Weight

• Very low or very high exploration-loss weight yields suboptimal MATH performance, indicating a need to balance exploration against the reference path.A low weight emphasizes the supervised reference-path loss; a high weight emphasizes explored branches.Source: §3.5 · p. 7, Figure 4 · p. 7

3.6 Analysis On Reasoning Path Length

• RPO benefits longer reasoning paths on MATH compared with ORPO, the highest-performing baseline.The authors suggest this may indicate fewer mistakes on questions requiring more complex reasoning.Source: §3.6 · p. 7, Figure 5 · p. 7

3.7 Case Study

• In the GSM8K case study, RPO produces a coherent step-by-step solution, while SFT and the other path-based baselines make errors or omit explanatory working.RFT, DPO, and ORPO calculate directly and err at the first step in the example.Source: §3.7 · p. 8, Table 1 · p. 8

3.8 Code-Based Reasoning

• For code-based reasoning, RPO uses Python programs instead of text reasoning paths and shows similar benefits over ORPO.The LLaMA-3-8B analysis compares code-based reasoning with the paper’s main text-based setting.Source: §3.8 · p. 8, Table 2 · p. 8

3.9 Effect of Contrastive Objectives

• RPO remains effective with a direct-preference branch-pair objective and a randomly selected correct reference path; using three correct reference paths can further enhance performance.The objective comparison reports consistent benefits across datasets, and the reference-path analysis reports robustness to path choice.Source: §3.9 · p. 8, Table 3 · p. 8, §3.10 · p. 8

Alignment and Preference-Based Optimization

• Preference-optimization methods compare responses as wholes, whereas reasoning errors arise at specific steps and branches; RPO considers each intermediate step.The cited methods include DPO, IPO, SimPO, and ORPO.Source: §4 · p. 9

Multi-step Reasoning in Language Models

• Recent studies construct step-wise labels automatically to avoid costly human annotations and train verifiers (reward models); in contrast, this work applies process supervision to preference optimization without requiring a separate reward model.Source: §4 · p. 9

Path Exploration in Artificial Intelligence

• Prior path-exploration work includes Monte Carlo Tree Search, Tree-of-Thought, and tree-based decoding; RPO explores model-generated branches and trains with contrastive feedback from both favorable and unfavorable branches.Earlier reasoning-task work also explored self-sampled solutions and tree search for path generation.Source: §4 · p. 9

5 Conclusion

• RPO encourages favorable reasoning branches at each step while penalizing unfavorable branches across diverse branch pairs.The framework uses the model’s own generated reasoning paths rather than large-scale human-annotated rationales.Source: §5 · p. 9
• RPO is presented as scalable because it uses model-generated reasoning paths instead of large-scale human-annotated rationales.The paper identifies multi-step tasks such as math word problems as applications.Source: §5 · p. 9
• Experiments on datasets of varying difficulties support RPO as an approach to enhancing reasoning for complex problem-solving scenarios.The conclusion frames this as a way toward more reliable and accurate problem-solving.Source: §5 · p. 9

Limitations

• RPO relies on correct reasoning paths from the base model; substantially underperforming models may struggle to generate them, limiting the approach’s effectiveness.The limitation concerns path generation during training.Source: Limitations · p. 10
• Generating and exploring multiple paths for each problem is more computationally intensive, though the authors describe this as a one-time training cost amortized over inference cases.The authors consider the performance trade-off worthwhile.Source: Limitations · p. 10
• Appendix A.6 reports Inter. F1 as an additional metric and says trained reasoning paths are more consistent with GSM8K ground-truth paths.The passage presents this as performance insight beyond accuracy.Source: Limitations · p. 10

A.1 Training and Hyperparameter Details

• λ is selected from {0.1, 0.3, 0.5, 0.7, 0.9} using MATH validation accuracy with LLaMA-3-8B, then used across datasets.Duplicate reasoning paths and branches are removed before training to maintain exploration diversity.Source: §A.1 · p. 13
• For fairness, each method uses at most one accepted or reference correct-answer path per question, and preference methods also use at most one rejected wrong-answer path.DPO includes SFT loss on the accepted path; all methods use a single training stage.Source: §A.1 · pp. 13–14
• Figure 6 presents evaluation accuracy (%) for training methods on math reasoning datasets and indicates improvement over the highest-performing baseline.The caption does not provide the underlying accuracy values.Source: Figure 6 · p. 14
• Figure 7 presents evaluation accuracy on MMLU STEM questions and indicates improvement over the highest-performing baseline.The caption does not provide the underlying accuracy values.Source: Figure 7 · p. 14
• Training methods use similar amounts of data measured in reasoning paths, but sample counts differ because outputs are filtered for correctness.Source: §A.1 · p. 14

A.2 Dataset Details

• GSM8K and MATH use their original training and testing splits.Source: §A.2 · p. 14
• MMLU STEM uses 3,000 training and 375 testing questions drawn from three listed subcategories.The categories are college computer science, high school computer science, and high school mathematics.Source: §A.2 · p. 14

A.3 Prompting

• Four-shot chain-of-thought prompting generates paths for the reasoning-generation stage and the RFT, DPO, and ORPO baselines.Source: §A.3 · p. 14
• Table 5 lists hyperparameter and training details.Source: Table 5 · p. 15
• Table 6 compares training data across methods and models on GSM8K.Source: Table 6 · p. 15

GSM8K CoT Prompt

• The examples work through school-absence and bird-count word problems using arithmetic steps before giving final answers.Source: GSM8K CoT Prompt · p. 15
• Further examples show stepwise discount and ingredient-cost calculations for shirt and cookie word problems.Source: GSM8K CoT Prompt · p. 15
• Table 7 reports GSM8K reasoning-quality analysis using Inter. Precision, Recall, and F1 metrics.Source: Table 7 · p. 15

MATH CoT Prompt

• The MATH prompt examples work through a square-root expression's domain and determinant and weight calculations.Source: MATH CoT Prompt · pp. 15–16
• Additional MATH examples solve a linear-system ratio and show the corresponding stepwise derivation.Source: MATH CoT Prompt · p. 16

MMLU (STEM Subset) CoT Prompt

• The additional STEM prompt examples also include a proportional-voting question and a projectile-speed calculation.Source: MMLU (STEM Subset) CoT Prompt · p. 17
• The STEM prompt examples include worked solutions about animal adaptations, survey proportions, projectile speed, and capacitor energy.Source: MMLU (STEM Subset) CoT Prompt · pp. 16–17
• For GSM8K, the reported framework improves both the final benchmark score and reasoning quality measured by Inter. F1; Table 7 covers Inter. Precision, Recall, and F1.Source: MMLU (STEM Subset) CoT Prompt · p. 18

A.4 Instruction-Tuned Model Experiments

• Experiments report benefits for LLaMA-3-8B-Instruct, which has undergone general instruction tuning, alongside improvements on base models.Source: §A.4 · p. 18

A.5 Commonsense and General Reasoning

• On commonsense and general reasoning tasks, RPO surpasses other preference-optimization methods for prompted multi-step reasoning, with a 6.6% Winogrande improvement over ORPO.Source: §A.5 · p. 18

A.6 Evaluation of Reasoning Quality

• Inter. F1 compares numerical objects consistent between a generated reasoning path and the ground-...Source: §A.6 · p. 18