Source-linked AI summary
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, Jiaya Jia
TL;DR
Long-chain mathematical reasoning requires accurate intermediate steps, while DPO struggles to identify errors inside incorrect answers. Step-DPO optimizes preferences over individual reasoning steps and constructs in-distribution pairs, achieving reported gains on mathematical benchmarks.
Problem
DPO offers limited benefits for long-chain mathematical reasoning because it struggles to identify detailed errors in incorrect answers.
Method
Step-DPO treats individual reasoning steps as preference units and constructs step-wise pairs by identifying erroneous steps and sampling correct replacements.
Results
Step-DPO improves performance across mathematical reasoning evaluations, including 70.8% on MATH and 94.0% on GSM8K for Qwen2-72B-Instruct.
Takeaways & Limitations
The results support fine-grained process supervision and in-distribution, self-generated preference data for improving long-chain mathematical reasoning.
Takeaways & Limitations
Some samples require additional filtering because correct final answers may contain erroneous intermediate reasoning steps.
Abstract
from arXiv · showhide
Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by learning from human feedback. However, Direct Preference Optimization (DPO) has shown limited benefits for long-chain mathematical reasoning, as models employing DPO struggle to identify detailed errors in incorrect answers. This limitation stems from a lack of fine-grained process supervision. We propose a simple, effective, and data-efficient method called Step-DPO, which treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Additionally, we have developed a data construction pipeline for Step-DPO, enabling the creation of a high-quality dataset containing 10K step-wise preference pairs. We also observe that in DPO, self-generated data is more effective than data generated by humans or GPT-4, due to the latter's out-of-distribution nature. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at https://github.com/dvlab-research/Step-DPO.
2 RELATED WORKS
Prior work improves mathematical reasoning through SFT data augmentation, continued pre-training, reinforcement learning, and step-by-step verification, but DPO provides only marginal benefits for mathematical reasoning despite simplifying alignment.
- SFT data augmentation methods improve reasoning abilities, but performance plateaus once the data reaches a certain volume.
- Continued pre-training on extensive, high-quality math datasets markedly improves mathematical reasoning capabilities.
- Reinforcement learning has been explored to mitigate hallucinations in mathematical reasoning, with several works emphasizing step-by-step verification.
- RLHF aligns models using reward-model and policy-model training, but performance depends heavily on reward-model quality and the pipeline is complex.
- DPO streamlines alignment by directly optimizing models with pair-wise preference data, yet offers only marginal benefits for mathematical reasoning.
3 STEP-DPO
Step-DPO applies preference optimization to individual reasoning steps, targeting the first error rather than rejecting entire answers. Its data pipeline constructs in-distribution step-wise preference pairs by locating errors and sampling correct replacements from the reference model.
- STEP-WISE FORMULATION: DPO can discard preceding correct reasoning steps because errors often appear midway through otherwise undesirable answers.This introduces training noise and limits benefits for long-chain mathematical reasoning.
- STEP-WISE FORMULATION: Step-DPO treats each intermediate reasoning step as the preference unit and targets the first erroneous step.The method provides granular supervision to locate, rectify, and avoid erroneous steps.
- STEP-WISE FORMULATION: Given a prompt and preceding correct steps, Step-DPO maximizes the probability of a correct next step and minimizes the probability of an incorrect one.
- IN-DISTRIBUTION DATA CONSTRUCTION: Each preference sample contains a prompt, preceding reasoning steps, a preferred step, and an undesirable step.
- IN-DISTRIBUTION DATA CONSTRUCTION: The construction pipeline collects erroneous answers, localizes the first incorrect step, and rectifies it with a correct step sampled from the reference model.Collected answers are filtered against ground-truth final answers before the preferred step is selected.
- IN-DISTRIBUTION DATA CONSTRUCTION: Samples may require additional filtering because an answer can be correct overall while containing erroneous intermediate reasoning steps.This filtering can be performed manually or with GPT-4.
- IN-DISTRIBUTION DATA CONSTRUCTION: Human or GPT-4 rectifications are out-of-distribution for the reference model, whereas self-generated in-distribution steps are more effective preferred answers.The paper attributes this preference to lower reference-model log-probability for out-of-distribution outputs and gradient decay issues.
4 EXPERIMENTS
Experiments evaluate Step-DPO across standard and competition-level math benchmarks, model types, and ablations. Results show strong gains from step-wise preference optimization, especially with in-distribution data and larger models.
- Experimental Setup: The experiments use Qwen2, Qwen1.5, Meta-Llama-3-70B, and DeepSeekMath models, evaluated mainly on MATH and GSM8K.The study also evaluates AIME 2024 and Odyssey-MATH competition problems.
- Experimental Setup: The Step-DPO phase uses 10K pair-wise preference data constructed from remaining SFT data and AQuA.The data construction process described in the paper produces the preference pairs used for Step-DPO training.
- Main Results: 70.8% MATH and 94.0% GSM8K accuracy are achieved by Step-DPO applied to Qwen2-72B-Instruct, surpassing several closed-source models.The cited results compare against GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro.
- Main Results: Approximately 3% MATH improvement is obtained for models exceeding 70B parameters after applying Step-DPO to SFT models.The reported models include Llama-3-70B-SFT and Qwen-2-72B-SFT.
- Main Results: 50.1% accuracy is achieved on Odyssey-MATH by Step-DPO applied to Qwen2-72B-Instruct, narrowing the gap with GPT-4o.The same Step-DPO training data is used for these competition-level problems.
- Ablation Study: Compared with vanilla DPO, Step-DPO provides larger benefits for mathematical reasoning when trained with 5K Step-DPO data.The ablation compares judging preferred versus undesirable outputs, reward margins, and final MATH performance.
- Ablation Study: In-distribution self-generated data is more effective than out-of-distribution data generated by GPT-4 for Step-DPO.The comparison is reported in the ablation study as a test of the data construction process.
- Demonstrations: Step-DPO corrects minor mistakes in previous models in the demonstrated comparison.The paper presents this comparison between Qwen2-72B-Instruct and Qwen2-72B-Instruct-Step-DPO.
5 CONCLUSION
The paper proposes Step-DPO as a data-efficient approach for long-chain mathematical reasoning. It optimizes individual reasoning steps and uses a 10K-pair dataset to improve performance, particularly for large models.
- 5 CONCLUSION: Step-DPO uses a single reasoning step as the fundamental unit for preference comparison instead of holistic answers.This provides fine-grained process supervision and supports localization of errors in incorrect answers.
- 5 CONCLUSION: A data construction pipeline creates 10K preference data pairs for Step-DPO.The conclusion describes the method and dataset as simple, effective, and data-efficient.
- 5 CONCLUSION: Step-DPO produces significant improvements, particularly for large models, when applied to long-chain reasoning problems.The paper presents these results as evidence that Step-DPO and its 10K dataset improve mathematical reasoning.