Source-linked AI summary
Iterative Reasoning Preference Optimization
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, Jason Weston
TL;DR
Reasoning-focused iterative preference optimization has typically lagged behind its gains for general instruction tuning. The paper introduces Iterative RPO, which repeatedly pairs correct and incorrect generated CoT responses and trains with DPO plus an NLL term. Across GSM8K, ARC-Challenge, and MATH, the method reports large accuracy improvements using training-set examples without extra data.
Problem
Iterative preference optimization methods typically make little improvement on reasoning tasks despite performing well for general instruction tuning.
Method
Iterative RPO repeatedly generates CoT candidates, pairs correct-answer winners with incorrect-answer losers, and trains with DPO plus NLL on winners.
Results
Iterative RPO improves accuracy across GSM8K, ARC-Challenge, and MATH and outperforms SFT, standard DPO, and other baselines.
Takeaways & Limitations
The experiments support a simple iterative recipe for improving LLM reasoning across the three evaluated tasks without human-in-the-loop or extra training data.
Takeaways & Limitations
The approach assumes correctness judgments for final answers, using gold labels and exact-match rewards in the experiments, not judgments of reasoning-step correctness.
Abstract
from arXiv · showhide
Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
1 Introduction
Iterative preference optimization has worked well for instruction tuning but has typically produced little improvement on reasoning tasks. Iterative RPO addresses this gap by repeatedly optimizing preferences between generated CoT candidates, yielding substantial gains across three reasoning benchmarks.
- Iterative preference methods improve instruction tuning but typically make little improvement on reasoning tasks.
- Iterative RPO generates multiple CoT candidates and answers, then pairs correct-answer responses as winners against incorrect-answer responses.
- The method trains with a modified DPO objective that adds an NLL term for winning reasoning responses.
- The generation and preference-optimization steps are repeated, using each updated model to improve reasoning on the next iteration.
2 Iterative Reasoning Preference Optimization
Iterative RPO assumes answer-correctness judgments for generated CoT responses, constructs winner–loser preference pairs, and trains successive models with DPO plus NLL. The updated model regenerates training data for the next iteration, while the setup uses fixed prompts and can rely on exact-match rewards.
- Initialization: The method assumes a base language model, training inputs, and a correctness measure for final answers rather than reasoning steps.
- Iterative training: Successive iterations regenerate preference data from the updated model, while experiments reuse a fixed prompt set and do not require a sophisticated reward model.
- Chain-of-thought & answer generation: Each current model generates multiple CoT-and-answer responses for every training input.
- Chain-of-thought & answer generation: In the experiments, binary rewards equal one for an exact match to the training answer and zero otherwise.
- Preference optimization: Preference pairs assign higher-reward responses as winners and lower-reward responses as losers.
- Preference optimization: The next model is initialized from the current model and trained with DPO on pairs plus length-normalized NLL on winning responses.
- Related work: Unlike STaR's SFT approach, preference optimization incorporates negative reasoning chains and answers into training.
3 Experiments
Across GSM8K, ARC-Challenge, and MATH, Iterative RPO improves accuracy over repeated iterations and outperforms SFT and standard DPO baselines. The experiments also show that the NLL term and iterative model updates are important, while multiple-choice answer spaces can introduce noise into generated CoT preferences.
- Test-time sampling: 88.7% GSM8K accuracy is achieved with majority voting over 32 samples, compared with 81.6% from a single generation at iteration 4.For ARC-Challenge, majority voting reaches 87.9%; for MATH, it reaches 29.1%.
- Results across tasks: Iterative RPO improves accuracy across GSM8K, ARC-Challenge, and MATH over repeated iterations.Reported sequences include 73.1% to 78.0% to 81.1% to 81.6% on GSM8K, 84.8% to 86.2% to 86.7% on ARC-Challenge, and 17.7% to 19.9% to 20.8% on MATH.
- 3.1 Math Word Problems: GSM8K: Doubling paired data improves first-iteration performance, but iterative model updates improve performance more than simply adding data.For STaR, twice as much data reaches 66.9% versus 65.2% with the original data, still below Iterative RPO.
- 3.1 Math Word Problems: GSM8K: 73.1% vs. 61.8%: standard DPO performs worse than Iterative RPO after one iteration, supporting the need for the additional NLL loss term.The performance gap remains when standard DPO starts from the SFT-tuned model.
- 3.2 ARC-Challenge Task: ARC-Challenge multiple-choice answers can be correct by luck, creating potential noise in the CoT generations used for preference optimization.With typically four answer choices, random guessing is correct 25% of the time.
4 Related Work
The work is positioned among iterative alignment methods and reasoning-improvement approaches, differing by how preference data are generated and optimized. Unlike several prior methods, it trains the reasoning generator directly with pairwise preferences.
- SPIN uses human-annotated winning responses and previous model generations as losing responses, whereas this work generates both winning and losing Chain-of-Thoughts from final-answer correctness.
- Iterative RLHF and offline preference methods improve general instruction following, but prior iterative approaches report only modest gains on reasoning tasks.
- STaR, Expert Iteration, and ReSTEM repeatedly filter or generate high-quality reasoning samples for SFT-like training rather than using pairwise preference optimization.
- V-STaR applies DPO to train a verifier that filters SFT generations, while this work applies DPO directly to train the reasoning generator.
5 Conclusion
The paper proposes Iterative Reasoning Preference Optimization for chain-of-thought reasoning, using correctness-based preference pairs and a modified DPO objective. Experiments report large improvements across GSM8K, MATH, and ARC-Challenge without human-in-the-loop supervision or extra training data.
- Iterative RPO generates multiple responses, forms preference pairs from final-answer correctness, and trains with modified DPO including an additional NLL term.
- Iterative RPO improves chain-of-thought reasoning performance on GSM8K, MATH, and ARC-Challenge using the same base model and training data as its baselines.
- The method does not require human-in-the-loop labeling or extra training data and is described as simple and efficient to implement.
A Limitations
The limitations concern data reuse, task scope, and the need for answer-based supervision. The approach is evaluated on three tasks and depends on correct answers plus a clear response-comparison metric.
- Each iteration discards preference data collected in previous iterations, although reusing those data could potentially improve performance.
- The experiments cover only three tasks, leaving performance on general instruction tuning tasks without a clear best answer unclear.
- The recipe requires correct answers and a clear metric for comparing generated responses with those answers.
B.1 Prompts
The prompts ask models to provide step-by-step reasoning followed by a consistently formatted final answer. Task-specific templates differ in answer formatting and, for MATH, include fixed in-context demonstrations.
- GSM8K: GSM8K prompts combine a question with solution and final-answer fields, using explicit placeholders for demonstrations or the problem to solve.
- ARC: Alternative ARC prompt variants retain step-by-step reasoning but use a less constrained final-answer format.
- MATH: MATH prompts ask for step-by-step reasoning, require the answer to be wrapped in \boxed, and conclude with a final-answer format.
- MATH: The MATH prompt includes four fixed in-context training examples so final answers can be formatted correctly in LATEX.
- ARC: ARC prompts request step-by-step reasoning and conclude with a final answer, typically selecting among four labeled options.