Source-linked AI summary
V-STaR: Training Verifiers for Self-Taught Reasoners
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, Rishabh Agarwal
TL;DR
Existing self-improvement methods discard many incorrect generated solutions, limiting their use of available training information. V-STaR uses correct and incorrect solutions to iteratively train a generator and DPO verifier, then ranks candidates at inference time. Across math and code-generation benchmarks, it improves test accuracy over prior self-improvement and verification methods.
Problem
Existing self-improvement methods discard incorrect generated solutions, even though these solutions can contain information about discrepancies and error patterns.
Method
V-STaR iteratively trains a generator on correct solutions and a DPO verifier on correct and incorrect generated solutions, using the verifier to rank candidates at inference time.
Results
6% to 17% absolute test-accuracy gains are reported in math reasoning and 4% to 12% in code generation over prior self-improvement and verification methods.
Takeaways & Limitations
V-STaR improves existing self-improvement and verification approaches by jointly using iteratively generated correct and incorrect solutions to train stronger generators and verifiers.
Takeaways & Limitations
The study did not test whether including the verifier in the training loop helps on other tasks or with different generator sampling strategies.
Abstract
from arXiv · showhide
Common self-improvement approaches for large language models (LLMs), such as STaR, iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability. However, these approaches discard the large amounts of incorrect solutions generated during this process, potentially neglecting valuable information in such solutions. To address this shortcoming, we propose V-STaR that utilizes both the correct and incorrect solutions generated during the self-improvement process to train a verifier using DPO that judges correctness of model-generated solutions. This verifier is used at inference time to select one solution among many candidate solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers, delivering a 4% to 17% test accuracy improvement over existing self-improvement and verification approaches on common code generation and math reasoning benchmarks with LLaMA2 models.
1 Introduction
Existing self-improvement methods retain correct self-generated solutions but discard many incorrect ones. V-STaR uses both types to train generators and DPO verifiers iteratively, improving selection among generated candidates.
- Self-improvement methods such as STaR, RFT, and ReSTEM fine-tune LLMs on self-generated correct solutions while discarding incorrect solutions.This is especially data-inefficient for challenging reasoning tasks, where incorrect generations can comprise a large portion of outputs.
- V-STaR trains a generator on correct solutions and a DPO verifier on pairs of correct and incorrect generated solutions collected during iterative self-improvement.The verifier ranks multiple generator outputs at test time and selects the highest-ranked candidate.
- V-STaR evaluates math reasoning on GSM8K and MATH and code generation on MBPP and HumanEval using LLaMA2 and CodeLLaMA models.
- 6% to 17% absolute accuracy gains are reported for math reasoning, while code-generation gains range from 4% to 12% over prior self-improvement and verification methods.The reported comparisons cover the evaluated math and code-generation benchmarks.
- V-STaR outperforms prior self-improvement and ORM verification methods and better uses adaptive test-time compute than several strong baselines.The comparison includes RFT, STaR, ORM verification, RFT plus a verifier, and self-consistency.
2 Preliminaries
The preliminaries describe supervised fine-tuning, self-improvement, test-time verification, and preference learning as foundations for V-STaR. These approaches respectively train generators, retain correct completions, rank candidates, or optimize preferences between preferred and dispreferred outputs.
- Preliminaries: Supervised fine-tuning trains a causal language model on problem-solution pairs using a negative log-likelihood objective.The problem input may describe a task, while the solution may be a rationale or generated code.
- Self-improvement approaches: Self-Taught Reasoner iteratively generates solutions, checks binary correctness using answers or test cases, and retains correct completions for subsequent training.
- Self-improvement approaches: RFT augments the original dataset with correct generated completions but is not iterative, whereas STaR† applies sampled completions across iterations.
- Test-time verification: Outcome-supervised verifiers score candidate correctness, and Best-of-k selects the highest-ranked solution from multiple generator outputs.Verifier training data contains generated solutions paired with binary correctness labels.
- Test-time verification: ORM verifiers combine language-modeling and binary-classification objectives when trained on correctness-labeled generated solutions.
- Evaluation setup: Figure 2 compares 7B V-STaR with baselines using Best-of-64 for verification methods and Pass@1 for other methods across in-domain and transfer tasks.The setup uses K = 48 generations per problem, with STaR† and V-STaR run for three iterations using 16 samples per iteration.
- Preference learning with DPO: DPO increases the reference-relative likelihood of preferred completions over dispreferred completions without using a separately trained reward model during fine-tuning.
3 V-STaR: Verifiers for self-taught reasoners
V-STaR uses iteratively generated correct and incorrect solutions to improve both a generator and a verifier. The verifier is trained with DPO to rank candidates at inference, while iterative collection yields stronger training data and broad benchmark gains.
- V-STaR training procedure: V-STaR retains correct and incorrect generated solutions to train a better generator and verifier iteratively.Correct solutions augment generator data, while both correctness classes form verifier data so it can learn from generator mistakes.
- Empirical results: Iterative data collection produced better verifier-training distributions and quality than a non-iterative approach with the same generation budget.The comparison is against the V-STaR [1 iter] baseline.
- Training verifiers with DPO: DPO trains the verifier to increase the likelihood of correct solutions and decrease the likelihood of incorrect solutions for each problem.Preference pairs are constructed from the Cartesian product of correct and incorrect solutions, with correct solutions preferred.
- Inference-time selection: At inference, the trained verifier scores and ranks generated candidate solutions so the highest-scoring completion is selected.Best-of-64 uses the verifier on candidate answers sampled from the generator.
4 Empirical results
V-STaR is evaluated on math and code-generation tasks, including out-of-domain transfer, using iterative generator–verifier training and Best-of-k evaluation. Across benchmarks, it consistently improves over baselines, while results also examine candidate scaling, iteration gains, verifier training, and training-loop design.
- Evaluation setup: V-STaR is trained on GSM8K and MBPP, then evaluated on GSM8K, a MATH subset, MBPP, and HumanEval using LLaMA2 and CodeLLaMA models.The MATH subset contains 150 Level 1 problems; models use LoRA fine-tuning.
- Evaluation setup: At inference, generators produce 128 candidates per problem; evaluation reports Pass@1 for generators and Best-of-64 for verifier-based methods.Best-of-64 and self-consistency use the proposed Best-of-k estimator.
- Main results: 6% to 17% absolute accuracy gains occur over prior self-improvement and verification methods in math, while code-generation gains range from 4% to 12%.The reported gains cover GSM8K, MBPP, the MATH subset, and HumanEval comparisons.
- Main results: Iterative V-STaR outperforms baselines and V-STaR [1 iter] on both out-of-domain tasks across model sizes.Models trained on GSM8K transfer to MATH, while models trained on MBPP transfer to HumanEval.
- Candidate scaling: Best-of-k accuracy saturates for k ≥16, while the gap between V-STaR and V-STaR [1 Iter] remains consistent.The comparison uses 7B models and 128 candidate solutions per test problem.
- Candidate scaling: V-STaR’s Best-of-k performance remains stronger than self-consistency for k ≤64, although the gap slightly decreases at larger k.The comparison is reported on GSM8K using 7B models and up to 1,000 generated solutions.
- Verifier analysis: Best-of-64 improves substantially after DPO verifier training, but the verifier’s generation ability degrades after only 2k updates.This contrasts verifier effectiveness with its retained usefulness as a generator.
- Iteration analysis: On MBPP, gains are larger across verifier iterations than generator iterations, highlighting the importance of iterative verifier improvement.The study reports improvements across iterations without signs of collapse.
5 Related work
Related work improves LLM reasoning through self-generated training data and learned verifiers that assess solution correctness. These approaches target multi-step reasoning while balancing solution quality, supervision, exploration, and test-time computation.
- Self-training and self-improvement: Self-improvement methods fine-tune LLMs on generated solutions that produce correct answers.This family includes STaR, reinforced self-training, rejection fine-tuning, and related approaches.
- Self-training and self-improvement: Self-improvement faces a difficult exploration problem because successful solutions must be discovered for fine-tuning.Prior work also questions whether LLMs can reliably correct their own reasoning.
- Training verifiers: Verifier research studies outcome supervision, process supervision, step-level rewards, combined verifiers, and auxiliary execution information.Other work also frames rationale generation as graph search guided by verifier-estimated step quality.
6 Conclusion
V-STaR uses correct and incorrect solutions from an iteratively trained generator to train a verifier, and reports stronger performance than prior approaches. The method is presented as simple, data-efficient, and applicable where correctness feedback is available during training.
- Conclusion: V-STaR uses correct and incorrect generated solutions from an iteratively trained generator to train a strong verifier.The approach is described as data-efficient and simple to implement.
- Conclusion: DPO is reported to train verifiers more effectively than the common method of Cobbe et al. (2021).
- Conclusion: V-STaR outperforms existing self-improvement and verification-based methods and may improve self-improvement loops when training correctness feedback is available.
A Algorithm
The algorithm begins with supervised data and iteratively generates, labels, and stores candidate solutions. It uses correct solutions for generator training and all labeled solutions to form preference pairs for DPO verifier training.
- A Algorithm: Each iteration fine-tunes a generator, samples k candidates for training queries, labels their correctness, and adds the labeled data to the generation buffer.
- A Algorithm: Correct solutions enter the generator buffer, while all generated solutions enter the verifier buffer for preference-pair construction.
- A Algorithm: The verifier is obtained by applying DPO to the SFT generator using preference pairs from the verifier buffer.
- A Algorithm: Training data-generation prompts include Python function-completion tasks with assertion tests and generated code solutions.
- A Algorithm: The prompt examples cover finding similar tuple elements, identifying non-prime numbers, and returning the largest integers from a list.
C Test accuracy of 13B V-STaR and baselines
Figure 8 compares 13B V-STaR with baselines using different accuracy protocols for verification-based and other methods. It also separates training-task results from transfer evaluations.
- C Test accuracy of 13B V-STaR and baselines: Verification-based methods are evaluated with Best-of-64, whereas other methods are evaluated with Pass@1.
- C Test accuracy of 13B V-STaR and baselines: The left panel reports test accuracy on training tasks.
- C Test accuracy of 13B V-STaR and baselines: The right panel reports transfer evaluations from GSM8K-trained models to a MATH subset and from MBPP-trained models to HumanEval.
D Candidate solutions ˆy1 and ˆy2 for a GSM8K problem x
The GSM8K problem asks for the distance from Andy’s house to the market after a 50-meter trip to school and return. Two candidate solutions produce different answers by interpreting the route differently.
- The problem states that Andy walks 50 meters from home to school and travels 140 meters in total before reaching the market.
- Candidate y1 counts the trip to school and back as 100 meters, leaving 40 meters to the market.
- Candidate y2 subtracts only the initial 50-meter walk from the 140-meter total, yielding 90 meters from school to the market.
E Example for Best-of-k using Eq. 3
The example contrasts a V-STaR solution with a majority-voting solution for calculating Wendi’s final chicken-feed meal. The solutions produce different answers because they use different interpretations of the feeding information.
- The problem describes 20 chickens receiving three cups of feed each per day, with 15 cups given in the morning and 25 in the afternoon.
- The V-STaR solution computes 60 cups daily, subtracts 40 cups already given, and answers 20 cups for the final meal.
- The majority-voting solution treats the 40 cups already distributed as the total amount and answers 2 cups per chicken.
G Best-of-k using Eq. 3 vs. Best-of-k from Lightman et al. (2024)
Figure 10 states that Eq. 3 estimates Best-of-k accuracy efficiently and without variance, whereas the formula-free process is repeated 32 times.
- Eq. 3 estimates Best-of-k accuracy efficiently and without variance.
- Without the formula, Best-of-K accuracy is obtained by repeating the process 32 times.