PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Verifiers: Reward Modeling as Next-Token Prediction

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal

arXiv:2408.15240v3cs.LG

TL;DR

Existing discriminative verifiers score candidate solutions without using pretrained LLMs’ generation capabilities, limiting verification approaches for Best-of-N reasoning. The paper trains generative verifiers with next-token prediction, including verification rationales and joint solution generation, and reports stronger performance across reasoning tasks, including 73% → 93.4% on GSM8K.

  • Problem

    Discriminative verifiers score solutions but do not use pretrained LLM capabilities such as chain-of-thought reasoning and additional inference-time computation.

  • Method

    GenRM recasts verification as next-token prediction, optionally generating chain-of-thought rationales before a correctness token and jointly training on verification and solution generation.

  • Results

    GenRM outperforms discriminative RMs, DPO verifiers, LLM-as-a-Judge, and self-consistency across algorithmic and math reasoning tasks, including 73% → 93.4% on GSM8K.

  • Takeaways & Limitations

    Generative verifiers unlock chain-of-thought reasoning, majority-voting inference-time compute, and unified solution generation and verification within one LLM.

Abstract

from arXiv · show

Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional test-time compute via majority voting for better verification. We demonstrate that GenRM outperforms discriminative, DPO verifiers, and LLM-as-a-Judge, resulting in large performance gains with Best-of-N, namely 5% $\rightarrow$ 45.3% on algorithmic tasks and 73% $\rightarrow$ 93.4% on GSM8K. In easy-to-hard generalization settings, we observe improvements of 28% $\rightarrow$ 44.6% on MATH, and 37.9% $\rightarrow$ 53.5% on MMLU abstract algebra. Furthermore, we find that training GenRM with synthetic verification rationales is sufficient to pick out subtle errors on math problems. Finally, we demonstrate that GenRM scales favorably with model size and test-time compute.

1. Introduction

LLM verifiers are commonly trained as discriminative scorers, leaving pretrained language models’ generation capabilities underused. GenRM instead uses next-token prediction to support chain-of-thought verification, majority voting, and unified solution generation and verification.

  • Motivation: Best-of-N generates and ranks multiple candidate solutions, so verification accuracy is central to selecting a correct final answer.The verifier ranks candidates produced by the LLM and selects the most suitable one.
  • Motivation: Discriminative reward models score solutions numerically but do not use LLM capabilities such as instruction tuning, chain-of-thought reasoning, or extra inference-time computation.LLM-as-a-Judge instead prompts an off-the-shelf generative model, rather than fine-tuning a verifier from a pretrained LLM.
  • GenRM: GenRM represents correctness as the probability of a next-token response such as ‘Yes’ or ‘No’, while retaining language-generation abilities.The approach trains verifiers with next-token prediction and can jointly support verification and solution generation.
  • GenRM-CoT: GenRM-CoT generates a verification rationale before predicting correctness, and majority voting averages ‘Yes’ probabilities across sampled rationales.This adds inference-time computation to verification and is intended to improve verification accuracy.
  • Results: 73% → 93.4% on GSM8K: GenRM-CoT improved Best-of-N problem-solving performance using a Gemma2-9B verifier on Gemini 1.0 Pro solutions.The reported results also include gains on algorithmic tasks and easy-to-hard generalization settings.

2. Preliminaries

These preliminaries define autoregressive next-token prediction, supervised fine-tuning, Best-of-N selection, and discriminative verifier scoring. Best-of-N depends on a verifier ranking sampled solutions, while discriminative verifiers estimate correctness with a separate numerical score.

  • Autoregressive language modeling: An autoregressive language model generates an output sequence one token at a time from the input context and previously generated tokens.The conditional probability of each token depends on the preceding tokens and the input context.
  • Autoregressive language modeling: Higher temperature increases sampling randomness, whereas temperature 0 produces deterministic greedy decoding.The temperature controls how probabilities are transformed from token logits.
  • Supervised fine-tuning: Supervised fine-tuning minimizes cross-entropy between predicted next tokens and the target response on input-response pairs.The loss is computed over the target sequence under the model’s autoregressive predictions.
  • Best-of-N: Best-of-N samples N candidate solutions, scores them with a learned verifier, and selects the highest-scoring solution as the final answer.A better verifier increases the chance of selecting a correct solution and improving test accuracy.
  • Discriminative verifiers: Discriminative verifiers fine-tune an LLM as a classifier that assigns r_θ(x, y) ∈ [0, 1] to estimate solution correctness.They are trained on correct and incorrect solutions using binary cross-entropy and do not use the LLM’s text-generation capabilities.

3. GenRM: Verification as Next-Token Prediction

GenRM reframes verification as next-token generation, representing correctness with token probabilities while retaining LLM generation capabilities. Its CoT variant generates verification rationales, averages scores across multiple reasoning paths, and can train on synthetic rationales.

  • Direct verification: GenRM represents solution correctness with the probability of a “Yes” or “No” token instead of a separate numerical score.This preserves the verifier’s text-generation capabilities while making verification another token-prediction task.
  • Direct verification: At inference, the likelihood of the “Yes” token becomes the score used to rerank candidate solutions.Using likelihood incorporates confidence in the correctness prediction rather than relying only on a binary decision.
  • Unifying generation and verification: GenRM jointly trains verification and correct-solution generation by mixing Dverify with Dcorrect in the SFT loss.The mixture ratio is controlled by λ > 0, and the paper describes positive transfer between generating a correct solution and judging correctness.
  • Chain-of-thought verifiers: GenRM-CoT generates a rationale or critique before predicting correctness, allowing verification to identify subtle reasoning errors.The rationale may be human- or LLM-generated, and inference conditions the correctness score on the self-generated reasoning trace.
  • Chain-of-thought verifiers: Majority voting averages correctness scores across K verification rationales, mitigating errors in individual reasoning paths and adding inference-time compute.Unless otherwise specified, reported GenRM-CoT results use 32 votes.
  • Synthetic verification rationales: Synthetic verification rationales are explored on GSM8K to reduce the cost and difficulty of obtaining human-generated rationales.The paper generates and filters rationales according to verification correctness, using a reference solution to improve rationale quality.

4. Experiments

The experiments compare generative verifiers with standard approaches across algorithmic, mathematical, and generalization settings, evaluating Best-of-N and verifier accuracy. GenRM-CoT improves verification, sample efficiency, generation-verification synergy, and scaling with model size and test-time voting.

  • Experimental setup: The evaluation compares GenRM with discriminative RMs, DPO, LLM-as-a-Judge, and self-consistency across algorithmic and mathematical reasoning tasks.Metrics include Best-of-N performance and test RM accuracy, while experiments also examine unified training, chain-of-thought verification, and test-time compute.
  • Tasks: Algorithmic tasks test verification length generalization by training on word-list lengths {2,3,4} and evaluating on lengths {5,6}.The tasks are Last Letter Concatenation and Word Sorting from Big-Bench.
  • Generative Verifiers Outperform Standard Verification Approaches: GenRM outperforms LLM-as-a-Judge and DPO verifiers, while performing comparably or slightly better than discriminative verifiers; GenRM-CoT nearly matches oracle verification on algorithmic tasks.On GSM8K, GenRM-CoT consistently outperforms other methods and detects subtle reasoning errors missed by discriminative or direct GenRM verifiers.
  • Generative Verifiers Outperform Standard Verification Approaches: GenRM-CoT achieves 6.4× better sample efficiency than discriminative verifiers on easy-to-hard MATH generalization without training on MATH.It also surpasses self-consistency and generalizes from grade-school math to harder MATH and college-level MMLU mathematics.
  • Synergy Between Generation and Verification: Unifying solution generation and verification with next-token prediction consistently improves verification across tasks, although too much solution-generation data can decrease GenRM verification performance.Adding CoT verification data to the generator’s training mix also improves its oracle-verifier Best-of-N scores.
  • Scaling Model Size and Inference-time Compute: GenRM-CoT scales gracefully with test-time voting and model size, outperforming greedy decoding within 2 votes and surpassing LLM-as-a-Judge across Gemma scales.Larger models perform better in both reward-modeling accuracy and Best-of-N performance, with GenRM-CoT benefiting from stronger text-generation and chain-of-thought capabilities.

5. Related Work

Prior verification methods mainly use discriminative scoring or prompted judging, while GenRM trains generative verification directly and produces its own reasoning traces.

  • Conventional reward models train as discriminative classifiers to predict solution correctness or preferences.
  • LLM-as-a-Judge prompts untrained language models with grading rubrics, templates, or many-shot examples rather than specifically training them as verifiers.
  • GenRM trains verifiers with next-token prediction and uses model-generated chain-of-thought directly during verification.
  • GenRM unifies solution generation and candidate verification within the same generative model.

6. Conclusion & Future Work

The paper concludes that GenRM improves verification by recasting it as generation, supporting chain-of-thought and additional inference-time computation while unifying generation and verification.

  • GenRM recasts verification as next-token prediction and outperforms discriminative verifiers.
  • GenRM enables chain-of-thought reasoning and inference-time compute for better verification.
  • A single GenRM can be co-trained for solution generation and verification, benefiting both capabilities.
  • Future work: Future work includes broader tasks, process-level supervision, reinforcement learning for CoT verifiers, tool use, and integration into LLM reinforcement-learning pipelines.

A. Training Data Generation for Verifiers

The appendix describes synthetic verification data for algorithmic and math tasks, using ground-truth or model-generated rationales and a zero-shot judge evaluation.

  • Algorithmic tasks use ground-truth verification chain-of-thought to study generative verification without noisy rationale supervision.
  • Last Letter Concatenation and Word Sorting train on shorter word lists and evaluate verification generalization to longer lists.
  • Grade School Math uses generated solutions, sampling correct and incorrect examples for verifier training and evaluating on held-out solutions.
  • GSM training rationales are model-generated using another correct solution as privileged training information, while the same model generates solutions and rationales.
  • The LLM-as-a-Judge evaluation uses a zero-shot prompt based on Gemini 1.0 Pro.

B. Hyper-parameters for Verifier Training

Verifier training uses balanced correctness data, optimizer and learning-rate schedules for GenRM, broad hyperparameter sweeps for discriminative models, and selected DPO settings.

  • Gemma-based verifiers select checkpoints by held-out validation accuracy and balance training data between 50% correct and 50% incorrect solutions.
  • GenRM verifiers: GenRM generally performs best with learning rate 2e-6, weight decay 1e-2, Adam, gradient clipping 1.0, warmup, and cosine decay.
  • Discriminative RMs: Discriminative reward models use special-token logits and sweep learning rates, weight decay, dropout, and learning-rate schedules.
  • DPO: DPO initializes from an SFT reference policy, trains on correct-incorrect solution pairs, and uses learning rate 1e-6 with β=0.1.

C. Additional Details

This section describes filtering synthetic verification rationales and a top-K scoring variant for Weighted Self-Consistency.

  • Synthetic verification CoT: Synthetic verification rationales are filtered using agreement with an answer checker to mitigate false negatives and false positives.Only rationales from solutions meeting the stated agreement criterion are retained.
  • Synthetic verification CoT: More than 50% of verification rationales must agree with the answer checker's correctness judgment for retention.
  • Weighted Self-Consistency: Weighted Self-Consistency slightly improves performance by summing only the top-K verifier scores for each answer.The method uses K=6 for GSM and K=4 for MATH.

D. Additional Results

Additional experiments examine training-loss weighting, data scaling, verifier comparisons, self-consistency, and DPO reward scoring across tasks.

  • Generation loss: Adding too much solution-generation data negatively impacts verification, while intermediate generation-loss weights perform best.Default weights are λ=1/3 for algorithmic tasks and λ=1/4 for GSM8K.
  • Data scaling: Increasing solutions per problem from 8 to 32 improves GenRM-CoT RM accuracy and Best-of-N accuracy.For GSM8K data scaling, both rationales per solution and solutions per problem are varied.
  • Verifier comparisons: GenRM without CoT performs slightly better than or comparably to Discriminative RM across tasks, while outperforming DPO verifiers.
  • Weighted Self-Consistency: On GSM8K, Weighted Self-Consistency increases solved problems only slightly from 93.4% to 93.5% with 16 solutions.The passage attributes the limited gain to saturated improvement potential.
  • DPO reward ablation: For DPO verifiers, using the final policy's sequence log probability directly performs better than subtracting the reference-policy log probability.

E. Examples Verification rationales from GenRM-CoT: GSM8K Test and MATH500

The appendix presents GenRM-CoT verification-rationale examples for GSM8K and transferred MATH evaluations, alongside a majority-voting score example.

  • GSM8K Test: Tables E.1–E.4 provide four GenRM-CoT verification-rationale examples.
  • Majority voting: A Gen-RM majority-voting example reports a score of 0.3267.
  • MATH500: Tables E.15–E.16 show GenRM-CoT examples for MATH transferred from GSM.
Loading 2408.15240v3…