PaperlayerView exact revision on arXiv

Source-linked AI summary

Critique-out-Loud Reward Models

Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu

arXiv:2408.11791v1cs.LG

TL;DR

Classic reward models predict preference scores without using the LLM’s generation capabilities, leaving response-quality reasoning implicit. CLoud generates a critique before predicting a scalar reward, improving RewardBench pairwise accuracy and ArenaHard Best-of-N win rate over classic reward models.

  • Problem

    Classic reward models do not use the underlying LLM’s language-modeling head, so they must reason about response quality implicitly in a single forward pass.

    Source: §1 · p. 1

  • Method

    CLoud generates a natural-language critique of the response and conditions its scalar reward prediction on the prompt, response, and critique.

    Source: §1 · p. 2

  • Results

    CLoud improves RewardBench pairwise preference accuracy by 4.65 and 5.84 percentage points for 8B and 70B models, respectively, and Pareto-improves ArenaHard Best-of-N win rate over classic reward models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On-policy training is essential to CLoud’s success in preference classification and Best-of-N, while self-consistency benefits reasoning tasks.

    Source: §1 · p. 2, §3.3 · p. 8

Abstract

from arXiv · show

Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.

1 INTRODUCTION

Classic reward models reason implicitly in one forward pass; CLoud generates a critique before scoring, improving RewardBench accuracy and Pareto-improving ArenaHard Best-of-N win rate.

1 INTRODUCTION

  • Classic reward models do not use the LLM’s language-modeling head, so response-quality reasoning must occur implicitly in a single forward pass.LLM-as-a-Judge uses reasoning traces but generally underperforms classic reward models on pairwise preference classification.Source: §1 · p. 1
  • CLoud generates a detailed critique from the prompt and response, then predicts a scalar reward conditioned on all three while retaining the LLM’s language-modeling head.The reward model is trained to critique response quality before scoring.Source: §1 · p. 2, Figure 1 · p. 2
  • CLoud reward models improve RewardBench pairwise preference classification accuracy by up to 4.65 and 5.84 percentage points for 8B and 70B base models, respectively, and Pareto-improve ArenaHard Best-of-N win rate.Source: §1 · p. 2
  • On-policy training is essential to CLoud's success in both preference classification and Best-of-N.The paper identifies this as a key training design choice.Source: §1 · p. 2
  • Self-consistency marginalizes multiple critique–reward predictions, improving reasoning-task accuracy by up to 0.70 and 0.49 percentage points for 8B and 70B models, respectively.This trades added inference compute for improved reward modeling.Source: §1 · p. 2
  • Training first fine-tunes the LLM on oracle critiques, then replaces them with model-generated critiques before training the scalar reward head.The passage describes training on prompts, chosen and rejected responses, and critiques.Source: Figure 2 · p. 3

2 METHODS

The methods extend pairwise preference reward modeling with critique-conditioned scoring, training the model to generate critiques and score responses from them. They also describe self-consistency decoding to estimate rewards across multiple critiques.

2.1 CLASSIC REWARD MODELS

  • Classic reward models use a pretrained base model and shallow reward head to score prompt-response pairs, learning to score chosen responses above rejected ones under the Bradley-Terry model.The chosen and rejected responses are judged by a human or a more powerful model.Source: §2.1 · p. 3

2.2 CLOUD REWARD MODELS

  • CLoud training fine-tunes the base model and LM head on oracle critiques, replaces them with self-generated critiques, then trains a critique-conditioned reward head to reduce training–inference critique distribution shift.Oracle critiques approximate human critiques and are generated by Llama-3.1-405B-Instruct.Source: §2.2 · p. 4
  • The CLoud joint loss combines reward modeling and critique-generation losses, preserving critique generation while discouraging optimization solely for reward scores.A hyperparameter weights the language-modeling loss.Source: §2.2 · p. 4
  • Self-consistency samples multiple critiques for a prompt-response pair, predicts a reward for each, and averages those rewards to estimate the response’s reward.The method uses sampling temperature when generating critiques.Source: §2.2 · p. 4

3 RESULTS

Across Llama-3-8B and 70B experiments, CLoud improves RewardBench preference classification and ArenaHard Best-of-N performance over classic reward models, while added inference compute helps only some task distributions.

3.1 SETUP

  • Training uses prompts from UltraFeedback and UltraInteract, Llama-3-8B-Instruct response pairs labeled by Llama-3.1-405B-Instruct, and oracle-generated critiques.Evaluation uses RewardBench pairwise classification and ArenaHard Best-of-N win rate.Source: §3.1 · p. 5

3.2 COMPARING CLASSIC AND CLOUD REWARD MODELS

  • 4.65 and 5.84 percentage points: CLoud exceeds classic reward models in average RewardBench pairwise preference accuracy for 8B and 70B, respectively.CLoud significantly outperforms classic reward models on every category at both scales.Source: Figure 3 · p. 6, §3.2 · p. 6
  • At Best-of-16, CLoud improves ArenaHard win rate over classic reward models by 1.84 and 0.89 percentage points for 8B and 70B, respectively.Across response counts, CLoud is a Pareto improvement: its win rate is equal to or higher than classic reward models at both scales.Source: Figure 4 · p. 6, §3.2 · pp. 6–7
  • Training off-policy drops average RewardBench accuracy by 5.60 and 3.03 percentage points for 8B and 70B, respectively.On-policy training significantly outperforms off-policy training on every category except Chat-Hard at 70B.Source: Figure 5 · p. 7, §3.2 · p. 8
  • At Best-of-16, off-policy training decreases ArenaHard win rate by 3.31 and 1.56 percentage points for 8B and 70B, respectively.On-policy CLoud is a Pareto improvement over off-policy CLoud for Best-of-N win rate.Source: Figure 6 · p. 7, §3.2 · p. 8

3.3 SELF-CONSISTENCY FOR CLOUD REWARD MODELS

  • Up to 0.70 and 0.49 percentage points: self-consistency improves RewardBench reasoning accuracy for 8B and 70B, but not most other categories.It also provides no ArenaHard Best-of-N win-rate gain; the benefit depends on the task distribution.Source: Figure 7 · p. 8, §3.3 · pp. 8–9
  • Only reasoning problems estimated to require 1–2 steps show consistent accuracy gains as critique samples increase; problems requiring more steps decline after eight critiques.The reasoning horizon is approximated by the average sentence count of the chosen and rejected responses.Source: §3.3 · p. 9

4 RELATED WORK

Related work positions CLoud against classic reward modeling, self-generated and oracle critiques, and LLM-as-a-Judge, while identifying where critique-based scoring differs from these approaches.

Critique-based feedback

  • Self-consistency lifts performance for 1–2-step prompts as critique count rises, while other reasoning-step groups degrade beyond eight critiques.The comparison is framed by response reasoning horizon.Source: Figure 8 · p. 9

Classic reward models

  • CLoud's improvements are orthogonal to alternative preference objectives, and future work could combine it with more complex reward-model objectives.The cited approaches include ranking-based, direct pairwise-preference, and multi-objective reward models.Source: §4 · p. 9

Critique-based feedback

LLM-as-a-Judge

5 CONCLUSION

CLoud reward models generate critiques before scoring responses, improving RewardBench accuracy over classic reward models and yielding a Pareto improvement on ArenaHard Best-of-N win rate. Self-consistency benefits are limited to reasoning problems, especially responses with short reasoning horizons.

Conclusion

  • Up to 4.64 and 5.84 percentage points higher average pairwise preference modeling accuracy on RewardBench for 8B and 70B base models, respectively, versus classic reward models.The paper also reports a Pareto improvement over classic reward models for ArenaHard win rate under Best-of-N decoding.Source: §5 · p. 10
  • CLoud preserves the underlying LLM’s language modeling capabilities, generates a response critique, and then predicts a scalar reward.The reward depends on the prompt, response, and self-generated critique.Source: §5 · p. 10
  • Self-consistency benefits CLoud reward models only on reasoning problems and is predominantly useful for responses with short reasoning horizons.The method marginalizes over sampled critiques to estimate reward.Source: §5 · p. 10

Oracle Critique Generation Prompt

The oracle-critique prompt asks an expert to analyze how well an assistant response answers the user’s query.

Oracle Critique Generation Prompt

A GENERATING ORACLE CRITIQUES

Oracle critiques are generated to approximate human critiques using Llama-3.1-405B-Instruct, with the prompt shown in Figure 9.

A GENERATING ORACLE CRITIQUES

B TRAINING HYPERPARAMETER SWEEP

The appendix compares training settings for classic and CLoud reward models at 8B and 70B, then reports the best-performing settings for each.

B TRAINING HYPERPARAMETER SWEEP

C SELF-CONSISTENCY FOR BON

The section tests self-consistency reward scoring for Best-of-N on ArenaHard. It finds no win-rate difference from greedy scoring at either model size.

C SELF-CONSISTENCY FOR BON

D EXAMPLE REWARD PREDICTIONS ON REWARDBENCH

The appendix illustrates CLoud reward prediction on randomly sampled RewardBench examples from the chat and reasoning categories, for both 8B and 70B models.

D EXAMPLE REWARD PREDICTIONS ON REWARDBENCH

Loading 2408.11791v1…