PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Reward Models with Synthetic Critiques

Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, Matthias Gallé

arXiv:2405.20850v2cs.CL

TL;DR

Reward models are important to RLHF, but preference annotations are costly and training can be difficult to interpret and generalize. This paper conditions reward models on LLM-generated critiques of completions; experiments report stronger performance, particularly with high-quality critiques and limited data, and estimate that one such pair can match 40 standard preference pairs.

  • Problem

    Reward-model training relies on human preference annotations and faces concerns about interpretability, noisy or biased labels, and whether models learn preferences rather than superficial features.

    Source: §1 · pp. 1–2

  • Method

    The method generates natural-language critiques for prompt-completion pairs and trains reward models to predict scalar rewards conditioned on those critiques.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    A high-quality critique-enhanced preference pair can be as valuable as 40 standard preference pairs, and critiques are especially effective in low-data settings and on reasoning- and adversarial-skill tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors characterize synthetic critiques as an accessible, cost-effective way to improve reward models, particularly when critiques are high quality and data are limited.

    Source: §6 · p. 9

    The study evaluates reward models on ranking test accuracy but does not test optimizing language models with critique-enriched reward models.

    Source: §7 · p. 9

Abstract

from arXiv · show

Reward models (RMs) play a critical role in aligning language models through the process of reinforcement learning from human feedback. RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation. Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalization performance on unseen distributions. We propose a novel approach using synthetic natural language critiques generated by large language models to provide additional feedback, evaluating aspects such as instruction following, correctness, and style. This offers richer signals and more robust features for RMs to assess and score on. We demonstrate that high-quality critiques improve the performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. Furthermore, incorporating critiques improves both the interpretability and robustness of RM training.

1 Introduction

Reward models serve as proxies for human preference in RLHF, but their training can be costly, difficult to interpret, and vulnerable to superficial features and distribution shifts. The paper proposes critique-conditioned reward models and reports stronger performance, especially with limited data and high-quality critiques.

1 Introduction

  • RLHF uses reward models as proxies for human assessment, with a trained model scoring competing completions to estimate which a user is likely to prefer.The introduction notes that Llama 2 used one million binary preference datapoints to train its reward model.Source: §1 · p. 1
  • Reward-model training raises concerns about interpretability, noisy or biased human preference labels, and whether models learn preferences or superficial features.The cited discussion also notes possible length bias and mismatch between reward predictions and human preferences at inference.Source: §1 · pp. 1–2
  • Synthetic critiques explicitly assess response quality across dimensions such as correctness and instruction adherence, and reward models trained on them predict scalar rewards.The proposed pipeline generates critiques for each prompt-completion pair, then trains and evaluates reward models using critique-enriched data.Source: §1 · p. 2
  • A high-quality critique-enhanced preference pair can be as valuable as 40 standard preference pairs, with critiques especially effective in low-data settings and on reasoning- and adversarial-skill tasks.The introduction also describes synthetic critique generation as accessible and cost-effective because open-source models can generate critiques efficiently.Source: §1 · p. 2

2 Related Work

Related work covers reward-model training and evaluation, known behavior and interpretability concerns, and uses of critiques as feedback for models and evaluators.

2 Related Work

  • Reward models are commonly trained on annotator preference data, while alternatives include direct preference optimization and prompting generative models to judge or score responses.The section also describes Bradley–Terry preference modeling as a common approach.Source: §2 · p. 3
  • REWARDBENCH evaluates reward models on completion pairs with subtle preference distinctions and reports that models struggle more with reasoning than general chat.Related concerns include length bias and limited interpretability in reward modeling.Source: §2 · p. 3
  • Prior work uses critiques and rationales to support model development, training, evaluation, self-improvement, and human annotation.Related evaluator work reports improved accuracy from rationale-and-rubric prompting and from training a judge to generate critiques.Source: §2 · p. 3

3 Methods

The method generates LLM critiques for individual completions and trains reward models to rank critique-enriched preference pairs. It compares critique-conditioned models with a no-critiques baseline.

3.1 Synthetic Critique Generation with LLMs

  • LLMs generate a point-wise critique for each chosen or rejected completion, assessing how well it meets prompt requirements, including instruction-following, truthfulness, and helpfulness.This produces critique-augmented training examples of the form (x, y, c).Source: §3.1 · pp. 3–4

3.2 Training RMs with critiques

  • The no-critiques baseline takes a prompt and completion and trains a scalar reward model with a binary ranking loss and a preference-rating margin.The margin is larger for responses with distinct ratings and smaller for similarly rated responses.Source: §3.2 · p. 4
  • Critiques are concatenated after each completion in the preference pair, and the reward model is trained on the augmented pair; inference also uses critique-enriched test data.The reward model replaces a pretrained language model’s final projection layer with a linear layer that predicts a logit.Source: §3.2 · p. 4

4 Experimental Setup

The experiments compare reward models across preference and benchmark datasets, critique generators, pretrained initializations, and a shared training setup with a separate learning-rate sweep for one initialization.

4.1 Datasets

  • The setup evaluates reward models using test accuracy, the proportion of preference pairs where the chosen completion receives the higher score.The training preference dataset contains 5,000 open-ended, multi-turn conversation examples with human preference labels.Source: §4.1 · p. 4
  • Evaluation covers REWARDBENCH’s chat, chat-hard, safety, and reasoning categories, as well as the PandaLM dataset.REWARDBENCH contains 2,985 prompts; PandaLM contributes 762 prompts with labels independently provided by three human evaluators.Source: §4.1 · p. 5

4.2 LLM Critique Generator

  • Six LLMs with varied architectures, sizes, and training data generate critiques for both training and test sets, matched by generator within each experiment.The generator pool includes LLaMA2-7B-Chat, LLaMA2-70B-Chat, Mixtral-8x7B-Instruct, Command R, Command R+, and GPT4-Turbo.Source: §4.2 · p. 5

4.3 Pretrained Models for RM initialization

  • Reward models are initialized from LLaMA2-7B-Base, Command-35B-Base, or Command R (35B).Command-35B-Base is the checkpoint from which Command R was obtained with further fine-tuning.Source: §4.3 · p. 5

4.4 Training Details

  • Models generally train for one epoch with batch size 32 and 155 steps; LLaMA2-7B-Base critique runs receive a separate learning-rate sweep.For that initialization, learning rates of 5 × 10^-5, 8 × 10^-5, and 1.6 × 10^-4 are swept; longer training was found to overfit.Source: §4.4 · pp. 5–6

5 Evaluation Results

Adding critiques improves reward-model test accuracy in most cases, with effects depending on critique quality and starting checkpoint. Scaling and task analyses find stronger benefits in low-data, reasoning, and adversarial settings.

5.1 Do synthetic critiques improve RM performance?

  • GPT4-Turbo critiques raised REWARDBENCH Avg. Score by 0.16 for LLaMA-7B Base RMs and 0.14 for Command-35B Base RMs.Table 2 covers test accuracy on REWARDBENCH and PandaLM across three pretrained checkpoints and critique variants.Source: Table 2 · p. 6, §5.1 · pp. 6–7
  • Weaker LLaMA2-7B-Chat critiques could hurt accuracy, while No-Critiques excelled on preference-tuned Command R-35B.The passage contrasts this preference-tuned checkpoint with base checkpoints, where critiques offer more room for improvement.Source: Figure 2 · p. 7, §5.1 · p. 7

5.2 What impacts do critiques have on RMs as training scales up?

  • Command R+ critiques deliver greater accuracy gains than No-Critiques, especially with 5k–50k examples; ordinary gains narrow near 200k, but 5k GPT-4 critiques retain a gap.The scaling comparison initializes RMs from the 35B Command R base model and evaluates REWARDBENCH Avg. Score.Source: §5.2 · p. 7, Figure 3 · p. 8
  • 5k Command R+ critiques match about 90k No-Critiques examples; 5k Command R or LLaMA2-70B-Chat match roughly 62k, while 5k GPT4-Turbo surpasses 200k.These comparisons use REWARDBENCH test accuracy.Source: §5.2 · p. 8

5.3 Analysis

  • Chat accuracy reaches 0.91 at 10k examples, while critique gains on Reasoning exceed 10 points and converge with the baseline only at 20k.Critiques also improve Chat Hard adversarial robustness across training-data sizes, with larger benefits on reasoning and adversarial tasks.Source: §5.3 · pp. 8–9, Figure 4 · p. 8
  • Critique-trained RMs outperform a generative-judge baseline on Reasoning and Chat Hard; adding critiques at inference also raises its Reasoning score by 0.3.The authors argue that benefits in both training and inference indicate critiques add a Chain-of-Thought component, rather than merely transferring knowledge.Source: §5.3 · p. 9

6 Conclusion

Synthetic critiques provide an accessible, cost-effective way to enrich reward models, with strongest benefits in low-data settings and improved interpretability.

6 Conclusion

  • High-quality critiques can make one critique-enhanced preference pair as valuable as 40 standard preference pairs in low-data settings.The strongest effects occur when the base RM is weak and the task requires reasoning skills.Source: §6 · p. 9
  • Critique-enhanced RMs show stronger performance on existing reward-model benchmarks.The conclusion also reports better scores in challenging settings such as Reasoning and Chat Hard when preference data are abundant.Source: §6 · p. 9
  • Because critiques are generated without human labor, the approach offers a more cost-effective way to obtain competitive RMs amid scarce, costly preference data.Source: §6 · p. 9
  • Using critiques as part of the reward model’s Chain-of-Thought enhances training and interpretability.Source: §6 · p. 9

7 Limitations

The authors identify limits in the evaluation scope, critique quality, and gains for strong or data-rich settings; the appendix also describes their critique prompt.

7 Limitations

  • The study evaluates RM ranking accuracy on existing benchmarks, not the downstream optimization of LLMs with critique-enriched RMs.The authors also note that critiques may be incorrect or contain hallucinations.Source: §7 · p. 9
  • Critique value diminishes with abundant preference data, and improvement is limited when starting from a strong preference-tuned model.The authors nevertheless report no final-performance reduction in the abundant-data regime and better scores in Reasoning and Chat Hard.Source: §7 · p. 9
  • The authors plan to test a wider range of pretrained checkpoints and prompting methods.Source: §7 · p. 9

A Prompt Template

Template for Point-wise Critique Generation

B Training Data Format

The training preference data use human-ranked pairs of sampled completions, with confidence margins, and the appendix includes example critiques from different models.

B Training Data Format

  • 86.6% of preference pairs were labeled slightly better, 12.6% better, and 0.8% significantly better.Source: B Training Data Format · p. 13
  • Each pair contains two completions sampled from a previous model generation and a ranking from one human annotator, with no ties.The data comprise multi-turn conversational tasks seeded with a synthetically generated initial prompt; annotators were paid more than minimal wage.Source: B Training Data Format · p. 13
  • Table 3 gives examples of point-wise critiques from different models for the same preferred response.Source: Table 3 · p. 14

C Critique Generation Examples

The appendix notes that Table 3 presents critiques from different models for one preferred response.

C Critique Generation Examples

D Scaling Performance By Task

Critique-augmented reward models show task-specific gains as training data scales, including stronger Reasoning performance and a smaller decline on adversarially challenging Chat Hard. On Reasoning, critiques also spread reward scores for pairs that the generative judge had tied.

D Scaling Performance By Task

  • On Reasoning, Command R+ Critiques consistently outperform No-Critiques, while on Chat Hard they lead to a smaller performance drop as training examples increase.Chat Hard prompts are designed with superficial features to trick LLM-based evaluators.Source: D Scaling Performance By Task · p. 14
  • Classifier-based reward models overwhelmingly outperform the generative judge baseline, which performs strongly on Chat, comparably on Safety, and poorly on Reasoning and Chat Hard.Source: D Scaling Performance By Task · p. 14
  • A 0.3 increase to 0.3685 occurs for Gen-Command R+ (+Critiques) when critiques spread scores on Reasoning pairs previously tied by the generative judge.The ties arose when the judge assigned the same score to preferred and worse completions.Source: D Scaling Performance By Task · p. 15

E Critique Quality Analysis

Critique quality varies with the generating model: larger models correlate with higher metacritique scores, while model families differ in accuracy and comprehensiveness. Human-annotated critiques also slightly improve test accuracy over GPT4-Turbo in early experiments.

E Critique Quality Analysis

Loading 2405.20850v2…