Source-linked AI summary
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar
TL;DR
Existing self-rewarding methods improve response generation while leaving judgment ability undertrained, despite the cost and limits of human supervision. Meta-Rewarding adds a meta-judge that evaluates the model’s judgments and jointly trains acting and judging. Llama-3-8B-Instruct’s LC win rate rises from 22.9% to 39.4% on AlpacaEval 2, with similar improvement on Arena-Hard.
Problem
Existing self-rewarding primarily improves acting rather than judging, while human-generated training data is costly, time-consuming, and limited by human capabilities.
Method
Meta-Rewarding uses a model as actor, judge, and meta-judge, turning response judgments and meta-judgments into preference data for joint iterative training.
Results
39.4% LC win rate on AlpacaEval 2 versus 22.9% for Llama-3-8B-Instruct, with similar improvement on Arena-Hard and performance exceeding enhanced Self-Rewarding.
Takeaways & Limitations
The results support self-improving model training without additional human feedback while improving both judging and instruction-following performance within the reported evaluations.
Takeaways & Limitations
Positional bias persists, score saturation reduces discrimination, and the judge shows limited improvement on non-self-generated responses.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
1 INTRODUCTION
Meta-Rewarding extends self-improvement by training an LLM’s judging ability alongside its response-generation ability, without extra human data. The method uses a meta-judge to evaluate judgments and reports improved benchmark performance while addressing length bias.
- Motivation: Human-generated instruction-tuning and preference data are costly, time-consuming, and constrained by human capabilities.These constraints motivate self-judging approaches for steering advanced AI systems.
- Motivation: Prior Self-Rewarding improves the model as an actor but overlooks improving its judging ability, risking rapid saturation or reward hacking across iterations.The paper argues that actor training depends on increasingly capable judgment.
- Method: Meta-Rewarding adds a meta-judge that evaluates the model’s own judgments and creates preference pairs for training its judging capability.The same model performs the actor, judge, and meta-judge roles.
- Method: The method jointly uses response preferences and judgment preferences to improve acting and judging, with the combined skills intended to enhance instruction following without extra human data.Both preference-pair sets train the model at the next iteration.
- Method: Length control combines judge scores with response length, selecting shorter responses when scores are close to counteract length bias during iterative training.The mechanism addresses response-length growth associated with judge preference for longer outputs.
- Results: 39.4% LC win rate on AlpacaEval 2, up from 22.9% for Llama-3-8B-Instruct, while Meta-Rewarding exceeds enhanced Self-Rewarding at 39.4% versus 35.5%.The paper also reports similar improvement on Arena-Hard.
2 META-REWARDING
Meta-Rewarding is an iterative self-improvement scheme in which one model acts as actor, judge, and meta-judge, training both response generation and judgment. It constructs preference data for each role, applies DPO, and controls length and positional bias during selection.
- Meta-Rewarding: The model assumes three roles: actor for generating responses, judge for scoring them, and meta-judge for comparing its own judgments.All three roles are performed by the same model in a self-play setup without additional human data.
- Iterative training: Each iteration generates response variations, scores them with multiple judge judgments, builds response preferences, and constructs judgment preferences before applying DPO.The improved model becomes the next iteration’s data generator and optimization initialization.
- Length control: Length-controlled selection chooses the shortest response in a top score tier as chosen and the longest response in a bottom score tier as rejected.The quality tier parameter ρ controls the trade-off between score-based selection and response length; ρ = 0 disables length control.
- Judge preference dataset: Judge-training data focuses on the response whose judgments have the highest score variance, then compares judgment pairs with the meta-judge.The same model evaluates each pair, while prompting both judgment orders and weighting wins by position mitigates positional bias.
- Meta-Rewarding: Meta-Rewarding trains judgment ability by having a meta-judge evaluate the judge’s judgments and create preference pairs for judge training.The meta-judge receives the original prompt, response, two judgments, and the judge rubric before selecting the better judgment.
- Judge preference dataset: Meta-judge battles are converted into Elo scores, and the highest- and lowest-scoring judgments become the chosen and rejected pair for judge training.The battle matrix aggregates weighted pairwise outcomes, and ties are broken randomly during pair selection.
3 EXPERIMENTS
Experiments evaluate Meta-Rewarding for instruction following and judging, comparing it with Self-Rewarding and examining length control, category coverage, and analysis of judge behavior. Across benchmarks, Meta-Rewarding improves instruction-following performance and judge agreement, while score and positional biases emerge during training.
- Instruction Following Evaluation: 39.44% LC win rate is achieved on AlpacaEval 2 after four iterations, up from 22.92% for seed Llama-3-8B-Instruct.The method also outperforms GPT-4 and the 38.77% LC win rate reported for SPPO.
- Ablations and Analysis: Length control prevents substantial response-length growth and contributes to higher performance than the Self-Rewarding baseline with the same mechanism.Without length control, responses become excessively verbose and LC win rate declines.
- Instruction Following Evaluation: +8.5% improvement over the seed model is achieved on Arena-Hard after four Meta-Rewarding iterations.The benchmark uses complex and challenging questions whose prompt distribution differs from the training prompts.
- Instruction Following Evaluation: Meta-Rewarding improves MT-Bench Turn 1 Score from 8.319 to 8.738 while reducing Turn 2 Score by no more than 0.1.This contrasts with Self-Rewarding + LC, which typically sacrifices more than 0.2 in Turn 2 score without improving Turn 1.
- Reward Modeling Evaluation: Meta-Rewarding improves judging correlation with GPT-4 relative to Self-Rewarding in both evaluation settings, with agreement-without-ties gains up to +12.34%.The increase exceeds +6% in the GPT-4 Chosen Pairs setting.
- Ablations and Analysis: Using Starling-RM-34B as an external reward model fails to increase first-iteration AlpacaEval LC win rate, reaching 24.63% versus 27.85%.The authors suggest length bias as a possible explanation.
- Ablations and Analysis: Meta-judge training shifts the judge toward higher scores and introduces preferences for non-integer scores such as 4.5, 4.75, and 4.9.The higher scores provide more granularity for distinguishing response quality, while the meta-judge also develops score and positional biases.
4 RELATED WORK
Prior alignment methods commonly depend on human preferences or stronger models, while this work emphasizes self-improvement of judgment skills.
- RLHF approaches train reward models from human preference data before reinforcement learning.
- LLM-as-a-Judge is used for evaluation and reward-model training, but some judge datasets rely on human data or stronger models.
- Super alignment seeks to align models that may surpass human capabilities, motivating AI-generated feedback for training AI.
5 LIMITATIONS
The study is constrained by its coarse five-point judging system and by persistent judge-training biases that limit discrimination and later improvements.
- The five-point judging system often creates ties and makes further improvements difficult to detect as responses approach the maximum score.The authors suggest more nuanced scoring or comparison-based evaluation as alternatives.
- Higher scores increasingly saturate the judging distribution, reducing the judge’s ability to discriminate between responses.
- Persistent positional bias hindered further improvements in Iteration 3, despite mitigation efforts.
6 CONCLUSION
Meta-Rewarding trains judging skill through a meta-judge while adding length control for AI-feedback training. Without additional human feedback, it improves benchmark performance and judging correlation.
- Meta-Rewarding uses a meta-judge to assign meta-rewards and select chosen and rejected judgments for preference optimization.
- The method adds length control to mitigate response-length explosion during training with AI feedback.
- The approach improves results on AlpacaEval, Arena-Hard, and MT-Bench without additional human feedback beyond the stated seed-model setup.
- Judging ability improves in correlation with human judges and strong AI judges such as gpt-4-1106-preview.
A.1 JUDGE PROMPT
The judge prompt evaluates responses with an additive five-point system, briefly justifies the total, and outputs a formatted score.
- The prompt asks the judge to review the user’s question and response using an additive 5-point scoring system.
- Points accumulate as the response becomes more relevant, complete, and useful for the user’s question.
- The evaluated response is inserted into the prompt using a response placeholder.
- The judge must briefly justify the total score and conclude with the format “Score: <total points>”.
- The judge assesses the response from the AI Assistant perspective and may use web-search knowledge as necessary.
A.2 GPT4 JUDGE PROMPT
The GPT-4 judge prompt asks an LLM to compare unordered model outputs for a given instruction and return only the identifier of the best model. It is adopted from AlpacaEval because of its reported correlation with human judges.
- Prompt structure: The prompt presents an instruction and unordered outputs associated with unique model identifiers for evaluation.The judge receives model outputs and their corresponding identifiers as structured inputs.
- Evaluation task: The judge selects the model whose output has the best quality and relevance.The required decision is a comparison among candidate model responses.
- Output format: The response must contain exactly one permitted model identifier and no additional formatting.The allowed outputs are the identifiers m or M.
- Prompt source: The prompt is adopted from AlpacaEval because it is reported to correlate highly with human judges.The passage attributes this rationale to prior validation of the AlpacaEval prompt.
A.3 TRAINING DETAILS
The training setup specifies separate SFT, DPO, Self-Rewarding, and Meta-Rewarding schedules, with iterative filtering and checkpoint selection. Evaluation passages describe prompt distributions, MT-Bench turn-specific changes, and judge agreement metrics.
- SFT training: SFT training runs for 10 epochs with learning rate 5 × 10−8 and global batch size 32, selecting the epoch-5 checkpoint.Cosine learning-rate scheduling is used, with checkpoints saved after every epoch.
- DPO training: DPO training runs for 10 epochs with learning rate 5 × 10−6, β = 0.1, and global batch size 32.Cosine learning-rate scheduling is also used for DPO.
- Meta-Rewarding training: Meta-Rewarding iterations increase ρ from 0 to 0.32 and then 0.4 while filtering long chosen judgments and selecting different epoch checkpoints.The initial judge-length threshold is 1100, decreases to 1000 in Iteration 2, and later iterations retain ρ = 0.32 or increase it to 0.4.
- Prompt distribution: Training prompts are closer in embedding distribution to AlpacaEval prompts, whereas Arena-Hard prompts are more concentrated in a subset of the distribution.The comparison is based on t-SNE embeddings computed with text-embedding-3-small.
- MT-Bench: MT-Bench shows substantial Turn 1 improvement, while Meta-Rewarding sacrifices little Turn 2 performance and improves it in Iterations 3 and 4.The passage contrasts this with a large Turn 2 drop for the Self-Rewarding baseline.
- Human agreement: Judge agreement with humans is measured on the Open Assistant test set using Spearman correlation with the averaged ground-truth ranking, with and without ties.The metric represents ranking agreement averaged over prompts.