Self-Generated Critiques Boost Reward Modeling for Language Models

1 Introduction

• Scalar-only reward models are difficult to interpret and may underuse LLM language-modeling abilities, while critique-based judges can pair explanations with discrete quality judgments.The paper frames combining critiques with scalar rewards as a way to bring together these approaches’ strengths.Source: §1 · p. 1
• Prior approaches such as Ye et al. (2024), Ankner et al. (2024), and Zhang et al. (2024) rely on strong teacher LLMs, which can be costly and unavailable for frontier models.Source: §1 · p. 2
• Critic-RM generates multiple candidate critiques, filters them for consistency with human preference labels, and refines selected critiques through summarization and ranking.The framework does not rely on strong LLM teachers.Source: §1 · p. 2
• A weighting strategy first prioritizes critique modeling, then gradually shifts toward predicting rewards from both the response and its critique.This balances critique generation against reward modeling, which is prone to overfitting.Source: §1 · p. 2
• Critic-RM outperforms baselines on RewardBench and three out-of-distribution reward modeling tasks, with gains in both in-domain and out-of-domain evaluations.The authors attribute its superior generalization to identifying and leveraging high-quality self-generated critiques.Source: §1 · p. 2

2 Related Work

• Earlier reward models commonly train on chosen-versus-rejected response rankings using Bradley-Terry loss.Source: §2 · p. 2
• Critique-based reward modeling approaches often rely on strong teacher models and lack a unified way to improve critique quality; Zhang et al. (2024) focuses on math verification.Teacher dependence can be costly and difficult to apply when no stronger teacher exists, while math verification is hard to map to subjective domains.Source: §2 · p. 3
• LLM judges often explain a response before assigning a discrete score or preference, but off-the-shelf evaluators may be biased or misled.Source: §2 · p. 3
• Unlike self-alignment work that generates preference labels or synthetic pairs, Critic-RM combines self-generated critiques with human-annotated preference pairs.Source: §2 · p. 3

3.1 Preliminaries

• Critic-RM uses an instruction-finetuned LLM as the shared backbone for critique generation and reward modeling, with separate language-modeling and reward heads.The language-modeling head is inherited from the backbone, while the reward head is randomly initialized.Source: §3.1 · p. 4

3.2 Critique-augmented Reward Model Training

• Critiques act as latent variables between responses and rewards, decomposing training into preference modeling conditioned on critiques and critique generation toward the oracle distribution.The objective approximates the intractable oracle critique distribution with the critic generator.Source: §3.2 · pp. 4–5

3.2.1 Critique-augmented Reward Prediction

• The reward model predicts preferences from responses augmented with critiques, placing each critique after its response and a special token at its end.The special token marks where the reward is calculated.Source: §3.2.1 · p. 5

3.2.2 Critique Generation & Filtering

• The backbone samples candidate critiques with 1–10 response-quality scores, then retains preference pairs whose mean score is higher for the chosen response.This instance-level filter aims to reduce noisy critiques and align them with human preference labels.Source: §3.2.2 · pp. 5–6
• Critique refinement either summarizes multiple critiques into meta-critiques or ranks them with a meta-judge and keeps the top-K.Sampling different permutations for summarization can produce more diverse training critiques.Source: §3.2.2 · p. 6
• Critique-generation training uses forward KL with the self-identified high-quality critiques as empirical supervision, rather than backward KL, which may cause policy and entropy collapse.The resulting loss is applied to critiques for both chosen and rejected responses.Source: §3.2.2 · p. 6

3.2.3 Joint Learning of Critique Generation and Reward Modeling

• A dynamic weight schedule emphasizes critique generation early and shifts toward reward learning in the final training round to mitigate reward-model overfitting.The schedule balances critique-generation and reward-modeling losses across training steps.Source: §3.2.3 · p. 6

3.3 Critic-RM Inference

• At inference, Critic-RM generates a critique before scoring the response augmented with it; inference-time scaling averages rewards across multiple critiques.The scaled procedure samples critiques at non-zero temperatures.Source: §3.3 · p. 7

4.1.1 Training Data

• Public preference data covers chat, helpfulness, reasoning, and safety; synthetic data adds math and safety examples, including cases where the model should not refuse.The synthetic math pairs label correct solutions as chosen and incorrect solutions as rejected.Source: §4.1.1 · p. 7

4.1.2 Evaluation Benchmarks.

• RewardBench evaluates 2,985 chat, reasoning, and safety examples; CrossEval, QA Feedback, and SHP test OOD generalization, while CriticBench evaluates critique and correction accuracy.Accuracy is the main metric for reward-model tasks; CriticBench uses F1 for critique accuracy and accuracy for correction accuracy.Source: §4.1.2 · pp. 7–8

4.1.3 Baselines

• Baselines include LLM judges, standard scalar-output reward models, and critique-based reward models; relevant methods are reimplemented with the same data and backbone for fair comparison.The study excludes techniques focused on standard reward-model objectives as orthogonal to its focus.Source: §4.1.3 · p. 8

4.1.4 Implemenation Details

• Main experiments use Llama-3.1-70B-Instruct, sample 10 candidate critiques, and select the top two; training runs for two epochs and removes overlapping evaluation prompts.Critique generation uses temperature 0.9; inference-time critique sampling, when used, uses temperature 0.95.Source: §4.1.4 · p. 8

4.2 Main Experiments: RewardBench

• 3.7%-4.7% over the standard Reward Model and 6.2%-7.3% over the Llama-3.1-405B judge: Critic-RM’s RewardBench gains hold with the same preference data.Table 2 reports results on RewardBench.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Out-of-Distribution (OOD) Evaluation

• 4% average margin over standard reward-model baselines across three OOD datasets; Critic-RM’s gains are more pronounced on challenging cross-ability tasks.Its performance is comparable to LLM judges with significantly more parameters.Source: §4.3 · p. 9, Table 3 · p. 10

4.4 Evaluation on Critiques

• Critic-RM generates more accurate critiques than strong baselines, including GPT-4, and its feedback helps policy models correct flawed reasoning.With Llama-3-8B as the policy model, it achieves high accuracy across five reasoning tasks.Source: §4.4 · p. 10, Table 4 · p. 10

4.5 Ablation Studies

• Constant or reverse loss-weight schedules hurt performance, while two training rounds improve it before gains plateau, making K = 2 a performance-efficiency trade-off.Figure 2 reports average accuracy on RewardBench.Source: §4.5 · pp. 10–11, Figure 2 · p. 10

• Critic-RM harnesses LLMs’ ability to generate and refine critiques as a self-improvement approach for reward modeling.Source: §5 · p. 11
• Table 6 illustrates Critic-RM identifying key errors in rejected responses and explaining why chosen responses are superior.Source: Table 6 · p. 12
• Critic-RM consistently outperforms baseline reward models across multiple datasets, with strong data efficiency and robust results even with limited labeled data.Source: §5 · p. 12

• Critic-RM requires the base LLM to have a certain level of critique-generation ability.Source: Limitation and Future Work · p. 12
• Generating critiques during inference adds computational overhead that may affect latency-critical real-time applications.Source: Limitation and Future Work · p. 12
• Critic-RM does not use iterative training in which models refine themselves over multiple rounds.Source: Limitation and Future Work · p. 12

• The selected CrossEval subtasks are Reasoning, Coding, English, Tool, Reasoning+Coding, Coding+Reasoning, and Tool+Coding4.Source: A Dataset Processing Details for… · p. 17
• For each prompt, the processing uses three responses associated with two ratings and retains response pairs with different average scores.Source: A Dataset Processing Details for… · p. 17
• The higher-scored response in each retained pair is designated chosen, yielding 1,181 response pairs in total.Source: A Dataset Processing Details for… · p. 17

B.1 Prompt Templates for Critique Generation

• Critique-generation prompts are customized by task, with Chat/Helpfulness, Code, or Math formats used for specified evaluation subsets.Source: §B.1 · p. 17
• Table 7 lists prompt formats for generating critiques for training and evaluation data.Source: Table 7 · p. 17
• Multilingual and multimodal capabilities are beyond the scope of the paper.Source: §B.1 · p. 17

B.2 Prompt Templates for Critique Refinement

• The appendix lists prompt templates for refining critiques in Table 8.Source: §B.2 · p. 18
• Table 8 provides prompt formats for generating critiques for critique refinement.Source: Table 8 · p. 18

B.3 Prompt Templates for Correction Generation

• The appendix lists prompt templates for correcting initial responses in Table 9.Source: §B.3 · p. 18
• Table 9 presents prompt formats for correcting initial responses.Source: Table 9 · p. 18

• The additional qualitative examples focus on Code reasoning and RewardBench’s Chat-hard subset, comparing standard reward models, CLoud, and Critic-RM on reward prediction.Underlining marks imprecise generation of rejected responses, while green text marks the central part leading to Critic-RM’s accurate predictions.Source: C Additional Case Studies · p. 19, Table 10 · p. 19
• Critic-RM identifies key strengths and weaknesses of responses, while CLoud falls short in delivering faithful assessments.Source: C Additional Case Studies · p. 19

• Table 11 presents comprehensive Critic-RM results with a detailed breakdown by category.Source: D Full Results for Critic-RM · p. 20
• Table 11 reports results for different Critic-RM variants, and IS denotes inference scaling.Source: Table 11 · p. 20