PaperlayerView exact revision on arXiv

Source-linked AI summary

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze Chen, Yiqun Liu, Min Zhang, Qingyao Ai

arXiv:2610.05370v2cs.AI

TL;DR

Pairwise GRMs can make correct preference judgments despite unreliable critiques, and human critiques are scarce. EnGRICH uses a training-time MetaCritic to turn limited human critiques into reusable rubric-based supervision; experiments report consistent improvements across seven benchmarks and downstream policy optimization.

  • Problem

    Because pairwise GRMs choose between only two responses, preference correctness can coexist with unreliable critiques that outcome supervision may reinforce.

    Source: §1 · p. 1

  • Method

    EnGRICH pairs a GRM with a training-time MetaCritic that learns response-specific rubrics from human critiques and uses their diagnoses for process rewards and guided exploration.

    Source: §1 · p. 2, §3.2 · p. 3, §3.4 · p. 4

  • Results

    EnGRICH consistently improves over competitive baselines across seven reward-model benchmarks and downstream policy optimization.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Limited human critiques can provide reusable evaluative criteria for outcome-only data while the trained GRM operates independently at inference.

    Source: §6 · p. 10

    The learned evaluative criteria may reflect biases or limitations in the underlying data and human feedback.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside preference judgments, providing finer-grained evaluation signals. Their effectiveness depends heavily on critique reliability. However, existing GRM training typically uses final preference correctness as outcome supervision. Because the preference outcome space is highly constrained, unreliable critiques can still yield correct outcomes and thus be reinforced. Recent work leverages human critiques for process supervision, but such critiques are scarce and are often reduced to scalar rewards, leaving their fine-grained evaluative information underutilized. We argue that evaluative criteria learned from human critiques can be generalized to broader outcome-only preference data. To this end, we propose \textbf{EnGRICH}, a GRM training framework that pairs the GRM with a training-time MetaCritic learned from a small set of human critiques. MetaCritic constructs response-specific rubrics and uses them to evaluate the evidence coverage and correctness of generated critiques. The resulting signals provide both process rewards for fine-grained credit assignment and structured guidance for exploring better critiques. During GRM training, MetaCritic is further optimized to generalize human-grounded evaluative criteria to outcome-only data. At inference, the trained GRM operates independently. Experiments across seven reward-model benchmarks show that EnGRICH consistently improves over competitive baselines, while further analyses validate the effectiveness of its core mechanisms.

1 INTRODUCTION

Because pairwise GRMs choose between only two responses, a correct preference judgment can coexist with an unreliable critique that outcome-based training reinforces. EnGRICH uses human-grounded critique evaluation to supervise both critique quality and preference judgments.

1 INTRODUCTION

  • Pairwise preference outcomes are constrained to choosing between two responses, so correctness alone may not ensure that a GRM's critique is reliable.Unreliable critiques can still be reinforced and affect downstream policy optimization and candidate selection.Source: §1 · pp. 1–2
  • Prior process-supervision methods use human critiques but reduce their diagnostic information to scalar rewards, while Rubric-ARM lacks human-critique supervision for rubric generation and requires both modules at inference.High-quality human critiques are costly, so only a small fraction of preference data includes them.Source: §1 · p. 2
  • EnGRICH pairs a GRM with a training-time MetaCritic that builds response-specific rubrics and assesses critique evidence coverage and correctness.Its signals provide process rewards for finer-grained credit assignment and structured guidance for exploring better critiques; MetaCritic is further optimized on outcome-only data, and inference uses the standalone GRM.Source: §1 · p. 2
  • Across diverse reward-model benchmarks, EnGRICH consistently outperforms outcome-supervised and prior process-supervised baselines, with analyses supporting its process-supervision and exploration mechanisms.Training uses limited human critiques together with outcome-only preference data.Source: §1 · p. 2

2 RELATED WORK

Prior GRM training relies largely on preference correctness, while critique-based methods add process supervision but often compress diagnostic information. EnGRICH instead learns human-grounded rubrics and uses their diagnoses for process rewards and targeted revision.

2 RELATED WORK

  • GRM training largely relies on final preference correctness, which may reinforce unreliable critiques that nevertheless yield correct judgments.RC-GRPO and RM-NLHF derive scalar process rewards from human critiques, while R-Align assesses rationale alignment; these methods largely compress fine-grained diagnostics into scalar or holistic signals.Source: §2 · p. 2
  • Rubric-ARM jointly learns a rubric generator and judge but trains rubric generation through preference correctness rather than explicit human-critique supervision, and requires both modules at inference.EnGRICH instead learns human-grounded, instance-specific rubrics from limited human critiques, generalizes them to outcome-only data, and uses diagnoses for process rewards and targeted revision.Source: §2 · p. 2
  • Natural-language feedback methods use critiques or reflections for refinement, while newer online-RL methods incorporate critique-guided refinements or feedback on failed rollouts [Scheurer et al., 2024; Madaan et al., 2023; Shinn et al., 2023; Aky¨urek et al., 2023; Zhang et al., 2026; Li et al., 2026].The paper distinguishes GRMs by assigning fine-grained supervision complementary roles in process credit assignment and guided exploration.Source: §2 · p. 3

3 METHOD

EnGRICH learns response-specific evaluative criteria from limited human critiques, extends them to outcome-only data, and uses keypoint diagnoses to train the GRM. Process rewards differentiate correct assessments, while guided exploration addresses groups of failed assessments.

3.1 PROBLEM FORMULATION

  • The setting partitions preference examples into a small human-critique subset and a larger outcome-only subset, and aims to generalize key evaluation points learned from the former.A GRM generates a comparative critique and preference judgment for two candidate responses.Source: §3.1 · p. 3

3.2 OVERVIEW

  • MetaCritic builds response-specific rubrics and diagnoses critique evidence coverage and correctness, providing process rewards and structured exploration guidance.It is further optimized on human-critique and outcome-only data; the GRM operates independently at inference.Source: §3.2 · pp. 3–4

3.3 FROM HUMAN CRITIQUES TO PROCESS SUPERVISION

  • MetaCritic's rubric generator proposes keypoints and its verifier labels each as MATCH, MISSING, or INCORRECT; Keypoint-F1 aggregates the labels into a process reward for coverage and correctness.Incorrect keypoints reduce both precision and recall, while missing keypoints reduce recall; MetaCritic is initialized using human-critiqued examples and human-derived targets.Source: §3.3 · p. 4

3.4 LEARNING WITH HUMAN-GROUNDED PROCESS SUPERVISION

  • During GRM optimization, preference correctness is combined with Keypoint-F1 only for correct assessments, providing bounded process differentiation among them.For groups where every valid assessment is incorrect, keypoint-guided revision can replace a failed assessment if the revision is valid and correct, restoring reward contrast; the exploration prompt withholds preference correctness.Source: §3.4 · pp. 4–5
  • The rubric generator is further optimized on human-critique data to retain fine-grained distinctions and on outcome-only data to generalize human-grounded criteria without reference rubrics.On the human-critique subset, reference-rubric scores refine rankings among equally correct critiques; on outcome-only data, preference correctness supplies coarser rankings, while the verifier remains fixed.Source: §3.4 · p. 5

4 EXPERIMENTS

Across seven reward-model benchmarks and two backbones, EnGRICH consistently improves performance. Its critique evaluations also report stronger critique metrics and response-revision gains, while experiments use a mixture of human-critiqued and outcome-only preference data.

4.1 EXPERIMENTAL SETTINGS

  • Experiments use a 60K preference mixture with 12K human-critique examples and 48K outcome-only examples, and compare EnGRICH with outcome- and process-supervised baselines.The study uses Qwen3-4B and Gemma3-12B backbones; the process-supervised baselines are RC-GRPO and RM-NLHF.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • EnGRICH consistently improves performance across all seven reward-model benchmarks and both backbones.Table 1 reports results across seven benchmarks and identifies the best and second-best controlled results within each backbone; Overall is the unweighted mean.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 CRITIQUE QUALITY EVALUATION

  • On HelpSteer3 validation examples where all compared GRMs predict the correct preference, EnGRICH achieves higher CC-F1 and lower OPI than baselines, with comparable assessment lengths.It also yields larger response-revision gains than direct-answer baselines across MATH-500, HumanEval+, and Arena-Hard-v2.0.Source: Figure 2 · p. 7, §4.3 · p. 7

5 ANALYSIS

The analyses report that EnGRICH’s gains extend to downstream policy optimization, while ablations and scaling results support the value of its core mechanisms. Its training adds overhead, but inference time and assessment length remain comparable to baselines.

5.1 ABLATION STUDY

  • Removing Keypoint-F1 most degrades critique quality, while freezing Mrub causes the largest average benchmark-performance drop.Removing guided exploration also generally degrades benchmark performance; restricting process supervision to DH underperforms full EnGRICH across all benchmarks and critique-quality metrics.Source: §5.1 · p. 8

5.3 TRANSFER TO POLICY OPTIMIZATION

  • EnGRICH-trained policies achieve the strongest performance across all three downstream benchmarks.The comparison uses Qwen3-4B-Instruct-2507 policies with Gemma3-12B as the GRM backbone.Source: Table 3 · p. 9, §5.3 · p. 9

5.2 SCALING HUMAN CRITIQUE SUPERVISION

  • Increasing human-critique supervision improves all three evaluated aspects, and rubric generalization further improves rubric quality across DH sizes.The study varies DH over {0, 3K, 6K, 12K} while keeping all 60K preference pairs and labels fixed; Mver gains only marginally from additional critiques.Source: §5.2 · pp. 8–9

5.4 EFFICIENCY

  • EnGRICH adds training overhead comparable to RC-GRPO and RM-NLHF, while its inference time and assessment length are comparable to baselines.Figure 4 reports mean wall-clock time per completed GRM update and per-sample inference time and output length averaged across seven benchmarks.Source: §5.4 · p. 9, Figure 4 · p. 9

6 CONCLUSION

EnGRICH uses MetaCritic to generalize evaluative criteria from human critiques to broader outcome-only data, while the trained GRM operates independently at inference. Experiments across seven reward-model benchmarks and downstream policy optimization show consistent improvements.

6 CONCLUSION

  • Across seven reward-model benchmarks and downstream policy optimization, EnGRICH demonstrates consistent improvements.Further analyses validate the complementary roles of process supervision, guided exploration, and rubric generalization.Source: §6 · p. 10
  • MetaCritic turns human critiques into reusable evaluative criteria that provide process rewards and structured guidance for critique exploration.The trained GRM operates independently at inference.Source: §6 · p. 10

A ALGORITHM DETAILS AND ANALYSIS

A.1 HUMAN-GROUNDED INITIALIZATION

  • For human-critiqued data, an offline annotator extracts reference rubrics, and the initialized rubric generator constructs rubrics used to train the verifier.The GRM receives only X, not the human critique or gold preference, as conditioning input.Source: §A.1 · p. 15

A.2 TRAINING PROCEDURE

  • Rubric learning uses original assessments, while GRM learning may use retained revised assessments, preventing a rubric from scoring a revision it helped generate.Source: §A.2 · pp. 15–16
  • When every original assessment is format-valid but outcome-incorrect, EnGRICH guides one revision with rubric keypoint labels while withholding the gold preference and correctness indicator.A format-valid, outcome-correct revision replaces its parent; otherwise the original group is retained.Source: §A.2 · p. 15

A.3 TOKEN-LEVEL OPTIMIZATION OBJECTIVES

  • The GRM batch objective averages per-assessment losses across inputs, with each retained assessment containing a critique and final preference.Source: §A.3 · p. 16
  • Accepted revisions are evaluated under X alone, and their policy shaping is a learning surrogate rather than an unbiased importance-sampling correction for diagnostic-conditioned behavior.The assessment must be re-encoded under X rather than retaining feedback in its context.Source: §A.3 · p. 17

A.4 PROPERTIES OF THE TRAINING SIGNALS

The training signals use rubric-based critique evaluation to refine rewards and provide contrast for optimization. Their guarantees are local, while computation reuses verifier outputs and the trained GRM operates independently at inference.

A.4 PROPERTIES OF THE TRAINING SIGNALS

  • Keypoint-F1 stays in [0, 1], reaches 1 only when every keypoint is MATCH, and process rewards refine correct assessments without reversing preference-correctness ordering.A format-valid correct assessment receives at least one more reward than any format-valid incorrect assessment.Source: §A.4 · pp. 17–18
  • Successful guided revision can restore reward contrast in an all-failed group, but it does not guarantee nonzero net gradients or improved test performance.Its expected advantages depend on the probability that the revision is accepted.Source: §A.4 · p. 18
  • Outcome-only rubric ranking supplies no signal for an all-correct or all-incorrect original group, whereas human-reference score differences can add within-correctness pairs.Ranking bounds characterize agreement with constructed targets, not an unobserved ground-truth critique ranking.Source: §A.4 · p. 19
  • The policy-shaping rule gives an accepted revision a positive, bounded multiplier on each log-probability gradient, but does not bound the full parameter-gradient norm.The multiplier depends on the current-to-old probability ratio, not absolute token probability.Source: §A.4 · p. 19

A.5 COMPUTATIONAL COST AND SIGNAL REUSE

  • Cached verifier diagnoses serve GRM rewards, exploration guidance, and rubric ranking; inference uses only the trained GRM, without rubric generation, verification, or guided revision.Evaluating pairwise rewards after scoring costs O(GX|PX|) scalar operations, separate from verification inference.Source: §A.5 · p. 20

B PROMPT DETAILS

  • The prompt interfaces separate GRM critique and choice generation, generate rubrics from task and responses alone, verify keypoints, and use diagnoses to guide revision without gold preference input.The rubric generator is updated online, while the critique verifier remains frozen.Source: B PROMPT DETAILS · pp. 20–21

C.1 TRAINING DATA

The training mixture combines a smaller human-critique subset with a larger outcome-only subset across preference domains. Human-derived critique supervision is available for one fifth of the original training pairs.

C.1 TRAINING DATA

  • The 60,000-pair training set contains 12,000 human-critique examples and 48,000 outcome-only examples, a 1:4 ratio.Both subsets have pairwise preference labels, but only the human-critique subset adds critique supervision.Source: §C.1 · pp. 21–22
  • The 12,000 human-critique examples come from HelpSteer3-Preference and span general, code, STEM, and multilingual domains.The released domain counts are 5,816 general, 2,717 code, 1,536 STEM, and 1,931 multilingual examples.Source: §C.1 · pp. 21–22
  • The 48,000 outcome-only examples cover general assistance, mathematics, code, safety, instruction following, and writing, with explicit preferred and rejected responses.Inputs over 4,096 tokens are filtered, and preferred responses are balanced across sides A and B; 60,000 counts original pairs only.Source: §C.1 · pp. 22–23

C.2 VALIDATION AND EVALUATION DATA

The evaluation uses seven reward-model benchmarks with benchmark-specific protocols and an unweighted overall mean. The accompanying setup specifies two backbones, shared optimization settings, and distinct training recipes for EnGRICH and its comparators.

C.2 VALIDATION AND EVALUATION DATA

  • The evaluation spans seven benchmarks: HelpSteer3, RewardBench 2, RM-Bench, SCAN-HPD, HREF-HA, LitBench, and WQ-LMArena.Overall is the unweighted arithmetic mean of the seven benchmark scores, not an example-weighted average.Source: §C.2 · pp. 23, 25
  • Protocols differ: RewardBench 2 reports the unweighted mean of five domain accuracies, RM-Bench the unweighted mean of four; HS3 and SCAN-HPD report pairwise preference accuracy, while HREF reports leave-one-out human-agreement.Source: §C.2 · pp. 23–25

D.1 GENERAL DETAILS

  • The Judge backbones are Qwen3-4B and Gemma3-12B, with frozen backbone weights and LoRA adapters; final checkpoints are evaluated using greedy generation.Evaluation filters inputs beyond benchmark token limits, producing tokenizer-dependent effective counts for HelpSteer3.Source: §D.1 · pp. 25–26, §C.2 · p. 25
  • Frozen-verifier EnGRICH trains the Judge and rubric generator for one pass, uses process-reward coefficient λ = 0.5, and permits at most one guided revision per all-failed group.Its verifier remains frozen; the recipe uses no online external-annotator calls, KL regularization, or entropy bonus.Source: §D.1 · p. 26
  • The comparison methods use distinct supervision: RC-GRPO and RM-NLHF use human-critique-derived process rewards, while the outcome-only GRPO baseline uses correctness rewards without a MetaCritic.The SFT baseline uses human critiques and gold preferences; DPO forms preference pairs from sampled correct and incorrect judgments.Source: §D.1 · pp. 26–27

D.2.1 DIRECT CRITIQUE EVALUATION

The direct critique evaluation uses a shared held-out set and judge prompt, then measures critique agreement, outcome–process inconsistency, and assessment length.

D.2.1 DIRECT CRITIQUE EVALUATION

  • The evaluation compares seven models on the same held-out HelpSteer3 question–response pairs using a shared Brief format and GPT-5.5 judge.Each model produces a critique followed by an A/B choice.Source: §D.2.1 · p. 27
  • CC-F1 measures critique agreement with human references on reference-eligible pairs where all seven models make valid, correct choices.GPT-5.5 constructs references, extracts claims, and performs semantic matching; matching uses maximum-weight one-to-one assignment.Source: §D.2.1 · p. 28
  • OPI is the fraction of each model’s valid, correct choices whose critique has core-argument F1 at or below 0.5.The evaluation follows RM-NLHF’s core-argument scoring prompt.Source: §D.2.1 · p. 28
  • Average assessment length counts tokens in the complete critique and A/B choice across eligible generated examples, including invalid outputs and incorrect choices.Source: §D.2.1 · p. 28

D.2.2 DOWNSTREAM REVISION EVALUATION

The downstream evaluation tests whether critique-guided selection and editing improve responses over direct answering, using three benchmarks and task-specific scoring. The section also specifies ablations that remove individual EnGRICH mechanisms.

D.2.2 DOWNSTREAM REVISION EVALUATION

  • Feedback-Edit improvement measures the complete candidate-sampling, selection, and editing procedure against direct answers from the same downstream model.The reported difference is in percentage points and includes candidate selection.Source: §D.2.2 · pp. 28, 31
  • Critique-guided refinement is evaluated on MATH-500, HumanEval+, and 500 hard prompts from Arena-Hard-v2.0, with task-specific scoring procedures.HumanEval+ reports pass@1 over 164 questions; MATH-500 uses answer equivalence, and code outputs must pass original and additional tests.Source: §D.2.2 · pp. 28, 30
  • Each GRM selects two responses from a shared pool of eight, generates a critique, and passes the selected responses and critique to the editor for a complete final answer.The editor is the downstream model that generated the candidates; it restricts edits to issues identified in the critique and returns a supplied response when no actionable change is identified.Source: §D.2.2 · pp. 29–30
  • Arena-Hard responses are compared with the released o3-mini reference in both orders, and the two win/tie/loss scores are averaged per question.The judge prompt also evaluates candidate and reference in both orders and is shared across direct answers and final Feedback-Edit responses.Source: §D.2.2 · p. 31, Figure 13 · p. 31

D.3 ABLATION DETAILS (§5.1)

  • The ablations separately remove the process reward, guided exploration, rubric generalization, or both process reward and exploration on outcome-only data.The last variant leaves preference correctness as the only GRM supervision on that subset.Source: §D.3 · p. 31

D.4 HUMAN-FEEDBACK SCALING EXPERIMENTAL DETAILS (§5.2)

The experimental details vary human-critique coverage while holding preference data fixed, then describe downstream policy optimization and efficiency measurement. Critique-quality evaluation includes direct and downstream assessments.

D.4 HUMAN-FEEDBACK SCALING EXPERIMENTAL DETAILS (§5.2)

  • Human-critique coverage varies across nested subsets of 0, 3,000, 6,000, and 12,000 examples while all 60K preference pairs and labels remain fixed.At zero coverage, MetaCritic components start from the base backbone; outside the exposed subset, human critiques and human-derived rubric targets are withheld.Source: §D.4 · p. 32

D.5 DOWNSTREAM POLICY OPTIMIZATION DETAILS (§5.3)

  • Downstream policies are trained with GRPO for 100 steps using questions alone, and frozen reward models are the only component varied across conditions.Pairwise judgments are converted into group-relative rewards through a double-elimination tournament; MetaCritic and human critiques are not used during policy optimization.Source: §D.5 · p. 33

D.6 EFFICIENCY DETAILS (§5.4)

  • Training-time efficiency reports amortized wall-clock seconds per completed GRM update, while inference time per sample is a pooled batch-processing measure, not single-request latency.Online update timing includes rollout, revision, rewarding, auxiliary updates, and GRM updates; offline initialization and teacher annotation are excluded.Source: §D.6 · p. 34

E.1 CRITIQUE QUALITY

  • Critique quality is assessed through direct critique evaluation and downstream response-revision evaluation on both Qwen3-4B and Gemma-3-12B.Source: §E.1 · p. 34

E.2 ABLATION STUDY

The appendix presents an ablation comparison, human-critique coverage scaling, and RL-based GRM efficiency analyses across two backbones.

E.2 ABLATION STUDY

  • The ablation study evaluates four EnGRICH variants on Qwen3-4B and Gemma3-12B, with Table 8 reporting results by backbone.The caption says boldface marks the best result within each backbone; no cell values are supplied here.Source: §E.2 · p. 34, Table 8 · p. 35

E.3 SCALING HUMAN CRITIQUE SUPERVISION

  • Figure 15 tracks benchmark performance, critique quality, and MetaCritic quality as human-critique coverage increases for both backbones.It distinguishes MetaCritic before online GRM training from its final checkpoint after training.Source: §E.3 · p. 34, Figure 15 · p. 36

E.4 EFFICIENCY

  • Figure 16 compares training and inference efficiency of RL-based GRMs on Qwen3-4B and Gemma3-12B.Inference time and output length are averaged per sample across seven benchmarks.Source: §E.4 · p. 35, Figure 16 · p. 36

Related papers

Loading 2610.05370v2…