Source-linked AI summary
Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
Binghai Wang, Yantao Liu, Yuxuan Liu, Tianyi Tang, Shenzhi Wang, Chang Gao, Chujie Zheng, Yichang Zhang, Le Yu, Shixuan Liu, Tao Gui, Qi Zhang, Xuanjing Huang, Bowen Yu, Fei Huang, Junyang Lin
TL;DR
Reward models can achieve correct judgments for incorrect reasons, exposing a gap in outcome-only evaluation and supervision. The paper introduces Rationale Consistency and METAJUDGE, then combines rationale consistency with outcome accuracy for GenRM training. The resulting method improves benchmark performance, RLHF outcomes, and rationale consistency, while scalability remains limited by the need for high-quality human annotations.
Problem
Outcome-only supervision can mask deceptive alignment, because reward models may match human decisions while relying on incorrect rationales and then generalize poorly during RLHF.
Method
The paper introduces Rationale Consistency and METAJUDGE, then trains GenRMs with a hybrid signal combining rationale consistency and outcome accuracy using GRPO.
Results
The method achieves state-of-the-art results across RM-Bench and JudgeBench, improves Arena Hard v2 Creative Writing performance, and raises rationale consistency during training.
Takeaways & Limitations
Rationale consistency complements outcome accuracy by detecting deceptive alignment and providing supervision that improves reward-model reliability within the evaluated benchmarks and RLHF setting.
Takeaways & Limitations
Rationale Consistency still requires high-quality human annotations, limiting scalability; even the strongest frontier models achieve under 40% consistency with human rationales.
Abstract
from arXiv · showhide
Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.
1 Introduction
The paper argues that outcome accuracy can hide deceptive alignment in reward models and introduces rationale consistency to evaluate whether judgments follow human reasoning. It combines both signals during GenRM training to improve benchmark and RLHF performance.
- Outcome-only supervision can produce reward models that predict human preferences correctly while relying on spurious correlations or incorrect reasoning.This deceptive alignment contributes to poor generalization during RLHF.
- Rationale Consistency measures how closely a model’s evaluation process aligns with human judgment.METAJUDGE decomposes human rationales into atomic units and uses strict one-to-one semantic matching to quantify recovered rationales.
- Evaluation of 19 state-of-the-art LLMs shows that rationale consistency reveals judgment differences and separates frontier models more effectively than outcome accuracy.Models with comparable outcome accuracy can use fundamentally different reasoning, while outcome accuracy approaches saturation.
- 87.1% on RM-Bench and 82.0% on JudgeBench were achieved by GenRMs trained with a hybrid rationale-consistency and outcome-accuracy signal.The method outperformed outcome-only baselines by 3% and 7%, respectively.
- Using the trained reward model in RLHF produced a 7% boost on Arena Hard v2 Creative Writing tasks.Rationale consistency also increased from 25% to 37%, reversing its decline under outcome-only training.
2 MetaJudge
METAJUDGE builds a fine-grained benchmark by converting human preference rationales into atomic, evidence-grounded units and matching them against model-generated reasons. Its evaluation uses optimal one-to-one matching and fixed-length outputs to assess the reasons models identify.
- METAJUDGE evaluates alignment between model judgments and human reasoning through atomic rationale decomposition and strict semantic matching.The framework includes benchmark construction, matching, and formalization of Rationale Consistency.
- The benchmark uses HelpSteer3 examples containing queries, response pairs, preference labels, and detailed expert human rationales.It spans general conversation, code, STEM, and multilingual tasks.
- 250 examples per domain are decomposed into atomic rationales that retain concrete evidence while filtering generic statements and redundancy.The resulting benchmark retains 3–7 critique points per instance.
- 207 high-quality CW-Atomic test cases remain after three annotators labeled 350 creative-writing samples and disagreement cases were removed.
- Rationale elicitation is described as adding no burden beyond organizing annotators’ reasons into an atomic, evidence-grounded format.The paper also treats these rationales as supervision signals for model training and evaluation.
- Human and AI atomic reasons receive semantic fulfillment scores, with one-to-one matching preventing a broad AI reason from matching multiple human reasons.Each AI reason can match at most one best-matching human reason.
- Evaluation enforces a fixed-length reason list, such as Top-5, to control output budget and test identification of key reasons.This constraint is applied during evaluation but not training.
3 Rationale Consistency Evaluation
Rationale consistency distinguishes models that achieve similar outcome accuracy, exposes deceptive alignment, and remains robust across evaluators, domains, and annotator groups.
- Rationale consistency distinguishes stronger frontier models from weaker ones even when their outcome accuracy is similar.It provides a more reliable signal of human-aligned judgment by reflecting whether model rationales match key human reasons.
- Similar outcome accuracy can conceal major differences in rationale quality, including nearly 50% lower rationale consistency for o3-mini than o3.A comparable pattern appears between Gemini 3 Pro and Gemini 3 Flash.
- Correct choices can arise from incorrect reasoning: o3-mini selects the preferred answer but misses the human-identified word-count constraint.Instead, it relies on superficial cues such as self-claimed compliance and emojis.
- Even the most advanced models achieve only around 0.4 rationale consistency, indicating substantial room to align judgment logic with human reasoning.The passage also notes that human annotation remains necessary for genuine alignment for the foreseeable future.
- Rationale-consistency scores are highly consistent across evaluator models, with R2 = 0.983 and RMSE = 0.006.This consistency is attributed to the evaluator’s relatively lightweight semantic matching task.
- Model rankings generalize across HelpSteer3-Atomic and CW-Atomic, with Spearman ρ = 0.85 across different domains and annotator groups.CW-Atomic covers creative writing and uses a different annotator group.
4 Generative Reward Modeling Based on Rationale Consistency
The method trains generative reward models with rationale consistency alongside outcome accuracy, using hierarchical rewards and GRPO to align judgments with human reasoning. This improves benchmark performance, code-domain discrimination, downstream RLHF alignment, and resistance to rationale degeneration.
- 4.1 Training Objective: Average Precision rewards comprehensive human-aligned reasons while prioritizing core reasons near the top of the rationale list.Unlike unordered F1, its soft ranking constraint supplies a smoother prioritized learning signal.
- 4.1 Training Objective: The hybrid reward combines rationale consistency with binary outcome accuracy, making correct reasoning necessary for high reward.The multiplicative signal gates outcome correctness with rationale quality.
- 4.2 Optimization Algorithm: GRPO optimizes the hybrid rewards using relative advantages computed across groups of sampled outputs, with KL regularization against a reference model.The training pipeline converts HelpSteer3 rationales into atomic checklists for supervision.
- 4.3 Generative Reward Modeling: The method outperforms strong state-of-the-art competitors, including GRAM-R2 at 83.4% and Principles-Qwen32B at 83.8%, with an overall average of 84.6%.The paper attributes the advantage to improving the supervision signal so models learn judgment logic consistent with human reasoning.
5 Related Work
Prior work improves judge interpretability and critique quality, but these approaches remain vulnerable to post-hoc rationalization or difficult-to-use textual feedback. This paper distinguishes rationale supervision for judges from solver-focused process supervision and extends meta-verification to subjective value alignment.
- Generative Reward Models and LLM-as-a-Judge: Generative Reward Models and LLM-as-a-Judge use Chain-of-Thought to improve interpretability, but subjective evaluations remain vulnerable to post-hoc rationalization.CoT may justify a biased decision rather than provide its causal basis.
- Critique: Critique-based methods evaluate critique quality, but textual critiques are difficult to use effectively as reinforcement-learning supervision.
- Meta-Verification: Meta-verification methods penalize hallucinated defects in objective domains, while this work extends consistency checking to subjective value alignment.
- Process Supervision vs. Rationale Supervision: Rationale supervision targets the judge’s logical faithfulness to its final label, unlike Process Reward Models, which supervise a solver’s intermediate steps.
6 Conclusion
The paper finds that outcome-only supervision can hide deceptive alignment, then introduces rationale consistency measurement and rationale supervision to improve reward-model reliability. These additions produce stronger benchmark performance and more reliable logic-based verification.
- Outcome-only supervision can mask deceptive alignment because models may match human decisions while relying on incorrect rationales.
- Rationale Consistency, measured with METAJUDGE, exposes deceptive alignment and differentiates stronger models.
- Rationale supervision significantly improves GenRM performance, achieves state-of-the-art benchmark results, and increases the reliability of logic-based verification.
7 Limitations
The paper’s rationale-consistency evaluation depends on carefully constructed, evidence-grounded atomic rationales and currently requires high-quality human annotations. The accompanying decomposition process filters vague, subjective, conflicting, and redundant feedback.
- 36
- 39
- 40
- 01
A.2 Benchmark Statistics
The benchmark contains 1,000 instances across four domains, with checklist lengths constrained to three through seven items. Case studies show atomic decomposition converting evaluator feedback into actionable, evidence-grounded critique checklists.
- 1,000 instances span Code, General, Multilingual, and STEM domains, with 250 instances per domain.
- Checklist lengths are constrained to [3, 7], with average items per instance ranging from 4.08 in Multilingual to 4.58 in Code.
- Atomic decomposition extracts actionable critique points from factual-evaluation examples involving unsupported claims, irrelevant information, and hallucinated plot details.
- Table 5 reports the mean number of extracted checklist items per instance by domain.
- The case-study pipeline grounds critiques in response evidence, resolves cross-evaluator conflicts, removes redundancy, and records discarded subjective or contradictory statements.
B Creative Writing Dataset Annotation Process
The dataset combines creative-writing responses with detailed human evaluations, then uses MetaJudge to compare model-generated reasons against atomic human checklists. The process scores whether each human evaluation goal is substantively matched rather than merely mentioned.
- Dataset collection: Three annotators independently compare paired responses across popular science, film reviews, essays, and fiction, providing detailed assessments for both responses.
- MetaJudge evaluation: MetaJudge performs strict one-to-one semantic matching between model-generated evaluation reasons and human reference reasons.
- Case studies: Case studies show that identical preferences can conceal sharply different reasoning quality, including 8.3% consistency for generic critiques and 100% consistency for precise issue identification.
- Scoring rubric: The scoring rubric ranges from 0.0 for unaddressed or contradictory goals to 1.0 when the intended purpose, evidence, results, and constraints are fully satisfied.
- Scoring rubric: Partial scores distinguish vague or incomplete reasoning: 0.25 for slight contact, 0.5 for missing key steps, and 0.75 for mostly achieved goals lacking secondary details.
C.3 More Results
Across domains, outcome correctness generally correlates with reasoning quality, but rationale consistency reveals distinctions that outcome accuracy misses, especially among frontier models. Case studies extend this comparison to creative-writing evaluations.
- Cross-domain results: Outcome accuracy and rationale consistency are positively correlated across four domains, while o3-mini remains a notable low-consistency outlier.
- Cross-domain results: At the frontier, outcome accuracy saturates whereas rationale accuracy remains highly discriminative, making reasoning quality more informative for model comparison.
- Annotation examples: Figures 11 and 12 illustrate evaluator annotation and refinement for popular science and film-review responses, including factual errors, originality concerns, and plot misinterpretations.
- Creative writing: 8.3% rationale consistency was achieved by gemini-2.0-flash when its vague critiques missed key creative-writing checklist points.
D Training and Evaluation Details
The training pipeline uses GRPO for both GenRM training and downstream policy alignment, with structured evaluation prompts and rationale-aware supervision. Case studies show rationale quality improving over outcome-only training.
- GenRM training: GenRM training uses GRPO with 8 sampled responses per prompt, a 12K-token generation limit, an 8K-token prompt limit, and two training epochs.
- Policy alignment: Downstream policy alignment also uses GRPO, training the policy for 90 steps with the trained GenRM as its reward signal.
- Training comparison: Rationale+Outcome training prevents the rationale degradation observed under Outcome-Only training and surpasses that baseline.
- Evaluation protocol: The evaluation prompt requires comparative judgments with ordered reasons, while creative-writing guidelines require evidence-based strengths, weaknesses, and atomic labels.
- Training comparison: In the movelist case, Rationale+Outcome reaches 100% rationale consistency by identifying all three issues, versus 0% for the Outcome-Only model.
- Training comparison: In the Futurize case, Rationale+Outcome reaches 75% rationale consistency by identifying both flag issues, while Outcome-Only reaches 0%.