PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin Song

arXiv:2610.27528v1cs.CL

TL;DR

Generative reward models can give correct preference verdicts with flawed rationales, and reference-based rationale supervision does not establish that a rationale supports its verdict. DECERT evaluates rationale quality without reference explanations and uses verdict correctness and certificate quality to construct preferences. Across two model families and multiple in- and out-of-distribution benchmarks, it improves preference judgment and rationale quality, with position-consistent accuracy gains of up to 14.8 points over corresponding base judges.

  • Problem

    A correct GenRM verdict does not ensure sound reasoning, while reference-based supervision may not establish whether a rationale validly supports its verdict.

    Source: §1 · p. 1

  • Method

    DECERT assesses decision certificates for soundness, coverage, and decision support, then combines verdict correctness and certificate quality to construct preferences.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Across two model families and multiple in- and out-of-distribution benchmarks, DECERT consistently improves preference judgment and rationale quality; position-consistent accuracy rises by up to 14.8 points over corresponding base judges.

    Source: §1 · p. 2, §5 · p. 8, §4.5 · p. 8

  • Takeaways & Limitations

    Outcome-matched rationale preferences distinguish the quality of justifications even when judgments reach the same correct decision.

    Source: §1 · p. 2, §5 · p. 8

    Experiments cover pairwise preference judging with two 8B model families and a limited benchmark set, so whether the gains hold for larger models or other judging settings remains unstudied.

    Source: §6 · p. 9

Abstract

from arXiv · show

Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.

1 Introduction

Generative reward models can produce rationales that conflict with their verdicts, and reference-based supervision does not establish whether a rationale validly supports its verdict. DECERT assesses decision certificates and uses verdict correctness and certificate quality to supervise judgments; experiments report better preference judgment and rationale quality.

1 Introduction

  • GenRMs generate rationales before selecting a preferred response, but a correct verdict can still have flawed reasoning, while reference agreement may not establish valid support.Conflicting rationales and verdicts can mislead users, jeopardize safe deployment, and obscure biases.Source: §1 · p. 1, Figure 1 · p. 2
  • DECERT assesses decision certificates along Soundness, Coverage, and Decision Support rather than judging rationale quality by agreement with a reference explanation.A certificate is a comparative argument intended to support its preference verdict.Source: §1 · pp. 1–2
  • DECERT prioritizes verdict correctness across judgments, then uses certificate quality to form rationale preferences among judgments with the same correct verdict.These outcome-controlled pairs provide targeted supervision favoring better-supported justifications.Source: §1 · p. 2
  • 14.8 points: position-consistent accuracy improves by up to this amount over corresponding base judges.The improvement does not require swap-based augmentation or explicit positional-consistency supervision.Source: §1 · p. 2
  • Across two model families and multiple in- and out-of-distribution benchmarks, DECERT consistently improves preference judgment and rationale quality over outcome-only supervision.The paper also reports that DECERT outperforms strong generative judge baselines.Source: §1 · p. 2

2 Related Work

Prior work studies preference learning, generative judging, and rationale supervision, while DECERT focuses on whether a judge’s comparative reasoning supports its verdict. It assesses that reasoning without reference explanations and uses judgments sharing the same correct verdict to supervise justification quality.

Preference Learning and Reward Modeling

  • Preference labels identify which response is preferred but do not explain the supporting evidence, motivating evaluation of generated rationales for reward modeling.Traditional reward models learn scalar rewards from pairwise preferences, while DPO directly optimizes policies on preference data.Source: §2 · pp. 2–3

LLM-as-a-Judge and Generative Reward Modeling

  • GenRMs combine reasoning with preference prediction, but generating rationales does not guarantee that their evidence and reasoning support the predicted preference.Prior work also identifies sensitivity to response ordering as a systematic evaluation bias (Wang et al., 2024a).Source: §2 · p. 3

Rationale Supervision versus Process Reward Models

  • Unlike process reward models that supervise solver reasoning steps, DECERT assesses whether a judge’s comparative rationale supplies sound evidence, covers material issues, and supports its verdict.Its rationale preferences compare judgments with the same correct verdict, focusing on justification quality rather than step-level solution correctness.Source: §2 · p. 3

Rationale Supervision for Generative Judges

  • DECERT assesses input-grounded validity and verdict consistency without reference explanations, then forms rationale preference pairs among judgments with the same correct verdict.Prior rationale-supervision approaches include reference explanations, human critiques, self-generated critiques, evaluation criteria, and analysis preferences (Wang et al., 2026a; Lai et al., 2026; Wang et al., 2026e; Yu et al., 2025a; Liang et al., 2025; Wang et al., 2025; Qin et al., 2026).Source: §2 · p. 3

3 Decision-Certificate Supervision

DeCert assesses generated rationales as decision certificates and uses their quality alongside verdict correctness to supervise preference judgments. Its assessment checks soundness, coverage, and decision support before constructing training preferences.

3.1 Problem Formulation

  • Judges with equal outcome reward can produce different rationale distributions, motivating supervision of rationale quality alongside verdict correctness.Training instances provide a reference preference but no reference rationale.Source: §3.1 · p. 3

3 Decision-Certificate Supervision

  • DeCert scores judgments using rationale quality and verdict correctness, then uses the combined score to select SFT targets and construct DPO preference pairs.When all sampled judgments have the correct verdict, it forms additional preference pairs based on rationale quality.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Decision-Certificate Assessment

  • Soundness checks evidence and inference validity, coverage checks whether material issues are addressed, and decision support checks whether verified information supports the verdict.The claim extractor structures information already in the rationale without adding missing reasons; a geometric mean prevents a low score on one dimension from being offset by high scores on others.Source: §3.2 · pp. 4–5

3.3 Preference Data Construction

  • When pools include both correct and incorrect verdicts, DeCert ranks candidates so any correct-verdict judgment outranks any incorrect one; all-correct pools are ranked by rationale quality.The highest- and lowest-scoring candidates form general preference pairs, while all-correct pools use the highest and lowest rationale-quality scores; all-incorrect pools are discarded.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate DECERT on in- and out-of-distribution benchmarks, rationale-quality measures, counterfactual evidence edits, and response-order robustness. Across these settings, DECERT improves judgment and rationale outcomes, including positional consistency.

4.1 Experiments Setup

  • Experiments use HelpSteer3 training data, two base models from different families, and HelpSteer3, RM-Bench, and JudgeBench evaluation; comparisons include outcome-only variants and strong judge baselines.The training construction uses about 10k SFT samples and 8k DPO pairs, including about 3k rationale preference pairs.Source: §4.1 · pp. 5–6

4.2 Overall Judgment Performance

  • DECERT improves HelpSteer3 accuracy by 9.1 points for Llama-3.1-8B and 6.3 points for Qwen3-8B, and outperforms outcome-only variants on both OOD benchmarks.On RM-Bench, overall accuracy rises from 63.3 to 67.0 for Llama and from 73.5 to 76.2 for Qwen; on JudgeBench, positional consistent accuracy rises from 39.1 to 42.6 and from 48.3 to 52.0, respectively. DECERT-Qwen achieves the best overall performance among compared open generative judges on both OOD benchmarks.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 What Does Rationale Supervision Add?

  • Adding rationale preference pairs raises PCA from 46.7% to 52.9% on RM-Bench and from 39.1% to 42.6% on JudgeBench.These pairs rank judgments with the same correct verdict by certificate quality, adding supervision unavailable from outcome labels alone.Source: Figure 3 · p. 6, §4.3 · p. 7

4.4 Rationale Quality and Evidence Sensitivity

  • DECERT reduces all seven observed rationale flaw rates; non sequiturs fall from 21.5% to 3.3%, while generic unsupported claims remain the most common residual flaw.The flaw categories are non-exclusive, and lower rates indicate better rationale quality.Source: §4.4 · p. 8, Figure 5 · p. 8
  • On 1k edited pairs, DECERT raises EAR from 64.5% to 88.9% and lowers SRR from 34.0% to 11.1%; Qwen shows the same pattern.For Qwen, EAR increases from 87.2% to 93.4%, while SRR decreases from 30.5% to 7.3%.Source: Figure 4 · p. 7, §4.4 · p. 8

4.5 Positional Robustness

  • DECERT improves positional consistent accuracy by 14.8 points for Llama and 9.3 points for Qwen, without swap augmentation or an explicit positional-consistency objective.The gains exceed corresponding original-order accuracy gains of 9.1 and 6.3 points.Source: §4.5 · p. 8

5 Conclusion

The conclusion presents DECERT as a reference-free way to assess and supervise generative reward model rationales as decision certificates. It reports improvements in judgment, rationale quality, evidence responsiveness, and response-order robustness.

5 Conclusion

  • DECERT assesses whether rationales are well-supported, sufficiently comprehensive, and consistent with their verdict, rather than matching them to reference explanations.It uses the resulting certificate assessments to construct outcome-matched rationale preferences, distinguishing justifications for the same correct decision.Source: §5 · p. 8
  • Across two model families and multiple in- and out-of-distribution benchmarks, DECERT consistently improves preference judgment and rationale quality over outcome-only supervision.Analyses report fewer unsupported claims and reasoning inconsistencies, and counterfactual evaluations find more appropriate responses to changed decision-relevant evidence.Source: §5 · p. 8
  • DECERT improves response-order robustness without swap augmentation or an explicit consistency objective, supporting rationale supervision as a complement to verdict supervision.The conclusion frames this as a reference-free approach to training more reliable generative judges.Source: §5 · p. 8

6 Limitations

The paper identifies limitations in DECERT's assessment tools, experimental scope, and what its rationale analyses establish.

6 Limitations

  • Certificate assessments rely on LLM-based analyzers, so extracted issues may depend on analyzer reliability and biases.This constrains confidence in the assessments used by the framework.Source: §6 · p. 9
  • Experiments cover pairwise preference judging with two 8B model families and a limited benchmark set, leaving larger models and other judging settings unstudied.Source: §6 · p. 9
  • Rationale quality and evidence-sensitivity analyses do not establish that generated rationales faithfully reflect the model's latent decision process.Source: §6 · p. 9

A.1 Evaluation Policies

The evaluation policies define how judgments are parsed, material issues matched to claims, and audited comparisons converted into conservative decision support. Unresolved criteria remain potential evidence against the verdict, and geometric aggregation performs better than arithmetic aggregation on two challenging benchmarks.

A.1 Evaluation Policies

  • Judgments are parsed into criterion-level observations, inferences, winners, and a final verdict; unparseable judgments receive zero scores and are excluded from preference-pair construction.Parsing requires at least one criterion block with a valid result field and an answer selecting A or B.Source: §A.1 · p. 12

A.2 Material-Issue Extraction and Matching

  • The material-issue analyzer extracts up to six issues without seeing sampled rationales or verdicts, and one-to-one matching prevents a generic claim from covering multiple issues.Claims must address the same issue with a compatible assessment direction; inputs with no verifiable issue are excluded from training.Source: §A.2 · p. 12

A.3 Constructing Criterion Intervals from Verified Conclusions

  • Criterion intervals retain values compatible with locally valid comparative predicates; inconsistent predicates or invalid dependency records lead to certificate rejection.When no verified comparative information exists, the full interval [−1, 1] is retained; the intervals are not statistical confidence intervals.Source: §A.3 · p. 12

A.4 Closed-Form Conservative Decision Margin

  • Geometric aggregation achieves better performance than arithmetic aggregation on two challenging benchmarks.The conservative margin subtracts unresolved uncertainty, so wider unresolved intervals reduce support even when verified evidence favors the verdict.Source: §A.4 · p. 13

B.1 Training Objectives

The training pipeline selects judgments with supervised fine-tuning and then optimizes preference pairs with DPO. The rationale-score threshold has limited benefit as filtering reduces data, while geometric aggregation outperforms arithmetic aggregation on two benchmarks.

B.1 Training Objectives

  • Training first uses supervised fine-tuning on selected judgments, then initializes DPO from the SFT checkpoint and optimizes with the standard DPO loss.For pair construction, λ = 0.5, and groups whose parseable judgments are all incorrect are discarded.Source: §B.1 · p. 13

B.2 Training Parameter

  • Training uses LoRA; SFT runs for one epoch and DPO for three epochs, with the DPO regularization parameter β = 0.1.SFT uses an initial learning rate of 2 × 10^-4, while DPO uses 2 × 10^-6.Source: §B.2 · p. 13

B.3 Model Checkpoints and Decoding

  • The evaluation uses the public model checkpoints and decoding temperatures listed in Table 5.The table identifies checkpoint identifiers and the temperatures used for evaluation.Source: §B.3 · p. 14

C.1 Comparison of Different Rationale Score Thresholds

  • Increasing the rationale score-gap threshold yields only slight improvement, while reducing the amount of filtered training data.The default threshold is δ = 0.25.Source: §C.1 · p. 14

C.2 Comparison of Different Score Aggregation Strategies

  • Geometric aggregation achieves better performance than arithmetic aggregation on two benchmarks.The arithmetic alternative is used to aggregate total scores for preference-data construction.Source: §C.2 · p. 14

C.3 Ablation of Rationale Scoring

Controlled edits test whether rationale scores detect fabricated evidence, missing issues, and reversed verdicts while tolerating valid trade-offs and irrelevant true details. Geometric aggregation is more sensitive to harmful edits than arithmetic aggregation, with a modest increase in false positives.

C.3 Ablation of Rationale Scoring

  • The ablation applies five controlled edits to 2,000 judgments: two validity-preserving edits and three that introduce flaws.The flawed edits fabricate evidence, remove verdict-supporting criterion blocks, or reverse the verdict while leaving the rationale unchanged.Source: §C.3 · p. 14
  • A harmful edit counts as detected when its score change is below −τ, with τ = 0.05; success on validity-preserving edits requires a score change of at least −τ.This detection threshold is distinct from the score-gap threshold δ used to select rationale preference pairs.Source: §C.3 · p. 14
  • Individual scores detect different flaws: soundness detects 90.5% of fabricated evidence, coverage detects 64.1% of missing key issues, and decision support detects 95.5% of reversed verdicts.Their respective detection rates on the other two flaw types are lower, supporting their use as complementary checks.Source: §C.3 · p. 14
  • 89.5%, 67.4%, and 92.0% of fabricated-evidence, missing-key-issue, and reversed-verdict edits are detected by geometric aggregation, versus 86.5%, 62.5%, and 90.0% for arithmetic aggregation.Geometric aggregation has lower success rates on valid trade-off edits (83.5% vs. 86.0%) and irrelevant true details (79.5% vs. 81.5%).Source: §C.3 · p. 14, Figure 6 · p. 15
  • Missing key issues remain the hardest harmful perturbation for the combined score, leaving omitted-support detection as an area for improvement.Figure 6 reports success rates on a shared 0–100% scale, with higher values better.Source: §C.3 · p. 15, Figure 6 · p. 15

D Deployment of Analyzer

The analyzer comparison evaluates alignment with human-annotated material issues and reports cost alongside issue-extraction quality. GPT-4o-mini offers nearly GPT-4o's recall at substantially lower per-token cost, while Qwen3-8B remains competitive on the reported metrics.

D Deployment of Analyzer

  • Analyzer alignment is measured against reference issues on 500 human-annotated examples, with the comparison reporting issue-extraction metrics and API prices.Prices are reported per 1M input/output tokens.Source: D Deployment of Analyzer · p. 15, Table 8 · p. 16
  • GPT-4o-mini achieves nearly the same recall as GPT-4o while costing approximately 16.7× less per token.Qwen3-8B's F1 decreases by only 4.6%, with precision nearly unchanged; GPT-4o-mini was selected for its balance of effectiveness and efficiency.Source: D Deployment of Analyzer · p. 15, Table 8 · p. 16

E Case Study: Same Verdict, Different Rationale Quality

A STEM case study shows that two judgments can reach the same correct verdict while their rationales differ substantially in quality. The accompanying prompt templates show how certificates and material issues are elicited and checked.

E Case Study: Same Verdict, Different Rationale Quality

  • The example shows why verdict consistency alone is insufficient: same-verdict rationale preferences can supervise distinctions that outcome correctness misses.The rejected rationale endorses an incorrect factual assessment and misses the decisive calculation error, despite favoring the correct response.Source: E Case Study: Same Verdict, Different… · p. 16

F Prompt Templates

E Case Study: Same Verdict, Different Rationale Quality

  • Both sampled judgments correctly prefer Response A, but the rejected rationale validates Response B’s incorrect five-minute travel-time claim while the chosen rationale identifies the decisive conversion error.Response A’s travel time is approximately 6.62 hours; both rationales receive nearly identical Decision Support scores, while they differ sharply in Soundness and Coverage.Source: E Case Study: Same Verdict, Different… · p. 16, Table 9 · p. 17

F Prompt Templates

  • Figure 7 elicits structured judgment certificates, and Figures 8 and 9 verify observations and match material issues to certificate claims.Figure 10 prompts an analyzer to identify material issues from the comparison instance and evaluation policy without seeing candidate judgments.Source: Figure 7 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20

Related papers

Loading 2610.27528v1…