PaperlayerView exact revision on arXiv

Source-linked AI summary

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

Chao Xue, Yao Wang, Mengqiao Liu, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Chenyao Lu, Lei Jiang, Yu Lu, Haibo Shi, Shuang Liang, Minlong Peng, Flora D. Salim

arXiv:2604.10072v4cs.CL

TL;DR

Existing GRMs apply CoT indiscriminately and often evaluate reasoning paths with coarse voting, creating efficiency and reward-fidelity gaps. E-GRM uses parallel-generation convergence to trigger CoT selectively and a hybrid regression-ranking scorer for fine-grained evaluation, with experiments reporting improved efficiency and answer accuracy across reasoning benchmarks.

  • Problem

    Existing GRMs incur unnecessary computation by applying CoT uniformly and use coarse voting mechanisms that inadequately distinguish reasoning quality.

  • Method

    E-GRM uses model-internal uncertainty from parallel-generation convergence to trigger CoT selectively and applies a lightweight scorer trained with a hybrid regression-ranking objective.

  • Results

    Across multiple reasoning benchmarks, E-GRM consistently improves inference efficiency and answer accuracy over existing reward models.

  • Takeaways & Limitations

    Model-internal uncertainty provides an effective task-agnostic signal for efficient reasoning-aware reward modeling.

  • Takeaways & Limitations

    The method incurs modest parallel-decoding overhead, may require domain-specific consensus-threshold calibration, and can degrade for reasoning styles unlike its training data.

Abstract

from arXiv · show

Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.

1 Introduction

E-GRM targets two weaknesses in reasoning-enhanced generative reward models: unnecessary CoT computation and coarse evaluation of reasoning paths. It uses model-internal uncertainty to trigger reasoning selectively and a hybrid-loss scorer to improve reward fidelity.

  • GRMs improve complex reasoning with CoT but face persistent efficiency and reward-fidelity challenges.
  • Uniformly applying CoT to every input adds unnecessary latency and FLOPs for queries solvable by direct inference.Existing adaptive methods often depend on external, task-specific signals or handcrafted heuristics.
  • Voting-based aggregation evaluates reasoning chains coarsely and cannot reliably distinguish subtly higher-quality paths.This limits the precision of learned reward functions and can hinder final answer accuracy.
  • E-GRM estimates reasoning necessity from convergence across parallel generations, bypassing CoT for simple queries while selectively triggering it for harder ones.The approach uses model-internal uncertainty as a task-agnostic signal rather than external difficulty estimators.
  • A lightweight discriminative scorer combines robust regression and ranking to assign fine-grained quality scores to individual reasoning chains.Its hybrid objective uses Huber loss for regression robustness and hinge loss for ranking discrimination.
  • Across diverse reasoning tasks, E-GRM improves inference efficiency and answer accuracy relative to standard GRM baselines.

2 Related Work

Related work develops reward models and inference-time reasoning methods, but existing approaches often overuse CoT, rely on voting heuristics, and struggle to assess reasoning quality across domains. E-GRM combines adaptive CoT triggering with lightweight fine-grained scoring to address these gaps.

  • Reward modeling methods include scalar, semi-scalar, and generative models intended to improve reasoning fidelity, sample selection, and inference behavior.
  • Generative reward models frame reward modeling as generation, producing textual feedback or nuanced scores beyond scalar values.
  • Inference-time scaling research includes prompting and other methods, but many approaches prioritize reasoning quality over runtime efficiency.
  • Existing frameworks often overuse reasoning on simple tasks, rely on ensemble or voting heuristics, and lack dynamic adaptation to reasoning depth.
  • E-GRM adaptively triggers CoT only when necessary and selects outputs through fine-grained reward modeling instead of voting.The design targets reasoning quality and inference efficiency for task-adaptive deployment across domains.

3 Methodology

E-GRM dynamically routes inputs between direct inference and CoT reasoning using model-internal uncertainty, then scores difficult reasoning paths with a lightweight discriminative model. Its training combines short- and long-reasoning supervision with paired preference optimization, while inference bypasses unnecessary CoT and selects high-quality candidates.

  • Dynamic CoT Triggering via Model-Internal Uncertainty: E-GRM uses parallel decoding convergence to determine whether an input receives direct inference or explicit CoT reasoning.The framework treats high consensus as evidence for short reasoning and low consensus as a trigger for deeper reasoning.
  • Dynamic CoT Triggering via Model-Internal Uncertainty: Consensus(x) ≥ τ routes a prompt to its consensus answer, whereas lower consensus triggers explicit CoT generation.The threshold τ is empirically set to 0.8.
  • Discriminative Scoring with Hybrid Loss: The lightweight scorer evaluates candidate reasoning paths with a hybrid objective combining robust regression and discriminative ranking.Huber loss supports regression robustness, while hinge loss enforces ranking consistency between responses of different quality.
  • Training Pipeline: After supervised fine-tuning, paired preference optimization extends GRPO by directly contrasting preferred and dispreferred responses.The paired reward uses the discriminative scorer alongside answer correctness, with KL regularization retained in the policy objective.
  • Training Pipeline: E-GRM trains on short- and long-reasoning samples, teaching direct answers for short cases and step-by-step reasoning for long cases.The SFT dataset combines Dshort and Dlong, with long samples containing teacher-generated reference CoT sequences.
  • Inference Procedure: At inference, E-GRM estimates consensus, generates CoT candidates only for low-consensus inputs, scores them, and returns the highest-scoring response.This five-step protocol allocates computation according to inferred problem complexity while evaluating challenging paths with the scorer.

4 Experiments

Experiments evaluate E-GRM across reward-modeling benchmarks, triggering CoT dynamically and scoring reasoning paths discriminatively. Results show strong benchmark performance, reduced inference cost, and benefits from the extended preference-optimization formulation.

  • Evaluation setup: E-GRM is evaluated on RM-Bench, RMB, and RewardBench against scalar, generative, and structured reasoning reward-model baselines.The evaluation covers reasoning, coding, instruction following, helpfulness, and harmlessness.
  • Overall performance: 79.2% average on RM-Bench is achieved by the 32B variant, including 80.0% in Math and 70.7% in Hard.The 7B-to-32B progression rises from 70.1% to 79.2%.
  • Overall performance: 0.743 overall on RMB surpasses GPT-4o at 0.738, with first-place Harmlessness scores of 0.696 BoN and 0.823 Pairwise.The results report balanced Helpfulness and Harmlessness performance across both evaluation metrics.
  • Overall performance: 91.5% overall on RewardBench exceeds the next-best GenRM, Self-taught-evaluator-llama3.1-70B at 90.0%.The 32B model scores 95.4% in Reasoning and 92.0% in Safety; its 14B and 7B variants score 88.4% and 85.3% overall.
  • Dynamic CoT triggering: 58% of MATH samples bypass CoT, reducing average inference latency by 62% and FLOPs by 49% without accuracy loss.The comparison is against a forced-CoT baseline.
  • Preference optimization: The Extended GRPO variant consistently, though modestly, improves accuracy and alignment over standard GRPO on MATH and HelpSteer2.The variants use the same preference data and all other components remain identical.
  • Component ablation: Removing dynamic triggering raises FLOPs by 49%, latency by 55%, and lowers accuracy by 3.2 percentage points, while removing discriminative scoring causes a 5.6-point accuracy drop.The ablation uses Qwen-Instruct-14B on MATH; majority voting has similar computational efficiency but substantially lower accuracy.

5 Conclusion

E-GRM addresses efficiency and reward-fidelity limitations in reasoning-enhanced language models by triggering Chain-of-Thought selectively and scoring reasoning paths more finely. Across multiple reasoning benchmarks, it consistently improves efficiency and accuracy.

  • E-GRM dynamically triggers Chain-of-Thought reasoning using model-internal uncertainty from parallel decoding convergence.The framework uses this signal to invoke reasoning only when necessary.
  • Its hybrid regression–ranking discriminative scorer evaluates diverse reasoning paths beyond conventional voting or consistency mechanisms.The scorer provides fine-grained reward estimates for individual reasoning paths.
  • Across multiple reasoning benchmarks, E-GRM consistently outperforms existing reward models in efficiency and accuracy.

6 Limitations

The framework has modest parallel-decoding overhead, while its consensus threshold may require calibration beyond the tested benchmarks. Future work targets more efficient uncertainty estimation, adaptive thresholds, and broader scorer generalization.

  • 6 Limitations: Parallel decoding increases latency, but the reported increase remains within 5% and does not significantly affect overall efficiency.The paper characterizes this overhead as modest in practice.
  • 6 Limitations: The consensus threshold τ may require calibration for specialized domains beyond the tested benchmarks.
  • 6 Limitations: Future work includes more efficient uncertainty estimation, adaptive threshold mechanisms, and broader scorer generalization techniques.

A.1 Benchmarks

The study evaluates reward models using three established benchmarks with distinct evaluation focuses, including multi-domain assessment through prompt-chosen-rejected trios. RewardBench covers general chat, challenging dialogue, logical reasoning, and safety-critical contexts.

  • A.1 Benchmarks: RewardBench uses prompt-chosen-rejected trios across four domains for reward-model assessment.The domains are general chat, challenging dialogue scenarios, logical reasoning, and safety-critical contexts.
  • A.1 Benchmarks: RewardBench contains 358 general-chat samples, 456 challenging-dialogue samples, 740 logical-reasoning samples, and 1,431 safety-critical samples.
  • A.1 Benchmarks: RewardBench’s hierarchical structure provides multi-dimensional evaluation capabilities.

A.2 Datasets

The study uses six complementary preference datasets spanning mathematical reasoning, code generation, and general instruction following. These resources include step-by-step mathematical solutions, multi-dimensional response ratings, and granular feedback on reasoning validity.

  • A.2 Datasets: The evaluation uses six complementary preference datasets spanning mathematical reasoning, code generation, and general instruction following.
  • A.2 Datasets: MATH contains 12,500 competition-level mathematics problems with step-by-step solutions across seven difficulty levels.The standard split includes 7,500 training problems and 5,000 test problems.
  • A.2 Datasets: UltraFeedback contains 100K diverse instructions paired with multiple model responses and rated for helpfulness, safety, factual accuracy, and coherence.The dataset uses adversarial prompting to elicit varied response qualities.
  • A.2 Datasets: The datasets provide granular feedback on reasoning validity, including algebraic manipulation and proof-strategy error patterns.

A.3 Baselines

E-GRM is evaluated against scalar reward-modeling approaches and uses different training and inference strategies for simple versus complex reasoning tasks. Its inference pipeline returns direct answers when possible and applies diverse candidate generation with discriminative scoring when further reasoning is needed.

  • Baselines: E-GRM is evaluated against three classes of reward-modeling approaches, including scalar evaluation models that assign numerical preference ratings without explicit reasoning.Examples include traditional reward models, SteerLM-RM, Nemotron-RM, Starling-RM, ArmoRM, and Skywork.
  • Training Stage: During training, SFT uses tailored objectives for short and long samples to support direct answers and explicit step-by-step reasoning, respectively.Long-task reasoning chains are designed to culminate logically in final answers; the combined training uses a fixed batch size of 512.
  • Inference Stage: At inference, the model first produces a greedy output and returns it when ContainsCoT(r(0)) detects no need for further chain-of-thought reasoning.This routes simple prompts through direct inference rather than additional candidate generation.
  • Inference Stage: For complex samples, E-GRM generates K = 8 candidates with varied temperature and top-p settings, then selects the highest-scoring response using the trained reward model.The design combines diverse decoding with discriminative scoring to enhance reasoning on complex instances.
  • Evaluation Framework: The evaluation framework audits prompt-response pairs for safety, legal compliance, domain expertise, reasoning coherence, actionable implementation, and other quality dimensions.Its criteria include prompt coverage, off-topic content, problem-solving closure, misleading potential, knowledge gaps, and minimum-score or rejection rules.
Loading 2604.10072v4…