PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense

Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Sharon Li, Jason E Weston, Ping Yu

arXiv:2510.07242v3cs.CLcs.LG

TL;DR

Binary verifiers provide reliable but coarse feedback, while reward models offer richer scores that can be misaligned with correctness. HERO combines the signals using verifier-stratified normalization and variance-aware prompt weighting; it outperforms RM-only and verifier-only baselines across mathematical reasoning regimes.

  • Problem

    Binary verifiers can under-credit partially correct or alternative answers, while unconstrained reward-model scores may be noisy or misaligned with correctness.

    Source: §1 · p. 2, §3.2 · p. 5

  • Method

    HERO bounds reward-model scores within verifier-defined correctness groups and uses variance-aware weighting to emphasize more challenging prompts.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    HERO consistently outperforms RM-only and verifier-only baselines across verifiable, hard-to-verify, and mixed regimes and across backbones.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results support a hybrid reward design that retains verifier stability while adding dense, trajectory-sensitive feedback across the evaluated math-reasoning regimes.

    Source: §6 · p. 10

    Naively blending continuous reward-model scores with binary correctness checks can destabilize training and misalign rewards with correctness semantics.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Post-training for reasoning of large language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle--many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.

1 Introduction

Binary verifiers provide reliable but coarse feedback, while reward models offer richer but potentially misaligned scores; HERO combines them to improve supervision. Across easy-to-verify, hard-to-verify, and mixed regimes, HERO outperforms RM-only and verifier-only baselines.

Motivation

  • Strict binary verification can under-credit valid partial or alternative answers and provide no useful group-relative gradient when all rollouts receive the same label.This sparsity can reduce sample efficiency and leave difficult prompts underutilized.Source: §1 · p. 2

Approach

  • HERO combines verifier-anchored signals with dense reward-model feedback through stratified normalization and variance-aware weighting.The normalization bounds reward-model scores within verifier-defined correctness groups, while weighting emphasizes harder prompts where scores better distinguish responses.Source: §1 · p. 2

Results

  • HERO consistently outperforms RM-only and verifier-only baselines across easy-to-verify, hard-to-verify, and mixed regimes and across LLM backbones.On hard-to-verify tasks evaluated with Qwen-4B-Base, HERO scores 66.3, versus 54.6 for RM-only training and 57.1 for verifier-only training.Source: §1 · p. 2
  • Ablations indicate that anchoring dense signals to verifiable correctness and adaptively reweighting difficult prompts are both critical for stability and efficiency.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe reward models as a source of continuous response-quality scores and verifiers as a source of sparse correctness signals. GRPO optimizes groups of candidate responses using relative advantages.

Dense reward via reward modeling

  • A reward model learns a scalar response-quality score and can guide reinforcement learning toward human preferences.It is typically trained on pairwise preference data by minimizing a Bradley–Terry loss.Source: §2 · pp. 2–3

Sparse reward via verifier

  • Verifiable-reward training uses deterministic checks to assign sparse correctness rewards, typically 1 for correct and 0 for incorrect responses.For math problems, a verifier compares a parsed solution with the reference under equivalence transformations.Source: §2 · p. 3

Group Relative Policy Optimization

  • GRPO compares multiple candidate solutions per prompt and assigns relative advantages, with clipping and a KL penalty to stabilize optimization.Its group-based formulation helps alleviate gradient sparsity relative to pure verifier rewards.Source: §2 · p. 3

3 Methodology

The methodology examines verifier and reward-model behavior on hard-to-verify math tasks, then introduces HERO to combine correctness-anchored dense feedback with prompt-difficulty weighting. The design addresses verifier brittleness and the instability of naively blending reward signals.

3.1 Motivation: Delving into Rule-based vs. RM-based Verifiers

  • The study evaluates verifier behavior on HardVerify_Math, a benchmark of difficult math-verification cases, and asks whether reward models trained on verifiable data generalize to them.Responses from three models are used for the verifier analysis.Source: §3.1 · pp. 3–4
  • Rule-based verifiers can be precise but overly conservative on hard-to-verify samples, whereas reward models provide dense distinctions such as partial correctness and proximity to the ground truth.These complementary strengths motivate anchoring supervision in symbolic verification while enriching it with reward-model signals.Source: §3.1 · pp. 3–4
  • On hard-to-verify tasks, reported verifier results include FPR=0.3% with recall=10.1%; math_verify.py (in verl) yields only 38.6% recall and remains brittle to mismatched list-versus-set orderings despite normalization heuristics for formatting differences or units.For AceMath-7B-RM, threshold 1 yields 91.7% recall with lower precision; raising the threshold improves precision while decreasing recall.Source: §3.1 · pp. 3–4

3.2 HERO: Hybrid Ensemble Reward Optimization

  • Naively blending continuous reward-model scores with binary correctness checks can destabilize training and misalign rewards with correctness semantics.Source: §3.2 · p. 5
  • HERO partitions responses by verifier outcome and min–max normalizes reward-model scores within each group, adding quality distinctions while preserving correctness ordering.This can create reward differences among rollouts even when all receive the same verifier label.Source: §3.2 · p. 5
  • HERO weights prompts according to reward-score variance: higher variance signals more disagreement and a richer training signal, while trivial low-variance prompts are down-weighted.The final design allocates more learning effort to challenging, informative prompts.Source: §3.2 · p. 5

4 Experiments

HERO is evaluated across easy-to-verify, hard-to-verify, and mixed training settings, using two model backbones and several test sets. It consistently outperforms RM-only and verifier-only baselines, while ablations examine dense reward ranges, prompt reweighting, and reward-model size.

4.1 Experimental setup

  • The setup samples 2,000 examples for each single-regime dataset and combines 1,000 easy-to-verify with 1,000 hard-to-verify examples for mixed training, using two SFT-initialized backbones.Evaluation covers six test sets: four easy-to-verify and two hard-to-verify; hard-to-verify outputs are judged by GPT-4o, while easy-to-verify results use pass@1 over eight seeds.Source: §4.1 · p. 6

4.2 Main results

  • On Qwen3-4B-Base, easy-to-verify training yields an average score of 62.0, versus 56.4 (RM-only) and 58.3 (rule-based training).On hard-to-verify training, HERO scores 56.8 versus 55.1 (RM-only) and 47.4 (rule-based); mixed training reaches 58.8 on verifiable tasks and 64.1 on hard-to-verify tasks.Source: §4.2 · p. 7
  • Across all three training regimes, HERO outperforms RM-only and verifier-only baselines; on hard-to-verify evaluations, it reaches 66.3 versus 54.6 (RM-only) and 57.1 (rule-based training).The reported hard-to-verify comparison is for HERO trained on easy-to-verify data.Source: §4.2 · pp. 6–7
  • Across training settings, Qwen3-4B-Base results favor HERO, while the reported Qwen and OctoThinker averages also show gains across verifiable and hard-to-verify evaluations.For Qwen3-4B-Base, mixed training averages 40.2 on verifiable and 33.2 on hard-to-verify tasks; for OctoThinker-8B-Hybrid-Base, hard-to-verify training yields 41.0/34.6 versus 35.4/30.9 (RM-only) and 34.6/27.8 (verifier-only).Source: §4.2 · pp. 6, 8

4.3 Additional ablations

  • Dense negative rewards improve performance when added to sparse verifier-positive rewards, from 59.4 to 61.4 on verifiable tasks and from 62.2 to 68.4 on hard-to-verify tasks.The ablation attributes the gains to negative rewards supplying broader feedback on diverse reasoning errors.Source: §4.3 · p. 9
  • Reward-range settings depend on task type: smaller ranges perform best on verifiable tasks, while larger ranges benefit mixed settings until excess range adds noise or overfitting.The reported examples are α = 0.05 for verifiable tasks and α = 0.1 or α = 0.2 for mixed tasks.Source: §4.3 · p. 9
  • Variance-aware reweighting improves both verifiable and hard-to-verify accuracy, with larger gains on hard-to-verify tasks (+3.8).It emphasizes high-variance prompts and downweights trivial ones.Source: §4.3 · p. 9
  • Replacing HERO’s 7B reward model with a 72B model yields only a marginal verifiable-task improvement (62.8 vs. 62.0) and lower hard-to-verify performance (71.4 vs. 73.2).The paper attributes HERO’s gains primarily to hybrid reward design rather than reward-model scaling.Source: §4.3 · pp. 9–10

5 Related Work

Prior work uses rule-based verification, learned judges, and other strategies to provide supervision for reasoning tasks, but each approach has limits. HERO retains rule-based correctness signals and combines them with dense learned rewards.

Reinforcement learning from verifiable rewards

  • Rule-based rewards support filtering, training, and evaluation, but can miss semantically correct outputs that differ textually; LLM-based verifier coverage and reward sparsity remain concerns.Related approaches include outcome-driven RL, rubric-anchored RL, verifier-free RL, and cross-domain RLVR.Source: §5 · p. 10

Reasoning on hard-to-verify tasks

  • LLM-as-judge methods offer nuanced assessments for hard-to-verify outputs, but may be unreliable on domain-specific or long-form data.Other cited approaches use likelihood estimation or latent reasoning steps and discard the verifier component.Source: §5 · p. 10
  • Unlike approaches that discard verifiers, HERO combines rule-based verification with continuous learned rewards, anchoring dense signals to symbolic correctness.The paper also describes Peng et al. (2025) as directly combining verifiable correctness signals and human preferences for agentic tasks.Source: §5 · p. 10

6 Conclusion

The conclusion presents HERO as a hybrid reward method that anchors reward-model scores to verifier-defined correctness and weights informative prompts. Across the tested regimes and backbones, it outperforms RM-only and verifier-only baselines.

6 Conclusion

  • HERO uses stratified normalization to anchor reward-model signals to verifier-defined correctness and variance-aware weighting to emphasize informative prompts.The conclusion says this design supplies dense, trajectory-sensitive feedback while mitigating gradient sparsity and RM-only drift.Source: §6 · p. 10
  • Across verifiable, hard-to-verify, and mixed regimes and across backbones, HERO consistently outperforms RM-only and verifier-only baselines.Source: §6 · p. 10
  • Future work includes stronger difficulty estimators, process-level rewards, and extensions beyond math reasoning.Source: §6 · p. 10

A.1 Experimental setup

The appendix specifies task-dependent reward ranges and fixed bounds for variance-aware reweighting. It also documents training configurations and the verification library used across experiments.

A.1 Experimental setup

  • For easy-to-verify tasks, α = β = 0.05; for mixed and hard-to-verify tasks, α = β = 0.1.The tighter range is intended to minimize noise with precise verifiers, while the relaxed range provides richer feedback where reward models contribute more.Source: §A.1 · p. 14
  • Variance-aware reweighting uses wmin = 0.4, wmax = 3.0, and logistic steepness k = 6 to down-weight trivial prompts and emphasize highly uncertain ones.Source: §A.1 · p. 14
  • Training configurations document OpenMathReasoning data, rule-based and reward-model signals, and use of the HuggingFace math_verify library for standardized verification.The configurations cover data preparation, optimization, rollout generation, reward specification, and trainer settings for the two reported backbones.Source: §A.1 · p. 14

A.2.1 Supervised Fine-Tuning dataset preparation

The authors use cold-start SFT to address instability when RL begins directly from base models, then construct training data from verifier-passing and verifier-failing OpenMathReasoning examples.

A.2.1 Supervised Fine-Tuning dataset preparation

  • Direct RL from base models was often unstable, so training began with two SFT epochs using responses generated by the base model itself.The responses were filtered for a correct final answer, English-only text, and no unstop issues; the final cold-start set contained 2,000 samples.Source: §A.2.1 · pp. 14–15

A.2.2 Training data filter from OpenMathReasoning

  • From 40k CoT examples with extractable answers, the authors sampled 2k verifier-passing examples and 2k verifier-failing examples for RL training.The two sets serve as verifiable and hard-to-verify training samples, respectively.Source: §A.2.2 · p. 15

A.2.3 Hard-to-verify evaluation benchmark from TextBookReasoning

The TextBookReasoning benchmark is filtered to retain challenging hard-to-verify questions, and GPT-4o is prompted to judge answer equivalence without solving the problem. The reported comparison finds HERO ahead of two model-based verifiers across the evaluated regimes.

A.2.3 Hard-to-verify evaluation benchmark from TextBookReasoning

  • Filtering first excluded dataset answers that passed the math_verify check, targeting questions and answers with complexity or ambiguity challenging for standard verifiers; this process yielded approximately 750 prompts for hard-to-verify task evaluation.Questions also had to be clear and unambiguous, factual and answerable, not require proof or explanation, and be direct questions rather than statements.Source: §A.2.3 · pp. 15–17

A.2.4 Hard-to-verify prompt

  • GPT-4o evaluates hard-to-verify answers by checking equivalence to the reference answer rather than re-solving the question.The prompt supplies the question, ground truth, and student answer, and asks for a yes/no correctness decision.Source: §A.2.4 · p. 17
  • HERO consistently outperforms General Reasoner and Qwen2.5-7B-Instruct across easy-to-verify and hard-to-verify regimes.Results are pass@1 averaged over 8 seeds.Source: §A.2.4 · p. 18

A.3 More experiments

HERO performs strongly against model-based verifiers across verifiable, hard-to-verify, and mixed settings, while reward-model-only training can exhibit reward hacking on harder data. The experiments also report that naive reward fusion is suboptimal and that reward-model gains on hard-to-verify data can coincide with accuracy collapse.

A.3 More experiments

  • HERO scores 56.5 versus General Reasoner’s 54.0 and Qwen2.5-7B-Instruct’s 54.5 on hard-to-verify tasks, and 64.1 versus 59.0 and 63.3 in the mixed setting.On verifiable samples, HERO’s average score is 62.0, compared with 58.4 for General Reasoner and 62.5 for Qwen2.5-7B-Instruct; the authors also report greater stability across datasets.Source: §A.3 · p. 18
  • Naive rule-based and reward-model fusion remained suboptimal, with scores ranging from 55.9 to 58.7 for verifiable tasks and 60.2 to 61.4 for hard-to-verify tasks.Without constraining continuous scores within verifier correctness criteria, the fusion can produce noisy or misaligned rewards and destabilize learning.Source: §A.3 · pp. 18–19
  • On OpenMath Non-Verifiable data, the reward mean rose from below 5 to over 30 within 100 steps while MATH500 accuracy fell from around 0.75 at step 50 to below 0.2 at step 100.Accuracy was effectively zero by step 150 despite reward mean exceeding 40; the mixed data also showed accuracy crashing to nearly zero by step 150 as reward continued rising.Source: §A.3 · p. 19

B.1 Reward model qualification ability

The reward model usually ranks verifier-correct responses above incorrect ones, but low-AUROC groups remain. Qualitative verifier examples show a precision–recall trade-off that motivates anchoring dense scores to rule-based correctness.

B.1 Reward model qualification ability

  • The reward model’s AUROC has a mean of 0.79 and median of 0.92; 56.8% of groups reach excellent AUROC (≥0.9), while 13.7% fall in the random/poor range (0.4–0.6).Roughly 80% of groups have AUROC above 0.7, though low outliers remain.Source: §B.1 · p. 20

B.2 Qualitative analysis of rule-based verifiers

  • Rule-based verifiers can miss formatting variants or structurally different valid answers, while permissive model judges can credit partial answers and risk over-crediting.The examples contrast strict symbolic correctness with model-judge flexibility, motivating HERO’s use of rule-based correctness to anchor dense signals.Source: §B.2 · pp. 20–21

C Limitations and Future Work

HERO's benefits are bounded by verifier quality, reward-model calibration, training complexity, and evaluation noise; the authors identify broader verifier coverage and adaptive supervision as future directions.

Loading 2510.07242v3…