Source-linked AI summary
Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
Kirill Pavlenko, Alexander Golubev, Simon Karasik, Boris Yangel
TL;DR
Structured generations can suffer credit-assignment problems when GRPO applies one scalar advantage across multiple objectives and segments. The paper introduces Blockwise Advantage Estimation with Outcome-Conditioned Baselines to route local advantages and estimate later-block values from within-group statistics. On joint math reasoning and uncertainty estimation, the method matches RLCR on ECE and reasoning accuracy and extends to two-attempt refinement, while its baselines require populated outcome strata and stable segment boundaries.
Problem
Standard GRPO applies one scalar advantage across structured completions, coupling unrelated objectives and complicating credit assignment; later-block baselines also depend on sampled prefixes.
Method
Blockwise Advantage Estimation assigns objective-specific advantages to corresponding text blocks, while Outcome-Conditioned Baselines estimate later-block values from within-group outcome strata without additional rollouts.
Results
Across three models, the method performs on par with RLCR on Expected Calibration Error and reasoning accuracy, with broader applicability to two-attempt math refinement.
Takeaways & Limitations
The framework offers a modular approach for optimizing sequential objectives in structured generations without reward scalarization across segments or additional inference.
Takeaways & Limitations
Outcome-Conditioned Baselines can become noisy when outcome strata are rare, and the framework assumes segment boundaries are known and stable.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.
1. Introduction
Standard GRPO applies one scalar advantage across all tokens, which can misassign credit when structured generations contain separate objectives. Blockwise Advantage Estimation routes objective-specific advantages to corresponding segments, while Outcome-Conditioned Baselines estimate later-segment values from within-group statistics without extra rollouts.
- Standard GRPO maps multiple segment rewards to one scalar advantage applied uniformly across the completion, complicating credit assignment and requiring hand-designed reward combinations.
- Blockwise Advantage Estimation computes separate proxy-advantages for non-overlapping text blocks and updates each block using its local objective signal.
- Later-segment advantages require conditioning on sampled prefixes, making intermediate-state baseline estimation a central technical challenge.
- Monte Carlo rollouts from intermediate states can provide unbiased estimates but incur prohibitive compute for long-context and multi-thousand-token generations.
- Outcome-Conditioned Baselines approximate boundary-state values using only current-group statistics, reducing variance without additional inference.
- Across three models, the method performs on par with RLCR for Expected Calibration Error and reasoning accuracy, with broader applicability to two-attempt math refinement.
2. Related Work
Prior work addresses coarse credit assignment through finer-grained advantages, stratification, or reward design. These approaches either require value-function fitting or additional intermediate sampling, or combine multiple objectives through engineered rewards.
- Token-level advantage methods can provide finer learning signals but require fitting value functions that are difficult in diverse, prompt-conditioned LLM state spaces.
- VinePPO and Segment Policy Optimization estimate intermediate advantages by resampling continuations from selected prefixes, incurring additional sampling costs.
- Stratified Advantage Normalization addresses cross-stratum bias by normalizing advantages within homogeneous trajectory groups.
- Reward-design methods mix multiple desiderata into scalar rewards, often requiring tuning to manage trade-offs or reward exploitation.
3. Preliminaries
GRPO normalizes rewards within prompt-specific groups and applies a shared advantage across completion tokens. The paper evaluates confidence calibration by comparing reported probabilities with verifier-determined correctness using metrics including Brier score and ECE.
- 3.1. Group Relative Policy Optimization: GRPO samples a group of completions for each prompt and evaluates each completion with a scalar reward.
- 3.1. Group Relative Policy Optimization: A small positive constant epsilon provides numerical stability in the normalization.
- 3.1. Group Relative Policy Optimization: The group-normalized advantage is applied uniformly to all tokens in a completion within a PPO-style clipped objective.
- 3.2. Calibration Metrics for Confidence: Confidence calibration measures how closely reported confidence matches empirical correctness, with correctness determined by a verifier.
- 3.2. Calibration Metrics for Confidence: The Brier score measures squared error between predicted probability and binary outcome, while ECE compares average confidence with empirical accuracy across bins.
- 3.2. Calibration Metrics for Confidence: ECE depends on the selected number of bins and their boundaries.
4. Method
The method decomposes structured math completions into objective-specific blocks and applies block-local advantages, then estimates later-block baselines from within-group outcome strata rather than extra rollouts.
- Task setup: Because standard correctness rewards do not incentivize calibrated confidence, the method treats correctness and uncertainty estimation as separate objectives.
- Task setup: The running task separates a solution block Xsol, which primarily controls correctness, from a confidence block Xconf, which primarily controls calibration.
- Blockwise decomposition: Each completion is decomposed into K contiguous blocks with objective-specific rewards, while per-block token losses are averaged and optionally weighted to control block influence.
- Blockwise decomposition: Blockwise Advantage Estimation replaces the single completion-level advantage with proxy-advantages applied only to tokens in their corresponding blocks.
- Conditional baselines: For later blocks, differing sampled prefixes make the Group Mean Baseline potentially miscalibrated because it averages rewards across different intermediate states.
- Conditional baselines: Outcome-Conditioned Baseline partitions each GRPO group by a discrete prefix outcome and estimates later-block advantages from subgroup statistics, avoiding additional rollouts.
5. Experiments and Results
Experiments evaluate baseline fidelity, calibration, accuracy, test-time scaling, and broader sequential refinement. Outcome-Conditioned Baselines generally provide strong conditional estimates and competitive performance without scalarizing rewards across segments.
- Baseline fidelity: The Outcome-Conditioned Baseline achieves the lowest RMSE against Monte Carlo advantages for both correct and incorrect solution strata.The evaluation uses 100 MATH500 prompts and 32 confidence-block rollouts per fixed solution prefix.
- Main results: OCB yields ECE 0.030 versus 0.059 for RLCR on MATH500 with Qwen2.5-3B-Instruct, alongside improved AUROC and Brier performance.The reported accuracy gap is modest.
- Main results: Unconditioned group means can lower ECE while degrading accuracy and Brier performance, and may fail severely under distribution shift.For 7B-Instruct on GSM8K, the reported values are ECE 0.125, Brier 0.146, and AUROC 0.641.
- Main results: RLCR often leads on accuracy or hard-OOD discrimination, while BAE+OCB remains competitive and often improves the accuracy–calibration trade-off without reward scalarization.The comparison spans in-domain and easy-OOD regimes as well as hard-OOD evaluation.
- Test-Time Scaling Evaluation: Across MATH500, GSM8K, and AIME23–25, increasing test-time samples improves accuracy for realizable selection rules, while Pass@k indicates remaining headroom from diverse candidates.OCB tracks RLCR closely and preserves the downstream benefit of calibrated confidence.
- Reward variants: Training the confidence block with BCE converges more slowly than Brier in accuracy but reaches strong performance after sufficiently long training.The comparison is reported for Qwen2.5-7B-Base trained with BAE.
- Beyond Self-Confidence: Under Two-Attempt Refinement, the second attempt consistently improves over the first across all sampling budgets.This experiment applies BAE+OCB to Qwen2.5-3B-Instruct on MATH500.
6. Discussion, Limitations and Future Work
Outcome-conditioned grouping reduces variance for later-block advantages without additional rollouts, but its effectiveness depends on populated strata and suitable segmentation. The paper positions BAE as a modular approach to multi-objective RL, while identifying richer conditioning and broader evaluation as future work.
- Discussion: OCB is most effective when later-block rewards depend primarily on coarse task-relevant outcomes rather than fine-grained prefix properties.
- Discussion: Outcome-conditioned grouping provides variance reduction for later-block advantages without requiring additional rollouts.
- Limitations: Rare outcome strata can increase variance and potentially degrade calibration metrics, making sufficiently large or diverse groups important in practice.
- Limitations: BAE assumes stable, well-defined block boundaries; fuzzy or incorrect segmentation can require design choices and reintroduce cross-objective interference.
- Future Work: Future work includes multi-bin or learned outcome conditioning and evaluation beyond verifiable rewards.
- Discussion: The paper frames BAE as replacing hand-tuned reward scalarization with structured credit assignment aligned with how models generate text.
B. Additional Method Details
The algorithm samples grouped completions, computes block rewards and intermediate outcomes, estimates outcome-stratified baselines, and applies blockwise advantages only to corresponding tokens. Its unbiasedness requires the outcome to capture all prefix dependence relevant to the conditional value.
- Algorithm: BAE samples a GRPO group, parses each completion into blocks, and computes per-block rewards with intermediate outcomes for later blocks.
- Algorithm: For later blocks, OCB computes a mean within each outcome stratum and uses it to form proxy advantages.
- Algorithm: The blockwise clipped objective applies each estimated advantage only to tokens in its corresponding block.
- Unbiasedness: OCB is unbiased when a prefix-derived outcome captures all dependence of the block reward on the sampled prefix relevant to the conditional value.
C. Reward Structure for 2 Attempts Experiment
The two-attempt experiment uses correctness-based rewards with a small diversity bonus for incorrect second attempts. This keeps correctness dominant while discouraging verbatim repetition.
- Reward Structure: Each attempt produces a verifiable answer scored by the same correctness verifier, with correctness defined relative to the ground-truth answer.
- Reward Structure: For the second attempt, the reward is 1 when correct, 0.1 when incorrect but different, and 0 when incorrect and repeated.
- Reward Structure: The shaped second-attempt reward mildly favors trying a different answer after failure while preserving correctness as the dominant signal.
D. Evaluation Hyperparameters
Evaluation samples independent long-form completions and measures answer accuracy and calibration using verifier-based extraction and fixed-width ECE bins. Figure 4 reports ECE across MATH500, GSM8K, and AIME23–25.
- Inference Protocol: Evaluation samples n = 16 independent completions per prompt at temperature T = 1.0, with a maximum generation length of 32k tokens.
- Inference Protocol: Final answers are extracted and checked with math-verify, with boxed formatting used to reduce parsing failures.
- Calibration Metrics: Calibration uses Expected Calibration Error with fixed-width confidence bins.
- Calibration Metrics: Figure 4 compares ECE across MATH500, GSM8K, and AIME23–25.
E. Completion Example with Reasoning and Self-Analysis
The completion examples separate reasoning, analysis, answers, and confidence, illustrating both uncertainty expression and failure to solve a mathematical problem. The examples also motivate reward-hacking concerns and connect calibration with test-time scaling.
- Completion format: Generations explicitly separate reasoning, analysis, answer, and confidence fields.The examples include tagged reasoning and analysis blocks followed by confidence values such as 0.6 and 0.001.
- Reasoning and self-analysis: A geometry completion derives a result from similar triangles but reports relatively low confidence because approximate values were used.The reasoning invokes proportional side lengths, while the analysis links approximation to uncertainty in the final answer.
- Reward behavior: Naive reward schemes can create a local optimum in which the model refuses to answer while minimizing confidence, a form of reward hacking.The passage states that mitigation would require hyperparameter tuning or additional reward terms.
- Reasoning and self-analysis: A difficult integer problem produces an inability-to-determine answer with confidence 0.001 after the model reports unfamiliarity with the equation.The response attributes its failure to lack of familiarity and context.
- Reward behavior: OCB with BCE optimization of the confidence block shows no signs of reward hacking.
- Test-time scaling: In the well-calibrated regime, test-time scaling depends more on sampled-solution quality and diversity than on marginal calibration improvements.Confidence primarily serves as a tie-breaker and robustness mechanism when accuracy and sample quality are preserved.
H. Group Size Ablation for OCB Method
Outcome-Conditioned Baselines estimate later-block values from outcome-defined subgroups within a GRPO group. Increasing group size slightly improves calibration but reduces accuracy and AUROC in the reported runs.
- OCB mechanism: Outcome-Conditioned Baselines average rewards within outcome-defined subgroups to estimate conditional values for later blocks.Larger groups generally provide more samples per outcome class, reducing subgroup-mean variance while increasing sampling cost.
- Experimental setup: The ablation compares GRPO group sizes G = 32 and G = 64 while holding the batch size, training steps, model, and other settings fixed.
- Results: Increasing group size from 32 to 64 slightly improves calibration across all three benchmarks but reduces accuracy and AUROC.The authors attribute this trade-off to seeing twofold fewer distinct examples during training.
I. Full Results with Standard Errors
The evaluation reports uncertainty for accuracy and calibration metrics using SEM. Pass@1 uses empirical combination-level variation, while globally computed calibration metrics use bootstrap resampling.
- Reporting: Evaluation metrics are reported in mean±SEM format.
- Pass@1 uncertainty: Pass@1 SEM is computed as σ/√n from metric values across sampled combinations.Here, σ is the sample standard deviation and n is the number of combination-level evaluations.
- Calibration uncertainty: ECE, Brier score, and AUROC SEM use 1,000 bootstrap resamples of confidence-correctness pairs.Each iteration recomputes the calibration metric, and SEM is the standard deviation of the resulting bootstrap distribution.