PaperlayerView exact revision on arXiv

Source-linked AI summary

Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, Xuwei Ding, Pengcheng Xu

arXiv:2610.00574v1cs.LGcs.CL

TL;DR

Multi-reward RL needs to optimize several behavioral objectives, but GDPO’s reward-wise normalization still permits uneven learning. The paper derives DARA from the relationship between advantage energy and active-group density, and reports faster acquisition of targeted behaviors than GDPO.

  • Problem

    GDPO preserves reward-specific relative information through reward-wise normalization, yet learning progress across objectives remains uneven, raising the question of how to derive reward weights from the underlying mechanism rather than repeated tuning.

    Source: §1 · pp. 1–2

  • Method

    DARA uses the relationship between batch advantage energy and active-group density to derive adaptive density-based weights that strengthen signals from less frequently active rewards.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    DARA reaches high format compliance on tool calling and near-saturated length compliance on mathematical reasoning in up to 26% and 65% fewer training steps than GDPO, respectively, while remaining competitive in final performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Active-group density is a measurable quantity for understanding and calibrating reward contributions in multi-reward RL.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance. Our code is available at https://github.com/zhaihaotian/DARA.

1 INTRODUCTION

Multi-reward RL must balance objectives such as correctness, format, and tool use, yet GDPO’s reward-wise normalization does not prevent uneven learning. The paper explains this imbalance through active-group density, derives DARA to calibrate reward contributions, and reports faster learning than GDPO on tool calling and mathematical reasoning.

Motivation

  • Multi-reward RL represents distinct behavioral requirements as separate signals because a single reward may not capture correctness, length, format, safety, and reliable tool use.GDPO preserves reward-specific relative information, but its experiments still show uneven learning across objectives.Source: §1 · pp. 1–2

Mechanism

  • Under idealized GDPO normalization, a reward’s batch-level advantage energy is proportional to its active-group density, revealing residual signal imbalance after reward-wise normalization.Active-group density is the fraction of rollout groups where a reward provides nonzero relative advantages.Source: §1 · p. 2

Method

  • DARA matches each reward’s energy to that of the highest-density reward, giving larger weights to less frequently active rewards when they provide useful comparisons.Its inverse-square-root density correction adapts weights to reward activity during training.Source: §1 · p. 2

Empirical evaluation

  • Up to 26% and 65% fewer training steps than GDPO were needed to reach high format compliance on tool calling and near-saturated length compliance on mathematical reasoning, respectively.DARA remained competitive in final performance.Source: §1 · p. 2

2 RELATED WORK

Prior work develops general LLM reinforcement-learning methods and multi-reward approaches that address reward collapse, weighting, conflict, or specialized training. DARA differs by deriving reward weights from the relationship between active-group density and advantage energy.

Reinforcement Learning for Large Language Models

  • GRPO estimates advantages through within-group reward comparisons without a separate value model [Shao et al., 2024], while GSPO, DAPO, and Dr. GRPO address sequence-level ratios, stability, efficiency, or length-related bias.Search-R1 [Jin et al., 2025] enables search-augmented reasoning, and GiGPO [Feng et al., 2025] improves credit assignment in long-horizon agent training.Source: §2 · p. 2

Multi-Reward Reinforcement Learning

  • Multi-reward methods weight rewards by within-group standard deviation, batch-level variation, or remaining reward distance, while other approaches filter conflicting rollouts, emphasize low-scoring dimensions, or merge specialized teachers.GDPO [Liu et al., 2026b] addresses reward collapse through reward-wise group normalization.Source: §2 · p. 3
  • DARA grounds aggregation in active-group density’s relationship to advantage energy, identifying imbalance left by reward-wise normalization and deriving weights to equalize reward energies.Its aggregation operates orthogonally to policy-update mechanisms and domain-specific task reward design.Source: §2 · p. 3

3 METHOD

The method analyzes reward-wise signals at the batch level: idealized GDPO energy scales with active-group density, motivating density-based weights in DARA. DARA calibrates those signals before aggregation and uses the same clipped policy-optimization objective as GDPO.

3.1 PRELIMINARIES

  • Reward-wise normalization preserves individual reward contrasts, but rewards may be active in different fractions of rollout groups, producing different batch-level contributions.GDPO normalizes each reward dimension independently before aggregation.Source: §3.1 · p. 3

3.2 FROM ACTIVE GROUPS TO ADVANTAGE ENERGY

  • Active-group density counts groups where a reward produces a nonzero relative advantage; for binary rewards, difficult or nearly saturated rewards can have fewer mixed-outcome groups than intermediate rewards.A group is active when its reward values differ across responses.Source: §3.2 · pp. 3–4
  • Under idealized GDPO assumptions, each active group contributes G −1 energy, so total batch advantage energy scales linearly with active-group density.This exposes residual imbalance after reward-wise normalization; under an explicit covariance condition, the energy also governs fluctuations in policy-gradient contributions.Source: §3.2 · p. 4

3.3 DENSITY-AWARE REWARD AGGREGATION

  • DARA derives density weights by matching each reward’s energy to the batch’s highest-density reward, amplifying less frequently active rewards when they provide useful comparisons.A cap limits amplification; zero-density rewards receive none, and capped weights may prevent exact energy matching.Source: §3.3 · p. 4
  • DARA-Asym applies the density-based amplification only to positive advantages, retaining negative coefficients at GDPO scale; DARA-Sym amplifies both signs.DARA-Asym is the main method, and the exact energy-matching result applies to uncapped DARA-Sym under the stated assumptions. Calibrated advantages are aggregated, batch-normalized, and used with GDPO’s clipped objective.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Across tool-calling and mathematical-reasoning experiments, DARA speeds acquisition of targeted behaviors and remains competitive in final performance. The experiments also examine performance trade-offs, reward interactions, and how learning changes as reward activity becomes sparse.

4.1 TOOL-CALLING; 4.2 MATHEMATICAL REASONING

  • Tool-calling experiments use format and correctness rewards on two Qwen2.5 models; mathematical-reasoning experiments use correctness and length rewards on three models.The tool-calling evaluation uses BFCL-v4, while mathematical reasoning is evaluated on five benchmarks and tracks pass@1 and Exceed.Source: §4.1 · p. 5, §4.2 · p. 7

4.1 TOOL-CALLING

  • Format reward reaches 0.8 at steps 14–15 for DARA, versus 19 for GDPO and 34 for GRPO.At step 60, both DARA variants also have the highest Average Accuracy and Average Format among compared methods; at step 100, DARA remains competitive.Source: §4.1 · pp. 5–6
  • Larger rollout groups raise format active-group density and bring the 0.8 reward threshold earlier; DARA reaches it before GRPO and GDPO at every tested size.The experiments vary group size from 4 to 32 while holding responses per step at 2,048.Source: §4.1 · p. 6, Figure 3 · p. 6
  • Adding a length reward lowers GDPO Average Format from 97.11% to 94.39%, while DARA's Average Format changes less and both variants exceed GDPO on Average Len.Average Format changes from 97.90% to 96.93% for DARA-Asym and from 97.55% to 97.43% for DARA-Sym.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 MATHEMATICAL REASONING

  • Length compliance rises faster than correctness during early mathematical-reasoning training, and DARA-Asym generally retains higher correctness reward than DARA-Sym.As compliance improves, mixed compliant and over-length rollout groups become rarer, making relative length comparisons sparser.Source: §4.2 · p. 7, Figure 4 · p. 7
  • At 99% training-side length compliance, DARA-Sym reaches the threshold in 59 steps on Qwen3-4B-Instruct and 50 on DeepSeek-R1-7B, about 47–65% fewer steps than GDPO.Near saturation, most sampled responses satisfy the length constraint, decreasing active-group density; DARA weights the remaining informative comparisons more heavily.Source: §4.2 · p. 8
  • Both DARA variants first reach 95% held-out length compliance at step 50, compared with step 70 for GDPO.On DeepSeek-R1-7B, DARA-Sym attains lower Exceed rates, while DARA-Asym retains higher accuracy.Source: §4.2 · p. 8, Figure 6 · p. 8
  • DARA-Asym achieves the highest Joint scores on both DeepSeek-R1 models, 45.14% and 57.83%, respectively, together with the highest accuracy.Table 3 reports aggregate mathematical-reasoning results at step 50.Source: §4.2 · pp. 8–9, Table 3 · p. 9

5 CONCLUSION

The paper links reward activity to batch-level advantage energy and introduces DARA to strengthen signals from less frequently active rewards. Experiments show faster, more stable learning on tool calling and mathematical reasoning, while final performance remains competitive.

5 CONCLUSION

  • DARA applies a density correction to strengthen signals from rewards that provide useful comparisons less frequently.The conclusion reports that it learns targeted behaviors faster and more stably, reaching high task performance in fewer training steps.Source: §5 · p. 10
  • Under idealized GDPO normalization, a reward’s batch-level advantage energy is proportional to its active-group density.The paper presents active-group density as a measurable quantity for understanding and calibrating reward contributions.Source: §5 · p. 10
  • The authors report that AI coding assistants helped implement training and evaluation scripts and process logs, while authors developed the research questions, analysis, and experimental design.They state that AI-assisted code and reported results were reviewed against raw outputs.Source: AI USE STATEMENT · p. 10
  • The paper says it will release code and training configurations, with derivations and task-specific experimental details documented in the appendices.The listed details include reward definitions, evaluation protocols, benchmarks, decoding parameters, and metric definitions.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DERIVATION OF THE DENSITY–ENERGY RELATION

Under the derivation’s idealized GDPO normalization, each active rollout group contributes fixed advantage energy, while inactive groups contribute none. Consequently, total reward-wise energy scales with active-group density.

A.1 DERIVATION OF THE DENSITY–ENERGY RELATION

  • Each active rollout group contributes exactly G −1 units of advantage energy under the derivation.This is the fixed per-active-group contribution used to obtain the batch-level density relation.Source: §A.1 · p. 12
  • Inactive groups contribute zero energy because identical reward values yield zero centered rewards and reward-wise advantages.The derivation counts reward k as active in exactly Bπk groups.Source: §A.1 · pp. 12–13
  • Reward-wise normalization therefore makes total advantage energy vary linearly with active-group density.With Bπk active groups and fixed energy per active group, the relation recovers Equation 5.Source: §A.1 · p. 13

A.2 DERIVATION OF THE DENSITY CALIBRATION

The appendix derives DARA’s density calibration and examines its implications for reward-wise gradient signals, group-size effects, and positive-only amplification. It also reports logged-batch energy comparisons for mathematical-reasoning training.

A.2 DERIVATION OF THE DENSITY CALIBRATION

  • Before the practical cap, density calibration removes the dependence of reward-wise advantage energy on the original active-group density.The derivation matches each reward’s calibrated energy to the reference energy.Source: §A.2 · pp. 13–14
  • At step 40, length energy is 31–46% of correctness energy under GDPO, while DARA matches them; after the cap binds, DARA’s median ratio is 0.39–0.49.The cap binds at steps 59, 62, and 48 for the three models; thereafter the GDPO ratio stays below 0.04.Source: §A.2 · p. 14, Figure 7 · p. 14, Figure 8 · p. 14

A.3 FROM ADVANTAGE ENERGY TO REWARD-WISE POLICY GRADIENTS

  • For binary rewards, each mixed group contributes a positive multiple of the per-prompt gradient that raises success probability; contributions across prompts may still point in different directions.DARA-Asym’s positive-only term contributes a positive multiple of that same per-prompt gradient.Source: §A.3 · p. 16
  • Under the stated covariance condition, density calibration equalizes gradient fluctuations up to factors 1 ± δ for uncapped rewards, while DARA-Sym batch normalization preserves their relative magnitudes.The policy-gradient analysis uses the density–energy identity as an idealization where that identity is invoked.Source: §A.3 · pp. 15–16

A.4 ACTIVE-GROUP DENSITY UNDER HETEROGENEOUS PROMPTS AND FIXED BUDGETS

  • At a fixed response budget, larger groups raise both active-group density and advantage energy per generated response, although fewer prompts are sampled.The density monotonicity also holds without independence when groups of different sizes are prefixes of one rollout process.Source: §A.4 · p. 17

A.5 ENERGY OF POSITIVE-ONLY AMPLIFICATION

  • DARA-Asym’s energy is not exactly matched like DARA-Sym’s, but it remains between the GDPO energy and the DARA-Sym weight-squared upper bound.This applies both before and after batch centering, and the transformation amplifies only positive coefficients.Source: §A.5 · p. 18

B ADVANTAGE COMPUTATION OF MULTI-REWARD METHODS

This section compares multi-reward advantage estimators and describes how DARA computes batch-specific advantages. Its implementation replaces GDPO advantages while keeping the policy objective unchanged.

B.1 COMPARISON OF ADVANTAGE ESTIMATORS

  • Reward-wise group normalization and rollout-batch normalization use their respective group and batch means and standard deviations, with ε added to the denominator.The DARA advantage computation is given for one rollout batch with K rewards; its resulting advantages replace GDPO advantages without changing the policy objective.Source: §B.1 · p. 18, §B.2 · p. 19
  • DARA calibrates each reward by its frequency of providing a nonzero within-batch comparison, unlike methods weighting by dispersion, attainment, or conflicting signs.Table 5 summarizes the statistics used to determine each method’s reward contribution.Source: Table 5 · p. 18, §B.1 · p. 19

B.2 PSEUDOCODE

  • The pseudocode computes active-group density, sets a reference density to the maximum across rewards, and caps density-based weights before producing response advantages.The returned response-level advantage is assigned to each generated token.Source: §B.2 · p. 19

C.1 IMPLEMENTATION AND OPTIMIZATION DETAILS

  • All methods use the same verl-based training stack, matched model, rollout, optimizer, and budget settings, with only the reward aggregation rule changed.Table 6 lists the complete configuration for the main tool-calling experiments.Source: §C.1 · pp. 19–20, Table 6 · p. 20

C.2 BFCL-V4 EVALUATION PROTOCOL

The BFCL-v4 protocol evaluates final checkpoints with a shared prompting and parsing interface, then aggregates task-subset scores and training runs as described.

C.2 BFCL-V4 EVALUATION PROTOCOL

  • Each final checkpoint is evaluated once on every BFCL-v4 case through the same ToolRL prompting and response-parsing interface.The official checkers assess function names and arguments for Live and Non-Live tasks, and execution states and dialogue outcomes for Multi-Turn tasks.Source: §C.2 · p. 20, Table 7 · p. 20
  • The reported Average equally averages the three BFCL-v4 subset scores, and each metric is reported as the mean across training seeds.Metrics are computed independently for each training run; the base model is evaluated once.Source: §C.2 · p. 20
  • Average Format is the arithmetic mean of Live, Non-Live, and Multi-Turn format scores, with each case scored from the assistant output’s required block order and formatting.When a case has multiple assistant outputs, their format scores are averaged within the case first.Source: §C.2 · p. 20

C.3 ADDITIONAL ANALYSIS CONFIGURATIONS

The additional analyses vary rollout-group size under a fixed response and update budget, and define a length reward for the three-reward experiments.

C.3 ADDITIONAL ANALYSIS CONFIGURATIONS

  • The rollout-group analysis varies G while holding the total at 2,048 responses and four optimizer updates per training step.Prompt batch and PPO batch sizes change together; other settings match the main experiments, with configurations listed in Table 8.Source: §C.3 · p. 20, Table 8 · p. 21
  • The three-reward experiments add a length reward that scores one when the extracted <think> segment contains at most 16 whitespace-delimited words, and zero when required thinking tags are absent.These runs use G = 4 and otherwise retain the main configuration.Source: §C.3 · p. 21
  • On BFCL-v4, Len. averages the length reward using the same output-to-case and subset aggregation as Format.The added length channel is combined with correctness and format in the three-reward experiments.Source: §C.3 · p. 21

D.1 IMPLEMENTATION AND OPTIMIZATION DETAILS

The mathematical-reasoning experiments use a shared training setup and matched comparisons, then construct correctness and length reward channels for the advantage estimators. DARA weights channels by how often they are informative and normalizes the combined response values across the batch.

D.1 IMPLEMENTATION AND OPTIMIZATION DETAILS

  • Mathematical-reasoning runs use the full DeepScaleR-Preview collection, filter prompts above 1,024 tokens, and report checkpoints from the first 100 updates.A seeded 200-example monitoring set is sampled from the training collection for online validation, not as an independent test benchmark.Source: §D.1 · p. 21, Table 9 · p. 21
  • Comparisons at a given scale share the backbone, data, seed, training budget, and throughput configuration unless stated otherwise.The experiments use DeepSeek-R1-1.5B and 7B, Qwen3-4B-Instruct, and Qwen3-4B-Thinking for the thinking-mode comparison.Source: §D.1 · p. 21
  • The actor uses FSDP, responses are generated with vLLM, and the training stack is verl-based.Table 9 lists the shared mathematical-reasoning training configuration.Source: §D.1 · p. 21, Table 9 · p. 21
  • Dynamic group filtering retains prompt groups with nonconstant summed sequence rewards, using the same retained groups across compared methods within a run configuration.Sampling continues toward 512 informative groups, subject to configuration-specific generation-batch caps.Source: §D.1 · p. 22

D.2 REWARD CONSTRUCTION AND ADVANTAGE ESTIMATORS

  • GRPO normalizes the fused scalar reward once, whereas GDPO normalizes correctness and length separately, sums their advantages equally, and applies masked batch whitening.The correctness and length rewards are binary channels; estimators without separate channels receive their sum as a scalar reward.Source: §D.2 · p. 22
  • DARA estimates each channel’s informative-group fraction using τ = 10^-8 and weights channels relative to the highest-density channel; its asymmetric and symmetric variants differ in scaling negative advantages.Combined response values are normalized across the batch using the sample standard deviation, then broadcast to valid response tokens; static controls fix correctness and length weights at 1 and 5.Source: §D.2 · pp. 22–23

D.3 MATHEMATICAL-REASONING EVALUATION PROTOCOL

The protocol evaluates checkpoints on a five-benchmark mathematical-reasoning suite using the training prompt, then reports accuracy, length, and joint metrics with specified aggregation and uncertainty procedures.

D.3 MATHEMATICAL-REASONING EVALUATION PROTOCOL

  • The suite covers five public mathematical-reasoning benchmarks, totaling 1,559 questions and 24,944 generations per checkpoint and evaluation seed.Evaluation uses the training prompt, and ground-truth rendering is normalized for specified AMC and OlympiadBench answer formats.Source: §D.3 · p. 23
  • Checkpoint-wise early-training analysis evaluates steps 10 through 100 in increments of 10.Source: §D.3 · p. 23

D.4 METRICS AND AGGREGATION

  • Accuracy is an equal-question-weighted mean sample accuracy, not the probability that at least one of 16 responses is correct.The metric is specified for benchmarks with Q questions and 16 sampled responses per question.Source: §D.4 · p. 23
  • The numerical-equivalence fallback runs only after strict grading returns zero, can only rescue exact numerical rendering differences, and excludes general symbolic, interval, radical, or multi-answer equivalence.Strict and fallback-adjusted accuracies are both retained in the raw summaries.Source: §D.4 · pp. 23–24
  • Exceed measures the fraction of generated responses whose vLLM-returned token count is strictly greater than 4,000.Source: §D.4 · p. 24
  • Joint Success requires correctness and the training-time length criterion; truncated responses remain in the denominator, while macro metrics are unweighted means with question-resampled 95% confidence intervals.Intervals use 2,000 nonparametric question resamples for each benchmark and evaluation seed.Source: §D.4 · p. 24

Related papers

Loading 2610.00574v1…