PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Yushu Li, Jiin Woo, Xiaoxiao Li, Youngsuk Park

arXiv:2610.21404v1cs.LGcs.AI

TL;DR

Reward aggregation can improve while primary-task performance degrades, leaving open when and why it favors auxiliary behaviors. The paper analyzes this failure and proposes TAG, which elects an anchor using isolated policy drift and gates auxiliary credit on anchor satisfaction. Across evaluated settings, TAG mitigates correctness degradation; on Qwen3-4B with a brevity reward, it achieves 0.581 pass@4 versus 0.220 for additive aggregation.

  • Problem

    The paper asks whether reward aggregation can undermine a primary objective even when that objective is explicitly rewarded.

    Source: §1 · p. 2

  • Method

    Trust-Anchored Gating elects an anchor based on isolated policy drift, then conditions auxiliary credit on anchor satisfaction.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    0.581 pass@4 for TAG versus 0.220 for additive aggregation on Qwen3-4B with a brevity reward; responses were 44% shorter than with correctness-only training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When the primary channel is anchored, TAG reverses auxiliary-favoring selection in the analytical setting while preserving incentives for joint success.

    Source: §7 · p. 11

    TAG assumes one reward channel is the primary objective, and low policy drift does not guarantee alignment with the unknown true reward.

    Source: §7 · p. 11

Abstract

from arXiv · show

To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes. We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.

1 INTRODUCTION

The paper asks whether reward aggregation can undermine an explicitly rewarded primary objective, and finds that symmetric aggregation can favor easier auxiliary behavior. It proposes Trust-Anchored Gating (TAG) and reports that it mitigates correctness degradation while retaining auxiliary benefits.

Problem

  • Symmetric aggregation can raise aggregate reward while correctness collapses, even when reward channels are individually meaningful.The paper calls this failure multi-reward hacking: aggregation itself can induce it, rather than only a flawed or incomplete proxy.Source: §1 · p. 2

Analysis

  • When auxiliary rewards are more reliably obtained, symmetric rules that reward single-channel success drive conditional-mean GRPO toward auxiliary-favoring behavior from any initialization.The result holds even when the policy initially favors the primary objective.Source: §1 · p. 2

Trust-Anchored Gating

  • TAG elects an anchor by isolated policy drift and gates auxiliary credit on anchor satisfaction, eliminating credit for auxiliary-only success while preserving extra credit for joint success.With the primary channel anchored, it converges to primary-favoring behavior from any initialization for any positive gating weight, whenever primary success remains possible.Source: §1 · pp. 2–3

Empirical validation

  • Across Qwen and Llama models, mathematics and program synthesis, TAG mitigates correctness degradation under conventional aggregation while retaining auxiliary benefits.On Qwen3-4B, TAG reaches 0.581 pass@4 versus 0.220 for additive aggregation, with responses 44% shorter than correctness-only training.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies proxy-reward over-optimization, multi-reward combination, and reward auditing; this paper focuses on how aggregation can itself favor an auxiliary channel over a primary objective.

Reward hacking and over-optimization

  • Prior reward-hacking work studies proxy over-optimization, imperfect proxies, and mitigation through regularization or aligned update directions.These approaches generally consider one proxy against the true reward, whereas this paper studies interaction among combined channels.Source: §2 · p. 3

Multi-reward and rubric-based RL

  • Multi-reward methods combine channels through weighted sums, constraints, normalization, policy interpolation, or rubric-based approaches, while classical multi-objective RL treats specified objectives as trade-offs.The paper asks whether aggregation drives optimization toward auxiliary behavior and notes that Pareto optimality among channels need not suffice against an unknown true reward.Source: §2 · p. 3

Reward auditing and reward selection

  • Reward auditing methods assess reward models or behaviors, while anchor election probes each channel alone and ranks candidates by induced policy drift.The paper positions anchor election earlier in the training pipeline and as attributing behavior to a channel.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper formulates post-training as scalar aggregation of multiple reward channels and studies when the aggregate can be hacked relative to an unknown true reward. Its analysis uses a two-channel, one-step decision model and conditional-mean GRPO dynamics.

3.1 PRELIMINARIES

  • GRPO replaces PPO’s value model with the mean reward of a response group, while Dr.GRPO also has an advantage variant without standard deviation.The setup writes the objective without clipping and KL-penalty terms.Source: §3.1 · pp. 3–4
  • Reward hacking is defined by a reversal in policy ordering: the proxy prefers one policy while the true reward prefers another.The pair is hackable when both inequalities are strict for policies in the specified policy set.Source: §3.1 · p. 4

3.2 MULTI-REWARD RL FOR POST-TRAINING

  • The multi-reward formulation maps a reward vector to a scalar through an aggregation rule, while an unknown function of the channels represents the true reward used to judge hacking.The aggregate is treated as the proxy reward.Source: §3.2 · p. 4
  • Multi-reward hacking occurs when an aggregate-optimal policy is strictly preferred by the proxy over an alternative that the true reward strictly prefers.It is a property of the aggregate, not of any individual channel.Source: §3.2 · p. 4
  • The analysis includes GRPO variants trained on aggregated reward and MO-GRPO, which standardizes each reward channel separately.The passage introduces the channel-specific standardized advantages.Source: §3.2 · p. 4

3.3 A TWO-CHANNEL MDP FOR ANALYSIS

  • The two-channel MDP uses binary primary and auxiliary rewards and three tabular-softmax actions with distinct outcome distributions.The response is abstracted away; each action produces a reward outcome, and expected aggregate reward is averaged over actions.Source: §3.3 · p. 4
  • Training is represented by a deterministic conditional-mean recursion for parameter updates after on-policy GRPO steps.The update uses a positive step size and averages the change conditional on the current parameters.Source: §3.3 · p. 4
  • Compatibility η measures the chance of joint success, while ρ measures primary success outside the joint-success component.η = 0 means no response satisfies both channels; larger η means a single behavior more often serves both objectives.Source: §3.3 · p. 5

4 FAILURE OF SYMMETRIC REWARD AGGREGATION

In the two-channel analysis, symmetric aggregation selects the auxiliary-favoring action when auxiliary success is easier to obtain. This selection constitutes hacking when the unknown true reward prefers the primary-favoring action.

4.1 ACTION SELECTION UNDER SYMMETRIC AGGREGATION

  • Under symmetric, monotone aggregation, conditional-mean GRPO converges to the auxiliary-favoring action a1 from any initialization.The result assumes η, ρ < 1, n ≥ 2, and positive step size; when channels are incompatible and ρ < 1, the same convergence follows.Source: §4.1 · p. 5

4.2 WHEN SYMMETRIC AGGREGATION HACKS THE TRUE REWARD

  • For a nondecreasing true reward, symmetric aggregation is multi-reward hacking exactly when it values the primary-favoring action a2 above the aggregate-optimal a1.For the illustrated linear true reward, the symmetric rules’ true reward falls toward 0.19 as a1 takes over, below 0.83 for a2.Source: §4.2 · pp. 5–6
  • On-policy sampling compounds a1’s advantage, and normalized and constrained variants also fail, motivating a remedy that breaks symmetry.Figure 3 shows symmetric methods handing the policy to a1 after initially raising a2, while TAG retains a2.Source: §4.2 · p. 6, Figure 3 · p. 6

5 BREAKING THE SYMMETRY WITH TRUST-ANCHORED GATING

TAG breaks aggregation symmetry by electing an anchor from isolated policy-drift probes and gating auxiliary credit on anchor satisfaction. With the primary channel anchored, it provably selects the primary-favoring action in the analyzed setting.

5.1 ANCHOR ELECTION: WHICH CHANNEL TO TRUST

  • TAG elects the channel with the smallest per-token reverse KL after short, isolated GRPO probes from the same base policy.The probes are used only for election; joint training restarts from the base policy.Source: §5.1 · p. 6
  • Reverse KL is sensitive to invented modes because it samples from the probe policy, unlike forward KL, which can miss how far the policy moved.For an invented mode at distance ∆, the figure shows reverse KL growing with ∆2 while forward KL stays at the added mass.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 TRUST-ANCHORED GATING: CONDITIONING CREDIT ON THE ANCHOR

  • Static auxiliary down-weighting is calibration-sensitive: on hard prompts, group standardization can erase the weight, so reweighting delays but does not prevent collapse.The reported weight sweep compares held-out primary reward for multiple auxiliary weights against TAG and primary-only training.Source: §5.2 · p. 7
  • TAG gives no additional credit when the anchor is unsatisfied, but distinguishes joint success from primary-only success to retain auxiliary incentives.When the primary channel is the anchor, deterministic conditional-mean GRPO converges to the primary-favoring action from any finite initialization for λ > 0 and ρ > 0.Source: §5.2 · pp. 7–8

6 EXPERIMENTS

Experiments compare reward aggregation methods with primary-only training across Qwen and Llama models, mathematics, and program synthesis. Conventional aggregation often trades correctness for auxiliary behavior, while TAG generally mitigates degradation and retains auxiliary benefits.

6.1 SETUP

  • The experiments use correctness plus synthetic auxiliary signals across Qwen3 models, with held-out pass@4 and auxiliary behavior rates as outcomes.The tested baselines include reward-level rules and advantage-level estimators; TAG uses the probe-elected anchor.Source: §6.1 · p. 8

6.2 AGGREGATION-INDUCED DEGRADATION AND TAG

  • Under Correct & brief, conventional aggregations collapse to bare-answer templates, while TAG reaches pass@4 of 0.453 and 0.581 with responses 33% and 44% shorter.The corresponding correctness-only pass@4 values are 0.481 and 0.634, so TAG trades 3–5 percentage points for shorter correct responses.Source: §6.2 · p. 8, Table 1 · p. 9
  • Under Correct & negative, GRPO, Dr.GRPO, and MO-GRPO drive negative answers nearly to 100% and pass@4 to 0.02–0.05, while TAG matches or exceeds primary-only pass@4.Static reweighting does not reliably prevent collapse: on Qwen3-1.7B tested weights yield primary reward 0.02–0.03, versus 0.31 for TAG and correctness-only training.Source: §6.2 · p. 9, Table 1 · p. 9

6.3 PROGRAM SYNTHESIS

  • TAG is the only tested arm that preserves the primary-only pass@4 reference in both program-synthesis settings, while Correct & fast also reduces mean latency.On Qwen3-1.7B latency is 307 vs. 359 ms, and on Qwen3-4B it is 434 vs. 486 ms; on Qwen3-4B, correct programs faster than the reference occur at 0.215 vs. 0.164 for primary-only.Source: §6.3 · p. 10, Table 2 · p. 10

6.4 ISOLATED DRIFT AND ANCHOR ELECTION

  • In anchor-election probes, auxiliary channels drift 2.8–23× farther than matched correctness probes on mathematics and 12–21× farther on program synthesis, so the rule elects correctness in every cell.Forward KL is less sharp: it fails on mathematics and is two to three times less sharp on program synthesis.Source: §6.4 · p. 10, Figure 7 · p. 11

6.5 GENERALIZATION ACROSS MODELS, SCALES, AND REWARD CHANNELS

  • Across Llama-3.2-3B, Llama-3.1-8B, and Qwen3-30B-A3B, conventional methods show correctness collapse in tested settings, while TAG matches or maintains near-reference accuracy.With three simultaneous rewards, additive aggregation and Rubric Dropout exploit auxiliary rewards and degrade correctness, whereas TAG suppresses the negative-answer shortcut while retaining brevity benefits.Source: §6.5 · pp. 10–11

7 CONCLUSION

The paper concludes that symmetric aggregation can improve aggregate reward while degrading primary-task performance, and presents TAG as a drift-elected, anchor-gated defense. Its claims and practical scope are bounded by assumptions about the primary channel, probe cost, and simplified theory.

7 CONCLUSION

  • Symmetric aggregation can drive GRPO toward auxiliary-favoring behavior when auxiliary rewards are easier to obtain, while TAG conditions auxiliary credit on an elected anchor.The paper reports experimental mitigation across mathematics and program synthesis while retaining benefits from compatible auxiliary rewards.Source: §7 · p. 11
  • TAG assumes one reward channel deserves primary status; handling multiple primary channels or none deserving priority remains future work.Anchor probes add computational cost, and low policy drift does not guarantee alignment with the unknown true reward.Source: §7 · p. 11
  • The theoretical guarantees are established in a simplified two-channel, binary-reward model, with broader reward settings and larger reward sets left for future work.The experiments use synthetic auxiliary channels in controlled training runs; no deployed system was attacked.Source: §7 · p. 11

A DEFERRED PROOFS

Appendix A.1–A.3 collect proofs and numerical companions for the paper’s theoretical results and related thresholds.

A DEFERRED PROOFS

  • Appendix A.1 proves Theorem 1 and derives the recursion in the incompatible regime.It also records extensions to other estimators and addresses the boundary cases ρ = 1 or η = 1.Source: A DEFERRED PROOFS · p. 15
  • Appendix A.2 provides the proof of Theorem 2.Source: A DEFERRED PROOFS · p. 15
  • Appendix A.3 supplies numerical companions for Figure 3, MO-GRPO thresholds, and the weight thresholds cited in Section 5.2.Source: A DEFERRED PROOFS · p. 15

A.1 PROOF OF THEOREM 1

The proof derives conditional-mean GRPO dynamics from centered group advantages and shows how their signs drive policy selection. When η, ρ < 1, the auxiliary-favoring action is selected from every finite initialization; boundary behavior and estimator-specific scope are treated separately.

A.1 PROOF OF THEOREM 1

  • Centering gives E[∆θj | θt] = απj mj, while the GRPO conditional mean advantages are continuous, bounded, and ordered by action reward.The proof uses monotonicity of the standardized advantage in the response’s own reward and a shared-component coupling to compare actions.Source: §A.1 · p. 15
  • When η, ρ < 1, the conditional-mean GRPO recursion converges to π1 → 1 from every finite initialization.The proof establishes π0 → 0 and π2 → 0; no condition on α beyond α > 0 is used.Source: §A.1 · p. 16
  • The convergence argument also applies to mean-baseline Dr.GRPO and exact policy gradient, but generally does not apply to MO-GRPO.For MO-GRPO, the stated selection result is limited to boundary cases; in the compatible regime it can select a1 below a ρ-dependent threshold and be bistable above it.Source: §A.1 · p. 16
  • For η = 0, the explicit drift makes the a1-versus-a0 log-odds increase strictly, although the a1-versus-a2 log-odds can move transiently toward a2.For ρ < 1, the recursion still converges to π1 → 1 from every finite initialization.Source: §A.1 · p. 17
  • On the boundary ρ = 1 or η = 1, GRPO and Dr.GRPO select whichever of a1 and a2 starts with greater probability; an exact tie persists.On the tie, a1 and a2 each tend to probability 1/2 while a0 tends to zero.Source: §A.1 · pp. 17–18

A.2 PROOF OF THEOREM 2

The proof shows that Trust-Anchored Gating makes auxiliary-only outcomes receive no positive advantage, then analyzes the resulting policy dynamics. With the primary channel anchored and η < 1, ρ > 0, the recursion converges to the primary-favoring action from every finite initialization.

A.2 PROOF OF THEOREM 2

  • TAG assigns zero reward to auxiliary-only and neither outcomes, so either receives strictly negative advantage when the group has a higher-reward response.If all group rewards coincide, every advantage is zero by convention; this holds for any λ ≥ 0 and any η, ρ.Source: §A.2 · p. 20
  • With η < 1 and ρ > 0, TAG converges to π2 → 1 from every finite initialization, with no condition on α beyond α > 0.The argument establishes π0 → 0 and π1 → 0; it also applies to the mean-baseline Dr.GRPO advantage.Source: §A.2 · p. 20
  • When η = 0, the two-level TAG drift is independent of λ and gives positive expected log-odds drift toward a2 over both alternatives.For η > 0, λ affects the update only in groups containing both joint success and a primary-only realization.Source: §A.2 · p. 20
  • A smaller λ, corresponding to a larger primary-only bonus, increases m2 − m1 at every state, strictly when the other responses include a joint success.The proof’s λ-dependent comparison concerns groups that contain both joint-success and primary-only outcomes.Source: §A.2 · p. 20
  • At η = 1 or ρ = 0, the TAG rewards of a1 and a2 have the same law, while at ρ = η = 0 every realized reward is zero and the logits remain constant.Selection can be slow near the auxiliary vertex when η is close to one or ρ is small; the convergence statement is asymptotic.Source: §A.2 · p. 21

A.3 NUMERICAL COMPANION

The numerical recursions show that symmetric GRPO constructions can select auxiliary-favoring behavior even with joint success, while TAG retains the primary-favoring action. Static reward weights have setting- and initialization-dependent safe regions, unlike TAG’s stated guarantee for every ρ > 0.

A.3 NUMERICAL COMPANION

  • The exact-recursion numerical companion enumerates realized action-outcome group counts and applies the expected logit update, with a sampled-group check at the boundary case.The deterministic runs use n = 8, α = 0.1, and 105 iterations; at the exact tie, 1,000 sampled seeds split nearly evenly among the three symmetric constructions’ auxiliary-only selections.Source: §A.3 · p. 21
  • At η = 0.1, all four symmetric constructions select a1 while TAG keeps a2; at η = 0.2, MO-GRPO keeps a2 while the three Theorem 1 constructions select a1.For η = 0.1, MO-GRPO takes 7,186 iterations to 0.99 versus 10,235 for GRPO, 22,445 for Dr.GRPO, and 19,637 for exact policy gradient; TAG takes 395. At η = 0.2, TAG takes 446 iterations.Source: §A.3 · p. 21
  • In the compatible regime, MO-GRPO’s primary vertex changes stability at a numerically observed η threshold, equal to 0.18 at ρ = 0.9 for n = 8.At η = 0.2 it remains at a2 from the specified initialization, unlike the three constructions covered by Theorem 1, which select a1.Source: §A.3 · p. 21
  • TAG selects a2 for every ρ > 0, whereas the GRPO safe weight region closes at low ρ and can depend on initialization.For low-ρ prompts, no single positive global auxiliary weight is uniformly safe under GRPO; the passage attributes this to the finite-group z-score behavior.Source: §A.3 · p. 22
  • The compatible-regime weight sweep finds that GRPO requires larger primary-to-auxiliary weight ratios as ρ falls, roughly tracking 1/ρ.The smallest grid ratios are 1.14, 1.30, 1.96, and 3.88 for ρ = 0.9, 0.7, 0.5, and 0.3, respectively.Source: §A.3 · p. 22
  • At ρ = 0.7, an intermediate weight band yields a mixed policy; at ρ = 0.3, the recursion is bistable and selects different actions from different initializations.The mixed policy is π = (0, 0.74, 0.26) at w2/w1 = 0.9; at ρ = 0.3 and w2/w1 = 0.2, the two initializations select a2 and a1, respectively.Source: §A.3 · pp. 22–23

B.1 METHOD DETAILS

The method probes each reward channel separately from the same base policy, then elects the channel with the smallest per-token reverse-KL drift as anchor. The probe policies are discarded before joint training resumes from the base policy.

B.1 METHOD DETAILS

  • Isolated probes reveal behavior induced by each reward channel before joint optimization can make the behavior’s source difficult to identify.After probing from the shared base policy, the probe policies are discarded and joint optimization restarts from that base policy.Source: §B.1 · p. 23
  • The election statistic evaluates the log-ratio under the probe policy because that policy generates the behavior created by isolated optimization.The opposite KL direction can be less sensitive to newly created response modes with low probability under the base policy.Source: §B.1 · p. 23
  • Per-token normalization divides the sequence-level reverse KL by expected generated-token count, making probes with different response lengths comparable.Without normalization, the summed per-response log-ratio gives sequence-level reverse KL by the chain rule.Source: §B.1 · p. 23
  • The channel with the smallest reverse-KL drift is elected as the anchor; correctness has the smallest drift in every setting reported in the paper..Source: §B.1 · p. 23
  • The election assumes at least one channel has no low-base-probability shortcut, and the convergence theorem is stated when the elected anchor is primary.In the three-action model, a saturated probe’s reverse drift equals −log π0(aj | x), so election favors the isolated optimum more typical under the base policy.Source: §B.1 · p. 23

B.2 DIRECTIONAL SENSITIVITY OF KL TO MODE INVENTION

The analysis explains why reverse KL is more sensitive than forward KL to probability mass placed in a mode rare under the base policy. Simulations show stronger detection of invented modes under benign drift, while mode dropping reverses the KL comparison.

B.2 DIRECTIONAL SENSITIVITY OF KL TO MODE INVENTION

  • Forward KL is bounded by the invented mass alone, while reverse KL can grow without bound as the invented region becomes rarer under the base policy.For a set A with invented mass at least ϑ and base probability q ≤ εϑ, the reverse-KL lower bound grows without bound as q approaches zero.Source: §B.2 · p. 24
  • A new mode contributes little to forward KL when base-policy samples never visit it, but reverse KL charges its probe-policy log-ratio when the probe visits it.This is why the paper’s collapsed behaviors—such as instant short answers, negative constant answers, and trivial programs—are treated as inventions.Source: §B.2 · p. 24
  • At ε = 5% and ∆ = 6, KLfwd = 0.0505 and KLrev = 0.704, a 14× ratio; for mode dropping, KLfwd = 8.31 and KLrev = 0.689.The mode-invention example uses a Gaussian base and a probe mixture with a benign bulk drift plus a far mode.Source: §B.2 · p. 24
  • With benign drift, the z ≥ 5 alarm region covers 22% of the grid for KLfwd and 63% for KLrev.At fixed ∆ = 6 and ε = 5%, forward-KL excess is swamped once δ ≳ 0.5, while reverse-KL excess retains the invented-mode contribution.Source: §B.2 · p. 24
  • The difference KLrev − KLfwd remains nearly drift-invariant for δ ≤ 1, while benign location drift pulls their ratio toward 1.The reported difference is 0.65 for δ ≤ 0.5 and 0.63 at δ = 1.Source: §B.2 · p. 24

C EXPERIMENTAL DETAILS AND FULL TABLES

The appendix reports the full aggregation grid, sensitivity checks, cross-family and program-synthesis evaluations, and illustrative outputs. Across the reported settings, TAG mitigates correctness degradation, while the examples show symmetric aggregation producing task-irrelevant outputs.

C EXPERIMENTAL DETAILS AND FULL TABLES

  • The appendix tables report raw held-out values; prose ratios divide held-out RP by the same-model, same-budget RP-only value, with held-out noise about ±0.02.Table 6 lists shared training and evaluation settings, and the main experiments use the anchor elected by the §6.4 probes.Source: C EXPERIMENTAL DETAILS AND FULL TABLES · p. 25, Table 6 · p. 25

C.1 MAIN GRID AT CAP 512: EVERY AGGREGATOR

  • Under Correct & brief, baseline aggregators collapse to 0.38–0.44× while TAG retains 0.91×/0.87×; under Correct & negative, TAG stays at 1.01× at both sizes.In the three-channel setting, additive and Rubric Dropout fall to 0.04–0.05×, while TAG retains 0.82× and 0.90× and keeps negative answers at the RP-only level.Source: §C.1 · p. 26, Table 7 · p. 27, Table 8 · p. 27
  • Accuracy stays within 0.254–0.282 across a sixteen-fold range of TAG λ values, versus 0.310 for the reference and 0.119 for additive aggregation.Larger λ raises the short-response rate and correct-and-short cell through λ = 2, after which both plateau; their ordering reverses between λ = 0.25 and λ = 0.5.Source: Table 9 · p. 27, §C.1 · p. 27, Figure 10 · p. 28

C.3 SECOND FAMILY, SCALE, AND A CONTROL

  • On the Llama Correct & negative grid, every symmetric and constrained aggregator collapses on both models, while TAG stays at the RP-only reference.On Qwen3-30B-A3B, TAG ends at 0.94× of the reference under Correct & brief and 1.00× under Correct & negative; the low-base-rate control reports additive at 0.172 versus reference 0.201 and TAG at 0.179.Source: §C.3 · pp. 28–29, Table 10 · p. 28, Table 11 · p. 29

C.4 PROGRAM SYNTHESIS

  • The program-synthesis evaluation uses 3,741 training and 185 held-out problems, and measures correctness by whether all tests pass in a sandbox.The Correct & fast reward is T < 60 ms, while Correct & no-op is T < 3 ms; evaluation re-executes programs in one session so reported latencies share a clock.Source: §C.4 · p. 29

C.5 REAL OUTPUTS ACROSS THE FOUR SETTINGS

  • For one selected held-out problem per setting, symmetric aggregation prints a constant without reading the input in both program-synthesis settings.Under Correct & fast, TAG replaces the reference’s O(N^2) loop with an O(N) prefix-sum pass.Source: §C.5 · p. 29, Figure 12 · p. 31

Related papers

Loading 2610.21404v1…