PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation

Zhongyi Li, Wan Tian, Xiang Xu, Yutian Xiao, Yikun Ban, Yijie Peng, Fuzhen Zhuang

arXiv:2609.36944v1cs.AI

TL;DR

Group-relative policy optimization depends on reward-derived advantages and sequence-level weights that can be sensitive to localized outliers. RoVR-GSPO robustifies the reward and ratio channels separately; it consistently improves over matched GSPO across mathematical reasoning and GovReport, with additional tool-call annotation gains.

  • Problem

    Group-relative methods rely on small-sample reward references and sequence-level likelihood weights, which localized reward and token-ratio perturbations can affect.

    Source: §1 · p. 1

  • Method

    RoVR-GSPO combines a robust reward reference and bounded residual credit with differentiable SoftRoVR aggregation of token log-ratios into sequence weights.

    Source: §1 · pp. 1–2

  • Results

    RoVR-GSPO consistently improves over matched GSPO across mathematical reasoning and GovReport, with additional gains on offline tool-call annotation scores.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support robust advantage and sequence-weight estimation as a practical statistical interface for group-relative language-model post-training.

    Source: §6 · p. 9

    Robust aggregation can reduce sensitivity to isolated scoring errors but does not guarantee reward-model validity, factual correctness, fairness, or safe deployment.

    Source: ETHICS STATEMENT · p. 9

Abstract

from arXiv · show

Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.

1 INTRODUCTION

Group-relative optimization depends on small-sample reward references and sequence-level likelihood weights, both vulnerable to localized perturbations. RoVR-GSPO separates these sensitivities with robust reward referencing and bounded credit, plus differentiable robust sequence weighting.

1 INTRODUCTION

  • Extreme rewards can make clean responses' pairwise advantage differences vanish after group normalization, even while standardized advantages stay bounded.The reward reference limits shared displacement; bounded residual credit protects focal leverage and the scale used to normalize clean responses.Source: §1 · p. 1
  • RoVR-GSPO uses a robust reward reference and bounded credit, while SoftRoVR supplies differentiable robust sequence weights to the clipped GSPO objective.The method makes reference, credit, and sequence-weight operations distinct objects for analysis and evaluation.Source: §1 · pp. 1–2
  • Localized token log-ratio perturbations can change sequence weights and clipping branches.The paper identifies reward perturbations and token-ratio perturbations as distinct update sensitivities.Source: §1 · p. 1
  • The paper derives conditional stability and efficiency results and connects them to downstream comparisons, channel-placement tests, and paired perturbation studies.“Variance reduction” refers specifically to the outer reference-estimation factor relative to median aggregation; SoftRoVR is a surrogate, not an exact importance ratio.Source: §1 · p. 2

2 BACKGROUND: FROM GROUP STATISTICS TO POLICY UPDATES

Mean–RMS group-relative advantages and sequence-level clipping can spread localized anomalies across a policy update. This section shows how a reward outlier erases clean-response contrast and how token-ratio excursions can alter weights and clipping.

2.1 MEAN–RMS GROUP-RELATIVE ADVANTAGES

  • As a reward outlier grows, clean responses' pairwise advantage differences approach zero, so group normalization erases their contrast.This motivates a residual transformation in addition to a bounded location estimate when using a shared scale.Source: §2.1 · p. 3
  • With raw-residual credits and a shared scale, bounded focal leverage and preserved clean-response contrast cannot both be achieved without an additional residual transformation.A bounded credit is used to make clean-signal fidelity explicit.Source: §2.1 · p. 3

2.2 SEQUENCE WEIGHTS AND CLIPPING ASYMMETRY

  • An upper-ratio excursion is clipped for positive advantage but remains on the qA branch for negative credit, making clipping asymmetric.Localized token perturbations can cross the upper boundary or magnify an already-active negative-advantage branch, affecting both sequence weight and optimization branch.Source: §2.2 · p. 3

3 DUAL-CHANNEL ROBUST ESTIMATION FOR GROUP-RELATIVE POLICY OPTIMIZATION

RoVR-GSPO separates robust reward referencing, bounded response credit, and differentiable sequence-weight estimation. Its design specifies the robustness trade-offs and block-resolution conditions that govern the guarantees.

3 DUAL-CHANNEL ROBUST ESTIMATION FOR GROUP-RELATIVE POLICY OPTIMIZATION

  • RoVR-GSPO maps prompt-level rewards to bounded normalized advantages and token log-ratios to differentiable sequence-weight surrogates before the GSPO loss.RoVR names the hard reward-reference estimator, SoftRoVR the differentiable ratio-side aggregator, and RoVR-GSPO the combined optimizer.Source: §3 · pp. 3–4

3.1 ROVR ROBUST REFERENCE ESTIMATION

  • Under the clean local normal-score model, RoVR changes the outer variance factor from π/2 to V_K, which approaches π/3 as the quantile grid is refined.Total reference-estimation variance also depends on the local sandwich variance; balanced unequal blocks affect effective sample size.Source: §3.1 · p. 5

3.2 REWARD-CHANNEL ADVANTAGE ESTIMATION

  • The bounded odd reward map preserves sign and local slope while saturating smoothly, with κ controlling the fidelity–robustness trade-off.The normalized bounded advantage is deterministically bounded and remains close to the raw residual for clean moderate rewards.Source: §3.2 · p. 5

3.3 SOFTROVR SEQUENCE-WEIGHT ESTIMATION FOR GSPO

  • SoftRoVR forms differentiable robust sequence weights from token log-ratios, and RoVR-GSPO combines these weights with bounded reward credit.Disabling either channel yields a placement ablation; the sequence-weight surrogate has a different likelihood functional from an exact importance ratio.Source: §3.3 · p. 6

3.4 RESOLUTION, FAILURE GEOMETRY, AND FALLBACK

  • Simultaneous worst-case protection against arbitrary blocks and within-block replacements requires N ≥ (2q + 1)(2s + 1).The condition sets guarantee resolution, not whether RoVR is defined; contiguous token blocks model burst contamination, while random partitions model independently assigned contamination.Source: §3.4 · p. 6

4 THEORETICAL GUARANTEES

The theoretical analysis bounds reference and credit behavior under contamination, studies outer reference efficiency under stated conditions, and relates ratio-weight error to clipping margins.

4 THEORETICAL GUARANTEES

  • If the honest-block score condition holds, the estimated reference deviation is bounded by C_B; the credit bound also caps dependence on a perturbed focal reward by 2κ.The credit guarantee includes the focal response and gives magnitude-independent control under the stated geometry and population separation.Source: §4 · p. 7
  • Under the stated local normal-score model, the outer reference-estimation factor improves on the median value π/2.Total variance also combines this factor with the local sandwich factor and, for unequal blocks, effective sample size.Source: §4 · p. 7
  • A clipping branch can change only when the true log-weight lies within e_m of a clipping boundary.The result links log-weight error to boundary distance; same-branch objective and gradient perturbations require explicit Jacobian conditions.Source: §4 · p. 7

5 EXPERIMENTS

Experiments compare RoVR-GSPO with matched GSPO across mathematical reasoning, GovReport summarization, and offline tool-call annotation, then examine channel ablations and controlled perturbations. RoVR-GSPO improves the reported downstream metrics, and the perturbation checks show robustness gains on both channels.

5.1 EXPERIMENTAL SETUP

  • The evaluation spans mathematical reasoning, long-context summarization, and offline tool-call annotation; comparisons match initialization, response and optimizer budgets, decoding, and evaluation code.Mathematical reasoning and the channel ablation use three independent runs; summarization and tool-call annotation are single-run estimates.Source: §5 · p. 7, §5.1 · pp. 7–8

5.2 DOWNSTREAM RESULTS

  • 2.08%/3.97% (Qwen3-4B-Base) and 1.27%/1.96% (OLMo-3-7B-Instruct) are five-benchmark average improvements under rule/InternLM2 rewards.RoVR-GSPO has a higher mean than matched GSPO in all 20 settings.Source: Table 1 · p. 8, §5.2 · p. 8
  • On GovReport, RoVR-GSPO improves ROUGE-1/2/L by 0.024/0.022/0.009 for Qwen3-4B-Base and 0.001/0.006/0.008 for OLMo-3-7B-Instruct; Figure 2 validation-reward trajectories are consistent with held-out ROUGE results.Source: Table 2 · p. 8, §5.2 · p. 8
  • On RLLA-4K, the highest observed mean Total reward was 2.009 for RoVR-GSPO versus 1.924 for GSPO, selected from four validations per run.Scores use 80 held-out prompts and one greedy response each; logs cover only the early portion of the configured training budget, and tools were not executed.Source: §5.2 · p. 8

5.3 CHANNEL PLACEMENT ABLATION

  • Across five benchmarks, RoVR-Adv, SoftRoVR, and RoVR-GSPO gain 1.00, 2.20, and 3.97 percentage points over GSPO; the full model leads in every benchmark.The full model exceeds SoftRoVR by 1.77 points on average.Source: Table 3 · p. 9, §5.3 · p. 9
  • The combined variant records the highest late-stage reward, while the robust variants show lower late-stage entropy (Figure 3).Source: §5.3 · p. 9
  • Offline perturbations show robust centering reduces reference displacement from 1.000σ to 0.060σ, while bounded credit limits scale inflation and SoftRoVR reduces log-weight displacement.Bounded credit limits scale inflation from 28.194× to 1.911× and raises clean-contrast retention from 0.035 to 0.520; SoftRoVR reduces log-weight displacement by about 26% for strong token spikes and 87% for 20% bursts, with fewer clipping flips under strong spikes.Source: §5.3 · p. 9

6 CONCLUSION

RoVR-GSPO separates robust reward-reference estimation, bounded credit, and sequence-weight aggregation. The paper reports improvements over matched GSPO baselines across mathematical reasoning and GovReport, with additional offline tool-call annotation gains.

6 CONCLUSION

  • Across mathematical reasoning and GovReport, RoVR-GSPO improves over matched GSPO baselines and adds gains on the offline tool-call annotation score.The conclusion describes the reward and ratio channels as addressing complementary sources of update sensitivity.Source: §6 · p. 9
  • Robust aggregation can reduce sensitivity to isolated scoring errors but does not guarantee reward validity, factual correctness, fairness, or safe deployment.The conclusion calls for independent reward audits and distribution-shift evaluation in deployment.Source: ETHICS STATEMENT · p. 9

A RELATED WORK

  • RoVR-GSPO retains the parent clipped objective while separating robust reference estimation, bounded credit, and sequence-weight aggregation.Related-work passages distinguish its reward-side bounded residual normalization from the ratio-side differentiable aggregation surrogate.Source: A RELATED WORK · p. 12

B.1 TRAINING DYNAMICS

  • GovReport Figure 2 pairs training and validation rewards for two backbones; Figure 3 tracks reward, actor entropy, and gradient norm across channel variants.Figure 3 distinguishes RoVR-Adv, SoftRoVR, full RoVR-GSPO, and GSPO.Source: Figure 2 · p. 13, Figure 3 · p. 13

B.2 LOCALIZED TOKEN-RATIO STRESS TEST

  • 26.30% and 26.09% are SoftRoVR’s reductions in log-weight displacement for positive and negative strongest single-token spikes, respectively.For a 20% burst, displacement falls by 87.44% and 87.39% in the positive and negative directions.Source: §B.2 · pp. 13–14
  • Long-burst clipping effects depend on perturbation direction and training stage, unlike the consistent reduction in aggregate displacement.The positive 20% burst flip-rate interval includes zero.Source: §B.2 · p. 14

B.3 ROBUSTNESS TO REWARD CONTAMINATION

The reward-contamination analysis separates protection of the reference from protection of the normalized advantage. Robust centering limits displacement, while bounded credit additionally limits scale inflation and preserves clean-response contrast.

B.3 ROBUSTNESS TO REWARD CONTAMINATION

  • The offline study perturbs one of 16 saved rewards while keeping responses fixed, and matches the RoVR reference across robust variants to isolate bounded-credit effects.It measures reference displacement, scale inflation, clean-response contrast retention, and clean-advantage RMS deviation.Source: §B.3 · pp. 14–15
  • 1.911× scale inflation and 0.520 clean-contrast retention are achieved with bounded credit at 16σ, versus 28.194× and 0.035 for center-only normalization.The robust reference shifts by 0.060σ, compared with 1.000σ for the arithmetic reference.Source: Figure 5 · p. 15, Table 5 · p. 15, §B.3 · p. 15
  • At a tested 4σ shift, bounded-credit normalization yields 1.881× scale inflation and 0.529 contrast retention, versus 6.918× and 0.145 for standard normalization.The passage links robust centering to shared reference displacement control and bounded credit to additional control of the perturbed response’s influence on scale and advantage.Source: §B.3 · p. 15

C EXPERIMENTAL PROTOCOL AND HYPERPARAMETERS

The appendix specifies training configurations and reporting conventions, then checks token-ratio sensitivity across negative perturbations and recorded training stages. Displacement reductions persist across stages, while clipping effects vary by direction and stage.

C EXPERIMENTAL PROTOCOL AND HYPERPARAMETERS

  • The appendix documents channel settings and notes that the token-ratio and reward-contamination studies evaluate update statistics on saved responses.Theoretical contamination guarantees are stated for B ≥ 2q + 1 and nmin ≥ 2s + 1.Source: C EXPERIMENTAL PROTOCOL AND… · p. 15, §C.2 · p. 16

C.1 TRAINING CONFIGURATION

  • The mathematical-reasoning runs use two backbones on a 7,500-example MATH split; GovReport uses the same backbones with eight summaries per input.The tool-call comparison uses matched files and scorer, but logs extend only through training step 100 and validation step 80 despite 500 requested steps.Source: §C.1 · pp. 15–16

C.2 RESOLUTION AND REPORTING CONVENTIONS

  • Comparisons match initialization, response budget, decoding, optimizer budget, and evaluation code, but GovReport reports single-run estimates and tool-call results use one run per method.Tables 1 and 3 report means and cross-run standard deviations over three independent runs; tool-call prompt order is not verified across runs.Source: §C.2 · p. 16

C.3 ADDITIONAL TOKEN-RATIO STRESS DIAGNOSTICS

  • For a negative 16σℓ spike, SoftRoVR reduces Dq by 26.09% and lowers the flip rate from 21.061% to 17.370%.For a negative 20% burst, Dq falls by 87.39%; the positive-burst flip interval includes zero.Source: §C.3 · p. 16, Figure 6 · p. 17
  • Across 35 complete steps, SoftRoVR reduces mean log-weight displacement by 85.29%–90.12% for 20% bursts, but burst flip effects vary by direction and stage.Pointwise intervals for displacement and strong-spike flips remain above zero; the steps come from one run and intervals are not adjusted for multiple comparisons.Source: §C.3 · p. 16, Figure 7 · p. 17

D IMPLEMENTATION DETAILS

The implementation appendix specifies the estimator and update procedure, numerical solver and differentiation choices, and supplementary reference-estimator experiments. The simulations validate the outer correction and show a clean-efficiency trade-off alongside lower variance under t3 data.

D.2 BARRON FAMILY AND CONVEX ROOT SOLVER

  • The appendix records the full Barron family but uses α = 1 for the main results, with safeguarded Newton or bisection solving its empirical score.Shapes α < 1 may have multiple stationary points and are not covered by the convex-root or finite-group theorem.Source: D IMPLEMENTATION DETAILS · p. 17, §D.2 · p. 18

D IMPLEMENTATION DETAILS

  • The implementation appendix specifies the numerical objects corresponding to the main-text definitions and describes the solver, tie convention, block handling, SoftRoVR path, and complexity.Main results use only α = 1.Source: D IMPLEMENTATION DETAILS · p. 17

D.1 ESTIMATOR AND UPDATE SUMMARIES

  • The update procedure checks declared failure budgets and balanced block sizes, falls back to a global M-center if checks fail, then returns bounded-credit advantages and selected sequence weights.The ratio path uses SoftRoVR when enabled and retains the canonical GSPO aggregate otherwise.Source: §D.1 · p. 18

D.3 BLOCKS, TIES, SCALES, AND DIFFERENTIATION

  • Block assignments must be independent of corruption labels for the stated guarantee; gradients stop through reward statistics but pass through the ratio-side unrolled solvers.The appendix also specifies tie conventions and notes that differentiation retains token-level intermediate states.Source: §D.3 · p. 19

E.1 DESIGN AND ESTIMANDS

  • The supplementary comparison uses shared observations and balanced partitions to assess reference-estimator variance and error against uncontaminated center zero.It compares six estimators under clean Gaussian and t3 distributions and matched contamination geometries.Source: E SUPPLEMENTARY EXPERIMENT:… · p. 19, §E.1 · p. 19

E SUPPLEMENTARY EXPERIMENT: REFERENCE-ESTIMATOR VARIANCE AND ROBUSTNESS

  • RoVR combines bounded local M-centers with tie-neutral composite-quantile aggregation and adds a policy-specific bounded-credit map to examine reference protection and clean-response contrast separately.The appendix describes its supplementary scalar reference-estimator comparisons as complementary to downstream policy comparisons.Source: E SUPPLEMENTARY EXPERIMENT:… · p. 19, §E.1 · p. 19

E.2 ORACLE OUTER-FACTOR VALIDATION

  • At K = 9, the theoretical outer factor is 1.0691 and the empirical oracle factor is 1.0747, while the empirical median factor is 1.5649.The passage says finite-sample agreement supports the composite-quantile correction’s implementation and stated outer-factor interpretation.Source: §E.2 · p. 19, Table 8 · p. 20, Figure 8 · p. 20

E.3 FINITE-SAMPLE COMPARISON

  • 0.007552 is the clean-Gaussian variance of the arithmetic mean, the most efficient comparator; under t3, RoVR variance is 0.004660 versus 0.004274 for the global M-center.Under t3, RoVR also has lower variance than the arithmetic mean and MOM in this finite configuration.Source: §E.3 · p. 21, Table 9 · p. 21

E.4 CONTAMINATION GEOMETRY AND INTERPRETATION

RoVR’s performance depends on the contamination geometry: it is competitive under dispersed point corruption and performs similarly to other block-robust estimators under one fully corrupted block. The theoretical discussion separates robustness guarantees, bounded-credit behavior, clean efficiency, and policy-scope results.

E.4 CONTAMINATION GEOMETRY AND INTERPRETATION

  • Under dispersed corruption, RoVR RMSE is 0.270 versus approximately 1.006 for mean, MOM, and VRMOM-style; under one corrupted block, RoVR is 0.117, alongside MOM at 0.117 and VRMOM-style at 0.113.Table 10 reports RMSE to the uncontaminated center under fixed +8 contamination, modifying 16 of 128 observations in each geometry.Source: Table 10 · p. 21, §E.4 · p. 21
  • Across shifts from 0 to 16, mean-like block estimators grow approximately linearly under dispersed corruption, while global and block-robust estimators saturate.For the declared one-block failure geometry, the median and composite-quantile outer aggregators can reject the bad block.Source: §E.4 · p. 21, Figure 11 · p. 22, Figure 12 · p. 22
  • The simulations reproduce the predicted outer-efficiency factor and show low RoVR error under both dispersed and coherent contamination geometries.The reported interpretation is conditional on the declared block-failure geometry.Source: §E.4 · p. 21

F DETAILED THEORETICAL STATEMENTS

  • The theoretical statements cover mixed-contamination guarantees, bounded-credit fidelity, clean transfer efficiency, and smoothing, clipping-branch, and policy-scope effects.The section also treats linear leave-one-out baselines and bias from robust credit shaping.Source: F DETAILED THEORETICAL STATEMENTS · pp. 21, 23

F.1 MIXED CONTAMINATION AT FIXED GROUP SIZE

These results distinguish finite-group robustness to mixed contamination from target and credit effects, clean-sample efficiency, and smooth policy-update behavior. Their guarantees depend on stated contamination timing, separation, and local regularity conditions.

F.1 MIXED CONTAMINATION AT FIXED GROUP SIZE

  • For fixed corrupted indices and a balanced partition sampled independently of them, Proposition F.2 bounds each block’s corruption load; the resulting guarantees exclude corruption selected after observing the partition.The later mixed-contamination results also require declared timing and population-separation conditions.Source: §F.1 · pp. 23–24

F.2 TARGET, BOUNDED RESPONSE ADVANTAGE, AND RETAINED CONTRAST

  • Robust location can shift the population target under asymmetry, while bounded credit changes reward utility; smaller κ limits focal leverage, whereas larger κ better preserves moderate clean residuals.The credit bound includes the perturbed focal response and caps its dependence on reward magnitude under the stated geometry and population separation.Source: §F.2 · pp. 24–25

F.3 CLEAN EFFICIENCY AND SCALE ORTHOGONALITY

  • Clean outer-efficiency transfer is conditional on local normal-score and equicontinuity conditions, which are not implied by a pointwise CLT, especially for tie-rich rule rewards.Scale orthogonality removes the first-order scale term under the local normal-score model.Source: §F.3 · pp. 26–27

F.4 SMOOTHING, CLIPPING BRANCHES, AND POLICY SCOPE

  • Smooth policy-stability bounds require matching clipping branches and explicit Jacobian conditions; no global policy-improvement claim follows at clipping kinks.The robust token surrogate is not an exact likelihood ratio, and bounded advantages are not an unbiased raw-reward gradient.Source: §F.4 · p. 28

G PROOFS FOR FAILURE PROPAGATION AND FINITE-GROUP ROBUSTNESS

  • The appendix’s fixed-group proof evaluates bounded empirical scores at deterministic thresholds, avoiding a curvature event along an unknown contaminated root path.It also establishes translation, constant-preservation, and tie-neutral properties of RoVR.Source: G PROOFS FOR FAILURE PROPAGATION AND… · p. 29

G.1 SINGLE-REWARD PROPAGATION AND THE CENTER-ONLY IMPOSSIBILITY

  • A center-only repair cannot prevent a single extreme reward from collapsing clean-response contrast when the raw residual and RMS remain in the normalization.The proof states that bounding focal credit requires a scale that grows with the extreme reward, which sends fixed clean residuals toward zero after division.Source: §G.1 · p. 29

G.2 INVARIANCE AND DETERMINISTIC ROOT SENSITIVITY

The proofs establish translation calibration and root sensitivity, then derive block-load and estimator-tail bounds under randomized contamination assumptions. They keep target, contamination, sampling, and credit-shaping effects distinct.

G.2 INVARIANCE AND DETERMINISTIC ROOT SENSITIVITY

  • Shifting every observation by a common amount shifts RoVR by exactly that amount, while the constant-input case makes the composite correction zero.The proof also reduces the B = 1 case to the local M-center.Source: §G.2 · p. 29

G.3 RANDOMIZED LOAD AND THE FIXED-THRESHOLD BLOCK TAIL

  • The randomized-load proof uses a hypergeometric block count and threshold-tail bounds to control the median block center under fixed, independently randomized contamination.The argument allows replacement values to adapt to clean observations, but requires the relevant block labels and partition to be independent of those values.Source: §G.3 · p. 30

H PROOFS FOR TARGET, CREDIT, AND CLEAN EFFICIENCY

  • The accompanying proofs keep functional bias, contamination bias, finite-sample error, and credit shaping separate rather than combining them into one error source.The appendix covers target, bounded-credit, unequal-block oracle variance, scale-orthogonality, and clean-transfer results.Source: H PROOFS FOR TARGET, CREDIT, AND CLEAN… · p. 31

H.1 ASYMMETRY AND POPULATION CONTAMINATION

The proofs characterize how asymmetry and contamination affect the robust target, how bounded credit controls response influence while retaining moderate contrast, and how clean efficiency and policy stability depend on explicit conditions. They also quantify smoothing and credit-distortion effects.

H.1 ASYMMETRY AND POPULATION CONTAMINATION

  • Under asymmetry, the robust location target can differ from the mean; contamination displacement is bounded by comparing the bounded contamination score with clean score separation.The proof also bounds the mean-to-target difference using curvature and, under a third-moment condition, a cubic-moment term.Source: §H.1 · p. 31

H.2 BOUNDED-ADVANTAGE STABILITY AND RETAINED CONTRAST

  • Bounded credit limits the influence of replaced rewards, while moderate clean residuals retain contrast through a positive lower bound on the credit map’s slope.The proof uses the 1-Lipschitz, bounded credit map and bounds changes in the scale and credits.Source: §H.2 · p. 31

H.3 UNEQUAL-BLOCK ORACLE VARIANCE AND SCALE ORTHOGONALITY

  • For independent Gaussian block centers, the oracle composite score has mean zero, and symmetry of the quantile grid cancels the first-order scale term.The proof derives the oracle variance using independence across blocks.Source: §H.3 · p. 32

H.4 CLEAN EFFICIENCY TRANSFER

  • Under the transfer conditions, the clean estimator’s asymptotic variance combines the honest-block fluctuation with the outer factor, whose sequential limit is π/3.The proof controls the arbitrary-block contribution as negligible under its stated conditions.Source: §H.4 · pp. 32–33

I PROOFS FOR SMOOTHING AND POLICY SCOPE

  • The smoothing and policy-scope proofs do not make the robust token surrogate an exact likelihood ratio or bounded advantage an unbiased raw-reward gradient.These are explicit boundaries on what the results establish.Source: I PROOFS FOR SMOOTHING AND POLICY SCOPE · p. 33

I.1 SMOOTHING AND TRANSLATION CALIBRATION

  • The logistic smoothing error is zero at ties and has an O(γ) expected bound under a bounded continuous margin density.The proof integrates the logistic-step discrepancy; atomic mass at zero contributes no discrepancy.Source: §I.1 · p. 33

I.2 BRANCH COUNTS AND LOCAL POLICY STABILITY

  • Policy-stability bounds compare objectives and gradients only when both inputs remain on the same differentiable clipping branch and the stated Jacobian conditions hold.The proof relates log-aggregate error to objective and gradient error under those conditions.Source: §I.2 · p. 33

I.3 LINEAR LEAVE-ONE-OUT DIRECTION AND BOUNDED-ADVANTAGE DISTORTION

  • With leave-one-out baselines and the stated sampling conditions, the linear score direction recovers the expected-reward gradient; bounded-credit distortion is separately controlled by residual magnitude and score norms.The proof bounds the credit-versus-linear difference using the cubic residual term and a lower scale bound.Source: §I.3 · p. 34
Loading 2609.36944v1…