STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

1 INTRODUCTION

• An unreliable reward can shift the group mean and scale before post-hoc weighting, while purely relative weights cannot express common low confidence.STAR therefore places reliability inside normalization and retains its absolute magnitude in the update.Source: §1 · pp. 1–2
• STAR-GRPO uses paired assessments to separate the quality score being optimized from the reliability that governs how strongly each rollout influences learning.The quality path selects or combines training and anchor scores; score disagreement determines reliability.Source: §1 · pp. 1–2
• The paper applies the reliability-first mechanism to token-interface exploitation and rubric-proxy overoptimization, which use different score-pair semantics.For rubric-proxy medical reasoning, STAR retains the proxy as the task reward while a rubric-free semantic anchor determines rollout influence.Source: §1 · p. 2
• Reliability enters the robust group fit before normalization, and an absolute-reliability factor preserves group-wide confidence in the bounded update.The analysis establishes coordinate and second-moment bounds, exact-centering conditions, and reliability-dependent attenuation guarantees.Source: §1 · p. 2
• STAR improves canonical quality while limiting token-interface reward exploitation; in medical reasoning, the independent-judge score rises from 0.2706 to 0.3174 as the proxy–judge gap and overclaim fraction decrease.The proxy–judge gap decreases from 0.2935 to 0.2318, and the overclaim fraction decreases from 0.2464 to 0.2204.Source: §1 · p. 2

2 RELATED WORK

• TOMPA studies token-interface attacks and representation engineering supplies internal shortcut signals for modifying GRPO advantages; STAR derives reliability from paired reward assessments [Zhang et al., 2026; Wu and Tang, 2026].Reward overoptimization and reward-model limitations motivate evaluating progress beyond the training score [Casper et al., 2023].Source: §2 · p. 3
• Reward-model ensembles and uncertainty penalties support conservative optimization, while robust reward training addresses preference artifacts and answer-level reliability weighting informs preference optimization [Coste et al., 2024; Zhai et al., 2024; Liu et al., 2024; Chen et al., 2026].STAR differs by placing reliability inside the group fit and retaining absolute attenuation afterward.Source: §2 · p. 3
• Dr. GRPO and DAPO analyze normalization and loss-reduction choices, while known LLM-judge biases motivate using a third judge never used in policy updates [Liu et al., 2025; Yu et al., 2025; Zheng et al., 2023].The independent judge measures whether proxy optimization transfers to semantic quality.Source: §2 · p. 3

3 PROBLEM SETUP AND THREAT MODEL

• Both evaluation settings provide a deployed or proxy score and an anchor score for the same rollout, with token-interface scoring using two representations and RQ2 using a rubric proxy and semantic anchor.The RQ2 score reference and statistical scope are specified separately.Source: §3 · p. 3
• The token-interface canonicalizer decodes with the policy tokenizer, applies prespecified normalization and chat-template rules, and retokenizes with the reward tokenizer.Positive deployed-minus-canonical discrepancy measures reward unsupported by this anchor.Source: §3 · p. 3
• The quality path sets how much deployed reward enters optimization, while contexts, discrepancy fitting, calibration, and final evaluation use prespecified or disjoint data rules.The default uses the deployed score, while κ = 0 selects the canonical reward path.Source: §3 · p. 3
• Paired views provide evidence about differential shifts, but do not establish that every observed discrepancy is an attack.An attack adding the same random shift to every evaluated view leaves pairwise differences unchanged, so discrepancy-only tests cannot distinguish it.Source: §3 · p. 4

4 RELIABILITY-FIRST GROUP-RELATIVE ADVANTAGES

• STAR specifies the score pair and quality path before estimating reliability, fitting a robust group baseline, and forming policy advantages.The formal-analysis procedure and experiment-specific solver and loss choices are documented separately.Source: §4 · p. 4

4.1 TRUSTED DISCREPANCY CALIBRATION

• Trusted discrepancies are used to fit context-specific locations and scales, followed by disjoint-split calibration of a reliability coefficient.The threshold sets when downweighting begins and λ controls its rate; the coefficient represents optimization influence, not a posterior probability that a response is clean. A fixed RQ2 reference has no split-calibration coverage claim.Source: §4.1 · p. 4

4.2 RELIABILITY-FIRST ROBUST ADVANTAGES

• Groups must meet total-reliability and effective-size thresholds to be admitted; if none qualify, the optimizer step is skipped.For non-admissible groups, reward advantages are zero and they do not enter the shared-scale fit.Source: §4.2 · p. 4
• The robust fit uses prompt-specific locations and a minibatch-shared scale, pooling normalization information while retaining prompt-specific centers.STAR explicitly reintroduces absolute group reliability into the final advantage.Source: §4.2 · p. 5
• Relative reliability reduces a suspicious rollout’s influence on the fitted baseline, while absolute reliability controls the strength of the group update.With εw = 0 and a fixed admitted set, multiplying all group weights by c in (0, 1] multiplies its advantages by c.Source: §4.2 · p. 5

5 THEORETICAL GUARANTEES

• Update bounds concern only the initial reward-side direction at the old policy, holding reward-side quantities fixed; statistical reliability also requires assumptions about paired assessments and reference data.For the experimental token-mean reduction, the corresponding length-weighted control links realized optimization reduction to rollout reliabilities.Source: §5 · pp. 5–6
• The fitting objective is jointly convex and has a unique minimizer when every admitted prompt has two distinct positive-weight rewards; direction bounds do not require an exact fit.Source: §5 · p. 6
• STAR converts rollout reliability ε into O(ε) coordinate bounds for both the suspicious rollout and, under exact centering, the induced clean-rollout perturbation.The exact minimizer result additionally assumes W_min < n, G_min < n, and a scale in a fixed positive interval.Source: §5 · p. 6
• For attacked scores separated from the clean-score threshold by Δ with probability at least 1−β, expected attacked-rollout reliability is at most β + (1−β)e^(−λΔ).This is the conditional reliability bound stated for the attacked group.Source: §5 · p. 6
• Under the stated conditional total-variation premise, adaptive-round clean downweighting probability is at most α + Eρ_t, extending finite-sample marginal split-calibration control.The first claim is identified as the standard finite-sample marginal split-calibration guarantee.Source: §5 · p. 6

6.1 RQ1: MITIGATING TOKEN-INTERFACE REWARD HACKING

• RQ1 compares TOMPA-GRPO's deployed-interface reward optimization with STAR's canonical-primary quality path, using deployed-versus-canonical score discrepancy to regulate reliability.The experiment uses Llama-3.2-1B-Instruct, Skywork-Reward-V2-Qwen3-8B, 10,000 WildChat training prompts, and 100 curated NoveltyBench validation prompts.Source: §6.1 · p. 7
• Canonical validation reward rises from 3.121 to 4.902 (57.1% relative) with STAR, while TOMPA-GRPO's observed reward rises from −1.894 to 9.641 and its responses reach the 2,048-token cap by step 235.Figure 2 contrasts the validation trajectories; STAR's mean rollout reliability stays between 0.9847 and 1.0000, while minimum individual reliability reaches 0.105.Source: §6.1 · pp. 7–8, Figure 2 · p. 7

6.2 RQ2: MITIGATING RUBRIC-PROXY OVEROPTIMIZATION

• RQ2 trains on RubricHub-Medical and evaluates on HealthBench-Hard, keeping the GPT-4o-mini rubric proxy as the task reward while using a rubric-free semantic anchor to estimate reliability.Claude-Sonnet-4-6 serves only as an independent validation judge and never enters the policy update.Source: §6.2 · p. 8
• The independent-judge score rises from 0.2706 to 0.3174, while the proxy–judge gap falls from 0.2935 to 0.2318 and overclaim fraction falls from 0.2464 to 0.2204.Criterion pass rate increases from 0.4752 to 0.5049 even as STAR's proxy score decreases from 0.5640 to 0.5492.Source: §6.2 · pp. 8–9, Figure 4 · p. 8
• RQ2 reliability remains active: rollout reliability ranges approximately 0.70–0.90, effective group size 11.3–15.2 out of 16, and non-admissible groups reach approximately 10.9%.Figure 5 tracks reliability, effective group size, and the training proxy–anchor gap; the training anchor differs from the independent evaluation judge.Source: §6.2 · p. 9, Figure 5 · p. 9

6 EXPERIMENTS

• Across both regimes, STAR limits poorly supported reward influence, improving canonical quality in token-interface exploitation and independent-judge quality while reducing proxy disagreement and overclaim in medical reasoning.In RQ2, STAR retains the proxy as the task reward and uses an independent semantic view to modulate its influence.Source: §6.2 · p. 9

7 CONCLUSION

• Paired assessments estimate rollout reliability before normalization; reliability shapes the robust group fit, and an absolute group-reliability factor attenuates bounded policy advantages.The construction provides coordinate and second-moment control, an auditable condition for exact centering, and reliability-dependent attenuation.Source: §7 · p. 9
• Across two reward-hacking settings, STAR limits poorly supported score influence, improving canonical reward in token-interface exploitation and independent-judge quality while reducing proxy disagreement and overclaim in medical reasoning.In RQ2, the proxy remains the task reward while an auxiliary semantic assessment regulates its influence.Source: §7 · pp. 9–10
• Deployment calls for independent semantic evaluation, appropriate human oversight, and protection of sensitive logged outputs and model interfaces.The paper also calls for safeguards proportionate to the systems when releasing attack descriptions and evaluation artifacts.Source: §7 · p. 10
• Equations (2)–(9), proofs, numerical protocols, paired-score interfaces, calibration parameters, optimizer settings, and realized configurations are provided for reproducibility.The paper states that these materials specify the advantage construction and reported experiments.Source: §7 · p. 10

A EXTENDED RELATED WORK

• STAR's score-discrepancy weighting acts on group statistics before normalization and preserves absolute reliability in final advantages, an intervention the paper distinguishes from related reliability-aware approaches.The paper positions this contribution as orthogonal to prior approaches that intervene at other points.Source: A EXTENDED RELATED WORK · p. 13
• The reward-side construction retains specified PPO likelihood-ratio and clipping mechanics, but its initial-direction bounds do not guarantee the full optimization trajectory.The paper also distinguishes equal-response from token-mean loss reductions.Source: A EXTENDED RELATED WORK · p. 13
• RQ2's prompts and rubrics do not make its reported scores evaluations under the original benchmarks' complete protocols, and RQ1 reports reward and length dynamics rather than NoveltyBench diversity.The paper identifies judge biases as a reason to treat response length as a diagnostic and uses a third judge for independent measurement.Source: A EXTENDED RELATED WORK · p. 13
• The adaptive-round statement relies on an explicitly stated conditional total-variation premise; neither that premise nor clean exchangeability is automatic in on-policy training.The paper distinguishes this premise from split-conformal guarantees under exchangeability.Source: A EXTENDED RELATED WORK · p. 13

B.1 PROOF MAP AND FORMAL SETUP

• Conformal validity requires exchangeability after the score map is fitted, whereas the self-tuned scale interpretation uses finite variance.The variance condition is not stated as necessary for conformal validity.Source: §B.1 · p. 15
• Mean separation alone does not guarantee detection power without variance or tail control.The scale lower bound also depends on truncated variance, so uniform scale comparability requires regularity that preserves a constant fraction of total variance.Source: §B.1 · p. 15, §C.1.1 · p. 18

B.2 IDENTIFIABILITY BOUNDARIES

• No detector based only on the observed reward can uniformly distinguish clean examples from attacks that lower the canonical reward while preserving the observed-reward law.Scalar transformations of that same observation remain non-identifying.Source: §B.2 · p. 16
• If an attack adds the same random shift to every evaluated view, pairwise differences remain unchanged and discrepancy-only tests have type-I plus type-II error of at least one.Under identical clean and attacked laws of the difference-based score, the lower bound is one.Source: §B.2 · p. 16

B.1 PROOF MAP AND FORMAL SETUP

• Trusted paired examples define context-specific discrepancy fits and a conformal threshold, while training rollouts and final test examples are disjoint from both calibration splits.The proof map distinguishes observable-score identification from bounds conditional on realized reward and weight arrays.Source: §B.1 · p. 14, C CALIBRATION AND RELIABILITY GUARANTEES · p. 16

C.1.1 ROBUST CONTEXT FIT

• The fitted location score has bounded magnitude.The supplied result states the bound without specifying its value here.Source: §C.1.1 · p. 17
• The empirical robust context-fit objective is jointly convex, and its interior minimizer is unique when positively weighted data contain at least two distinct values.The uniqueness statement is conditional on an interior minimizer.Source: §C.1.1 · p. 17
• For a nondegenerate mean-zero variable with finite variance, the self-tuned population scale approaches the ordinary standard deviation in the large-sample regime.The stated result assumes z^2 < 2n.Source: §C.1.1 · p. 18

C.1.2 FINITE-SAMPLE CALIBRATION AND TRANSFER

• Under the stated one-context deviation premises, a union bound gives simultaneous fitting guarantees with probability at least 1 − δD across retained contexts.Rare contexts handled by the frozen fallback hierarchy are excluded; the result lifts one-context guarantees to the finite retained collection.Source: §C.1.2 · pp. 18–19

C.2 CONFORMAL RELIABILITY, SEPARATION, AND DRIFT

• Conditional on the fitting split, exchangeable clean calibration and test pairs give finite-sample rank validity, while attack power requires additional tail separation.Empirical-quantile localization uses a DKW event; the passage explicitly distinguishes power from validity and does not imply exact group-conditional conformal validity.Source: §C.1.2 · p. 19, §C.2 · pp. 21–22
• For adaptive policy rounds, calibration guarantees transfer under a conditional total-variation stability premise, with drift ρt adding to the nominal level α.For random drift, the bound uses α + Eρt; if drift is uncontrolled, calibration diagnostics rather than nominal conformal levels are needed.Source: §C.1.2 · p. 20, §C.2 · p. 23

C.1.3 SUPPORTING CALCULUS AND BOUNDARY CONDITIONS

• The robust scale equation has a unique solution under the stated nondegeneracy condition, and its self-tuned large-sample interpretation converges to σ.The supporting calculus also shows a positive-semidefinite one-observation Hessian and a positive-definite sum for two distinct residuals.Source: §C.1.3 · pp. 20–21

C.2 CONFORMAL RELIABILITY, SEPARATION, AND DRIFT

• The soft reliability weight equals one below its threshold, is nonincreasing and λ-Lipschitz, and attenuates sufficiently separated attack scores.The stated clean-rollout guarantee follows from rank validity; quantitative attack power requires a tail condition, not merely a mean shift.Source: §C.2 · p. 22
• For groups where every rollout is suspicious, STAR’s absolute mean-reliability factor and primary abstention rule prevent relative weights from canceling the shared suspicion.The discrepancy baseline is external to the current group, so a common positive shift is not redefined as normal.Source: §C.2 · p. 22

D RELIABILITY-FIRST ADVANTAGES AND OPTIMIZATION

• The weighted robust objective yields zero-sum bounded advantages and a controlled initial reward-side update direction at the old policy.Reliability enters the location–scale fit before normalization, while an absolute group-reliability factor carries reliability into the update.Source: D RELIABILITY-FIRST ADVANTAGES AND… · p. 23, §D.1 · p. 23

D.1.1 WEIGHTED LOCATION–SCALE FIT

• Because GRPO uses few rollouts, STAR fits one robust location per prompt and one shared minibatch scale from weighted residuals rather than separate location–scale pairs.The shared scale is an update normalization, not a claim that prompts have identical population variance; the unconstrained location minimizer satisfies the weighted location equation.Source: §D.1.1 · pp. 23–24
• The weighted objective is jointly convex, and its constrained minimizer is unique when each admitted prompt has two distinct positive-weight quality rewards.The deterministic convexity and advantage results condition on observed rewards and weights; identifying a fitted location with a population parameter requires an additional estimating-equation or cross-fitting condition, and STAR does not identify it with human utility.Source: §D.1.1 · p. 24

D.1.2 OBJECTIVE GEOMETRY AND NUMERICAL CONDITIONS

• Strict convexity requires distinct residuals within each prompt to identify its location and aggregate variation to identify the shared scale.For fixed scale, location minimization is globally safe by bisection; alternating exact minimization converges to a global minimizer under the stated compact-level-set conditions.Source: §D.1.2 · p. 25

D.1.3 STAR ADVANTAGE AND DETERMINISTIC BOUNDS

• For every admissible rollout, STAR’s self-tuned score satisfies |φ_b,i| ≤ 1.This bound supports distribution-free magnitude control.Source: §D.1.3 · p. 25
• At an exact fit, the unconstrained location coordinate satisfies its first-order equation even when the shared scale is at a boundary.Relative weights enter the fitted location and scale, while absolute group reliability attenuates the policy update and avoids common-scale cancellation.Source: §D.1.3 · p. 25

1. Conditional zero sum: if P

• When all rollout reliabilities equal one, STAR reduces to the self-tuned anchored-quality score divided by 1 + εw.Source: §1 · p. 26
• If each rollout reliability is at most e^−λ∆, the individual STAR advantage is also bounded by e^−λ∆.Source: §1 · p. 26
• Commonly scaling one group’s reliability weights by c leaves relative weights, effective size, fitted parameters, and φ unchanged, while STAR advantages scale by c when εw = 0.With a positive numerical floor, every coordinate is multiplied by a factor no greater than c.Source: §1 · p. 26
• For ordinary GRPO, a single outlier among G rewards leaves each clean advantage at −1/√(G − 1), regardless of the outlier magnitude.Post-hoc downweighting the outlier does not undo the clean-rollout signs assigned by the contaminated normalization.Source: §1 · pp. 26–27
• With the location equation satisfied, all clean coordinates are equal and zero sum bounds each clean STAR advantage by ε/n.The clean-coordinate bound holds for every admitted group, including when its scale is shared with other groups.Source: §1 · p. 27

• The finite-ϵ bound holds for every admitted group, including when its scale is shared across groups.The passage states the bound as clean | ≤ ϵ/n.Source: G . Hence |ASTAR · p. 27
• At the limit, the exact location equation gives µ⋆ = 0, and compactness yields bµϵ → 0 for the full sequence.The fitted location lies in [0, M], with scale in a fixed compact interval.Source: G . Hence |ASTAR · p. 27
• The displayed advantage bounds imply that all coordinates vanish; the comparison with a fixed stabilizer is algebraic, not an empirical ablation.With fixed δ > 0, adding δ to σw gives O(ϵ) coordinates for fixed M, with constants that can depend on M/δ; STAR’s |Ai| ≤ wi bound is uniform over finite rewards and positive scales.Source: G . Hence |ASTAR · p. 27

D.1.4 ALGEBRAIC CONSEQUENCES AND DESIGN VARIANTS

• A common within-prompt εw preserves zero sum when the weighted location equation holds, while bounded nonlinear h breaks exact affine equivariance to control optimizer gradients.With identity h, positive affine transformations of anchored quality scores transform locations and shared scale correspondingly while leaving normalized scores and reliability factors invariant.Source: §D.1.4 · p. 27
• Because νb ≥ w̄b, STAR’s RMS denominator yields no larger coordinate magnitudes and a sharper energy bound than the mean-denominator variant.With one unit weight and all others zero, the mean-denominator factor is one when εw = 0; STAR’s factor is 1/G.Source: §D.1.4 · p. 27

D.2 INITIAL POLICY DIRECTION AND ITS SCOPE

• The stated advantage bounds apply to the initial reward-side direction at the old policy, not the full clipped policy update.The clipped objective is introduced to identify that direction within the training update.Source: §D.2 · p. 27

D.2.1 POLICY OBJECTIVE AND OLD-POLICY GUARANTEE

• At the old policy, the individual response contribution is bounded by LπLiwi/T under the token-mean reduction.The token-mean setup includes skipped responses in T and assigns their reward advantages zero.Source: §1 · p. 28

1 GASTAR

• The direction bound concerns the specified aggregation batch and does not bound later PPO epochs, optimizer state, parameter displacement, or the KL term.At later epochs, bounded advantages alone do not bound the surrogate or parameter gradient without an explicit ratio and policy-score condition.Source: §1 · p. 29

E.1 ALGORITHMS AND NUMERICAL SAFEGUARDS

• The algorithms specify canonical rendering, paired-score reliability, group admission, robust fitting, and the policy loss; solver failure skips the policy step.The training protocol retains skipped entries and uses a global-minibatch shared scale.Source: §E.1 · pp. 29–30

E.2.1 GUARANTEE SCOPE AND DESIGN CONDITIONS

• STAR’s guarantees use paired evidence, calibrated reliability, robust group statistics, and explicit abstention for groups with insufficient reliable mass.Under clean exchangeability, split conformal calibration controls marginal clean downweighting at level α; the adaptive-round statement requires its conditional total-variation premise.Source: §E.2.1 · p. 31
• Paired scoring requires two reward assessments per rollout, while the robust fitting pass is O(BG) with parallel prompt locations and a one-dimensional shared scale.RQ1 uses split-calibrated token-interface discrepancies; RQ2 uses a fixed cross-evaluator discrepancy reference.Source: §E.2.1 · pp. 31–32

E.2.2 REPRODUCIBILITY RECORD

• Reproducibility requires recording tokenizer and interface details, frozen context construction, numerical parameters, loss aggregation, and solver diagnostics.The record calls for batch-level residuals, iterations, boundary indicators, skipped-group counts, advantage statistics, and update norms.Source: §E.2.2 · p. 32
• Calibration, policy training, validation, audit, and final evaluation use six immutable, disjoint identifier pools; reusing evaluation examples in calibration invalidates the intended protocol.Source: §E.2.2 · p. 32

F REALIZED EXPERIMENTAL CONFIGURATIONS

• At step 855, Table 2 reports that STAR improves the canonical score while suppressing the runaway deployed-interface reward exploited by TOMPA-GRPO.The table caption states that scores are raw reward-model outputs and length is in policy tokens.Source: Table 2 · p. 33

F.1 RQ1: REALIZED TOKEN-SPACE CONFIGURATION

• RQ1 compares directly mapped policy token identifiers with a canonical view that decodes, normalizes, and retokenizes the same sampled response.STAR’s fit uses the bounded canonical quality input 10 tanh(Rcanon/10).Source: §F.1 · p. 33
• Over STAR’s full recorded training horizon, canonical reward rises from −4.154 to 6.850, while observed reward changes from −1.439 to −0.663 and mean discrepancy changes from 2.715 to −7.513.Source: §F.1 · p. 34

F.2 RQ2: REALIZED RUBRIC-PROXY CONFIGURATION

• Table 7 reports that STAR improves all external-quality and reward-hacking diagnostics while optimizing the same scalar proxy reward as GRPO.Its Δ column is STAR-GRPO minus GRPO at the latest common eligible checkpoint.Source: Table 7 · p. 35

F.1.1 CANONICAL-GRPO DIAGNOSTIC CONTROL

• The canonical-only diagnostic control’s higher raw canonical reward and shorter outputs do not establish a causal effect of removing reliability or superior human-perceived quality.Each method has one run, and the reported variation does not estimate uncertainty across training seeds.Source: §F.1.1 · p. 35

F.2 RQ2: REALIZED RUBRIC-PROXY CONFIGURATION

• RQ2 keeps the rubric-conditioned proxy as the training reward and uses a rubric-free semantic assessment as the anchor for reliability.Its training proxy–anchor gap drives STAR reliability, whereas the validation proxy–judge gap is an external diagnostic.Source: §F.2 · pp. 35, 37
• In RQ2, invalid proxy or anchor calls receive zero reliability; STAR abstains from the optimizer step if reliable mass is insufficient for every group.Source: §F.2 · p. 37