PaperlayerView exact revision on arXiv

Source-linked AI summary

ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control

Eric Frankel, Banghua Zhu, Sewoong Oh, Lillian J. Ratliff

arXiv:2609.34374v1cs.LGcs.AIstat.ML

TL;DR

Human-label scarcity makes LLM post-training costly, while pseudo labels provide more signal but can carry bias. ABC-Align adaptively trades bias for variance reduction and reports better performance than prior semi-supervised baselines in the studied scarce-feedback RLHF, DPO, and GRPO settings.

  • Problem

    Human labels are expensive and slow to collect, while pseudo-label methods can introduce systematic bias and general-purpose correction methods can have high variance when human annotations are scarce.

    Source: §1 · pp. 1–2

  • Method

    ABC-Align uses abundant pseudo-label signal with a controlled-bias gradient estimator and adaptively tunes the bias–variance trade-off using human-labeled data.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 7

  • Results

    ABC-Align improves over prior semi-supervised methods in the studied scarce-feedback settings across reward-model training, DPO, and GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When teacher-gradient bias is small, ABC-Align can move closer to pseudo-label-dominated Θ(1/N) variance scaling, at the cost of an explicit bias term.

    Source: §1 · p. 2

    The formulation focuses on scarce-label settings with n ≪ N, rather than establishing results beyond that regime.

    Source: §2 · p. 3

Abstract

from arXiv · show

Language model post-training is often bottlenecked by the need for human-collected preference data, which is expensive and difficult to scale. Reinforcement learning from AI feedback (RLAIF) style approaches that leverage pseudo labels offer an abundant alternative but introduce systematic biases that degrade downstream alignment. Recent general-purpose semi-supervised methods correct for teacher bias using a small set of human-labeled examples, but suffer from high variance especially when human annotations are scarce. To this end, we propose ABC-Align, leveraging abundant pseudo label signal to minimize variance and applying a lightweight, adaptive correction grounded in the human-labeled subset. The correction strength is tuned automatically during training using plug-in estimates of the relevant bias--variance quantities. On LLM alignment with RLHF, DPO, and GRPO where human feedback is scarce, we empirically demonstrate that ABC-Align achieves superior performance over prior semi-supervised baselines in a series of experiments on an increasing scale. Our code is available at https://github.com/SewoongLab/abc-align .

1 Introduction

Human labels are costly and difficult to scale, while pseudo-label approaches can introduce systematic bias. ABC-Align addresses this tension with adaptive bias control and reports improvements over prior semi-supervised methods in the studied LLM alignment settings.

Motivation

  • Human labels support preference-based methods such as RLHF and DPO as well as outcome-based GRPO, but annotation is expensive, slow, and subject to inter-annotator disagreement.Pseudo labels offer a potential low-cost increase in training signal, but LLM judges can favor verbosity or sycophancy, and self-play can plateau or degrade.Source: §1 · p. 1

Prior work

  • Prior semi-supervised methods, including prediction-powered inference and its training extensions, use trusted labels to correct pseudo-label bias while targeting unbiased, lower-variance estimates.The paper positions ABC-Align against this family of approaches.Source: §1 · p. 2

Motivation and approach

  • ABC-Align permits controlled gradient bias in exchange for lower variance, potentially approaching Θ(1/N) scaling when teacher-gradient bias is small.For a perfect teacher, it removes the labeled-data variance contribution; otherwise, it yields a controlled bias–variance trade-off.Source: §1 · p. 2
  • Unbiased prediction-powered gradients have a Θ(1/n) variance floor except in a special perfect-correlation regime, so more pseudo labels cannot reduce that floor.Here n is the number of human-labeled examples.Source: §1 · p. 2

Contributions

  • ABC-SSL uses two parameters to exchange bias and variance, with an adaptive controller that tunes them online from mini-batch statistics.The paper characterizes the oracle bias–variance trade-off and analyzes convergence for smooth non-convex objectives.Source: §1 · p. 2

Empirical validation

  • ABC-Align improves over prior semi-supervised methods in experiments spanning reward-model training, DPO, and GRPO, including full-parameter Llama-3.2-3B fine-tuning.The largest experiments use DPO on UltraFeedback and GRPO on GSM8K-style data.Source: §1 · p. 2

2 Background and Problem Formulation

The formulation considers scarce labeled data alongside abundant unlabeled inputs that can receive pseudo labels. It contrasts biased semi-supervised learning with prediction-powered methods and their training-time extensions, which preserve unbiasedness.

Setup

  • The setup assumes i.i.d. labeled examples and unlabeled inputs, with a pseudo-labeling function applicable to both; it focuses on n ≪ N.In alignment, labels may be human preferences and pseudo labels may be preference orderings from an LLM.Source: §2 · p. 3

Semi-Supervised Learning

  • Supervised learning ignores unlabeled data, while classical semi-supervised learning combines labeled and pseudo-labeled losses but is biased unless pseudo labels are perfect.Inaccurate pseudo labels can degrade the learned parameters, motivating methods that use unlabeled data without introducing their bias.Source: §2 · p. 3

Prediction-Powered Inference

  • Prediction-powered inference uses a zero-mean correction estimated on labeled examples, allowing pseudo-label information to be added without changing the loss expectation.Its parameterized family includes supervised learning at λ = 0, vanilla PPI at λ = 1, and the doubly robust loss at λ = N/(N + n).Source: §2 · p. 3

Prediction-Powered Training

  • For iterative model training, PP-SSL applies prediction-powered corrections to stochastic gradients and tunes λ online while keeping the gradient estimator unbiased.The formulation motivates ABC-Align’s relaxation of that unbiasedness constraint to seek a different bias–variance trade-off.Source: §2 · pp. 3–4

3 ABC-SSL: Adaptive Bias Control for Semi-Supervised Learning

ABC-SSL uses a two-parameter gradient family to trade controlled bias for variance reduction, then selects parameters with an online controller or plug-in estimates. Its analysis characterizes oracle choices and gives convergence and end-to-end guarantees, including a strict improvement over statically tuned PP-SSL under stated conditions.

3 ABC-SSL: Adaptive Bias Control for Semi-Supervised Learning

  • ABC-SSL relaxes the unbiasedness constraint with a two-parameter gradient family containing PP-SSL and the doubly robust gradient as special cases.Its online controller learns both parameters from samples and achieves regret against the best fixed pair.Source: §3 · p. 4

3.1 An Adaptive Bias Control (ABC) Gradient Estimator

  • The parameter a controls conditional bias, while b scales a zero-mean correction and contributes only to variance.Figure 1 shows the selected configuration approaching pseudo-only when the teacher is accurate and an interior bias–variance trade-off when it is biased.Source: §3.1 · p. 5, Figure 1 · p. 5

3.2 Oracle-optimal parameters

  • The oracle parameters adapt to teacher quality: they favor pseudo-gradients with a good teacher and move toward stronger human correction as teacher bias grows.In the good-teacher regime, the oracle approaches a = 0 and b = 1; as bias grows without bound, a approaches 1.Source: §3.2 · p. 6
  • When S = Θ(1), every unbiased-family estimator has an irreducible Θ(1/n) variance floor, while the optimal ABC estimator improves on it when S > 0 and B_t^2 is finite.The improvement gap is significant when B_t^2 < S/n.Source: §3.2 · p. 6

3.3 Online controller and end-to-end rate

  • For offline reward-model and DPO experiments, the controller uses a corrected-norm surrogate whose expected value tracks per-step MSE up to a parameter-independent term.Projected AdaGrad tunes both parameters over the feasible set; the online route avoids estimating per-example gradient covariances.Source: §3.3 · p. 7
  • Under the stated assumptions and controller-regret conditions, ABC-SSL’s end-to-end rate strictly improves on the best statically tuned PP-SSL.The convergence bound uses the time-averaged ABC mean-square error, controlled by the parameters, in place of labeled-only variance.Source: §3.3 · p. 8

T < ¯V PP-SSL

The stated comparison is against the best static PP-SSL tuning; a comparison with a per-round retuned PP-SSL oracle is not pursued. When the controller keeps a bounded away from 1, the convergence bound retains a persistent bias contribution.

T < ¯V PP-SSL

  • The improvement is against the best static PP-SSL tuning; comparison with a per-round retuned PP-SSL oracle would require dynamic-regret analysis, which the paper does not pursue.The static comparator matches the one used for the regret bound.Source: T + ¯V PP-SSL · p. 8
  • If the controller keeps a bounded away from 1, the bound retains a persistent bias floor; a two-timescale variant drives this contribution to zero asymptotically at the cost of an additional state.The practical footprint of that variant is discussed separately.Source: T + ¯V PP-SSL · p. 8

4 ABC-Align for Prediction-Powered Alignment

ABC-Align is instantiated for reward-model training, DPO, and GRPO. The offline methods use the corrected-norm controller, while on-policy GRPO uses a finite-sample plug-in to address policy-driven gradient shifts.

4 ABC-Align for Prediction-Powered Alignment

  • ABC-Align covers reward-model training for RLHF, offline DPO, and on-policy GRPO, using different adaptation strategies for offline and policy-shifting settings.The offline methods use the corrected-norm controller; GRPO replaces it with a finite-sample plug-in for the per-step oracle.Source: §4 · pp. 8–9

4.1 Background

  • GRPO uses group-relative advantages from multiple completions per prompt instead of PPO’s learned value baseline, with a clipped policy-ratio objective and KL anchor.Reward-model training and DPO are offline preference-learning problems; DPO optimizes the policy directly without learning an explicit reward model.Source: §4.1 · p. 9, §4.2 · p. 10

4.2 ABC-Align for offline preference learning: RM and DPO

  • For offline reward-model training and DPO, ABC-Align applies Algorithm 1 verbatim and updates the joint corrected-norm controller parameters at each step.Human-labeled examples use hard preference labels, while unlabeled examples use teacher-provided soft verdicts on response pairs.Source: §4.2 · p. 10
  • The offline losses share a scalar-margin structure: labeled and teacher-based gradients use the same per-example Jacobian and differ in their residual scalars.This shared factorization lets two weighted backward passes produce both gradient aggregates from one labeled forward pass.Source: §4.2 · p. 10

4.3 ABC-Align for on-policy GRPO

  • Because GRPO’s gradient distribution shifts with the policy, ABC-Align estimates the per-step oracle from the fresh labeled prompt-group batch instead of tracking a fixed-pair controller.ABC mixing applies only to source-specific advantage gradients; the KL gradient is treated as a deterministic offset. The same-round plug-in targets the MSE landscape but does not guarantee the deployed update’s realized MSE, which includes additional cross-terms.Source: §4.3 · pp. 10–12

5 Experiments

The experiments evaluate ABC-Align across synthetic preferences, reward-model training, DPO, and GRPO, comparing gradient estimators under scarce human feedback. Across these settings, results favor ABC-Align, including under teacher bias and changing on-policy data.

5 Experiments

  • The experiments progress from a controlled synthetic preference task to real reward modeling, DPO, and on-policy GRPO, comparing methods that differ only in their gradient estimator.The LLM experiments use Naive combined in place of the Doubly Robust baseline.Source: §5 · p. 12, §5.1 · p. 12

5.1 Synthetic preference experiment on linear reward model training

  • When teacher bias is low, ABC-Align nearly matches pseudo-only performance; as bias rises, it shifts toward stronger human correction and degrades smoothly.Its gains concentrate on hard pairs, where the 50 human labels leave unbiased estimators variance-limited.Source: §5.1 · p. 13, Figure 2 · p. 13

5.2 Reward model training

  • 0.77 vs. 0.70: on Skywork held-out pairwise accuracy at ρ = 0.20 and 1% labels, ABC-Align leads the next-best baseline, PP-SSL.ABC-Align is strongest across the tested label fractions and flip rates.Source: §5.2 · p. 14, Figure 5 · p. 15
  • ABC-Align achieves the best accuracy on RewardBench overall and human-preference splits, and on PPE human-preference and math best-of-k.At label fraction 0.05 and ρ = 0.10, it exceeds PP-SSL by 2.2 pp on RewardBench overall and by 2.7 pp on PPE human-preference.Source: §5.2 · p. 15, Figure 6 · p. 16
  • With a structurally biased real teacher, ABC-Align leads at moderate teacher–human agreement and shifts toward labeled-only estimation as agreement approaches chance.Naive combined falls below labeled-only near chance-level agreement.Source: §5.2 · p. 15, Table 1 · p. 16
  • The fixed ABC-Align mixture peaks near 0.69, above PP-SSL’s fixed-mixture peak near 0.67; online ABC-Align tracks its peak better than online PP-SSL.The comparison uses a fixed-mixture ablation at label fraction 0.05 and ρ = 0.10.Source: §5.2 · p. 15, Figure 7 · p. 17

5.3 DPO-based policy alignment

  • 0.72 at 5% labels: ABC-Align leads held-out DPO preference accuracy at every tested label fraction, with PP-SSL next and Naive combined and labeled-only below.At 5% labels, ABC-Align matches labeled-only accuracy at 30% labels; DPO fine-tunes all policy parameters.Source: §5.3 · p. 17, Figure 8 · p. 18

5.4 On-policy alignment with GRPO

  • 0.721 to 0.752: held-out GSM8K accuracy rises for ABC-Align as the labeled fraction increases from 0.05 to 0.30, leading all baselines at each fraction.PP-SSL trails by 1.5–2 points; the advantage is largest at the smallest label fraction.Source: §5.4 · p. 18

6 Related Work

Related work spans preference- and outcome-based LLM alignment, pseudo-labeling, and prediction-powered methods. ABC-Align differs from prior training approaches by permitting a controlled bias–variance trade-off rather than requiring an unbiased gradient estimate.

Preference-based post-training

  • Preference-based methods use pairwise comparisons, while reasoning models also use outcome-based GRPO; all three alignment families rely on costly, slow, noisy human ground truth.The section discusses reward-model-based RLHF, DPO, and GRPO.Source: §6 · pp. 18–19

Pseudo labels for alignment, and why they are not free

  • Pseudo labels are abundant but can encode systematic judge biases, and training on them can degrade alignment or retain irreducible error without a real-data anchor.The cited examples include verbosity, position, self-preference, sycophancy, and judge–generator lineage biases.Source: §6 · p. 19

Prediction-powered inference and training

  • PPI-style methods anchor pseudo labels to trusted data, but the unbiased class faces limits including a Θ(1/n) variance floor and constraints on label savings or pseudo-label correlation.ABC-Align is contrasted with evaluation-time PPI applications and is described as leaving the unbiased class.Source: §6 · pp. 19–20

Prediction-powered training

  • PP-SSL, PPI-SVRG, and doubly robust self-training use unbiased estimators; ABC-Align adds a second parameter to control bias against variance reduction.The section identifies these methods as closest to ABC-Align and says unbiasedness pins them to the variance floor.Source: §6 · p. 20

Semi-supervised and noise-robust preference learning

  • Other preference-learning methods clean labels or assume a noise model, whereas ABC-Align uses human labels to measure teacher-gradient bias.The human-labeled set is characterized as a measuring instrument rather than additional training signal to augment.Source: §6 · p. 20

Biased gradient estimators

  • Prior biased-gradient work studies variance reduction or convergence, but ABC-SSL contributes online control of bias measured against trusted human labels.The section notes biased estimators in stochastic optimization and RLHF, including GRPO advantages.Source: §6 · p. 20

7 Conclusion

The conclusion presents ABC-SSL as a two-parameter estimator that admits controlled bias to move beyond the variance floor of unbiased prediction-powered methods. The appendix proofs establish its bias relation, MSE structure, and convexity properties.

7 Conclusion

  • ABC-SSL uses a two-parameter gradient estimator that admits controlled bias decoupled from variance reduction, with adaptive parameters and a joint regret guarantee.The conclusion states a strict improvement over the best statically tuned PP-SSL estimator at rate O(T^-1/2) for a broad parameter range.Source: §7 · pp. 20–21
  • ABC-Align dominates labeled-only, naively combined, and PP-SSL baselines across reward-model training, DPO, and GRPO scenarios.The conclusion describes these as three instantiations of ABC-SSL for LLM alignment.Source: §7 · p. 21

A.1.1 Proof of Lemma 1

  • The ABC estimator’s bias is (1 − a_t) times the pseudolabel gradient bias.The proof gives E_t[ĝ_t] = ∇L(θ_t) + (1 − a_t)B(θ_t).Source: §A.1.1 · p. 30

A.1.2 Proof of Proposition 1

  • The per-step conditional MSE decomposes into squared bias and variance, with its variance derived from the labeled and pseudolabeled sample contributions.The proof expresses the labeled-sample contribution through the covariance of each sample’s combined gradient.Source: §A.1.2 · p. 30

A.2.1 Convexity of the per-step MSE

  • For fixed θ_t, the per-step MSE V_t(a, b) is a convex quadratic in the two estimator parameters.The proof derives convexity by expressing summands as squared affine functions or squared norms of affine maps.Source: §A.2.1 · p. 31

A.2.2 Proof of Proposition 2

The proof characterizes an oracle-optimal parameter pair for the limiting MSE and refines it for finite unlabeled sample size. It also identifies degenerate conditions and the asymptotic discrepancy between the finite-sample and limiting solutions.

A.2.2 Proof of Proposition 2

  • The oracle minimizer is characterized for the limiting MSE, and its a* coordinate lies in [0, 1] without imposing that range as a constraint.The proof notes that the relevant squared-bias and residual-variance quantities are nonnegative.Source: §A.2.2 · p. 31
  • When the teacher bias and residual variance both vanish, the limiting MSE is zero along the specified curve of parameter pairs.The proof identifies any pair on that curve as optimal in this degenerate case.Source: §A.2.2 · p. 32

A.2.3 Finite-N refinement

  • For fixed θ_t, the finite-N minimizer differs from the limiting oracle minimizer by O(1/N).The proof also notes that a constrained minimizer over a bounded feasible set generally differs from the unconstrained pair.Source: §A.2.3 · p. 32

A.2.4 Proof of Corollary 2

The proof lower-bounds the variance of unbiased estimators and compares it with the optimal ABC estimator. It also points to the online controller pseudocode in the following section.

A.2.4 Proof of Corollary 2

  • When S = Θ(1), every unbiased-family estimator has a labeled-batch variance floor of at least S/n; optimal ABC strictly improves when S > 0 and B_t^2 is finite.The gap is significant when B_t^2 < S/n, the stated good-teacher regime.Source: §A.2.4 · p. 32

A.3 Online controller and end-to-end rate (Section 3.3)

  • The full pseudocode for ABC-SSL with the joint corrected-norm controller appears at the end of the section.Source: §A.3 · p. 33

A.3.1 Proof of Theorem 1

The proof establishes convergence bounds in terms of ABC’s time-averaged estimator MSE and shows that adaptive control can improve on the best fixed unbiased PP-SSL tuning under stated conditions. It also justifies the surrogate, controller, and split-batch bias estimate used by the algorithm.

A.3.1 Proof of Theorem 1

  • At η = 1/β, the right-hand side is 2β∆0/T + ¯V, recovering Eq. (17); a variance-tuned analogue gives the same bound with ¯V replaced by an a-posteriori variance term.Source: §A.3.1 · p. 33

A.3.2 Proof of Lemma 2

  • The expected joint surrogate equals the per-step MSE plus the parameter-independent term ∥∇L(θt)∥2, so minimizing it is equivalent to minimizing MSE.This identity holds for any conditionally unbiased bHt.Source: §A.3.2 · p. 34

A.3.3 Split-batch unbiased estimator of Ht

  • The split-batch construction gives an unbiased estimate of Ht by partitioning the labeled batch into two independent halves and averaging symmetric cross-products.The construction assumes an even batch size, dropping one sample otherwise.Source: §A.3.3 · p. 34

A.3.4 Proof of Proposition 3

  • Projected AdaGrad achieves the controller’s regret rate without knowing T in advance, while a general O(log T) rate is unavailable when the surrogate Hessian is singular.The Hessian can be singular when dt and ct are linearly dependent; the faster rate would require a positive lower bound on its minimum eigenvalue.Source: §A.3.4 · pp. 34–35

A.3.5 Proof of Corollary 3

  • Under Eq. (19), the adaptive in-class oracle strictly improves on the best fixed PP-SSL tuning when the centered labeled and pseudo-label gradients are not L2-collinear on average.The stated comparison is against the best static parameter on the unbiased slice.Source: §A.3.5 · p. 35, T + ¯V PP-SSL · p. 35

T + ¯V PP-SSL

  • The time-averaged ABC MSE is bounded by the in-class oracle value, which is no greater than the best fixed PP-SSL value.The first comparison follows from the controller regret bound; minimizing over the full feasible set cannot do worse than minimizing over the unbiased slice.Source: T + ¯V PP-SSL · p. 35

A.3.6 Algorithm pseudocode

  • Algorithm 1 updates the gradient-mixture parameters by projected AdaGrad on a quadratic surrogate, using split-batch gradients and an estimate of Ht.In experiments, EMA smoothing of the Ht estimate is a variance-reduction heuristic that forfeits conditional unbiasedness and adds a drift term; the theoretical statements concern the raw estimate.Source: §A.3.6 · p. 36

B.1 Data generation

  • The synthetic data use a linear Gaussian reward, noisy preference labels, and a teacher whose reward direction departs further from ground truth as µ increases.The annotator noise level makes roughly 30% of pairs ambiguous.Source: §B.1 · pp. 36–37

B.2 Training details

The synthetic reward-model experiments use a shared training setup to compare gradient estimators. A plug-in oracle controller closely tracks the headline corrected-norm controller in this setting.

Training details

  • The linear reward model is trained with Bradley–Terry negative log-likelihood using Adam at learning rate 10−3, batch size 32, and 2,000 iterations.The model uses the same training loop and optimizer across methods, which differ only in the gradient estimator.Source: §B.2 · p. 37
  • The joint corrected-norm controller and the finite-sample plug-in oracle track each other closely during training, supporting the surrogate as a tracker of the per-step oracle without per-example covariance estimates.The plug-in companion is a sanity check in a setting where per-example gradients are essentially free.Source: §B.2 · p. 37
  • PP-SSL tunes λt online by projected gradient descent on ∥ˆgt∥2, while the comparison methods share the practical minibatch loop and optimizer.The PP-SSL tuning uses step size 0.01 and clips λt to [0, 1].Source: §B.2 · p. 37

B.3 Metrics

The evaluation centers on held-out preference accuracy and reports that ABC-Align’s advantage is concentrated on hard comparisons. The appendix describes supplementary metrics, experimental conditions, and out-of-domain benchmark tables.

B.3 Metrics

  • Preference accuracy is the primary reported metric; reward MSE and Bradley–Terry log-loss show consistent trends but are not reported in the main text.Preference accuracy measures agreement with ground-truth preference direction, while log-loss measures calibration of predicted probabilities.Source: §B.3 · p. 37

C Tabular versions of the main-text figures

  • The out-of-domain reward-model benchmark table covers label fraction 0.05 and teacher flip rate ρ = 0.10, with ∆ defined as ABC-Align minus the best baseline.The supplied caption gives the table’s scope and comparison convention, not cell values.Source: Table 2 · p. 38

B.4 Subgroup analysis

  • ABC-Align’s advantage is concentrated on hard pairs, where labeled-only and unbiased estimators are variance-limited with n = 50; pseudo-label methods perform similarly on easy pairs.The passage attributes the hard-pair gains to leaning more heavily on pseudo labels while applying a controlled correction.Source: §B.4 · p. 38

B.5 Experimental conditions

  • Synthetic experiments use 50 labeled pairs and 5,000 unlabeled pairs for the main curve, averaged over 25 trials; Figure 2 error bars show ±1 standard error.The appendix uses 10 trials for the teaser heatmap and trajectory and MSE ablations.Source: §B.5 · p. 38

D LLM experiment details: RM, DPO, and GRPO

  • The LLM experiment configurations share architecture, tokenizer, optimizer, and update budget within each experiment, varying only the gradient estimator.The listed evaluation spans in-domain Skywork accuracy and out-of-domain RewardBench and PPE benchmarks.Source: D LLM experiment details: RM, DPO, and… · p. 38

D.1 Reward model training

The LLM experiments adapt ABC’s controller to reward-model training, offline DPO, and on-policy GRPO. An appendix extension adds a fast bias-tracking update to asymptotically remove residual squared bias, subject to additional assumptions.

D.1 Reward model training

  • The reward-model comparison freezes the Llama-3.2-3B-Instruct backbone and trains only its scalar score head to isolate gradient-estimator effects from full-model optimization.Evaluation combines Skywork validation accuracy with RewardBench and PPE.Source: §D.1 · pp. 38–39

D.2 DPO-based policy alignment

  • For full-model DPO, a one-sided streaming estimate preserves conditional unbiasedness while limiting extra memory to one full-model gradient buffer; experimental EMA smoothing forfeits exact unbiasedness and adds drift.The controller uses projected AdaGrad, while PP-SSL adapts its interpolation parameter online.Source: §D.2 · p. 39

D.3 On-policy GRPO

  • GRPO uses labeled prompts with verifiable correctness rewards and unlabeled prompts scored by a teacher; a split-batch plug-in selects (at, bt) as the policy and gradient distribution shift across iterations.The comparison sweeps labeled fractions {0.05, 0.10, 0.30} and reports held-out GSM8K pass@1.Source: §D.3 · p. 39

E Two-Timescale Debiasing of the ABC-Align Estimator

  • The two-timescale extension modifies the slow gradient with a fast bias-tracking update, aiming to eliminate residual squared bias without changing the controller.The appendix presents this construction as orthogonal to the controller choice.Source: E Two-Timescale Debiasing of the… · p. 39

E.1 Motivation

  • Because the controller can trade bias for variance, its convergence guarantee retains a residual bias floor; the two-timescale construction targets that floor while preserving controller variance reduction.The fast auxiliary iterate tracks the unknown bias online.Source: §E.1 · p. 40

E.2 Debiased estimator and algorithm

  • The auxiliary ξt tracks the bias through an EMA-like update, and the slow estimator subtracts (1 − at)ξt; driving tracking error to zero removes the estimator’s bias.This auxiliary state enters the slow update, not the controller, and requires no additional batch-gradient computation.Source: §E.2 · p. 40

E.3 Assumptions

  • The two-timescale convergence analysis requires regularity of B, bounded second moments, and a decaying step-size schedule; one stated schedule yields an O(k−2/3) tracking rate.A horizon-tuned polynomial finite-horizon rate is also noted for a constant slow step size.Source: §E.3 · pp. 41–42

E.4 Convergence guarantee

The section gives asymptotic and finite-horizon convergence guarantees for two-timescale ABC-Align, then discusses implementation costs and assumptions that bound those guarantees.

E.4 Convergence guarantee

  • Theorem 2 replaces the persistent bias floor with a tracking-error term (1 − at)^2Vt that vanishes asymptotically, without constraining at.Source: §E.4 · p. 42
  • O(T^-1/3) bounds the expected gradient norm at a uniformly sampled iterate when both constant step sizes are tuned to horizon T.Source: §E.4 · pp. 45–46
  • O(1/log T) is the rate for the ηt-weighted average gradient norm under the concrete decaying-step schedule ηt = c2t^-1.Source: §E.4 · p. 46

E.5 Practical considerations

  • Algorithm 2 stores one model-gradient-sized vector and adds two scalar-vector operations per step, with no additional batch-gradient computations.Source: §E.5 · p. 46
  • Removing the residual bias floor requires smoothness, iterate-boundedness and noise-moment assumptions, plus a schedule with ηt/γt → 0.Source: §E.5 · p. 47

Related papers

Loading 2609.34374v1…