PaperlayerView exact revision on arXiv

Source-linked AI summary

On KL-Regularized Policy Optimization

Yifan Zhang

arXiv:2610.08963v1cs.LGcs.AIcs.CL

TL;DR

Asynchronous agent RL trains on trajectories from stale samplers whose probabilities can differ from the trainer’s, while clipping can bias updates and response groups are costly for long episodes. KLPO anchors the KL regularizer at the sampler and regresses the resulting log-ratio condition without importance weights. Its token-level gradients can be computed from terminal returns, and independent Monte Carlo KL estimates preserve unbiasedness.

  • Problem

    Asynchronous agent RL uses trajectories from stale samplers whose probabilities can differ from the trainer’s, while clipping can bias updates and response groups are costly for long episodes.

    Source: §1 · pp. 1–2

  • Method

    KLPO anchors the KL regularizer at the sampler, regresses the improvement’s log-ratio condition, and profiles the intercept into a sampler mean plus a sampler-to-trainer KL divergence.

    Source: §1 · p. 2

  • Results

    Token-level gradients follow from terminal returns through sampler-centered scores or a trajectory residual, and independent Monte Carlo KL estimates keep them unbiased.

    Source: §5.1 · p. 11

  • Takeaways & Limitations

    Token-level KLPO uses one rollout per prompt without a critic, learned normalizer, or response group, fitting asynchronous agentic training.

    Source: §8 · p. 15

    Top-K and binary KL approximations lose exact identities; their tail gap can make updates depend on reward offsets and leave the PMD target nonstationary.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-$K$ and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

1 Introduction

Asynchronous agent training routinely uses trajectories from stale samplers whose probabilities can differ from the trainer’s, while common corrections can bias updates or require costly response groups. KLPO anchors its regularizer at the sampler and fits the resulting log-ratio condition by regression, enabling critic-free token-level updates from terminal returns.

1 Introduction

  • KLPO fits the sampler-anchored improvement’s log-ratio condition by regression, avoiding importance weights while covering staleness and execution mismatch.Replay across checkpoints becomes a mixture of per-sampler regressions.Source: §1 · p. 2
  • Profiling the regression intercept replaces the unknown log-partition function with the signal’s sampler mean plus a sampler-to-trainer KL divergence.The profiled gradient is the regularized-return gradient with the likelihood ratio removed.Source: §1 · p. 2
  • SPPO and GPO are response-level KLPO, REBEL its pairwise profiled form, and BPO’s exact trajectory loss equals the profiled token-level objective.Each practical loss differs from its KLPO instance by an identified approximation.Source: §1 · p. 2
  • Token-level policy mirror descent gradients follow from terminal returns through sampler-centered scores or a single trajectory residual, including with stochastic tool outputs.Both routes avoid a critic.Source: §1 · p. 2
  • Independent Monte Carlo estimates of the local KL keep both gradients unbiased, while top-K and binary approximations fall short by the divergence inside the merged tail.Replay across versions remains a fixed mixture of per-version regressions.Source: §1 · p. 2

2 Background

The background describes asynchronous collection in which workers generate trajectories using sampler checkpoints while a learner trains a separate policy. The setup records sampler-specific probabilities and versions so each trajectory can be evaluated against its collection policy.

2 Background

  • Reused trajectories retain their collection sampler version, and the learner fits each trajectory against that sampler.Reuse increases lag without changing the sampler checkpoint that generated the trajectory.Source: §2 · pp. 2, 4
  • The sampler and trainer can differ because trajectories use earlier checkpoints and because inference execution can change probabilities even at matched parameters.The sampler probabilities include the transforms actually used during collection and also serve as the KL reference.Source: §2 · p. 3
  • A response includes termination and its generated tokens; in agentic tasks, each history also contains tool outputs, while response length counts policy-generated tokens.Source: §2 · p. 4
  • The analysis assumes bounded responses and rewards, normalized positive policies on common support, and sampler-matched collection for each version.Deterministic transitions and terminal rewards support samplewise telescoping identities; with tools, this requires deterministic tool outputs.Source: §2 · p. 4

3 KL-Regularized Policy Optimization

KLPO starts from a sampler-anchored KL-regularized improvement and regresses its log-ratio optimality condition. Profiling removes the unknown normalizer, while response-level and token-level instances connect the framework to existing objectives and expose different practical constraints.

3 KL-Regularized Policy Optimization

  • KLPO applies sampler-anchored KL-regularized improvement to a context and outcome scored by an improvement signal.This primitive covers contexts such as prompts or token histories and outcomes such as complete responses or next tokens.Source: §3 · pp. 4–5, §3.1 · p. 5

3.1 Sampler-Anchored Improvement

  • The unique improvement target is a Gibbs policy proportional to the sampler probability times the exponentiated signal scaled by β.Its logarithm gives an optimality condition between the trainer-to-sampler log-ratio and the signal shifted by a context-dependent constant.Source: §3.1 · p. 5

3.2 Regressing the Optimality Condition

  • Regressing the log-ratio optimality condition identifies the same target under any replay law covering the support, without an importance multiplier in the gradient.The regularized-return and regression gradients differ by a likelihood ratio and coincide when trainer and sampler match; restricted model classes can have different minimizers.Source: §3.2 · p. 5

3.3 Profiling the Normalizer

  • Profiling the intercept replaces the unknown log-partition function with a sampler mean plus a sampler-to-trainer KL divergence, and the profiled loss reaches zero at the target when realizable.The profiled and fixed-normalizer losses share their realizable minimizer, though their gradients differ away from it.Source: §3.3 · p. 6

3.4 Instances

  • Sequence-level KLPO needs a prompt mean reward and complete-response KL that a single response cannot supply, limiting its fit for asynchronous training.At token level, the profiled intercept is a local KL computable from the two next-token distributions, but the residual still contains an advantage ordinarily requiring a critic.Source: §3.4 · p. 6
  • Token-level targets multiply to the response-level Gibbs target only when accumulated local normalizers are constant across responses.Regressing against optimal soft values instead of sampler advantages recovers the response-level target.Source: §3.4 · p. 6
  • SPPO and GPO are response-level KLPO, REBEL is its pairwise profiled form, and BPO’s exact trajectory loss equals the profiled token objective.AWR and RPG share the Gibbs target but fit it differently.Source: §3.4 · p. 7

4 Token-Level KLPO without a Critic

Section 4 develops a profiled token objective and two critic-free ways to compute its gradient from terminal returns. Token regression centers trainer scores under the sampler; sequence regression uses a trajectory residual.

4.1 The Profiled Token Objective

  • The profiled token objective has the token PMD target as its unique zero-loss policy when that target is realizable.Profiling the intercept at each history yields the profiled token objective.Source: §4.1 · p. 8

4.2 Token Regression with Sampler-Centered Scores

  • Terminal returns give token-regression gradients through trainer scores centered under the sampler, with one response per prompt when the score mean is action-independent.Continuation noise adds variance but no bias when the score is centered under the sampler that generated the action and continuation; the log-ratio term moderates the update as the trainer approaches its target.Source: §4.2 · pp. 8–9

4.3 Sequence Regression by Bellman Telescoping

  • Sequence regression squares a single trajectory residual; its token residuals have zero conditional mean and telescope into terminal reward under Assumption 2.3.Random tool outputs add residual noise whose contribution to the loss does not depend on θ, so this route requires neither a value estimate nor a group of responses nor deterministic transitions.Source: §4.3 · pp. 9–10
  • Under Assumptions 2.1 and 2.2, sequence regression and token regression agree in population, and the sequence objective matches exact BPO.The prompt-value term in the BPO residual does not affect the gradient and can be dropped.Source: §4.3 · pp. 9–10

5 Estimating the Local KL Correction

The section compares independent Monte Carlo estimates of the local KL correction with cheaper top-K and binary coarsenings. Independent draws preserve the full-KL gradient identities in expectation, while coarsening introduces an exact tail-divergence gap and changes the update.

5.1 Monte Carlo KL

  • For token regression, averaging independent auxiliary-token samples leaves the expected gradient unchanged for every M ≥1.A single estimate can be negative, and clipping it at zero destroys unbiasedness.Source: §5.1 · p. 11
  • Independent auxiliary draws preserve the full-KL sample gradients in expectation, and sequence regression uses a leave-one-out surrogate to avoid bias from reusing draws.Sequence regression requires at least two draws per prefix; the estimates are single tokens rather than continuations, so they need no further generation or reward.Source: §5.1 · p. 11
  • Fresh auxiliary draws are needed for conditional unbiasedness at every update; using the rollout action as its own draw makes the correction bracket vanish.The unbiasedness statements condition on a trainer chosen independently of the draws.Source: §5.1 · p. 11

5.2 TopK-KL and Binary KL

  • Top-K and binary KL fall short of the full KL by exactly the divergence within the merged tail.Top-K is exact when the tail contains one token; binary KL uses the sampled token as its head.Source: §5.2 · p. 12
  • Top-K and binary approximations lose the zero-mean identities, so reward offsets can change updates and the two gradient routes may disagree.MC-KL preserves exact identities in expectation at the cost of variance and M records per position; Top-K is deterministic and nearly exact when sampler mass is concentrated, while binary KL is cheapest and crudest.Source: §5.2 · p. 12

6 Learning from Lagged Samplers

KLPO handles lagged samplers by keeping each trajectory tied to its collection policy and combining version-specific updates. It supports single-rollout asynchronous training without clipping importance ratios, a critic, a learned normalizer, or a second response.

6.1 Mixtures of Sampler Versions

  • Complete, unfiltered rollouts from each recorded sampler yield expected gradients equal to the mixture of per-version objectives, weighted by version frequencies.Sampler quantities must remain version-specific, while trainer-dependent quantities are recomputed at each reuse.Source: §6.1 · p. 13

6.2 Consequences for Asynchronous Training

  • KLPO has no importance ratio to clip as checkpoints age or trajectories grow, because its regression gradient removes the trainer-to-sampler ratio.Errors in logged probabilities or stored conditionals can still shift the log-ratio or KL correction.Source: §6.2 · p. 13
  • With full KL or MC-KL, one complete rollout gives an unbiased stochastic gradient without a value function, prompt normalizer, or second response.The learner trains on completed trajectories without waiting for another response to the same prompt.Source: §6.2 · p. 13, §6 · p. 13
  • Both token-gradient routes apply to interactive trajectories with stochastic tool outputs, but their returns estimate the ordinary sampler Q-function, not an optimal soft Q-function.The response-level target can reweight tool outcomes the policy cannot control, so no policy may attain it.Source: §6.2 · p. 13

7 Related Work

The related work positions KLPO as log-ratio regression on a sampler-anchored target, connecting it to policy-improvement projections, preference optimization, token-level PMD, and asynchronous critic-free RL.

KL-regularized policy improvement

  • REPS, MPO, and AWR fit the regularized improvement target by weighted maximum likelihood, while PMD analyzes repeated steps and RPG uses a reverse-KL projection.KLPO fits the same target through its log-ratio condition; the projections share an unrestricted optimum but can differ in restricted model classes.Source: §7 · p. 14

Log-ratio regression and preference optimization

  • SPPO, GPO, and REBEL regress log-ratios or log-ratio differences onto preference signals, and KLPO places them within its framework through the signal f.This accommodates general preferences, which can be cyclic.Source: §7 · p. 14

Token-level PMD and score centering

  • KLPO token regression applies sampler score-centering with coefficient R −βℓθ,u, while its sequence regression retains the residual and allows random tool outputs.BPO’s practical loss adds group normalization, binary KL, smoothing, and clipping.Source: §7 · p. 14

Off-policy and critic-free RL for language models

  • Unlike clipped ratios or group-relative advantages, KLPO changes the objective so sampler probability enters a log-ratio rather than a multiplicative weight.At token level it fits ordinary sampler advantages and learns no value function, unlike value-function-based approaches discussed in this section.Source: §7 · p. 14

8 Conclusion

The paper concludes that sampler-anchored log-ratio regression yields KLPO’s token-level, critic-free update and recovers several preference-optimization methods as special cases. Its response-level instance instead fits a response Gibbs target and depends on normalization information.

8 Conclusion

  • KLPO fits the sampler-anchored Gibbs target by regressing its log-ratio condition, with token-level gradients available from terminal returns and independent Monte Carlo KL estimates.The conclusion also identifies SPPO, GPO, REBEL, and BPO as special cases and calls for large-scale evaluation, finite-sample and convergence analysis, and value learning for stochastic environments.Source: §8 · pp. 14–15

A Response-Level KLPO (SKLPO)

  • SKLPO regresses the response Gibbs condition with one normalizer per prompt and sampler version; unlike token-level KLPO, it cannot avoid that information.Its response residual multiplies every token score, whereas sequence regression uses a path sum of local KL corrections and targets local PMD.Source: A Response-Level KLPO (SKLPO) · p. 18, §A.2 · p. 19

A.1 The Response Gibbs Target

  • The response Gibbs target is defined relative to the collection sampler, and a finite trainer step need not reach it.Subtracting a prompt-dependent baseline from the reward shifts the normalizer but leaves the target unchanged.Source: §A.1 · p. 18

A.2 Regressing the Response Condition

  • With a realizable target, the SKLPO regression objective vanishes exactly at the target policy; in a restricted model class, the regression and target minimizers can differ.The response log-ratio sums token log-ratios before squaring, with the reward and normalizer entering once.Source: §A.2 · p. 19

A.3 Outer Length Weighting

  • Positive response weights preserve the target but change how a restricted model trades off errors across responses.At equal absolute residual, a length-100 response contributes ten times as much loss as a length-1000 response; dividing by total token count is a different batch reduction.Source: §A.3 · p. 20

A.4 Replay across Sampler Versions

Across sampler versions, replay fits version-specific targets, while normalization choices determine what can be learned from limited rollout data. The later derivations connect profiled and terminal-reward objectives to Bellman and soft-value formulations under stated conditions.

A.4 Replay across Sampler Versions

  • Distinct full-support samplers generally induce competing Gibbs targets, so replay’s best fit depends on their relative frequencies and need not be the on-policy return gradient.The zero-loss target exists only when the sampler targets coincide; for a common reward and β, full-support samplers share a target only when they are equal.Source: §A.4 · pp. 20–21

A.5 The SKLPO Update

  • SKLPO reuses each response’s historical sampler probabilities, but one-response training needs independent normalization information or a shared predictor.A learned normalizer is an auxiliary scalar predictor, not a token-value critic.Source: §A.5 · p. 21, §B.1 · p. 22, §B.3 · p. 24

B.2 Fixed Normalizers; B.3 Profiling within a Prompt

  • Profiling centers the response signal within a prompt, while fixed-normalizer plug-in estimates trade unbiasedness of the partition estimate against a downward-biased log-normalizer.The exponential estimate e^(bc/β) is unbiased for Z, whereas E[bc] ≤ c∗; empirical centering changes when sampler versions or outcome-dependent selection are pooled.Source: §B.2 · p. 22, §B.3 · p. 23

B.4 Centering with Length Weights

  • Length-weighted centering must use the same outer weight as the loss, and its intercept generally differs from both the ordinary mean and fixed c∗.The unweighted (G−1)/G bias formula does not carry over to random-length weights.Source: §B.4 · p. 24

B.5 Fixed Token Normalizers

  • A fixed token normalizer can be computed or estimated from next actions given exact sampler advantages, but those advantages require same-sampler continuation information.A sampled action probability supplies the log-ratio, not the fixed normalizer or conditional score mean.Source: §B.5 · pp. 24–25

B.6 A Terminal-Reward Form of the Token Objective

  • With exact local KLs, terminal-reward regression has the same population gradient as the token objective when the rollout law and fixed baseline are held fixed.Dropping the KL derivative generally changes the gradient, and using the sampled action as the baseline cancels the learning signal.Source: §B.6 · p. 25

C.1 The Regularized Bellman Target

  • Sampler-regularized Bellman equations define the optimal values and policy, and their values converge to the unregularized optimum as β decreases to zero.At finite β, future KL costs enter continuation values; exponentiating ordinary Q⋆ or sampler value does not generally solve this Bellman system.Source: §C.1 · p. 26

C.2 Soft-Value Token Regression

  • In deterministic generation, soft-value token regression and SKLPO share the response Gibbs target, while local PMD targets differ; stochastic tools can break the deterministic equivalence.Soft-value regression needs values at every prefix, whereas SKLPO needs the reward and prompt normalizer; unrestricted trajectory tilts can change outcomes outside the policy’s control.Source: §C.2 · p. 26, Figure 3 · p. 27

C.3 The Implicit Action-Value Representation

This section characterizes how KLPO regression represents regularized policies and action values across sequence- and token-level objectives, including the roles of sampler-based residuals and Monte Carlo KL estimates. It also examines when KL estimates remain unbiased and the limitations of finite-head and binary approximations, while noting caveats involving exact-fit results, stochastic tools, and reused auxiliary draws.

C.3 The Implicit Action-Value Representation

  • Under replay coverage and realizability, zero-loss regression recovers the optimal soft advantage, but exact fit does not guarantee neural optimization outcomes.The policy log-ratio represents action-value differences; absolute Q-values require a Bellman-consistent state baseline. With stochastic tools, unrestricted trajectory tilts can change outcomes beyond the policy’s control.Source: §C.3 · p. 27

C.4 Policy Regression with an Optimal Q-Function

  • With state-action coverage and realizability, regression against fixed optimal soft Q-values identifies the optimal policy; a sampler’s terminal return cannot generally replace that target.The regression applies to action policies under stochastic transitions.Source: §C.4 · p. 28

D.1 The Two Instances

  • Profiling removes the fixed normalizer from both KLPO instances, but only the token-level intercept needs local KL information; the response-level intercept needs additional quantities.The token-level advantages can be removed from the residual, as described in Section 4.Source: §D.1 · p. 28, Table 4 · p. 28

D.2 The Sequence-Regression Route

  • Summing token residuals yields a trajectory residual that matches BPO’s under deterministic transitions; random tool outputs add only a trainer-independent term.The sequence-regression route differentiates through full local KLs and estimates them with at least two independent draws per prefix; leave-one-out residuals preserve an unbiased gradient. The two routes share a population gradient but differ sample by sample.Source: Figure 4 · p. 29, §D.2 · p. 29, §E.6 · p. 35

D.3 The Token-Regression Surrogate

  • With the specified stop-gradients, the token-regression surrogate differentiates to the critic-free score-centered gradient using a complete rollout and an independently supplied conditional score mean.The token-regression loss generally remains nonzero at its target because terminal returns contain continuation noise; score centering computes the profiled-objective gradient but is not an importance-corrected current-policy return gradient.Source: §D.3 · p. 29, §E.3 · pp. 33–34

D.4 Sequence Regression and the Response-Level Instance

  • Sequence regression and SKLPO both square one residual per response but differ in correction and target: a trainer-dependent path sum of local KLs versus a fixed prompt normalizer.Their targets are local policy mirror descent and the response Gibbs policy, respectively, and coincide only in the special case noted in Section 3.4.Source: §D.4 · p. 30

D.5 TopK-KL and Binary-KL Surrogates

  • TopK-KL approximates the sampler tail as proportional to the trainer’s, while binary KL partitions the sampled token from all others and uses a complementary-probability correction, not an importance ratio.The binary partition depends on the sampled action; BPO’s practical loss additionally linearizes the residual and applies group normalization, smoothing, and clipping.Source: §D.5 · pp. 30–31

D.6 Reusing Auxiliary Draws

  • Conditionally unbiased gradients require fresh auxiliary draws from the historical sampler after trainer updates; reusing fixed draws instead defines a sample-average objective without the stated expectation identities.Fresh draws require stored sampler conditionals or the ability to rerun the historical sampler.Source: §D.6 · p. 31

E.6 Monte Carlo KL and Leave-One-Out Residuals

  • Fresh auxiliary tokens from the historical sampler restore conditional unbiasedness, while reusing rollout actions or sampling from the trainer changes the estimator.The identities assume the trainer is independent of the auxiliary records; after updating on fixed records, recomputing their log-probabilities gives a fixed-sample surrogate. Leave-one-out uses all M draws symmetrically without enumerating pairs; with M = 1, sequence regression is undefined but token regression remains valid.Source: §E.6 · pp. 35–36

E.7 TopK-KL

  • A finite top-K head is only an approximation: its gradient error depends on sampler-centered signal, so reward offsets can change the update.The head-and-tail partition generally has a nonzero KL gap; conditional centering, baseline invariance, and route equivalence do not extend to finite heads. Detach the whole coefficient, not just the tail ratio; a singleton tail recovers full KL, while a sampled-action singleton head gives the binary partition.Source: §E.7 · p. 36

E.8 Binary KL

  • Binary-KL identities apply to the binary objective, whose corrected log-ratio can have nonzero sampler mean, breaking baseline invariance and allowing reward offsets to change gradients.With two vocabulary tokens, the binary partition is the full vocabulary.Source: §E.8 · p. 37

E.9 Mixtures of Sampler Versions

This appendix establishes how KLPO behaves across sampler versions and policy-improvement formulations, then examines normalizer profiling and empirical centering. Its identities rely on specified sampling and coverage conditions, and some gradient or target equivalences do not extend to restricted models or singleton groups.

E.9 Mixtures of Sampler Versions

  • With complete, unfiltered rollouts, both expected gradients equal the version-frequency-weighted mixture objective; outcome-dependent replay selection can break these identities.A shared trainer at zero loss must match every component target, and distinct full-support samplers cannot share the same Gibbs target for a common reward and β.Source: §E.9 · p. 37

F.1 Response Targets and Positive Weights

  • The response Gibbs target is shared by regularized-return and log-ratio regression objectives under realizability, but their restricted-model projections can differ.The unregularized state and action values converge to their regularized counterparts as β ↓ 0.Source: §F.1 · p. 38, §F.2 · p. 38

F.3 Implicit Soft Advantages

  • Token soft-value regression identifies local conditionals, while the induced response Gibbs policy matches only under the stated factorization conditions.Under deterministic transitions, token residuals telescope along the trajectory; the regularized Bellman policy links soft values to the token target.Source: §F.3 · pp. 38–39, §F.5 · p. 39, §F.4 · p. 39

F.6 Terminal-Reward Token Regression

  • Terminal-return token gradients can avoid a critic, but dropping the KL derivative requires the sampler-value baseline rather than an arbitrary baseline.With a different baseline, the conditional residual can have nonzero mean, so the KL derivative generally cannot be omitted.Source: §F.6 · p. 39

G RPG–URKL and Gradient Identities

  • At matched policies, the unweighted SKLPO and RPG population gradients agree, while RPG’s per-response gradient scales the SKLPO gradient by the trajectory ratio.The unweighted regression lacks RPG’s prompt-baseline invariance because its zero-loss target depends on the correct normalizer.Source: G RPG–URKL and Gradient Identities · p. 40

H.1 The Normalizer as a KL Divergence

  • The normalizer is a pure KL term; this removes the response partition function as BPO does in its PMD optimality condition (Song et al., 2026, Section 3.1), though the identity still involves the unknown optimum.Source: §H.1 · pp. 40–41

H.2 Binary Rewards

  • For binary rewards, a success-rate estimate can provide a plug-in normalizer, but it must be specific to the prompt and collection sampler.A global rate or an exponential average across changing policies generally estimates a different quantity.Source: §H.2 · p. 41

H.4 Empirical Centering and Singletons

  • Empirical centering has finite-group bias at fixed parameters, and centering a singleton response makes its residual loss identically zero.For independent samples, the expected centered squared residual is (G − 1) Var(hθ)/G.Source: §H.4 · pp. 41–42

H.5 Data Selection and the Normalizer

This section treats data selection, weighted profiling, and normalizers for single rollouts. The regression measure and sampler-specific normalizer must be matched, while learned intercepts require generalization and can yield an imperfect policy gradient.

H.5 Data Selection and the Normalizer

  • Under model misspecification, the best fit can differ between selected-data and sampler-conditioned losses, and finite data do not guarantee support coverage.A profiled intercept under the selected response law identifies the Gibbs target under the stated support and realizability assumptions, with finite-sample behavior depending on that law.Source: §H.5 · p. 42

H.6 Weighted Profiling

  • Positive fixed weights change the regression measure but preserve the original Gibbs target under full coverage and realizability.The sampler in the log-ratio and the fixed Gibbs normalizer remain those of the original target.Source: §H.6 · p. 42

H.7 Learned Normalizers for Single Rollouts

  • A learned normalizer must describe the specified prompt and sampler version; selection, length weighting, or pooling versions changes the quantity being learned.For replay, separate estimation data or held-out folds are needed for the independence premise; detaching does not undo fitting on the same record.Source: §H.7 · p. 43
  • For binary rewards, a single success or failure can update the success predictor but cannot justify using that same label as its own normalizer estimate.For bounded rewards, the alternative predictor targets an exponential moment under the specified sampler.Source: §H.7 · p. 43
  • An imperfect learned intercept does not give the exactly profiled policy gradient, and a flexible predictor can memorize singleton residuals instead of generalizing.A shared prompt-level predictor avoids a token-value critic, not all auxiliary value estimation.Source: §H.7 · pp. 43–44

I.1 The Local Importance Multiplier

These sections compare KLPO gradients and residuals across local, response-level, and token-level formulations, then spell out limits involving stale data, stochastic environments, coverage, and sequence length. Several equivalences hold only under explicit sampling, target, or feasibility conditions.

I.1 The Local Importance Multiplier

  • At a fixed history, importance sampling multiplies the return-gradient contribution by the trainer-to-sampler ratio, while KLPO’s regression gradient has no such multiplier or clipping threshold.Outcome-dependent admission can change the action distribution required by the return-gradient identity.Source: §I.1 · p. 44

I.2 Comparison with FlashREINFORCE

  • FlashREINFORCE and SKLPO both average token scores within a response, but SKLPO uses a complete-response optimality residual rather than a batch advantage.The methods place collection probabilities differently: FlashREINFORCE uses a multiplicative ratio, while SKLPO uses its logarithm in the response residual.Source: §I.2 · p. 45

I.3 Centering under the Behavior Distribution

  • Behavior centering must use the actual replay distribution, and its resulting score identity is not generally an unbiased on-policy return gradient.A prompt-dependent reward offset cancels in the centered form.Source: §I.3 · p. 45

I.4 Single Rollouts and Signed Feedback

  • Single-rollout signed feedback pushes up success likelihood and down failure likelihood without a token-value critic, but stale responses can violate the sign and finite updates need not reach the target odds.Asynchronous single-rollout SKLPO still requires a normalization estimator; empirical centering one response gives zero loss.Source: §I.4 · p. 46

I.5 Feasible Updates in Interactive Environments

  • A trajectory Gibbs target may be infeasible when tools are stochastic or initial conditions vary, even though common stochastic transition factors cancel from likelihood ratios.A statewise regression offers a separate route for stochastic environments, but its continuation values must come from the same sampler as its denominator.Source: §I.5 · pp. 46–47

I.6 A Stochastic-Tool Counterexample

  • With one forced action and a fair random terminal reward, unrestricted trajectory tilting raises success probability despite the action being unable to change it; the regularized Bellman value remains 1/2.This counterexample separates the unrestricted trajectory tilt from the feasible policy update.Source: §I.6 · p. 47

J.3 BPO Equation (25) and Score Centering

  • The exact BPO trajectory loss equals token PMD regression in population, but practical BPO approximations fall outside that equality; the response and token targets also agree only under a constant path normalizer.Token reductions, per-response length division, and length-averaged log-ratios can change gradient identities or the residual.Source: §J.2 · p. 48, §J.4 · p. 49, K Detailed Comparisons with Prior… · p. 50

L.2 Coverage and Uniqueness

  • Coverage failures can destroy uniqueness, and length weighting can change gradients even when it preserves realizable component targets.A nonvanishing mean token log-ratio can make the length-weighted loss and its bound grow linearly with sequence length.Source: §L.2 · p. 51, M Sequence Length Weighting and… · p. 51

N Implementation Details

The appendix specifies token- and response-level updates, replay records and reuse, numerical safeguards, and the SKLPO update for responses from collection policies.

N Implementation Details

  • Token-level replay stores sampler log-probabilities and collection versions, scores trainer tokens, and forms route-specific losses using the selected KL correction.Stored sampler probabilities should include collection transforms such as temperature and truncation.Source: N Implementation Details · p. 52, §N.1 · p. 52

N.1 Records, Reductions, and Reuse

  • Reuse fixes sampler records, rewards, and collection versions but recomputes trainer-dependent quantities; unbiased MC-KL gradients require fresh historical-sampler draws per update.Prompt, padding, and tool-output tokens are excluded from the loss.Source: §N.1 · p. 52

N.2 Numerical Safeguards

  • Active TopK-KL floors can break correspondence to the exact KL gradient, while binary complementary log-probabilities use stable log1mexp arithmetic.The TopK-KL correspondence assumes inactive floors.Source: §N.2 · p. 53

N.3 The SKLPO Update

  • For unfiltered responses to one prompt from one sampler, Equation (B.2) estimates the normalizer without length weights; pooled versions need separate justification, and singleton group means are unsuitable.With one response per prompt, use a supplied normalizer or the shared learned intercept instead.Source: §N.3 · p. 53

Related papers

Loading 2610.08963v1…