PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

Yu Li, Yufa Zhou, Tian Lan

arXiv:2610.40945v1cs.LGcs.AI

TL;DR

Training-inference mismatch makes language-model reinforcement learning off-policy, and its engine-numerics and stale-weight sources require different corrections. The paper separates these components, follows their effects through training, and introduces DLC to correct each according to its dynamics. Under stale sampling, DLC stays near the on-policy noise floor on two Qwen3 models, while importance sampling loses stability at the lag-ratio effective-sample-size threshold.

  • Problem

    Engine numerics and stale sampler weights both create training-inference mismatch, but correction performance depends on the mismatch source.

    Source: §1 · p. 2

  • Method

    The paper evaluates the trainer at sampler weights to separate numerical discrepancy from policy lag, then centers the discrepancy, clips the lag ratio, and resynchronizes at an effective-sample-size threshold.

    Source: §1 · p. 2, §7.1 · p. 11, §8 · p. 15

  • Results

    DLC keeps stale-sampler deviation within twice the noise floor on Qwen3-0.6B and Qwen3-1.7B; on Qwen3-1.7B, importance sampling leaves the noise floor at step 11 when effective sample size falls below one half.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Mismatch corrections need to account for whether an error component stays fixed or feeds back through the trainer’s updates.

    Source: §1 · p. 2, §8 · p. 15

Abstract

from arXiv · show

On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. \textit{We ask which property of a mismatch source decides the correction it needs.} Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a \textit{numerical discrepancy} of the engine and a \textit{policy lag} of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the \textbf{discrepancy-lag correction} (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.

1 Introduction

Training-inference mismatch combines engine numerics with stale sampler weights, but these sources evolve differently during training. The paper separates their effects, traces how policy lag feeds back into updates, and derives a correction that handles both.

1 Introduction

  • Prior results show that correction performance depends on the mismatch source, motivating the question of which source property determines the correction it needs.Score centering stabilizes quantized sampling but falls behind importance sampling under staleness, while PPO shows the opposite pattern.Source: §1 · p. 2
  • The trainer’s forward pass at the sampler’s weights splits sampled-token error into a numerical discrepancy and a policy lag, which remain approximately fixed and grow with updates, respectively.After centering removes drift, the remaining discrepancy bias stays fixed while the policy-lag bias grows with accumulated updates.Source: §1 · pp. 2–3
  • The policy lag’s token and prefix terms enter the next update, closing a feedback loop that ratio weighting cuts only until the lag-ratio effective sample size falls below one half.On Qwen3-1.7B, that threshold is reached at step 11, three steps before the Gaussian prediction.Source: §1 · p. 3
  • DLC centers the numerical discrepancy, clips the lag ratio against sampler weights, and resynchronizes the sampler when the lag-ratio effective sample size falls below one half.These choices target the approximately fixed discrepancy and the growing policy-lag loop separately.Source: §1 · p. 3
  • Under stale sampling, DLC keeps deviation within twice the noise floor on Qwen3-0.6B and Qwen3-1.7B, and resynchronization prevents collapses left by clipping alone.On Qwen3-1.7B, importance sampling leaves the noise floor at step 11, when the lag-ratio effective sample size falls below one half.Source: §1 · p. 3

2 Background

The background distinguishes infrastructure mismatches from the weight-version and engine-numerics differences that reach the algorithm. It reviews corrections for off-policy sampling and staleness, while motivating separate treatment of mismatch components according to whether they evolve with training.

2.1 Training-inference mismatch

  • Once sampling, reporting, token sequences, and batch composition are aligned, the algorithm must correct only the changing weight-version lag and the approximately fixed engine-numerics discrepancy.The paper classifies the former as policy lag and the latter as numerical discrepancy.Source: §2.1 · p. 3
  • Prior experiments report source-dependent correction behavior: score centering works best with quantized sampling, while importance sampling with centering performs better under staleness.PPO survives staleness in those experiments but collapses under quantization and synthetic weight noise.Source: §2.1 · p. 4

2.2 Off-policy correction and staleness

  • Existing off-policy and asynchronous methods clip or weight ratios, control update statistics, or modify updates; the paper positions DLC as assigning corrections by whether each mismatch factor enters the training loop.DLC uses the lag ratio’s second moment to decide when to resynchronize sampler weights.Source: §2.2 · pp. 4–5

3 Setup

The setup defines the sampler-weighted update and how common corrections treat its components, then specifies controlled sources of mismatch. Training runs compare deviations against on-policy references on fixed prefixes.

3.1 Sampler, trainer and corrections

  • At a prefix, the sampler-weighted update contains a drift term equal to expected advantage times mean score, which can be nonzero on prefixes carrying the learning signal.Group-centering advantages does not eliminate this drift because correct-answer and post-error prefixes can have opposite expected advantages.Source: §3.1 · p. 5
  • Importance sampling weights sampled-token updates by a truncated probability ratio, while score centering subtracts the sampler’s mean score and leaves the covariance term.PPO instead clips the trainer-to-reference-policy ratio.Source: §3.1 · p. 5

3.2 Mismatch sources and training

  • The experiments compare reference, FP8-quantized, 64-step-stale, and combined quantized-stale samplers, with temperature-one sampling and aligned token ids.The stale sampler retains its initial weights during the 40 training steps.Source: §3.2 · p. 5
  • Qwen3-0.6B and Qwen3-1.7B are trained on Countdown for 40 steps across three runs, with deviation measured as full-vocabulary KL on fixed initial-model prefixes relative to paired on-policy training.The noise floor is the mean KL between independently trained on-policy models; test accuracy uses 64 held-out Countdown problems.Source: §3.2 · pp. 5–6

4 Numerical Discrepancy and Self-Induced Policy Lag

The corrections behave differently under quantization and stale sampling: score centering handles quantization but falls behind importance sampling with centering under staleness. An exact split of sampled-token error distinguishes engine numerical discrepancy from trainer policy lag, which grows with the trainer’s updates.

4.1 Source dependence of the corrections

  • Under staleness, score centering reaches 32 times the noise floor and 12.2% test accuracy, while importance sampling with centering stays at 2.1 times the floor and ends at 32.0%.Under quantization, the corrections stay at the noise floor, while score centering matches on-policy training’s 46.5% test accuracy.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Exact split of the sampled-token error

  • Evaluating the trainer at the sampler’s weights splits sampled-token error exactly into policy lag and numerical discrepancy.The split uses a trainer forward pass at the sampler’s weights to isolate the effects of policy staleness and engine numerics.Source: §4.2 · pp. 6–7
  • Quantized-sampler numerical discrepancy stays near 0.02 nats per token, while stale-sampler policy lag grows by two orders of magnitude on both model sizes.The quantization sequence offset stays constant, whereas the stale offset grows by about 0.7 nats per step.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Numerical discrepancy is independent of the accumulated update, so centering removes its first-order bias; with both sources, the accumulated update stays within 10% of stale runs.In combined-source runs, the numerical discrepancy retains its quantization-only size, while lag reaches the stale value by step 40.Source: §4.2 · p. 7

5 Training Loop of the Policy Lag

Under staleness, accumulated updates create policy lag, which leaves residual correction bias that enters the next update and can amplify further movement. Score centering leaves a lag-dependent token term; importance sampling removes it within its validity range, while the resulting feedback loop adds a positive quadratic contribution to excess update norm.

5.1 Links of the loop

  • The loop runs from accumulated update to token lag, residual correction bias, and the next update; negative eigenvalues amplify bias-driven movement until synchronization resets Δ.To first order, bias is linear in accumulated update, and a step multiplies the bias-driven part of Δ by 1 − ηλ along an eigenvector with eigenvalue λ.Source: §5.1 · p. 8

5.2 Residual bias of each correction

  • An 8.9-nat quantization sequence offset leaves residual bias at gradient noise; under staleness, score-centering bias leaves the floor near step 12 as lag-offset covariance grows.The offset’s size alone does not bias the update; the policy lag supplies covariance with the gradient.Source: §5.2 · p. 9
  • Importance sampling removes the token term only within ratio truncation at 2; truncated tokens retain excess ratio weight.Score centering keeps the token term (A_t − Ā_t)ℓ_t, which importance sampling removes within its validity range.Source: §5.2 · p. 9
  • Score-centering bias grows linearly with ∥Δ∥, reaching 2.5 times gradient noise by step 40, while importance-sampling bias stays within the noise.The measured residual bias agrees with the predicted forms; the prediction aligns above gradient noise, mainly through the prefix term.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 Loop gain

  • At step 20, score-centering mean lag is −0.66 for negative-advantage tokens versus −0.37 for positive-advantage tokens, while importance-sampling groups agree.Source: Figure 3 · p. 8, §5.3 · p. 10
  • Excess update norm follows a positive quadratic in ηk: the pooled fit gives c = 0.52 (R2 = 0.987), versus 0.90 for the linear-only fit.The quadratic coefficient is positive in each of the six fits by learning rate and run.Source: Figure 3 · p. 8, §5.3 · p. 10

6 Validity Range of Importance Weighting

Importance weighting is effective only while the lag ratio remains sufficiently concentrated; on Qwen3-1.7B its effective sample size crosses the one-half threshold before the Gaussian prediction. PPO’s clipping behavior under mismatch also depends on where its reference policy is set.

6.1 Tail of the ratio

  • On Qwen3-0.6B, importance-sampling mean lag stays below 0.3 nats in magnitude through step 40; at step 40, 11% of tokens have ratio below 0.5 versus 28% predicted by a Gaussian lag.Most tokens remain near ratio one, with lag concentrated on a few tokens.Source: §6.1 · p. 10
  • On Qwen3-1.7B, importance-sampling effective sample size falls below one half at step 11, three steps before the Gaussian prediction, and deviation leaves the floor from that step.The crossing occurs at mean lag −0.15 nats per token, compared with the Gaussian prediction of −0.35.Source: §6.1 · p. 10, Figure 4 · p. 11

6.2 Reference policy of PPO

  • A sampler-weight PPO reference puts policy lag inside the clipped ratio and numerical discrepancy outside it; a trainer-before-batch reference leaves policy lag outside the ratio.Source: §6.2 · pp. 10–11
  • With the trainer-before-batch reference, PPO under staleness reaches 699 times the noise floor, versus 534 without clipping; sampler-weight-reference DLC stays at 1.9.With sampler weights as reference, the clipped ratio contains policy lag and leaves numerical discrepancy outside it as drift.Source: §6.2 · p. 11

7 Discrepancy-Lag Correction

DLC combines centering of numerical-discrepancy drift, clipping of the policy-lag ratio, and sampler resynchronization when the lag ratio loses effective sample size. Under staleness, it stays close to on-policy training and avoids off-support collapses that sampler-prefix deviation can miss.

7.1 Method

  • DLC clips the lag ratio against sampler weights to remove the policy-lag token term, then centers the weighted score at each prefix to remove numerical-discrepancy drift.The centering term estimates the sampler’s mean using head log-probabilities and a tail estimate.Source: §7.1 · pp. 11–12
  • When the lag ratio’s effective sample size falls below half the batch, DLC copies trainer weights to the sampler, resetting the lag and the clip reference.Importance sampling with resynchronization lowers deviation from 115 to 2.6 times the floor on Qwen3-1.7B.Source: §7.1 · p. 12, §7.2 · p. 13

7.2 Deviation and test accuracy

  • On Qwen3-0.6B, DLC reaches 1.9 times the floor under staleness and 2.2 with both mismatch sources; on Qwen3-1.7B, it stays within 1.2 times the floor through step 40 and ends at 0.6.The comparison is deviation from on-policy training, measured as a multiple of the noise floor.Source: §7.2 · p. 13, Table 4 · p. 14
  • On Qwen3-1.7B, DLC ends at 51.9% average test accuracy versus 57.6% for on-policy training, with each run within 7 percentage points of its counterpart.By comparison, importance sampling with resynchronization averages 36.6% because one run collapses.Source: §7.2 · pp. 14–15, Table 4 · p. 14

7.3 Mass off the support of the initial policy

  • Without resynchronization, two of three DLC runs fall below initial-model accuracy; their off-support share reaches about 90%, while it remains at or below 11% under DLC.The deviation measured on sampler prefixes stays at the floor during the collapses, and the shift begins on prefixes the stale sampler does not reach.Source: Figure 5 · p. 14, §7.3 · p. 15

8 Conclusion

The conclusion distinguishes numerical discrepancy from policy lag by their different training dynamics. DLC addresses both and, with resynchronization, keeps stale-sampler training close to on-policy training while preventing collapses revealed by off-support responses.

8 Conclusion

  • Engine numerical discrepancy stays approximately fixed, whereas policy lag grows from trainer updates and feeds back through residual centering bias into later updates.Ratio weighting cuts the feedback loop only until the lag ratio’s effective sample size falls below one half.Source: §8 · p. 15
  • DLC centers the discrepancy, clips the lag ratio, and resynchronizes at the effective-sample-size crossing; it stays within twice the noise floor and prevents collapses revealed by off-support share.Source: §8 · p. 15

Reproducibility

The reproducibility notes identify the training and measurement setup, while the notation appendix specifies how token indices and sampler expectations are used.

Reproducibility

  • Training uses verl with vLLM samplers, three runs per configuration on Qwen3-0.6B and Qwen3-1.7B, and fixed-checkpoint offline measurements on fixed probe prefixes.Source: Reproducibility · p. 15

A Notation

  • The notation appendix states that indexed quantities belong to the sampled token at its prefix and that E_q ranges over sampler responses.Source: A Notation · p. 19, Table 5 · p. 19

B Experimental Details

The experiments use a fixed Countdown training configuration and define deviation, residual-bias, and off-support measurements. Table 6 compares two ways of reporting step-40 deviation on Qwen3-0.6B.

B Experimental Details

  • Each batch has 64 Countdown prompts with eight responses each; training uses reward ±1 minus the group mean, token-averaged loss, and one SGD step at learning rate 10^-2.The setup uses BF16 compute, FP32 master weights, and no momentum, warmup, or weight decay.Source: B Experimental Details · p. 19
  • Table 6 reports Qwen3-0.6B step-40 deviation using both the ratio of mean KL to mean floor and the mean of per-run ratios; the correction ordering is the same except for a DLC/importance-sampling-with-centering swap under staleness.Source: Table 6 · p. 19
  • Deviation is full-vocabulary KL to paired on-policy training averaged over fixed probe prefixes; residual bias compares corrected and on-policy gradients on the same regenerated batch.Off-support share is the fraction of trainer-response sequence KL on tokens with p_θ0(y_t) < 10^-3, measured over 256 responses.Source: B Experimental Details · p. 19

C Step Cost, a Larger Model and Continuous Asynchrony

The section compares DLC under stale and continuously asynchronous sampling, alongside its cost and the prefix-level behavior of score centering and importance weighting. On Qwen3-8B, score centering does not contain policy lag, while DLC remains near the noise floor; continuous-asynchrony results also report an admission variant that defers low-ESS responses.

C Step Cost, a Larger Model and Continuous Asynchrony

  • DLC adds a sampler top-128 report and a forward pass at sampler weights, while its once-per-run resynchronization takes at most 1.5 seconds per copy.The extra forward pass includes about 19 of 54 seconds from a reference forward whose output does not enter the loss at zero KL coefficient.Source: C Step Cost, a Larger Model and… · p. 20, Table 7 · p. 20
  • On Qwen3-8B, DLC ends at 49.6% test accuracy and 3.1 times the noise floor, versus 51.8% and 2.9 for importance sampling with centering and 55.3% for on-policy training.Score centering reaches 89 times the floor, indicating that policy lag again exceeds its reach on the larger model.Source: Table 8 · p. 20, C Step Cost, a Larger Model and… · p. 20
  • In continuous asynchrony, DLC ends at 1.2 times the floor and 55.9% test accuracy; the admission variant reaches 0.8 times the floor after deferring 0.3% of responses.Admission defers a response when the lag-ratio effective sample size falls below one half and regenerates it with newer weights.Source: Table 9 · p. 21, C Step Cost, a Larger Model and… · p. 21

D Drift and the Corrections at a Prefix

  • At a prefix, score centering leaves the update unchanged under a constant advantage shift, while truncated importance weights can restore drift when the ratio exceeds 2.The sampler reports probabilities on its head; the tail estimate uses the trainer's probabilities scaled to match the sampler's tail mass.Source: D Drift and the Corrections at a Prefix · pp. 21–22

E Exact Split of the Sampled-Token Error

The section separates sampled-token error into numerical discrepancy and policy lag, then traces how correction residuals and trainer updates evolve. It also compares linearized bias predictions with exact weights and derives when bias feedback can grow within a synchronization period.

E Exact Split of the Sampled-Token Error

  • The exact split evaluates both policies at the sampler's weights: numerical discrepancy depends on the engine and sampled token, while policy lag changes as the trainer moves.The sequence offset decreases as the sequence KL from sampler weights to trainer increases.Source: E Exact Split of the Sampled-Token Error · pp. 22–23

F Residual Bias of the Corrections

  • The first-order residual-bias terms include prefix, continuation, and token effects; the continuation term depends on errors after each token and their covariance with return.Under staleness, the token term compares the sampled advantage with its prefix mean and the token lag.Source: F Residual Bias of the Corrections · pp. 23–24

G Exact-Weight Recomputation of the Residual Bias

  • Exact-weight recomputation reduces the predicted-to-measured residual-bias norm ratio from up to 3400 to between 1.9 and 7.9, with cosine above gradient noise of at most 0.07.The comparison uses the same checkpoints, batches, and backward passes for linearized and exact weights.Source: G Exact-Weight Recomputation of the… · p. 24, Table 10 · p. 25

H Unrolling the Loop

  • Within a synchronization period, negative-eigenvalue components of the bias-driven update grow faster than the on-policy component, while positive-eigenvalue components saturate.The excess update norm of score centering over importance sampling has a positive quadratic term when below-prefix-mean advantages dominate the relevant score projections; the fit separates fixed bias from loop feedback.Source: H Unrolling the Loop · pp. 25–26

I Effective Sample Size and Truncation under a Gaussian Lag

This section relates lag-ratio concentration to importance-weight validity, then examines PPO's reference choice and DLC's clipping, centering, and loss construction. It identifies an effective-sample-size threshold for resynchronization and describes how DLC handles discrepancy and lag at a prefix.

I Effective Sample Size and Truncation under a Gaussian Lag

  • On Qwen3-1.7B, the lag-ratio effective sample size falls below one half at step 11, when mean lag is −0.15 nats rather than the Gaussian prediction of −0.35.The section attributes the earlier crossing to a heavy tail in the ratio; truncation at 2 limits the excess ratio excluded from the weight.Source: I Effective Sample Size and Truncation… · p. 27

J Reference Policy of PPO

  • With PPO's reference at the trainer before each batch, the clip bounds only within-batch changes and leaves both policy lag and numerical discrepancy outside the ratio.At the first update per batch, the ratio is one and the clip is inactive.Source: J Reference Policy of PPO · p. 27

K Properties of DLC

  • DLC's clip has a validity range like importance-sampling truncation, so it resynchronizes when that range ends; copying weights resets lag but leaves numerical discrepancy unchanged.The prefix analysis also states that, up to the tail estimate, DLC removes drift from discrepancy and lag.Source: K Properties of DLC · p. 28

L Loss of DLC

  • The DLC loss adds a centering term to the PPO loss with sampler weights as reference, using head log-probabilities from the forward pass at sampler weights.The implementation averages both terms over batch tokens and clamps the ratio to the interval from e^-20 to e^20.Source: L Loss of DLC · p. 29
Loading 2610.40945v1…