Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL

Chenlu Ye; Xuanchang Zhang; Yifan Hao; Zhou Yu; Ziji Zhang; Abhinav Gullapalli; Hao Chen; Jing Huang; Tong Zhang

Source-linked AI summary

Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL

Chenlu Ye, Xuanchang Zhang, Yifan Hao, Zhou Yu, Ziji Zhang, Abhinav Gullapalli, Hao Chen, Jing Huang, Tong Zhang

arXiv:2603.19470v3Mar 19, 2026cs.LGcs.AI

TL;DR

Off-policy staleness and training–inference mismatch destabilize LLM RL through heavy-tailed importance ratios and sharp policy geometry. ALP adds learnable hidden-state perturbations and uses one perturbed-policy ratio against the inference policy; across single-turn and multi-turn tasks, it improves stability and final performance while controlling ratio tails and KL spikes.

Problem
Policy staleness and training–inference mismatch create heavy-tailed importance ratios and unstable optimization, limiting training stability and exploration in LLM RL.
Method
ALP injects learnable perturbations into hidden states across layers and uses a single importance ratio between the perturbed training policy and unchanged inference policy.
Results
Across single-turn and multi-turn reasoning tasks, ALP improves training stability and final performance while avoiding importance-ratio-tail explosions and KL spikes.
Takeaways & Limitations
Hidden-state perturbation, especially across all layers, is the most effective tested design and supports more robust, exploratory off-policy RL.
Takeaways & Limitations
ALP assumes inference-time mismatch can be modeled as small zero-mean system-induced noise and that perturbations are not too large.

Abstract

from arXiv · show

Off-policy problems such as policy staleness and training--inference mismatch have become a major bottleneck for training stability and further exploration in LLM RL. The distribution gap between the inference and updated policies grows because of the techniques to enhance inference efficiency, leading to heavy-tailed importance ratios. Heavy-tailed ratios arise when the policy is locally sharp, which further inflates gradients and can push updates outside the trust region. To address this, we propose Adaptive Layerwise Perturbation (ALP), which injects small learnable perturbations into the input hidden states of each layer during updates and uses the resulting perturbed policy as the numerator of the importance ratio against the unchanged inference policy in the objective. Intuitively, by adding controlled noise to intermediate representations, ALP prevents the updated policy from deviating too sharply from the inference policy and enlarges the policy family to cover inference-time mismatch noise. Hence, the flattened distribution can naturally tighten the gap between the updated and inference policies and reduce the tail of importance ratios, thus maintaining training stability. This is further validated empirically. Experiments on single-turn math and multi-turn tool-integrated reasoning tasks show that ALP not only improves final performance, but also avoids blow-up in the importance-ratio tail and KL spikes during iterative training, along with boosted exploration. Ablations show that representation-level perturbations across all layers are most effective, substantially outperforming partial-layer and logits-only variants.

1 Introduction

ALP frames off-policy instability as both a ratio-correction and policy-geometry problem, then unifies staleness and training–inference mismatch through layerwise perturbation. Theory and experiments indicate smoother, more stable optimization with improved robustness and exploration.

Motivation: Off-policy effects from repeated updates, inference-engine differences, and agentic distribution shifts produce heavy-tailed ratios, KL spikes, and brittle optimization.These effects make robust off-policy RL an urgent problem for LLMs.
Motivation: Prior mismatch corrections can over-truncate valid updates by separately truncating two ratios, inducing bias and early plateau.TIS/MIS use an auxiliary proxy-policy ratio and threshold-based truncation or masking.
Method: ALP injects learnable perturbations into input hidden states across layers and applies a single ratio with the perturbed training policy as numerator.The inference policy remains unperturbed in the objective.
Mechanism and results: Perturbation smooths local optimization geometry, suppresses heavy-tailed ratio excursions, tightens the training–inference gap, and encourages exploration.The full-training comparison reports stable ALP behavior where unperturbed Bypass becomes sharp and its ratio tail explodes.
Theory: Theory bounds updated-to-inference KL when σ2 matches or exceeds inference-distribution bias, increasing the probability that updates remain within a trust region.The analysis also connects perturbation to loss smoothing and less attraction to sharp optima.
Experiments: Across single-turn and multi-turn reasoning tasks, ALP improves stability and performance, while all-layer perturbation is most effective in ablations.Partial-layer variants closer to lower layers perform better than those restricted to upper layers.

2 Adaptive Layerwise Perturbation

The section introduces ALP as a perturbation-based approach that modifies the training policy and importance ratios to address training–inference mismatch. Its analysis links perturbation to mismatch coverage, trust-region behavior, and smoother optimization geometry.

2.1 Prior Approaches: Prior objectives use token- or sequence-level importance ratios to correct differences between training and inference policies, but Bypass does not effectively stabilize mismatch.Bypass can exhibit ratio-tail blow-up even from the same checkpoint and rollouts.
2.2 Our Approach: ALP adds Gaussian perturbations to layer inputs and forms a perturbed training policy, while leaving the inference policy unperturbed.The perturbation is sampled across layer representations and token positions; only the training policy is perturbed.
2.2 Our Approach: The objective uses token-level or sequence-level ratios based on the perturbed training policy, preserving the corresponding loss formulations while changing the training-policy numerator.The supplied passages identify both ratio granularities and state that perturbation applies only to the training policy.
2.3 Analysis: Robustness of ALP: ALP models inference as training behavior with system-induced bias and enlarges the effective policy family to cover mismatch noise.The analysis states that perturbation should be large enough to cover mismatch noise but small enough to preserve the original policy.
2.3 Analysis: Robustness of ALP: Perturbation is theoretically associated with a higher likelihood of remaining close to the rollout distribution and within a trust region.The stated trade-off is between covering mismatch noise and preserving the original policy.
2.3 Analysis: Robustness of ALP: ALP also performs local averaging that reduces effective curvature and discourages optimization toward sharp, fragile regions.Figure 2 supports this mechanism with a flatter surrogate and a narrower conditional log-ratio quantile envelope, especially for low-probability tokens.

3 Experiments

Single-turn experiments evaluate ALP across math benchmarks and training dynamics, finding strong average performance and more stable optimization than several baselines.

Evaluation setup: The single-turn evaluation compares ALP with Seq-Bypass, Seq-MIS, Token-MIS, and vanilla GRPO across five math benchmarks.The reported metric is average@32 at temperature 1.0.
Evaluation setup: Token-ALP attains the highest average score across five single-turn math benchmarks under average@32 evaluation.Evaluation uses temperature 1.0 and a 4096-token generation limit.
Evaluation results: Seq-ALP achieves a 67.52 overall average score on Qwen3-4B, exceeding GRPO at 66.24, Token-MIS at 65.33, and Seq-Bypass at 64.12.The result suggests ALP’s gain persists at a larger model scale.
Training dynamics: Both token-level and sequence-level ALP keep gradient norms controlled, entropy reasonable, and rollout-training KL near zero throughout training.GRPO and Seq-MIS instead show late-stage gradient-norm and KL spikes with reward collapse.

4 Multi-Turn Agentic Reasoning

Multi-turn tool-integrated reasoning experiments show that ALP combines strong benchmark performance with stable dynamics and improved exploration, with sequence-level perturbation particularly effective in this setting.

Experimental setup: The TIR setup uses a Python interpreter, zero-RL training from Qwen2.5-7B-base, and episodes allowing up to five code-execution turns.The rollout batch size is 512 and the maximum response length is 16K.
Main results: Seq-ALP achieves the highest overall average accuracy, 50.53, on multi-turn TIR benchmarks.Its gains are broad-based, although AIME25 slightly favors Token-MIS.
Training dynamics: Seq-ALP maintains controlled train-inference KL, higher but stable entropy, and stable convergence across both KL metrics.Token-MIS is unstable in train-inference KL, while Seq-MIS shows policy-update KL spikes.
Token versus sequence level: Sequence-level ALP performs best in multi-turn settings, where greater drift and mismatch can make token-level ratio bias more consequential.The paper presents this as a plausible efficiency–stability explanation rather than a definitive mechanism.
Exploration: Across rollout budgets k = 16–256, ALP attains the highest Pass@k on AIME24 and AIME25.The advantage becomes more pronounced as k increases, indicating more effective exploration and solution diversity.
Layer ablation: The multi-turn ablation reports that all-layer perturbation outperforms partial-layer perturbations.This complements the broader finding that representation-level perturbations are most effective when applied across layers.

5 Ablations

ALP’s ablations show that perturbing hidden representations across all layers is most effective, outperforming partial-layer and logits-only alternatives in single- and multi-turn reasoning.

Perturbation targets: All-layer perturbation consistently achieves the strongest average accuracy across single-turn benchmarks, while logits-only perturbation is least effective.The tested settings include all layers, early layers, middle layers, late layers, and output logits only.
Overall pattern: The ablation ranking is all layers > partial layers > logits-only, supporting representation-level family enlargement over output noise alone.Logits-only perturbation is substantially weaker than hidden-state perturbation.
Multi-turn ablation: All-layer perturbation attains the best overall score across multi-turn agentic benchmarks, whereas partial-layer perturbations trail behind.The result is evaluated under sequential feedback and tool-induced distribution shift.
Mechanistic interpretation: Perturbing all layers produces the smallest probability-ratio tail deviations, while late-layer-only perturbation exhibits heavier tails.The authors attribute this to broader enlargement of the effective policy family and better support overlap with the rollout distribution.

6 Conclusions

The paper introduces ALP as a unified approach to stabilize off-policy LLM RL by addressing policy staleness and training–inference mismatch. Across single- and multi-turn reasoning tasks, ALP improves stability and final performance, with all-layer hidden-state perturbation performing best in ablations.

Contribution: ALP injects learnable hidden-state perturbations into the updated policy and optimizes one importance ratio against the rollout distribution.The formulation targets policy staleness and training–inference mismatch together.
Empirical findings: ALP improves training stability and final performance across single-turn and multi-turn reasoning tasks.The paper also reports that perturbation tightens train–inference discrepancy and smooths sharp local optimization geometry.
Ablation conclusion: Hidden-state perturbation across all layers is the most effective ablation configuration.The conclusion identifies broad representation-level perturbation as especially effective.
Future directions: Fully asynchronous RL and Mixture-of-Experts settings are identified as promising future directions because off-policy effects and systems mismatch may be more severe there.

A Additional Related Works

Related work addresses off-policy optimization through ratio aggregation, mismatch correction, consistency enforcement, and theoretical perturbation analyses. These approaches target instability from stale policies, inference systems, and distributional mismatch.

Off-policy correction: Prior methods stabilize off-policy optimization by changing importance-ratio aggregation, correcting training–inference mismatch, or clipping and masking outliers.The cited approaches include sequence-level ratios, Bypass, TIS/MIS, and other ratio corrections.
Systems approaches: Strict bitwise-consistent inference and higher precision can reduce training–inference mismatch but may be computationally inefficient or difficult under production constraints.The stated constraints include changing execution engines, rollout quantization, and asynchronous rollouts.
Technical lemmas: The technical development defines Fisher information through Lemma 1 and uses Gaussian perturbations in Lemma 2 to define a perturbed policy as an expectation over noise.The proof then differentiates this expected policy with respect to the input.
Proof technique: The Gaussian-perturbation proof uses integration by parts to move derivatives onto the Gaussian density, whose gradient has a closed form and vanishing boundary term.

C Formal Statement and Proof for Theorem 1

The formal analysis imposes bounded posterior perturbation means and uniform domination of the perturbed policy by the original policy. Under these conditions, the theorem bounds training–inference mismatch when σ is properly learned.

Conditions: Condition 1 bounds the squared ℓ2 norm of the posterior mean perturbation by O(dσ2).Here d is the input dimension and σ2 is the Gaussian prior variance.
Conditions: Condition 2 requires the perturbed policy to be uniformly dominated by the original policy up to a constant factor α.The condition specifies 0 < α ≤ 1.
Proof: The proof bounds the KL distance using Condition 2, Lemma 2, and Condition 1.The derivation applies the condition-based inequality, the Gaussian perturbation identity, and the posterior-mean bound.
Theorem consequence: With Conditions 1 and 2, properly learning σ can effectively control training–inference mismatch during training.

D Proof for Theorem 2

The proof introduces bounds on the importance ratio and perturbation-radius measure, then combines these with Gaussian-measure properties to establish the stated result.

The proof defines b as the maximum importance ratio I(x, θ) and introduces a constant c in the interval (0, 2).
For any ϵ > 0, the perturbation standard deviation σ can be chosen subject to a bound on the measure of an r-radius ball.
The argument applies the bound to arbitrary x and a before completing the proof through the preceding inequalities.
A Gaussian distribution assigns less probability to the high-importance-ratio set than to a centered ball with the same volume C.

E Additional Experimental Details

The appendix specifies the datasets, task setup, and baseline ratio-control settings used in the single-turn and multi-turn experiments.

Single-turn details: Single-turn training merges math data from Guru RL-92k and a 75K OpenR1 subset after score filtering and prompt-content deduplication.Guru contains 91.9K samples spanning six reasoning-intensive domains.
Baseline settings: Table 5 is identified as documenting the parameter settings for importance ratios.
Baseline settings: Token-level importance ratios use lower clipping 0.2 and higher clipping 0.2 or 0.28, while sequence-level ratios use 0.5 and 3.0.
Baseline settings: MIS uses a masking threshold of 2.0.
Multi-turn details: Multi-turn experiments use Tool-Integrated Reasoning with a Python interpreter and mask void turns that contain neither tool calls nor final answers.

E.1 Hyperparameter sensitivity

ALP is evaluated across several math benchmarks while varying perturbation noise and learning rate; performance remains stable across a broad hyperparameter range and improves during training.

The sensitivity study varies initial Gaussian noise standard deviation and perturbation-parameter learning rate, reporting average@32 across five math benchmarks.The benchmarks are Math500, Minerva Math, OlympiadBench, AIME24, and AIME25.
Figure 7 compares final-task performance for GRPO, MIS, Bypass, and ALP in the single-turn setting.
Figure 8 reports mean adaptive noise scale, perturbation-induced probability differences, and their 75th-percentile tail across training steps.
Performance curves are tightly clustered across a broad range of (std, lr) values and improve monotonically over training.
The default configuration is std = 1 × 10−4 and lr = 5 × 10−4, selected from final-stage performance.
The mismatch diagnostic plots training probability πtrain against rollout-time probability πinferθold, with tighter concentration around y = x indicating smaller mismatch.Each panel reports the Pearson correlation coefficient r.