Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

1 INTRODUCTION

• RL-trained LLM agents are sensitive to deployment-time perturbations; pruning 20% of FFN channels nearly collapses WebShop performance.The introduction contrasts this with pruning and post-training quantization on static, single-turn tasks, where performance is largely retained.Source: §1 · p. 1
• Robustness to policy perturbation in long-horizon LLM agents remains largely unstudied, and existing methods usually target one perturbation type.The paper frames robustness to a general family of perturbations as a more challenging open problem.Source: §1 · p. 2
• A single early wrong action can derail a long-horizon trajectory, so perturbation injection must be controlled to preserve training stability.The method keeps perturbations relatively mild along sensitive dimensions.Source: §1 · p. 2
• The work formulates perturbation robustness for RL-trained LLM agents under a general distribution of policy perturbations.The introduction presents this as the first study of this setting.Source: §1 · p. 2
• The paper extends TRPO monotonic-improvement analysis to perturbations, deriving a sufficient condition for stable improvement and showing update limit points are perturbation-robust.This is the paper’s theoretical contribution.Source: §1 · p. 2
• SPrPO turns the analysis into a practical condition for controlling perturbations during policy optimization.The method is named Stable Perturbation-Robust Policy Optimization.Source: §1 · p. 2
• Across ALFWorld and WebShop, SPrPO shows more stable training optimization and better preserves task performance under Gaussian noise, dropout, pruning, and quantization.The experiments include structured and unstructured pruning.Source: §1 · p. 2

2 RELATED WORK

• The related-work review is brief here, with more detailed discussion deferred to Appendix B.Source: §2 · p. 2
• Robust RL introduces perturbations adversarially for worst-case performance or stochastically for expected performance; stochastic approaches avoid explicit inner optimization.The cited work typically concerns control settings, while this paper focuses on policy perturbations rather than action or transition-dynamics noise.Source: §2 · p. 2
• LLM perturbation training includes input-representation and low-rank adversarial perturbations, exploration-oriented RL noise, and methods targeting quantization robustness.Examples include adaptation to quantization errors and adaptive control of quantization-induced noise during RL.Source: §2 · p. 2

3 BACKGROUND AND PROBLEM FORMULATION

• The section first defines perturbation robustness, then reviews the TRPO lower bound and monotonic-improvement theory.Source: §3 · p. 3

3.1 PROBLEM SETUP AND PERTURBATION-ROBUST POLICY

• An LLM policy generates textual actions from states containing task context and interaction history, producing trajectories and discounted returns in a discounted MDP.Source: §3.1 · p. 3
• Policy perturbations act on parameters or hidden representations; perturbations within radius δ form the set Ξδ used to define robustness.The passage names Gaussian noise, random dropping, pruning, and quantization as examples.Source: §3.1 · p. 3
• Probabilistic robustness allows degradation greater than ε only with probability β under perturbations bounded by δ, whereas uniform robustness requires a stronger guarantee.The paper adopts probabilistic robustness as the more practical criterion because uniform robustness can be overly restrictive.Source: §3.1 · p. 3

3.2 TRPO LOWER BOUND AND MONOTONIC IMPROVEMENT PROPERTY

• The section introduces a policy update from θ to θ′ and measures its improvement as the difference in their returns.Source: §3.2 · p. 3
• The TRPO lower bound combines surrogate improvement under the candidate policy with a penalty that grows with policy deviation.Source: §3.2 · p. 3
• When deviation is controlled, increasing surrogate improvement guarantees monotonic improvement in the true return.The surrogate term measures improvement relative to the current policy.Source: §3.2 · p. 3

4.1 POLICY IMPROVEMENT UNDER PERTURBATIONS

• A positive expected perturbed surrogate does not rule out degradation for a substantial subset of perturbations, motivating a high-probability improvement guarantee.The surrogate is estimated under the perturbed current policy’s state-visitation distribution.Source: §4.1 · p. 4
• Because deployment perturbations are generally unknown, the analysis uses Gaussian perturbations as a tractable proxy whose variance can be controlled through dimension-dependent scales.The stated motivation is that deployment improvement variance is difficult to characterize directly.Source: §4.1 · p. 4
• For Gaussian perturbations, the stability condition links expected surrogate improvement and policy drift to sensitivity-weighted perturbation variance, yielding nonnegative improvement with probability at least 1−β.It favors smaller variance in more sensitive dimensions and can tolerate larger variance when surrogate improvement is larger.Source: §4.1 · p. 5

4.2 CONVERGENCE UNDER STABLE POLICY UPDATES

• Under the stated assumptions, stable updates have nonempty limit points, and every limit point is (δ, ϵ, β̄)-probabilistic perturbation robust.The result assumes a compact policy space, continuity of perturbed return almost everywhere, proper initial variance, and a sufficiently small per-step degradation probability.Source: §4.2 · pp. 5–6

• SPrPO applies controlled Gaussian hidden-space perturbations and estimates sensitivity and improvement statistics from rollout samples.The method is presented as a practical realization of the stability condition for LLM policy optimization.Source: §5 · p. 6
• The method perturbs intermediate FFN channels with multiplicative Gaussian noise rather than modifying the full parameter space.Applying noise after the up projection and activation is equivalent to perturbing corresponding columns of the down-projection matrix, with simpler implementation and more stable training.Source: §5 · p. 6
• SPrPO reuses the current rollout batch to estimate channel sensitivity and expected improvement, using squared PPO gradient norm as an improvement proxy.After the policy update, it replays the batch for one additional forward and backward pass without extra rollouts.Source: §5 · pp. 6–7
• Channel perturbation standard deviation increases with estimated improvement and decreases with channel sensitivity; ρ sets the base scale and α controls sensitivity adaptation.α = 0 gives sensitivity-independent noise, while α = 1 gives inverse-square-root sensitivity scaling.Source: §5 · p. 7

6.1 EXPERIMENT SETUP

• The evaluation uses Qwen2.5-1.5B-Instruct with GiGPO on ALFWorld and WebShop, comparing Vanilla with multiplicative Gaussian and SAM perturbations.Evaluations cover ALFWorld validation tasks and 500 WebShop tasks, with channel-wise Gaussian, dropout, structured pruning, and weight-space pruning and quantization perturbations.Source: §6.1 · pp. 7–8

6.2 MAIN RESULTS: ROBUSTNESS TO DIVERSE POLICY PERTURBATIONS

• SPrPO generally outperforms the baselines in success rate and retention across perturbations, although all methods can collapse at extreme strengths.Figure 3 shows trends across increasing perturbation magnitudes; Table 1 reports success rate and retention, and WebShop INT4 success falls to single digits.Source: Table 1 · p. 8, Figure 3 · p. 8, §6.2 · p. 9

6.3 REWARD LANDSCAPE VISUALIZATION

• Reward landscapes show sharp performance cliffs for Vanilla and flatter surfaces for SPrPO on both benchmarks.The visualization samples two fixed multiplicative Gaussian perturbation directions and plots success rate over their magnitudes.Source: Figure 4 · p. 9, §6.3 · p. 9

6.4 ABLATION AND HYPERPARAMETER STUDY

• At equal perturbation magnitude, SAM-style perturbations collapse task performance and isotropic Gaussian noise substantially degrades it, while reducing noise on sensitive channels preserves more original performance.Reducing gradient-norm scaling lowers clean success by 7.87 percentage points; scaling otherwise increases noise strength as training progresses.Source: §6.4 · p. 9
• A small ρ gives too little perturbation exposure, an overly large ρ corrupts training, and α values that are too small or too large reduce robustness gains.Small α leaves sensitive channels over-perturbed, whereas larger α can make them barely perturbed during training.Source: §6.4 · p. 9

7 CONCLUSION AND FUTURE WORK

• The paper concludes that SPrPO improves robustness across diverse policy perturbations while theoretically characterizing policy improvement under perturbations.Appendix A contains complete proofs of the theoretical results.Source: §7 · p. 10, REPRODUCIBILITY STATEMENT · p. 11
• Further analysis could consider different model structures, agent tasks, and perturbation settings.Source: §7 · p. 10
• The authors report using generative AI tools for algorithm implementation assistance and language polishing, while taking responsibility for the paper's content.Source: AI USE STATEMENT · p. 11

A.3 PROOF OF THEOREM 2

• Every limit point of the update sequence is (δ, ϵ, β̄)-probabilistic perturbation robust under νδ.The proof establishes this after showing nondecreasing bounded return and deriving bounds for the conditional perturbation distribution.Source: §A.3 · pp. 16–17

B ADDITIONAL RELATED WORK

• Related robust-RL work includes adversarial worst-case and stochastic expected-performance perturbations; this paper adopts the stochastic approach for long-horizon LLM agents.Source: B ADDITIONAL RELATED WORK · p. 17
• Prior LLM perturbation studies cover single-turn tasks and reasoning, whereas this work focuses on long-horizon interactions where perturbations can alter later observations.The paper motivates explicit attention to optimization stability and careful control of injected perturbations in this setting.Source: B ADDITIONAL RELATED WORK · p. 17

C.1 ENVIRONMENT DETAILS

• ALFWorld uses household tasks and WebShop shopping instructions; training pools and evaluations are 3,553 and 274 tasks for ALFWorld, and 11,587 and 500 for WebShop.Episodes are capped at 50 steps for ALFWorld and 15 for WebShop.Source: §C.1 · p. 18
• At each step, agents receive task instructions, interaction history, environment observations, and admissible actions, then generate structured environment actions.Source: §C.1 · p. 18
• Training and evaluation disable explicit reasoning traces; the paper also uses sparse trajectory-level rewards and a −0.1 format penalty for malformed actions.The no-thinking choice is motivated by optimization and computational considerations and reduces training time from roughly seven days to two.Source: §C.1 · p. 18

C.2 TRAINING DETAILS

• All methods fine-tune Qwen2.5-1.5B-Instruct with LoRA on attention and FFN projections, use GiGPO for advantage estimation, and optimize the same PPO clipped objective.Source: §C.2 · p. 18
• Gaussian, SAM, and SPrPO apply channel-wise multiplicative perturbations to the intermediate FFN representation before its down-projection.Parameter-space noise and dropout during training led to collapse in the authors' experiments.Source: §C.2 · p. 18
• Vanilla uses unperturbed GiGPO; the Gaussian baseline uses isotropic noise, while SAM constructs an adversarial channel perturbation from the PPO-loss gradient.SAM at ρ = 0.1 caused collapse, so the experiments used ρ = 5 × 10^-3 for reasonable performance.Source: §C.2 · p. 19
• SPrPO estimates channel-wise sensitivity from the training-surrogate gradient and scales perturbations upward with ρ and downward with sensitivity raised to α.Sensitivity statistics are estimated with an additional forward/backward pass that replays the current batch after a policy update.Source: §C.2 · pp. 19–20
• SPrPO adds no environment rollouts; its main training overhead is one extra forward/backward pass per policy update for channel-sensitivity estimation.The paper reports training-cost measurements averaged in seconds per step, excluding periodic validation and checkpoint saving.Source: §C.2 · p. 20, Table 4 · p. 20

C.3 EVALUATION DETAILS

• Evaluations apply each perturbation only at test time to a fixed trained policy.WebShop uses all 500 goals; ALFWorld uses 140 ID and 134 OOD validation gamefiles.Source: §C.3 · p. 20
• Gaussian noise scales from σ = 0.02 to 0.20, and channel-dropout probabilities from p = 0.01 to 0.05, on both benchmarks.Gaussian perturbation is multiplicative on FFN activations; dropout independently removes FFN channels.Source: §C.3 · p. 20
• Structured pruning removes the lowest-scoring diagonal-Fisher FFN channels, sweeping r = 0.05–0.20 on both benchmarks.Scores use a fixed calibration set of 128 interaction sequences; compared methods share the set, while scores are computed separately for each model.Source: §C.3 · p. 21
• SparseGPT unstructured pruning removes weights from FFN projections while keeping attention weights dense, with benchmark-specific sparsity sweeps.The sweep is r = 0.10, 0.20, 0.30, 0.40 on WebShop and r = 0.10, 0.30, 0.50, 0.70 on ALFWorld.Source: §C.3 · p. 21
• Round-to-nearest quantization covers all transformer linear weights at INT8, INT6, and INT4, using groups of 128 weights per scale.Each weight is scaled, rounded to the nearest integer code, and clipped to the representable range.Source: §C.3 · p. 21

D MORE EXPERIMENT RESULTS

• Reported robustness results are mean ± standard deviation over three seeds, with boldface marking the highest mean, including ties.The table description does not provide numerical comparisons or identify a winner.Source: D MORE EXPERIMENT RESULTS · p. 21
• The robustness tables cover ALFWorld overall success, ALFWorld ID/OOD success, and WebShop success rates.The supplied captions identify these table scopes but do not provide cell values.Source: Table 5 · p. 22, Table 6 · p. 22, Table 7 · p. 23
• The section also includes WebShop hyperparameter results and component ablations, both reported as success rates.Vanilla (ρ = 0) and full SPrPO (ρ = 0.2, α = 1) reuse Table 1 results; other configurations use separate ablation evaluation batches.Source: Table 8 · p. 23, D MORE EXPERIMENT RESULTS · p. 23, Table 9 · p. 24

E QUALITATIVE EXAMPLES UNDER POLICY PERTURBATIONS

• Under perturbation, Vanilla’s malformed action tags accompany failure, while SPrPO retains valid tags and completes the ALFWorld soapbar task in 9 steps.The passage cautions that repairing Vanilla’s first admissible command’s tags does not establish that the episode would succeed.Source: E QUALITATIVE EXAMPLES UNDER POLICY… · p. 25
• In the CD task, Vanilla undoes prior progress and fails at the 50-step limit with only one CD in the safe; SPrPO delivers both in 9 steps.The examples describe executable but unproductive Vanilla actions, including removing CD 2 from the safe and relocating CD 1.Source: E QUALITATIVE EXAMPLES UNDER POLICY… · p. 26
• On the identical Step-5 input, SPrPO proceeds to purchase and succeeds in 5 steps with task score 1.0, while Vanilla abandons the product, repeatedly clicks search-query text, and fails at the 15-step limit without purchase (score 0.0).Source: E QUALITATIVE EXAMPLES UNDER POLICY… · p. 27
• Vanilla repeatedly clicks the already selected size through the 15-step limit and does not purchase; SPrPO purchases from the same decision input, succeeding in 5 steps with task score 1.0.Source: E QUALITATIVE EXAMPLES UNDER POLICY… · p. 28