PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments

Yuxin Chen, Xiaodong Cai, Junfeng Fang, Zhuowen Han, Yu Wang, Yaorui Shi, Yi Zhang, Qi Gu, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua

arXiv:2605.27209v1cs.AI

TL;DR

LLM agents can degrade in complex, dynamic environments despite strong benchmark performance, amid a mismatch between idealized training and stochastic real-world interactions. NoisyAgent trains agents with user and tool perturbations, and the paper reports improved performance in noisy environments as well as gains on standard benchmarks.

  • Problem

    Agent training commonly assumes curated instructions and stable environments, while real-world interactions are stochastic and imperfect, a mismatch relevant to robustness and performance in complex, dynamic settings.

    Source: §1 · pp. 1–2

  • Method

    NoisyAgent incorporates user-instruction and tool-execution perturbations into agent RL training on a subset of rollouts, progressively increasing noise difficulty.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    NoisyAgent consistently improves performance in noisy, dynamic environments and also yields gains on standard, idealized benchmarks.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The findings highlight the importance of aligning agent training with real-world interaction characteristics.

    Source: §7 · p. 10

    The framework covers two representative noise sources and common failure patterns, but not all uncertainty, including more complex, compositional, and dynamically evolving noise.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have facilitated the widespread deployment of LLMs as interactive agents capable of reasoning, planning, and tool use. Despite strong performance on existing benchmarks, such agents often exhibit notable degradation when deployed in real-world settings, where environments are inherently stochastic and imperfect. We argue that this discrepancy arises from a fundamental mismatch between idealized training settings and real-world interaction dynamics, where current paradigms rely on carefully curated task instructions and stable, well-controlled environments. To address this gap, we propose NoisyAgent, an agentic training framework that explicitly incorporates environmental imperfections into the agent learning process. We identify two major sources of interaction noise in real-world scenarios: user noise, which captures ambiguity and variability in user interaction, and tool noise, which reflects failures and anomalies in tool execution. We introduce such perturbations into the training pipeline by modifying user interaction patterns and simulating tool execution results within the training environment. To stabilize training while encouraging agents to handle increasingly challenging imperfections, noise is applied to only a subset of rollouts and progressively increased in difficulty as the model adapts to the current noise level. Extensive experiments demonstrate that our approach consistently improves agent robustness under noisy and dynamic environments. Our analysis reveals that training under noise conditions also yields performance gains on idealized benchmarks, suggesting that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making behaviors. Our findings highlight the importance of modeling interaction imperfections for bridging the gap between agent training and real-world deployment.

1 Introduction

NoisyAgent addresses the mismatch between idealized agent training and imperfect real-world interactions by training agents with user and tool noise. The introduction reports improved performance on noisy environments and standard benchmarks.

1 Introduction

  • Agents often degrade in complex, dynamic settings because training with curated instructions and stable environments differs from stochastic, imperfect real-world interactions.The introduction identifies variability in user behavior and noisy, incomplete, or failed tool outputs as sources of this mismatch.Source: §1 · pp. 1–2
  • NoisyAgent perturbs user instructions and tool results on a subset of rollouts, then adaptively increases noise difficulty and ratio as robustness improves.Robustness is measured by the performance gap between idealized and perturbed environments on the same tasks.Source: §1 · p. 2
  • The paper argues that exposure to uncertainty can encourage recovery from errors, ambiguity resolution, and adaptation to unexpected outcomes, supporting more generalizable reasoning and decisions.It frames appropriately designed noise as controlled instability and implicit difficulty augmentation, while warning that naive noise injection can destabilize training.Source: §1 · pp. 2–3
  • NoisyAgent improves performance on benchmarks with real-world noise and also yields gains on standard, idealized benchmarks.The paper presents these findings as evidence of enhanced robustness in imperfect and dynamic environments.Source: §1 · p. 3

2 Preliminary

The preliminary section formalizes agent training as sequential decision-making and describes a scalable environment-construction pipeline whose clear-user and stable-tool assumptions leave a gap with imperfect real-world interaction.

2.1 Agentic Reinforcement Learning

  • Agentic tasks are modeled as POMDPs in which a policy chooses user or tool actions from observations and seeks to maximize cumulative trajectory reward.Observations include user feedback and tool results, and actions change the environment and produce subsequent observations.Source: §2.1 · p. 3
  • RLVR scores final states or trajectories with a verifier, while GRPO normalizes each rollout’s reward against its sampled group to compute advantages [28] [29] [30].GRPO extends PPO [31].Source: §2.1 · p. 3

2.2 Scaling Environment for Agentic Training

  • A domain graph enables automated environment scaling by supplying domain-specific tools and a unified database schema for executable training environments [34].Each environment couples a user-side task and interaction pattern with tool-side environment dynamics.Source: §2.2 · p. 3
  • User-side construction samples tool chains to create task queries and interaction patterns, while tool-side construction implements APIs, databases, valid transitions, and verifiable execution paths.Tool chains are instantiated as reference executions, and the tool set is expanded along the domain graph while preserving correctness and verifiability.Source: §2.2 · p. 4
  • Because this pipeline assumes clear, helpful user interactions and stable tool behavior, its idealized environments mismatch real-world settings, where interactions are imperfect.Source: §2.2 · p. 4

3 Methodology

NoisyAgent trains agents with user- and tool-side imperfections, then adapts noise difficulty to improve robustness while preserving training stability. Its methodology combines automatic noise injection with a curriculum that mixes clean and perturbed rollouts.

3.1 Automatic Noise Injection

  • A stochastic generator perturbs agent–environment interactions to model user-side ambiguity and variability and tool-side failures and anomalies.The pipeline is designed to incorporate these imperfections into synthesized agent-training environments.Source: §3.1 · p. 4
  • User-side noise makes instructions ambiguous, inconsistent, or redundant while preserving the underlying task objective and solvability.Perturbations modify user interaction patterns before the task starts, increasing interaction difficulty and stochasticity without invalidating the task.Source: §3.1 · p. 4
  • Tool-side noise perturbs some rollout results with failures, incomplete outputs, misleading information, or redundant details while keeping interactions executable.The perturbations simulate imperfect tool behavior during agent rollouts.Source: §3.1 · pp. 4–5

3.2 Adaptive Noise Training

  • Hybrid GRPO training combines clean and noisy rollouts, computes advantages separately for each group, and optimizes over their union to prevent either group dominating.Group-wise reward normalization stabilizes optimization across heterogeneous interaction conditions.Source: §3.2 · p. 5
  • Noise scheduling begins in clean environments and increases perturbed-rollout proportion and noise difficulty when the clean-to-noisy performance gap falls below a threshold.The gap measures how much a noise type degrades task success; difficulty reflects tool-perturbation frequency and severity of user-side anomalies.Source: §3.2 · p. 5

4 Experiments

NoisyAgent improves performance in both noisy and idealized evaluations. The experiments also find that controlled, progressively challenging noise training supports stable learning and more efficient, clearer interaction.

4.1 Experiment Settings

  • Experiments compare Qwen3-8B and Qwen3-32B NoisyAgent with GRPO, DAPO, and GSPO on noisy AgentNoiseBench and standard τ 2-Bench and VitaBench, reporting four-run Avg@4 and Pass@4.AgentNoiseBench evaluation uses its τ 2 and Vita subsets; NoisyAgent is based on GSPO, with noisy trajectories capped at 50% of rollouts.Source: §4.1 · p. 6

4.2 Main Results

  • NoisyAgent achieves the best AgentNoiseBench results across domains and model scales in Avg@4 and Pass@4, and outperforms all baselines across domains and metrics on τ 2-Bench and VitaBench.The ideal-setting results indicate that noise training does not harm clean-benchmark performance.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Analysis

  • Removing any framework component degrades performance; applying noise to all rollouts causes the largest drop, while progressive noise scheduling improves learning effectiveness and stability.The ablations cover controlled injection, scheduling, noise, and RL training on Delivery with Qwen3-8B.Source: §4.3 · p. 7, Table 3 · p. 7
  • NoisyAgent’s early performance is comparable to the no-noise baseline, but its gains grow as noise difficulty increases, with the widening advantage most pronounced under noisy evaluation.The initial moderate noise can slightly degrade performance before the model adapts; Figure 2 shows training dynamics on Vita-Bench Delivery with Qwen3-8B.Source: §4.3 · pp. 7–8, Figure 2 · p. 8
  • Under noise, NoisyAgent reduces tool calls from 13.9 to 11.4 per episode (18%) and increases output tokens from 2,014 to 4,248 (2.1×), while ideal-setting tool use remains similar across methods.The reported ideal-setting tool usage is 6.7–7.4 calls; the authors interpret the noisy-setting pattern as fewer redundant interactions alongside more explicit responses.Source: §4.3 · p. 8

5 Related Work

LLM-agent research has advanced from hand-crafted pipelines to reinforcement learning and broader environment design, but largely under idealized conditions. Robustness research documents user and tool noise, while learning agents to withstand realistic noise remains underexplored.

5.1 LLM as Agent

5.2 Robustness of Agent

  • User interactions can be noisy and evolving, while incomplete, outdated, or erroneous tool outputs can cascade into downstream decision failures [69] [71] [72] [73] [74] [75] [76] [77] [78] [79] [82].Prior studies examine prompt perturbations, clarifications, and multi-turn dialogue, alongside errors in external tool execution.Source: §5.2 · p. 9
  • Robustness benchmarks characterize noise and show degradation under realistic perturbations, but existing work focuses primarily on evaluation rather than learning robust behaviors [35] [83] [85].AgentNoiseBench provides a unified taxonomy of user-side and tool-side noise with controllable perturbations.Source: §5.2 · p. 9

6 Limitation

The framework models representative user- and tool-side noise but does not cover all real-world uncertainty. Evaluation is concentrated in synthesized, controlled settings, leaving broader real-world and in-domain benchmarking for future work.

6 Limitation

  • NoisyAgent models representative user- and tool-side failure patterns, but does not exhaustively cover uncertainty that may be more complex, compositional, or dynamically evolving.Future work could extend the framework to richer and more diverse interaction patterns.Source: §6 · p. 9
  • Experiments primarily use synthesized environments that approximate real-world interaction dynamics, rather than extensive real-world or in-domain benchmarks.The paper identifies broader real-world and in-domain benchmarking as future work.Source: §6 · p. 9
  • The framework is described as applicable to any agentic environment through structured noise augmentation, but high training costs and systematic out-of-distribution robustness evaluation motivated a focus on controlled settings.Source: §6 · p. 9

7 Conclusion … A.3 Rollout and Generation Configuration

NoisyAgent trains LLM agents with realistic interaction imperfections, improving robustness in noisy environments and performance on idealized benchmarks. The appendix provides configurations for reproducing the experiments.

7 Conclusion

  • NoisyAgent models instruction and tool noise, injects them automatically, and adaptively mixes clean and perturbed rollouts while increasing noise difficulty based on robustness.The framework targets the gap between idealized training and real-world deployment.Source: §7 · pp. 9–10
  • The method consistently improves agent performance in noisy, dynamic environments and on standard idealized benchmarks.The authors suggest that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making.Source: §7 · p. 10

A Training Configuration Details

A.1 Model and Infrastructure

  • Experiments use Qwen3-8B and Qwen3-32B backbones trained in BF16 with vLLM v0.8.5; models use RoPE with θ = 106 and RMSNorm with ϵ = 10−6.Source: §A.1 · p. 16

A.2 Optimization Hyperparameters

A.3 Rollout and Generation Configuration

A.4 Method-Specific Configurations

The section compares method-specific loss configurations: GRPO uses its original formulation, DAPO and GSPO use asymmetric clipping with dynamic temperature scaling, and NoisyAgent adds its noise-aware curriculum to GSPO’s configuration.

A.4 Method-Specific Configurations

  • NoisyAgent inherits GSPO’s loss configuration and adds a noise-aware curriculum, as summarized in the method-specific comparison in Table 7.The passage does not specify additional loss-setting changes for NoisyAgent.Source: §A.4 · p. 16, Table 7 · p. 17
  • GRPO follows its original formulation with a fixed clip range and batch-level advantage normalization.Source: §A.4 · p. 16
  • DAPO and GSPO use an asymmetric clip range [0.2, 0.28], dynamic temperature scaling, and buffer-level advantage normalization.These configurations follow their respective original implementations.Source: §A.4 · p. 16

A.5 Noise-Aware Training Configuration · A.6 Training Data Configuration

The configuration combines controlled noise injection with a curriculum that raises noise as the model adapts. Training uses tasks from τ 2-Bench and VitaBench, while baseline methods train only in clean environments.

A.5 Noise-Aware Training Configuration

  • Noise is injected into a subset of each task’s parallel rollouts, with the proportion starting at ρ = 0 and capped at 50%, increasing by fixed steps when performance plateaus.For each task, Nnoise of N rollouts are noisy and the rest are clean.Source: §A.5 · p. 17
  • When the clean–perturbed performance gap ∆ falls below threshold θ, training increases noise difficulty through more frequent tool perturbations and more severe user anomalies.This progression is intended to increase interaction complexity while maintaining training stability.Source: §A.5 · p. 17
  • The configuration treats user-side noise as ambiguity, inconsistency, redundancy, or out-of-scope input, and tool-side noise as failures, incompleteness, misleading outputs, or redundancy.Controlled injection and noise scheduling are the two components of the strategy.Source: §A.5 · p. 17

A.6 Training Data Configuration

  • Multi-domain training draws from τ 2-Bench’s Airline, Retail, and Telecom tasks filtered to medium-to-low pass-rate difficulty, and VitaBench’s Delivery, In-Store, and OTA tasks.Source: §A.6 · p. 17
  • GRPO, DAPO, and GSPO baselines train exclusively in clean environments, whereas NoisyAgent progressively introduces noisy trajectories through controlled injection.Source: §A.6 · p. 17
  • Training filters out rollout groups in which all rewards are identical, whether all-pass or all-fail, to preserve meaningful gradient signal.Source: §A.6 · p. 17

A.7 Evaluation Protocol · A.8 Computational Resources · B Discussion

Evaluation uses GPT-4.1 as the user simulator and Claude-Sonnet-4.5 as the evaluator, with repeated final runs; training resources scale from 32 to 64 NVIDIA H800 GPUs by model size.

A.7 Evaluation Protocol

  • Final experiments are repeated 4 times, with Avg@4 reporting average score and Pass@4 reporting the fraction of tasks solved in at least one run.The final evaluation reports both metrics across the four runs.Source: §A.7 · p. 17
  • GPT-4.1 serves as the user simulator and Claude-Sonnet-4.5 as the evaluator throughout training and final evaluation.Source: §A.7 · p. 17
  • Training evaluation occurs every 5 steps with 4 rollouts per task, assessing both ideal noise-free and noisy settings.Source: §A.7 · p. 17
  • In the τ 2-Bench Retail noisy case study, the base model fails the final action after API noise, while NoisyAgent stays on task.Both agents complete the information-gathering phase correctly.Source: Figure 3 · p. 18

A.8 Computational Resources

  • Qwen3-8B training uses 32 NVIDIA H800 GPUs, while Qwen3-32B training uses 64 NVIDIA H800 GPUs.Source: §A.8 · p. 18
  • A 100-step training run takes approximately 3–5 days, depending on model scale and domain complexity.Source: §A.8 · p. 18

B.1 Case Study … B.4 Safeguards

The case study shows that NoisyAgent more reliably turns correct understanding into task-directed action under noisy conditions. The accompanying ethics, impact, and safeguards sections describe synthetic-data practices, potential deployment risks, and controls for responsible evaluation.

B.1 Case Study

  • Under intermittent API failures and corrupted fields, NoisyAgent completes both gaming-item returns, while the base model abandons the task without invoking the return API.Both models gather the necessary information, but diverge during execution.Source: §B.1 · p. 18

B.2 Code of Ethics

  • The work reports no human subjects or real user data, and describes synthetic data, controlled pipelines, and review for consistency, correctness, and safety.Experiments use controlled simulations and do not directly interact with real users or external systems.Source: §B.2 · p. 19

B.3 Broader Impacts

  • Training agents on diverse interaction uncertainties may support more reliable and adaptive systems in customer service, recommendation, and task automation.The work frames these as potential implications for real-world deployment.Source: §B.3 · p. 19
  • Robust agents could be misused for manipulation or exploitation, and without safeguards may propagate biases or make incorrect decisions on ambiguous inputs.The authors encourage further study of fairness, safety, and alignment in real-world applications.Source: §B.3 · p. 19

B.4 Safeguards

  • Safeguards include validated synthetic data, avoiding real user data, and modular noise injection to limit uncontrolled or unrealistic perturbations.The controlled synthesis process is designed to avoid harmful content and preserve evaluation validity.Source: §B.4 · p. 19

Related papers

Loading 2605.27209v1…