Source-linked AI summary
Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments
Yuxin Chen, Xiaodong Cai, Junfeng Fang, Zhuowen Han, Yu Wang, Yaorui Shi, Yi Zhang, Qi Gu, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua
TL;DR
LLM agents often degrade in realistic, stochastic environments because training typically assumes curated instructions and stable tool interactions. NoisyAgent trains agents with user and tool perturbations, improving robustness in noisy environments and performance on standard benchmarks.
Problem
LLM agents can perform strongly on idealized benchmarks yet degrade in complex, dynamic environments because training relies on curated instructions and stable conditions.
Method
NoisyAgent incorporates user and tool perturbations into agentic training, applying them selectively and increasing their difficulty through a curriculum.
Results
NoisyAgent consistently improves agent performance and robustness under noisy, dynamic environments while also yielding gains on standard idealized benchmarks.
Takeaways & Limitations
Controlled exposure to environmental noise supports more robust and generalizable reasoning and decision-making in LLM agents.
Takeaways & Limitations
The framework models representative user-side and tool-side failure patterns but does not exhaustively cover more complex, compositional, or dynamically evolving uncertainty.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have facilitated the widespread deployment of LLMs as interactive agents capable of reasoning, planning, and tool use. Despite strong performance on existing benchmarks, such agents often exhibit notable degradation when deployed in real-world settings, where environments are inherently stochastic and imperfect. We argue that this discrepancy arises from a fundamental mismatch between idealized training settings and real-world interaction dynamics, where current paradigms rely on carefully curated task instructions and stable, well-controlled environments. To address this gap, we propose NoisyAgent, an agentic training framework that explicitly incorporates environmental imperfections into the agent learning process. We identify two major sources of interaction noise in real-world scenarios: user noise, which captures ambiguity and variability in user interaction, and tool noise, which reflects failures and anomalies in tool execution. We introduce such perturbations into the training pipeline by modifying user interaction patterns and simulating tool execution results within the training environment. To stabilize training while encouraging agents to handle increasingly challenging imperfections, noise is applied to only a subset of rollouts and progressively increased in difficulty as the model adapts to the current noise level. Extensive experiments demonstrate that our approach consistently improves agent robustness under noisy and dynamic environments. Our analysis reveals that training under noise conditions also yields performance gains on idealized benchmarks, suggesting that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making behaviors. Our findings highlight the importance of modeling interaction imperfections for bridging the gap between agent training and real-world deployment.
1 Introduction
NoisyAgent addresses the gap between idealized agent training and stochastic, imperfect real-world deployment by incorporating user and tool noise into training. The framework improves robustness on noisy environments while also yielding gains on idealized benchmarks.
- Motivation: LLM agents perform strongly on benchmarks but degrade in complex, dynamic, and realistic settings.The introduction attributes this discrepancy to differences between benchmark conditions and real-world interaction dynamics.
- Motivation: Existing agent-training paradigms rely on curated instructions and stable environments, unlike inherently stochastic and imperfect real-world interactions.Users may vary in interaction style, while deployment environments can contain uncertainty and imperfections.
- Method: The framework trains agents with structured perturbations and exposure to diverse imperfections rather than relying on post-hoc adaptation.Its training design combines clean and noisy trajectories and introduces noise progressively according to the agent’s adaptation, as described in the overview.
- Method: NoisyAgent incorporates environmental imperfections into agentic reinforcement learning through user noise and tool noise.User noise models ambiguity and variability in user interactions; tool noise simulates execution anomalies from external tools.
- Results: NoisyAgent improves performance on noise-augmented benchmarks and consistently gains on standard idealized benchmarks.These results indicate enhanced robustness under imperfect, dynamic environments alongside benefits in conventional evaluation settings.
2 Preliminary
Agentic training tasks are modeled as POMDPs in which agents act through user interactions and tool calls, receiving corresponding feedback and tool results. RLVR with GRPO optimizes these trajectories, while automated environment construction improves scalability but often assumes idealized user and tool behavior.
- Agentic training formalizes each task as a POMDP with states containing environment state, interaction history, and task prompt.The supplied formulation defines the agent’s state as st = (senv, ht, q).
- The agent selects user-interaction or tool-calling actions from Auser ∪ Atool and receives user feedback or tool results from Ouser ∪ Otool.Environment transitions map the current state and action to a next state and observation.
- RLVR uses a verifier to assign trajectory-level scalar rewards, while GRPO computes rollout advantages relative to group reward statistics.GRPO is presented as an extension of PPO using normalized group rewards, with Âi = (ri − µ)/σ.
- Automated environment scaling synthesizes executable domains from domain specifications, tool sets, database schemas, task queries, and interaction patterns.The pipeline constructs a structured domain graph D and uses it to generate user-side objectives and tool-grounded environments.
- These scalable environments assume clear, helpful user interactions and stable tool behavior, producing idealized training settings that mismatch inherently imperfect real-world deployment.The construction process is described as reliable and verifiable, but its assumptions limit realism.
3 Methodology
NoisyAgent bridges idealized training and noisy deployment by injecting user- and tool-side imperfections into agentic environments and adapting noise difficulty during training. Its hybrid strategy combines clean and perturbed rollouts to improve robustness while preserving stable optimization.
- Automatic Noise Injection: NoisyAgent introduces an automatic noise injection pipeline that incorporates user- and tool-side perturbations into synthesized agentic training environments.User-side noise models ambiguity and variability in interaction patterns, while tool-side noise models failures and anomalies in external execution.
- Automatic Noise Injection: User-side injection modifies pre-task interaction patterns to simulate ambiguous, inconsistent, and redundant user behavior while preserving the underlying task objective.The perturbations increase interaction difficulty and stochasticity without invalidating task solvability.
- Automatic Noise Injection: Tool-side injection randomly perturbs a subset of rollout tool results to simulate failures, incomplete outputs, misleading information, and redundant details while maintaining executable interactions.The perturbation process is applied during agent rollouts and models imperfect tool behavior.
- Hybrid Training: Hybrid training combines idealized and perturbed environments, partitioning parallel rollouts into noisy and clean groups and computing group-wise advantages before optimizing jointly.Separate reward normalization prevents either group from dominating optimization and stabilizes learning under heterogeneous interaction conditions.
- Noise Scheduling: Progressive noise scheduling begins with fully idealized environments and increases noise as the model adapts, controlling both the perturbed-rollout proportion ρ = Nnoise/N and noise difficulty.Difficulty reflects tool-perturbation frequency and the severity of user-side interaction anomalies, with adjustments based on the clean–perturbed performance gap.
4 Experiments
Experiments show that NoisyAgent improves robustness across noisy environments and also delivers consistent gains on idealized benchmarks. Ablations, training dynamics, and interaction analyses attribute these gains to structured noise injection and curriculum-based adaptation.
- Noisy evaluation: NoisyAgent consistently achieves the best AgentNoiseBench performance across domains and model scales, surpassing GSPO and DAPO in Avg@4 and Pass@4.Standard reinforcement-learning methods show substantially diminished gains and notable relative degradation under noise.
- Noisy evaluation: Structured perturbations help agents handle ambiguous instructions and imperfect tool feedback, recover from intermediate failures, and maintain progress toward task completion.The training framework targets both user ambiguity and tool-execution imperfections.
- Ideal evaluation: NoisyAgent consistently improves performance across τ 2-Bench and VitaBench, outperforming all baselines across domains and metrics in idealized settings.The results indicate that noise-aware training does not harm clean-environment performance.
- Ablation study: Ablations remove controlled injection, noise scheduling, noise, or training to isolate the contributions of hybrid trajectories, curriculum complexity, perturbations, and optimization.Controlled injection mixes clean and noisy rollouts, while scheduling varies perturbation complexity during training.
- Training dynamics: NoisyAgent initially tracks the clean-trained baseline, then adapts as increasingly challenging perturbations are introduced and achieves a larger performance improvement, especially under noisy evaluation.Moderate noise can initially cause slight degradation because perturbed trajectories are harder, whereas the clean-trained baseline improves only moderately.
- Interaction pattern: Under noisy conditions, NoisyAgent reduces tool usage from 13.9 to 11.4 calls per episode (18%), while ideal-setting methods use similar totals of 6.7–7.4 calls.GSPO produces only marginal tool-usage change in the noisy setting.
5 Related Work
Related work frames LLM agents as tool-using, planning-capable systems that increasingly interact with dynamic environments. It also highlights brittleness in hand-crafted agent pipelines and growing concerns about robustness under environmental shifts and user-side interaction perturbations.
- Agent Development: LLM agents have advanced from passive text generation to tool use, multi-step planning, and interaction with dynamic environments.This evolution accompanies improvements in reasoning and instruction-following capabilities.
- Agent Development: Early agent approaches manually specify reasoning–action patterns, tool schemas, and memory mechanisms on top of frozen models.The passage characterizes these prompt-level designs as effective but brittle.
- Robustness: Agent robustness has become a critical concern because performance degrades substantially under distributional shifts in environment dynamics.Prior work also examines how prompt, clarification, and multi-turn dialogue perturbations affect agent behavior.
6 Limitation
The framework’s noise modeling is limited to two representative real-world interaction sources—user-side and tool-side perturbations—and a set of common practical failure patterns. Further aspects remain for future exploration.
- Scope of the limitation: The study focuses on whether incorporating real-world interaction noise improves agent-policy robustness.This focus defines the framework’s primary investigative goal.
- Scope of the limitation: The framework models two representative noise sources: user-side and tool-side perturbations.These sources are selected as representative forms of interaction noise.
- Scope of the limitation: Its noise design covers a set of common failure patterns observed in practice, leaving additional aspects for future work.The passage characterizes the modeled patterns as broad but not exhaustive.
7 Conclusion … A.3 Rollout and Generation Configuration
NoisyAgent addresses the robustness gap between idealized training and real-world deployment by modeling instruction and tool imperfections with adaptive noise-aware training. The paper reports improved performance in noisy environments and gains on idealized benchmarks, alongside configurations for reproducing the experiments.
- 7 Conclusion: NoisyAgent targets limited robustness caused by omitting environmental imperfections during agent training.The framework explicitly incorporates stochasticity and imperfections into the learning process by modeling instruction noise and tool noise.
- 7 Conclusion: The adaptive strategy combines clean and perturbed rollouts while progressively increasing noise difficulty according to model robustness.This design is intended to support stable optimization as agents adapt to increasingly challenging noise levels.
- A Training Configuration Details: The training configuration details are provided to support reproduction of the experiments.The appendix states that this section contains the complete training configurations.
- A.1 Model and Infrastructure: Qwen3-8B and Qwen3-32B serve as backbone models, trained in BF16 precision with vLLM v0.8.5 for rollout generation.All models use RoPE with θ = 106 and RMSNorm with ϵ = 10−6.
- A.2 Optimization Hyperparameters: Table 5 summarizes the optimization hyperparameters shared across all methods.The supplied passage identifies the table as the source for these shared optimization settings but does not provide their individual values.
- A.3 Rollout and Generation Configuration: Table 6 details the rollout generation settings used in the experiments.The supplied passage identifies the table as the source for rollout-generation configuration but does not provide its individual settings.
A.4 Method-Specific Configurations
The paper compares method-specific loss configurations and uses distinct normalization and clipping choices across GRPO, DAPO, GSPO, and the proposed method. The proposed method inherits GSPO’s loss configuration and adds a noise-aware curriculum.
- Method-specific configurations: Table 7 compares loss configurations across the different training methods.It is presented as the method-specific loss-configuration comparison.
- Method-specific configurations: GRPO follows its original formulation with a fixed clip range and batch-level advantage normalization.These settings are retained from the original GRPO formulation.
- Method-specific configurations: DAPO and GSPO use an asymmetric clip range [0.2, 0.28], dynamic temperature scaling, and buffer-level advantage normalization.The configurations follow their respective original implementations.
- Method-specific configurations: The proposed method inherits the GSPO loss configuration and adds a noise-aware curriculum.The curriculum is applied on top of the inherited GSPO configuration.
A.5 Noise-Aware Training Configuration · A.6 Training Data Configuration
The training configuration combines controlled noise injection with adaptive noise scheduling, while training data spans τ 2-Bench and VitaBench domains. Baselines use clean environments, whereas NoisyAgent progressively introduces noisy trajectories and filters uninformative rollout groups.
- A.5 Noise-Aware Training Configuration: Noise-aware training consists of controlled injection and noise scheduling, corresponding to the framework’s two training components.Controlled injection regulates exposure to perturbed rollouts, while scheduling governs difficulty and progression.
- A.5 Noise-Aware Training Configuration: Controlled injection sets ρ = Nnoise/N, with each task’s rollouts split between noisy and clean environments.For each task, N rollouts are generated in parallel, including Nnoise noisy rollouts and N −Nnoise clean rollouts.
- A.5 Noise-Aware Training Configuration: The maximum noise proportion is 50% of total rollouts, with training starting at ρ = 0 and increasing by a fixed step s.The configuration constrains the noise scale to ρ ≤0.5.
- A.5 Noise-Aware Training Configuration: Noise scheduling increases difficulty and noise scale when the clean-versus-perturbed performance gap ∆ falls below threshold θ.Difficulty reflects tool-side perturbation frequency and user-side anomaly severity.
- A.5 Noise-Aware Training Configuration: The framework models user-side noise as ambiguous, inconsistent, redundant, or out-of-scope interactions, and tool-side noise as failures, incomplete, misleading, or redundant outputs.These two axes define the noise types used in training.
- A.6 Training Data Configuration: Multi-domain training combines τ 2-Bench Airline, Retail, and Telecom tasks with medium-to-low pass-rate difficulty and VitaBench Delivery, In-Store, and OTA tasks.The setup trains on tasks from two benchmarks simultaneously.
- A.6 Training Data Configuration: GRPO, DAPO, and GSPO baselines train exclusively in clean environments, while NoisyAgent progressively introduces noisy trajectories through controlled injection.Groups with all-pass or all-fail rewards are filtered to preserve meaningful gradient signal.
A.7 Evaluation Protocol · A.8 Computational Resources · B Discussion
Evaluation uses repeated multi-run assessment in ideal and noisy settings, with GPT-4.1 simulating users and Claude-Sonnet-4.5 evaluating outcomes. The appendix also documents a noisy retail failure case and the substantial GPU resources required for training.
- A.7 Evaluation Protocol: GPT-4.1 simulates users and Claude-Sonnet-4.5 evaluates agents throughout training and final evaluation.Training evaluations occur every 5 steps with 4 rollouts per task in both ideal and noisy settings.
- A.7 Evaluation Protocol: Final experiments are repeated 4 times and reported with Avg@4 and Pass@4.Avg@4 is the average score across 4 runs, while Pass@4 is the fraction of tasks solved in at least one run.
- A.7 Evaluation Protocol: The base model fails with reward = 0.0 after identifying the items, diverting to desk-lamp recommendations instead of calling the return API.The conversation ends without processing either return.
- A.7 Evaluation Protocol: In the noisy retail case, both agents complete information gathering, but the base model fails after API noise while our model stays on task.This case study contrasts final-action execution under tool noise.
- A.8 Computational Resources: Qwen3-8B training uses 32 NVIDIA H800 GPUs, whereas Qwen3-32B training uses 64 NVIDIA H800 GPUs.Each 100-step training run takes approximately 3–5 days, depending on model scale and domain complexity.
B.1 Case Study … B.4 Safeguards
The case study shows that noise mainly disrupts agents’ transition from understanding to action, while NoisyAgent maintains task focus and executes required operations. The paper also describes ethical, broader-impact, and safeguard considerations for responsible deployment.
- B.1 Case Study: Under intermittent API failures and corrupted fields, both models gather user, order, and item information, but only NoisyAgent completes both required returns.The task involves returning a mechanical keyboard and gaming mouse from separate orders with refunds to the original payment method.
- B.1 Case Study: The base model understands user intent but loses focus and fails to invoke critical return APIs, whereas NoisyAgent executes the necessary actions without deviation.This illustrates a behavioral divergence during the execution phase rather than the information-gathering phase.
- B.1 Case Study: 78% of 23 tasks where NoisyAgent succeeds but the base model fails show correct information gathering followed by failure to execute the critical action.The findings suggest noise primarily affects the transition from understanding to action.
- B.1 Case Study: Curriculum training improves goal-directed behavior and reliable action execution when intermediate observations are corrupted or inconsistent.The result summarizes the observed benefit of training under progressively challenging noisy conditions.
- B.2 Code of Ethics: The work states that it complies with the NeurIPS Code of Ethics and uses synthetic or controlled data rather than real user data or sensitive personal information.It also describes strict data construction and validation procedures, although the supplied passage is truncated.
- B.3 Broader Impacts: The approach could support more reliable and adaptive agents in customer service, recommendation, and task automation, but more capable agents may also introduce misuse risks.The broader-impact discussion connects robustness improvements to real-world deployment while acknowledging potential risks.
- B.4 Safeguards: Safeguards use controlled synthesis for environments, interaction data, and noise perturbations, with validation and refinement to promote correctness, consistency, and avoidance of harmful content.The passage also states that the framework does not rely on real user data, though the supplied text is truncated.