Source-linked AI summary
Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents
Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, Gennady Pekhimenko
TL;DR
Real-world LLM agents often have low task success rates, while prior reliability work has primarily focused on improving the agents themselves. This paper analyzes agent-environment failures and introduces targeted environment optimizations, reporting higher success rates across benchmarks without modifying the agent or underlying LLM.
Problem
Practical LLM agents have low success rates on real-world tasks, and prior work has primarily focused on improving the agent or underlying LLM rather than its environment.
Source: §1 · p. 1Method
The authors analyze 142 failed tasks and 3,656 interaction turns, classify six failure modes, and design Aegis environment optimizations to address them.
Source: §1 · p. 2Results
6.7-12.5% average success-rate improvements are reported across five state-of-the-art agent benchmarks without modifying the agent or underlying LLM.
Source: §1 · p. 2Takeaways & Limitations
The evaluations show that targeted environment optimizations improve agent task success rates across varied workloads and models.
Source: §8 · pp. 11–12User instruction-following failures are included in the taxonomy, but the environment optimizations do not directly address them because the user instructions are unchanged.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Large Language Models (LLMs) agents augmented with domain tools promise to autonomously execute complex tasks requiring human-level intelligence, such as customer service and digital assistance. However, their practical deployment is often limited by their low success rates under complex real-world environments. To tackle this, prior research has primarily focused on improving the agents themselves, such as developing strong agentic LLMs, while overlooking the role of the system environment in which the agent operates. In this paper, we study a complementary direction: improving agent success rates by optimizing the system environment in which the agent operates. We collect 142 agent traces (3,656 turns of agent-environment interactions) across 5 state-of-the-art agentic benchmarks. By analyzing these agent failures, we propose a taxonomy for agent-environment interaction failures that includes 6 failure modes. Guided by these findings, we design Aegis, a set of targeted environment optimizations: 1) environment observability enhancement, 2) common computation offloading, and 3) speculative agentic actions. These techniques improve agent success rates on average by 6.7-12.5%, without any modifications to the agent and underlying LLM.
1 Introduction
The paper reframes agent reliability as an environment-design problem, analyzing failure traces to build a taxonomy and develop targeted system optimizations. It reports higher task success without modifying the agent or underlying LLM.
1 Introduction
- The paper addresses low real-world agent success rates by studying system-environment optimization, a direction described as largely overlooked alongside agent and LLM improvements.The introduction notes that o3 improved customer-service success rates by only 2-3% over o1.Source: §1 · p. 1
- Aegis enhances environment observability, offloads deterministic computation, and speculatively bundles related tool calls to address identified failure modes.The bundled calls aim to reduce turn count and token consumption.Source: §1 · p. 2
- Environment optimizations also raise success rates for fine-tuned xLAM-2 8B and reduce LLM inference API costs.The reported averages are 3.3-8.8% higher success rates and 7.1-17.7% lower inference API cost.Source: §1 · p. 2
- 6.7-12.5% average success-rate improvements are reported across five state-of-the-art agent benchmarks without modifying the agent or underlying LLM.The authors also report 3.3-8.8% average gains with fine-tuned xLAM-2 8B and a 7.1-17.7% reduction in LLM inference API cost.Source: §1 · p. 2
- The analysis covers 142 failed tasks and 3,656 interaction turns, yielding a taxonomy of six agent-environment failure modes.The taxonomy groups failures into exploration, exploitation, and resource exhaustion.Source: §1 · p. 2
2 Background
LLM agents use tools to interact with environments, while benchmark correctness checks their tool-call traces and resulting environment states. The paper contrasts model-centered reliability work with a workflow for analyzing failures and optimizing environments.
2.1 Large Language Model Agents
- Agents receive tool descriptions and task instructions, then reason and optionally call tools as the environment returns responses.Benchmark correctness compares the actual trace with expected tool-call order and final environment state.Source: §2.1 · p. 3
2.2 The Accuracy Challenge of LLM Agents
- Real-world tasks challenge agents through specialized knowledge, complex environments, and large-scale data, while failures can require human takeover or task re-execution.Existing agent-for-system approaches improve the LLM but require a costly, complex pipeline of data preparation, training, and evaluation.Source: §2.2 · p. 3
- The paper's methodology analyzes workloads with a subtask abstraction, identifies recurring environment-interaction failures, and targets them with environment optimizations.The three steps are workload analysis, failure taxonomy, and environment optimizations.Source: §2.2 · pp. 3–4
3 Agentic Workload Analysis
The section establishes a subtask-based framework for analyzing agent-environment interactions, then uses recurring subtasks and transitions across workloads to identify opportunities for environment optimization.
3.1 Methodology
- Across five benchmarks, 250 tasks were split into 30-task analysis sets and 20-task evaluation sets per workload.Failure analysis and optimization design use the analysis sets; evaluation on unseen tasks tests generalizability.Source: §3.1 · p. 4
3.2 The Subtask Abstraction
- The HTN-inspired subtask abstraction represents task components and dependencies, helping localize where an agent departs from a correct solution.Complex tool calls can be split into logical operations to distinguish, for example, eligibility-judgment errors from execution errors.Source: §3.2 · p. 4
3.3 Exploration and Exploitation Subtasks
- Exploration subtasks gather new information, while exploitation subtasks act on knowledge already acquired; the distinction guides different environment optimizations.Source: §3.3 · p. 4
3.4 Workload Analysis Insights
- All workloads except file system have subtasks appearing in over 70% of tasks, often early steps that retrieve data needed later.Merged-graph node weights represent the fraction of tasks containing a subtask; Figure 4 plots node and edge weight distributions.Source: §3.4 · p. 5, Figure 4 · p. 5
- In retail, retrieving a user profile is followed by retrieving order history in every task, motivating speculative execution of common transitions.Figure 3 encodes merged subtask graphs, with node and edge thickness representing weights and color distinguishing exploration from exploitation.Source: §3.4 · p. 5, Figure 3 · p. 5
4 Failure Taxonomy
The taxonomy locates failures at the first unsuccessful subtask across benchmark traces and distinguishes information-gathering, information-use, and resource-limit failures. Exploitation failures dominate several workloads, while resource exhaustion is especially common in CRM and retail.
4 Failure Taxonomy
- Failures are assigned to the first unsuccessful subtask across 142 traces from five workloads and three models, covering 3,656 interaction turns.Transient errors that agents correct without affecting the final outcome are excluded.Source: §4 · p. 5
4.1 Exploration Failures
- Exploration failures account for 30%, 27%, and 29% of failures under GPT 4.1, GPT 4.1 mini, and o3, respectively.Medical state-space navigation failures account for 91%, 100%, and 90% under those models; state awareness errors are distinct and occur in file-system tasks.Source: §4.1 · pp. 5–6
4.2 Exploitation Failures
- Exploitation failures comprise 75% of airline, 88% of retail, 60% of file-system, and 33% of CRM failures under GPT 4.1.They occur when agents have gathered the necessary information but use it incorrectly.Source: §4.2 · p. 6
- Tool-output processing failures account for 25% of airline, 67% of retail, and 33% of CRM failed tasks under GPT 4.1.The errors include comparisons, calculations, retrieval, and sorting; these account for 33%, 25%, 25%, and 17% of tool-output failures, respectively.Source: §4.2 · p. 6
- Airline domain-rule violations account for 21%, 42%, and 50% of failures under GPT 4.1, GPT 4.1 mini, and o3, respectively.The taxonomy distinguishes invalid actions from incorrectly concluding that a valid action is forbidden.Source: §4.2 · p. 7
- User-instruction failures are most frequent in file-system tasks: 40%, 40%, and 89% of failures under GPT 4.1, GPT 4.1 mini, and o3.Environment optimizations do not directly address this category because they do not modify user instructions.Source: §4.2 · p. 7
4.3 Resource Exhaustion Failure
- Resource exhaustion accounts for 45% of retail and 83% of CRM failures; CRM is primarily token-limited, while retail is primarily turn-limited.Large tool outputs and repetitive queries contribute to exhaustion; Figure 6 breaks down tokens and turns by subtask for failed GPT 4.1 tasks.Source: §4.3 · p. 7, Figure 6 · p. 7
4.4 Subtask Failure Analysis
- The brittle subtasks identified as optimization targets are reservation-modification judgment, product filtering, file or directory finding, medical vital lookup, and CRM average-handle-time computation.Figure 7 plots subtask failure rate against frequency; subtasks high on both dimensions are treated as the highest-leverage targets.Source: §4.4 · pp. 7–8, Figure 7 · p. 8
5 System Environment Optimizations
Aegis targets agent-environment failures through environment-only changes, leaving the task, agent logic, and correctness criteria unchanged. Its optimizations improve observability, offload common computations and rules, and speculate on likely next actions.
5 System Environment Optimizations
- Aegis modifies the environment without changing the task, agent logic, correctness criteria, or existing tool set.The design adds no new tools; it augments existing ones.Source: §5 · p. 8
5.1 Enhancing Environment Observability
- Environment lookahead broadens the observation window, while explicit state-change confirmations make current location retrievable from the latest tool response.For example, a changed working directory can be confirmed along with its contents.Source: §5.1 · p. 8, Figure 8 · p. 8
5.2 Offloading Common Computational Patterns
- The environment can return precomputed results, such as products sorted by price, alongside a tool’s primary output.This is opportunistic: agents may not need the extra results, but the paper says their availability outweighs slightly longer responses.Source: §5.2 · pp. 8–9
- Domain-rule guardrails reject invalid actions, while rule hints in tool outputs help agents identify valid actions they might otherwise miss.For instance, reservation information can indicate whether each reservation is modifiable.Source: §5.2 · p. 9
5.3 Speculative Agentic Actions
- Speculative execution bundles results from likely consecutive tool calls, saving input-generation tokens and an agent-environment round trip when the prediction matches.For example, the environment can run calc_avg after get_cases or retrieve orders within get_user_profile; a miss wastes tokens, so low miss cost matters.Source: §5.3 · p. 9
6 Evaluation
Across five workloads, environment optimizations improve success rates, including on the unseen evaluation set. They reduce several targeted failure types and generally lower inference cost, with a cost increase in the medical workload; gains also extend to a fine-tuned model.
6.1 Success Rate
- 10.3%, 6.7%, 12.5%, 10.0%, and 7.5% improvement in success rate across five workloads; the analysis and evaluation sets improve by 11.8% and 7.0%, respectively.The unseen evaluation-set improvement is reported as evidence that the optimizations generalize to new tasks.Source: §6.1 · p. 9, Figure 10 · p. 10
6.2 Failure Breakdown
- Addressing navigation failures in medical exposed resource-exhaustion failures, while resolving exploitation failures in airline and retail exposed instruction-following failures.The authors describe this as evidence that agent failures compound and support a holistic approach.Source: §6.2 · p. 10
- 60% fewer Tool Output Processing failures in airline, from 5 to 2; exploration optimizations eliminated the majority of exploration failures in airline, file system, and medical.Table 3 reports GPT 4.1 analysis-set failure counts across workloads.Source: Table 3 · p. 10, §6.2 · p. 10
6.3 Cost Analysis
- 17.4%, 7.1%, and 17.7% lower monetary cost for GPT 4.1, GPT 4.1 mini, and o3, respectively; their task turns fell by 16.8%, 7.2%, and 16.6%.The paper attributes turn reductions to speculative agentic actions.Source: Figure 11 · p. 10, §6.3 · p. 10
- 29–33% higher medical-workload cost after optimization, as improved observability led agents to retrieve more information than they did before failing early.The authors consider this a favorable tradeoff.Source: §6.3 · p. 10
6.4 Fine-tuning
- With xLAM-2 8B, success rates improve by 3.3% on the analysis set and 8.8% on the evaluation set; file-system performance shows no added improvement.The paper presents environment optimization as complementary to fine-tuning and reports remaining file-system failures as mostly instruction-following failures.Source: Figure 12 · p. 11, §6.4 · p. 11
- Environment optimizations require modifying tool APIs, while fine-tuning involves a resource-intensive, multi-stage pipeline and risks overfitting and catastrophic forgetting.The authors recommend prioritizing environment optimization and considering fine-tuning if those improvements are insufficient.Source: §6.4 · p. 11
7 Related Works
Prior work analyzes agent-centric failures and develops specialized environments for particular domains. This work complements those lines by examining agent-environment failures and seeking environment-optimization principles across workloads.
Failure Analysis of LLM Agents
- Agent-centric taxonomies categorize failures such as incorrect inter-agent communication and flawed reasoning or planning.Other work localizes failure steps or measures misalignment with user expectations.Source: §7 · p. 11
Specialized Agent Environments
- Unlike domain-specific interface efforts, this work seeks generalizable environment-optimization principles across workloads.It also uses failure analysis to develop, implement, and empirically validate environment optimizations.Source: §7 · p. 11
- Specialized interfaces have improved performance in games, web browsing, and software engineering.Examples include screenshots, refined observation and action spaces, and custom agent-computer tools.Source: §7 · p. 11
8 Conclusion
The conclusion frames the system environment as a first-class component of reliable agentic systems. It presents a methodology for analyzing agent-environment interactions, targeted optimizations, and evaluations showing improved task success across workloads and models.
8 Conclusion
- The system-for-agent paradigm treats the agent’s environment as a first-class component in building more reliable agentic systems.Source: §8 · p. 11
- The paper’s methodology analyzes agent-environment interactions to produce a taxonomy of common failure modes and targeted system-level optimizations.Source: §8 · p. 11
- Evaluations show that the environment optimizations substantially improve task success rates across a variety of workloads and models.Source: §8 · p. 12