Source-linked AI summary
Evaluation and Benchmarking of LLM Agents: A Survey
Mahmoud Mohammadi, Yipeng Li, Jane Lo, Wendy Yip
TL;DR
Evaluating LLM agents is difficult because they operate in dynamic, interactive environments involving planning, tools, memory, and collaboration. This survey organizes the field with a two-dimensional taxonomy, highlights enterprise-specific requirements, and reports that current agents struggle with consistency in retail and airline-booking tasks.
Problem
Existing evaluation approaches and surveys provide limited holistic coverage of agents’ dynamic behavior and rarely address enterprise requirements such as secure data access, reliability, and compliance.
Method
The survey proposes a taxonomy organized by evaluation objectives and evaluation process, covering interaction modes, data, metrics, tooling, and environments.
Results
The survey identifies enterprise-specific challenges and reports that current agents struggle with consistency when evaluated using pass^k in retail and airline booking.
Takeaways & Limitations
Systematic agent assessment must account for behavior, capabilities, reliability, safety, interaction settings, and enterprise constraints rather than treating agents like standalone language models.
Takeaways & Limitations
Offline evaluations are cheaper and simpler but generally provide less accurate representations of agent performance because static cases lack interaction nuance and are prone to error propagation.
Abstract
from arXiv · showhide
The rise of LLM-based agents has opened new frontiers in AI applications, yet evaluating these agents remains a complex and underdeveloped area. This survey provides an in-depth overview of the emerging field of LLM agent evaluation, introducing a two-dimensional taxonomy that organizes existing work along (1) evaluation objectives -- what to evaluate, such as agent behavior, capabilities, reliability, and safety -- and (2) evaluation process -- how to evaluate, including interaction modes, datasets and benchmarks, metric computation methods, and tooling. In addition to taxonomy, we highlight enterprise-specific challenges, such as role-based access to data, the need for reliability guarantees, dynamic and long-horizon interactions, and compliance, which are often overlooked in current research. We also identify future research directions, including holistic, more realistic, and scalable evaluation. This work aims to bring clarity to the fragmented landscape of agent evaluation and provide a framework for systematic assessment, enabling researchers and practitioners to evaluate LLM agents for real-world deployment.
1 Introduction
LLM agents operate dynamically through reasoning, planning, tools, memory, and interaction, making their evaluation more complex than evaluating LLMs alone. This survey addresses fragmented coverage with a taxonomy and enterprise-focused challenges.
- Motivation: LLM agents act in dynamic, interactive environments, so evaluating them requires assessing reasoning, planning, tool use, memory, and real-world effects.These properties make standard text-generation and question-answering evaluations insufficient.
- Research gap: Existing surveys often focus narrowly on LLM evaluation or specific capabilities rather than providing a holistic view of agent evaluation.Enterprise requirements are also rarely addressed in the existing literature.
- Contributions: The survey organizes prior work by evaluation objectives and evaluation process, covering behavior, capabilities, reliability, safety, interaction modes, data, metrics, tooling, and environments.The two dimensions distinguish what to evaluate from how to evaluate it.
- Contributions: The survey highlights enterprise challenges involving role-based access control, reliability guarantees, long-term interaction, and compliance requirements.These challenges accompany deployment in enterprise settings.
- Organization: The paper examines the taxonomy, evaluation objectives, evaluation processes, enterprise challenges, and future research directions in successive sections.The structure follows the survey’s two-dimensional organization.
2 Taxonomy for LLM-based Agent Evaluation
The survey’s taxonomy separates what is evaluated from how evaluation is conducted. It provides a framework for comparing agents across goals, methods, and deployment conditions while recognizing that diverse settings require tailored strategies.
- Taxonomy structure: The taxonomy has two axes: Evaluation Objectives, which specify what to assess, and Evaluation Process, which specifies how to assess it.The framework is represented as a hierarchical tree.
- Evaluation Objectives: Evaluation Objectives cover agent behavior, capabilities, reliability, and safety and alignment.Capabilities include tool use, planning and reasoning, memory and context retention, and multi-agent collaboration.
- Evaluation Process: Evaluation Process covers interaction mode, evaluation data, metric computation methods, evaluation tooling, and evaluation contexts.These range from static inputs and synthetic data to human or LLM judgments, instrumentation, leaderboards, simulations, browsers, and APIs.
- Purpose: The taxonomy supports systematic comparison and analysis of LLM agents across goals, methodologies, and deployment conditions.It functions as both a conceptual framework and a practical guide.
- Scope boundary: Single-turn versus multi-turn interaction, multilingualism, and multimodality may require tailored metrics and evaluation strategies.The taxonomy remains applicable across these variations, but the evaluation methods need adaptation.
3.1 Agent Behavior
Agent behavior evaluation treats the agent as a user-facing black box and measures task completion, output quality, latency, and cost. These dimensions span both outcome attainment and the quality and practicality of interaction.
- Agent behavior: Agent behavior evaluates overall user-perceived performance as a black box, including task completion, output quality, latency, and cost.This is the highest-level evaluation view and offers direct insight into user experience.
- Task completion: Task completion measures whether an agent reaches predefined goals or satisfies task-success criteria.Success Rate, Task Goal Completion, Pass Rate, and binary goal-achievement indicators are used.
- Task completion: Task completion is applied across coding, scientific programming, research reproduction, interactive applications, and web environments.Representative benchmarks include SWE-bench, ScienceAgentBench, CORE-Bench, PaperBench, and AppWorld.
- Output quality: Output quality covers accuracy, relevance, clarity, coherence, and adherence to agent specifications or task requirements.An agent can complete a task while still producing a poor user experience.
- Output quality: Response Relevance and Factual Correctness can evaluate tool-grounded, context-aware answers produced by agents.These metrics overlap with evaluation approaches for retrieval-augmented generation systems.
- Latency and cost: Latency is measured with TTFT for streaming interactions and End-to-End Request Latency for asynchronous use cases, while cost is estimated from input and output tokens.Latency affects synchronous user experience, whereas token counts approximate usage-based monetary efficiency.
3.2 Agent Capabilities
Agent capability evaluation examines the mechanisms underlying performance, including tool use, planning and reasoning, memory, and multi-agent collaboration. The survey describes metrics and benchmarks for both isolated decisions and long-horizon execution.
- Capability scope: Capability evaluation targets tool use, planning and reasoning, memory and context retention, and multi-agent collaboration.This provides a more granular view of agent strengths and weaknesses than external behavior alone.
- Tool use: Tool-use evaluation asks whether an agent knows when to invoke a tool, selects the right tool, and supplies executable parameters.These decisions support retrieval, external actions, and interaction with environments.
- Tool use: Invocation Accuracy, Tool Selection Accuracy, Retrieval Accuracy, and Mean Reciprocal Rank measure different stages of tool selection and retrieval.The metrics distinguish deciding to call, choosing among candidates, retrieving from larger toolsets, and ranking results.
- Tool use: Parameter-name F1 evaluates identifying required function parameters and assigning values, while AST checks may miss semantic errors in those values.Syntactic validity does not necessarily establish semantic correctness.
- Planning and reasoning: Planning selects tools in an appropriate order, while reasoning supports context-aware decisions before or during execution.T-Eval evaluates planning by comparing predicted tools with a reference.
- Planning and reasoning: Step Success Rate measures the percentage of generated plan steps executed successfully, complementing program-similarity comparisons against annotated references.These approaches evaluate planning quality during execution and against reference plans.
- Memory and context retention: Memory evaluation tests retention and application of information across long dialogues and long-horizon tasks, using recall and consistency metrics.Benchmarks include conversations exceeding 40 turns and studies with dialogues exceeding 600 turns.
- Multi-agent collaboration: Multi-agent collaboration requires methods that assess natural-language coordination, strategic reasoning, and decentralized problem-solving.These capabilities differ from coordination based on predefined reinforcement-learning reward structures.
3.3 Reliability
Reliability evaluation examines whether agents remain consistent, robust, and trustworthy across repeated tasks and changing conditions, including tool or environmental failures.
- Reliability covers consistency, robustness to variations, and trustworthiness, extending evaluation beyond best-case task performance to average- and worst-case scenarios.
- Consistency: The pass^k metric measures whether an agent succeeds on all k repeated attempts, providing a stricter consistency measure than pass@k.The τ-benchmark uses pass^k to capture consistency requirements for mission-critical deployments.
- Robustness: Robustness tests whether agents maintain performance under paraphrases, misleading context, typos, dialects, and other input variations.
- Robustness: Adaptive resilience evaluates whether agents adjust their strategies when environments change rather than stalling or failing.WebLinX examines behavior when web-page structure changes during execution.
- Robustness: Tool-use robustness includes recovering gracefully from API errors, null responses, and other unexpected tool outputs.Possible recovery behaviors include retries, switching tools, or explaining the issue to the user.
3.4 Safety and Alignment
Safety and alignment evaluation addresses harmful behavior, fairness, transparency, and compliance with domain-specific legal or policy constraints.
- Safety evaluation examines adherence to ethical guidelines, avoidance of harmful behavior, and compliance with legal or policy constraints in high-stakes domains.
- Fairness and Transparency: Fairness and transparency matter because biased decisions can reduce trust and reinforce systemic inequalities, including in financial applications.
- Fairness and Transparency: Explainability evaluations assess whether agents provide clear reasoning paths that support user trust in recommendations and autonomous decisions.AutoGuide, MATSA, FinCon, and Rjudge represent guideline-driven or structured transparency approaches.
- Safety Evaluation: Toxicity evaluation measures harmful language and bias using test sets, automated detectors, human raters, and failure rates under provocative inputs.RealToxicityPrompts and CoSafe target toxic outputs and adversarial attempts to induce safety violations.
- Compliance: Compliance testing uses domain-specific scenarios to verify that agents respect confidentiality, regulated advice boundaries, and established guidelines.Enterprise frameworks may incorporate proprietary cases and metrics for domains such as finance and law.
4 Evaluation Process
The evaluation process spans interaction modes, datasets, metrics, tooling, and evaluation environments, balancing realism, cost, scalability, and objectivity.
- Interaction Mode: Offline evaluation uses static datasets and test cases, whereas online evaluation uses reactive simulations, human interactions, or live monitoring.
- Interaction Mode: Offline evaluations are cheaper and simpler but often miss response nuance, propagate errors, and represent system performance less accurately.
- Interaction Mode: Online evaluation adapts to user or simulated-environment interactions and can reveal domain-specific issues that static testing misses.
- Interaction Mode: Web-agent simulators verify action sequences such as clicking links and filling forms in reactive environments.Examples include MiniWoB, WebShop, and WebArena.
- Tooling: Evaluation-driven Development integrates continuous offline and online evaluation into development and production monitoring to detect regressions and adapt to new use cases.
- Datasets and Benchmarks: Agent benchmarks combine human-annotated, synthetic, and interaction-generated data to assess reasoning, workflows, tool use, planning, safety, and robustness.Examples include ScienceAgentBench, TaskBench, ToolBench, AppWorld, WebArena, AgentHarm, and AgentDojo.
- Metrics Computation Methods: Code-based metrics are deterministic and reproducible for well-defined outputs but are inflexible for open-ended qualitative responses.
- Metrics Computation Methods: LLM-as-a-Judge evaluates nuanced and subjective responses using qualitative criteria supplied through instructions.
5 Enterprise-Specific Challenges
Enterprise evaluation must account for security, permissions, repeatability, long-horizon operation, and domain-specific compliance rather than task success alone.
- Enterprise deployments demand performance alongside predictable reliability, regulatory compliance, data security, and maintainability.
- Role-Based Access Control: Role-Based Access Control makes agent permissions context-dependent because agents must follow the access constraints of the users they represent.
- Role-Based Access Control: Permission-sensitive evaluations can authenticate user identity and deny access to other users’ information through role-specific task restrictions.IntellAgent embeds these constraints into task generation.
- Reliability Guarantees: Enterprise agents must behave reliably across time and usage scenarios because occasional success is insufficient for compliance and auditing frameworks.
- Reliability Guarantees: Reliability measurement requires repeated trials and representative datasets, creating substantial computational and benchmarking overhead for complex agent tasks.
- Reliability Guarantees: The τ-benchmark applies pass^k in retail and airline-booking domains and reports that current agents struggle with consistency.
- Long-Horizon Interactions: Long-horizon evaluation is needed because short benchmarks cannot capture performance drift, context retention, or cumulative decision effects over extended operation.
- Long-Horizon Interactions: Long-running simulations and extended dialogues provide initial tools for studying emergent behavior and memory over multi-day or 600-turn interactions.
6 Future Research Directions
The survey identifies four future directions for LLM agent evaluation: holistic frameworks, realistic settings, automated and scalable techniques, and time- and cost-bounded protocols. Together, these directions target evaluation that is holistic, realistic, scalable, and efficient for reliable and trustworthy agents.
- Holistic Evaluation Frameworks: Holistic evaluation should assess multiple interdependent competencies rather than isolated dimensions such as task success, planning quality, or tool use.Real-world agents must balance multiple competencies simultaneously.
- More Realistic Evaluation Settings: More realistic settings should incorporate dynamic multi-user interactions, role-based access controls, and domain-specific knowledge through deployment trials or enterprise workflow simulations.These settings are intended to bridge laboratory evaluation and production environments.
- Automated and Scalable Evaluation Techniques: Automated evaluation should reduce human effort and improve reproducibility in costly multi-turn and long-horizon assessments.Proposed approaches include synthetic test-case generation, simulated environments, LLM-as-a-judge, and Agent-as-a-judge.
- Time- and Cost-Bounded Evaluation Protocols: Time- and cost-bounded protocols should balance evaluation depth with efficiency to support iterative agent development.Repeated trials and human-in-the-loop assessments can be time- and resource-intensive.
- Overall Direction: The survey concludes that evaluation methods should be holistic, realistic, scalable, and efficient for building reliable and trustworthy LLM-based agents at scale.This conclusion consolidates the four proposed research directions.