PaperlayerView exact revision on arXiv

Source-linked AI summary

Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

Mia Lassiter, Brinnae Bent

arXiv:2609.11018v1cs.AIcs.MA

TL;DR

The term “AI agent” lacks a standard definition, complicating evaluation, comparison, and reproducibility. The paper surveys agent evaluation across five dimensions of agenticness and introduces the Agent Compendium to organize criteria and evaluation methods. It provides a structured account of methods while noting that evaluation remains limited or inconsistent in some areas.

  • Problem

    “AI agent” lacks a widely accepted definition, complicating evaluation and reproducibility when systems with different capabilities are described using the same term.

    Source: §1 · p. 1

  • Method

    The paper organizes its review around five dimensions of agenticness and maps explicit agent criteria to existing metrics, benchmarks, and evaluation frameworks through the Agent Compendium.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    The review and compendium provide a structured account of agent evaluation methods, while identifying areas where evaluation remains limited or inconsistent.

    Source: §1 · p. 2, §2.2 · p. 3

  • Takeaways & Limitations

    The compendium offers a public-facing structure for accessing evaluation methods for components of agenticness.

    Source: §2.2 · p. 3

    Navigation evaluation methods differ across models, and some metrics or capabilities apply only to models with specific modalities or abilities.

    Source: Navigation Metrics · p. 22

Abstract

from arXiv · show

The term agent in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research. We address this ambiguity through a survey organized around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, we examine how the underlying capability has been conceptualized across prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. This review provides a structured account of the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent. We additionally introduce the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified through this review. Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents.

1 Introduction

“AI agent” lacks a widely accepted definition, and researchers disagree about which capabilities qualify a system. This ambiguity complicates evaluation and reproducibility, communication, and distinctions between agents and agentic AI.

1 Introduction

  • Researchers disagree about which capabilities qualify a system as an AI agent, complicating evaluation and reproducibility when systems with different capabilities share the label.The rapid integration of LLMs into agent architectures adds uncertainty about which capabilities to evaluate and which benchmarks to use (Zhu et al., 2025).Source: §1 · p. 1
  • Different definitions also complicate communication and can misalign user expectations with agent behavior, including uncertainty about capabilities, prompting, and human takeover.Shome et al. (2026) report that some participants hesitated to prompt agents because they were uncertain how extensively the agents might act or respond.Source: §1 · pp. 1–2
  • Ambiguity contributes to conflating AI agents with agentic AI, although a system’s agent status and its degree or form of agentic behavior are distinct questions.Source: §1 · p. 2
  • The Agent Compendium organizes evaluation methods around agency capabilities, mapping explicit criteria for identifying agents to existing metrics, benchmarks, and frameworks.It aims to support reproducibility, comparability, clearer communication, and better alignment between user expectations and system behavior.Source: §1 · p. 2

2 Related Work

Prior work frames agenticness as a degree of agent-associated characteristics across five dimensions. Existing evaluations cover environments, capabilities, and evaluation infrastructure, but the review identifies no standardized repository of methods for evaluating agenticness components.

2.1 Agenticness Framework

  • Agenticness is the degree to which a system exhibits agent-associated characteristics, organized across environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence.The dimensions concern environmental understanding and impact, behavioral improvement, operation without external guidance, objective pursuit, and consistency across memory, context, and intent.Source: §2.1 · p. 2

2.2 Existing Agent Evaluations

  • Zhu et al. (2025) categorize LLM-agent evaluations by environment and capability, including code, web, OS, mobile, scientific, and game settings, and planning, reflection, interaction, memory, and general skills.Source: §2.2 · p. 2
  • MMAU evaluates task-completion skills including understanding, reasoning, planning, problem-solving, and self-correction across domains, but does not assess performance in interactive environments.Source: §2.2 · p. 2
  • HAL combines a unified harness, regularly updated multidimensional leaderboard, and automated agent-log analysis to address evaluation infrastructure, cost, and shortcut-detection challenges.Its harness standardizes evaluation across nine benchmarks, but those benchmarks measure specific capabilities rather than components of agenticness dimensions (Kapoor et al., 2025).Source: §2.2 · p. 2
  • The review identifies no standardized repository of methods for agent characteristics; it proposes a public-facing compendium of evaluation methods for agenticness components.The repository is intended to improve information accessibility and fill this niche.Source: §2.2 · pp. 2–3

3 Evaluation Methods

This section organizes evaluation methods around agent capabilities and surveys metrics, benchmarks, and frameworks for environmental interaction. It covers architecture, navigation, tool use, spatial analysis, consistency, and robustness.

3 Evaluation Methods

  • The review categorizes existing evaluation metrics within five dimensions of agenticness, treating evaluation as a basis for reproducible AI-agent research.Environmental interaction includes perception, understanding, and manipulation of the environment, with architecture, navigation, tool use, geospatial analysis, consistency, and robustness as components.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Agent Architecture

  • Agent architectures are deliberative, reactive, or hybrid: deliberative systems reason over symbolic representations, reactive systems use condition-action rules, and hybrid systems combine both approaches.Deliberative architectures face challenges representing real-world states and processes; purely reactive agents are limited to situations considered in advance.Source: §3.1.1 · p. 3

3.1.2 Navigation

  • Navigation evaluation includes success normalized by path length, while tasks range from point- and object-goal navigation to language-guided navigation, embodied question answering, and visual question answering.Language-guided tasks place agents in unseen environments and provide natural-language instructions; VQA differs from EQA because agents have a fixed view and cannot control their trajectories.Source: §3.1.2 · pp. 3–4

3.1.3 Tool-Use

  • Tool-use evaluations assess calls, tool names, and arguments; MultiCAT-Bench categorizes ten evaluations, while the Tool Dexterity Index scores optimal use, misuse, and ignored better tools.TDI assigns +1 for optimal use, -1 for misuse, and -0.5 for ignoring a better tool.Source: §3.1.3 · p. 4

3.1.4 Spatial Analysis

  • Spatial analysis distinguishes centimeter-, meter-, and kilometer-scale tasks, and describes episodic and spatial memory as supporting remembered experiences and spatial relationships.The review identifies four spatial failure modes—spatial hallucination, reference-frame confusion, scale insensitivity, and temporal drift—and reports no published framework for tracking them across agents.Source: §3.1.4 · p. 6

3.1.5 Consistency

  • Consistency covers outcome, trajectory, and resource variation across repeated tasks, while the Agent Stability Index includes four drift dimensions that measure consistency.The four ASI dimensions named are output semantic similarity, decision pathway stability, confidence calibration, and tool sequencing consistency.Source: §3.1.5 · p. 6

3.1.6 Robustness

  • Robustness covers performance under training–deployment perturbations and distinguishes resilience to adversarial attacks, infrastructure failures, environmental changes, and prompt reformulations.Adversarial robustness includes threats such as prompt injection; fault robustness concerns infrastructure errors, environmental robustness concerns changes preserving semantic content, and prompt robustness concerns semantically equivalent instructions.Source: §3.1.6 · pp. 6–8

3.2 Learning and Adaptation

Learning and adaptation covers agents’ use of resources and adjustment of behavior to new conditions. The section surveys measures of efficiency, planning, reliability, agentic capabilities, predictability, and generality.

3.2 Learning and Adaptation

  • Learning and adaptation concerns using resources to perform tasks and adjusting behavior to new conditions.Its listed components include efficiency and cost, trajectory and planning, robustness and reliability, agentic capabilities, predictability, and generality.Source: §3.2 · p. 9

3.2.1 Efficiency and Cost

  • Efficiency measures capture time and resource use, including computational efficiency, cognitive efficiency, latency, token usage, and token cost per task.Task difficulty is also relevant: as it increases, cost rises and efficiency decreases (Wang et al., 2025).Source: §3.2.1 · p. 9

3.2.2 Trajectory and Planning

  • Planning comprises task decomposition and execution, followed by reflection on a plan’s merits and shortcomings using model insights, human input, or environmental feedback.Action validity, loop rate, and plan adherence are among the metrics used to characterize why agents succeed or fail.Source: §3.2.2 · p. 9

3.2.3 Reliability

  • Reliability is consistent performance over time and across repetitions, unlike robustness, which concerns success under perturbations.The section describes measuring outcome variation across seeds for each task.Source: §3.2.3 · p. 10

3.2.4 Agentic Capabilities

  • The agentic-capabilities material spans tool selection, task decomposition, environmental failure adaptation, context preservation, and common-sense reasoning.It also describes context-tracking metrics and a hierarchy of capabilities, including Level 5 common-sense reasoning through inference and application of knowledge.Source: §3.2.4 · pp. 10–11

3.2.5 Predictability

  • Predictability is assessed through calibration, discrimination, and the Brier score, which together address confidence and output reliability.Calibration compares stated confidence with empirical success; discrimination tests whether confidence separates successes from failures.Source: §3.2.5 · p. 12

3.2.6 Generality

  • Generality measures operation across contexts and tasks, with frameworks distinguishing levels by minimum performance thresholds or the breadth of mastered tasks.Morris et al. (2023) also discuss task performance and capability breadth, while Kasirzadeh and Gabriel (2025) propose five levels based on mastered-task breadth.Source: §3.2.6 · pp. 13–14

3.3 Autonomy

Autonomy concerns how independently an agent operates, and prior frameworks assess it through task performance or the roles of agents and users. The section also distinguishes autonomy from capability and surveys safety measures.

3.3 Autonomy

  • Autonomy is the ability to operate without human intervention or guidance, but autonomous rational action has not historically been required for agenthood.Fully autonomous agents are included in taxonomies despite not yet being in operation.Source: §3.3 · p. 14
  • Autonomy frameworks assess task performance, the agent’s role, or the user’s role in interaction; one framework describes full autonomy as performing tasks without oversight or control.Morris et al. (2023) associate higher autonomy levels with progressing AGI capabilities and describe risks at different levels, including over-reliance and labor displacement.Source: §3.3 · pp. 14–15
  • Capability benchmarks alone are insufficient for evaluating autonomy because autonomy can be designed independently of capability; evaluations should measure requested user involvement.Feng et al. (2025) describe a highest level where users observe through activity logs but cannot change the agent’s trajectory.Source: §3.3 · p. 15
  • As autonomy increases, harmful behaviors and the importance of AI safety increase; safety measures include compliance, harm severity, risk, policy violation rate, and human intervention rate.Risk is defined as violation probability multiplied by expected severity.Source: §3.3 · p. 16, Table 20 · p. 16

3.4 Goal-Directed Behavior

Goal-directed behavior concerns forming, understanding, and pursuing objectives. The section reviews intentionality, goal complexity, and end-to-end goal execution measures.

3.4.1 Intentionality

  • FIT-Eval estimates intentionality dimensions, maps them to levels, and uses those levels as governance thresholds for AI-system risk management.Its dimensions cover stable purpose under ambiguity and adversarial redirection, foresight, proactive behavior, temporal commitment, and coherence of reasoning.Source: §3.4.1 · pp. 16–18

3.4.2 Goal Complexity

  • Goal complexity is evaluated through hierarchical planning, required plan length relative to human steps, and multi-objectivity under constraints.Kasirzadeh and Gabriel (2025) propose five gradations of goal complexity.Source: §3.4.2 · p. 18

3.4.3 Goal Execution

  • Goal execution metrics are task-based or resource-based, covering completion, terminal state, speed, API calls, and tool interaction.Task-based metrics address task outcomes, while resource-based metrics address resource use.Source: §3.4.3 · p. 18

Task-Based Metrics

The section describes task completion and speed measures, alongside a gradation that includes generating goals and interpreting underspecified objectives.

Task-Based Metrics

  • Goal completion rate is the percentage of tasks for which an agent successfully achieves the intended goal.Source: Task-Based Metrics · p. 18
  • Decision turnaround time measures the time from task initiation to completion and is identified as critical for time-sensitive tasks.Source: Task-Based Metrics · p. 18
  • At the unbounded level, an agent can generate goal structures and interpret underspecified objectives; the described capabilities extend beyond traditional complexity classifications.Source: Task-Based Metrics · p. 19

Resource-Based Metrics

The section outlines resource-oriented measures for task efficiency, collaboration, and tool selection, then frames resilience and adaptation around disruptive events. It also identifies three proposed resilience metrics.

Resource-Based Metrics

  • The cognitive efficiency score measures tokens and tool calls per successfully completed task.The passage describes a low score as imperative for resource management and environmental sustainability.Source: Resource-Based Metrics · p. 20
  • The collaboration quality index assesses communication clarity, responsiveness, and contextual awareness in an agent’s collaborative abilities.Source: Resource-Based Metrics · p. 20
  • The Tool Dexterity Index scores optimal tool use, misuse, and ignored better tools as +1, -1, and -0.5, respectively.It assesses whether an agent selects the optimal tool for each situation.Source: Resource-Based Metrics · p. 20

3.4.4 Resilience and Adaptation

  • Resilience assessments distinguish resource-depletion disruptions from situations lacking established policies, testing resource maintenance and new-plan generation, respectively.Al Shikh et al. (2025) propose multi-step task resilience, chain robustness, and adaptability delta metrics.Source: §3.4.4 · p. 20

3.5 Temporal Coherence

Temporal coherence links an agent’s successive states to preserve logical consistency and support dynamic stability. The section describes coherence through an active processing window and context anchors, and hysteresis as a way to monitor drift.

3.5 Temporal Coherence

  • Temporal coherence makes each step’s state depend on the preceding state, preserving logical consistency and enabling return to stability after disturbance.Its components include coherence and hysteresis.Source: §3.5 · pp. 20–21

3.5.1 Coherence

  • The active processing window dynamically includes new inputs and excludes old states, with its size determined by memory bandwidth and information density.Context anchors resist meaning shifts; coherence drift occurs when interpretation moves beyond their bounds.Source: §3.5.1 · p. 21
  • Sustained positive drift acceleration is described as a strong indicator of anchor failure and imminent temporal incoherence.The passage presents persistent instability as detected through second-order change.Source: §3.5.1 · p. 21

3.5.2 Hysteresis

  • Low hysteresis index H implies hypersensitivity and anchor degradation, while reduced hysteresis indicates context anchoring is failing and coherent drift is beginning.The passage defines probe state s(x)_i as the state resulting from a standardized probe input x at step i.Source: §3.5.2 · p. 21

4 Future Directions

The section identifies evaluation gaps and recommends research directions across hybrid architecture, navigation, task mastery, and temporal challenges. Its recommendations target measures or benchmarks suited to these specific gaps.

Hybrid Agent Architecture

  • Hybrid architectures lack an evaluation of the proportion of deliberative versus reactive characteristics, motivating a proposed hybrid architecture evaluation.The authors recommend pairing it with reactivity degree, defined as maximum available slack percentage to deliberate.Source: Hybrid Agent Architecture · p. 22

Navigation Metrics

  • Navigation metrics vary across models, and measures such as audio-noise metrics or PASTURE do not apply universally, motivating model-agnostic navigation metrics.PASTURE evaluates language-driven zero-shot object navigation capabilities.Source: Navigation Metrics · p. 22

Task Mastery

  • Task mastery lacks defined criteria; the authors suggest adapting accuracy level and success frequency as assessment criteria.The agent generality ontology uses breadth of task mastery without defining what it means to master a task (Kasirzadeh and Gabriel, 2025).Source: Task Mastery · p. 22

Temporal Challenges

  • The authors recommend benchmarks for state awareness and memory systems to reveal temporal reasoning capacity and related temporal challenges.The passage contrasts immediate state awareness with limited historical awareness against advanced state management using hierarchical memory structures.Source: Temporal Challenges · p. 22

5 The Agent Compendium

The Agent Compendium is a growing collection of criteria and evaluation approaches for defining AI agents. It is intended to remain current through automated literature monitoring and expert review.

5 The Agent Compendium

  • The Agent Compendium collects criteria, metrics, benchmarks, and approaches for defining AI agents.It is available at agent.duketrustlab.com.Source: §5 · p. 22
  • An AI-assisted literature review tool will flag new publications and preprints that may introduce or apply measures related to agent characteristics.Source: §5 · p. 22
  • The research team will review flagged sources for relevance, methodological quality, and fit with the existing taxonomy.The authors describe this expert review as preserving scholarly oversight, transparency, and consistency.Source: §5 · p. 22

6 Conclusion

The conclusion presents definitional ambiguity as a barrier to reproducible research and clear communication, and describes the Agent Compendium as a public resource linking five components of agenticness to evaluation methods.

6 Conclusion

  • The lack of a widely accepted definition undermines research reproducibility and scientific communication, and contributes to conflating AI agents with agentic AI.Source: §6 · p. 23
  • The paper builds on a proposed spectrum of agenticness comprising environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence.Source: §6 · p. 23
  • The Agent Compendium maps metrics, benchmarks, and frameworks to those five components and is presented as a publicly accessible resource for scholars, researchers, and the general public.Source: §6 · p. 23

Related papers

Loading 2609.11018v1…