AgentOps: Enabling Observability of LLM Agents

1 Introduction

• LLM agents are autonomous systems that perceive context, reason, plan, and execute workflows using external tools, knowledge bases, and other agents.They are motivated by limitations of LLMs in handling complex tasks autonomously.Source: §1 · p. 1
• Agents combine design-time components with dynamically generated runtime artifacts across context processing, planning, workflow execution, and continuous evolution.These pipelines may use external tools, knowledge bases, and other agents.Source: §1 · p. 1
• Autonomous, non-deterministic behavior can produce unintended outcomes, while continuous evolution complicates alignment with quality and safety goals.Agents may interact with shifting environments and produce different outputs from identical inputs.Source: §1 · p. 2
• Shared accountability across agent owners, foundation-model providers, and external tool or agent providers complicates failure-source identification and incident accountability.Observability must therefore support tracing behavior, artifact evolution, and associated data.Source: §1 · p. 2
• Existing agent DevOps tools largely emphasize LLM metrics and prompt management, with limited support for goals, plans, and tools.The paper identifies this as insufficient observability from both system and pipeline perspectives.Source: §1 · p. 2
• AgentOps proposes a specialized DevOps paradigm that traces agent artifacts and associated data to provide holistic observability.The paper presents an artifact relationship model and taxonomy intended to guide monitoring, logging, and analytics infrastructure.Source: §1 · p. 2

2 Methodology

• The study systematically maps existing tools to understand their features and limitations and identify artifacts and data requiring traceability.Figure 1 summarizes the search process.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Data Sources

• Researchers combined GitHub and Google Search to broaden coverage across open-source and proprietary AgentOps-relevant tools.The searches used targeted AgentOps, DevOps, LLMOps, and observability terms.Source: §2.1 · p. 2, §2.2 · p. 3

2.3 Selection Criteria

• Tool inclusion required core observability functions, agent- or application-level tracing, a formal release version, and accessible public documentation.These criteria distinguish AgentOps-focused tools from general model-level operations.Source: §2.3 · p. 3, Table 1 · p. 3

2.4 Search Process

• The search screened GitHub results and added six tools from Google Search, producing a final set of 17 tools.The Agent-and-DevOps query produced 618 repositories but contributed only one tool after relevance screening.Source: §2.4 · p. 4

2.5 Data Extraction

• Data were extracted from repositories, official product websites, and project documentation using predefined data items.Table 2 specifies the extraction fields used for categorization.Source: §2.5 · p. 4, Table 2 · p. 4

3 Mapping Study Result

• The study presents results from a systematic mapping study of tools relevant to AgentOps, with Table 3 listing the tools included in that mapping.Source: §3 · p. 4, Table 3 · p. 4

3.1 AgentOps-Relevant Tools

• 17 tools were analyzed, and 14 had accumulated thousands of GitHub stars by November 2024.Examples include AgentOps at 1.7k stars, AgentNeo at 1k, Langfuse at 6.5k, and PortKey at 6.3k.Source: §3.1 · p. 5, Table 4 · p. 5
• The leading listed LLM-application tools included Langfuse, PortKey, Arize, TraceLoop, and DataDog.Their reported star counts were 6.5k, 6.3k, 3.9k, 3.4k, and 2.9k, respectively.Source: §3.1 · p. 5
• The tool landscape may omit relevant projects because of keyword mismatches or incomplete documentation, motivating future monitoring of emerging tools.The paper states that future work will incorporate additional tools.Source: §3.1 · p. 5

3.2.1 Customization

• AgentOps tools extend agents with marketplace toolkits, multiple knowledge bases, and customized fine-tuned models for specific use cases.Source: §3.2.1 · p. 5

3.2.2 Prompt Management

• Prompt management provides versioning, experimentation, and monitoring to compare templates and detect risks such as code injection and secret leakage.Prompt playgrounds support testing templates with different models before deployment.Source: §3.2.2 · p. 5

3.2.3 Evaluation

• Evaluation assesses agents against criteria and benchmarks while also tracking execution steps and intermediate outputs for governance.The process includes evaluation datasets, criteria, metrics, and testing against user requirements or comparable systems.Source: §3.2.3 · p. 5

3.2.4 Feedback

• Human feedback can be attached to execution traces or LLM generations and used to improve agent designs or retrain models.Explicit feedback is simpler but often lower-quality and less frequent than implicit behavioral feedback.Source: §3.2.4 · p. 6

3.2.5 Monitoring

• Monitoring tracks metrics such as latency and cost, associates feedback with spans, and uses traces to debug errors and performance problems.It can help identify root causes of unexpected results, errors, or latency issues.Source: §3.2.5 · pp. 6–7

3.2.6 Tracing

• Tracing captures an agent’s process from user prompt to final output, helping developers inspect decisions and identify root causes across complex workflows.All tools listed in Table 3 implement tracing functionality.Source: §3.2.6 · p. 7

3.2.7 Guardrails

• Guardrails support AI-safety-by-design, and tracing their activation, execution, and outcomes can generate safety cases for auditing.Arize Guards is an existing implementation, while other tools list guardrails as planned features.Source: §3.2.7 · p. 7

4 Taxonomy of AgentOps

• The taxonomy is intended as a developer template for implementing AgentOps tools that trace agent artifacts and associated data.Future work includes validating it through real-world case studies and developing an AgentOps prototype.Source: §4 · p. 7

4.1 Entity-Relationship Model for Agent Artifacts

• The artifact relationship model links user goals, knowledge bases, reasoning, plans, workflows, tasks, tools, LLM calls, evaluations, and guardrails through nested spans.Plans organize objectives into workflows; workflows contain tasks, while evaluations and guardrails connect across the lifecycle.Source: §4.1 · p. 7

4.2 Taxonomy of AgentOps

• A trace contains one or more spans, with a root span representing a complete request and child spans exposing its constituent steps.The model uses parent-child structure to provide deeper context for operations within a request.Source: §4.1 · p. 7, §4.2 · p. 8
• The taxonomy records common span metadata including names, timestamps, duration, inputs, outputs, errors, tracebacks, metrics, events, parent IDs, and links.These attributes support temporal analysis, debugging, hierarchy tracking, and relationships among spans or external references.Source: §4.2 · p. 8
• Reasoning and planning spans capture context, retrieved knowledge, rules, outcomes, goals, constraints, historical plans, and other information shaping decisions.Planning metadata describes the intended sequence of operations and the conditions under which it must operate.Source: §4.2 · p. 10
• Workflow, task, and tool spans represent execution structure, documenting task dependencies, operational context, execution history, task status, tool configurations, and tool responses.Tool spans are nested within task spans, while workflows organize the tasks required to achieve a plan.Source: §4.2 · p. 10
• Evaluation spans store test cases, metrics, and results to assess whether agent actions and outputs meet expected standards.The taxonomy treats evaluation as a structured assessment of correctness and quality against predefined criteria.Source: §4.2 · p. 11
• Guardrail spans record governance constraints, triggered actions, and targeted artifacts, while LLM spans record model identity, version, parameters, and interactions.Guardrails may block, validate, or filter artifacts such as goals and tools; LLM parameters include temperature and maximum tokens.Source: §4.2 · p. 11

• The rapid proliferation of tools means the study may not identify all relevant AgentOps tools, despite using multiple data sources.The sample includes open-source tools such as AgentOps and Langfuse and commercial platforms such as Datadog.Source: §5 · p. 11
• The proposed lifecycle coverage may omit data attributes, including trace links and interactions between different steps.The authors supplemented their findings with academic literature [9] [10] and identify these omissions as future-work gaps.Source: §5 · p. 11

• AgentOps is proposed as a specialised DevOps paradigm tailored for LLM agents to enable observability.Source: §6 · p. 11
• The study develops an entity-relationship model and comprehensive taxonomy to structure agent artifacts and guide developers designing AgentOps tools.Future work will validate the taxonomy through real-world case studies and develop an AgentOps tool prototype.Source: §6 · pp. 11–12