Source-linked AI summary
TRAIL: Trace Reasoning and Agentic Issue Localization
Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, Rebecca Qian
TL;DR
Agentic workflows require scalable evaluation of complex, structured traces, but existing approaches are difficult to apply systematically. TRAIL introduces a fine-grained taxonomy and 148 annotated traces from realistic benchmarks, finding that leading LLMs remain poor at trace debugging. The dataset is primarily text-only and contains many sparse tail categories.
Problem
Existing agent evaluation requires granular analysis of complex structured traces, while prior approaches rely on end-to-end evaluation or inadequately structured traces.
Method
TRAIL introduces a fine-grained taxonomy spanning reasoning, planning, and execution, and constructs an annotated benchmark from GAIA and SWE-Bench traces.
Results
GEMINI-2.5-PRO achieved only 11% combined joint accuracy on TRAIL, while three of eight models could not process the full context.
Takeaways & Limitations
TRAIL provides an open foundation for scalable, systematic evaluation of agentic workflows and exposes limitations in current LLM trace analysis.
Takeaways & Limitations
TRAIL primarily covers text-only inputs and outputs, and many tail error categories have very few examples.
Abstract
from arXiv · showhide
The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.
1 Introduction
TRAIL addresses the need for granular, structured-trace evaluation of agentic workflows by introducing an error taxonomy and an ecologically grounded benchmark. Its evaluations find that leading LLMs perform poorly at trace debugging, with reasoning-chain length affecting performance.
- Motivation and approach: TRAIL targets step-level analysis of structured, opentelemetry-based agentic workflows rather than only end-to-end outcomes.The benchmark is designed for traces extending beyond current model context limits and covers single- and multi-agent applications.
- Contributions: The taxonomy defines fine-grained agentic error categories across reasoning, planning, and execution.It adds system-execution and planning failures such as API errors and task orchestration errors.
- Findings: 11% combined joint accuracy was achieved by GEMINI-2.5-PRO, the best-performing evaluated model on both splits.Current leading LLM families perform modestly at predicting both error categories and locations.
- Availability: TRAIL is fully open-source under the MIT License and will include a HuggingFace leaderboard for future agentic-workflow evaluation.The dataset is positioned as a foundation for scalable evaluation research.
2 Relevant Work
Prior work establishes the value of agent benchmarks, trace observability, and LLM-based evaluation, but TRAIL focuses on ecologically valid, structured traces with granular execution and planning failures. The section situates TRAIL at the intersection of these developments.
- LLM-based evaluation: LLM judges have seen limited application to trace evaluation because the task requires robust reasoning over large contexts.This distinguishes trace analysis from broader LLM-as-a-judge settings.
- Agentic evaluation: Multi-agent frameworks have shown minimal performance gains compared with single-agent counterparts in prior work.This motivates evaluating both system types rather than assuming multi-agent setups are superior.
- Agent benchmarks: Existing agentic benchmarks cover realistic coding, general-assistant, and web tasks, including SWE-Bench and GAIA.SWE-Bench tests real-world GitHub issue resolution, while GAIA requires reasoning, tool use, and multimodality.
- Traces and taxonomies: Prior trace-analysis work emphasizes observability for diagnosing nondeterministic agent behavior and performing root-cause analysis.Related applications include diagnostic log analysis in cloud incidents and security contexts.
- TRAIL’s position: TRAIL differs through ecological validity, coverage of single- and multi-turn systems, and emphasis on execution and planning failures.Its taxonomy particularly targets critical failure patterns beyond agentic reasoning and coordination.
3 Agentic Error Taxonomy
The TRAIL taxonomy organizes agentic failures into reasoning, planning and coordination, and system execution, covering errors from information handling and decisions through outputs, configuration, APIs, resources, context, and task management.
- Taxonomy structure: The taxonomy spans three areas: reasoning, planning and coordination, and system execution.It is intended to provide a comprehensive framework for classifying agentic errors.
- Reasoning: Hallucination categories include fabricated or ungrounded text and invented tool outputs or capabilities.The latter concerns agents misunderstanding tool functions or claiming nonexistent results.
- Reasoning: Information-processing failures cover poor retrieval and misinterpretation of retrieved outputs.These categories address agents’ difficulty reasoning effectively over retrieved information.
- Planning and coordination: Decision-making categories include incorrect problem identification and tool-selection errors at individual workflow steps.Appropriate tool selection is linked in the taxonomy discussion to lower cost and greater efficiency.
- Execution and outputs: Output-generation failures include formatting errors and instruction non-compliance, especially for structured tool calls.The examples concern precise JSON or code formatting and complex or ambiguous instructions.
- System execution: System-oriented categories include configuration, API, resource, context, and task-management failures.Examples include authentication and rate-limit errors, resource exhaustion, context-handling failures, goal deviation, and task orchestration errors.
4 TRAIL Benchmark
TRAIL is a benchmark for analyzing long, structured OpenTelemetry agent executions through fine-grained error annotations across realistic information-retrieval and software-engineering tasks.
- TRAIL contains 148 annotated traces and 841 errors, averaging 5.68 errors per trace across GAIA and SWE-Bench Lite.
- The benchmark covers open-world search and GitHub issue localization and repair, selected for their need for environment and search-space exploration.
- GAIA traces use hierarchical multi-agent orchestration, while SWE-Bench traces use a single CodeAct agent with sandbox, Python, and repository-ingestion tools.
- The traces are collected with OpenTelemetry-compatible OpenInference instrumentation to support real-world tracing and observability workflows.
- Four software-engineering and log-debugging experts annotated the traces, with a separate 63-trace set indicating high inter-annotator agreement.
- The dataset includes long-tail, high-impact error types, while Output Generation errors dominate and Formatting Errors plus Instruction Non-compliance comprise 353 of 841 errors.
5 Results
TRAIL evaluations show that trace debugging is difficult for current LLMs, especially with long inputs, while reasoning models and higher test-time reasoning generally perform better.
- Long-Context Reasoning: TRAIL inputs approach or exceed several models' context limits, while typical required outputs exceed 1K tokens and reach approximately 3.7K at maximum.
- All performance metrics are anti-correlated with input length, supporting the hypothesis that longer raw traces increase task difficulty.
- Reasoning vs Non-Reasoning Models: Reasoning models other than O1 achieve 1.5–8 times the best non-reasoning model's Joint Accuracy.
- Does Reasoning Effort Matter?: Category F1 for O3 decreases from 0.296 to 0.277 to 0.264 as reasoning effort changes from high to medium to low.
- Performance Across Categories: Context Handling Failures are nearly universally scored at F1 0.00, with CLAUDE-3.7-SONNET the exception at 0.18.
- Performance Across Categories: For Tool Selection Errors, most models score 0.00–0.08, while GEMINI-2.5-PRO, CLAUDE-3.7-SONNET, and O3 reach 0.26, 0.27, and 0.53.
- Performance Across Categories: Goal Deviation and Poor Information Retrieval favor the Gemini models, whereas Formatting Errors show non-monotonic performance with GPT-4.1 and Gemini models outperforming several reasoning models.
6 Conclusion
TRAIL combines a taxonomy with 148 expert-curated agentic problem instances and shows that current LLM judges struggle to evaluate complex traces under context constraints.
- TRAIL combines a new agentic-error taxonomy with 148 expert-curated problem instances and 841 unique errors from GAIA and SWE-Bench.
- GEMINI-2.5-PRO reaches 18% Joint Accuracy on GAIA and 5% on SWE-Bench, while three of eight models cannot process the full context.
- The results indicate that existing models struggle with systematic evaluation of complex agentic traces, motivating a framework for scalable workflow evaluation.
Limitations
TRAIL is primarily text-only, while multimodal agentic systems require extending its taxonomy to cover errors such as multimodal tool use; many tail categories also have few examples.
- TRAIL primarily covers text-only inputs and outputs, leaving multimodal tool-use errors outside its current focus.
- Many tail error categories contain very few examples, making LLM-Judge correctness especially important for high-impact failures.
- Future work could generate synthetic traces for high-impact, low-occurrence categories by modifying existing traces to induce catastrophic irrecoverable failures.
Ethics Statement
Annotators were selected for being adults with computer-science expertise, and traces were screened for personally identifiable, explicit, or biased content before annotation.
- Annotators were selected based only on being at least 18 years old and having computer-science expertise.
- Each trace took 30–40 minutes to annotate, and annotators were paid $12.66 per trace.
- Traces were manually checked to exclude personally identifiable information, explicit content, and biased content before annotation.
A.2 Long Context Leaderboard Rankings vs TRAIL
TRAIL’s model rankings largely match rankings from LongBenchv2 and LongContextBench, except that O3 underperforms Gemini models on TRAIL.
- TRAIL reproduces the LongBenchv2 ranking GEMINI-2.5-PRO > GEMINI-2.5-FLASH > O1.
- TRAIL’s ranking largely matches fiction.live’s LongContextBench ordering across the evaluated models.
- Unlike LongContextBench, TRAIL places O3 below GEMINI-2.5-PRO and GEMINI-2.5-FLASH.
A.3 Evaluation Setup
The evaluation tests state-of-the-art closed and open models on TRAIL using structured traces, multiple agent settings, span-level annotations, and rubric-based quality checks. It also examines reasoning effort, context demands, model-specific performance, and annotation reliability.
- Model selection: The study evaluates closed and open-source models selected for reasoning, agentic capabilities, or long-context support.Closed models include O1, O3, GPT-4.1, Claude 3.7 Sonnet, and Gemini 2.5 Pro/Flash; open models include Llama-4 Scout and Maverick.
- Reasoning effort: Reasoning effort is varied for O3 on the GAIA split to measure performance changes across low-to-high API settings.
- Annotation: TRAIL evaluation uses raw structured traces and measures both overall trace quality and span-level error annotations.Annotators mark span IDs, error categories, evidence, descriptions, and impact, while also rating instruction adherence, plan optimality, security, and reliability.
- Model-specific results: GEMINI-2.5-PRO is strongest overall, including Goal Deviation (0.70), Poor Information Retrieval (0.50), Tool Output Misinterpretation (0.67), and Environment Setup Errors (0.57).
- Context demands: Raw trace inputs approach or exceed model context limits, while task outputs can also require substantial generation capacity.
- Human agreement: GEMINI-2.5-PRO has the highest SWE Bench correlation with human judgment (average 0.817), while CLAUDE-3.7-SONNET scores highest on GAIA (average 0.738).