Source-linked AI summary
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Asaf Yehudai, Lilach Eden, Michal Shmueli-Scheuer
TL;DR
Existing agent-evaluation tools rely on basic observability or rigid, hand-crafted categories that cannot adapt to bespoke tasks. Agentic CLEAR dynamically evaluates agents at system, trace, and node levels, producing textual diagnostics; across four benchmarks and seven configurations, it aligned with human-annotated errors and provided predictive signal for task success.
Problem
Existing evaluation tools use basic observability or rigid, hand-crafted categories that cannot dynamically adapt to bespoke agent tasks.
Method
Agentic CLEAR automatically generates structured textual diagnostics at system, trace, and node levels by evaluating execution traces and aggregating recurrent issues.
Results
Across four benchmarks and seven agentic configurations, Agentic CLEAR aligned with human-annotated errors and provided meaningful predictive signal for task success.
Takeaways & Limitations
Agentic CLEAR supports multi-level agent evaluation at scale without predefined error taxonomies or hand-crafted rubrics.
Takeaways & Limitations
Rubric-based evaluation can misrepresent success when tasks require implicit policy adherence, particularly adversarial tasks where correct behavior is refusal.
Abstract
from arXiv · showhide
Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.
1 Introduction
Agentic systems can fail through subtle modes, repeated loops, misaligned sub-agents, and error propagation that final outputs alone may not reveal. Agentic CLEAR addresses this gap with automatic, dynamic, multi-level evaluation that generates textual diagnostics and aggregates recurrent failures across traces.
- Agentic systems’ subtle failure modes, repeated loops, misaligned sub-agent behavior, and cross-step error propagation are difficult to detect from final outputs alone.
- Existing observability platforms mainly provide trace logging, basic metric aggregation, or coarse full-trace LLM judging, leaving developers to inspect many traces manually.
- Static error categories and hand-crafted evaluation engineering cannot dynamically adapt to bespoke tasks faced by everyday agent developers.
- Agentic CLEAR performs automatic, dynamic, granular evaluation, producing step-level and full-trace feedback before aggregating execution traces to surface recurrent failures and quality degradation.
- Agentic CLEAR is delivered as a pip-installable Python package with easy integration and an interactive UI for deep-dive trace analysis.
- Experiments across varied benchmarks, agents, and models show that Agentic CLEAR surfaces execution failures without human-engineered tests and provides actionable, high-level insights.
2 Agentic CLEAR Method
Agentic CLEAR evaluates agent execution traces at step and trace levels, then aggregates feedback into system- and node-level insights. This hierarchical pipeline links interpretable findings to the execution steps or traces that triggered them.
- Multi-Level Output: The hierarchical approach provides interpretable insights at node, trace, and system resolutions, giving developers visibility into agent behavior at different granularities.This spans fine-grained node and trace views through the full system view.
- Trace Generation: Agentic CLEAR runs a target agent across a dataset of tasks, producing traces composed of LLM input-output pairs generated by distinct agent nodes.The traces are denoted T = {t_n}^N_{n=1}, with each trace containing a sequence of calls attributed to nodes.
- Trace Evaluation: The first evaluation stage uses LLM judges to score and critique individual steps and assess the quality of each complete trace.Step-wise evaluation examines each input-output pair, while trace-wise evaluation considers the complete execution trace.
- Insight Aggregation: CLEAR clusters and summarizes instance-level feedback to identify component-specific node failures and holistic system behaviors.Node-level aggregation groups input-output pairs by node, while system-level aggregation combines trace-level judgments.
- Insight Attribution: Each aggregated insight is linked to the specific execution step or trace that triggered it.These links connect global findings back to the underlying agent behavior.
3 Agentic CLEAR Framework
Agentic CLEAR is a modular, extensible evaluation pipeline distributed as a Python package and designed to analyze LLM interactions from standardized traces. Its interface supports hierarchical diagnosis through system-, node-, and trace-level views with generated scores, issues, and justifications.
- Integration and customization: Agentic CLEAR supports independent or combined end-to-end evaluation levels, allowing users to tailor workflows to their evaluation needs.The package is available on PyPI under a permissive Apache 2.0 License.
- Integration and customization: OpenTelemetry-compatible, LangFuse-formatted traces are converted into an intermediate representation capturing LLM inputs, outputs, and required metadata.Other trace formats require only minimal preprocessing to reach the same intermediate state.
- Evaluation design: Judge modes evaluate step-level criteria, trace-level execution and deliverable quality, or dynamically generated task-specific rubrics.Each prompt requests a brief textual justification before the score, while users can customize dimensions, prompts, or the judge implementation.
- Interactive dashboard: The dashboard provides system, node, and trace perspectives for reconstructing topologies, surfacing recurring issues, filtering localized errors, and inspecting execution details.Trace View includes overall evaluations, step-level dimension scores, rubric evaluation, and the judge’s natural-language reasoning.
4 Experimental Setup
Agentic CLEAR is evaluated on execution traces spanning four benchmarks and seven settings, using traces consolidated into a unified representation. Two judge models evaluate every trace dataset, after which CLEAR aggregates evaluations for issue discovery.
- Trace datasets: The evaluation uses traces from SWE-Bench Verified Mini, GAIA, AppWorld, and τ 2-Bench, generated by leading agent architectures and LLMs.
- Trace datasets: Seven settings support comparative analyses across models, agents, and benchmarks, with traces consolidated into a unified intermediate representation schema.
- Judge models: The judges are OSS-120B in high thinking mode, representing a leading open-source model, and GPT-5, representing a closed-source model.
- Evaluation procedure: Trace-wise evaluations run across all seven trace datasets using both judge models, then pass to CLEAR aggregation for issue discovery.
5 Agentic CLEAR Issues Results
Agentic CLEAR reveals recurring systemic weaknesses alongside benchmark-, agent-level, and node-specific failure patterns. Its diagnoses vary with the backbone model and judge, while exposing actionable pitfalls such as planning-stage hallucinations and execution bugs.
- Universal Error Patterns: 195 trace-level issues revealed recurring weaknesses, including redundant tool usage and insufficient error handling or recovery.These patterns reflect systemic weaknesses across current agent systems.
- Domain-Specific Issues: Benchmark-specific analysis found distinct weaknesses, with GAIA dominated by sourcing and verification failures.The passage frames these as domain-specific issues beyond shared errors.
- Domain-Specific Issues: Planning nodes showed task-decomposition and API-selection issues, whereas execution nodes surfaced functional bugs such as missing pagination handling.The evaluation pinpoints role-specific pitfalls behind failure modes.
- Domain-Specific Issues: Hallucinations occurred mainly during planning, such as listing APIs absent from the supplied catalog, rather than during execution.This localization helps developers fine-tune the relevant components.
- Backbone Model and Agent Differences: GPT-4.1 and Claude 4.5 Sonnet shared most HAL system-level failures on GAIA, including source-verification gaps, tool misuse, and output-format noncompliance.Both backbones produced nearly identical output-compliance issues in the reported comparison.
- Judge Selection: 67 vs. 130 characters on average, OSS-120B generated shorter, broader, and more generic issues than the other judge.Both judges uncovered diverse recurring issues, but differed in depth, specificity, wording, and emphasized behavior.
6 Analysis
Agentic CLEAR aligns its taxonomy-free issues with human-annotated error categories and evaluates whether its scores predict trace success. It recovers broad error coverage, achieves strongest agreement with GPT-5, and shows benchmark- and method-dependent predictive performance.
- Error-category alignment: TRAIL’s 12 nonexecution categories cover 94% of ground-truth labels and provide the taxonomy used to evaluate Agentic CLEAR’s reasoning and planning issues.Agentic CLEAR semantically maps system-level issues to TRAIL categories using full or partial matches.
- Error-category alignment: All 15 GPT-5 issues and 12 OSS-120B issues map to TRAIL categories, covering 12 and 10 of the 12 relevant categories, respectively.The mapping was performed with Claude Opus 4.6 and verified by the authors.
- Error-category alignment: GPT-5 achieves the strongest full+partial agreement, with macro-F1 0.459 and micro-F1 0.497, while the frequency baseline reaches micro-F1 0.459 and macro-F1 0.199.The low macro-F1 of the frequency baseline reflects poor coverage of infrequent error categories; GPT-5 outperforms OSS-120B.
- Success prediction: AppWorld is most predictable, with every result above 0.75 and GPT-5 reaching at least 0.82 AUC across methods, whereas τ 2-Bench remains at or below 0.62.GAIA and SWE-Bench Verified Mini vary by method, agent, and model, motivating further research on trace judges.
- Success prediction: Trace-level scores are the strongest success predictors across configurations, generally outperforming rubric and step-wise methods, with GPT-5 usually exceeding OSS-120B.Rubric prediction can fail when requirements are implicit, while step-wise prediction is better suited to agents with distinct, self-contained components.
- Rubric analysis: Generated rubrics use only task descriptions and may describe workflows qualitatively, whereas AppWorld’s gold criteria are programmatic state checks against precomputed outcomes.For shopping tasks, rubrics capture retrieval, parsing, cart addition, and checkout, while gold constraints verify the resulting state.
7 Related Work
Agentic CLEAR is situated within emerging work on standardizing agent evaluation and benchmarks for evaluating judges. It differs from meta-evaluation approaches that rely on fixed error taxonomies and test judges’ classification of erroneous agent steps.
- General Agent Evaluation: Recent work emphasizes the importance of standardizing agentic evaluation and has taken initial steps toward achieving it.These efforts address runtime and execution layers across environment types and agent-evaluation protocols.
- General Agent Evaluation: Environment-agnostic agent evaluation remains an emerging direction addressed by Agentic CLEAR and related standardization efforts.The cited efforts span runtime and execution layers across environment types and standardized evaluation protocols.
- Agent Meta-Evaluation: Agent meta-evaluation benchmarks assess whether judges detect erroneous agent steps and classify them into predefined categories.This work extends broader research on meta-evaluation of LLMs.
- Agent Meta-Evaluation: Unlike these approaches, which assume a fixed error taxonomy, Agentic CLEAR is presented as operating beyond that fixed-taxonomy evaluation setup.The cited meta-evaluation works evaluate judges’ recovery of a predefined error taxonomy.
8 Conclusions · A Prompts
Agentic CLEAR is an automatic evaluation framework that generates scalable, multi-level textual insights into agent behavior without predefined taxonomies or hand-crafted rubrics. Across four benchmarks and seven configurations, it aligned with human-annotated errors and predicted task success, while future work targets execution analysis and judge reliability; prompts are available in the code repository.
- 8 Conclusions: Agentic CLEAR is an automatic evaluation framework for agent behavior.It produces multi-level textual insights at scale.
- 8 Conclusions: It avoids predefined error taxonomies and hand-crafted rubrics.This enables evaluation without manually specifying fixed error categories or scoring rules.
- 8 Conclusions: 4 benchmarks and 7 agentic configurations supported the evaluation study.The framework was demonstrated across four benchmarks and seven agentic configurations.
- 8 Conclusions: Agentic CLEAR aligned with human-annotated errors.The experiments demonstrated alignment between its feedback and human-annotated errors.
- 8 Conclusions: It provided meaningful predictive signal for task success.The conclusions identify task-success prediction as a demonstrated capability.
- 8 Conclusions: Future work includes analyzing system execution alongside reasoning and planning.This extends the framework’s analysis beyond reasoning and planning to system execution.
- 8 Conclusions: Future work also targets improved judge capabilities and reliability across settings.The paper identifies judge improvement and reliability across settings as additional directions.
- A Prompts: The prompts are presented in the code repository.The repository provides the prompts used by the paper.
B Data … B.3 Agents
The evaluation data span four agent benchmarks, two trace datasets, and three agents covering software engineering, digital assistance, customer service, general AI, and enterprise or research automation. These resources combine hidden-test evaluation, simulated-user interaction, real-world questions, and annotated execution traces.
- B.1 Benchmarks: SWE-Bench Verified Mini contains 50 human-validated Python software-engineering tasks evaluated through patches and hidden unit tests.Each task includes a GitHub issue and repository snapshot.
- B.1 Benchmarks: AppWorld evaluates user-assistance agents on realistic day-to-day digital tasks through executable Python code and AppWorld APIs, while τ 2-Bench evaluates policy-compliant completion and violation rejection.τ 2-Bench covers retail, airline, and telecom domains using LLM-simulated users.
- B.1 Benchmarks: GAIA comprises 466 human-designed real-world questions requiring abilities such as web browsing, multimodal handling, and multi-file handling.The benchmark evaluates general AI assistants.
- B.2 Traces Data: HAL provides diverse-domain execution traces for automated evaluation of hidden failure modes, agent-behavior issues, and unsafe real-world actions.It is designed to standardize agent benchmarking across domains.
- B.2 Traces Data: TRAIL contains execution traces with human-annotated errors for testing whether LLM judges identify reasoning, planning, and system-execution failures.The annotations use a predefined taxonomy.
- B.3 Agents: CUGA is an open-source enterprise-automation system using multi-agent orchestration, dynamic reasoning, and API integrations under strict policy compliance.It is designed to handle complex tasks.
- B.3 Agents: The HAL Generalist Agent operates across HAL’s unified evaluation framework, while Hugging Face’s Open Deep Research Agent navigates the web and produces citation-backed answers over long trajectories.The research agent synthesizes information for complex research queries.
C Issue Examples
Agentic CLEAR surfaces issues across configurations and analysis levels, revealing both universal and benchmark-specific system failures. Its multi-level analysis also distinguishes system-wide failure modes from finer-grained planning errors in individual nodes.
- Cross-benchmark system issues: Table 4 identifies shared system-level issues across GAIA and SWE-Bench Verified Mini, including inefficient workflows and tool misuse.Four of the top 10 issues are shared across both benchmarks.
- Cross-benchmark system issues: The remaining Table 4 issues are benchmark-specific, with GAIA exposing source-verification and data-processing problems.SWE-Bench Verified Mini instead surfaces engineering-oriented failures.
- Multi-level analysis: For the TaskDecompositionAgent on AppWorld, system-level analysis captures incomplete execution and entity-resolution errors.The system-level and node-level issues were generated from the same CUGA traces.
- Multi-level analysis: Node-level analysis pinpoints planning-stage errors, including wrong app assignments and unsupported capability assumptions.Some themes recur at both system and node levels, but with different granularity.
D Agentic CLEAR Issues to TRAIL Mapping
Agentic CLEAR’s system-level issues are mapped to the TRAIL taxonomy for outputs generated with GPT-5 and OSS-120B. Tables 6 and 7 provide these mappings for GAIA.
- System-level issue mapping: Tables 6 and 7 map Agentic CLEAR’s system-level issues to TRAIL error categories for GPT-5 and OSS-120B, respectively.Both mappings concern GAIA evaluations.
- GPT-5 mapping: Table 6 presents the mapping from GPT-5 system-level issues on GAIA to TRAIL error categories.The categories include language-only, tool-related, misinterpretation, identification, and non-compliance errors; “—” denotes no full match.
- OSS-120B mapping: Table 7 presents the mapping from OSS-120B system-level issues on GAIA to TRAIL error categories.The table uses the same abbreviations as Table 6.