PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents

Xuannan Liu, Xiao Yang, Zekun Li, Peipei Li, Ran He

arXiv:2601.06818v1cs.CL

TL;DR

LLM-agent hallucinations can propagate through multi-step workflows, but single-turn detection does not identify the responsible step or explain its cause. AgentHallu introduces a benchmark and attribution task for this gap, finding that even leading models achieve limited localization accuracy, especially for tool-use hallucinations.

  • Problem

    Single-turn hallucination evaluations do not identify where or why hallucinations originate in sequential agent workflows, where intermediate errors can propagate into final answers.

  • Method

    AgentHallu provides 693 high-quality trajectories spanning 7 frameworks and 5 domains, with 5 categories, 14 subcategories, and human annotations for judgments, responsible steps, and causal explanations.

  • Results

    The best-performing model achieves 41.1% step-localization accuracy, while tool-use hallucinations reach only 11.6%.

  • Takeaways & Limitations

    Evaluations of 13 leading LLMs show that automated hallucination attribution remains challenging across hallucination categories, prompting methods, and trajectory lengths.

  • Takeaways & Limitations

    The benchmark may not capture emerging hallucination patterns as agent frameworks evolve and currently focuses on text-based rather than multimodal trajectories.

Abstract

from arXiv · show

As LLM-based agents operate over sequential multi-step reasoning, hallucinations arising at intermediate steps risk propagating along the trajectory, thus degrading overall reliability. Unlike hallucination detection in single-turn responses, diagnosing hallucinations in multi-step workflows requires identifying which step causes the initial divergence. To fill this gap, we propose a new research task, automated hallucination attribution of LLM-based agents, aiming to identify the step responsible for the hallucination and explain why. To support this task, we introduce AgentHallu, a comprehensive benchmark with: (1) 693 high-quality trajectories spanning 7 agent frameworks and 5 domains, (2) a hallucination taxonomy organized into 5 categories (Planning, Retrieval, Reasoning, Human-Interaction, and Tool-Use) and 14 sub-categories, and (3) multi-level annotations curated by humans, covering binary labels, hallucination-responsible steps, and causal explanations. We evaluate 13 leading models, and results show the task is challenging even for top-tier models (like GPT-5, Gemini-2.5-Pro). The best-performing model achieves only 41.1\% step localization accuracy, where tool-use hallucinations are the most challenging at just 11.6\%. We believe AgentHallu will catalyze future research into developing robust, transparent, and reliable agentic systems.

1 Introduction

AgentHallu frames hallucination attribution as identifying where an agent’s initial hallucination arises and why, beyond binary single-turn judgments. It introduces a diverse, manually annotated benchmark and evaluates leading models under challenging attribution conditions.

  • Motivation: Sequential agent workflows can propagate intermediate hallucinations into downstream tool calls and incorrect final answers.The paper illustrates this with a planning error that misdefines regions X, Y, and Z before tool use.
  • Motivation: Existing evaluations mainly classify single-turn responses as factual or hallucinated, leaving the responsible step and causal explanation unaddressed.AgentHallu defines attribution around the questions of where and why a hallucination originates.
  • AgentHallu: AgentHallu contains 693 trajectories from 7 agent frameworks across 5 domains, organized into 5 hallucination categories and 14 subcategories.The dataset applies three-stage filtering and includes binary labels, responsible steps, and plain-language causal explanations curated by humans.
  • Evaluation: The benchmark evaluates attribution using step localization accuracy and G-EVAL scores for open-ended explanations.These dimensions measure responsible-step identification and explanation quality, respectively.
  • Results: 41.1% step-localization accuracy is achieved by the best-performing model, while tool-use hallucinations reach only 11.6%.The evaluation covers 13 leading models, including 5 proprietary and 8 open-source systems.

2 Related Work

Prior hallucination work largely evaluates factuality or solvability in single-turn settings, whereas AgentHallu targets attribution within multi-step agent trajectories. Related agent research expands capabilities through reasoning, tool use, and multi-agent coordination.

  • Hallucination Detection: Hallucination detection research distinguishes factual from hallucinated content across world knowledge, science, and math domains.RAGTruth further reports hallucinations across tasks involving retrieval-augmented generation.
  • Hallucination Attribution: Prior work also studies solvability detection, solution planning, and missing-tool analysis, but remains confined to binary judgment in single-turn responses.AgentHallu introduces attribution for multi-step agent trajectories as a distinct task.
  • Agent Systems: LLM-based agents extend task automation through chain-of-thought, in-context learning, instruction following, and function calling.These mechanisms support capabilities beyond agents’ internal knowledge.
  • Agent Systems: Multi-agent systems compose specialized agents using structured debate, role-play dialogue, and central orchestration.Recent systems use orchestrators to assign tasks to specialized agents.

3 Task Formulation

The task models agent trajectories as sequences of thought, action, and observation units, judges hallucination by divergence from ground truth, and attributes the earliest causally sufficient error. Attribution uses counterfactual correction and rollout to identify the responsible step.

  • Trajectory Representation: Each interaction unit combines an internal reasoning state, an invoked tool action, and feedback from the tool response.The trajectory is represented as a sequence of these structured interaction units.
  • Trajectory Representation: The study restricts analysis to coherent, seemingly plausible answers rather than non-hallucination failures.This defines the scope of the trajectories considered in the task formulation.
  • Hallucination Judgment: A trajectory is judged hallucinated when its produced answer diverges from the task’s ground-truth solution.The judgment objective is binary.
  • Hallucination Attribution: A step is hallucinated when replacing it with its correct counterpart and rolling out subsequent steps changes an incorrect result into the ground-truth result.This counterfactual definition identifies steps whose correction is causally sufficient.
  • Hallucination Attribution: When multiple hallucinated steps exist, the initial error is treated as the primary source of hallucination.The objective therefore selects the earliest hallucination-responsible step.

4 AgentHallu Dataset

AgentHallu is a benchmark of annotated multi-step agent trajectories designed to analyze hallucinations through standardized trajectory structure, taxonomy, and human attribution labels. It combines diverse queries and agent frameworks with filtering and annotation procedures intended to support reproducible evaluation.

  • Dataset Overview: 693 annotated trajectories cover 443 hallucinated and 250 non-hallucinated instances across five domains and seven mainstream LLM-based agents.Each trajectory step is standardized as a thought, action, and observation triplet.
  • Dataset Overview: Each instance contains a query, a standardized multi-step trajectory, and human-curated binary, responsible-step, and causal-explanation annotations.Queries span world knowledge, science, math, general assistant, and tool use.
  • Trajectory Construction: The benchmark uses seven agent frameworks to generate diverse trajectories from queries drawn across five realistic domains.The framework set includes SmolAgents, OpenDeepSearch, OpenManus, OctoTools, Magentic-One, OWL, and function-calling agents.
  • Trajectory Construction: A three-stage filtering process removes non-completing failures, excessively short trajectories, and other unsuitable cases before benchmark retention.Short trajectories are excluded because they lack sufficient reasoning depth for step localization.
  • Annotation: The hallucination taxonomy is empirically grounded through pilot analysis and consensus refinement, while annotation proceeds from binary judgment to attribution and taxonomy classification.Ten graduate-level AI annotators iteratively label and refine cases, with disagreements resolved through discussion and peer cross-validation.
  • Evaluation: The evaluation reports hallucination judgment with F1, recall, and accuracy, and attribution with step-localization accuracy and G-EVAL.Table 4 evaluates LLMs under standard prompting using these distinct judgment and attribution metrics.

5 Experiments

Experiments show that hallucination attribution remains substantially harder than judgment, with performance worsening for longer trajectories and tool-use cases. Step-by-step prompting improves localization but substantially increases token cost.

  • Main Results: 41.1% localization accuracy is the best reported attribution result, while GPT-5 reaches 70.2% judgment F1.Gemini-2.5-Pro achieves 41.1% localization accuracy and a 2.4 G-EVAL score.
  • Main Results: Open-source models average 10.9% localization accuracy versus 35.9% for proprietary models.DeepSeek-V3.1 is the strongest open-source model at 19.2%, while the open-source average is comparable to random.
  • Main Results: Tool-induced hallucinations are consistently hardest to localize, whereas Gemini-2.5-Pro reaches 64.4% accuracy on reasoning hallucinations.The authors associate tool-use difficulty with verifying environmental state in action–observation loops.
  • Prompting Methods: Step-by-step prompting raises average attribution accuracy from 24.3% to 36.6%.It incrementally processes context for more focused localization, unlike standard prompting’s competitive judgment performance.
  • Prompting Methods: 17,514 average input tokens are required by step-by-step prompting, versus 6,312 for standard prompting.The increase reflects additional decisions from multi-turn prompting.
  • Trajectory Length: 29.9% to 11.4% is the average attribution decline from the shortest to longest trajectory levels.The levels contain trajectories with Nstep ≤5, 6 ≤Nstep ≤10, and Nstep ≥11, respectively; judgment also degrades as trajectories lengthen.
  • Dataset Comparison: AgentHallu produces about 18.2% lower average binary F1 than three existing hallucination detection datasets.The comparison uses HaluEval, FELM, and RAGTruth with three advanced LLMs.
  • Thinking Mode: Qwen3-32B judgment F1 rises from 51.8 to 58.6 with thinking mode, while attribution accuracy rises from 21.2 to 23.5.The paper attributes these gains primarily to enhanced self-verification.

6 Conclusion

The paper introduces automated hallucination attribution for LLM-based agents and AgentHallu, a benchmark for locating hallucination-responsible steps and explaining their causes. Evaluations show that attribution remains challenging across categories, prompting methods, and trajectory lengths.

  • Conclusion: AgentHallu contains 693 trajectories spanning 7 agent frameworks, 5 domains, 5 hallucination categories, and 14 subcategories.Its annotations include binary labels, responsible steps, and causal explanations curated by humans.
  • Conclusion: The benchmark evaluates automated identification of the initial hallucination step and explanation of why it occurs.The paper frames this as a distinct task for multi-step agent trajectories.
  • Conclusion: Evaluations on 13 leading LLMs reveal substantial variation across hallucination categories, prompting methods, and trajectory lengths.

Limitations

AgentHallu has two main limitations: its taxonomy may not keep pace with emerging agent behaviors, and its benchmark excludes multimodal interactions.

  • Scope and coverage: The taxonomy may not fully anticipate emerging hallucination patterns as agent frameworks, tools, and interaction protocols evolve.The authors recommend continuously expanding the dataset as new agent capabilities appear.
  • Scope and coverage: The benchmark focuses on text-based trajectories and does not cover agents grounded in images, audio, or other modalities.The authors identify multimodal extension as a direction for future work.

Ethical Considerations

AgentHallu is designed with privacy, content-safety, and research-use constraints by excluding sensitive data and using controlled or publicly accessible sources.

  • Data safeguards: AgentHallu contains no personally identifiable information or offensive content.The benchmark follows stated data-integrity and ethical standards.
  • Data safeguards: The benchmark uses publicly accessible datasets and repositories plus agent trajectories generated under controlled settings.Sensitive or restricted data sources are explicitly avoided.

A.1 Dataset and Code Release

The release provides AgentHallu’s dataset, source coverage, agent configurations, backbone diversity, licensing information, and reproducibility details. The benchmark contains 693 heterogeneous trajectories spanning multiple frameworks, domains, and interaction depths.

  • Dataset: AgentHallu contains 693 trajectories across one non-hallucination category, five hallucination categories, and fourteen subcategories.The category distribution is intentionally balanced for broad coverage of hallucination types.
  • Dataset: Queries include recurring tool-invocation and information-seeking patterns while covering diverse user intents rather than narrow prompt templates.The query distribution is visualized as a word cloud.
  • Dataset: Trajectory lengths exclude one- and two-step cases, extend to 43 steps, and are broadly distributed across interaction depths.This supports coverage of longer-horizon reasoning and diverse execution patterns.
  • Agent configurations: Seven representative agent frameworks generate trajectories spanning diverse reasoning paradigms and interaction patterns.The frameworks include SmolAgents, OpenDeepSearch, OpenManus, OctoTools, Magentic-One, OWL, and function-calling agents.
  • Agent configurations: The framework mix covers CodeAct and ReAct agents, search refinement, human-in-the-loop interaction, standardized tools, and coordinator-based multi-agent designs.These configurations are described across the included agent frameworks.
  • Data sources: The six general frameworks cover seven knowledge-intensive datasets, while function-calling agents are used exclusively for BFCL V3 queries.Together, the sources yield 693 trajectories across heterogeneous agent designs and data sources.
  • Backbone diversity: Multiple LLM backbones are used across frameworks to broaden behavioral diversity and reduce reliance on a single model family.The configurations include GPT, Claude, Qwen, and Llama backbones.
  • Release and licensing: The source datasets carry varied licenses, including MIT, Apache-2.0, and an unspecified license for GAIA.The release also states CC BY-NC-SA 4.0 terms for sharing and adapting AgentHallu itself.

B More Details on Evaluation

Evaluation uses deterministic LLM settings and compares standard versus incremental prompting across hallucination judgment, responsible-step localization, and causal-explanation quality. The protocol also examines trajectory length, evaluator alignment, and retrieval-subcategory performance.

  • Evaluation setup: Evaluation fixes sampling by disabling sampling or setting temperature to zero, with a maximum output length of 1024 tokens.Experiments run on eight NVIDIA GeForce A100 GPUs and are described as fully reproducible.
  • Prompting methods: Standard prompting evaluates the complete query and trajectory in one pass to judge hallucination, identify the earliest responsible step, and explain its cause.Step-by-Step prompting instead evaluates trajectory prefixes incrementally and stops at the first detected hallucination.
  • Evaluated metrics: Hallucination judgment uses macro-F1, macro-recall, and accuracy, while attribution reports responsible-step localization accuracy on hallucinated samples.Attribution is restricted to hallucinated outputs so non-hallucinated cases do not dominate localization scores.
  • Evaluated metrics: Causal-explanation quality is scored by GPT-5 using G-EVAL on a 1–5 ordinal scale against human-annotated explanations and trajectories.The score follows a fixed rubric measuring explanation accuracy.
  • Evaluator analysis: G-EVAL scores remain close to human ratings across five representative models, and GPT-5 explanations are not favored by the GPT-5 judge.This analysis probes potential evaluator bias.
  • Subcategory analysis: Retrieval hallucination analysis compares step-localization accuracy across three retrieval subcategories and six LLM judges.The comparison is presented in Figure 10.

C.2 More Analysis on Subcategories

The section analyzes hallucination subcategories, prompting templates, and representative attribution cases across five hallucination categories. It also describes the benchmark’s role in diagnosing where and why hallucinations arise in multi-step agent workflows.

  • Retrieval hallucination: Gemini-2.5-Pro achieves the strongest attribution performance across the three retrieval-hallucination subcategories, while query misalign remains challenging for all models.The analysis highlights Gemini-2.5-Pro’s advantage on summarize misalign, involving errors during evidence aggregation and compression.
  • Tool-use hallucination: All models perform poorly on most tool-use subcategories, whereas Claude-4.5-Sonnet identifies parallel conflict hallucinations with notably higher accuracy.The passage links the stronger parallel-conflict performance to explicit and verifiable contradictions from concurrent tool executions.
  • Prompting and evaluation: Step-by-step prompting evaluates each trajectory incrementally and stops once the model detects a hallucination.Standard prompting instead provides the query and full trajectory for joint judgment and attribution.
  • Prompting and evaluation: G-EVAL scores each causal explanation on a five-point ordinal scale using the trajectory and human-annotated explanation under a rubric prioritizing explanation accuracy.The evaluation therefore assesses explanations against human-annotated references rather than localization alone.
  • Qualitative analysis: Qualitative case analysis examines representative planning, retrieval, reasoning, human-interaction, and tool-use hallucinations from five models.The cases assess both hallucination identification and explanations of where and why hallucinations arise in agentic workflows.
  • Significance: AgentHallu supports systematic diagnosis of where and why hallucinations arise during multi-step agent execution, including settings with high-stakes downstream consequences.The benchmark is positioned as supporting reliability and transparency research for agentic systems.
Loading 2601.06818v1…