Source-linked AI summary
When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation
Lauren Pothuru
TL;DR
Agentic RAG can transform an early retrieval fault into later query drift, a downstream answer error, or recovery, making causal attribution difficult from post-hoc traces. AgenticRAG-FP injects a certified fault at a specified hop, re-executes the suffix, and evaluates diagnosers against that known intervention. In the completed structural sweep, coverage-based attribution was 0.91 at hop 1 but 0.00 at hops 2 and 3, while the smaller content study showed exploratory evidence favoring frozen-hop repair at depth 2.
Problem
The paper asks whether post-hoc traces identify a known injected hop after downstream propagation or recovery, rather than merely revealing a later symptom.
Method
AgenticRAG-FP injects a certified fault at a selected hop, allows the agent to generate the downstream suffix, and scores diagnoses against the retained intervention label.
Results
0.91 at hop 1 and 0.00 at hops 2 and 3: coverage-based attribution loses its post-hoc signal in the completed strict dense MuSiQue sweep.
Takeaways & Limitations
Propagation depth should be an explicit evaluation axis, while content-fault method comparisons should be distinguished from broader evidence of post-hoc signal loss.
Abstract
from arXiv · showhide
Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the completed strict dense Claude Haiku 4.5 sweep on 80 three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories). A smaller content-corruption study changes an answer-bearing or bridge fact in topically intact evidence. At depth 2, where 18 failed cases remain after filtering, coverage-based diagnosis is 0.00 and a frozen-hop counterfactual probe is 0.67 in an exploratory pooled comparison. Depth-3 content estimates are descriptive only because they contain three failed cases. These results make propagation depth an explicit evaluation axis for diagnosing agentic RAG failures while distinguishing broad evidence of post-hoc signal loss from small-sample method comparisons.
1 Introduction
AgenticRAG-FP introduces a certified interventional benchmark to test whether diagnosers can recover an injected failure hop after downstream propagation or recovery. Its completed structural results show strong hop-1 attribution but complete coverage-signal loss at later hops, while content-fault probe comparisons remain bounded.
- AgenticRAG separates visible end-to-end errors from their causes because early retrieval faults can drift later queries, propagate, or be repaired downstream.
- AgenticRAG-FP corrupts a trace prefix at hop h, re-executes the remaining trajectory, and scores diagnoses against the certified intervention.
- The benchmark supports structural retrieval faults and certified content corruption with deterministic absorbed, resisted, and derailed labels.
- 0.91 at hop 1 and 0.00 at hops 2 and 3: coverage-based attribution loses its post-hoc signal across later structural depths.This pattern concerns information available after suffix re-execution rather than end-to-end task accuracy.
- The study compares frozen-hop repair with suffix regeneration for content faults, treating depth-2 results as illustrative and depth-3 results as descriptive.
2 Related Work
Prior work provides retrieval, active-agent, diagnosis, and multi-hop benchmark foundations, but standard traces and answer labels do not certify which hop caused a failure. AgenticRAG-FP addresses this gap through controlled causal intervention.
- RAG and active retrieval make evidence selection responsive to uncertainty or evolving generation, so early retrieval changes can alter later questions.
- Trace inspection and repair methods can localize suspicious hops, but inspection alone cannot establish whether a hop caused the final answer.
- HotpotQA, MuSiQue, FRAMES, and CRAG supply multi-hop or dynamic question-and-corpus substrates, but their standard labels do not identify the causal hop.
- AgenticRAG-FP supplies the missing causal label through a controlled intervention.
- The intervention perspective fixes a fault and hop before trajectory observation, making recovery informative rather than treating it as an evaluation nuisance.
3 Task and Trace Model
The trace model represents each multi-hop agent trajectory as question, hop-level queries and evidence, answer, reference answer, and cost, with explicit failure stages and certified-hop identifiability. The framework distinguishes causal attribution from merely detecting an incorrect answer or broad failure type.
- Each example pairs a question and reference answer with a retrieval corpus, from which the agent generates a multi-hop trace.
- A trace records each hop’s sub-query and retrieved evidence, plus the final answer, reference answer, and total token cost.
- Each hop is assigned retrieval, tool, answer, or none as its failure stage.
- Retrieval failures include empty or irrelevant evidence, query drift, false-premise evidence, stale evidence, and wrong facts in relevant documents.
- A diagnoser is identifiable at depth h when it reliably recovers the injected hop, a stricter criterion than detecting a wrong answer or broad failure stage.
- The trace’s causal target is the injected hop rather than a downstream symptom, because certification distinguishes causes, effects, and benign consequences.
4 Interventional Benchmark
AgenticRAG-FP uses live interventions that alter a selected hop and let the agent generate a new downstream suffix, preserving the certified target while exposing propagation and recovery. It combines structural and content faults with post-hoc and counterfactual diagnosers.
- A live intervention replaces evidence or a sub-query at hop h, then resumes the agent so the downstream suffix responds to corrupted context.
- The benchmark retains the failure family and selected hop while allowing suffix changes across retrieval, query, and termination interventions.
- Intervention families cover missing evidence, misleading evidence, and altered agent decisions, with different visibility to post-hoc inspection.
- Recovery remains an evaluated outcome because it measures resilience, while failed trajectories are the subset used for root-cause localization.
- Content corruption changes one fact while preserving topical relevance, modeling stale, corrupted, or upstream-extraction errors without obvious off-topic retrieval.
- Answer facts are prioritized for corruption, followed by bridge entities carried from hop evidence into later queries, with reproducible replacements.
- Frozen-hop repair holds other hops fixed, whereas suffix regeneration repairs a candidate and regenerates downstream evidence.
5 Metrics
The paper separates exact-hop attribution among failures that persist after live re-execution from recovery, answer-level corruption outcomes, and diagnosis cost. Secondary localization metrics provide graded alternatives to exact-hop scoring.
- Attribution identifiability: Exact-hop accuracy evaluates whether a diagnoser recovers the injected hop among trajectories that remain incorrect after suffix re-execution.Recovered trajectories are excluded because they contain no final failure to attribute.
- Localization beyond exact hop: Stage accuracy, hop-tolerance, ancestor-hit rate, and mean absolute hop error provide secondary localization diagnostics beyond exact-hop accuracy.These metrics distinguish broad failure recognition from precise intervention localization.
- Counterfactual recovery: Recovery measures the fraction of live interventions that still produce a correct answer, independently of successful fault localization.High recovery indicates that the intervention did not propagate into an end-to-end failure.
- Deterministic generation outcomes: Content faults receive deterministic absorbed, resisted, or derailed labels based on whether the generated answer contains the changed span and remains correct.Correctness takes precedence when an answer contains both gold and corrupted values.
- Diagnosis cost: Cost per correct diagnosis divides total diagnosis tokens by correct localizations, separating localization quality from online deployability.An accurate method may still be unsuitable when correct localization requires disproportionate re-execution or judging.
6 Experimental Setup
Experiments evaluate live structural and content interventions across multi-hop datasets, model backbones, retrieval settings, and propagation depths. Diagnosers receive the same post-intervention trace, while active probes generate their own counterfactual repairs.
- Datasets and corpora: The benchmark uses HotpotQA and MuSiQue, with FRAMES and CRAG providing additional multi-hop and RAG substrates.HotpotQA and MuSiQue include annotated supporting facts; FRAMES uses Wikipedia passage corpora and CRAG includes dynamic and false-premise questions.
- Backbones and retrieval: The headline structural condition uses Claude Haiku 4.5, dense retrieval, three-hop MuSiQue examples, and a four-probe budget.BM25, GPT-4o-mini, and local-model conditions are retained only where complete and sufficiently populated; static controls are appendix controls.
- Structural interventions and depth eligibility: Structural faults inject empty, irrelevant, false-premise, or stale evidence at hops 1–3 before re-executing the downstream suffix.A depth-specific case is eligible only when the base trace answered correctly and the actual injected hop matches the requested depth.
- Content-corruption conditions: Content corruption evaluates GPT-4o-mini and Claude Haiku 4.5 on HotpotQA and MuSiQue with BM25 across depths 1–3.Each condition begins with n=40 base examples, and traces are cross-family judged to reduce self-diagnosis.
- Evaluation units: All diagnosers receive the same post-intervention trace, while active probes create counterfactual repairs whose evidence counts as part of diagnosis.For content conditions, span selection distinguishes answer-fact, bridge-entity, and salient-fallback corruptions.
7 Results
Results show that post-hoc coverage signal declines sharply with structural propagation depth, while content-fault probe comparisons remain exploratory. Deterministic answer outcomes further show that corruption can be absorbed, resisted, or derailed.
- Structural attribution across depths: The strict dense Claude Haiku 4.5 MuSiQue sweep reports loss of post-hoc coverage signal after suffix re-execution across the requested depths.Table 2 reports accuracy with bootstrap 95% intervals and per-depth counts of failed injected trajectories.
- Counterfactual probe performance: 0.00 coverage and 0.67 frozen-hop repair are observed at content-fault depth 2 in an exploratory pooled comparison over 18 failed cases.Depth-3 content estimates use only three failed cases and are reported descriptively.
- Counterfactual probe performance: At depth 2, Suf-Regen is 0.11 because suffix regeneration can retrieve clean evidence and remove the content fault under test.A descriptive bridge-entity slice reports Suf-Regen 1.00 versus Propagation-Aware 0.73 with n=11.
- Answer outcomes after content corruption: 0.15 of hop-1 content-fault cases are absorbed verbatim, 0.58 resisted, and 0.26 derailed.These deterministic outcomes cover all content-fault interventions, including trajectories that recover.
- Answer outcomes after content corruption: 20 of 22 absorbed cases are answer-fact corruptions, while salient-entity corruptions are absorbed in 0 of 40 cases.Query contamination occurs in only 3 of 129 cases, so changed values usually affect reasoning rather than later sub-queries.
8 Discussion
Suffix re-execution can erase the local signature of an injected fault, making exact-hop attribution depend on counterfactual scope and downstream information flow. Frozen-hop and suffix-regeneration probes therefore answer different causal questions, while recovery must remain distinct from localization.
- Coverage-based localization can lose the injection signature after suffix re-execution because later retrieval may compensate for or propagate the original fault.The final trace then reflects the later trajectory rather than the local intervention.
- A recovered answer does not imply that the injected fault was absent; it can indicate that later retrieval repaired the trajectory without restoring an identifiable trace signature.Answer correctness and diagnosability are separate outcomes.
- Frozen-hop repair tests whether changing one candidate hop alters the answer with later hops fixed, whereas suffix regeneration rebuilds downstream hops after repairing the candidate.The probes preserve different parts of the causal chain.
- Frozen-hop repair reached 0.67 versus 0.11 for suffix regeneration at depth 2, but the small samples support a mechanism-oriented rather than definitive comparison.Frozen-hop repair can preserve downstream corruption, while suffix regeneration can remove it.
- Exact-hop evaluation distinguishes causal positions that answer-only evaluation would collapse into a single correctness outcome.
- Path-aware or cascaded diagnosis could select probes by suspected fault type and report uncertainty when multiple hops remain causally entangled.The appropriate probe depends on whether downstream dependence should be preserved or regenerated.
9 Limitations
The main structural comparison is concentrated in a strict dense Claude Haiku 4.5 MuSiQue sweep, while content-fault comparisons remain limited by recovery and small failed-case counts. The paper therefore avoids broad factorial or depth-3 comparative claims.
- The main structural sweep has adequate failed-case counts at all requested depths, but HotpotQA has only 9 and 2 failed trajectories at its valid depths.HotpotQA is treated as descriptive and excluded from the main comparison.
- The evaluation does not provide a complete backbone-by-dataset factorial comparison, and CRAG is excluded from headline results because it is normalized as single-turn.
- Clean-corpus content corruption allows later re-retrieval to heal faults, leaving 18 failed cases at depth 2 and 3 at depth 3.The depth-2 comparison is exploratory, and no depth-3 method comparison is made.
- Future content studies should pre-specify larger failed-case targets and include persistent corpus corruption to separate clean-index recovery from recovery caused by later reasoning.Additional limitations include strict exact-hop scoring, judge sensitivity, positional priors, and corpus-construction effects.
10 Conclusion
AgenticRAG-FP evaluates causal attribution by injecting certified faults and allowing downstream trajectories to change. Its main structural result shows post-hoc coverage signal loss at deeper hops, while content-fault comparisons remain exploratory and depth-3 estimates are descriptive only.
- AgenticRAG-FP injects a known fault and evaluates diagnoses against the resulting downstream response.
- Coverage-based attribution is 0.91 at hop 1 and 0.00 at hops 2 and 3 in the completed strict dense Claude-MuSiQue sweep.The corresponding failed-trajectory counts are 43, 36, and 21.
- The evaluated methods include rule-based diagnosis, Doctor-RAG, LLM-Judge, Propagation-Aware frozen-hop repair, and Suf-Regen suffix regeneration.
- Only three content-corruption trajectories remain failed at depth 3, so those point estimates support no comparison or mechanism claim.
C Additional content-fault outcome details
Additional content-fault results show recovery varies by fault type and that depth-2 curves distinguish Propagation-Aware from coverage gating. Depth-3 estimates are omitted when no injected trace remains failed.
- Content-fault recovery ranges from 0.54–0.85, compared with 0.00–0.71 for structural faults.Hop-1 absorbed rates are 0.18 / 0.19 / 0.11 / 0.11 across the four conditions.
- At hop 2 on both HotpotQA BM25 backbones, Propagation-Aware reaches 0.75 while coverage gating and suffix regeneration are zero.Hop 3 is omitted when no injected trace remains failed.
- At depth ≥2, the discordant-pair comparison contains 1 versus 6 pairs, too few for a McNemar test.The direct comparison is therefore Propagation-Aware versus coverage gating, not versus all post-hoc methods.