PaperlayerView exact revision on arXiv

Source-linked AI summary

SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation

Chenyang Zhu, Jiayu Yao, Kushal Chawla, Youbing Yin, Nathan Wolfe, Pengshan Cai, Jingyu Wu, Spencer Hong, Sangwoo Cho, Shi-Xiong Zhang, Daben Liu, Sambit Sahu, Erin Babinsky

arXiv:2606.24626v1cs.AI

TL;DR

Long agentic traces challenge fault attribution because trajectories can exceed model context windows. SAFARI actively investigates trace segments with tools and iterative hypotheses. It outperforms existing methods on the reported benchmarks and retains precision when faults lie beyond the model's context window.

  • Problem

    Fault attribution remains difficult on long agentic trajectories, which can exceed model context windows and coincide with benchmark performance drops as trace length increases.

    Source: §1 · p. 1

  • Method

    SAFARI uses an Investigator to query trace segments iteratively, form hypotheses, and verify findings rather than passively ingesting the full trajectory.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    SAFARI outperforms state-of-the-art methods by 20% on Who&When and by 19% on TRAIL GAIA, and maintains 0.58 precision at ς = 5.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Active Investigation offers a robust alternative to linear context ingestion for fault attribution, including when the target fault lies beyond the model's native context window.

    Source: §5 · p. 4

    At a 1M-token context budget, SAFARI performs slightly worse than RAFFLES and its own 200k-budget variant, partly because STM summarization can lose information when raw logs fit in context.

    Source: §4 · p. 4

Abstract

from arXiv · show

As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows. Current methods for effectively diagnosing agent failures load the full trajectory into an LLM's context window, which suffers from attention dilution and fails when agentic traces inevitably exceed context limits. To address this, we introduce SAFARI (Scaling long-horizon Agentic Fault AttRibution via active Investigation), a framework that replaces linear context loading with a tool-augmented diagnostic loop. By equipping LLMs with a specialized toolbox to read and search trajectory segments alongside a persistent Short-Term Memory (STM) for cross-turn reasoning, SAFARI effectively decouples diagnostic accuracy from architectural context limits. Our experiments demonstrate that SAFARI outperforms state-of-the-art results by 20% on the Who&When dataset within a 1M token budget, and by 19% on TRAIL GAIA subset on a 25K token budget. Most significantly, SAFARI maintains a 0.58 precision even when the target fault resides 5x beyond the model's native context window, a scenario where traditional evaluators fail entirely.

1. Introduction

Longer agentic trajectories make fault attribution increasingly difficult, with existing systems losing performance as horizons grow. SAFARI addresses this with active, hypothesis-driven investigation rather than passive trajectory ingestion.

1. Introduction

  • Fault detectors lose performance on long-horizon Who&When and TRAIL trajectories, some of which exceed the context windows of available models by millions of tokens.The paper identifies scalable, cost-effective fault detection as a need for these longer traces.Source: §1 · p. 1
  • Prior approaches include passive processing of full trajectories and active methods, but the paper notes continuing challenges in correctly locating faults over long horizons.The cited active-investigation work motivates strategically storing and retrieving information instead of ingesting the entire context.Source: §1 · p. 1
  • SAFARI investigates traces through iterative, hypothesis-driven exploration, using specialized search tools and expert judges to verify findings.The paper evaluates it on Who&When and TRAIL and reports gains over existing methods in resource-constrained settings.Source: §1 · p. 2

2. Methodology: SAFARI

SAFARI attributes faults by selectively investigating trajectory segments instead of loading the full trace into the model context. Its loop combines trace-navigation tools, claim verification, and persistent short-term memory.

2. Methodology: SAFARI

  • The task is to identify the earliest uncorrected error in a trajectory by gathering evidence through selective queries rather than ingesting the entire trace.Initial prompts provide the trace length, investigation metadata, and agent-system data structures.Source: §2 · p. 2
  • The Investigator uses read(offset, limit) to traverse segments and search(pattern) to find matching spans, values, or error signatures.These tools support navigation without exhausting the model's context window.Source: §2 · p. 2
  • Before concluding, the Investigator submits up to three atomic claims for evaluators to check against quoted evidence, and weakly supported claims prompt further investigation.The default confidence threshold for concluding is 70.Source: §2 · p. 2
  • SAFARI preserves the task goal, hypotheses, evidence, investigative gaps, and past tool calls in short-term memory as the conversation history grows.The loop ends with a supported prediction or after 30 iterations, when it forces a final answer from the gathered evidence.Source: §2 · p. 2

3. Experimental Setup

The experiments use Who&When and TRAIL, comparing SAFARI with long-context and iterative fault-attribution baselines. Two evaluation protocols test performance under fixed context budgets and when fault positions exceed those budgets.

3. Experimental Setup

  • Who&When and TRAIL provide complementary trajectory challenges, including token-dense TRAIL/SWE-bench traces and TRAIL/GAIA traces reaching 122 steps.Who&When includes algorithm-generated and hand-crafted subsets, with the latter having longer trajectories.Source: §3 · pp. 2–3
  • Baselines include full-trajectory one-shot prompting, step-by-step scanning, and RAFFLES; evaluation reports step-level and tolerant accuracy, plus TRAIL-specific precision measures.TRAIL's metrics account for its labels of multiple faults rather than one decisive fault.Source: §3 · p. 3
  • The scaling factor ς is the root-cause fault's token position divided by the evaluator's context budget; values near or above 1 indicate a context ceiling or attention dilution risk.Posinfo is the fault's token position, and Lencontext is the evaluator's token budget.Source: §3 · p. 3
  • To test long-horizon capability, the study either fixes the context budget across trajectories or fixes ς and adjusts context length proportionally to fault position.The experiments use Claude-Opus-4.6 with a 1M-token context as the primary backbone; most dataset faults occur within the first 200k tokens.Source: §3 · p. 3

4. Results

SAFARI’s results span fixed-budget and beyond-context tests across TRAIL and Who&When. Its advantages are strongest in constrained settings, with a limitation when raw logs fit within the context window.

Fixed Budget Performance

  • At a 25K token budget on TRAIL (GAIA), SAFARI outperforms RAFFLES by 7% in precision and 19% in strict precision.Figure 2 compares performance across datasets, budgets, and metrics.Source: §4 · p. 3, Figure 2 · p. 4
  • SAFARI with a 100K token context window achieves comparable performance to RAFFLES with a 1M token context window.SAFARI navigates the trace to extract evidence.Source: §4 · p. 4
  • At a 1M token budget, SAFARI performs slightly worse than RAFFLES and its own 200k-budget variant.The authors attribute this partly to the STM trade-off between accurate summarization and information loss when raw logs fit in context; potential conflicts between raw logs and STM remain for future study.Source: §4 · p. 4
  • On Who&When, SAFARI exceeds state-of-the-art results by 20% on the Hand-crafted subset, even when traces fit within the context window.The reported improvements over RAFFLES are attributed to the flexible Atomic Reasoning Evaluator and short-term memory.Source: §4 · p. 4
  • The STM acts as a logical anchor that preserves investigative intent and supports coherent planning across turns.This role is shown through example reasoning traces in Appendix E.Source: §4 · p. 4

Fixed Scaling Factor

  • At a scaling factor of ς = 5 on TRAIL (GAIA), SAFARI achieves 0.58 precision; it degrades on the more challenging SWE-Bench subset.Table 2 reports performance across fixed scaling factors for TRAIL GAIA and SWE-Bench.Source: §4 · p. 4, Table 2 · p. 4

5. Conclusion

SAFARI uses active investigation as an alternative to linear context ingestion for fault attribution. It outperforms state-of-the-art methods on Who&When and constrained-budget TRAIL GAIA, and retains precision when faults lie beyond the model’s native context window.

  • SAFARI outperforms state-of-the-art methods by 20% on Who&When with a 1M token budget and by 19% on TRAIL GAIA with a 25K token budget.Source: §5 · p. 4
  • SAFARI maintains 0.58 precision when the target fault is 5x beyond the model’s native context window, where traditional evaluators fail entirely.Source: §5 · p. 4
  • Active Investigation selectively queries and reasons over trajectory segments as an alternative to linear context ingestion for fault attribution.Source: §5 · p. 4

A. Performance Benchmark of SAFARI

The appendix benchmark tables cover TRAIL and Who&When performance across context budgets, context sizes, and methods.

  • Table 3 compares methods and context budgets on TRAIL benchmarks.Source: Table 3 · p. 6
  • Table 4 reports Who&When fault-attribution step-level accuracy across context sizes and methods.Source: Table 4 · p. 6

B. Latency Analysis

SAFARI’s latency adapts to the token budget: it converges with baseline speeds at high budgets and trades latency for precision at low budgets. Its latency also scales with trajectory complexity.

B. Latency Analysis

  • At 25k tokens, SAFARI trades latency for precision by iteratively synthesizing information across disjoint trajectory segments.At 1M tokens, it converges with baseline speeds through retrieval throughput and parallel evidence verification.Source: B. Latency Analysis · p. 7
  • At 1M tokens, SAFARI converges with baseline speeds by maximizing retrieval throughput and parallelizing evidence verification.Source: B. Latency Analysis · p. 7
  • Across GAIA and SWE-bench, SAFARI’s latency overhead remains proportional to task complexity.Table 5 reports latency in seconds across the two TRAIL subsets, token budgets, and methods.Source: B. Latency Analysis · p. 7, Table 5 · p. 7

C. Ablation studies

The ablation compares full SAFARI with a version that excludes STM, showing that STM’s contribution depends on the dataset and token budget. Its advantage is clearest for GAIA across budgets and for token-dense SWE-bench traces at 100k tokens.

C. Ablation studies

  • The ablation removes STM, leaving the agent reliant on raw conversation history; evicted tool observations are lost unless retrieved again.That re-retrieval can consume available turn budget.Source: C. Ablation studies · p. 7
  • On TRAIL GAIA, STM improves attribution across all evaluated token budgets.Table 6 compares full SAFARI with SAFARI ex. STM; the caption reports successful-attribution counts and notes N = 31 for SWEBench.Source: C. Ablation studies · p. 7, Table 6 · p. 7
  • On SWE-bench, STM’s impact is less pronounced at 1M and 200k tokens but becomes significant at 100k.These traces have fewer steps but are exceptionally token-dense, creating a trade-off between accurate summarization and information loss.Source: C. Ablation studies · p. 7
  • At 100k tokens, the persistent memory buffer’s benefit outweighs summarization loss, allowing full SAFARI to outperform the ablated version.Source: C. Ablation studies · p. 7

D. Future Work

Future work proposes testing SAFARI beyond fault attribution, including long-horizon tasks and domains where errors are defined by human alignment or semantic intent.

D. Future Work

  • One proposed direction is evaluating SAFARI on general-purpose long-horizon tasks beyond fault attribution.Source: D. Future Work · p. 8
  • The authors also propose extending SAFARI to open-ended reasoning and creative synthesis.Source: D. Future Work · p. 8
  • In those domains, errors are defined through human alignment and semantic intent rather than binary success metrics.Source: D. Future Work · p. 8

E. Example reasoning traces of SAFARI

The examples show the Investigator maintaining open questions in Short-Term Memory, using targeted searches, and evaluating candidate faults through separate checks. One TRAIL GAIA trace verifies step 3 as a fault; another follows an uncorrected error while noting uncertainty about quoted trace text.

E. Example reasoning traces of SAFARI

F. Prompts

The prompts guide investigators to trace failures backward through agent trajectories, identify and verify the earliest uncorrected causal fault, and preserve and evaluate evidence across context limits before submitting an attribution.

F. Prompts

  • The task is to return the single first-mistake step ID, using only actual trace evidence and supplying both the step ID and reasoning when submitting.The prompt prohibits fabricated IDs or tool results and requires a valid integer from the trace.Source: F. Prompts · pp. 10–11
  • When conversation history is capped, the prompt makes concise STM the means of carrying forward key facts, step IDs, verbatim evidence, and causal chains.It advises recording important findings early, restating critical evidence in each evaluation block, and not assuming earlier information remains in context.Source: F. Prompts · p. 10
  • Tool use is constrained to one call per turn, with the investigator required to wait for results, avoid duplicate calls, and reserve turns for evaluation and submission.The prompt also instructs investigators to track remaining turns and skip further investigation when only two turns remain.Source: F. Prompts · pp. 11–13
  • The prompt requires targeted search rather than sequential scanning, followed by reading each candidate step’s full content and confirming its step ID before judgment or submission.Investigators are told to search for a candidate step, read its specific trace region, and avoid guessing IDs or repeating identical calls.Source: F. Prompts · pp. 11–12
  • A decisive fault must be genuine, the earliest fault that caused failure, and uncorrected; investigators should trace backward from the visible wrong output to its cause.The prompt distinguishes the point where failure becomes visible from where it originated, and asks whether correcting the candidate would have changed the final outcome.Source: F. Prompts · p. 12
  • Ask at most three atomic questions; the judge returns confidence scores, and an average of at least 70 permits submission, while lower scores require new evidence and another evaluation.Source: F. Prompts · p. 13

Investigation strategy

  • The investigation proceeds from the final output to the causal origin, then confirms the earliest genuine fault that propagated without correction.Investigators first state the final output, task requirement, and failure; they search backward using specific wrong values, actions, or entities, then directly read candidate steps to verify the fault.Source: F. Prompts · p. 14
  • Before submission, evaluation reasoning must quote the candidate action, establish why it was wrong, and show it was neither preceded nor corrected by another fault.If the score is below 70, investigators gather new evidence for low-scoring questions and evaluate again; after three calls, they submit their best answer.Source: F. Prompts · p. 14

Listing 5. Investigator Agent User Prompt

  • When a trace exceeds the context window, the prompt warns that older messages may be dropped and directs investigators to preserve key information in short-term memory.When the trace fits, it instead recommends reading large contiguous sections first to build a holistic picture.Source: F. Prompts · p. 15
  • The submitted step must be the assistant’s wrong action, not the following observation that reveals the failure.The prompt specifies a zero-based step_id for the first faulty span or message.Source: F. Prompts · p. 16
Loading 2606.24626v1…