PaperlayerView exact revision on arXiv

Source-linked AI summary

RAFFLES: Reasoning-based Attribution of Faults for LLM Systems

Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Charlotte Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, Daben Liu

arXiv:2509.06822v3cs.AIcs.CL

TL;DR

Long-horizon, multi-component LLM systems are difficult to evaluate for where and when failures arise, while outcome-focused approaches and single-pass judges struggle to localize faults. RAFFLES uses an iterative Judge-Evaluator pipeline to assess candidate faults and rationales. It improves step-level attribution across agentic and mathematical-reasoning benchmarks, reaching 51.59% on Who&When Algorithmically-Generated.

  • Problem

    Long-horizon, multi-component LLM systems expose failures that outcome-focused metrics and single-pass judges struggle to localize, while manual diagnosis is costly.

    Source: §1 · p. 1

  • Method

    RAFFLES is an offline iterative Judge-Evaluator architecture in which Evaluators assess candidate faults and the Judge’s rationales and provide feedback for refinement.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    51.59% step-level accuracy on Who&When Algorithmically-Generated, up from 38.10%, alongside reported improvements on Hand-Crafted and ReasonEval benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results support automated step-level fault attribution as an approach to reducing the manual diagnostic burden of complex LLM systems.

    Source: §7 · p. 9

    The publicly available Who&When dataset is insufficient for comprehensive validation and generalization, and large-scale, high-quality agentic fault-attribution datasets are scarce.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The advent of complex, interconnected long-horizon LLM systems has made it incredibly tricky to identify where and when these systems break down. Evaluation capabilities that currently exist today are limited in that they often focus on simple metrics, end-to-end outcomes, and are dependent on the perspectives of humans. In order to match the increasing complexity of these many component systems, evaluation frameworks must also be able to reason, probe, iterate, and understand the nuanced logic passing through these systems. In this paper, we present RAFFLES, an offline evaluation architecture that incorporates iterative reasoning. Specifically, RAFFLES operates as an iterative, multi-component pipeline, using a central Judge to systematically identify faults and a set of specialized Evaluators to assess the quality of the candidate faults as well as rationales of the Judge. We evaluated RAFFLES with several benchmarks - the Who&When dataset to identify step-level faults in multi-agent systems and the ReasonEval datasets to diagnose step-level mathematical reasoning errors. RAFFLES outperforms strong baselines, achieving an accuracy of over 20% and 50% on the Who&When Hand-Crafted and Algorithmically-Generated datasets, and over 80% on the ReasonEval datasets. These results demonstrate a key step towards introducing automated fault detection for autonomous systems over labor-intensive manual review.

1 Introduction

Long-horizon, multi-component LLM systems create failures that outcome-focused evaluation and single-pass judges struggle to localize, leaving costly manual diagnosis. RAFFLES compares iterative evaluation architectures and reports gains in step-level fault attribution on agentic and mathematical-reasoning benchmarks.

1 Introduction

  • Current outcome-focused strategies struggle with long contexts and subtle agentic failures, leaving root-cause diagnosis to manual review that can take minutes to tens of minutes per instance.A single subtle error can cascade into unexpected behavior.Source: §1 · p. 1
  • Evaluation architectures should grow from single judgments toward aspect-specific routing, tool use, iteration, and eventually fully autonomous evaluators.The paper frames this progression as parallel to increasing LLM-system complexity.Source: §1 · p. 2, Figure 1 · p. 2
  • The RAG example distinguishes evaluating a final summary from locating and characterizing failures in retrieval versus generation.Adding retrieval tools and reflection loops expands agent activity, interaction effects, and decisions to track.Source: §1 · p. 2
  • Accuracy rose from 38.10% to 51.59% on Who&When Algorithmically-Generated, from 18.20% to 27.59% on Hand-Crafted, and from 73.58% to 84.91% on ReasonEval MR-MATH-Invalid.The ReasonEval result on MR-GSM8K-Original rose from 75.46% to 83.78%.Source: §1 · pp. 2–3
  • RAFFLES structures Judge reasoning, adds Evaluators to critique rationales and decisions, and feeds their feedback into iterative refinement.The paper describes this as a post-hoc offline architecture for detecting decisive, trajectory-altering faults.Source: §1 · p. 2

2 Related Work

Related work spans automated judging and reasoning, fault attribution in agentic systems, and error localization in reasoning chains. The paper positions RAFFLES as combining structured reasoning and iterative refinement across these areas.

Reasoning and automated evaluators

  • LLM-as-a-judge and advances in LLM reasoning underpin automated evaluation, while prior enhancements add planning, self-checking, debate, and judge juries.RAFFLES combines structured problem decomposition, natural-language confidence estimation, and iterative refinement.Source: §2 · p. 3

Fault attribution in agentic systems

  • Agentic fault-attribution work includes debugging tools, customized LLM judges, and binary-search or step-by-step methods; newer methods use simulation or simulation-free approaches.The paper mainly follows the simulation-free category.Source: §2 · p. 3

Fault attribution in reasoning chains

  • Reasoning-chain fault attribution advances through both fine-grained step-level benchmarks and detection methods, including training-free approaches and fine-tuned Process Reward Models.RAFFLES benchmarks its framework on reasoning-chain attribution to assess performance and versatility.Source: §2 · p. 3

3 Background

The background formalizes step-level errors and distinguishes faults by whether intervention could change a failed trajectory’s outcome. A decisive fault is the earliest critical fault, further constrained by severity.

Fault definitions

  • A step-level fault is an action judged incorrect by a local function when its estimated error probability exceeds a predetermined threshold.The local judge commonly uses a case-specific taxonomy such as hallucination or execution error.Source: §3 · pp. 3–4

First step-level fault attribution

  • The framework also distinguishes first-error attribution, which identifies the earliest error in a multi-step trajectory and is relevant to ReasonEval reasoning chains..Source: §3 · p. 4

Fault definitions

  • A trivial fault cannot be replaced by any alternative action that would make a failed trajectory succeed; it may be minor, later corrected, or superseded by an unavoidable later fault.The intervention denotes the alternative trajectory unfolding after the action at time t is replaced.Source: §3 · p. 4
  • A critical fault is an action in a failed trajectory for which an alternative replacement yields success, so it directly impacts the outcome.This contrasts with a trivial fault, which has no bearing on the final result.Source: §3 · p. 4

Decisive fault attribution

  • A decisive fault is the first critical fault, refined here as a non-trivial, high-severity error aligned with the Who&When annotation guidance.The paper adds a step-level severity function to formalize this aspect of fault attribution.Source: §3 · p. 4

4 RAFFLES

RAFFLES is an offline iterative Judge-Evaluator system that attributes faults using three criteria and refines candidate steps from evaluator feedback. Its stopping rules select a high-confidence decisive fault or return none.

4 RAFFLES

  • RAFFLES targets faults that satisfy fault-condition, primacy, and decisiveness criteria: the step is erroneous, earliest among critical faults, and capable of changing the failed outcome.The Judge and Evaluators are designed to reason over these criteria with confidence.Source: §4 · p. 4
  • The Judge proposes a candidate step and separate rationales for each decisive-fault criterion, while three Evaluators assess the corresponding criterion-specific rationale.The Judge is prompted to reference the execution log for rationale faithfulness.Source: §4 · pp. 4–5
  • An additional rule-based Evaluator checks whether the proposed candidate step is consistent with the log.The Evaluators also report confidence estimates for their assessments.Source: §4 · p. 5
  • Evaluator outputs are stored in memory and returned to the Judge in later iterations, allowing it to refine its candidate-step selection.The Judge incorporates evaluator rationales and confidence scores.Source: §4 · p. 5
  • Iteration ends when evaluator confidence exceeds 350 or the maximum iteration count is reached; the highest-confidence candidate is selected, with none returned when no decisive fault is identified.At the iteration limit, selection uses the candidate with the highest confidence in the history.Source: §4 · p. 5
  • RAFFLES prompts are task-agnostic, so a new dataset requires only minor revisions to the problem statement.The framework is presented as structured iterative reasoning between the Judge and Evaluators.Source: §4 · p. 5

5 Experiments

The experiments compare RAFFLES with four baseline architectures, evaluate fault attribution on Who&When and ReasonEval datasets, and use strict and tolerant step-level accuracy metrics.

5.1 Baseline Methods

  • $0.1188 per average 9-turn trajectory and 15.55 seconds average latency were reported for RAFFLES using claude-sonnet-4-20250514, with parallelized calls reducing latency by over 50%.The full cost and latency analysis is in Appendix D.Source: §5.1 · p. 5
  • Four baselines span one-shot judgment, sequential and binary-search routers, and a planner-directed Tool-Caller limited to three judge calls.The one-shot judge evaluates the full log; the routers narrow the fault location sequentially or by halving the log.Source: §5.1 · pp. 5–6

5.2 Dataset

  • Who&When includes Algorithmically-Generated data with 191 agents across 126 logs averaging 9 steps, and Hand-Crafted data with 5 agents across 58 logs averaging 51 steps.Both subsets use logs from real and dynamic agentic systems.Source: §5.2 · p. 6
  • ReasonEval uses 83 MR-MATH-Invalid trajectories with an incorrect reasoning step and 2,777 MR-GSM8K original or reversed samples, omitting Decisiveness because critical faults are inapplicable.MR-MATH-Invalid contains 159 trajectories with correct output; the evaluated subset is the 83 with an incorrect reasoning step.Source: §5.2 · p. 6

5.3 Evaluation Metrics

  • Strict Step-Level Accuracy requires the exact fault step, while Tolerant Step-Level Accuracy counts predictions within s steps of the ground-truth step.The paper describes small tolerance windows, such as s ≤ 2, as often sufficient to guide manual inspection and debugging.Source: §5.3 · p. 6

6 Results

RAFFLES outperforms fault-attribution baselines on Who&When and improves over an LLM-judge baseline across ReasonEval datasets. The results also show that performance declines with longer trajectories and that iterative gains are not always monotonic.

6.1 RAFFLES Results on Who&When

  • 31% with Llama 3.3 70B and 55% with Claude Sonnet 4 on Algorithmically-Generated, and 53% with Llama 3.3 70B on Hand-Crafted: RAFFLES beats Tool-Caller.The paper also reports that RAFFLES outperforms Chat-LLM across nearly all tolerance thresholds in both datasets.Source: §6.1 · p. 6
  • Partial-trajectory methods struggle on long logs, while Tool-Caller’s Planner can fail to generate high-quality candidates; RAFFLES combines global Judge context with focused Evaluator analysis.The paper characterizes this combination as a global, end-to-end perspective alongside local analysis of the hypothesized fault step.Source: §6.1 · p. 7
  • 35% on Algorithmically-Generated and 52% on Hand-Crafted: RAFFLES exceeds the best simulation-free method, and it surpasses leading simulation-based methods by 8.7% on Algorithmically-Generated.The paper reports new state-of-the-art step-level accuracy on Who&When.Source: Table 4 · p. 8, §6.1 · p. 8
  • On long-horizon logs averaging 50 steps, RAFFLES improves from 3.45 baseline to 22.41 for Claude and from 6.90 baseline to 20.69 for Llama 3.3 70B, although accuracy declines as trajectory length grows.RAFFLES still outperforms single-LLM baselines across context lengths.Source: §6.1 · p. 7, Figure 2 · p. 8
  • 20.6% of Algorithmically-Generated and 17.2% of Hand-Crafted examples retain an incorrect prediction across both K = 2 iterations, with the authors linking these cases to overconfident Evaluators.The authors report that attribution is repeatedly verified until Evaluators reach high confidence and that RAFFLES revises hypotheses across iterations.Source: §6.1 · p. 8

6.2 RAFFLES Results on ReasonEval

  • 15.40% on MR-MATH-Invalid, 11.08% on MR-GSM8K-Reversed, and 11.03% on MR-GSM8K-Original: Claude Sonnet 4 RAFFLES (K = 2) improves over the LLM-judge baseline.The ReasonEval results are reported in Table 2.Source: Table 2 · p. 7, §6.2 · p. 8
  • At K = 2, iterative reasoning improves step-level accuracy for three models on almost all datasets, but gains are not always monotonic and depend on trajectory length and fault position.The authors report convergence of target and predicted fault-step distributions, and call for high-confidence early stopping and a maximum iteration count.Source: §6.2 · p. 9

7 Conclusion

RAFFLES is a post-hoc iterative evaluation architecture that refines fault hypotheses and critiques the reasoning behind them. It improves fault attribution on Who&When and ReasonEval benchmarks over the reported baselines.

7 Conclusion

  • Over 51% step-level accuracy on Algorithmically-Generated and over 27% on Hand-Crafted, versus published simulation-free bests of 38% and 18%, respectively.RAFFLES also showed a 15.4% improvement over baseline on mathematical reasoning fault attribution.Source: §7 · p. 9
  • RAFFLES uses a Judge-Evaluator iterative architecture that builds hypotheses, refines reasoning, and critiques both candidate faults and their rationales.The conclusion attributes its results to error correction and iterative structured reasoning for problems that cannot be solved in a single step.Source: §7 · p. 9

Limitations

The paper identifies scarce, high-quality agentic fault-attribution datasets as a barrier to comprehensive validation and generalization. It also notes a scope boundary and latency trade-off for RAFFLES on mathematical reasoning.

Limitations

  • The scarcity of large-scale, high-quality agentic fault-attribution datasets limits comprehensive validation and generalization of emerging methods.The authors call for larger, more diverse public benchmarks and reliable high-fidelity synthetic-data methods.Source: Limitations · p. 9
  • RAFFLES was tested on ReasonEval, but developing a superior process reward model was outside the paper’s primary focus.The authors suggest broader reasoning-model fault attribution and systematic comparison with specialized reward-based approaches as future work.Source: Limitations · p. 9
  • RAFFLES’s iterative reasoning has higher latency than a fine-tuned process reward model, but remains viable for less time-sensitive applications.The paper names in-depth reasoning-process analysis and generating high-quality synthetic training data as such applications.Source: Limitations · p. 9

Ethical Considerations

The paper reports an internal risk evaluation and limits its study to English-language datasets focused on agentic systems and mathematical problems. Its dataset discussion contrasts short ReasonEval trajectories with much longer Who&When logs.

Ethical Considerations

  • The study is limited to English-language datasets focused on agentic systems and mathematical problems.The authors report an internal risk evaluation, proper citation and licensed use of prior works, datasets, and models, and anonymization of all datasets.Source: Ethical Considerations · p. 9

A Discussion on Datasets

  • ReasonEval subsets average fewer than 200 words per trajectory, versus 1,500 words for Algorithmically-Generated and 7,460 for Hand-Crafted Who&When logs.The authors characterize the Hand-Crafted subset's length and intricacy as making it their most demanding benchmark.Source: A Discussion on Datasets · p. 13

A.1 Limitations of Who&When Dataset

Who&When provides agent-step fault labels on multi-agent logs, but its Hand-Crafted subset has pronounced agent imbalance and the dataset contains some mismatched ground-truth labels.

A.1 Limitations of Who&When Dataset

  • Who&When logs include the problem statement, agent communications, the original task’s ground-truth answer, and an annotated erroneous agent and step.The dataset is described as one of the first open-source datasets addressing fault attribution in agentic systems.Source: §A.1 · p. 13
  • 57% accuracy is achievable by always guessing WebSurfer on Hand-Crafted; the authors also note subjectivity in assigning faults between planning and executing agents.There are 55 steps on average in Hand-Crafted versus 8 in Algorithmically-Generated, but only five possible agents across the logs.Source: §A.1 · p. 13
  • 3 of 126 Algorithmically-Generated cases and 3 of 58 Hand-Crafted cases have ground-truth agent-step pairs that do not match the logs.The paper leaves correction of these cases outside its scope and advises future researchers to account for them.Source: §A.1 · pp. 13–14

B Alternative Evaluation Metrics

The section explains the choice of step-level accuracy and reports RAFFLES comparisons on Who&When and ReasonEval. Reported results favor RAFFLES, while some additional state-of-the-art comparisons were not independently reproduced.

B Alternative Evaluation Metrics

  • Step-level accuracy is the unified primary metric; agent-level accuracy is less informative because WebSurfer is the root cause in 57% of Who&When trajectories.ReasonEval also reports step-level and solution-level F1 in Appendix C because its dataset includes both correct and incorrect trajectories.Source: B Alternative Evaluation Metrics · p. 14

C Extended results on Who&When Dataset and ReasonEval Dataset using alternative metrics

C.1 Who&When

  • 67.46% agent-level accuracy on Algorithmically-Generated and 67.24% on Hand-Crafted, plus 51.59% step-level accuracy on Algorithmically-Generated, are reported as state of the art.Across model families, RAFFLES ranks first for step-level accuracy in almost every tested setting and first or a close second for agent-level accuracy.Source: §C.1 · p. 14, Table 8 · p. 15

C.2 ReasonEval

  • RAFFLES outperforms reported state-of-the-art results on ReasonEval, including fine-tuned Llemma-34B and untuned GPT-4.The authors describe this as evidence that the framework can extend to fault attribution in reasoning chains.Source: §C.2 · p. 14

D Latency and Cost Analysis of RAFFLES

RAFFLES trades longer processing time for iterative refinement, while its reported per-log costs are below the stated human-debugging estimates.

D Latency and Cost Analysis of RAFFLES

  • Latency is generally higher than the Chat-LLM baseline; parallelizing Evaluators reduces the two-iteration run to about four LLM calls, and Claude latency falls by over 50%.A typical two-iteration run uses two sequential Judge calls and six Evaluator calls before parallelization.Source: D Latency and Cost Analysis of RAFFLES · p. 15
  • At two maximum iterations, estimated Claude costs are $0.1188 per Algorithmically-Generated log and $0.306 per Hand-Crafted log, versus estimated junior-engineer costs of $2.5 and $12.5.The estimates assume four LLM calls per iteration and human debugging times of three minutes for nine turns and ten minutes for 50 turns.Source: D Latency and Cost Analysis of RAFFLES · pp. 15–16

E Extended Analysis of Iterative Reasoning

This section examines RAFFLES’s iterative reasoning through its effects on ReasonEval and Who&When predictions, alongside examples of the Judge–Evaluator process and supporting prompt materials.

E Extended Analysis of Iterative Reasoning

  • 70.35% to 75.40% with ±2 tolerance is the Algorithmically-Generated dataset’s accuracy improvement by iteration four, while Hand-Crafted accuracy initially falls before recovering and stabilizing.The authors attribute the Hand-Crafted pattern to its larger, more complex search space, which impedes convergence.Source: Figure 3 · p. 16, E Extended Analysis of Iterative… · p. 16
  • Accuracy and F1 improve across all ReasonEval GSM8K subsets as iterations increase from K = 0 to K = 2.The comparison uses Claude Sonnect 4; the supplied passages provide no per-subset values.Source: E Extended Analysis of Iterative… · p. 16, Table 13 · p. 18
  • On Hand-Crafted Who&When, early-step predictions initially create a spike, then converge toward the ground-truth distribution as evaluator critiques prompt the Judge to consider later steps.The authors hypothesize that early-step bias reflects the “lost in the middle” phenomenon (Liu et al., 2023a) and that the “first mistake” framing contributes to it.Source: E Extended Analysis of Iterative… · pp. 16, 19, Figure 4 · p. 18
  • More than a quarter of steps change between iterations one and two, but revisions decrease substantially by iteration four, motivating a maximum iteration count K to limit diminishing returns.The authors relate the diminishing returns to patterns in prior self-refinement research (Madaan et al., 2023).Source: E Extended Analysis of Iterative… · p. 19

F.1 Baseline prompt

  • The baseline prompts adapt a one-shot prompt to the Llama chat template and use task-specific Step-by-Step and binary-search instructions based mostly on the Who&When paper (Zhang et al., 2025a).The Step-by-Step and binary-search approaches share a prompt template, with task-specific descriptions for Step-by-Step logic.Source: §F.1 · p. 20

F.2 Tool-Caller prompt

  • The Tool-Caller uses a globally informed Planner to select a step for a Judge to evaluate, but it lacks explicit iterative planning and RAFFLES’s multiple evaluators.Its prompt directs the Planner to inspect a likely fault with few calls, and limits tool calling to fewer than three times.Source: §F.2 · p. 22, Tool-Caller Prompt · p. 23

F.3 RAFFLES Prompt

  • RAFFLES uses a generic fault prompt adapted to reasoning traces by replacing “agents” with “reasoning chains,” with evaluators checking the candidate fault’s logic, first-mistake status, and whether later steps corrected it.The Judge is instructed to select the first relevant, uncorrected mistake, using a best-guess fallback when no single step meets all criteria.Source: §F.3 · pp. 24–27

G Example of RAFFLES reasoning process

  • An example of the RAFFLES reasoning process is provided in Figures 5 and 6.Source: G Example of RAFFLES reasoning process · p. 28
  • A Who&When example log shows RAFFLES’s iterative reasoning process arriving at the correct decisive fault.Source: Figure 5 · p. 29
  • In the continued interaction, separate Judge reasoning blocks go to specialized Evaluators, who assess rationale soundness and score their confidence.Source: Figure 6 · p. 30
Loading 2509.06822v3…