ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents

1 Introduction

• Final-outcome and tool-call metrics provide limited insight into whether agents can localize failures, diagnose causes, and recover from corrupted intermediate states.Prior evaluations also miss fine-grained behaviors in multi-turn parallel execution, including dependency assessment and replanning after errors.Source: §1 · p. 1
• Its taxonomy covers 14 error types across planning dependencies, tool selection, and argument matching.The taxonomy is based on agent execution trajectories in real-world multi-turn parallel scenarios.Source: §1 · pp. 1–2
• ParaRecover evaluates intermediate decision-making, error localization, and corrective replanning across 10,626 instances and two difficulty levels.LEVEL-1 includes errors from the most recent execution; LEVEL-2 includes multi-turn propagation and varied error types.Source: §1 · p. 2
• The SDE rubric assesses Structural Integrity, Diagnostic Reasoning, and Evolutionary Strategy to evaluate reflection, backtracking, and decision capability after errors.Source: §1 · p. 2
• Even state-of-the-art models show weaknesses in dependency reasoning, implicit failure localization, and efficient recovery under multi-turn error propagation.Source: §1 · p. 2

2 Problem Definition

• Agent execution is represented as a DAG whose nodes are subtasks or tool operations and whose edges encode completion dependencies.A node is executable only after its dependencies are satisfied, and tool invocations proceed layer-wise along the graph’s topological structure.Source: §2 · p. 2

2.1 ParaRecover

• ParaRecover evaluates error localization and recovery in parallel tool-use agents through a DAG-based, process-level benchmark.The benchmark builds on BUTTON tasks with compositional dependencies and uses a controlled simulated tool-execution environment.Source: §2.1 · p. 3, §2.1.1 · p. 3

2.1.2 Error Taxonomy

• Its 14 error types span three dimensions and extend local tool-call failures to include parallelization and dependency errors found in rollout traces.These categories support subsequent trajectory perturbation and error-aware data augmentation.Source: §2.1.2 · pp. 3–4

2.1.3 Data Composition

• The benchmark combines manually curated erroneous rollout trajectories with augmented trajectories created by injecting predefined errors into correct trajectories.LEVEL-1 covers recent single-round errors, while LEVEL-2 includes co-existing errors and cross-round propagation in long-horizon corrupted contexts.Source: §2.1.3 · p. 4

2.1.4 The SDE Rubric

• The SDE rubric evaluates structural integrity, diagnostic reasoning, and evolutionary strategy as dimensions of multi-turn execution, failure localization, and replanning.Its structural dimension covers DAG correctness and executability; diagnostic reasoning assesses failure analysis, while evolutionary strategy assesses replanning efficiency and precision.Source: §2.1.4 · pp. 4–5
• The SDE rubric scores five sub-dimensions per category, including dependency correctness, failure-root localization, and replanning efficacy.Problem-Solving Efficacy considers final task success and the number of rounds required relative to the ground-truth optimum.Source: §2.1.4 · pp. 4–5

3.1 RQ1: How do mainstream large language models perform on our benchmark?

• Claude Opus 4.6 achieves the highest average scores, 69.23 on LEVEL-1 and 66.85 on LEVEL-2, while models score higher on LEVEL-1 overall.Across models, average benchmark scores remain below 70, and Evolutionary Strategy is the weakest dimension, reflecting difficulty producing precise, efficient plan updates.Source: §3.1 · pp. 6–7

3.2 RQ2: How does agent performance vary across different types of errors?

• Multi-turn errors challenge both representative models more than single-turn errors; Toolcall_Redundancy is hardest to localize, while structure- and tool-related errors also pose difficulties.GPT-4o-mini declines broadly on LEVEL-2, whereas DeepSeek-V4-flash changes less across most metrics; parameter-related explicit errors tend to score higher than implicit errors such as redundant calls and state drift.Source: §3.2 · p. 7

3.3 RQ3: Is it reliable to evaluate abstract rubrics using a LLM-as-a-Judge paradigm?

• LLM-as-a-Judge achieves average EMA of 0.8067 and average MAE of 0.1208, and a GPT-4o judge produces similar overall patterns to DeepSeek-V4-flash thinking.LEVEL-1 rankings remain unchanged across judges, with only minor top-model rank permutations on LEVEL-2.Source: Table 6 · p. 7, §3.3 · p. 8

3.4 RQ4: Can fine-grained evaluation metrics provide actionable guidance for improving model capabilities?

• Few-shot prompting yields average scores of 44.47 on LEVEL-1 and 42.24 on LEVEL-2; SFT and ParaRecover-DPO improve SDE scores, with DPO performing best on both levels.For DPO, higher-scoring trajectories are preferred over lower-scoring trajectories from the same task; the resulting preferences are used to train from the SFT model.Source: §3.4 · pp. 8–9, Table 7 · p. 8
• Across both levels, ParaRecover-DPO improves all judge-free execution metrics; on LEVEL-1, tool-use efficiency rises from 20.43 to 26.20.E5 rises from 36.50 to 40.85, invalid DAG rate falls from 10.25 to 5.90, and Pass@1 rises from 91.84 to 94.30 on LEVEL-1.Source: §3.4 · p. 9, Table 8 · p. 9

• ParaRecover evaluates error localization and recovery in parallel tool-use agents at the process level.The framework introduces a fine-grained error taxonomy, challenging error trajectories, and the SDE rubric.Source: §4 · p. 9
• The benchmark uses the SDE rubric to measure agent capabilities.The rubric is introduced alongside the error taxonomy and challenging trajectories.Source: §4 · p. 9
• High task completion rates do not necessarily indicate reliable recovery and replanning, especially under multi-turn error propagation.SDE-based supervision can effectively improve agents’ reflective recovery behaviors.Source: §4 · p. 9

5 Limitation

• The simulated execution environment may not capture the unpredictability, latency, and dynamic state changes of live real-world APIs.Simulation supports reproducibility and controlled error injection.Source: §5 · p. 9
• LLM-as-a-Judge evaluation of abstract SDE dimensions may introduce bias in complex or ambiguous cases.The approach achieves relatively high agreement with human annotations.Source: §5 · p. 9
• DAG workflows capture dependencies in parallel multi-turn scenarios but do not fully represent loops, conditional branching, or asynchronous interactions.The paper identifies more realistic environments, hybrid human-AI evaluation, and more general execution structures as future work.Source: §5 · p. 9

A.1 LLM Agents

• LLM agents are described as goal-driven systems that plan, act, and interact with external environments, including through tool-augmented designs.Examples include research, code, and web agents; the text identifies high-quality benchmarks as crucial for expanding agent capabilities.Source: §A.1 · p. 12

• Existing tool-use benchmarks mainly assess tool selection, parameter filling, and task completion, while remaining outcome- or call-level oriented.This makes it difficult to diagnose where an agent fails within a long trajectory.Source: §A.2 · p. 12
• Trajectory-aware benchmarks assess dimensions such as tool selection, parameterization, ordering, and trajectory preferences, but focus mainly on step or tool-use correctness or preference judgments.Examples include T-Eval, TRAJECT-Bench, AgentRewardBench, and Plan-RewardBench.Source: §A.2 · p. 12
• ParaRecover evaluates whether agents can inspect flawed intermediate trajectories, localize erroneous steps, and decide how to recover and replan.This error-aware trajectory evaluation targets parallel tool-calling scenarios and complements existing tool-use and trajectory-evaluation benchmarks.Source: §A.2 · p. 12

• Pass@1 remains over 88% across models, while E5 scores are generally low, exposing a disparity between task completion and execution-round efficiency.E5 measures deviation of actual execution rounds from ground-truth rounds; the contrasting trends indicate erroneous planning and redundant tool calls.Source: B Analysis of the Discrepancy Between… · p. 13
• 18 execution rounds versus 6 required rounds produced an E5 score of 0.33 after the agent misattributed an empty response caused by an incorrect parameter type.The agent retried the entire workflow rather than attributing the failure to the call and retrying it, triggering subsequent errors.Source: B Analysis of the Discrepancy Between… · p. 13
• The case illustrates why final task completion rates alone do not capture error attribution and replanning failures in multi-turn parallel scenarios.The described cascade arose from misattributing a tool failure to an upstream task.Source: B Analysis of the Discrepancy Between… · p. 13

• Table 12 lists 14 error types and their corresponding descriptions.Source: C Error Description · p. 13

E Related Prompts

• The prompts specify DAG-based planning, simulated tool execution, and LLM-as-a-Judge evaluation across the SDE dimensions.The tool agent checks ready DAG nodes and invocation validity, then returns error prompts or simulated results.Source: E Related Prompts · pp. 13–15

D Error Type Distribution

• Incorrect parameter types, invalid parameter values, redundant calls, and missing critical calls are the most frequent errors; larger models show lower error rates.All taxonomy categories also occur in real-world agent execution.Source: D Error Type Distribution · p. 13, Figure 3 · p. 14

Prompt for Diagnostic Reasoning Evaluation

• Diagnostic reasoning is evaluated for plan and tool evidence anchoring, root-cause localization, impact recognition, and evidence sufficiency with restraint.The prompt supplies the original task, available tools, previous plan and results, replanner thought, and updated plan as evaluation inputs.Source: Prompt for Diagnostic Reasoning… · p. 15

Prompt for Diagnostic Evolutionary Strategy

• Evolutionary strategy evaluates minimal modification, closed-loop revision, goal consistency, and the rationality of replanning changes.The prompt evaluates the replanner’s thought and updated plan against the original task and prior execution context.Source: Prompt for Diagnostic Evolutionary… · pp. 15–16

• 10,626 instances are divided into LEVEL-1’s 4,033 and LEVEL-2’s 6,593 instances.LEVEL-1 has 1,421 real-world error instances and 2,612 error-augmented instances; LEVEL-2 has 1,391 real-world error instances and 5,202 error-augmented instances. LEVEL-1 covers single-round errors, while LEVEL-2 covers multi-turn error types.Source: F Data Distribution · p. 16

• Training used Qwen3-8B with LoRA for SFT and DPO on one server equipped with eight NVIDIA RTX 3090 GPUs.LoRA used rank 32, scaling factor 64, and dropout 0.05.Source: G Training Configuration · p. 16
• SFT ran for three epochs with a 4,096-token maximum sequence length, while DPO was limited to one epoch.SFT used a 1 × 10^-4 peak learning rate; DPO used 5 × 10^-6 and KL coefficient β = 0.1.Source: G Training Configuration · p. 16

H Maximum Number of Execution Rounds

• Completion requires reaching the response node within 40 execution rounds.The cap is based on preliminary experiments indicating that longer trajectories typically signal comprehension bias or an infinite loop.Source: H Maximum Number of Execution Rounds · p. 17