Source-linked AI summary
PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases
Sri Vatsa Vuddanti, Aarav Shah, Satwik Kumar Chittiprolu, Tony Song, Sunishchal Dev, Kevin Zhu, Maheep Chaudhary
TL;DR
Tool-augmented agents remain vulnerable to runtime tool failures, while existing training does not explicitly teach recovery. PALADIN combines failure-injected recovery training with inference-time retrieval of recovery examples. Across backbones, it reports high recovery, task-success, and catastrophic-success rates with efficiency penalties within one step.
Problem
Existing tool-agent training does not explicitly expose agents to structured runtime tool failures, and standard task-success metrics do not measure recovery capabilities.
Source: §1 · pp. 1–2Method
PALADIN trains agents on failure-injected, recovery-annotated trajectories and retrieves similar recovery examples when execution errors occur.
Source: §1 · p. 2, §2 · p. 2, §2.4 · p. 4Results
Across backbones, PALADIN reports >79% RR, >78% TSR, and >80% CSR, with efficiency penalties within one step.
Source: §7 · p. 9Takeaways & Limitations
The paper concludes that execution-level robustness can be taught, generalized across models, and achieved without prohibitive cost.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Tool-augmented language agents frequently fail in real-world deployment due to tool malfunctions--timeouts, API exceptions, or inconsistent outputs--triggering cascading reasoning errors and task abandonment. Existing agent training pipelines optimize only for success trajectories, failing to expose models to the tool failures that dominate real-world usage. We propose \textbf{PALADIN}, a generalizable framework for equipping language agents with robust failure recovery capabilities. PALADIN trains on 50,000+ recovery-annotated trajectories constructed via systematic failure injection and expert demonstrations on an enhanced ToolBench dataset. Training uses LoRA-based fine-tuning to retain base capabilities while injecting recovery competence. At inference, PALADIN detects execution-time errors and retrieves the most similar case from a curated bank of 55+ failure exemplars aligned with ToolScan's taxonomy, then executes the corresponding recovery action. This approach generalizes to novel failures beyond the training distribution, retaining 95.2\% recovery performance on unseen tool APIs. Evaluation across PaladinEval and ToolReflectEval demonstrates consistent improvements in Recovery Rate (RR), Task Success Rate (TSR), Catastrophic Success Rate (CSR), and Efficiency Score (ES). PALADIN improves RR from 32.76% to 89.68% (+57% relative) over ToolBench and outperforms the strongest baseline CRITIC (76.34%) by +13.3%. Against vanilla agents, PALADIN achieves 89.86\% RR (+66% relative improvement from 23.75%). These results establish PALADIN as an effective method for building fault-tolerant agents capable of robust recovery in real-world tool environments.
1 INTRODUCTION
PALADIN addresses tool-agent brittleness under realistic runtime failures by teaching recovery through trajectory-level supervision. The introduction reports improved recovery and task success, alongside a broader robustness evaluation.
1 INTRODUCTION
- Real-world tool failures such as timeouts, malformed outputs, and silent call failures can leave agents hallucinating success or deadlocked.Existing training pipelines do not explicitly expose agents to structured tool failures.Source: §1 · p. 1
- Execution-level robustness—detecting, diagnosing, and recovering from runtime failures—remains underaddressed by work focused on comprehension robustness or call-level reflective fixes.Task-success-focused evaluation also leaves recovery capabilities unmeasured; PALADIN introduces TSR, RR, CSR, and ES, with the latter three novel.Source: §1 · p. 2
- 50,000+ recovery-annotated trajectories are constructed through systematic failure injection spanning diverse ToolScan error modes.At inference, PALADIN detects execution errors and retrieves recovery strategies from 55+ curated failure exemplars.Source: §1 · p. 2
- 57–66% relative improvements in recovery rates across model sizes are reported, while PALADIN improves RR and CSR at the cost of lower ES.Figure 1 compares Recovery Rate and Task Success Rate across LLMs and shows PALADIN ahead of CRITIC on both.Source: §1 · p. 2, Figure 1 · p. 2
2 METHODOLOGY
PALADIN equips tool-augmented agents to detect, diagnose, and recover from runtime failures through failure-annotated training and taxonomy-driven retrieval.
2.1 PROBLEM SETUP
- PALADIN formalizes execution-level robustness as detecting, diagnosing, and recovering from runtime failures, which existing training ignores.Its goal is to maximize task success while remaining robust to failures.Source: §2 · p. 2, §2.1 · p. 3
2.2 FAILURE INJECTION AND RECOVERY ANNOTATION
- Failures aligned to ToolScan’s seven canonical classes are injected into ToolBench traces; GPT-5 repairs traces truncated at failure, yielding 50K+ recovery-annotated trajectories.A separate dictionary of 55+ exemplar failures and recovery strategies operationalizes the taxonomy.Source: §2.2 · p. 3, Figure 3 · p. 5
2.3 PALADIN TRAINING OBJECTIVE
- PALADIN adds recovery supervision to the causal language-modeling objective, using LoRA adapters for parameter-efficient adaptation while preserving base competence.The recovery loss applies to tokens following Recovery: tags that mark corrective steps.Source: §2.3 · p. 4
2.4 INFERENCE-TIME RECOVERY VIA TAXONOMIC RETRIEVAL
- At inference, PALADIN retrieves the most similar failure exemplar from its curated bank and executes the associated recovery action.Similarity is measured over error signatures, enabling recovery across diverse failure scenarios.Source: §2.4 · p. 4
2.5 RECOVERY AS IMPLICIT POLICY LEARNING
- PALADIN learns recovery behaviors directly from annotated trajectories, producing proactive, compositional strategies such as retrying, switching tools, and graceful termination.The implicit policy maps dialogue context and observed failures to recovery actions.Source: §2.5 · p. 4
2.6 METRICS FOR EXECUTION ROBUSTNESS
- PALADIN evaluates robustness with TSR, RR, CSR, and ES; RR, CSR, and ES are novel metrics beyond traditional task success.ES measures efficiency through the average number of steps to complete a task.Source: §2.6 · p. 4
3 EXPERIMENT AND EVALUATION SETUP
The evaluation combines controlled and naturalistic failure benchmarks, a simulator for consistent execution and scoring, and baselines spanning different supervision and recovery strategies.
3.1 BENCHMARKS AND EVALUATION PROTOCOL
- PaladinEval combines clean and failure-augmented ToolBench prompts; the Generalization Set tests novel errors, while ToolReflectEval preserves its protocol with up to three attempts.ToolReflectEval uses ToolAlpaca-derived prompts without modification or cherry-picking.Source: §3.1 · p. 5
3.2 SIMULATION ENVIRONMENT
- The simulator reproduces ToolBench execution with systematic failure injections or replayed ToolReflect errors, logging dialogue context, observed errors, and recovery attempts.A GPT-5 grader applies explicit guidelines to score TSR, RR, CSR, and ES consistently across runs.Source: §3.2 · pp. 5–6
3.3 BASELINE AGENTS
- Baselines include vanilla, CRITIC, ToolBench Agent, and ToolReflect Agent, spanning clean-trajectory training, reactive correction, and recovery-annotated supervision.CRITIC accesses recovery protocols with probability p = 0.7; the ToolReflect Agent makes iterative, local self-corrections.Source: §3.3 · p. 6
4 RESULTS
PALADIN improves recovery, task success, and catastrophic success across model–dataset pairs, generalizes across backbones, and benefits from inference-time exemplar retrieval. These gains come with measurable but bounded efficiency tradeoffs.
4.1 OVERALL PERFORMANCE
- PALADIN achieves the highest Recovery Rate in 7/8 settings (+13.6% average), top Task Success Rate in 6/8 (+10.2%), and best Catastrophic Success Rate in 6/8 (+9.2%).All gains are statistically significant (p < 0.01, bootstrap N = 1000).Source: §4.1 · p. 6, Table 1 · p. 7
- Efficiency decreases by 0.183 on LLaMA-3.1-8B but improves by 0.012 on Qwen-2.5-14B; no baseline achieves simultaneously higher safety and efficiency.Recovery and TSR correlate strongly (r = 0.91, p < 0.001), while CSR and Efficiency correlate negatively (r = −0.72, CI: [−0.79, −0.63]).Source: §4.1 · p. 6
4.2 GENERALIZATION
- Across unseen errors, PALADIN secures > 79% RR, > 78% TSR, and > 80% CSR across the reported backbones.AM-Thinking V1 records the highest RR overall at 96.1%, alongside TSR of 81.2% and CSR of 88.7%.Source: §4.2 · pp. 6–7
4.3 ABLATIONS
- Removing inference-time exemplar matching reduces RR, TSR, and CSR across all four reported backbones, with retrieval contributing up to 20–30 points in robustness metrics.For example, LLaMA-8B RR falls from 79.8% to 48.6% without matching.Source: Figure 4 · p. 7, §4.3 · p. 8
4.4 KEY TAKEAWAYS
- The authors characterize PALADIN’s gains as combining recovery and safety improvements, modest efficiency costs, and consistent generalization.Their ablation attributes robustness to both training-time failure exposure and inference-time retrieval.Source: §4.4 · p. 8
5 RELATED WORKS
Prior tool-agent work improves planning, call quality, or interface reliability, but often leaves multi-step recovery from runtime failures underdeveloped. PALADIN instead targets recovery across failure-rich trajectories.
5.1 EXECUTION-LEVEL ROBUSTNESS
- ToolReflect’s paired bad-versus-good call training emphasizes single-call correction, while PALADIN trains diagnosis, replanning, and multi-turn recovery across failure-rich trajectories.Critic-style correction is reactive and can be poor without diverse tool-error taxonomies.Source: §5.1 · p. 8
5.2 DIAGNOSTICS AND BENCHMARKS
- ToolScan labels seven recurring tool-use failure patterns but does not curate executable recovery paths or train agents on full recovery trajectories.BugGen motivates realistic failure-rich datasets because handcrafted or mutation-based synthesis can have narrow coverage.Source: §5.2 · p. 8
5.3 PRIOR TOOL SYSTEMS & INTERFACE RELIABILITY
- ToolBench lacks recovery protocols, RoTBench evaluates noisy documentation and ambiguous specifications, and ToolFuzz tests documentation–tool mismatches before execution.These approaches address different stages or settings than runtime detection, diagnosis, and recovery.Source: §5.3 · p. 9
6 DISCUSSION
The discussion presents recovery as learnable and transferable, while emphasizing the tradeoff between catastrophic failures and efficiency. It also identifies exemplar retrieval and targeted retries as important parts of the approach.
DISCUSSION
- PALADIN’s recovery supervision raises LLaMA-8B TSR from 17.5% to 78.7%, and RR exceeds 79% across Gemma, Qwen, LLaMA, and AM-Thinking.The authors frame recovery as a transferable training signal rather than an emergent feature of scale.Source: §6 · p. 9
- Retry intensity lowers catastrophic failures from 82.4% to 19.3% while modestly reducing efficiency, and no baseline dominates on both safety and cost.CSR and Efficiency correlate at r = −0.72 in the reported analysis.Source: §6 · p. 9
- Inference-time exemplar matching contributes up to +30 points in robustness, while targeted retries use fewer than one additional step on average.The authors describe the retries as data-driven minimal corrective actions rather than brute-force exploration.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents execution-level robustness as teachable and scalable, with PALADIN generalizing across backbones and unseen failures. It also notes a documentation-accuracy assumption and complementary upstream interface validation.
7 CONCLUSION
- Across backbones, PALADIN achieves > 79% RR, > 78% TSR, and > 80% CSR while keeping efficiency penalties within one step.The conclusion says its recovery behaviors generalize to unseen failures.Source: §7 · p. 9
A CRITIC-STYLE APPROACH
- The critic-style baseline invokes a Gemma-12B critic after each tool call and permits at most three recovery attempts per error.Its executor selects from predefined recovery actions in an oracle dataset.Source: A CRITIC-STYLE APPROACH · p. 14
EVALUATION EXAMPLE: SERVICEDEPENDENCYFAILURE
- In the service-dependency example, the expected recovery is to identify the failing service, wait for it to become healthy, and retry.The scenario treats the upstream outage as an external failure rather than malformed input.Source: EVALUATION EXAMPLE:… · p. 14
B TOOLFUZZ
- ToolFuzz found over 20x more specification-related failures than prompt-engineering baselines across 32 LangChain and 35 custom tools.It addresses documentation–tool mismatches, which the authors distinguish from PALADIN’s runtime robustness focus.Source: B TOOLFUZZ · p. 14
7 CONCLUSION
- The PALADIN evaluation assumes tools have correct instructions at test time; ToolFuzz is presented as complementary validation of tool interfaces upstream.The authors identify adaptive retry controllers and integration with production error logs as future work.Source: §7 · p. 9, B TOOLFUZZ · p. 14
C RUNTIME ERRORS
This section describes how PALADIN builds recovery supervision from injected tool failures, recovery mappings, and teacher-generated trajectories, and fine-tunes open-weight backbones with LoRA. It also covers its runtime failure catalog and retrieval bank of ToolScan-aligned exemplars used to guide recovery.
C RUNTIME ERRORS
- The runtime-error catalog presents common failures alongside simulated outputs and recovery strategies.The cited caption establishes the catalog’s scope but does not provide individual table entries.Source: Table 2 · p. 15
D SYSTEMATICALLY AUGMENTED TRACES
- Deterministic failure injection creates variants of ToolBench-style traces with labeled errors and annotated recovery paths for training and controlled evaluation.Injected failures include timeouts, 5xx responses, rate limits, malformed outputs, and partial results; recovery actions include diagnosis, replanning, parameter fixes, retries, and tool switches.Source: D SYSTEMATICALLY AUGMENTED TRACES · p. 16
E RECOVERY DICTIONARY
- PALADIN’s recovery dictionary contains over 50 execution-level failure types paired with strategies grounded in real debugging and fallback practices.It follows ToolScan’s taxonomy and draws on forums, documentation, and engineering logs; its mappings are converted into annotated ToolBench-format trajectories.Source: E RECOVERY DICTIONARY · p. 17
G PYTHON DICTIONARY
- For 400/422 errors, the example recovery path checks request syntax, payload structure, headers, and parameters, then tests a minimal payload to isolate the cause.The dictionary illustrates staged diagnosis and correction rather than a single retry action.Source: G PYTHON DICTIONARY · p. 18
H.1 MODEL INSTANTIATIONS
- PALADIN was fine-tuned on Gemma-27B, Qwen-2.5-14B-Instruct, AM-Thinking V1, and LLaMA-3.1-8B-Instruct using the full 50K recovery-annotated corpus.RoPE scaling extended LLaMA-3.1-8B’s context to avoid truncating long recovery traces.Source: §H.1 · p. 20
H.2 LORA CONFIGURATION
- Training used LoRA adapters across attention and MLP projections, with one epoch over 50K trajectories and an effective batch size of 8.The setup used rank 16, α = 32, dropout 0.0, 8192-token contexts, bf16, paged AdamW, and gradient checkpointing; fine-tuning ran on h200sxm GPUs, with RoPE scaling extending LLaMA’s context to 128K tokens.Source: §H.2 · p. 20, §H.3 · p. 20, §H.4 · p. 20
I.1 FAILURE INJECTION PROCEDURE
- Failure simulation truncates ToolBench rollouts at the first execution error, then uses a GPT-5 Teacher and recovery dictionary to generate multi-turn repairs.The simulator deterministically specifies error type, manifestation, and turn; supervision combines dictionary strategies with teacher-guided rewriting, while about 20% of clean traces are audited or rewritten to preserve baseline competence.Source: §I.1 · p. 21, §I.2 · p. 21
F TOOLSCAN ERROR TYPES
- The recovery supervision aligns with ToolScan’s seven error classes, including partial execution, invalid intermediate reasoning, and re-entrant failures.These examples call for continuation, plan correction or backtracking, and retry or fallback logic, respectively.Source: F TOOLSCAN ERROR TYPES · p. 17, §I.2 · p. 21
I.3 RECOVERY EXEMPLAR DISTRIBUTION
- Over 55 exemplars cover ToolScan’s seven canonical error types, with signature-distance matching retrieving recovery actions that steer trajectories back to stability.The types are Tool Hallucination, Argument Hallucination, Invalid Tool Invocation, Partial Execution, Output Hallucination, Invalid Intermediate Reasoning, and Re-entrant Failures.Source: §I.3 · p. 21
I.4 DATA SPLIT
The final corpus combines recovery-rich and clean traces, then uses a single-epoch supervised fine-tuning regime and controlled evaluation.
I.4 DATA SPLIT
- The corpus contains roughly 50K trajectories in an 80/20 mix of recovery-rich and clean traces, trained with single-epoch CLM SFT.LLaMA-3.1-8B used RoPE scaling to extend effective context to 128K tokens for long recovery trajectories.Source: §I.4 · p. 21
J PALADINEVAL BENCHMARK
PaladinEval provides deterministic, taxonomically labeled testing of recovery, honesty, and efficiency, emphasizing multi-turn recovery after explicit failures. Its balanced failure coverage enables comparisons with ToolReflectEval, with gains in recovery, task success, and catastrophic success alongside efficiency trade-offs.
J PALADINEVAL BENCHMARK
- PaladinEval combines controlled simulators, taxonomic labels, and standardized metrics to compare recovery competence, honesty, and efficiency across methods.Unlike ToolReflectEval’s single-call reflective corrections, it emphasizes trajectory-level, multi-turn recovery in noisy execution settings.Source: J PALADINEVAL BENCHMARK · p. 22
J.1 BENCHMARK DESIGN
- Episodes inject one labeled execution failure at a specified turn, then log recovery attempts, retries, tool switches, and termination decisions through completion.Fixed seeds and deterministic tool outputs support reproducibility across backbones and runs.Source: §J.1 · p. 22
J.2 TASKS AND COVERAGE
- PaladinEval comparisons with ToolReflectEval show consistent gains in RR, TSR, and CSR, with efficiency trade-offs from retry-heavy strategies.The suite covers all seven ToolScan error categories under uniform sampling rules.Source: §J.2 · p. 22
J.3 SAMPLING PROCEDURE
- Sampling covers all seven ToolScan error classes and supports per-class and macro-averaged reporting of RR, TSR, CSR, and ES.The first failure’s class, manifestation, and turn are specified deterministically; subsequent tool responses are fixed.Source: §J.3 · p. 22
J.4 ADAPTING TOOLREFLECTEVAL
- ToolReflectEval is rerun under the same deterministic simulator and normalized metrics, complementing PaladinEval’s multi-turn recovery focus with single-call critique.Together, the benchmarks offer complementary perspectives on robustness.Source: §J.4 · p. 22
J.5 FILTERING AND DEDUPLICATION
- Filtering removes ambiguous or duplicate error signatures and non-reproducible episodes, while retaining clean episodes for competence checks but not recovery scoring.Trajectories are truncated at the first failure.Source: §J.5 · p. 22
K.1 METRIC VARIANTS AND FORMULAS
- RR, CSR, and ES are novel execution-robustness metrics beyond task success; diagnostic efficiency variants preserved model rankings and were omitted from main results.RR is the fraction of encountered failures recovered; ES is one minus the average steps to complete a task.Source: §K.1 · p. 23
K.2 ABLATION: REMOVING INFERENCE-TIME RETRIEVAL
Removing inference-time retrieval substantially reduces robustness, while recovery-annotated training and zero-shot transfer support stable recovery behavior. The paper identifies retrieval as a primary robustness driver and a safety net for novel APIs.
K.2 ABLATION: REMOVING INFERENCE-TIME RETRIEVAL
- 20–30-point drops across backbones show that removing exemplar matching sharply reduces robustness.PALADIN maps observed failures to recovery actions using 55+ ToolScan-aligned exemplars.Source: §K.2 · p. 23
K.3 ABLATION: TRAINING DATA COMPOSITION
- Failure-only training degrades multi-turn behavior, recovery, task success, safety, and efficiency, while explicit recovery annotations support stable, compositional recovery.Agents otherwise overfit to retry-once heuristics and miss plan-shift and tool-switch transitions.Source: §K.3 · p. 23
K.4 ROBUSTNESS: ZERO-SHOT TRANSFER TO UNSEEN TOOLS
- On held-out APIs under the same simulator and error taxonomy, zero-shot transfer preserves a large fraction of recovery performance, with smaller absolute drops in CSR than RR or TSR.Schema-mismatch handling and malformed-output repair transfer strongly; authentication and pagination account for most residual errors, which retrieval can help address.Source: §K.4 · p. 23
K.5 TAKEAWAYS
- Inference-time retrieval complements diverse failure training, recovery annotations, and zero-shot transfer by serving as a safety net for novel APIs.The paper identifies retrieval as a primary robustness driver and recovery-annotated trajectories as critical to chained recovery.Source: §K.5 · pp. 23–24