Source-linked AI summary
Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures
Isham Kalappurackal Mansoor, Abhishek Phadke, Pratip Rana
TL;DR
LLM agents often treat tool responses as reliable signals of whether actions succeeded, although non-atomic tool behavior can make responses diverge from external state. This paper introduces a verify-before-retry wrapper that checks postconditions and uses idempotent execution. In simulated workflows, it reduces duplicate side effects while maintaining high task success across tested settings.
Problem
LLM agent frameworks commonly assume atomic tool calls, but timeouts, delayed visibility, and partial updates can make responses diverge from actual state.
Source: §1 · p. 1, §2.2 · p. 3Method
A lightweight wrapper checks whether the intended postcondition holds before retrying and uses idempotency keys when a retry is warranted.
Source: §4.1 · p. 5, §9 · p. 11Results
The wrapper reduces duplicate side effects across tested settings; task success improves for activate_customer, while record_invoice task success is already high.
Source: §6.4 · p. 9Takeaways & Limitations
Evaluating tool use should measure whether agents reach correct final states safely, including whether they avoid redundant external writes.
Source: §6.4 · p. 9The evaluation uses a controlled simulator, two representative workflows, and hand-designed postcondition verifiers rather than production APIs with more diverse behaviors.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Large Language Model (LLM) agents rely on external tools to perform multistage tasks. Existing agent frameworks typically assume that tool calls are atomic and return binary success or failure signals. However, real-world systems exhibit non-atomic behaviors such as timeouts after dispatch, delayed visibility, and partial state updates. These mismatches lead to reliability issues including duplicate actions, task success, and unnecessary tool executions. A lightweight, verification-aware tool wrapper is introduced that augments tool calls with postcondition verification, verify-before-retry logic, and idempotency keys. The approach is evaluated in a controlled simulated environment with injected non-atomic failures across multiple task templates. The results demonstrate that the proposed method significantly reduces duplicate actions, while maintaining comparable task success rates. Overall, the findings suggest that strengthening tool interaction semantics is a promising direction for improving LLM agent reliability without requiring modifications to the underlying language model.
1 Introduction
LLM agents depend on tool responses, but real tools can change external state without returning a response that reliably reflects what happened. The introduction proposes a verification-aware wrapper to check effects and make retries safer.
1 Introduction
- Existing systems often treat tool calls as atomic, although timeouts, delayed visibility, and stale-version conflicts can separate the agent’s response from actual world state.This mismatch can produce duplicate actions, incorrect task completion, and cascading failures in long workflows.Source: §1 · p. 1
- The proposed wrapper adds postcondition validation and failure-aware retries so agents can check external effects, not just select tools.The comparison diagram contrasts a verification-aware strategy with a naive approach.Source: §1 · p. 1, Figure 1 · p. 2
- The work formalizes non-atomic tool failure and proposes verify-before-retry checks, aiming to prevent duplicate actions without modifying the underlying LLM.The contribution passage identifies a taxonomy of four failure modes.Source: §1 · p. 2
2 Background & Problem Formulation
This section models tool calls in terms of actions, responses, and world states, then explains how non-atomic execution breaks the assumption that responses faithfully indicate effects. It identifies practical failure modes and frames their implications for agent reliability.
2.1 How LLM Agent Tool Calls are Modeled
- A tool call is modeled as an action, its response, and the ground-truth world states before and after dispatch.This representation separates what the agent receives from what happens in the environment.Source: §2.1 · p. 2
- Atomicity means the response faithfully represents the state transition, with the action visibly succeeding or failing without ambiguity.The passage relates this to database transactions that commit completely or roll back without exposing partial effects.Source: §2.1 · pp. 2–3
2.2 Why Real Tools Are Non-Atomic
- Timeouts can hide an action that executed, eventual consistency can make a successful write appear absent, and partial execution can leave only some intended effects applied.These behaviors are presented as consequences of latency, weak read-after-write guarantees, and service or isolation failures.Source: §2.2 · p. 3
- Non-atomic responses can make an agent’s belief diverge from ground truth even when both the agent and tool are individually correct.The section narrows a broader concern about complex multi-agent failures to this response-state mismatch.Source: §2.2 · p. 3
2.3 Four Failure Modes of Non-Atomic Tools
- The related-work discussion notes that existing tool-use approaches may omit explicit checks of invocation preconditions and output postconditions.It connects this omission with invalid results entering reasoning and inconsistent representations of world state.Source: §2.3 · p. 3
- The framework tests four failure modes: timeout-after-dispatch, delayed visibility, partial success, and stale conflicts.These cover lost or delayed responses, stale verification reads, incomplete compound effects, and state changes by concurrent actors.Source: §2.3 · p. 3
3 Related Work
The related work describes agent systems that use tools and retries, often assuming responses accurately reflect outcomes. This work connects those assumptions to distributed-systems failure semantics and positions verification before retry as a system-level intervention.
3.1 LLM Agent Reliability
- ReAct and Toolformer exemplify agent approaches centered on iterative tool use, while tool-use benchmarks assume clean, deterministic behavior.The section argues that missing, delayed, or inconsistent responses challenge this assumption.Source: §3.1 · pp. 3–4
3.2 Retry and Error Handling in Agents
- Reflection and retry methods can help correct poor reasoning or planning, but retrying after an action already succeeded can duplicate it.A lost response to a successful payment is given as an example of this distinction.Source: §3.2 · p. 4
3.3 Distributed Systems Foundations
- Distributed systems distinguish at-least-once from exactly-once execution, and idempotency keys can prevent repeated requests from producing duplicate effects.Eventually consistent systems can also delay visibility of updates, a behavior the section connects to agent failure modes.Source: §3.3 · p. 4
3.4 Positioning of this Work
- The proposed system-level framework combines postcondition verification with idempotent execution to address ambiguous tool responses without changing the underlying language model.It is presented as integrable with existing agent frameworks.Source: §3.4 · p. 4
4 Method: Verified Tool Wrapper
The wrapper separates tool effects from their responses, verifies intended postconditions before retrying, and uses idempotency keys to make warranted retries safer. Its workflow tests and verifier design account for duplicate side effects, partial writes, and delayed visibility.
4.1 Design Principles
- Retries follow independent postcondition checks, while idempotency keys enable server-side deduplication when a retry arrives after delayed success.This separates the effect channel from the response channel and avoids treating no response as no effect.Source: §4.1 · pp. 4–5
4.2 Task Suite
- The activate_customer and record_invoice workflows test duplicate side effects and incomplete writes, respectively, using task postconditions rather than object existence alone.The first requires exactly one welcome message; the second requires an invoice record with required fields.Source: §4.2 · p. 5
4.3 Task Success Definition
- Success means every expected action effect is fully and correctly reflected in world state; existence-only checks can accept partial success.Postconditions encode domain knowledge, so adoption requires engineering effort per task.Source: §4.3 · p. 5
4.4 Postcondition Verifier
- Postcondition verifiers return true, false, or unknown; they are read-only, but incomplete or overly strict checks can misclassify state and trigger retries.The three outcomes accommodate state that may not yet be observable under eventual consistency.Source: §4.4 · p. 6
4.5 Idempotency Key Protocol
- A deterministic idempotency key is reused across attempts to let supporting APIs deduplicate effects; without key support, concurrent retries can still produce duplicates.Verification offers best-effort protection where keys are unsupported, while keys provide stronger guarantees.Source: §4.5 · p. 6
4.6 Model and Agent Implementation
- The methods hold the underlying model, tools, and decoding fixed; the baseline retries directly without state checks or idempotency keys, while the wrapper permits at most one gated retry.Ambiguous intermediate states are resolved inside the wrapper rather than surfaced to the model.Source: §4.6 · p. 6
4.7 Verify-Before-Retry Algorithm
- Algorithm 1 takes an action, verifier, idempotency key, and retry limit, and is described as ensuring retries occur only when necessary and safely.The procedure includes waiting with backoff and checking the verifier against current state.Source: §4.7 · p. 7
5 Experimental Setup
The experiments use a controlled simulated tool environment to compare retry and verification strategies under injected faults. The setup specifies the fault conditions and evaluates task completion and duplicate side effects.
5.1 Simulated Tool Environment
- The controlled simulator exposes external world state and injects failures that can complete, partially complete, or become temporarily invisible despite unreliable observations.This provides control over failure timing while preserving the tool-agent action-observation loop.Source: §5.1 · p. 7
5.2 Compared Methods
- The comparison includes direct retry without state checks, verification without retry, and verify-before-retry, distinguishing when retrying may fix failure or duplicate an already-completed action.Verification-only reports failure when the intended postcondition is not verified; verify-before-retry retries in that case.Source: §5.2 · pp. 7–8
5.3 Failure Injection Protocol
- Three fault levels control probabilities for timeout after dispatch, delayed visibility, partial success, and conflict; runs use a fixed seed reused across methods.Table 1 is identified as the source of the simulated environment's fault probabilities.Source: §5.3 · p. 8, Table 1 · p. 8
5.4 Metrics
- The evaluation metrics include task success rate, whether final world state satisfies all task postconditions, and duplicate action rate, whether an action produces repeated side effects.Proportions across repeated runs are reported, with Wilson score confidence intervals and Cohen’s h used where appropriate.Source: §5.4 · p. 8
6 Results
Across fault levels, verification-aware handling preserves strong task success while sharply lowering duplicate side effects. The ablation identifies verification—not automatic retries—as the main driver, and the summary argues reliability evaluation should include execution safety.
6.1 Reliability Gains Grow as Faults Worsen
- For activate_customer, verify-before-retry maintained 100% task success across fault levels versus baseline success of 92% at low fault and 64% at high fault.For record_invoice, baseline success was 100%, 96%, and 96% across low, medium, and high fault; wrapper success was 100%, 96%, and 100%.Source: §6.1 · p. 8, Figure 2 · p. 9
6.2 Verification Sharply Cuts Duplicate Side Effects
- 20%, 44%, and 72% activate_customer duplicate side effects under baseline fell to 0%, 16%, and 20% with the wrapper.For record_invoice, duplicate side effects fell from 32%, 52%, and 76% under baseline to 0%, 16%, and 20%; the largest reductions were statistically significant.Source: §6.2 · p. 8, Figure 2 · p. 9
6.3 Verification, Not Retry, Drives the Gains
- About 80% task success and 20% duplicate actions for verify-only, versus baseline's about 58% and 42%, show verification's central role.Full reached about 72% task success and 28% duplicate actions; these separate ablation-run rates should be compared with one another, not Figures 2 or 4.Source: §6.3 · p. 9
6.4 Summary
- The wrapper checks external state before authorizing retries; reliability evaluation should consider safe execution, not only whether the final state is correct.The discussion notes that duplicate side effects were reduced across tested settings, including tasks where final state may appear correct despite redundant writes.Source: §6.4 · p. 9
7 Discussion
The discussion concludes that verify-before-retry improves reliability under non-atomic failures, while warning that the controlled evaluation and verifier quality constrain the evidence. It argues reliability evaluation should account for safe execution, not only final-state correctness.
Discussion
- Verify-before-retry improves reliability under non-atomic failures, with gains tied to avoiding unnecessary retries and reducing duplicate side effects while maintaining high task success.The discussion says this effect is especially evident when duplicate actions violate task objectives, and also applies when redundant writes leave the final state apparently correct.Source: §7 · p. 9
- An agent may eventually reach the correct outcome through an unsafe execution path, so final-state success alone can obscure reliability failures.The paper recommends evaluating whether the agent reached that state safely.Source: §7 · p. 10
- Stronger tool execution semantics—such as idempotency keys, operation identifiers, or status endpoints—could support safe recovery without changing the underlying language model.The discussion frames this as a direction for future agent frameworks and tool APIs.Source: §7 · p. 10
- Evidence is bounded by a controlled simulator, two workflows, and hand-designed verifiers; outdated or incomplete verifier state can still leave failures unresolved.The evaluation may not capture all real-world failure modes, including stale reads, authentication failures, or incomplete verification logic.Source: §7 · p. 10
8 Future Work
Future work targets more nuanced recovery decisions, stronger postcondition verification, and evaluation beyond the controlled simulator and its two representative workflows.
8 Future Work
- The current framework makes binary completion decisions; future work could use confidence from multiple signals to choose whether to retry, wait, seek human help, or stop safely.The suggested signals include postcondition verification, tool responses, idempotency information, and execution history.Source: §8 · p. 11
- Framework reliability depends on verifier quality, and complex workflows need semantic checks of external-state completeness and correctness rather than simple object existence.The passage identifies automatically constructing or learning postcondition verifiers as a way to improve practicality.Source: §8 · p. 11
- Evaluation was limited to a controlled simulator with two representative workflows, leaving generalization to real APIs and broader agent benchmarks for future study.The proposed settings include databases, email systems, spreadsheets, and web services, with challenges such as eventual consistency and rate limiting.Source: §8 · p. 11
9 Conclusion
Non-atomic tool execution can leave an agent uncertain about whether an action took effect, making unchecked retries risky. The paper presents verify-before-retry as a lightweight wrapper that checks the intended postcondition before retrying, and concludes that external verification and careful retry policies are needed for more dependable agents.
9 Conclusion
- Non-atomic tool execution can apply an action even when the agent receives an error, timeout, stale read, or incomplete response.Retrying without checking the external state can create duplicate actions, incorrect final reports, or false success states.Source: §9 · p. 11
- The lightweight wrapper checks whether the intended postcondition is already satisfied before retrying, avoiding a retry after success and enabling a better-evidenced retry after failure.The paper reports improved reliability without changes to the underlying LLM.Source: §9 · p. 11
- Reliable agents require more than model reasoning: external verification, careful retry policies, and explicit checks of world state are practical steps toward dependability.The conclusion highlights workflows where duplicate or incorrect actions can have real consequences.Source: §9 · p. 11