Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents

1 Introduction

• Tool writes can time out after taking effect, so blind retries risk duplicate actions while giving up risks skipped work.Distributed systems address this ambiguity through interface remedies such as idempotency keys, conditional writes, and queryable operation status.Source: §1 · p. 1
• LIMBO varies model, harness, contract, and recovery policy across service-boundary faults while grading committed side effects against a ground-truth ledger.It includes late commits, redelivery, and partial batches alongside realistic optional keys and read paths.Source: §1 · p. 2
• 0.5% of episodes: instructed frontier models duplicated writes after lost acknowledgements, while weaker models duplicated in 18%.On these recoverable faults, the model explained 53% of the explained variance in duplicates.Source: §1 · p. 2
• 56% of frontier episodes with an in-flight request and 74% with redelivery produced duplicates; the contract explained 81% of variance.Offering keys on every write reduced duplicates from 28% to 4%, with agents attaching keys in 98% of episodes.Source: §1 · p. 2
• Three production harnesses and a minimal scaffold behaved almost identically, while transparent client retries reduced exactly-once success from 72% to 50%.With keys everywhere and a guard that attaches them, exactly-once success reached 99%.Source: §1 · p. 2
• Agents reported task completion in 90% of episodes that produced a duplicate.Source: §1 · p. 2

2 Related work

• Sun (2026) studies observation-equivalent outcomes, IdempotencyBench (Gopnal Swamy, 2026) measures scripted-agent violations, and Mansoor et al. (2026) test a verification-and-key wrapper.LIMBO adopts counterfactual pairing, reproduces verify-before-retry, and adds late commits, redelivery, production harnesses, contract manipulation, and a belief-visible guard.Source: §2 · pp. 2–3
• Tool-failure benchmarks measure recovery under observable failures, but the cited studies do not track whether a failed write actually took effect.Consequently, task-success scoring does not penalize a duplicate side effect.Source: §2 · p. 3
• Transactional and compensating systems provide runtime mechanisms, while LIMBO measures how often and where they are needed and what a minimal contract-level mechanism buys.AgentRewind treats external calls as irreversible and replays them from its log.Source: §2 · p. 3
• Stateful benchmarks grade against world state, and LIMBO extends this approach with an effect ledger while focusing on verification after failures.Its harness experiments measure behavior under identical faults, complementing source-code studies of harness architectures.Source: §2 · p. 3

3 Recovery under uncertain side effects

• A timeout may hide absent, committed, late, partial, or duplicated outcomes, while reads can be stale or unavailable.The agent observes only the response and what its query tools later reveal.Source: §3 · p. 3
• Observation-equivalent states give the same response and the same pre-visibility reads, so recovery must rely on evidence or actions safe across those states.A timeout can leave absent, committed, and in-flight outcomes indistinguishable.Source: §3 · pp. 3–4
• Exactly-once success requires all required effects to remain at their target counts and no committed effect to exceed its required count, even if an excess is later undone.A refunded double charge still counts as a duplicate side effect.Source: §3 · p. 4
• Any observation-based policy that completes an absent-write world within time T duplicates with probability one in a late-commit world when the delay exceeds T plus read lag.Verify-then-retry works when the write is visible by the read, but fails while the original request remains in flight.Source: §3 · p. 4
• A known in-flight bound permits waiting until that bound plus read lag before verification, but the delay costs that much latency and any finite assumed bound leaves an unbounded tail exposed.Idempotency keys remove this verification-versus-waiting dilemma.Source: §3 · p. 4
• Reissuing with the same key is exactly-once across all five outcome states when the service executes each key at most once and resumes interrupted keyed batches.The policy repeats until acknowledged.Source: §3 · p. 4

4 The Limbo benchmark

• LIMBO combines six simulated services, twelve parameterized task templates, service-boundary fault injection, and a grader that reads the ground-truth effect ledger.Source: §4 · p. 5
• Service contracts model optional idempotency keys, writes that otherwise create new records, eventual read visibility, and a platform with no write listing endpoint.The sandbox also provides virtual waiting, simulated escalation, and a structured final report.Source: §4 · p. 5
• Twelve operational workflow templates specify targets and focal writes, with each instruction asking for every action exactly once unless that cue is removed.Focal writes are annotated for idempotency, verification, and reversibility.Source: §4 · p. 5
• Faults attach to matching focal calls, and ambiguous modes return identical observations across hidden outcome states.Late commits occur after 90 simulated seconds or after deterministic heavy-tailed delays of 40 seconds to 2 hours.Source: §4 · pp. 5–6, Table 2 · p. 6
• The grader reports task success, exactly-once success, duplicates, collateral damage, and overclaim using committed-effect and final-state records.It also classifies recovery behavior from the agent’s own tool calls.Source: §4 · p. 6
• Thirty-one scripted-agent unit tests validate fault semantics, policy behavior, escalation reporting, and detection of collateral damage and overclaiming.The MCP deployment keeps the fault plan and ground truth with the experiment runner and resets harness state between episodes.Source: §4 · p. 6

Models

• Nine models from five providers were evaluated through one gateway, using provider-default sampling and reasoning settings.The evaluated set spans several capability tiers.Source: §5 · p. 7

Minimal scaffold

• The minimal scaffold uses a fixed prompt, task-specific tools, and up to 40 tool calls in a standard function-calling loop.Parallel calls run in order, and the scaffold prompts the model to continue after turns without tool calls.Source: §5 · p. 7

Production harnesses

• Three production harnesses—GitHub Copilot CLI, Hermes, and Codex CLI—were compared with the scaffold through the same model gateway.Each harness retained its own prompts, tool-call protocol, context management, and retry logic.Source: §5 · p. 7

Contract variants

• The native contract supports keys for billing and one social platform, while the keys-everywhere variant extends key semantics to every non-idempotent write.The extension includes resumable batches, late commits, and redelivered requests.Source: §5 · p. 7

Recovery conditions

• Recovery conditions range from prompt changes to middleware wrappers, including verification, waiting, key attachment, and two ground-truth oracles.The wrappers do not see ground truth except for the state and outcome oracles; in harness runs they execute inside the MCP server.Source: §5 · p. 7

Experiments

• E1 varied nine models across fault modes and focal writes, while E2 compared four models across recovery conditions and contract variants.E2 included vanilla, prompt, wrapper, and oracle conditions; E2k repeated vanilla and guard under keys everywhere.Source: §5 · p. 7
• E3 and E3k compared harnesses with and without guard; E4 tested prompt and tool-description changes, an unanswered operator, and guard ablations.E3 and E3k used the same model across all four harnesses for their keys-everywhere design.Source: §5 · p. 8
• E5 tested waiting intervals up to 3,600 seconds under heavy-tailed delays, and E6 paired episodes with and without the closing exactly-once instruction.Comparisons used matched worlds, while regression models included template and template-by-instance random intercepts.Source: §5 · p. 8

6.1 What frontier models get right, and where they fail (RQ1)

• Frontier models duplicate 56% of late-commit and 74% of redelivery episodes; redelivery gives no error, so only an idempotency key sent in advance can prevent it.For lost acknowledgements, frontier models duplicate only 0.5%; weaker models duplicate in 18% of those episodes.Source: §6.1 · pp. 8–9, Figure 2 · p. 9, Table 3 · p. 9

6.2 Where exactly-once lives depends on the fault (RQ2)

• 53% of explained duplicate variance is attributable to the model on read-back-resolvable faults, while the contract accounts for 81% on unresolvable faults.The decomposition differs by fault class; the pooled analysis obscures this distinction.Source: Table 4 · p. 10, §6.2 · p. 10
• On lost acknowledgements, read-before-retry duplicates were 13.4% on eventually consistent paths and 0.8% on strongly consistent paths.Writes with no read-back path were duplicated less often than writes with a read-back path; agents often trusted reads that could not reveal lagging or in-flight effects.Source: §6.2 · pp. 10–11
• With keys on every write and a guard attaching them, late-commit duplicates fall from 68% to 7%, redelivery duplicates from 74% to 0%, and exactly-once success reaches 99%.Without a guard, the keys-everywhere contract lowers late-commit duplicates from 61% to 9% and redelivery duplicates from 74% to 7%.Source: §6.2 · p. 11, Table 5 · p. 11

6.3 Waiting is not a substitute for keys (E5)

• With heavy-tailed delays, one hour of waiting reaches 84% exactly-once success at 49.9 minutes per episode, while keys plus a guard reach 94% at 1.5 minutes.With a fixed 90-second delay, waiting 120 seconds reaches 99% exactly-once success; a short, known bound can therefore make waiting effective.Source: §6.3 · pp. 11–12, Figure 4 · p. 12, Table 6 · p. 12

6.4 Agents do not know what happened (RQ3)

• 90% of episodes that produced a duplicate ended with the agent reporting completion, and 80% also listed no operation as uncertain.The comparison between irreversible and reversible writes was inconclusive for equivalence within ±5 percentage points.Source: §6.4 · pp. 12–13, Table 7 · p. 13

6.5 Mitigations and harnesses (RQ4)

• Across four harnesses, native duplicate rates are 27%, 29%, 30%, and 26%; with keys available, the shared model duplicates in 0% of episodes in each.The keys-plus-guard result also transfers across harnesses without modifying them.Source: §6.5 · pp. 13–14, Table 8 · p. 14, Table 9 · p. 14

6.6 Robustness (E4, E6)

• Removing read-path lag and consistency guidance changes duplicate rates from 22% to 34%; removing the exactly-once cue raises resolvable-fault duplicates from 12% to 22%.Two prompt paraphrases yielded 21% and 22% duplicate rates, against 22% for the original.Source: §6.6 · p. 14, Table 10 · p. 15

Implications for tool and protocol designers

• A standard idempotency-key argument for non-idempotent tools, declared read-back for every write, and documented visibility and in-flight bounds are proposed protocol additions.Without an in-flight bound, verification alone cannot guarantee exactly-once behavior; an ineffective read-back can mislead agents into trusting it rather than escalating.Source: §7 · p. 15

Implications for harness builders

• Harness retries can duplicate non-idempotent writes, while a key-attaching guard can pin keys per intent and track writes with unknown outcomes.The evaluated guard is under 200 lines; the three production harnesses and minimal scaffold behaved almost identically with the same model.Source: §7 · p. 15

Implications for model developers and evaluators

• For failures no read-back can resolve, the model-side safe response is to escalate or report uncertainty, but no evaluated model does so reliably.Benchmarks that grade only final state can miss duplicates later compensated; observation-equivalent pairs and an effect ledger expose them.Source: §7 · p. 15

Limitations

• The study uses simulated services, one model gateway, three restricted harnesses, and short-to-medium tasks, limiting how directly results represent other settings.The heavy-tailed delay distribution is a modeling choice, and the guard is a simple baseline rather than an optimal policy.Source: §7 · p. 16

Broader impact

• The studied failure mode can cause double charges, repeated customer messages, and duplicate deployments; the authors release a benchmark, adapters, and a mitigation using synthetic data and simulated services.The mitigation can be deployed without retraining.Source: §7 · p. 16

Conclusion

• Exactly-once behavior depends on the failure: the model can handle observable lost acknowledgements, but unresolved in-flight or redelivered writes require contract support.Waiting for an unknown in-flight bound is costly and incomplete, while key-less writes cannot be made exactly-once by reasoning alone.Source: §8 · p. 16
• Offering idempotency keys on every write, which agents use, and harnesses that attach and track keys close most of the remaining gap across models and harnesses.Source: §8 · p. 16
• Agents rarely report duplicates they cause, and Limbo, its harness adapters, and episode traces are available to support further work.Source: §8 · p. 16

Scaffold system prompt (vanilla)

• The vanilla scaffold prompt instructs an operations agent to complete tasks through tools, avoid asking the unavailable user questions, and report uncertain operations when finishing.Source: Scaffold system prompt (vanilla) · p. 19

Reliability rules appended in the aware condition

• The aware-condition reliability rules tell agents that timeouts and server errors may follow committed actions, advise checking before repeating, and recommend reusing supported idempotency keys.Source: Reliability rules appended in the… · p. 19

A.1 Tools

• Table 11 catalogs tools and native agent-facing descriptions; contract variants remove consistency notes or add idempotency_key semantics to non-idempotent writes.Source: §A.1 · p. 19, Table 11 · p. 20

Reflection message (reflect condition)

• The reflect condition adds a prompt to reconsider the cause of a tool error and choose the next action.Source: Reflection message (reflect condition) · p. 20

Continuation nudge

• The continuation nudge tells the agent to keep using tools and call finish when done.Source: Continuation nudge · p. 20

Harness preamble (prepended to the task in E3)

• The E3 preamble limits agents to ACME MCP tools, prohibits shell, file, web, and code use, and directs them to escalate when operator intervention is needed.Source: Harness preamble (prepended to the… · p. 20

A.3 Task templates

• Templates span cross-platform posting, multi-write deployments, non-atomic batch logging, refunds, and customer notices, with focal writes identified for each.Each template is instantiated with seeded parameters and specifies required effects and focal writes.Source: §A.3 · pp. 20–21

• Table 12 lists preregistered hypotheses and outcomes, with intervals at 95% unless otherwise stated.Source: Table 12 · p. 21
• Table 13 compares the guard with other conditions on exactly-once success using paired, Holm-adjusted McNemar tests.Pairs share model, task instance, focal write, and fault mode.Source: Table 13 · p. 21
• Figure 5 plots exactly-once success against mean tool calls per episode across recovery conditions and models in E2.Marker area represents duplicate rate.Source: Figure 5 · p. 22
• Figure 6 reports statistically indistinguishable recovery-behavior distributions across models for the notexecuted and committed panels.The figure covers the focal timeout in the minimal-scaffold, vanilla E1 setting.Source: Figure 6 · p. 22

• A late commit duplicated the LinkedIn announcement after the agent checked the post list and retried the timed-out publish.The trace shows the first publish timed out, the read-back succeeded, and a second publish succeeded before another list call.Source: C Case studies · p. 22
• A proactive idempotency key made the Mastodon retry safe after its initial publish timed out.The trace shows the retry carrying the key and the episode completing exactly once.Source: C Case studies · p. 23
• After a LinkedIn timeout, the agent listed posts and escalated to a human instead of retrying the publish.The case outcome was exactly once, with the agent reporting completion.Source: C Case studies · p. 23
• A mail-send HTTP 500 was treated as failure, so retrying the message produced a duplicate despite a completed report.The trace shows the first mail call returning an HTTP error and the second succeeding.Source: C Case studies · p. 23