Source-linked AI summary
Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy
Mazhar Shaikh, Anurag Rajkumar Bombarde, Harshal Pathak
TL;DR
The paper examines how service-mesh reliability assumptions fail for effectful agent delegations. Through a retrospective study of 147 incidents across 81 runs, it identifies identity and evidence adequacy as cross-cutting requirements and derives seven delegation-level primitives, while stopping short of claiming controlled effectiveness.
Problem
Service-mesh retry, timeout, and error-rate mechanisms assume idempotence, latency as failure, and cost-free discards, assumptions violated by effectful agent delegations.
Method
The paper retrospectively analyzes 147 numbered incidents across 81 production runs, using durable records and mutation-based confirmation where available.
Results
The study finds violations of all three assumptions, including successful-call loops, duplicate external effects, misrouted repairs, and enforcement that blocks correct work.
Takeaways & Limitations
Five subsystem failures shared non-discriminating identities, supporting delegation reliability based on discriminating identities, movable attributable evidence, and runtime-proven enforcement.
Takeaways & Limitations
The observational single-system study has no baseline arm, permits confounding, and does not support effect-size claims for the deployed primitives.
Abstract
from arXiv · showhide
Autonomous agents increasingly perform bounded software tasks under an orchestrator that retries, resumes, and budgets them. The machinery such orchestrators reach for is the service mesh's: retry, timeout, and error-rate circuit breaking. We report a failure study of a production agentic software-delivery platform over 147 numbered incidents spanning 81 runs, each with a measured cost and, in most cases, a mutation proof reproducing the failure. All three assumptions those primitives rest on are violated in practice, and we quantify the consequences: a loop of fifty-four consecutive successful tool calls no error-rate breaker could see; a progress signal constant by construction, guaranteeing a false trip on the third repair round and driving one run from six of six components to three; twenty-one events accumulated across six invocations of one delegation, making a correct, idempotent component unwinnable; a misrouted failure that woke five components for a two-component fault, leaving three bystanders regressing working code; and twelve incidents in which the enforcement layer blocked correct work, the most expensive costing 107 agent turns and zero accepted writes. We find one cross-cutting cause and its dual. Identity adequacy: in five separate subsystems an identity that failed to discriminate produced a confident wrong answer, and two of them derived the corrective rule independently. Evidence adequacy: a reliability decision may be taken only on evidence capable of moving, attributable to what it measures, and deterministic under identical conditions. From the findings we derive seven reliability primitives whose enforcement unit is the delegation rather than the message, and specify the controlled evaluation the study motivates but does not constitute.
I. INTRODUCTION
The paper studies reliability failures in effectful agent delegations, where service-mesh assumptions about idempotence, failure signals, and free discards do not hold. It reports a retrospective production failure study and derives reliability primitives while distinguishing observed findings from untested intervention effects.
- Motivation: Service-mesh retry, timeout, and error-rate machinery assumes idempotence, latency as failure, and cost-free discarded requests.Agent delegations instead generate effectful operations at inference time and consume tokens whether their work is kept.
- Study and evidence: 147 incidents across 81 identified runs form a retrospective operational record from a production agentic software-delivery platform.Costs came from durable platform records, and diagnoses were mutation-tested or otherwise confirmed where reported.
- Observed failures: A verifier made 54 successful tool calls over 11 minutes, leaving every error-rate breaker unable to fire until a human stopped the run.No error path was reached, and the workload-sized step budget permitted continued spinning.
- Observed failures: A service accumulated 21 events across six invocations despite an intact producer, making the correct component unwinnable because duplication occurred in the ledger.The event log remained outside the transactional workspace and therefore outside its cleanup mechanisms.
- Contributions: The paper contributes seven measured findings, a cross-cutting identity-adequacy result, and delegation-level Agent Mesh primitives.It also specifies a controlled evaluation motivated by the findings but does not claim how much the proposed primitives help.
- System under study: The platform is a deterministic-DAG delivery system whose closed seven-tool surface makes effects observable at the tool boundary.The measured delivery agent spans 66,185 lines across 59 modules.
IV. FINDINGS
The findings show that agent failures often arise from successful actions that stop changing outcomes, while progress signals and containment boundaries can misclassify or preserve those effects. The resulting evidence supports progress-aware breakers and adequacy checks rather than inherited error-rate assumptions.
- A. F1: agents fail by ceasing to converge, not by erroring: 54 consecutive successful calls made an error-rate breaker structurally blind because the delegation stopped changing outcomes without producing exceptions.The loop ended only when a human noticed it.
- B. F2: a progress signal can be constant by construction: A progress fingerprint based on failed-check identifiers stayed constant because those identifiers described the recovery strategy rather than the round.Two additional recovery paths similarly fingerprinted constants by replay.
- B. F2: a progress signal can be constant by construction: The stall detector tripped on the third round while the graded budget refund could never fire, because both guards consumed constant evidence.Every model-driven repair in the affected class was therefore declared stalled on its third round.
- B. F2: a progress signal can be constant by construction: One run fell from six of six components with cross-service acceptance executing to three after a false stall decision.Evidence had improved from five failing tests to two, with three newly passing and both survivors one dictionary key from green.
- B. F2: a progress signal can be constant by construction: Signal adequacy must be demonstrated: a no-progress signal over identifiers constant by construction guarantees a false trip rather than conservatively stopping.The platform consequently uses nested progress-based scopes and content-keyed session progress.
C. F3: effects outlive the delegations that commit them
Transactional containment protects effects inside the workspace but not external effect classes. The measured consequence is that replayed delegations can expose prior effects, while only explicitly enumerated recovery actions receive durable deduplication.
- Containment boundary: Transactional containment prevents duplicate commits for the effect class it contains, using copy-on-write overlays and declared writable-path checks.A crashed or retried delegation commits nothing twice within that boundary.
- Uncontained effects: Package installation, per-service database writes, post-write hooks, and model-provider calls remain undeduplicated effect classes.Provider retries are described as the most expensive and least visible because tokens are spent without a ledger of prior spending.
- Deduplication: Recovery actions are deduplicated with a durable lease keyed by failure, strategy, strategy version, and budget key, plus restart reconciliation.The design works for developer-enumerated keys, whereas the measured duplicate-effect failure occurred in an unenumerated class.
D. F4: misrouted failure attribution damages correct work
Failure attribution can turn a localized fault into instructions for correct components to modify working code. Misrouted repair briefs also waste substantial diagnostic effort, while enforcement decisions can make correct delegations unwinnable.
- Failure routing: A failure caused by two components woke five, and three bystanders regressed working code after routing based on declared scenario identifiers.The rule conflated test coverage, runtime participation, and ownership of the failed transition.
- Diagnostic cost: One 943-tool-turn repair window spent 451 reads or searches across 71 minutes re-deriving causality from briefs that nominated only unmodifiable files.The discarded causality accounted for 48% of the effort.
- Enforcement failures: Twelve enforcement incidents blocked correct work, and the most expensive consumed 107 agent turns with zero accepted writes.A repair clamp restricted reads and writes to a misdiagnosed target set, preventing access to the defective file.
- Enforcement failures: The enforcement failures reflect encoded assumptions that can diverge from the checked environment, paralleling parameter and upgrade misconfiguration patterns.The paper presents this connection as a sub-pattern rather than as a causal result of the related work.
- Enforcement failures: A mistaken acceptance can be worse than a mistaken rejection because it can foreclose every remaining repair route.An appeal mechanism accepted a correct diagnosis against the wrong file, whereas rejection left the delegation a path forward.
F. F6: some effects are observable only when the whole system runs
Whole-system execution exposes effects and failures that isolated tool, sandbox, and service checks miss. The resulting enforcement and recovery decisions can block correct work, misdiagnose progress, or regress unaffected components.
- Whole-system observability: Three escapes passed every narrower boundary but appeared when the generated application ran as real servers and was probed through gateway and browser paths.The escapes included missing frontend compilation, routes that produced 404s on prefetch, and authentication mismatched with gateway-injected identity headers.
- Enforcement failures: A repair clamp restricted both reads and writes to a misdiagnosed target set, costing 107 turns and zero accepted writes before the component was lost.The defective file lay outside the diagnosis, so the delegation could not read it; twelve incidents ended with correct work blocked or recovery exhausted.
- Enforcement failures: A seed gate matched a handler by callee name, producing 3 corrections plus 2 stronger-model rounds before the component was lost.The platform’s generated dispatch seam was interpreted as having no seed, illustrating how enforcement rules can reject correct implementation.
- Progress and evidence: A constant failure identifier caused a false stall verdict on repair round three, reducing one run from 6/6 completed components to 3/6.The identifier was the failing check name, which stayed unchanged whether the agent fixed four defects or none.
I. Workload and a fully-traced run
The fully traced run records a long, multi-component execution in which repeated failures and stalls were handled through repair, escalation, and re-entry. It is presented as an example of the machinery converging rather than failing.
- One run lasted four hours fifty-five minutes across five components, ten scheduler cycles, and two attempts.Its persisted record contained 12 terminal component failures, 23 stall verdicts, 3 escalated rounds, and 3 stale checkpoints.
- Nine consecutive scheduler cycles in the first attempt ended with the cross-service acceptance component failing.
- The second attempt reached 21 of 21 required outcomes after the run resumed and repaired its failures.
- The machinery rejected, repaired, and approved work while the stall detector fired 23 times without terminating a still-converging component.Three stronger-model rounds were spent through the escalation path.
V. THE CROSS-CUTTING FINDING: IDENTITY ADEQUACY
The paper identifies identity adequacy as a cross-cutting failure cause: non-discriminating identities produce confident wrong answers across five subsystems. It pairs this with evidence adequacy, requiring signals to be discriminating, movable, attributable, and deterministic before reliability decisions act on them.
- Identity adequacy: Five subsystem failures shared an identity that failed to discriminate and consequently produced a confident wrong answer.
- Identity adequacy: The failures involved progress fingerprints, commit identifiers, topology nodes, scenario coverage, and component-coupling measurements.
- Evidence adequacy: Evidence adequacy requires signals capable of moving, attributable to the rule measured, and deterministic under identical conditions.Ambiguous routing evidence must support abstention rather than confident exclusion.
- Evidence adequacy: An identity or evidence failure can make a confident answer wrong or answer a question that was never measured.The paper states that both halves must be checked separately.
- Identity adequacy: Two independently developed subsystems reached the same non-collapsing identity rule for different objects.The topology graph separates identical-shaped state in different processes, while the work planner measures coupling per component.
- Delegation-level primitives: The proposed primitives enforce reliability at the delegation level, covering declarations, budgets, leases, effect commits, retries, and resumes.The design is positioned against model-call, tool-invocation, and commit boundaries rather than one architecture.
VII. DEPLOYMENT OUTCOMES
Deployment outcomes show targeted improvements across signal adequacy, enforcement, routing, verification, and declaration admission. The platform completed a fully green traced run, while the test-driven substrate’s best acceptance result remained seven of eight tests.
- Signal adequacy: Repeated-evidence stops became rare, while a converging replay that previously died on round three proceeded and a stuck replay still stopped.The behavior was verified by reverting the rule in both directions.
- Enforcement-layer exoneration: Five files were admitted through clamp yielding in one run, versus a structurally zero count before the change.
- Failure routing: Blast radius fell from five components to two on the arrangement case.
- Running-system verification: Three escapes were caught by running-system verification that no other boundary reached, and each was subsequently closed at its own layer.
- Declaration admission control: Two services complied with a newly declared physical schema contract on the first attempt, with no correction round.The previous undeclared arrangement had cost seven of eight acceptance tests twice from two different causes.
- Overall deployment outcome: The platform reached a live browser preview and completed the traced run fully green, while the test-driven substrate’s best acceptance result remained seven of eight tests passing.
A. Ablation: removing the verification ladder
The verification-ladder ablation removes selected safeguards in production to identify which defects each rung absorbs. It is diagnostic rather than a controlled delivery comparison, because mode assignment is architectural and the ablated runs still reached live previews.
- Ablation setup: The direct implementation mode removes cluster slicing, test preflight, oracle and mutation qualification, red validation, and freeze for oversized components.It reached live browser previews and therefore serves as a production ablation of the verification ladder.
- Ablation scope: The two arms differ in workload as well as treatment because mode selection is architectural rather than randomized.Thus the design is quasi-experimental, not a controlled delivery evaluation.
- Diagnostic result: The ablation identifies which defects each rung absorbed: every listed defect surfaced only after its rung was removed and was later closed at a layer operating in both modes.
- Mechanism reversions: The constant-vocabulary filter was validated in both directions: reverting it falsely stalled a converging replay on round three, while reverting diagnostics fallback prevented a stuck replay from stopping.
- Interpretation: The ablated runs reached live previews, so the study does not show that the verification ladder is necessary for delivery.It instead shows what each rung was holding up and that the exposed defects matched predicted identity-or-evidence failures.
- Motivated evaluation: The proposed controlled evaluation would compare naive mesh policies with the primitives over the same workload using duplicate-effect rate and work destroyed per fault.Its advance bounds include a 943-turn repair window and a 107-turn zero-write component, with a designed kill criterion.
- Implementation boundary: The effect ledger remains specified rather than built, and the study demonstrates the harm it addresses rather than its effectiveness.Other not-yet-built instruments include persisted effect traces, sidecar observation, distinct refusal verdicts, and per-tenant ledger partitioning.
IX. THREATS TO VALIDITY
The study’s principal validity boundary is observational rather than controlled: incidents came from one changing, self-diagnosed platform, so findings characterize failures but do not establish primitive effect sizes. Additional constraints include survivorship, strict-platform bias, and unbuilt evaluation instruments.
- Observational, not controlled: No baseline arm and concurrent system changes make deployment outcomes before/after observations rather than controlled effect sizes.Incidents were observed rather than induced, and confounding is possible.
- Single system, self-diagnosed: One platform was diagnosed by its builders, so independent replication remains necessary despite mutation-tested guards and reproduced failures.Withdrawn diagnoses were retained, but these safeguards do not substitute for independent replication.
- Survivorship in the corpus: The corpus over-represents failures interesting enough to record, so its incident composition cannot support frequency inference.Individual incident costs are measured, but the corpus is not a sample.
- The effect ledger is unbuilt: The proposed effect ledger is specified rather than demonstrated, so the study shows duplicate-effect harm but not that the primitive works.The relevant delta remains unmeasured.
- Strict-platform bias: Strict enforcement creates more opportunities to block correct work, while permissive orchestration offers weaker guarantees; observed frequencies are platform-specific.The trade-off is presented as potentially general, but the measured frequencies belong to this platform.
X. POSITIONING AND RELATED WORK
Related work falls into authorization, transport, durable execution, orchestration, and capability-comparison lanes, but none addresses whether agent delegations converge safely under retries and enforcement. The paper’s comparison is capability-based because no shared benchmark exists.
- Authorization and transport: Authorization systems govern whether actions are permitted, while transport systems enforce JSON-RPC syntax; neither addresses committed effects or converging delegations.The study instead asks whether authorized actions were already committed and whether enforcement itself is wrong.
- Durable execution: Durable workflow systems reconstruct durable state by deterministic replay, with exactly-once activity semantics relying on developer-supplied idempotency keys at known boundaries.The paper sharpens this distinction empirically by showing where its platform’s durable design stops.
- Capability comparison: Table VI compares capabilities rather than performance because existing systems do not accept the same workload and no shared benchmark exists.Effect-level deduplication is absent when operation sets are generated at inference time, and enforcement correctness is unclaimed across surveyed approaches.
- Agent orchestration: Multi-agent orchestration frameworks provide delegation and tool-use structures but generally rely on ordinary application-code safeguards such as exception retries, ceilings, and timeouts.These affordances are the machinery whose adequacy this study examines.
C. Empirical failure studies
The paper differs from prior empirical agent-failure work by studying infrastructure-level reliability around software-repair delegations rather than reasoning or coordination errors. It connects this gap to tracing, benchmarks, and framework-neutral orchestration while emphasizing observational limits.
- Empirical failure studies: Cemri et al. study cross-framework task-level failures, whereas this work studies single-system infrastructure failures recorded with operational costs.The two studies differ in scope, annotation source, and diagnostic process.
- Automated program repair: The paper’s concern is upstream of patch quality: whether repair runs, routes failures correctly, stops on movable evidence, and avoids blocking correct work.This distinguishes orchestration reliability from automated program repair’s focus on generating correct patches.
- Tracing, attribution, and evaluation: Effect-transition spans record whether effects were committed, supplying routing evidence that call-level distributed-tracing spans cannot provide.This distinction matters when the failure is an absent write rather than a failed request.
- Scope: The findings are stated for hierarchical agent–subagent orchestration generally, because they depend on effectful delegations rather than a particular framework.The stated scope includes inference-time operation generation, token cost, and retry, resume, or peer repair.
- Capability gap: Table VI identifies empty capability columns for inference-time effect-level deduplication and enforcement correctness across prior approaches.The table comparison is capability-based rather than quantitative.
- Empirical failure studies: The study reports that 147 recorded failures violated service-mesh assumptions about idempotence, latency signals, and free discards, with concrete consequences across retries, breakers, ledgers, attribution, and enforcement.Reported examples include a 54-call invisible loop, 21 events across six invocations, and 12 enforcement-blocked incidents.
- Cross-cutting result: Five subsystems exhibited the same identity-discrimination failure, while the derived primitives require discriminating identities, movable evidence, and runtime proof that enforcement does not block correct work.The study presents this as a cross-cutting result, not as a controlled effectiveness estimate.