PaperlayerView exact revision on arXiv

Source-linked AI summary

Stale Does Not Mean Unsafe: Guard Precision for Tool-Using LLM Agents under Infrastructure State Races

Zihao Zheng, Jiayu Long, Baichuan Li, Junyi Yao

arXiv:2609.29522v1cs.AI

TL;DR

Tool-using agents can commit actions based on state that changes after their read, but not every race makes a commit unsafe. This paper replays frozen proposals across paired infrastructure races to compare guard precision. All three guards eliminate unsafe commits, but freshness guards block many benign races while a complete predicate guard blocks none.

  • Problem

    The paper asks which state changes should block a commit, since changed state can leave an action’s safety predicate intact and freshness-only blocking can forfeit availability.

    Source: §I · p. 1

  • Method

    The study pairs race types across infrastructure tasks and counterfactually replays frozen proposals to compare guard granularities, verification, and model-side gates.

    Source: §I · pp. 1–2, A. Counterfactual development data · p. 3

  • Results

    All three commit-time guards eliminate unsafe commits, but freshness guards block 92–95% of benign races while the complete predicate guard blocks none.

    Source: §I · p. 2, B. Guard granularity: safety is easy,… · p. 5

  • Takeaways & Limitations

    Precise enforcement depends on a complete semantic contract; freshness heuristics and model self-assessment do not substitute for it.

    Source: §VII · p. 6, §X · p. 8

    The deterministic simulator cannot reproduce every production race, latency distribution, or operator convention, and its balanced conditions do not estimate production incident prevalence.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Tool-using language-model agents increasingly mutate schedulers, data pipelines, object stores, and access-control systems. Between an agent's read and its commit, external state can change, but not every change makes the commit unsafe. We separate invalidating races, which break a declared safety predicate, from predicate-preserving and irrelevant races, and ask how precisely runtime guards distinguish them. Our deterministic simulator separates visible from authoritative state and injects five non-atomic failure mechanisms across 16 infrastructure tasks in four domains; frozen agent proposals are replayed counterfactually under every controller without an LLM judge. We evaluate three commit-time guard granularities (global epoch, read-set version, semantic commit predicate), multi-level verification, and model-side gates on three locally hosted quantized model families (Qwen3-4B, Phi-4-mini, Gemma4-8B; 3,456 trajectories on one GPU). All three guards eliminate unsafe commits, but their availability differs sharply: freshness-based guards needlessly block 92-95% of benign races, forfeiting up to 43% of safe task completions, while the complete predicate guard blocks none. That precision is contract-dependent: deleting a single declared clause converts exactly its fault family into unsafe commits (up to 7.9%). Model-side signals do not substitute: verbal confidence is miscalibrated (ECE approximately 0.37), action agreement matches a random gate, a cautionary prompt leaves the direct unsafe rate essentially unchanged, and after a freshness-guard block agents re-commit unsafely from refreshed but still-incomplete reads. Under degraded telemetry a hidden concurrent mutation remains observationally clean, bounding every selective policy. Precise runtime enforcement therefore requires semantic contracts, not freshness heuristics or model self-assessment.

I. INTRODUCTION

State can change between an agent’s read and commit, but a changed state does not always make a mutation unsafe. The paper tests how precisely commit-time guards can distinguish unsafe races from harmless ones.

I. INTRODUCTION

  • A commit-time guard that blocks on every state change can prevent unsafe actions but also forfeit availability on changes that leave the safety predicate intact.A notice that state changed may prompt an agent to retry the same unsafe action.Source: §I · p. 1
  • Prior work covers tool failures, stale-state execution, contracts, transactional settlement, and verified wrappers; this paper isolates which changes should invalidate a frozen infrastructure mutation.It does not claim a new failure class, contract language, authorization lifecycle, read-set reconstruction method, or general false-block theory.Source: §I · p. 1
  • The benchmark pairs invalidating, predicate-preserving, and irrelevant races across 16 mutation tasks in four infrastructure domains and five non-atomic failure mechanisms.Frozen proposals are replayed under every controller, with authoritative state reserved for deterministic execution and scoring.Source: §I · pp. 1–2
  • The evaluation compares global-epoch, read-set-version, and semantic-predicate guards with multi-level verification and model-side gates, including contract-deletion and cost-sweep tests.The comparisons report unsafe commits and benign interventions.Source: §I · p. 2
  • Across models, freshness guards sacrifice availability on 92–95% of benign races, while confidence and agreement gates do no better than matched-random controls.A cautionary prompt does not reliably reduce unsafe proposals, and agents can re-commit unsafely after a freshness-guard block.Source: §I · p. 2

II. RELATED WORK

Related work addresses mutable tool environments, agent calibration and abstention, and verified or transactional execution. This paper studies invalidation precision while holding the mutation proposal and commit hook fixed.

A. Dynamic tools and mutable state

  • ToolMaze [1], STALE [2], STT-Arena [3], and TOCTOU-Bench [12] study tool perturbations, memory revision, plan invalidation, or check/use races; S-Bus reconstructs read sets.The paper treats read-set versioning as an invalidation baseline and asks when a dependency change is semantically harmless.Source: A. Dynamic tools and mutable state · p. 2
  • Commit-Time Authorization [8] and AID-Guard build authorization-to-effect protocols; this paper instead varies the invalidation condition and measures unsafe commits and benign interventions.The authors neither reproduce nor rank those protocols.Source: A. Dynamic tools and mutable state · p. 2

B. Calibration and abstention

  • Calibration and abstention work studies process features, verbal overconfidence, and stopping decisions [13] [15] [16]; this paper evaluates confidence and cautionary prompts alongside evidence purchases and commit-time guards.The paper’s action space includes graded evidence purchases and guards between acting and escalating.Source: B. Calibration and abstention · p. 2

C. Verified and transactional execution

  • Runtime-gating theory characterizes false-block/miss frontiers, while ToolGate [4], Atomix [5], Verified Tool Calls [6], and Proof-carrying agents [17] develop verified or transactional execution mechanisms.The paper positions its predicate guard as one granularity point in a controlled evaluation, not a new contract system or authorization protocol.Source: C. Verified and transactional execution · p. 2
  • Agent-Diff deterministically evaluates enterprise API outcomes; this work likewise scores state transitions, adding hidden state, paired races, counterfactual replay, and failure-distribution shift.The empirical replanning test is specific to mutable infrastructure rather than a general enforceability or certification theory.Source: C. Verified and transactional execution · p. 2

III. PROBLEM FORMULATION

The formulation distinguishes changes by whether they affect the declared safety predicate or the versioned read set. Controllers choose among execution, evidence purchases, commit-time guards, or escalation.

III. PROBLEM FORMULATION

  • A predicate-preserving race advances the mutated read resource’s version while leaving the declared predicate intact; an irrelevant race leaves the versioned read set untouched.The formulation distinguishes these from invalidating changes that break the declared safety predicate.Source: §III · p. 2
  • The guards compare a global epoch, read-set version, or the declared commit predicate, while other controller actions execute, buy evidence, or escalate.Each guard performs one authoritative check at identical cost.Source: §III · pp. 2–3
  • For snapshot promotion, the declared predicate requires validation passed, not already promoted, policy allowed, and intact integrity; ALLOW sends the frozen proposal to the ordinary executor.The visible snapshot is shown at version 1.Source: Table I · p. 3
  • The loss combines weighted unsafe commits and failed safe task success with verification and escalation costs; guard precision requires zero unsafe commits and zero benign interventions.Verification costs are 0, 1, or 4, escalation costs 10, and the semantic-check cost is separately swept from 1 to 4.Source: §III · p. 3

IV. RUNTIME CONTROLLERS

The runtime-controller evaluation uses frozen proposals for deterministic counterfactual scoring and tests how commit-time guards behave when their declared contracts omit a condition.

A. Counterfactual development data

  • For each decision, the simulator freezes the proposed mutation and replays controller actions to label safety, completion, benign intervention, and cost without an LLM judge.The controlled setting isolates the guarding question; shadow checks, transaction logs, or incident review could provide analogous deployment supervision.Source: A. Counterfactual development data · p. 3

B. Commit-time guards and contract incompleteness

  • The three guards share one implementation and differ only in the compared payload; the complete predicate rechecks operational status, policy authorization, integrity, and whether the effect was already applied.Stress tests delete one declared condition—policy, integrity, idempotence, or operational status—to model an incomplete contract.Source: B. Commit-time guards and contract… · p. 3

C. Atomicity and deployment mapping

Commit-time enforcement depends on an atomic check-and-mutation hook, while support-aware routing must operate only on observable signals. Hidden concurrent changes can remain indistinguishable from clean observations, limiting selective policies.

C. Atomicity and deployment mapping

  • A checked condition and durable mutation must share one linearization point; a gateway that reads state before calling an unrelated write API does not satisfy this assumption.Read-set guards map to compare-and-swap or entity-tag preconditions; predicate guards require a transactional check and effect.Source: C. Atomicity and deployment mapping · p. 3
  • The simulator assigns each guard one authoritative snapshot read, so its unit-cost comparison isolates semantics rather than implementation engineering.Real predicates may need extra reads or joins, so the cost sweep tests how large that premium can become before another approach is preferred.Source: C. Atomicity and deployment mapping · p. 3

D. Support-aware cost routing

  • Transparent and learned routers choose checks from observable failure signals, with the learned router falling back to strong verification outside development support.Its portable features are observation-age bucket and normalized previous-status family; fault identity and authoritative outcomes are prohibited.Source: D. Support-aware cost routing · p. 3
  • A concurrent mutation can share the visible age-and-status signature of a clean observation, so any policy executing that signature inherits nonzero risk.The evaluation boundary keeps authoritative state available to the hook and deterministic oracle, not as a controller feature.Source: D. Support-aware cost routing · p. 3, Figure 1 · p. 4

A. Infrastructure tasks, failures, and races

The benchmark pairs hazardous and benign races across infrastructure tasks, then tests them under canonical and degraded telemetry. Its main design uses frozen proposals from three model families, with an additional out-of-family replication.

A. Infrastructure tasks, failures, and races

  • The benchmark covers 16 mutation templates across four infrastructure domains, with instructions, visible and authoritative state, evidence levels, and expected state diffs.Table II summarizes the task families.Source: A. Infrastructure tasks, failures, and… · p. 4
  • Five non-atomic failure mechanisms are paired with hazardous and benign instances, distinguishing invalidating, predicate-preserving, and irrelevant races from clean controls.Each model makes 1,152 decisions; deterministic scoring checks preconditions, duplicate effects, target-state changes, and integrity without an LLM judge.Source: A. Infrastructure tasks, failures, and… · p. 4
  • Degraded telemetry aliases failure statuses, narrows age separation, and hides one concurrent mutation behind a fresh normal status, while authoritative outcomes remain unchanged.Source: A. Infrastructure tasks, failures, and… · p. 4

B. Models and inference

  • The three main models are Qwen3-4B, Phi-4-mini-instruct, and Gemma4-8B; Llama-3.2-3B replicates the proposal grid as an out-of-family check.Models use constrained JSON output, common prompts, and one resident model on a single 16 GB GPU.Source: B. Models and inference · p. 4
  • The main design generates 3,456 frozen proposals from 16 templates, four instances, six conditions, three models, and three seeds.A 432-proposal pilot precedes the main design; all replies, timings, proposals, and replayed outcomes are stored as JSONL.Source: B. Models and inference · p. 4

C. Controllers, baselines, and metrics

The study compares runtime guards, verification strategies, and model-side gates using frozen thresholds and held-out tests. Model confidence, agreement, and cautionary prompting do not reliably identify or prevent unsafe proposals.

C. Controllers, baselines, and metrics

  • Controllers include direct execution, fast and strong verification, model-side gates, three commit-time guards, evidence routers, and escalation.Primary metrics are unsafe commits, safe task success, and benign interventions; secondary metrics include cost, coverage, expected loss, worst-group risk, and regret.Source: C. Controllers, baselines, and metrics · p. 4
  • Thresholds and feature choices are frozen on development templates and canonical telemetry, with held-out, degraded-telemetry, fault-family, and domain tests.Uncertainty uses template-clustered bootstrap intervals, and controller differences use paired trajectories.Source: C. Controllers, baselines, and metrics · p. 4
  • The prompt-caution control repeats the 3,456-proposal grid, while replanning tests 1,584 actions after evidence or a freshness-guard block with an ordinary refreshed read.The guard-block arm matches decisions where the read-set version guard fires.Source: C. Controllers, baselines, and metrics · pp. 4–5

A. Model signals are not safety signals

  • Confidence ECE is near 0.37 for all three models, action agreement is no better than matched-random gating, and cautionary prompting leaves Qwen and Gemma’s 41.7% direct unsafe rate unchanged.Phi’s caution prompt lowers direct unsafe rate from 34.6% to 30.1%, while safe success falls from 51.9% to 49.3%; confidence discrimination remains near chance or below.Source: A. Model signals are not safety signals · p. 5

B. Guard granularity: safety is easy, precision is not

Commit-time guards all prevent unsafe commits, but freshness-based guards sacrifice safe completions. Predicate precision depends on complete contracts, and a block followed by an ordinary refresh does not reliably prevent unsafe recommits.

B. Guard granularity: safety is easy, precision is not

  • All three commit-time guards eliminate unsafe commits, but predicate guarding preserves 76.6% safe task success versus 43.2% for version and 34.9% for epoch guards.The global epoch guard blocks 94.6% of predicate-preserving and 91.7% of irrelevant races; the read-set version guard blocks 94.6% of predicate-preserving races.Source: B. Guard granularity: safety is easy,… · p. 5
  • Macro expected loss rises from 3.674 at cp = 1 to 6.674 at cp = 4, with predicate guarding dominating strong verification below a fourfold premium.It ties strong verification at cp = 4 and remains below the version guard’s 6.828 across the scanned range.Source: B. Guard granularity: safety is easy,… · p. 5

C. Predicate precision is contract-dependent

  • Deleting the policy, operational-status, or duplicate-check clause yields 7.9%, 6.4%, or 4.0% macro unsafe commits, respectively, while deleting the redundant integrity clause leaks none.Each omitted clause leaks its covered fault family; the duplicate check subsumes the integrity clause.Source: C. Predicate precision is… · p. 5

D. Blocking is not enforcing: replanning after evidence

  • After a freshness-guard block and ordinary refresh, 14.1% of replans are unsafe and 31.9% of recommitting agents act unsafely.The refreshed read can retain the original blind spot; returned authoritative evidence still leaves 6.9% of replans unsafe.Source: D. Blocking is not enforcing:… · p. 5

E. Shifted telemetry and the observability limit

Under degraded telemetry, the learned support-aware router still leaves unsafe commits because a hidden concurrent mutation resembles a supported fresh, normal state. Transfer across task domains is stronger than across failure mechanisms, while a semantic predicate guard needs no telemetry signature.

E. Shifted telemetry and the observability limit

  • 7.1–8.3% unsafe commits remain under degraded telemetry for the learned support-aware router because a hidden mutation shares its supported fresh/normal signature.It strong-verifies the 50% of degraded-telemetry examples outside development support; the residual risk occurs within support.Source: E. Shifted telemetry and the… · p. 5, Table VI · p. 6
  • On canonical telemetry, the transparent evidence router has no unsafe commits at cost 1.83; under degraded telemetry, it has 7.9% unsafe commits at cost 2.17.Source: E. Shifted telemetry and the… · p. 5
  • All four leave-one-domain-out tests transfer with 0 unsafe at cost 1.33, while all five held-out fault families fall back to strong verification at cost 4.00.Policies transfer across task domains more readily than across failure mechanisms.Source: E. Shifted telemetry and the… · p. 6
  • At unsafe cost 25 the evidence router minimizes measured loss, but uniform strong verification is optimal at costs 100–250; the atomic predicate guard needs no telemetry signature.The predicate guard rechecks semantics rather than inferring risk from observations.Source: E. Shifted telemetry and the… · p. 6

VII. DISCUSSION

  • An atomic semantic re-check is strictly better than freshness guarding at equal cost when the contract is complete, but contract completeness becomes the maintenance burden.Where available, compare-and-swap, idempotency keys, or transactional settlement realize predicate guarding natively; block notices should identify the violated clause to support replanning.Source: §VII · p. 6
  • The study uses small quantized local models for reproducibility on one commodity GPU, but does not establish performance for frontier hosted models.The authors identify larger models and real shadow-mode infrastructure traces as follow-up work.Source: §VII · p. 6

VIII. THREATS TO VALIDITY

The paper’s simulator, balanced scenarios, normalized costs, and declared-predicate taxonomy constrain how its findings transfer to production. Its reproducibility materials document the controlled evaluation, while the conclusion warns against treating the benchmark as evidence of production deployment safety.

VIII. THREATS TO VALIDITY

  • The deterministic simulator cannot reproduce every production race, latency distribution, or operator convention; its templated tasks and normalized costs further constrain transfer.The race taxonomy depends on the declared predicate, and quantized checkpoints served through one backend leave backend and quantization confounds.Source: §VIII · p. 6
  • Balanced hazardous and benign pairs support diagnosis, but aggregate rates are not estimates of production incident prevalence.Verification costs are normalized units, so the fourfold break-even point is a sensitivity result, not a universal operating point.Source: §VIII · p. 6

IX. REPRODUCIBILITY AND ETHICS

  • The artifact records prompts, model and run details, telemetry, proposals, race labels, counterfactual outcomes, and scorer versions; code, manifests, and raw trajectories are slated for release.Unit tests cover execution, paired conditions, telemetry degradation, taxonomy partitioning, and clause-deletion leak sets.Source: §IX · p. 6
  • Figure 2 contrasts coarse guards’ benign interventions, incomplete predicates’ leaked hazards, model overconfidence, and telemetry shift.The complete predicate guard sits at the origin in panel (a).Source: Figure 2 · p. 7
  • Each missing predicate clause converts exactly its fault family into unsafe commits without changing availability.Table VIII summarizes this contract-incompleteness stress test.Source: Table VIII · p. 7
  • The benchmark should not be interpreted as evidence that a guard is safe for unsupervised production deployment.The experiment runs in a deterministic simulator and does not touch production infrastructure or human accounts.Source: §IX · pp. 6–7

X. CONCLUSION

The conclusion argues that stale state is not necessarily unsafe: atomic commit-time rechecks can eliminate unsafe commits, but guard precision determines benign-race availability. It favors complete, clause-audited predicates while noting that model-side signals, retries, and hidden concurrency do not close the gap.

X. CONCLUSION

  • Freshness-based guards forfeit most benign-race availability, while semantic predicate guarding forfeits none and remains preferred up to a fourfold check-cost premium.That preference depends on the predicate contract being complete.Source: §X · p. 8
  • Deleting a declared clause leaks exactly the hazards associated with that missing clause, making predicate-guard precision contract-dependent.The conclusion calls for complete, clause-audited commit predicates.Source: §X · p. 8
  • Model-side signals, cautionary prompts, and post-block retries do not close the gap, while hidden concurrency bounds every observation-driven policy.The conclusion recommends cost-aware evidence routing and conservative fallbacks.Source: §X · p. 8
Loading 2609.29522v1…