PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents

Chenhao Wu, Haoxuan Jia, Yang Liu, Yingguang Yang, Yuhan Lin, Chongyang Zhang, Hao Zheng, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Shang Luo, Kefu Xu, Jifeng Zhu, Bin Chong

arXiv:2608.27141v1cs.CRcs.AI

TL;DR

Autonomous agents preserve work and adversarial progress across iterations, but common safeguards reset safety state at each trajectory. This paper formalizes the resulting composition failure, introduces LoopHarness with persistent non-decaying state, and reports both theoretical bounds and evaluation results. The framework limits expected unauthorized irreversible actions by a horizon-independent constant while retaining high clean-task completion in the reported experiments.

  • Problem

    Existing agent safeguards are scoped to single trajectories even though autonomous loops preserve state across many iterations, leaving cross-iteration fragmented attacks outside their observation window.

  • Method

    LoopHarness wraps an inner trajectory defense with persistent provenance, risk, memory-integrity, arbitration, and compounding-risk governance components.

  • Results

    0.1% overall ASR with 96.9% CleanGC was reported for B4, compared with 88.4–97.6% ASR for B0–B3; fully colluding checker conditions retained 0.1% ASR and reduced CleanGC to 92.9%.

  • Takeaways & Limitations

    Persistent, non-decaying loop-level safety state addresses fragmented evidence and yields an expected unauthorized-commit bound that is constant in the horizon, with a model-free component.

  • Takeaways & Limitations

    The theory bounds an expectation rather than guaranteeing absence, is a scaling statement rather than an empirical crossover claim, and depends on an environment-sensitive detection floor.

Abstract

from arXiv · show

Large language model agents are increasingly deployed as autonomous loops. Starting from one human goal, such a system repeatedly discovers work, plans, executes tool calls, verifies outcomes and persists state across many unattended iterations. The agent safeguards in wide use, however, are defined over a single trajectory, and their safety state is re-initialized when the next trajectory begins. We show that this is a failure of composition rather than an implementation detail. Our central result is a separation: against an attack whose evidence is fragmented across several iterations, every trajectory-scoped monitor has a true-positive rate equal to its false-positive rate, however expressive it is, because the evidence it would need never appears in the window it sees, whereas a monitor retaining cross-iteration state separates the two perfectly. We further show that the obvious repair of carrying a geometrically decaying risk score is insufficient, because the cooling-off period a patient adversary must wait is a constant that does not grow with the horizon $N$. We then present LoopHarness, which restores a persistent, non-decaying safety state at the loop level. Under mediated commits and an arbiter detection floor $δ_M$, it bounds the expected number of unauthorized irreversible actions by $B+m-1+m/δ_M$, a constant in $N$, of which the $B+m-1$ term is decided by a model-free rule and therefore survives a fully colluding verifier. We give a complete evaluation protocol on native Agent-SafetyBench tasks with paired clean and attacked episodes, an outer-state attack suite whose decisive evidence exists only across iterations, per-module ablations, and an adaptive white-box red team.

1 INTRODUCTION

Autonomous loops preserve adversarial progress across iterations while common safeguards reset safety state per trajectory. The paper formalizes this composition failure and proposes LoopHarness with persistent state and horizon-independent guarantees.

  • Autonomous agents repeatedly discover, execute, verify, persist state, and continue work from one human goal across unattended iterations.
  • Existing safeguards and benchmarks remain scoped to individual rollouts or episodes, while attacks can plant content that activates later.
  • LoopHarness retains grounded risk evidence, memory integrity, stopping arbitration, provenance monitoring, and compounding-risk control across iterations.
  • The paper proves that trajectory-scoped monitors cannot separate fragmented attacks from benign traffic and that geometric decay provides only constant cooling-off.
  • E[U1:N] ≤ B + m −1 + m/δM is horizon-independent, with its deterministic term surviving a fully colluding verifier.

2 PRELIMINARIES

The paper models autonomous agent loops as repeated inner trajectories connected by persistent memory and control state. It distinguishes operator authorization from the harness’s deterministic proxy and defines the evaluation quantities used later.

  • Autonomous agent loops: A goal specification includes instruction text, capability allowlists, anchors, and task contracts governing risk clearance and completion limits.
  • Autonomous agent loops: Each episode runs N iterations that discover work, admit items, execute inner trajectories, verify outcomes, persist memory, and decide whether to continue.
  • Authorization: Authorization is defined by a hidden operator oracle, while the harness uses a deterministic proxy combining capability tokens, scope, provenance, and contract coverage.
  • Single-trajectory lifecycle defenses: The inner harness represents current single-trajectory defenses including input sanitization, contextual verification, privilege ceilings, capability mediation, and rollback.
  • Per-trajectory reset: Per-trajectory reset reinitializes safety state to θ0 independently of previous trajectories, enabling stateless analysis but exposing cross-iteration exploitation.
  • Evaluation quantities: Evaluation reports compounding attack-success rate, AUC, clean goal completion, per-iteration ASR, over-halt, under-halt, and detection timing.

3 LOOPHARNESS

LoopHarness wraps single-trajectory defenses with persistent admission, integrity, risk, arbitration, and governance state. Its non-decaying latch blocks repeated reopening opportunities and supports horizon-independent bounds on unauthorized commits.

  • LoopHarness: LoopHarness executes a fixed lifecycle: rehydrate state, admit and gate work, run the inner harness, arbitrate outcomes, persist accepted state, update risk, and continue or halt.
  • LoopHarness: IPM admits work only after scope, trust, rejection-history, authorization-record, and lineage checks pass.
  • Cross-iteration risk cumulant: XRC folds each iteration’s maximum grounded attack risk and flags into persistent state, while structural flags latch the discount permanently.
  • Cross-iteration risk cumulant: Impact risk and verifier uncertainty affect current routing but never enter retained risk or latch state, preventing ordinary high-impact work from degrading later iterations.
  • Memory integrity: MIG authenticates append-only memory, enforces monotone trust, re-verifies checkpoints, and recalls only a bounded suffix of trusted entries.
  • Guarantees: A trajectory-scoped monitor has TPR = FPR on s-fragmented attacks, whereas cross-iteration state can attain TPR 1 at FPR 0.
  • Guarantees: Geometric decay reopens a gated capability after k⋆ constant iterations, while a latch prevents reopening after passive waiting.
  • Guarantees: E[U1:N] ≤ B + m −1 + m/δM does not depend on N, and B + m −1 holds even when the checker is fully colluding.

4 EXPERIMENTS

The evaluation uses native Agent-SafetyBench simulators, paired clean and attacked episodes, outer-loop attack suites, and controlled configurations to test LoopHarness. LoopHarness substantially lowers attack success while preserving clean-task completion, and non-decaying XRC is critical against cooling-off attacks.

  • Protocol: All tasks come from Agent-SafetyBench simulators, with one fulfillable benchmark task scheduled per iteration under a shared goal.The scheduled feed understates the admission problem that IPM would face in deployment.
  • Configurations: B0–B4 isolate the inner harness, decaying cumulant, naive arbiter, post-hoc supervisor, and complete five-component LoopHarness.The configurations are implemented through one switch object, enabling matched comparisons.
  • Results: B4 reduced overall ASR to 0.1% versus 88.4–97.6% for B0–B3 while retaining 96.9% CleanGC.Matched removals restored 96.0–97.5% ASR on their intended attack families.
  • Results: B1 and B1.5 cASR rose from 2.9% at N = 5 to 6.2% at N = 10, while B4 stayed at 0% with no risk-charged commits.The held-out Track-A sweep used N ∈ {5, 10, 20}.
  • XRC retention: In the matched A14 sweep, XRC-decay increased ASR from 0.0% to 97.0% as retained risk fell, whereas XRC-latch kept ASR at 0.0% across all intervals.CleanGC remained 95.5–97.5%, but the evaluated latch lacked an authenticated clearance transition.
  • Statistical protocol: The protocol fixes model roles, temperature-zero calls, seeds, caching, content-hash episode randomness, frozen hyperparameters, and paired bootstrap comparisons.Each reported matrix uses one frozen cohort, and clean completion is distinguished between paired targets and all slots.

5 RELATED WORK

Related work spans autonomous-agent capabilities, persistent attacks, single-trajectory defenses, oversight, runtime verification, and durable-workflow semantics. The paper positions loop-level non-decaying retention as adding a safety property beyond bookkeeping and existing trajectory-scoped controls.

  • Agent loops: Reasoning-and-acting, tool-learning, self-reflective, embodied, software-engineering, and simulated-society agents instantiate the long-running loops studied here.Capability suites measure these systems over long horizons, while safety evaluation has remained organized around single tasks.
  • Persistent attacks: Persistent attacks poison memory or retrieval so later interactions are compromised; Track B removes the lexical payload and ends with a native benchmark action.The paper distinguishes this outer-loop setting from indirect prompt injection within one rollout.
  • Oversight and control: The arbiter follows control-evaluation work on monitors under model subversion, while decorrelation addresses correlated failures in debate and scalable oversight.The governor draws on safe-reinforcement-learning shields, and gate checks and rehydration implement runtime verification over traces.
  • Durable execution: Durable-workflow practices contribute exactly-once completion, authorization recency, and integrity-checked persisted state, but retention adds Corollary 4 beyond ledger bookkeeping.The paper identifies missing transaction semantics in agent harnesses and separates them from cross-iteration risk retention.

6 CONCLUSION

The paper characterizes single-trajectory safety as a composition failure for unattended loops and responds with persistent, non-decaying loop-level state. It reports horizon-independent protection and identifies detection quality, multi-agent extension, and escalation operations as important boundaries.

  • Conclusion: A trajectory-confined monitor cannot separate fragmented attacks from benign work, and geometric decay leaves a constant cooling-off period.The conclusion frames both results as failures of composing single-trajectory safeguards across iterations.
  • Conclusion: LoopHarness bounds expected unauthorized irreversible actions by a horizon-independent constant, with a deterministic component requiring no model call.The conclusion identifies persistent, non-decaying loop-level safety state as the response.
  • Extensions: The stated extensions are improving the detection floor and carrying the same safety state into multi-agent loops.The multi-agent case changes which entity resets state, from the loop to a delegated sub-agent.
  • Scope: Attack contracts are instantiated only in sandboxed benchmark simulators rather than live systems, accounts, or third parties.The attacks are described structurally through violated contracts such as provenance, authorization atomicity, memory integrity, risk retention, and exact-once completion.
  • Limitation: A halt-and-escalate mechanism depends on a staffed escalation path; without one, the governor can become an availability failure.This is the paper’s explicit residual deployment risk.

B.1 PROOF OF PROPOSITION 2

A trajectory-scoped monitor cannot distinguish an s-fragmented attack from benign iterations because each observed inner view has the same distribution. Persistent state enables perfect separation, while repeated per-iteration risk can compound unauthorized actions over the horizon.

  • No trajectory-scoped monitor, however expressive, separates an s-fragmented attack from benign traffic because their observed distributions are equal.Each attack fragment matches a benign inner view, so the monitor never sees the jointly dangerous evidence within one trajectory.
  • 1 −(1 −q)^s episode detection on fragmented iterations exactly matches the alarm probability on s benign iterations.Requiring episode detection 1 −β forces q ≥1 −β1/s, making the clean-episode alarm rate over horizon N at least 1 −βN/s.
  • A persistent-state monitor alarms when all s fragments accumulate, detecting the attack with probability one while never alarming on benign episodes.
  • Under repeated per-iteration exposure with conditional violation probability at least ε, cumulative ASR reaches at least 1 −(1 −ε)^n and expected unauthorized actions grow as Θ(N) when Ui ≤m.The lower bound assumes ε remains constant across iterations; the upper bound follows from Ui ≤m almost surely.
  • The constant-ε compounding statement is a modelling assumption and is empirically wrong for deterministic guardrails that block the same payload on every retry.The fragmented-attack separation does not depend on that assumption.

B.3 PROOF OF PROPOSITION 3 AND COROLLARY 4

Geometrically decaying risk retention only delays gate reopening by a constant number of iterations independent of horizon N. A loop-structural latch instead keeps the gate closed after qualifying evidence, although nonstructural evidence does not trigger this protection.

  • k⋆=⌊x⌋+1 is the cooling-off period before a geometrically decaying gate reopens, and it is independent of N.The constant is fixed by γ, ρ and θ; when ρ < θ/γ, k⋆=1, reproducing reset behavior.
  • A loop-structural flag latches the discount, maintaining Ξi ≥ρ ≥θ and preventing the gate from reopening after iteration τ.
  • Reset and geometric decay each admit the gated item for Ω(N) iterations because their post-evidence delay does not grow with N.
  • Nonstructural grounded evidence leaves γi=γ↓, so the constant cooling-off bound still applies even in a non-decaying configuration.Suppressing structural cues is identified as the adversary’s principal lever against retention.

B.4 PROOF OF THEOREM 5

Theorem 5 combines a deterministic budget argument with an arbiter-based probabilistic argument to bound unauthorized irreversible actions independently of horizon N. The deterministic component remains valid even when the checker colludes with the writer model, while contract-invisible actions are limited by policy scope.

  • Deterministic half: B + m −1 unauthorized commits are bounded surely for every N and adversary, without probabilistic assumptions or model outputs.Rejected commits monotonically accumulate in the governor counter, which halts at budget B; at most m additional commits execute before the next test.
  • Probabilistic half: m/δ_M bounds the expected contract-invisible unauthorized-commit contribution, with a geometric tail bound independent of N.Arbiter detection at least δ_M stochastically dominates the undetected sequence, and each relevant iteration contributes at most m commits.
  • Uniform form: The resulting split bound is constant in N, while the uniform alternative mB/δ is looser when π is small.The uniform form counts productive iterations using the combined charge-or-flag probability, whereas the split form separates deterministic and probabilistic contributions.
  • Checker collusion: The deterministic bound holds for every checker, including a writer model operating under adversarial instructions, so collusion can affect only the m/δ_M term.The argument uses only the deterministic authorization proxy and assumptions (A1) and (A3).
  • Policy boundary: Contract-invisible commits are accepted by the declared policy, making their residual unauthorized class the gap between operator-declared scope and intent.Narrowing A or strengthening Π raises π, but completely closing the gap requires a policy specification not provided here.
  • Integrity assumptions: Tamper detection covers authenticated log state and counters, while truncation after the most recent checkpoint remains outside the guarantee.The residual truncation window is bounded by the checkpoint period, specified here as one iteration.

C ADDITIONAL RESULT TABLES

The additional tables test component interactions, compounding success over horizon, availability, verification assumptions, and safety–utility–cost trade-offs. They show strong full-system safety alongside an availability boundary and a substantial dependency between retained risk state and enforcement.

  • Component interactions: Removing CRG increased A14 ASR from 0.0% to 97.0%, disproving the prediction that every unlisted off-diagonal effect was below 2 percentage points.The result reveals a dependency between XRC state retention and CRG enforcement.
  • Safety–utility frontier: B4 achieved 100.0% FullGC and 0.0% over-halt on clean Track-A episodes while maintaining zero risk-charged commits at every horizon.Its 15.7% under-halt rate indicates that preventing unsafe commits did not require halting every attack episode.
  • Availability: 0.0% of B4 episodes achieved FullGC after a latch-only attack, exposing the availability boundary of persistent latching.The availability condition isolates the failure mode in which safety state latches without an authenticated clearance transition.
  • Compounding attack success: B1 and B1.5 had overlapping compounding success curves, while B4 remained at zero; under B1 and B1.5, per-iteration successes occurred only at n = 3 and n = 8.Figure 2 reports 95% parent-cluster bootstrap intervals over 97 OOD parents and 485 attack episodes per configuration.
  • Adaptive red team: Table 6 stress-tests the empirical premise behind the m/δ_M term using 18 adaptive variants, but its mitigation rate is not a direct estimator of uniform detection.The reported rate is conditioned on mounted attack attempts rather than contract-invisible unauthorized commits.

D COMPONENT SPECIFICATIONS

The component specifications define the controller’s configuration space, structural latch conditions, tiering rules, and evaluation setup. The design separates loop-structural evidence from review-only conditions and ensures that tool classification does not distort safety metrics.

  • Evaluation configurations: The evaluation varies cue suppression, attempt spacing, and delivery channel across 18 adaptive variants per parent under B4 and A-collude.Table 6 groups rows by the varied axis and reports Attempts plus the minimum mitigation rate over variants with mounted attempts.
  • Configurations: Every configuration uses the same controller code path; B4 is full LoopHarness, while each ablation removes exactly one component and B0 runs neither inner nor outer defenses.The configuration matrix preserves comparability across baseline and ablated settings.
  • Evaluation protocol: All hyperparameter values are fixed on the development split and transferred unchanged to the held-out split.Concrete values and sweep results are provided in supplementary material.
  • Structural flags: The latch responds to discrete loop-structural contract violations, while verifier indecision and audit bookkeeping never latch the cumulant.Review-only conditions such as arbiter review requests and unavailable judges are excluded by design.
  • Tool tiering: Token-level tool-name rules give read verbs precedence because naive substring matching would misclassify read-only tools and inflate commit counts and apparent baseline severity.The classification rule is therefore part of the metric validity, not merely an implementation detail.

E ATTACK CONTRACTS

The attack-contract taxonomy separates inherited single-trajectory families from LoopHarness suites whose violations depend on outer loop state. Track B attacks establish state across iterations and pair attacked episodes with clean twins.

  • Taxonomy: Track A covers A6–A10 with in-trajectory evidence, whereas Track B covers A11–A15 whose violations require outer state except A13’s within-iteration evidence conflict.A1–A5 are inherited from SafeHarness and are listed only for numbering continuity.
  • Track B contracts: A11 resubmits rejected semantic work with earlier authorization, so prevention requires joining re-entry detection with authorization recency.The attack uses an advisory record followed by later resubmission of the same semantic work.
  • Track B contracts: A14 places an early grounded risk observation before the risk-gated native target, requiring retention across Proposition 3’s cooling-off period.The attack stays below the global force-halt threshold while delaying the target.
  • Track B contracts: A15 resubmits a completion anchor after success, so prevention requires an exact-once completion contract enforced through a persistent ledger.The contract tests whether prior completion state survives across iterations.

F IMPLEMENTATION AND REPRODUCIBILITY

The implementation uses one switch-parameterized controller so configurations share the same code path, while the regression suite tests the behaviors on which the theory depends.

  • Implementation: A single switch-parameterized controller ensures every Table 7 configuration exercises the same code path, preventing ablations from silently becoming no-ops.The monotone-trust policy resides in the memory guard, so removing that guard genuinely removes the protection.
  • Implementation: The execution scheduler independently marks advisory, authorization, lineage, and risk-observation records as non-executable.
  • Reproducibility: The regression suite pins latch modes, budget accounting, forced halt and recovery, tamper detection, rollback, restart survival, and monotone-trust auditing.
  • Reproducibility: The suite also tests intake gates, capability-token mediation, tool-description integrity, and the two-stage inner cascade.
Loading 2608.27141v1…