Source-linked AI summary
The Irreversibility Budget: Fleet-Level Risk Accounting and Admission Control for Agent Operating Systems
Bardia Mohammadi, Laurent Bindschaedler
TL;DR
Agent fleets can collectively overdraw risk because existing controls authorize effects individually, especially under shared triggers. The paper introduces a trusted runtime that prices residual loss and charges a cumulative budget across principals. In controlled evaluations, local gates overdrawed by up to 48×, while the budget kept correctly charged runs within the risk limit; conservative, dependency-aware pricing remains unresolved.
Problem
Existing controls check effects individually and lack cumulative accounting across agents, workflows, and tenants, leaving heterogeneous, adversarially declared, and correlated exposure difficult to manage.
Method
The irreversibility budget treats residual exposure as a first-class resource that a trusted runtime prices below the agent, reserves across hierarchical principal ledgers, and admits before commit.
Results
48×: per-effect gates overdrawed the tenant’s risk limit at one thousand agents, while the budget held every correctly charged run within the limit.
Takeaways & Limitations
Cumulative irreversibility accounting gives agent operating systems a mechanism for fleet-level admission control beyond independent effect gates.
Takeaways & Limitations
The study establishes an accounting mechanism rather than a deployed system and does not show that value-at-risk, calibration, or distributed-ledger availability is solved.
Abstract
from arXiv · showhide
Fleets of LLM agents now externalize effects that cannot be fully undone: they move money, deploy code, delete data, and disclose information. Current controls check one effect at a time, so a fleet of individually authorized agents can overdraw its principal's risk under a shared trigger while every local gate stays correct. We propose the irreversibility budget, a cumulative account of residual value-at-risk that a trusted runtime maintains for each principal across agents, workflows, and tenants. Treating irreversibility as a first-class resource, the runtime charges each effect its residual loss below the agent and denies the marginal effect once the aggregate would overdraw the budget. Getting the price right is hard, because effects are heterogeneous, adversarially declared, and correlated. We perform a controlled study in which per-effect gates admit fleet-level overdraws of up to 48 times the tenant's risk limit while the budget holds every correctly charged run within that limit. Conservative, dependency-aware pricing remains the central open problem for a deployable design.
1 Introduction
Agent fleets can create irreversible external exposure that local controls do not cumulatively manage. The paper proposes an irreversibility budget and evaluates it against fleet-level overdraws.
- Agent fleets move money, deploy code, delete data, and disclose information, creating residual exposure that depends on effects, recovery, and concurrent spending.
- Existing controls check individual effects, so jointly triggered agents can remain locally authorized while committing an unapproved aggregate position.
- The irreversibility budget is a cumulative residual value-at-risk account maintained by a trusted runtime for each principal.
- The runtime charges each effect below the agent’s trust boundary and denies or escalates the next effect when the shared balance would exceed its authorized budget.
- 2.4× average and 48× at one thousand agents: local gates overdraw the risk limit, while the budget holds every run within the limit.
- The paper contributes a typed cumulative admission quantity, a trusted hierarchical ledger runtime, and a controlled feasibility study identifying dependency-aware pricing as an open requirement.
2 Background and Motivation
The paper frames fleet-level exposure as a missing operating-system resource: local permissions do not ensure aggregate safety when effects differ, agents can misdeclare them, and shared triggers correlate actions.
- Fifty procurement agents may respond simultaneously to a shared price-spike trigger, making the example’s parameters a fleet-level evaluation scenario.
- The same aggregate-accounting problem extends beyond procurement to payroll, infrastructure automation, deletion, customer communication, and incident response.
- Counting effects as identical is unsafe for irreversible actions and unusably conservative for reversible ones, so reversibility must affect pricing rather than act as a binary gate.
- Existing mechanisms control local effects or domain-specific counters but do not maintain a heterogeneous exposure account across agents.
- Heterogeneous effects rule out face-value caps, while adversarial agents can split orders or misdeclare effect types to evade trusted charges.
- Shared triggers can correlate losses, so deployable admission must account before commit, canonicalize types, and price dependence.
3 Resource Model
The resource model defines irreversibility as a priced, budgeted exposure rather than a count of actions. Risk pressure then turns remaining budget into an admission-scheduling signal.
- The model separates an effect charge, a principal’s budget, and a risk-pressure signal that drives admission.
- For each effect, charge c(e) is residual loss after compensation, recovery, and revocation over a configured horizon.
- At confidence level γ, c_γ(e) is the γ-quantile of the residual-loss distribution; charges may be scalar or vector-valued.
- Each principal receives a windowed budget B_p, and the runtime maintains reserved plus committed exposure against it.
- Short replenishment windows bound bursts, while longer windows bound campaigns.
- Risk pressure tightens admission as free budget falls and can ultimately freeze externalization, functioning as a scheduling signal rather than a safety proof.
4 The Runtime
The runtime makes exposure observable and prevents shared-budget double spending below the agent’s trust boundary. It prices, reserves, and schedules effects along the agent-to-tenant principal path under explicit ledger assumptions.
- The runtime observes exposure before commit, prevents double spending, and decides admission as budgets become scarce.
- Risk-typed specifications declare effect classes, bounds, reversibility paths, compensation, authority requirements, and dependency hints for trusted pricing.
- Canonicalization binds effect types to tool identity, signed specifications, endpoint schemas, and settlement receipts rather than caller declarations.
- Charges begin conservatively, are calibrated by replay and stress cases, audited against receipts, and adjusted for declared or inferred dependencies.
- The reserve algorithm prices an effect below the agent, checks every ledger on its principal path atomically, and reserves the charge only if all budgets permit it.
- Hierarchical ledgers use reserve, confirm, and cancel operations so speculative branches can refund exposure after cancellation or successful compensation.
- Under rising risk pressure, the scheduler tightens confidence, demotes low-value spenders, expires reservations, batches approvals, or uses separately audited exception allocations.
- The guarantee is limited to declared charge units and assumes authenticated principals, strongly consistent authorities, idempotent lifecycles, and trusted effect specifications.
5 Evaluation
The controlled evaluation tests whether fleet-level budgets prevent aggregate overdraw, how safety scales with liveness, and how typed pricing behaves under misdeclaration and correlated recovery. Across these tests, local gates overdraw under composition, while pre-commit budgeting improves safety but depends on conservative pricing.
- 5.1 Composition (RQ1): Local gates overdraw at 2.4× tolerance on average, while the budget overdraws in none of 300 runs and settles near 0.48× tolerance.The budget’s 95% upper bound on overdraw probability is 1.3%.
- 5.2 Scale and alternatives (RQ2): 48× tolerance is reached at a thousand agents under local gates, while the budget remains near 0.48× at every fleet size.Fragmentation transfers 3.9× tolerance past local gates, whereas the ledger admits only $459k because charges attach to exposure rather than call count.
- 5.3 Liveness (RQ3): 93% of routine traffic survives with B = R/2, while enlarging the budget to B = 2R permits overdraw in 33% of runs.Per-workflow ledgers raise an innocent sibling’s throughput from 48% to 89%, and rate-limited re-authorization restores post-burst liveness to 93%.
- 5.4 Typing and pricing (RQ4): Typed charging executes 1.8× the value of a face-value cap and 1.7× the strongest pooled cap when charges are right.The gain shrinks to 1.2× as effect classes converge.
- 5.4 Typing and pricing (RQ4): Misdeclaring colluding transfers as refundable overdraws in 77% of runs, while recovery depression from 0.75 to 0.25 makes additive charging overdraw in 59% of runs.A five-tick detection lag still overdraws 39% of runs during a fifty-tick burst.
- 5.5 Trace evidence (RQ5): Shared context propagates the same external effect across 84% of a heterogeneous fleet on average, confirming the correlated-trigger regime targeted by the budget.The effect-selection statistic reaches 𝑧 up to 198 against an independent-agent null.
6 Discussion and Limitations
The proposed budget introduces important limitations around pricing, correlation, distributed enforcement, adversarial spending, and liveness. These constraints define requirements for a deployable system rather than being fully solved by the study.
- Pricing and budget setting: Value-at-risk pricing is convenient but may be unsuitable for non-commensurable losses and is not subadditive.The paper notes that vector budgets may be needed for data exposure, availability damage, and reputational harm, while expected shortfall is coherent.
- Correlation: Correlation-aware charging remains open because shared triggers can make independent-looking effects jointly risky.The paper proposes workflow correlation classes, shared-trigger detection, and marginal charges under an explicit dependency model.
- Distributed accounting: Strong distributed budgets require coordination, while advisory budgets reconcile later and cannot enforce safety.The runtime should distinguish these modes, shard by principal hierarchy, and report the availability cost of strong authorities.
- Adversarial spenders: Adversarial agents can fragment actions, hoard reservations, misdeclare types, or manipulate refunds, making canonicalization and audited re-authorization necessary.The shared budget itself becomes a target, so re-authorization must be rate-limited rather than unlimited.
- Liveness: Budget enforcement can replace unsafe commits with denial, contention, starvation, or reservation denial-of-service.Reservation expiry, workflow sub-budgets, fair-share admission, replenishment horizons, and overrides are identified as scheduling policies.
7 Related Work
The paper combines mechanisms from database transactions, operating-system accounting, financial risk control, and agent safety. Its distinction is applying them together as a cross-principal admission resource for heterogeneous agent effects.
- Positioning: The novelty is combining established mechanisms as a cross-principal admission resource for typed, heterogeneous, adversarial effects.The paper positions irreversibility budgeting as a joint use of mechanisms from four existing areas.
- Database and OS primitives: Database escrow and saga compensation provide reserve-confirm-cancel, while resource containers shift charging from processes to principals.Coordination avoidance separates invariant-preserving local updates from updates requiring agreement.
- Agent effect controls: Existing agent effect controls protect individual effects or branches, whereas the budget adds cumulative accounting across agents, workflows, and tenants.Those controls regulate settlement time, semantic authority, or payload validity rather than fleet-wide exposure.
- Risk pricing and safe learning: Prior risk-pricing and safe-learning approaches price single actions, trajectories, or one agent’s cumulative safety cost.This paper moves pricing below untrusted agents and aggregates across principals, making collusion and shared drivers first-order concerns.
- Governance and operational caps: Additive governance caps and conventional operational limits do not address typed residual-loss accounting across dense, correlated fleets.The paper contrasts its approach with human re-authorization, credit-card limits, cloud quotas, and pre-trade checks.
8 Conclusion
The conclusion presents irreversibility budgeting as a missing accounting layer for fleets of tool-using agents. It reports fleet-level overdraw prevention while identifying conservative, dependency-aware pricing as unresolved.
- Conclusion: No runtime tracks the irreversible exposure that fleets of tool-using agents build up together.The proposed budget charges effects before commit and denies an effect when the fleet would overdraw.
- Conclusion: 48× is the maximum reported overdraw by local gates, while the budget held every correctly charged run within the risk limit.The result is stated across fleet-level evaluation runs.
- Conclusion: Conservative, dependency-aware pricing remains open, but irreversibility as a first-class resource provides an accounting location for agent operating systems.The conclusion frames the budget as an accounting abstraction rather than a solved pricing system.