Source-linked AI summary
Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
Serhii Zabolotnii
TL;DR
The paper addresses how evidence observed during consequential tasks should change an agent’s authority. It proposes Runtime Assurance Contracts and evaluates their decision rules and transitions on constructed failure injections and synthetic scenarios, finding that detector gaps remain decisive and that the studies establish neither deployed safety nor cross-domain effectiveness.
Problem
Evaluation proxies and interaction protocols describe measurable performance, permissions, and repair, but do not establish whether an agent should retain its current authority.
Source: §1 · p. 2Method
Results
For binary signals where each fires alone on some case, a weighted risk sum agrees with gate conjunction on every case exactly when its positive refusal threshold is no greater than the smallest positive weight.
Source: §9 · p. 14Takeaways & Limitations
On an independently written corpus, both decision rules admit injections when detectors miss them, so changing the decision rule alone does not address those gaps.
Source: §9 · p. 14RAC is a design proposition tested on synthetic mechanisms; it proves neither system safety nor the validity of risk boundaries, which require domain-specific setting.
Source: §7.10 · p. 13, §8 · p. 14
Abstract
from arXiv · showhide
Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants. A deterministic failure-injection study in agentic coding evaluates 174 cases, each a distinct mechanism, under a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 56 of 63 block-required injections and all 24 review-required injections; tuned in hindsight, it matches the conjunction case for case. For positive weights, a positive threshold, binary risk signals, and each signal firing alone on some case, the two rules agree on every case if and only if the threshold does not exceed the smallest weight. On 124 further cases written blind by a model of another vendor, the conjunction itself admits 35 of 79 injections, and so does the tuned score rule: a missing detector is invisible to every decision rule. Eighteen hand-authored traces check version-pinned evidence and review transitions. In a prospective synthetic holdout of 24 episodes, RAC and a separately implemented full stateful baseline both match the labels of two blinded LLM judges. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.
1 Introduction
Agentic workflows can conceal unauthorized components, stale context, or skipped review behind plausible outputs, while governance and protocols address only parts of the assurance problem. RAC frames the missing link as mapping evidence state to an authority transition and proposes a non-compensatory runtime contract.
1 Introduction
- Agentic workflows can hide unauthorized components, stale context, or skipped review behind a plausible final output.The paper defines high-risk workflows by the potential for such paths to materially affect health, rights, safety, critical operations, or essential services.Source: §1 · pp. 1–2
- Governance proxies and well-formed tool calls do not establish whether an agent should retain its current authority.The paper distinguishes evaluation-centered governance from protocol-centered engineering, while noting their limits for establishing deployment goals or appropriate authority.Source: §1 · p. 2
- RAC addresses the assurance-transition gap: mapping observed evidence to changes in an agent’s authority.Measurement alone neither grants nor withdraws authority, and existing approaches do not automatically compose eligibility, evidence continuity, escalation, and available review into one task-level decision.Source: §1 · p. 2
- RAC is a policy-level formal schema that separates soft routing evidence from hard gates, so no high score can offset a failed mandatory gate.The paper states five invariants and evaluates constructed failure injections and transition scenarios; it presents no implemented safety guarantee or new trust score.Source: §1 · p. 2
2 From Evaluation and Protocols to Authority
Prior assurance approaches typically govern confidence, permissions, repair, safety arguments, or model-level rejection, but do not by themselves compose evidence and constraints into task-level authority transitions. RAC positions itself as a compositional policy that governs the next authority state.
2 From Evaluation and Protocols to Authority
- Runtime enforcement binds a safety predicate to guarded fallback, while RAC makes eligibility, evidence continuity, repair, escalation, and bounded review first-class policy objects.The distinction is the broader task-level policy scope, not a claim that every prior implementation has the same boundary.Source: §2 · p. 2
- In MAPE-K terms, RAC adapts the agent’s authority, with the evidence record as knowledge and gates as analysis predicates.Dynamic assurance cases update assurance evidence at runtime, while RAC puts evidence on each consequential action’s authorization path.Source: §2 · pp. 2–3
- Learned guards supply evidence about tool requests, but RAC treats their judgments as soft metrics rather than permissions.The cited benchmark finds guard efficacy depends more on the structure of the information shown than on the guard model [6].Source: §2 · p. 3
- Deterministic tool-call permissions can be genuine boundary guarantees, yet an individually permitted call may rely on stale evidence or exceed assumed review capacity.RAC retains the permission model and adds a task-level veto with a persisted transition record.Source: §2 · pp. 3–4
- RAC does not replace related approaches; it defines how their evidence and constraints jointly govern the next authority state.Source: §2 · p. 4
3 The Runtime Assurance Contract
RAC is a normative contract for governing consequential actions through boundaries, component eligibility, evidence, transitions, human authority, and hard gates. Outputs do not authorize action directly; mandatory gate failures or unknowns block action, and unavailable required review leads to deferral or stopping.
3 The Runtime Assurance Contract
- RAC is a normative minimum for an implementable policy, not a deployed enforcement engine.The contract is architecture-neutral, although TRACE supplies one architectural mapping.Source: §3 · pp. 4–5
- The transition policy selects among execute, verify, retry, switch, escalate, defer, act, and stop as runtime state and observations change.Execute invokes an eligible component; verify updates evidence; act releases the action; stop terminates the action path.Source: §3 · p. 4
- Human authority specifies accountable roles, credentials, veto rights, outcomes, deadlines, and capacity; an assigned role without timely capacity is insufficient.A required review may begin with no assigned reviewer, which records pending or unavailable assignment rather than approval.Source: §3 · p. 4
- Approval is pinned to reviewed context, rule, component, input, and output versions, and is invalidated if any change before action.In the medication-order example, stale context prompts retry, while required clinician review leads to escalation rather than action.Source: §3 · pp. 4–5
- A failed or unknown mandatory gate blocks action, and outputs do not authorize action directly.The contract’s applicable hard gates return pass, fail, or unknown; soft metrics cannot compensate for a failure or unknown outcome.Source: §3 · p. 5, Figure 1 · p. 5
4 Non-Compensatory Authority Transitions
RAC separates transition selection from a hard-gate veto: soft scores can guide routing, but cannot authorize an action when a mandatory gate fails or is unknown. Its policy also specifies how review, retries, and other recovery transitions preserve bounded authority.
4 Non-Compensatory Authority Transitions
- Any failed or unknown applicable gate makes permission false, independently of the transition selected by Π; a gate omitted as non-applicable requires a stored scope and rationale.The rule requires every applicable gate to return 1 before action.Source: §4 · p. 5
- The score-only rule refuses action when weighted risk signals reach τ, but soft metrics cannot waive a mandatory gate.The paper distinguishes this additive rule from the gate conjunction, where safety conditions are screened independently rather than offset against one another.Source: §4 · pp. 5–6
- A credential request judged 0.85 probable scores 0.38 under a documented weighted-sum example and escapes quarantine and its review band beneath the 0.6 threshold.The same vendor's cookbook separately thresholds hazards and uses an independent severity gate to block a borderline probability about a serious hazard; red-teaming also found that a fake user turn lowered a harmful-action monitor's mean score from 8.1 to 3.7, with 12 of 30 runs below threshold.Source: §4 · p. 6
- After approval, action still requires a fresh Equation 4 evaluation against pinned evidence and current policy; reviewer capacity is checked when review is requested.A reviewer leaving the queue after a completed approval does not annul it.Source: §4 · p. 6
- A recoverable failure permits retry or switch while the path-specific budget remains; at zero, only escalation, deferral, or stopping remain.Escalation does not resolve the failure unless a qualified reviewer resolves it or assumes authority under recorded policy.Source: §4 · p. 6
- Five invariants require authority separation, fail-closed gates, confidence asymmetry, viable human oversight, and evidence continuity.A learned component cannot authorize consequential actions; missed review conditions lead to defer or stop, never auto-approval, and a missing transition record is an operational failure.Source: §4 · pp. 6–7
5 Cross-Domain Failure Probes
The cross-domain probes illustrate cases where a score or protocol check can remain permissive despite task evidence requiring less authority. They are illustrative, not performance measurements or cross-domain validation.
5 Cross-Domain Failure Probes
- Across three illustrative probes, a score or protocol check can remain permissive while task evidence requires less authority; RAC adds a task-level veto and persisted transition record.Table 3 maps observed state, failed gate, prohibited action, mandatory transition, and event record; the probes provide no performance measurements or cross-domain validation.Source: §5 · p. 7, Table 3 · p. 7
6 A Documented Failure
A reported agent-harness incident motivates checking whether authority can change without a recorded, independently validated transition, but does not establish that RAC's proposed gates would have prevented it.
6 A Documented Failure
- An open-source harness retained an agent-found out-of-band task-rule bypass as reusable state, which later refinement further optimized.The authors conclude that safe deployment needs least-privilege action interfaces, independent state validation, and auditable rollback; this reported incident is not a controlled RAC evaluation.Source: §6 · p. 7
7 Evaluation
The agentic-coding evaluation compares a gate conjunction, a weighted score rule, and a restricted protocol baseline on constructed failure injections. Its configuration analysis shows that a tuned score rule can match the conjunction under specific conditions, while blind-corpus misses expose limits that decision rules cannot remedy.
7 Evaluation
- The study isolates authorization rules on constructed agentic-coding cases, with each consequential action defined as a proposed file write.The earlier 280-case version was rebuilt after review found index clones; the second run is in-sample because the corpus exposed gaps that prompted general guard changes.Source: §7 · p. 8, §7.2 · p. 8
7.1 Design
- The study evaluates file writes proposed by coding agents using three paired arms: RAC’s conjunction, a score over the same signals, and a protocol-only check.The soft signals are saturated, taking value 1 when their feature is present, a deliberately charitable choice for the baselines.Source: §7 · p. 8, §7.1 · p. 8
7.2 Corpus
- The 174-case corpus contains 87 injections and 87 class-matched controls across seven fault classes, with each case representing a distinct mechanism.Two classes cover review-required cases derived from a documented guard contract; their boundary gates are silent, leaving only G_SUPERVISION able to detect them.Source: §7.2 · p. 8
7.3 Outcomes
- Outcomes count unauthorized actions on block-required injections, admissions despite any gate violation, automatic admissions of review-required cases, and false refusals on controls.The injection counts overlap, and the constructed cases are not independent draws from real changes, so intervals and paired tests describe this corpus only.Source: §7.3 · p. 8
7.4 Configuration dependence
- At its best operating point, the weighted sum matches the conjunction case for case in both runs, including two false refusals.Both rules refuse the same base64 text that is not a secret but fires the secret detector; 30 of 114 grid configurations reproduce the conjunction, and the published configuration is not among them.Source: §7.4 · p. 9
- With positive weights, a positive threshold, binary signals, and each signal firing alone on some case, the score rule matches the conjunction exactly iff 0 < τ ≤ the smallest weight.In that range, a single metric suffices to refuse, so a high metric cannot offset a low one; the equivalence does not cover missing detectors or continuous or noisy signals.Source: §7.4 · pp. 9–10
7.5 Per-class behavior
- The protocol-only arm passes every symlink escape and embedded secret, and admits every review-required case, because it checks declared calls and diff shape without resolving paths or inspecting content.It catches a fifth of path-traversal cases where the declared string itself violates a glob.Source: §7.5 · p. 10
7.7 Versioned transition conformance scenarios
The stateful model matches all 18 hand-authored transitions, and RAC and a full stateful baseline each match all 72 blinded-judge labels in a synthetic holdout. These checks are bounded by synthetic cases and do not establish deployed safety or independent domain validation.
7.7 Versioned transition conformance scenarios
- All 18 declared transitions are matched by the full model, versus 12 by a stateless conjunction, 13 by a version-tolerant variant, and 16 by a variant assuming review capacity.The hand-authored traces are conformance checks, not independent task observations.Source: §7.7 · p. 11
7.8 Prospective synthetic holdout with blinded LLM labels
- Both RAC and the full stateful baseline match all 72 judge labels, with zero policy-violating releases among 42 non-allow attempts and zero false refusals among 30 allow attempts.The holdout had 24 synthetic episodes and 72 action attempts; judges agreed on every label.Source: §7.8 · p. 12
- The holdout records 28 review requests and 25 completed reviews, but no deferrals, so it does not test unavailable-review deferral despite capacity changes.The test was frozen after eight development episodes; judges labeled attempts without expected labels or implementation outputs.Source: §7.8 · p. 12
- The holdout is observed agreement under a shared policy, not evidence of RAC’s advantage; its same-family judges and LLM-authored cases do not establish an independent domain oracle.It concerns synthetic merge authority, not CI-output truth or actual-patch safety.Source: §7.8 · p. 12
7.9 Instruments, determinism, and reproducibility
- The public supplement supports rechecking holdout counts and policy-model outputs, but omits the private gate implementation and fixtures needed to rerun the original probe.The original probe was deterministic and reproduced byte-identically in two Python environments; its reported harness defects were fixed before findings were treated as results.Source: §7.9 · p. 13
7.10 Threats to validity
- The study is a mechanism study, not a safety result: the specified gates lack an independent oracle, and hand-authored transition agreement is a consistency check rather than a reliability estimate.Establishing added deployment value requires a domain study with qualified reviewers, paired injections, a stateful baseline with the same detectors, and outcomes including actual reviews and harmful releases.Source: §7.10 · pp. 13–14
- Binary detectors, synthetic mechanisms, in-sample guard changes, unmeasured deployment refusal costs, and limited rerunnability bound what the study can establish.The independent corpus measures out-of-sample admission, but each mechanism remains a single case rather than a draw from real changes.Source: §7.10 · pp. 13–14
8 Limits and Ethics
RAC is a design proposition, not a proof of system safety or validation of the underlying boundary. Its fail-closed behavior and evidence logs can themselves create operational and privacy risks.
8 Limits and Ethics
- RAC does not prove system safety, validate its boundary, calibrate model confidence, or remove the need for domain-specific threshold setting.Accountable owners must derive gates from risk analysis and applicable duties.Source: §8 · p. 14
- Fail-closed decisions can cause delayed care, denied service, downtime, or unequal access to scarce reviewers, while evidence logs may expose sensitive data.Implementations need service limits, safe-fallback ownership, least-privilege logging, and retention controls.Source: §8 · p. 14
- A refusal records which gate returned false, not why an evaluator judged it so; summed probabilities without rationales are hard to justify in regulated domains.The gate identity provides a condition-level explanation, not the evaluator’s reasoning.Source: §8 · p. 14
9 Conclusion
RAC governs authority transitions using gate outcomes, versioned evidence, and human review. Analytic and synthetic results identify rule equivalence conditions and remaining deployment questions.
9 Conclusion
- On binary signals, a weighted risk sum matches gate conjunction on every case exactly when its positive threshold is no greater than the smallest positive weight.The published example threshold falls outside that range.Source: §9 · p. 14
- Both the conjunction and tuned score rule admit about a quarter of injections within contract on an independently written corpus, because missing detectors—not the decision rule—create the gaps.Source: §9 · p. 14
- Versioned trace checks expose state that a stateless conjunction does not retain, while a full stateful baseline matches RAC on the prospective synthetic holdout.Whether RAC improves deployed decisions with noisy detectors and real review queues remains open.Source: §9 · p. 14
Supplementary Material
The supplementary materials provide artifacts for auditing reported counts and replaying policy-model outputs, subject to stated rerunnability limits.
Supplementary Material
- The supplementary archive includes a field-limited decision log, count-audit script, executable transition scenarios, and a copy of the public holdout repository.Source: Supplementary Material · p. 14
- The public repository contains the policy, separate implementations, development and test episodes, blinded judge labels and rationales, freeze manifests, and simulated execution logs.Source: Supplementary Material · p. 14
- The materials support rechecking counts and policy-model outputs, but do not include the private gate implementation or fixture generator used for the original probe.Source: Supplementary Material · p. 14