PaperlayerView exact revision on arXiv

Source-linked AI summary

Audit-First Rollback Semantics for Safety-Critical Deployment Pipelines

Xue Qin, Simin Luan, Cong Yang, Zhijun Li

arXiv:2609.00406v1cs.SEcs.DC

TL;DR

Deployment runtimes lack a clear specification for keeping audit records truthful about live state when a transition crashes. This paper pairs audit-first rollback with provisional state machines and reports 66.7 percentage points more consistent trials than fail-open in its evaluation.

  • Problem

    How to ensure audit/live coherence at committed terminals under fail-stop crashes remains loosely specified, despite its importance when safety-critical systems rely on audit records.

    Source: §1 · pp. 1–2

  • Method

    The paper defines audit-first rollback semantics and pairs them with provisional state machines that use explicit rollback closures when local failures occur.

    Source: §7.3 · p. 10, §11 · p. 14

  • Results

    400 of 600 additional trials were consistent under audit-first versus fail-open, a 66.7% absolute consistency difference.

    Source: §6.3 · p. 8

  • Takeaways & Limitations

    The paper concludes that safety-critical deployment runtimes should treat the audit chain as the source of truth when it disagrees with live state.

    Source: §11 · p. 14

    The fault-injection evaluation ran in a single Python process rather than with a live deployment subprocess and TCP setup.

    Source: §10 · p. 14

Abstract

from arXiv · show

Distributed deployment runtimes carry a coherence obligation that classical fault-tolerance frameworks do not name directly: the live state a component is configured to run and the audit chain that records how it got there must agree at every terminal configuration. Prior works mainly focus on individual aspects of the deploy-time fault surface (canary controllers, configuration rollback, signed attestations), leaving the cross-cutting question of audit/live coherence under fail-stop crash only loosely specified. Yet a key systems question remains unresolved: how can a deployment runtime guarantee that the audit chain answers truthfully about live state even when a transition crashes mid-flight? We present audit-first rollback semantics, a fault-tolerance mechanism that guarantees audit/live coherence at every committed terminal under fail-stop crashes during transition phases. The mechanism pairs with provisional state machines, pipelines whose "active but not yet promoted" states carry an explicit rollback contract and a bounded deadline. We instantiate both in a runtime deployment system and run a dependability evaluation against a fail-open variant of the same pipeline across twelve fault-injection points spanning three structural failure classes. Across 1,200 injected-failure trials, audit-first rollback achieves 100% audit/live-state coherence (600/600, Wilson 95% CI [0.994,1.000]) with per-cell p95 recovery latency below 500 ms (SLO PASS in 12/12 cells); the fail-open variant preserves coherence on only 33% of trials (200/600, Wilson 95% CI [0.297,0.372]). We further lift the construction to a cross-bridge coordination protocol with a safety argument for fleets of fail-stop bridges, leaving fleet-scale empirical evaluation to follow-on work.

1. Introduction · 2. Motivation

A provisional deployment can leave live state and the audit chain describing different outcomes when a fail-stop crash occurs mid-transition. The paper argues that safety-critical runtimes should treat the audit chain as authoritative and restore live state to its last consistent point before recording a terminal outcome.

2.2. Why Cloud-Default Fail-Open Is Wrong for Embodied Agents

  • Cloud runtimes favor fail-open because continued partial service can be preferable to unavailability, but the paper argues this posture is structurally wrong when safety-critical systems depend on truthful audit records.The framing identifies audit-chain integrity and live-state availability as the properties at stake under an unhandled fail-stop crash.Source: §1 · p. 1, §2.2 · p. 3

1. Introduction

  • Audit-first rollback makes the audit chain authoritative: after a failure in a provisional transition, live state is reverted to its last consistent point before a terminal record is written.The target is audit/live coherence at every committed terminal under fail-stop crashes.Source: §1 · p. 2
  • Provisional state machines separate committed from provisional states, with each provisional state carrying a rollback closure and bounded lifetime.Unlike two-phase commit, their second phase is time-driven, such as a canary window or bounded soak, rather than coordination-driven by participant votes.Source: §1 · p. 2
  • The dependability evaluation covers twelve fault-injection points under two postures across 1,200 injected-failure trials, measuring coherence with Wilson 95% CIs and per-cell p95 recovery latency against an SLO.The supplied passage describes the evaluation design and metrics, not its outcomes.Source: §1 · p. 2
  • The construction is lifted to a multi-bridge protocol under bounded asynchrony, with safety propositions for per-bridge truthfulness and eventual fleet convergence.The introduction says these per-bridge invariants compose without strengthening the per-bridge fault model.Source: §1 · p. 2

2.1. Two Failure Modes That Diverge Live State from Audit

  • A metrics-provider TypeError after provisional promotion can leave the audit chain reporting FAILED while the active-version map still serves the new version.A separate failure occurs when an external mutation makes the rollback closure raise, leaving live state on the new version despite a ROLLED_BACK audit record.Source: §2.1 · p. 2
  • Both failure modes arise because live-state mutation and audit writing are not atomic, leaving a crash window for disagreement unless a rule and mechanism enforce convergence.The motivating cases arise when a pipeline flips live state before reaching a terminal audit record.Source: §2.1 · pp. 2–3

2.2. Why Cloud-Default Fail-Open Is Wrong for Embodied Agents

  • For embodied robots, divergence can enable unsafe motions, poison episodic-memory records, and break operators’ audit-based understanding; on failure, live state should be reverted to match the audit chain and that convergence audited.The passage also gives a clinical-pipeline example in which the wrong model version may be used while the audit chain reports the new one as live.Source: §1 · p. 2, §2.2 · p. 3

3. Construction

The construction bounds provisional states with rollback contracts and routes exceptions through rollback before writing a terminal audit record. Its committed-state guarantees distinguish successful promotion or rollback from failure requiring operator reconciliation.

3.1. Provisional State Contract

  • Each provisional state has a rollback closure capturing the committed snapshot, a bounded deadline to commit or roll back, and no path to end the pipeline provisionally.The closure must be installed on the job before live-state mutation; every reachable provisional state has an outgoing edge to a committed state.Source: §3.1 · p. 3

3.2. Audit-First Rollback Semantics

  • On an exception inside a provisional state, the runtime attempts rollback before writing the audit record, marking ROLLED_BACK on success or FAILED if rollback raises.The audit chain is written last with knowledge of the rollback outcome, so its terminal record reflects what happened to live state.Source: §3.2 · p. 3

3.3. Implementation Pattern

  • A try/except guard wraps the canary provisional region: on an exception it calls rollback before recording a terminal audit row, distinguishing ROLLED_BACK from FAILED.The canary stage is the implementation’s canonical provisional region.Source: §3.3 · p. 4

3.4. State-Machine Guarantees

  • Every provisional state reaches PROMOTED, ROLLED_BACK, or FAILED by its deadline; PROMOTED and ROLLED_BACK preserve audit/live coherence, while FAILED signals reconciliation.A bridge crash between the rollback call and store update is outside these guarantees.Source: §3.4 · p. 4

4. Informal State-Machine Specification

The paper gives an informal state-machine account of audit-first rollback, adapted from Schneider to single-process audit/live coherence. It states termination, terminal coherence, and guarded crash-safety properties, while leaving mechanized refinement checking out of scope.

4. Informal State-Machine Specification

  • The specification is a deliberately small, informal adaptation of Schneider ’s state-machine approach from N-replica replication to single-process audit/live coherence.Its propositions have informal justifications; mechanized refinement checking in TLA+ or P is identified as future work outside the paper’s scope.Source: §4 · p. 4

4.1. Specification

  • Provisional transitions mutate live state and record a rollback closure without appending audit; terminal promotion appends its audit record, while rollback or failure records ROLLED_BACK or FAILED.On failure, the rollback closure is attempted before the audit write; if it raises, live state is unspecified and the audit records FAILED.Source: §4.1 · p. 5
  • Every run reaches a committed terminal, subject to each provisional state’s finite deadline and the bounded-deadline obligation.The stated argument relies on a finite provisional-state set and deadlines that force outgoing transitions.Source: §4.1 · p. 5
  • At committed terminals, the last audit record identifies promoted live state, successful rollback to pre-deploy state, or failure requiring manual reconciliation.For exceptions during provisional-state transitions, the next configuration is either ROLLED_BACK with pre-deploy live state or FAILED with unspecified live state; in either case, the terminal audit record truthfully describes the outcome.Source: §4.1 · p. 5

4.2. Refinement Obligation

  • Implementation refinement requires each trace to match a specification trace with the same audit-chain prefix, but the paper does not prove this mechanically.Because crash-safety is established at the specification level, the implementation can use a try/except guard around each provisional region instead of maintaining a runtime invariant.Source: §4.2 · p. 6

5. Implementation

The runtime deployment system for embodied robots implements provisional state machines and audit-first rollback for single-capability upgrades. Its canary stage uses a bounded provisional window, with rollback closures and an append-only audit chain supporting terminal-state recording.

Implementation

  • The runtime deployment system for embodied robots upgrades a single capability on a Franka Panda arm through an eight-state pipeline.Source: §5 · p. 6
  • PENDING, CANARY_RUNNING, and CANARY_PROMOTED are provisional states; CANARY_RUNNING persists through a bounded soak window where most failure modes manifest.Source: §5 · p. 6
  • The audit chain is an append-only episodic-memory ring buffer with structured evolution records and SQLite write-through in WAL mode.Source: §5 · p. 6
  • At job creation, the canary rollback closure snapshots the prior version; when called, it restores that version and emits a rollback audit record.Source: §5 · p. 6
  • A per-capability asyncio lock prevents concurrent upgrades from racing; a second request receives HTTP 409 until the terminal audit record is written.Source: §5 · p. 6

6. Evaluation

The evaluation tests whether audit-first rollback maintains audit/live-state coherence under injected fail-stop exceptions while meeting recovery SLOs. It uses a targeted 12-point fault grid and does not claim to be a throughput benchmark or a stress test.

6.1. Setup

  • The comparator toggles only the rollback call; 50 trials per posture and injection point measured terminal status, live state, and whether the latest terminal audit record matched that state.The 12-point grid comprised 1,200 trials total.Source: §6.1 · p. 7, §6.2 · p. 7

6.2. Twelve Crash Injection Points

  • The targeted 12-point grid covers three structural failure classes: metric-side raises, rollback-internal raises, and audit-write-side raises, with four variants per class.Variants exercise different timings, exception types, and a concurrent rollback path.Source: §6.2 · p. 7

6.3. Results

  • 600/600 audit-first trials were coherent (Wilson 95% CI [0.994, 1.000]), versus 200/600 fail-open trials (Wilson 95% CI [0.297, 0.372]).Audit-first achieved SLO PASS in 12/12 cells, with per-cell p95 from 57 ms to 335 ms; aggregate p95 was 335 ms for audit-first and 325 ms for fail-open. Audit-first rolled back Classes A and C, while Class B ended FAILED with the to-version live and an audit reason recording the original error and rollback exception; fail-open silently diverged in A and C but coincidentally matched in B.Source: Table 1 · p. 8, §6.3 · p. 8, Figure 2 · p. 9

6.4. Comparison Framing

  • The grid follows lineage-driven fault injection, not randomized chaos testing, and its outcomes jointly cover every structural path through the specified guard.Fail-open divergence in Classes A and C follows directly from omitting rollback; the empirical tests instead show audit-first convergence across trials, specified Class B behavior, and SLO compliance in every cell.Source: §6.4 · pp. 8–9

6.5. What This Evaluation Does Not Exercise

  • The evaluation covers fail-stop exceptions caught within one process, but not uncatchable OS signals, cross-bridge or multi-host fault injection, or a live uvicorn subprocess and TCP setup.Those cases are left to a write-ahead journal layer or follow-on validation, including fleet-scale empirical evaluation.Source: §6.5 · p. 9

7. Multi-Bridge Coordination

The section lifts audit-first rollback from individual bridges to fleet coordination, presenting a protocol and safety argument. The multi-bridge protocol is not empirically measured here.

7.1. Fault Model for Multi-Bridge Operation

  • The model assumes independently fail-stop bridges with persistent, replayable local audit chains and a fail-stop coordinator with its own persistent decision log.Unlike Lamport or Ongaro and Ousterhout, it requires neither a stable bridge majority nor a replicated coordinator because the coordinator gates a go/no-go signal rather than consensus on a value.Source: §7.1 · p. 10

7.2. Two-Level Coherence

  • Fleet coherence requires each bridge’s audit chain to truthfully reflect its live state and all chains eventually to record a consistent fleet decision or a failed terminal.The construction requires eventual agreement, not strong linearisation; local truthfulness holds while fleet-level agreement propagates.Source: §7.2 · p. 10

7.3. Protocol Sketch

  • Local failures write a failed terminal before cross-bridge messages; successful canaries hold provisional mutations, and the coordinator’s COMMIT or ABORT triggers promotion or rollback.The coordinator treats a missing vote as ABORT-VOTE and records its decision persistently before broadcasting it.Source: §7.3 · p. 10

7.4. Safety Argument: Multi-Bridge Lifting

  • Per-bridge truthfulness holds despite coordinator or peer messages, while fleet convergence requires coordinator recovery and eventual network delivery and yields terminals consistent with the fleet decision.Proposition 4 is local; Proposition 5 requires that the coordinator not crash permanently and that the network eventually deliver in-flight messages within Δmax.Source: §7.4 · pp. 10–11

7.5. Why Not Paxos or Raft

  • Unlike Paxos and Raft, the coordinator supplies a single go/no-go bit while each bridge’s append-only audit chain remains the canonical record of local truth.Cross-bridge atomic broadcast, Byzantine-fault-tolerant audit aggregation, and adversarial operator-side reconciliation are out of scope.Source: §7.5 · pp. 11–12

8. Related Work

Related work positions audit-first rollback among coordination, recovery, deployment, and specification techniques. Its distinguishing concern is keeping the audit chain truthful about live state, including when rollback fails.

Transactional coordination, sagas, and crash-only software

  • Audit-first rollback makes the audit chain, rather than live filesystem state, the recovery target; unlike crash-only recovery’s single authoritative store, it assumes two stores and chooses between them on failure.Its second phase is time-driven: the pipeline exits provisional canary state when the soak window expires; it uses one rollback closure rather than threading saga compensations [18]. It may yield FAILED rather than clean ROLLED_BACK and require manual reconciliation.Source: §8 · p. 12

Lineage-driven fault injection and canary deployment patterns

  • Prior deployment patterns specify canary rollout and progression gates, while audit-first rollback supplies the per-bridge audit-versus-live tiebreaker they leave unspecified.Its twelve-point failure-injection grid draws methodological inspiration from Alvaro et al. and is adjacent to larger-scale FATE and DESTINI; Meta’s Twine provides a fleet-scale deployment substrate without prescribing the tiebreaker.Source: §8 · p. 12

Specification-driven design and refinement

  • Audit-first rollback follows specification-driven design [9], matching its specified failure branch to an implementation try/except without mechanised refinement.Its scope is one provisional region per pipeline; the proposed multi-bridge extension would be a natural target for mechanised refinement checking.Source: §8 · pp. 12–13

Recovery-oriented computing

  • Audit-first rollback specializes recovery-oriented computing by restoring the audit chain’s last consistent point through the provisional region’s rollback closure.It complements rollback-free recomputation by Loreti et al. and differs from Xu et al.’s focus on recovery throughput; refinement simulations link the specification and implementation.Source: §8 · p. 13

Supervision trees and let-it-crash

  • Audit-first rollback resembles Erlang OTP supervision through statically attached, automatic recovery, but restores live state from an audit-chain snapshot rather than a supervisor-held default.Source: §8 · p. 13

Log-as-source-of-truth

  • Like Aurora’s redo-log approach, audit-first rollback treats the log as ground truth and makes derived state converge, but restores an in-memory active-version map from a captured snapshot.Source: §8 · p. 13

9. Discussion

Audit-first rollback requires a structured, append-only audit chain and guarantees state–audit coherence, not behavioral correctness. It composes beneath canary controllers, leaving rollback decisions to them while governing rollback recording and coherent terminal states.

Discussion

  • The construction requires a structured, append-only audit chain to identify a canonical rollback point, and it preserves state–audit coherence without making the new version behaviorally correct.Behavioral safety remains the responsibility of upstream validators and canary success criteria; the audit-chain prerequisite is also implicit in log-as-source-of-truth systems such as Aurora-style storage.Source: §9 · p. 13
  • Audit-first rollback composes beneath canary controllers: they decide when to roll back, while its guard determines how rollback is recorded and what counts as a coherent terminal.Retrofitting an existing canary deployment requires per-pipeline insertion of the §3.3 try/except guard, with no change to the controller’s external interface.Source: §9 · p. 14

10. Limitations and Threats to Validity

The paper identifies five limitations that bound its claims. They concern evaluation scope, uncatchable crashes, the comparison baseline, synthetic fault selection, and the unimplemented multi-bridge protocol.

Single-process evaluation scope

  • The 1,200 fault-injection trials ran in one Python process with an in-process ASGI client, not a uvicorn subprocess over TCP.A live-process replication is a follow-on validation, particularly for the latency tails reported in Figure 2.Source: §10 · p. 14

Uncatchable signals are out of scope

  • OS-level kill -9 signals are outside the fail-stop-via-exception model and can leave a lost in-memory revert with an open CANARY_RUNNING audit record.Recovery after restart is manual; a write-ahead journal and replay could close this gap, at the cost of disk traffic per provisional flip.Source: §10 · p. 14

Baseline is restricted to a single fail-open variant

  • The comparison measures only a byte-identical fail-open variant, not industry-standard canary controllers such as Argo Rollouts, Spinnaker, or Flagger.This isolates the audit-first rule from implementation differences; comparisons with external controllers could broaden applicability but would mix in their rollout heuristics.Source: §10 · p. 14

The chaos grid is synthetic, not production-derived

  • The 12 injection points provide designed-of-experiments coverage of specified failure surfaces, following Alvaro et al., rather than replaying production incidents.The grid offers construction-surface coverage but does not reflect production-traffic frequencies; a production-derived grid could weight points by observed frequency.Source: §10 · p. 14

Multi-bridge protocol is designed but not implemented

  • The cross-bridge protocol is designed and supported by a safety argument, but has not been implemented or evaluated in a fleet-scale N-bridge fault-injection sweep.An empirical multi-bridge evaluation is a next step, alongside mechanised refinement checking against the per-bridge state machine.Source: §10 · p. 14

11. Conclusion and Future Works

The paper presents audit-first rollback as a mechanical way to keep the audit chain authoritative when live state and audit records disagree, and reports its dependability results. It also extends the construction to cross-bridge coordination and identifies four future research directions.

Conclusion

  • Across 1,200 trials, audit-first achieved 100% audit/live-state coherence versus 33% for fail-open, with the p95 ≤500 ms recovery-latency SLO met in 12/12 cells.The evaluation used fail-stop fault injection.Source: §11 · p. 14
  • Provisional state machines with explicit rollback closures and a small try/except pattern enforce audit-first semantics in roughly 30 lines per region.The construction fits pipelines that already maintain a structured audit chain; it adds small implementation overhead and signals FAILED if rollback cannot complete.Source: §11 · p. 14
  • The multi-bridge lifting extends the per-bridge construction to a cross-bridge coordination protocol with a safety argument under bounded async.Source: §11 · p. 14

Future Works

  • Future work proposes empirical multi-bridge evaluation, a write-ahead journal, mechanised refinement checking, and incident-frequency-weighted chaos testing.These would address the design-versus-implementation gap, uncatchable-signal limitation, prose-only proof arguments, and lack of empirical calibration, respectively.Source: §11 · p. 15

A. Reproducibility

The paper provides an open, containerisable reference artefact for reproducing its chaos-grid experiments, while distinguishing reproducible consistency counts from production-specific latency measurements. A sign-check script validates four stated hypotheses, and the production-runtime sweep takes about 160 seconds on one CPU core.

A. Reproducibility

  • The standalone reference artefact includes the audit-first guard, audit-chain interface, rollback closure factory, per-capability lock, 12-injection harness, and 1,200-trial traces and aggregates.It is released under Apache License 2.0, requires Python ≥3.11 with standard library only, and includes a Dockerfile for containerised replication.Source: A. Reproducibility · p. 15
  • The sign-check script verifies four hypotheses: audit-first consistency in every cell, fail-open consistency in Class-B cells, fail-open inconsistency in Class-A and Class-C cells, and audit-first wall-clock p95 within 500 ms.A clean run prints PASS for all four hypotheses.Source: A. Reproducibility · p. 15
  • The open reference reproduces the consistency counts: audit-first 600/600 and fail-open 200/600; its latency rerun matches the qualitative trimodal shape, not production absolute values.The 1,200 trials were collected on a closed-source production runtime; the reference uses in-process mocks, omitting production subprocess, transport, job-store, supervision, and marketplace components described as out of scope for the §6 claims.Source: A. Reproducibility · p. 15
  • The production-runtime 1,200-trial sweep completes in approximately 160 s on a single CPU core, and an integration regression test checks rollback and matching audit after a metric-side canary crash.The reference grid runs in comparable time.Source: A. Reproducibility · p. 15

CRediT authorship contribution statement

Xue Qin is credited with conceptualization, methodology, software, formal analysis, investigation, and the original draft; the other authors contributed investigation, validation, editing, and/or supervision.

CRediT authorship contribution statement

Loading 2609.00406v1…