PaperlayerView exact revision on arXiv

Source-linked AI summary

Audit-First Rollback Semantics for Safety-Critical Deployment Pipelines

Xue Qin, Simin Luan, Cong Yang, Zhijun Li

arXiv:2609.00406v1cs.SEcs.DC

TL;DR

The paper addresses how deployment runtimes can keep audit records truthful about live state when provisional transitions crash. It combines audit-first rollback with provisional state machines, evaluates the design against fail-open recovery, and reports perfect measured coherence for audit-first across the tested fail-stop injections while identifying fleet-scale evaluation as future work.

  • Problem

    Crash during a non-atomic provisional deployment can leave live state ahead of the audit chain, undermining truthful retrospective safety reasoning.

  • Method

    The paper combines audit-first rollback, which reverts live state before writing a terminal audit record, with provisional state machines carrying rollback contracts and bounded deadlines.

  • Results

    100% audit/live-state coherence was achieved by audit-first rollback across 600 trials, versus 33% for fail-open, with p95 recovery latency below 500 ms in all 12 cells.

  • Takeaways & Limitations

    The construction provides a mechanically enforceable source-of-truth rule for safety-critical deployment runtimes and supports a safety argument for cross-bridge coordination.

  • Takeaways & Limitations

    The evaluation uses a single-process fail-stop-via-exception model and does not exercise uncatchable OS signals or cross-bridge and multi-host fault injection.

Abstract

from arXiv · show

Distributed deployment runtimes carry a coherence obligation that classical fault-tolerance frameworks do not name directly: the live state a component is configured to run and the audit chain that records how it got there must agree at every terminal configuration. Prior works mainly focus on individual aspects of the deploy-time fault surface (canary controllers, configuration rollback, signed attestations), leaving the cross-cutting question of audit/live coherence under fail-stop crash only loosely specified. Yet a key systems question remains unresolved: how can a deployment runtime guarantee that the audit chain answers truthfully about live state even when a transition crashes mid-flight? We present audit-first rollback semantics, a fault-tolerance mechanism that guarantees audit/live coherence at every committed terminal under fail-stop crashes during transition phases. The mechanism pairs with provisional state machines, pipelines whose "active but not yet promoted" states carry an explicit rollback contract and a bounded deadline. We instantiate both in a runtime deployment system and run a dependability evaluation against a fail-open variant of the same pipeline across twelve fault-injection points spanning three structural failure classes. Across 1,200 injected-failure trials, audit-first rollback achieves 100% audit/live-state coherence (600/600, Wilson 95% CI [0.994,1.000]) with per-cell p95 recovery latency below 500 ms (SLO PASS in 12/12 cells); the fail-open variant preserves coherence on only 33% of trials (200/600, Wilson 95% CI [0.297,0.372]). We further lift the construction to a cross-bridge coordination protocol with a safety argument for fleets of fail-stop bridges, leaving fleet-scale empirical evaluation to follow-on work.

1. Introduction

The paper identifies audit/live-state divergence during crash-prone provisional deployment phases as a safety-critical coherence failure. It proposes audit-first rollback semantics with provisional state machines so the audit chain remains truthful about live state.

  • Problem: A crash after provisional activation can leave the new version serving while the audit chain still records only SHADOW_PASSED.The two views then disagree about both the running version and the successfully promoted version.
  • Motivation: Fail-open recovery trusts live state and permits a partial deployment to keep serving, a posture the paper argues is structurally wrong when audit records support safety-critical reasoning.The audit chain is used by regulators, operators, and downstream systems to reconstruct what actually happened.
  • Contribution: Audit-first rollback instead reverts live state to the audit chain’s last consistent point before writing a terminal record.This makes the audit chain the source of truth for safety-critical agents.
  • Construction: The mechanism pairs provisional state machines with explicit rollback contracts and bounded lifetimes for active-but-not-yet-promoted states.The paper distinguishes committed states, where views agree, from provisional states, where live state has changed without a terminal audit record.
  • Evaluation and scope: The paper evaluates the mechanism under twelve fault-injection points and 1,200 injected-failure trials, and extends it with a multi-bridge safety argument.It reports coherence measurements and proposes cross-bridge properties without claiming novelty for state machines, transactional rollback, or canary deployment itself.
  • Problem: The underlying failure exists because live-state mutation and audit-chain writing are non-atomic, creating a crash window in every provisional region.Without an explicit source-of-truth rule and convergence mechanism, a mid-flight failure can preserve disagreement.

3. Construction

The construction formalizes provisional deployment states with rollback and deadline obligations, then enforces audit-first semantics through a guarded rollback-before-audit pattern. Its terminal records distinguish successful rollback from rollback failure so the audit chain reflects the observed live-state outcome.

  • State model: A pipeline partitions states into committed states, where live state and audit agree, and provisional states, where live state has changed without a matching terminal audit record.Terminal states are committed by definition.
  • State model: Every provisional state carries a rollback closure that restores the most recent committed snapshot and a bounded deadline for reaching a committed state.The closure is installed before live-state mutation and captures the required pre-transition snapshot.
  • Rollback semantics: On an exception inside a provisional region, the guard catches it, runs rollback synchronously or with a bounded timeout, and records a terminal outcome afterward.A successful closure produces ROLLED_BACK; a closure exception produces FAILED.
  • Rollback semantics: Writing the audit record after the rollback attempt makes ROLLED_BACK signify a successful revert while FAILED signals possible divergence requiring human reconciliation.The audit chain therefore records the rollback result rather than predicting it.
  • Source of truth: The semantic race is audit-first: when the audit chain says ROLLED_BACK, the system treats the deployment as though it never happened and forces live state to agree.The label describes source-of-truth priority, not the clock order of the audit write.
  • Implementation pattern: The implementation uses a try/except guard around the post-provisional region, with nested handling that preserves both the original and rollback exceptions in FAILED records.The canonical provisional region is the canary stage, while the provisional promotion itself lies outside the guarded rollback scope.
  • Properties: The construction guarantees termination and live-state coherence at committed states through outgoing terminal transitions and bounded-deadline obligations.Reachable committed terminals include PROMOTED, ROLLED_BACK, and FAILED.

4. Informal State-Machine Specification

The specification models deployment as transitions over live state and an append-only audit chain, with provisional states carrying rollback behavior and bounded deadlines. Its propositions establish termination and truthful audit/live coherence at committed terminals, including crash outcomes.

  • State-machine model: The pipeline is a labelled transition system whose transitions partition into progress, rollback, and failure classes and carry live and audit effects.The model includes finite states, provisional states, committed terminals, and a distinguished start state.
  • Transition semantics: Provisional progress mutates live state without appending an audit record, while terminal progress appends the terminal record without further live mutation.Each provisional transition records its rollback closure as a job-local field.
  • Failure semantics: Failure transitions apply rollback best-effort and append ROLLED_BACK on success or FAILED when the rollback closure raises.The FAILED branch explicitly leaves live state unspecified and records the divergence.
  • Termination: Every run reaches a committed terminal because provisional states have finite bounded deadlines and transitions decrease remaining provisional budget or terminate.This termination claim is stated modulo the bounded-deadline obligation.
  • Coherence and crash safety: At committed terminals, PROMOTED corresponds to post-deploy live state, ROLLED_BACK to pre-deploy live state, and FAILED identifies a state requiring manual reconciliation.The terminal audit record therefore describes the outcome truthfully, including crash handling inside provisional guards.
  • Refinement obligation: Implementation satisfaction is framed as refinement: every implementation trace must have a specification trace with the same audit-chain prefix.The paper gives informal arguments rather than machine-checked refinement proofs.
  • Implementation consequence: The implementation can use a try/except guard around each provisional region because crash-safety is established at the specification level.This correspondence is presented as the source of the reported small implementation footprint.

5. Implementation

The runtime instantiates provisional state machines and audit-first rollback for capability upgrades on an embodied-robot deployment system. Its eight-state pipeline uses an append-only audit chain, rollback closures, and per-capability locking to coordinate upgrades.

  • Runtime instantiation: The implementation drives capability upgrades on a Franka Panda arm through an eight-state deployment pipeline.The runtime targets embodied robots and transitions one capability version at a time.
  • Pipeline states: The pipeline contains provisional PENDING, CANARY_RUNNING, and CANARY_PROMOTED states alongside promotion and failure terminals.CANARY_RUNNING represents a flipped live state while the soak remains ongoing.
  • Audit chain: The audit chain is an append-only episodic-memory ring buffer whose records include event type, intent, capability, version transition, and reason.SQLite write-through and WAL mode provide the chain’s durability mechanism.
  • Rollback closure: The canary rollback closure captures live-state and version context, restores the from-version entry, and emits a rollback audit record when invoked.The closure is constructed when the job is created.
  • Concurrency control: A per-capability asyncio.Lock serializes overlapping upgrades, while concurrent requests for the same capability receive HTTP 409 Conflict.The lock remains held until the terminal audit record is written.

6. Evaluation

The evaluation tests audit-first rollback against a fail-open variant using targeted fail-stop injections across twelve structural failure points. Audit-first preserves audit/live-state coherence in every trial while meeting the recovery SLO in every cell.

  • Evaluation design: The evaluation measures audit/live-state coherence and per-cell p95 recovery latency against explicit rollback-signal SLOs.It uses Wilson 95% confidence intervals for coherence and PASS/FAIL judgments for the SLO.
  • Evaluation design: The comparison isolates audit-first rollback by toggling only the rollback rule; the two pipeline variants are otherwise byte-identical.The fail-open branch omits rollback and records FAILED directly.
  • Evaluation design: 1,200 trials cover twelve injection points across metric-side, rollback-internal, and audit-write-side failure classes.The design runs 50 trials per posture at each injection point, with four variants per structural class.
  • Consistency results: Audit-first achieves 600/600 consistent trials across all twelve injections, while fail-open achieves 200/600.Under fail-open, the four consistent cells are the rollback-internal class-B variants, where FAILED with the to-version live coincidentally matches the specified outcome.
  • Latency results: Audit-first has p50 = 63 ms and p95 = 335 ms across 600 non-promoted trials, with SLO PASS in 12/12 cells.The per-cell p95 range is 57–335 ms against p95 ≤500 ms and p99 ≤1 s objectives.
  • Evaluation scope: The workload is a targeted, specification-driven fault-injection grid rather than a chaos-engineering stress test.Its results establish coverage of the construction’s identified structural paths, not population-level behavior across external systems.

7. Multi-Bridge Coordination

The multi-bridge lift separates local audit truthfulness from fleet-level convergence: each bridge preserves its own audit/live coherence while a coordinator arranges a shared commit-or-abort outcome.

  • System Model: The fleet model coordinates N independently fail-stop bridges through one coordinator, with each bridge retaining its audit chain on persistent storage.The coordinator is not replicated, and the model assumes bounded message delivery or eventual drops.
  • Two-Level Coherence: The protocol requires both per-bridge coherence and cross-bridge convergence to a consistent fleet-level decision.Per-bridge truthfulness must hold during propagation, while fleet convergence is required within finite time after the coordinator decides.
  • Protocol: Each bridge runs its canary locally, records failures before cross-bridge messaging, or records shadow_passed and sends PREPARE while retaining its provisional mutation.The coordinator subsequently determines whether participating bridges write promoted or rolled_back terminals.
  • Safety Argument: Proposition 4 guarantees local audit truthfulness regardless of messages from the coordinator or other bridges.The local guard acts before cross-bridge communication, and later terminal writes preserve the local propositions.
  • Safety Argument: Proposition 5 guarantees eventual fleet convergence when the coordinator recovers and the network eventually delivers in-flight messages.The coordinator persists one decision per deployment and re-broadcasts it after restart; already-terminal bridges ignore the broadcast.
  • Relation to Consensus: The fleet protocol is a single-coordinator go/no-go exchange rather than consensus on a replicated value, because audit truth resides on each bridge.Coordinator failure does not destroy per-bridge audit truth, though fleet liveness depends on eventual coordinator recovery and network healing.

8. Related Work

The paper situates audit-first rollback among coordination, specification, recovery, supervision, log-as-source-of-truth, and fail-stop approaches. It adapts these ideas to deployment pipelines while emphasizing audit/live-state coherence under crashes.

  • Transactional coordination, sagas, and crash-only software: Audit-first rollback differs from classical coordination by making provisional-state exit time-driven rather than coordinator-vote-driven.Two-phase commit commits after all participants vote prepare, whereas the pipeline exits its provisional canary state on a deadline.
  • Lineage-driven fault injection and canary deployment patterns: The evaluation enumerates twelve explicit structural failure points, forming a small coverage proof for the construction’s intended failure surfaces.The methodology draws on lineage-driven fault injection and is adjacent to larger-scale FATE and DESTINI evaluations.
  • Specification-driven design and refinement: The paper adopts specification-driven design at smaller scale, using a state-machine specification without the mechanised refinement step used by related work.Related systems connect distributed-protocol specifications to implementations through TLA+, Coq, or generated C++ code.
  • Recovery-oriented computing: Audit-first rollback specializes recovery-oriented computing by restoring the audit chain’s last consistent point through a rollback closure attached to the provisional region.This makes recovery a first-class deployment-pipeline operation rather than an afterthought.
  • Supervision trees and let-it-crash: The rollback closure is a deployment-pipeline analogue of an OTP supervision strategy: it is attached at job creation and triggered automatically after caught exceptions.Both approaches declare recovery behavior statically and execute it automatically on worker or transition failure.
  • Log-as-source-of-truth: Audit-first rollback treats the audit chain as deployment state’s source of truth, requiring live state to converge to its most recently committed point.This applies log-as-source-of-truth reasoning to deployment pipelines rather than relational storage.
  • Fail-stop semantics: When rollback cannot restore live state, the system records FAILED and halts forward progress, applying fail-stop reasoning instead of permitting silent live-state divergence.The design favors detectable cessation over arbitrary or silently corrupted output.

9. Discussion

The discussion makes the audit chain an explicit prerequisite and explains how audit-first rollback composes with existing canary controllers. The construction operates beneath controller decisions by defining coherent rollback recording and terminal states.

  • Audit-chain prerequisite: Audit-first rollback requires a structured, append-only audit chain because unstructured logs provide no canonical last consistent point.The paper treats this prerequisite as explicit at the deployment-runtime level.
  • Cross-paper composition with operational tools: The construction composes with Argo Rollouts, Spinnaker, and Flagger by inserting the audit-first guard beneath their canary-control layer.Controllers decide when to roll back; the guard determines how rollback is recorded and what constitutes a coherent terminal.
  • Cross-paper composition with operational tools: A retrofit into an existing canary deployment is described as a per-pipeline insertion of the try/except guard.The guard provides the audit-first rollback semantics without replacing the canary controller’s decision layer.

10. Limitations and Threats to Validity

The paper identifies five limitations as explicit threats to validity that bound its claims. The stated evaluation scope is a single Python process with an in-process ASGI client rather than a real subprocess and TCP setup.

  • Known limitations: The paper names five known limitations as distinct threats to validity that bound its claims.These limitations are separated from the discussion points in Section 9.
  • Single-process evaluation scope: The 1,200 fault-injection trials run in a single Python process with an in-process ASGI client.This is the stated single-process evaluation scope.
  • Single-process evaluation scope: The evaluation does not use a real uvicorn subprocess with TCP socket setup, although the measured architectural property is stated to be independent of that setup.The property under measurement is audit-first versus fail-open semantics.

11. Conclusion and Future Works

The construction makes audit-first rollback a practical coherence property for safety-critical deployment runtimes, with strong fail-stop evaluation results and a cross-bridge safety extension. Future work targets empirical fleet evaluation, mechanized verification, journaling, and production-calibrated fault coverage.

  • Conclusion: 1,200 trials report 100% audit/live-state coherence under audit-first fault injection versus 33% under fail-open.Per-cell p95 recovery latency satisfies the p95 ≤500 ms SLO in 12/12 cells.
  • Conclusion: The construction uses provisional state machines, explicit rollback closures, and a small try/except pattern to enforce audit-first rollback.It is described as fitting inside roughly 30 lines per provisional region and as implementable where a structured audit chain already exists.
  • Conclusion: A cross-bridge coordination protocol extends the per-bridge construction with a safety argument under bounded asynchronous execution.The paper leaves fleet-scale empirical evaluation for follow-on work.
  • Future Works: Future work includes empirical multi-bridge evaluation, a write-ahead journal, mechanized refinement checking, and production-derived chaos-grid calibration.The journal would address the uncatchable-signal limitation at the cost of additional disk traffic per provisional flip.

A. Reproducibility

The reproducibility materials provide an open reference artefact, rerunnable chaos-grid and sign-check scripts, and documentation of equivalence to the production experiments. The materials support verification of the paper’s principal coherence and latency hypotheses.

  • Open reference implementation: The standalone artefact includes the audit-first guard, audit-chain interface, rollback closure factory, per-capability lock, and 12-injection chaos-grid harness.It uses Python ≥3.11 and the standard library only, under Apache License 2.0.
  • Open reference implementation: The sign-check verifies four hypotheses covering audit-first consistency, fail-open class-specific consistency, and the audit-first 500 ms recovery budget.A clean run prints PASS for all four hypotheses.
  • Production runtime artefact: The open reference implementation is functionally equivalent to the production runtime for the chaos-grid and latency experiments.The comparison uses the same audit-first rule and 12-cell injection grid against in-process mocks.
  • Production runtime artefact: The full 1,200-trial production sweep completes in approximately 160 s on a single CPU core.A production integration regression test asserts rollback and matching audit recording after a canary crash following provisional promotion.

CRediT authorship contribution statement

The authors’ contributions span conceptualization, methodology, software, formal analysis, investigation, validation, supervision, and writing.

  • CRediT authorship contribution statement: Xue Qin led conceptualization, methodology, software, formal analysis, investigation, and the original draft.Qin also contributed to writing review and editing through the listed authorship roles.
  • CRediT authorship contribution statement: Simin Luan contributed investigation, validation, and writing review and editing.
  • CRediT authorship contribution statement: Cong Yang and Zhijun Li contributed supervision, with Yang also contributing writing review and editing.
Loading 2609.00406v1…