Source-linked AI summary
Supervision Without Blocking: An Evidence-Driven Redesign of OTP-Style Fault Tolerance for Cooperative Runtimes
Sam Uwejan, Erdogan Dogdu
TL;DR
OTP-style supervision on cooperative runtimes leaves open which guarantees to retain when synchronous blocking is removed. The paper audits those guarantees and reports a runtime that closes the cataloged defect classes while preserving OTP contracts and adding supervisor responsiveness and handler-turn isolation.
Problem
Cooperative runtimes remove the blocking on which OTP supervisors’ consistency contracts depend, while the costs and possibilities of that change have not been evaluated.
Source: §I · p. 1Method
The authors examine pinned OTP sources, three Rust actor frameworks, actor-model literature, and blind audits, then replace push signaling with supervisor-owned completions and per-incarnation lifecycle mechanisms.
Source: §I · p. 1, §IX · p. 6Results
The runtime closes all 24 cataloged defect classes in 241 tests and preserves OTP’s verified contracts while adding responsiveness during group restarts and handler-turn isolation.
Source: §I · p. 1Takeaways & Limitations
The guarantee analysis makes the evidence and costs of each kept, added, or traded property explicit.
Source: §I · p. 1, §IX · p. 6The taxonomy is a lower bound from one controlled framework and three fixed-revision codebases; performance remains unmeasured, and peer-to-peer monitoring is not yet designed.
Source: §VII · p. 5, §VIII · p. 5
Abstract
from arXiv · showhide
Actor frameworks on mainstream runtimes advertise Erlang/OTP-style supervision, but OTP's consistency contracts assume a synchronous, single-threaded supervisor, and ports inherit that assumption's absence as unexamined risk. We evaluated every supervision guarantee against pinned OTP documentation and source read verbatim, against the source of three further Rust actor frameworks at fixed revisions, and against the actor-model literature, recording the evidence behind each design decision. Blind adversarial audits and the cross-framework study surfaced 24 defect classes, of which eight reduce to one root cause: reconstructing, by explicit mechanism, the atomicity a synchronous supervisor gets implicitly. We introduce a supervision runtime that closes all 24 cataloged classes through supervisor-owned completion values, per-incarnation identity, and a single lifecycle state machine, validated by 241 tests including reverted-fix reproductions. The result holds OTP's verbatim-verified contracts while adding two properties OTP does not offer: a supervisor that remains responsive throughout group restarts, and message handlers that are never cancelled mid-execution. We delimit the forcing guarantees no cooperative runtime can honor.
I. INTRODUCTION
OTP-style supervision has spread beyond the BEAM, but its consistency contracts rely on a synchronous supervisor, leaving cooperative-runtime implementations to decide what to preserve and how to replace the blocking. The paper evaluates those guarantees against primary sources, framework code, and adversarial audits, then presents ANON and its tested results.
I. INTRODUCTION
- OTP-style supervision has migrated beyond the BEAM, and production systems depend on those implementations honoring its contracts.OTP’s supervisor is single-threaded and blocks during restart and termination; cooperative schedulers remove that blocking to remain responsive, without prior evaluation of its costs or possibilities.Source: §I · p. 1
- The study evaluates guarantees using pinned OTP documentation and source, three Rust frameworks at fixed revisions, actor-model literature, and blind adversarial audits.It records the evidence behind each design decision.Source: §I · p. 1
- The design gives the supervisor owned completions, per-incarnation identity, one lifecycle state machine, and signal-first termination.It reproduces documented OTP semantics where supported and departs where the evidence favors another choice.Source: §I · pp. 1–2
- ANON closes all 24 cataloged defect classes and passes 241 tests, including reproductions that fail when fixes are reverted.It preserves OTP’s verbatim-verified contracts while keeping the supervisor responsive during group restarts and preventing handler cancellation mid-execution.Source: §I · p. 1
- The authors catalog 24 defect classes, reduce them to seven mechanism families, and identify one dominant root cause.They also document two families recurring in examined external frameworks.Source: §I · pp. 1–2
- The contribution includes a three-way account of reproduced OTP contracts, properties beyond OTP, and forcing guarantees cooperative runtimes cannot honor.The account pairs each guarantee with evidence and cost.Source: §I · p. 2
II. RELATED WORK
The paper situates its contribution in actor-model theory, OTP supervision, actor frameworks outside the BEAM, and research on actor bugs. Its focus is runtime supervision defects and the lifecycle signaling choices of Rust frameworks.
Supervision in Erlang/OTP
- OTP implements supervised restart as a synchronous, single-threaded process,, while this paper treats that synchronicity as load-bearing rather than incidental.It traces eight defect classes in ports to the absence of that synchronicity.Source: §II · p. 2
Actor frameworks outside the BEAM
- The paper’s contribution is a source-level comparison of lifecycle signaling in three Rust frameworks at pinned revisions, finding convergence on a shape none chose deliberately.The frameworks are actix, ractor, and archived bastion.Source: §II · p. 2
Studies of actor concurrency defects
- Unlike studies of application-level actor bugs, this work classifies supervision-runtime defects found through audits and cross-framework source examination.The paper claims no novelty in the actor model itself; its contribution concerns what fault tolerance demands from a host scheduler,,.Source: §II · p. 2
Structured concurrency
- The paper applies structured concurrency’s parent-owned task-lifetime discipline to supervision bookkeeping, making completions supervisor-owned values.It states that this removes the lost-signal class by construction; Tokio exposes owned task sets [2].Source: §II · p. 2
III. STUDY METHOD AND RACE TAXONOMY
The study uses three evidence sources designed to avoid relying on framework documentation claims: blind audits, primary-source verification of OTP semantics, and examination of other frameworks’ source.
A. Audit Protocol
- Independent blind reviews audited the full source and changes, requiring exact code locations and concrete failure interleavings for each defect.Accepted findings were retraced in code; paired reviews used complementary scopes.Source: A. Audit Protocol · p. 2
- OTP behavior claims were checked against pinned Erlang documentation, and supervisor.erl, including three design-critical termination and restart behaviors.The verified behaviors include unlink-and-flush on termination, escalation to direct untrappable kill, and re-inclusion of terminated children in group restarts.Source: A. Audit Protocol · p. 2
- The study examined ractor, actix 0.13.5, and archived bastion at pinned revisions using fixed lifecycle questions.Questions covered restart synchronicity, death signaling, kill delivery, and teardown on abort.Source: A. Audit Protocol · p. 2
B. The Root Cause
Eight of the 24 cataloged defects arise from reconstructing the atomicity a sequential OTP supervisor gets implicitly. The taxonomy organizes the 24 classes into seven mechanism families, each describing a distinct lifecycle failure.
B. The Root Cause
- Eight of 24 defect classes stem from a cooperative supervisor having to interleave death handling, manual commands, and restart completions while reconstructing OTP’s implicit atomicity.OTP processes a child’s exit message to completion before handling other events, except that failed restart retries are requeued to permit interleaving.Source: B. The Root Cause · p. 2
C. Mechanism Families
- A stale liveness flag can misclassify a crash as a manual stop, bypassing the restart budget and potentially deleting a restartable worker’s specification.The reported consequences occurred in the studied framework.Source: C. Mechanism Families · pp. 2–3
- Bounded-channel death and restart notifications can be dropped or delayed; bastion discards lifecycle send results, while ractor and actix avoid that exposure by blocking inline.The latter choice blocks the supervisor.Source: C. Mechanism Families · p. 3
- A child’s terminal state could become visible before its registry name was released, causing restart name collisions or refusal of a completed termination..Source: C. Mechanism Families · p. 3
- Reused incarnation identities let a late death report from an old instance mark its live successor dead.The window became exposed when termination grew faster.Source: C. Mechanism Families · p. 3
- Interleaved group restarts, solo restarts, and manual stops could wedge a group restart or violate a caller’s termination-completion contract.These interleavings are not expressible by a sequential supervisor.Source: C. Mechanism Families · p. 3
- Force-aborting a supervisor could orphan its subtree, while auxiliary timer and stream forwarders could outlive registrations or die silently on panic.The examined frameworks also showed teardown orphaning instances.Source: C. Mechanism Families · p. 3
- Stop and kill signals could wait for mailbox capacity, allowing escalation itself to block without a deadline..Source: C. Mechanism Families · p. 3
IV. AN OWNERSHIP-BASED SUPERVISION RUNTIME
The runtime replaces notification with ownership across lifecycle mechanisms and maps six mechanisms to defect families.
- Ownership replaces notification across the runtime’s lifecycle mechanisms.Source: §IV · p. 3
- The section presents six mechanisms and maps each to defect families.Source: §IV · p. 3
A. A Monotone Stop Lane
The runtime uses a monotone stop lane, supervisor-owned completions, and a single lifecycle ledger to manage child stopping, completion, and state transitions.
A. A Monotone Stop Lane
- A stop lane stores the highest-severity pending request and raises it synchronously, without mailbox-capacity dependence.Signals are observed at turn boundaries, so an in-flight handler is not cancelled.Source: A. A Monotone Stop Lane · p. 3
B. Owned Death and Restart Completions
- Supervisor-owned watcher and restart task sets return completion values, while per-incarnation fate cells record each child’s real termination reason.A fate is written after the child task completes its destructors, proving its registry name is free.Source: B. Owned Death and Restart Completions · p. 3
C. A Single Lifecycle Ledger
- The ledger restricts liveness to four states and one transition function, replacing scattered flags that caused misclassification.Manual stops commit synchronously, and completed-stop classification uses the fate cell’s termination reason.Source: C. A Single Lifecycle Ledger · p. 3
D. Per-Incarnation Identity
Per-incarnation identity prevents stale reports from matching successors; signal-first teardown and ledger-based group restarts address termination and interleaving defects.
D. Per-Incarnation Identity
- Monotonic incarnation numbers key fate, watcher, and adoption decisions, preventing a superseded instance’s death report from matching its successor.The BEAM enforces an analogous non-reuse property for process identifiers.Source: D. Per-Incarnation Identity · p. 3
E. A Signal-First Termination Cascade
- Teardown drops supervision state so child guards signal down the tree; a per-system reaper force-aborts children that exceed a bounded grace window.Linked processes receive a trappable killed signal, while direct kill is untrappable and immediate.Source: E. A Signal-First Termination Cascade · pp. 3–4
F. Group Restarts over the Ledger
- Ledger-based group restarts settle failed members to down for revival and restore terminated members in their original slot order.Manual termination during a group stop settles the ledger synchronously, preserving the caller’s completion contract within a message handler.Source: F. Group Restarts over the Ledger · p. 4
V. GUARANTEE ANALYSIS
The guarantee analysis distinguishes reproduced OTP contracts, properties added beyond OTP, and guarantees a cooperative runtime cannot honor. The runtime preserves termination truth while staying responsive, adds handler-turn isolation, and bounds its forcing guarantees by its host substrate.
- Guarantee analysis covers lifecycle guarantees, not all OTP semantics, and separates reproduced contracts, added properties, and cooperative-runtime limits.Source: §V · p. 4, Table I · p. 4
- On termination return, the child is fully gone and its name is free, allowing immediate same-name respawn while the supervisor processes other events.Source: §V · p. 4
- The supervisor remains responsive during group restarts, terminations, and escalations, reconstructing atomicity that blocking supplied implicitly.Eight defect classes arise because that implicit atomicity must be rebuilt explicitly.Source: §V · p. 4
- A begun handler always completes, with kill latency bounded by one handler turn rather than one scheduling step.The runtime declines mid-execution cancellation, which would leave local state at an arbitrary await point and impose cancellation-safety obligations on handlers.Source: §V · p. 4
- A non-yielding handler defeats both signal and task-abort backstops, while an operating-system kill of the host runs no teardown.Source: §V · p. 4
VI. EVALUATION
The evaluation tests whether the redesign closes its cataloged defect classes, using the framework’s test suite and real failure-path regressions. It establishes correctness properties, not performance, and cannot rule out undiscovered defect classes.
VI. EVALUATION
- The evaluation asks whether the redesign closes the cataloged defects and states the boundary of its evidence directly.Its evidence is the framework’s test suite, not a performance comparison.Source: §VI · p. 4, C. What This Evaluation Does Not Show · p. 5
A. Setup
- 241 passing tests cover unit, integration, and documentation tests; race regressions ran 10 times each, and the full suite ran three times per change batch.All tests used framework version 0.10.0 on Tokio’s multi-threaded scheduler.Source: A. Setup · p. 5
B. Per-Family Regression Evidence
- Integration tests drive real failure paths across all mechanism families, including competing crash and manual-stop intents, simultaneous deaths, delayed stale fates, restart interleavings, and saturated-mailbox kill delivery.Two classes have revert-verified regressions that failed before the fix and passed across 10 runs; remaining classes are closed by construction and tests pin replacement behavior.Source: B. Per-Family Regression Evidence · p. 5
C. What This Evaluation Does Not Show
- The evaluation demonstrates correctness properties, not performance; throughput, latency, and memory comparisons remain follow-up work.Test-based evidence also cannot establish that no further defect classes exist.Source: C. What This Evaluation Does Not Show · p. 5
VII. THREATS TO VALIDITY
The taxonomy and closure claims are bounded by the frameworks examined and by test-based evidence. Audit classifications are checkable but can vary, and examining the authors’ own framework risks favorable framing.
Scope
- The taxonomy is a lower bound, not a census: it covers one primary framework and three others at single pinned revisions, so unimplemented mechanisms could conceal additional classes.The authors’ control of the primary subject also shapes the taxonomy’s depth.Source: §VII · p. 5
Evidence standard
- Tests, including adversarial interleavings, cannot prove the absence of unexplored schedules; closure claims are therefore scoped to cataloged classes.Mitigations include repeated runs, revert verification for two classes, and blind audits.Source: §VII · p. 5
Audit reproducibility
- Independent auditors could identify a different class set or draw family boundaries differently, although per-class code citations and failure interleavings make the findings checkable.The categorization remains a matter of judgment.Source: §VII · p. 5
Author position
- Auditing and redesigning their own framework risks favorable framing; the authors count only code-traced findings and quote examined revisions in cross-framework comparisons.They also give limitations of the redesign prominence alongside its gains.Source: §VII · p. 5
VIII. DISCUSSION AND LIMITATIONS
The discussion links push-based lifecycle signaling to cooperative-runtime constraints and bounds how far the taxonomy generalizes. The conclusion reports closure of the cataloged classes with explicit ownership mechanisms, while leaving performance measurement and peer observation open.
VIII. DISCUSSION AND LIMITATIONS
- Four independently developed frameworks converged on push-based lifecycle signaling, which the authors attribute to async-runtime constraints; channels require less additional code than supervisor-owned task sets.Structured-concurrency primitives made the ownership alternative ergonomic only recently,.Source: §VIII · p. 5
- The taxonomy uses framework-neutral mechanism families and finds two in other examined codebases, but its evidence covers only four Rust frameworks.The authors expect applicability to never-blocking supervisors using push signaling on cooperative runtimes.Source: §VIII · p. 5
- Turn-boundary delivery bounds kill latency by one handler turn, while a never-yielding handler and a host-process kill fall outside the forcing guarantees.Runtime bookkeeping cost is unmeasured; peer-to-peer actor monitoring and an ownership-safe observation plane remain open.Source: §VIII · p. 5
IX. CONCLUSION
- The redesign closes all cataloged classes in the studied framework using supervisor-owned completions, per-incarnation identity, one lifecycle state machine, and a signal-first cascade.The conclusion reports validation by 241 tests, including two revert-verified reproductions, and identifies performance measurement and peer observation as follow-ups.Source: §IX · pp. 5–6