PaperlayerView exact revision on arXiv

Source-linked AI summary

FailBench: Evaluating Fault Tolerance Across Distributed Training Architectures

Khaled Aljbab, Amine Barrak

arXiv:2610.07688v1cs.DC

TL;DR

Fault-tolerance mechanisms are typically evaluated within their own architecture, leaving little quantitative evidence for cross-architecture selection. FailBench compares mechanisms across distributed-training architectures and finds that trade-offs shift with architecture and state size, with no universal winner.

  • Problem

    Fault-tolerance mechanisms are usually evaluated only within their own architecture, leaving practitioners without quantitative cross-architecture evidence for choosing among them.

    Source: §I · p. 1

  • Method

    FailBench provides a unified harness to compare seven distributed-training architectures, eight crash-fault-tolerance mechanisms, and a no-FT baseline across 142 tuples.

    Source: §I · p. 2, §XI · p. 13

  • Results

    Fault tolerance has no universal winner: checkpointing trade-offs shift with architecture and state size, while A6 gossip raises post-fault sample throughput 17.9% without detectable loss-progress improvement.

    Source: §XI · p. 13

  • Takeaways & Limitations

    Mechanism choice should use a measurement-grounded decision framework that accounts for failure rate, state size, checkpoint interval, signal availability, and service-level constraints.

    Source: §XI · p. 13

    Validation of A6’s learning-efficiency result on a real training workload remains open.

    Source: §X · p. 12

Abstract

from arXiv · show

Distributed deep learning relies on data, pipeline, tensor, and hybrid parallelism, yet fault-tolerance mechanisms are typically evaluated only on the architecture for which they were designed. This leaves practitioners with little guidance when choosing mechanisms across architectures. FailBench provides a unified evaluation harness covering seven distributed training architectures, eight crash-fault-tolerance mechanisms and a no-FT baseline, and single, concurrent, and cascading fail-stop failures. We evaluate 142 (architecture, mechanism, trace) combinations on an 8xV100 cluster, with per-rank checkpoint states ranging from 205 MB to 2.7 GB. Three findings emerge. First, no mechanism is universally best: on A2, disk checkpointing has the lowest steady-state overhead (0.5%), in-memory replication restores fastest (~17 ms), and just-in-time checkpointing avoids periodic steady-state checkpoint cost but incurs ~0.9 s upon failure. When process-group re-formation takes seconds, mechanisms differ more in runtime overhead than restore speed. Second, gossip training increases sample throughput by 17.9% after losing a worker, yet shows no detectable improvement in loss progress over a matched no-fault baseline. Third, mechanism cost depends strongly on architecture: in-memory replication overhead ranges from 3.7% to 176%. We translate these findings into a decision framework for selecting fault-tolerance mechanisms and release FailBench as an open artifact.

I. INTRODUCTION · II. BACKGROUND · A. Distributed Training Architectures

Distributed training faces routine failures across architectures with distinct communication patterns and failure exposures; FailBench addresses the lack of cross-architecture evidence for choosing fault-tolerance mechanisms.

II. BACKGROUND

  • Fault-tolerance approaches include checkpointing, replicated pipeline stages, coded computation, and elastic or decentralized training that continues at reduced cluster size.Checkpoint variants include in-memory replication, asynchronous or lazy disk writes, and just-in-time capture before preemption.Source: §I · p. 1

I. INTRODUCTION

  • Prior evaluations compare fault-tolerance mechanisms only within their own design spaces, leaving practitioners without a common cross-architecture basis for choosing a strategy.The paper identifies the missing comparison as head-to-head measurement on the same cluster under a common fault schedule.Source: §I · p. 1
  • Across 142 tuples, six configurations halt after one worker crash, A1-async and A6 gossip degrade gracefully, and checkpointing spans 0–3.8% steady-state overhead and 17–900 ms restore time with no dominant mechanism.Gossip’s raw throughput rises post-fault without improving learning efficiency.Source: §I · p. 2
  • FAILBENCH pairs seven architectures with eight crash-fault-tolerance mechanisms and a no-FT baseline, and evaluates 142 architecture–mechanism–trace tuples.The study injects fail-stop crashes at 8 GPUs with 0.2–2.7 GB of per-rank state; Byzantine faults, fail-slow stragglers, silent corruption, and coded computation are out of scope.Source: §I · pp. 1–2

A. Distributed Training Architectures

  • A1’s server-versus-worker role and synchronization policy shape failure impact; A2 stalls on any worker loss, while A3’s impact depends on the failed tree position.A1 compares synchronous gather/broadcast with asynchronous per-worker updates; A2 uses ring all-reduce, and A3 uses tree all-reduce.Source: A. Distributed Training Architectures · p. 2
  • A4 pipeline-stage loss halts execution, while A5 tensor-parallel rank loss breaks the distributed matrix operation.A4 moves activations and gradients between stages; A5 combines partial outputs with all_reduce.Source: A. Distributed Training Architectures · p. 2
  • A6 decentralized peer-to-peer SGD can degrade gracefully after worker loss because it averages with live neighbors and reconfigures without a central coordinator.Timed-out neighbors are removed from the live set.Source: A. Distributed Training Architectures · p. 2
  • A7 combines data, pipeline, and tensor parallelism in a 2×2×2 configuration, so recovery cost depends on which parallelism axis contains the failed worker.Its failure topology combines all three axes.Source: A. Distributed Training Architectures · p. 2

B. Fault Tolerance Mechanisms · Checkpoint-class recovery (FT1–FT4): · State replication (FT5–FT6):

FailBench studies eight crash-fault-tolerance mechanisms plus a no-FT baseline, spanning checkpointing, state replication, and runtime reconfiguration. The mechanisms trade persistence, communication, memory, and recovery limitations in different ways.

B. Fault Tolerance Mechanisms

  • FailBench evaluates eight crash-fault-tolerance mechanisms plus a no-FT baseline across checkpointing, state replication, and runtime reconfiguration.Source: B. Fault Tolerance Mechanisms · p. 2

Checkpoint-class recovery (FT1–FT4):

  • Periodic disk checkpointing (FT1) saves model, optimizer, and RNG state every T steps, bounding lost work by T while persistence cost scales with model size,.Source: Checkpoint-class recovery (FT1–FT4) · p. 2
  • In-memory replication (FT2) mirrors the latest checkpoint to peer GPU or host memory, avoiding file-system recovery access at the cost of peer memory and communication.Source: Checkpoint-class recovery (FT1–FT4) · p. 2
  • Asynchronous/lazy disk checkpointing (FT3) stages state in a CPU buffer for background persistence, removing synchronous disk flushes and enabling shorter effective intervals at comparable throughput cost, [12].Source: Checkpoint-class recovery (FT1–FT4) · pp. 2–3
  • Just-in-time checkpointing (FT4) captures state only on preemption signals, eliminating periodic checkpoint work but incurring emergency transfer costs and failing to protect against unannounced crashes.The implementation captures Bamboo/Gemini-JIT's lead-time-reactive behavior but omits shadow-rank lineage replay.Source: Checkpoint-class recovery (FT1–FT4) · p. 3

State replication (FT5–FT6):

  • Continuous replication (FT5) mirrors full model and optimizer state to a peer each step, proxying compute-redundant systems' steady-state bandwidth and memory costs but pessimistically estimating recovery latency –.Source: State replication (FT5–FT6) · p. 3
  • K-replica redundancy (FT6) writes state to k=2 peers so any survivor can provide recovery state, but does not implement Reed–Solomon gradient coding or compare against it,.Source: State replication (FT5–FT6) · p. 3

Reconfiguration and baseline (FT7–FT9): … IV. THE FAILBENCH FRAMEWORK

FAILBENCH pairs distributed-training architectures with fault-tolerance mechanisms under a common fault injector and metric collector, addressing prior evaluations that usually compare mechanisms only within their target design spaces. Its framework records compatibility across architecture–mechanism pairs, including cases that cannot execute without redesign.

Reconfiguration and baseline (FT7–FT9):

  • FT7 redistributes data across surviving ranks at a reduced world size for architectures with per-rank data shards; FT8 uses PyTorch Elastic or Horovod Elastic to re-form the process group after worker loss.FT8 resumes training on the reduced cluster without application-level intervention.Source: Reconfiguration and baseline (FT7–FT9) · p. 3
  • FT9 is the no-FT control: its hooks are no-ops, recovery is declined, and the runner propagates the training exception, exposing each architecture’s inherent failure behavior in RQ1.Asynchronous gradient submission without a barrier is treated as an A1-async architecture property, not an FT mechanism.Source: Reconfiguration and baseline (FT7–FT9) · p. 3

III. RELATED WORK

  • Prior crash-fault-tolerance work falls into checkpointing, redundancy and coding, and decentralized or elastic optimization.Source: §III · p. 3

A. Fault Tolerance in Distributed Deep Learning

  • Checkpoint research spans adaptive intervals, peer-memory replication, asynchronous or lazy persistence, and just-in-time saves, with systems typically benchmarked against checkpoint variants on their target architecture and workload.Examples include CheckFreq, Gemini, DataStates-LLM, ByteCheckpoint [12], and just-in-time checkpointing.Source: A. Fault Tolerance in Distributed Deep… · p. 3
  • Redundancy systems replicate pipeline stages or use redundant scheduling, while gradient coding and coded matrix multiplication reconstruct aggregate updates from worker subsets; evaluations usually compare each system with vanilla parallelism in its target regime.Examples include Bamboo, Oobleck, Varuna, SWIFT, gradient coding, and coded matrix multiplication.Source: A. Fault Tolerance in Distributed Deep… · p. 3
  • Decentralized and asynchronous optimization accepts convergence costs for graceful degradation, while elastic APIs reconfigure process groups as clusters shrink or grow.Decentralized SGD uses peer-to-peer averaging, with convergence bounds under topology changes and failures,; asynchronous parameter servers and Hogwild! tolerate stragglers and crashes,, and elastic frameworks include PyTorch, Horovod [20], Megatron-LM [3], and DeepSpeed.Source: A. Fault Tolerance in Distributed Deep… · pp. 3–4

IV. THE FAILBENCH FRAMEWORK

  • The framework reports 142 tuples across a 7×9 architecture–mechanism compatibility matrix, pairing training architectures with mechanisms under a common fault injector and metric collector.The implementation is released as an artifact.Source: A. Fault Tolerance in Distributed Deep… · p. 4, §IV · p. 4

A. Fault Tolerance in Distributed Deep Learning

  • Prior evaluations typically compare mechanisms within their target design spaces; FAILBENCH distinguishes viable, degenerate, and structurally infeasible pairs, excludes unimplemented gradient coding, and scopes faults to fail-stop crashes.MLPerf-style benchmarks disclose fixed configurations but do not inject faults, while general-purpose chaos tools inject faults without training-specific metrics.Source: A. Fault Tolerance in Distributed Deep… · pp. 3–4, Reconfiguration and baseline (FT7–FT9) · p. 3

A. Design Goals … V. EXPERIMENTAL METHODOLOGY

FailBench uses shared design principles and an architecture–mechanism compatibility matrix to structure fault-tolerance evaluation, then applies deterministic rank-level fault injection and a defined experimental methodology.

A. Design Goals

  • Four design goals are common plug-in interfaces, reproducibility up to CUDA floating-point nondeterminism, rank-targeted architecture-agnostic fault injection, and consistently defined metrics.Architectures and mechanisms implement two abstract base classes; a seed and fault schedule reproduce a run.Source: A. Design Goals · p. 4

B. Architecture–Mechanism Compatibility Matrix

  • Table II classifies architecture–mechanism pairs as viable, degenerate, or structurally infeasible to show how mechanisms transfer across architectures.Viable pairs support implementation as designed and evaluation by FailBench.Source: Table II · p. 4, B. Architecture–Mechanism… · p. 4
  • FT7 on A4, A5, and A7 is degenerate because those architectures lack per-rank data shards to redistribute, reducing it to checkpoint-based restart.Degenerate pairs are implementable but collapse into a no-op or an already measured alternative.Source: B. Architecture–Mechanism… · p. 4
  • Across three fault traces, the matrix yields 180 candidate architecture–mechanism–trace tuples after excluding degenerate cells, while FailBench reports 142.FT2 is structurally infeasible on A6 because its globally addressable mirror rank protocol conflicts with A6’s per-edge gossip subgroups.Source: B. Architecture–Mechanism… · p. 4

C. Fault Injector

  • The injector applies deterministic, architecture-agnostic fault schedules by issuing an in-process SIGKILL to a target rank.Per-step measurements and fault events are logged in jsonl and aggregated offline.Source: C. Fault Injector · p. 4

V. EXPERIMENTAL METHODOLOGY

  • The experimental methodology covers the execution platform, fault-injection protocol, workloads, sweep parameters, metrics, and statistical procedures.Source: §V · p. 4

A. Execution Platform: Matilda HPC … D. Sweep Parameters

FailBench evaluates seven architectures on a shared 8-GPU, two-node Matilda HPC platform using architecture-matched workloads and fixed crash traces. Its injector applies comparable fail-stop events across architectures, with injection timing adjusted for state size and stabilization needs.

A. Execution Platform: Matilda HPC

  • Each Matilda compute node has four V100 16 GB GPUs, 32 CPU cores, and 192 GB host memory; 8-GPU NCCL ring all-reduce reaches 9.7 GB/s effective bus bandwidth.Nodes use HDR InfiniBand and BeeGFS, with jobs submitted through SLURM.Source: A. Execution Platform: Matilda HPC · p. 4
  • The software stack uses PyTorch 2.3.0, CUDA 12.1, NCCL 2.20.5, and Python 3.10, with all architectures implemented as FAILBENCH plug-ins.Source: A. Execution Platform: Matilda HPC · p. 4

B. Fault Injection Techniques

  • The F1 injector kills one worker with SIGKILL at t∈[30, 120] s, or at t=180 s for A6; injection may be delayed for large-state architectures so just-in-time mechanisms can save.Survivors detect failure through architecture-specific communication or detection paths.Source: B. Fault Injection Techniques · pp. 4–5
  • Three traces cover a single fail-stop crash, simultaneous crashes targeting ranks r and r+2, and three cascading crashes spaced 30 seconds apart.For cascading faults, time to detection and recovery latency are recorded separately for each event.Source: B. Fault Injection Techniques · pp. 4–5
  • The architecture-agnostic injector holds fault type, rank selection, count, and inter-arrival spacing fixed across architectures, varying injection time with state size; A6 waits 180 seconds for subgroup stabilization.The experiments assume SLURM, NCCL, and the kernel TCP/IB stack are correct.Source: B. Fault Injection Techniques · p. 5

C. Workloads

  • Data-parallel architectures A1, A2, A3, and A6 train ResNet 50 with approximately 200 MB of state per rank, while A4, A5, and A7 use Megatron-style MLP stacks with 2.5 to 2.7 GB per rank.ResNet uses micro batch 32 per rank and fp32 SGD with momentum 0.9.Source: C. Workloads · p. 5
  • The MLP workloads reproduce Transformer feed-forward expansion, nonlinearity, and contraction, preserving state, tensor, communication, collective, and process-group properties while omitting attention, embeddings, and sequence processing.They capture communication and state behavior of large Transformer training without implementing a complete language model.Source: C. Workloads · p. 5

D. Sweep Parameters

  • All experiments use 8 V100 GPUs across two nodes, with architectures mapped to their corresponding parameter-server, ring, tree, pipeline, tensor-parallel, gossip, or 2×2×2 hybrid layouts.The layouts are 1 PS plus 7 workers for A1; 8-rank ring, tree, and gossip for A2, A3, and A6; 8-stage pipeline for A4; TP group of 8 for A5; and DP×PP×TP=2×2×2 for A7.Source: D. Sweep Parameters · p. 5

E. Statistical Protocol · F. Metric Definitions · G. Restore Time Model and Validation

The protocol distinguishes directly measured recovery from analytical lower bounds, and validation shows realized restore times exceed the model’s bounds. The sections define the reported metrics and explain the sampling, measurement, and estimation procedures.

E. Statistical Protocol

  • Sample sizes range from n=1 to 7 according to wall-clock cost and experiment type, while smoke-run-confirmed structurally infeasible cells provide no viable Trestore.Source: E. Statistical Protocol · p. 5
  • FT7 and FT8 activate only after failure, so they add no steady-state overhead, and their Trestore values are analytical lower bounds because external rendezvous and process reinitialization are not executed by the runner.Examples include PyTorch Elastic and a c10d rendezvous server.Source: E. Statistical Protocol · p. 5
  • Crash experiments without fault tolerance report means with 95% Student’s t confidence intervals, while fault-tolerance experiments report means and half ranges because small samples do not support reliable normality assumptions.Outliers are retained, and same-seed runs replay the same deterministic fault schedule.Source: E. Statistical Protocol · p. 5

F. Metric Definitions

  • The four reported metrics are steady-state overhead, time to detect, effective state restore cost, and recovery time; total fault-to-resume latency is TTD + Trecover.Throughput with and without FT is denoted Ton and Toff; recovery latency entries are marked m when measured and a when analytical, and TPG covers process-group rebuilding while Tinit covers mechanism reinitialization.Source: F. Metric Definitions · p. 5

G. Restore Time Model and Validation

  • For halting architectures, a 600 s process-group watchdog terminates the process before the Python recovery path can log Trestore, so it is estimated from the dominant state-transfer path.Surviving ranks block on communication with the failed rank until the watchdog fires.Source: G. Restore Time Model and Validation · p. 6
  • Analytical Trestore uses state size divided by PCIe Gen 3 bandwidth for in-memory FT2 and FT4 restore, or measured filesystem read bandwidth for disk-based FT1 and FT3.For FT4, measured emergency-save cost is added separately; Trecover is measured when training resume is observed, otherwise it is a lower bound combining Trestore, TPG, and Tinit.Source: G. Restore Time Model and Validation · p. 6
  • M1 validation used n = 10 cold-cache iterations for approximately 205 MB ResNet 50 and 2.6 GB MLP states, measuring disk loads with torch.load deserialization and pinned-memory-to-GPU PCIe transfer.The disk path loaded serialized .pt state dictionaries; the PCIe path transferred from a preloaded buffer.Source: G. Restore Time Model and Validation · p. 6
  • Measured restore time is approximately 20%–120% above the analytical bound, which the paper treats as a lower bound rather than an exact time.The smaller PCIe gap reflects transfer and synchronization overhead; the larger disk gap also includes torch.load deserialization costs, especially for small checkpoints with many tensors.Source: Table III · p. 6, G. Restore Time Model and Validation · p. 6

H. Measurement Provenance Across the 142 Tuples … C. Results

The no-FT F1 measurements reveal different failure exposure across architectures: several configurations halt after worker loss, while asynchronous and gossip training continue; gossip’s throughput gain does not improve loss progress.

H. Measurement Provenance Across the 142 Tuples

  • Of 142 reported tuples, 133 are viable and 9 structurally infeasible; among viable tuples, 10 have directly observed recovery or continuation and 123 use analytical Trestore or reconfiguration bounds.All 9 structurally infeasible cases halted in smoke tests; steady-state overhead is zero by construction for reactive-only mechanisms such as FT7 and FT8.Source: Table IV · p. 6, H. Measurement Provenance Across the… · p. 6

A. Motivation

  • The no-FT baseline characterizes whether surviving ranks continue training and how quickly they observe worker loss, establishing what recovery mechanisms must improve and revealing topology effects.The measurements use F1, a single SIGKILL of one worker rank.Source: A. Motivation · p. 6

B. Approach

  • For F1 no-FT runs, the study records survivor continuation, time-to-detect, post-fault throughput, and job outcome, with the A6 fault delayed to t=180 s.Sample sizes are n = 5 for six halting configurations, n=3 for A1-async, and n=7 for A6; other faults are injected at t=30 s.Source: B. Approach · p. 6

C. Results

  • A1-sync, A2, A3, A4, A5, and A7 halt after F1 as surviving ranks block on communication requiring the failed rank; TTD ranges from 600.13 ± 0.03 s (A4) to 615.24 ± 0.58 s (A7).The dependencies include parameter-server synchronization, all-reduce, pipeline communication, and combinations of DP, PP, and TP paths.Source: C. Results · p. 6, Table V · p. 7
  • A1-async completes all three runs after detecting loss in 5.26 ± 0.01 s, with throughput falling from 226.8 ± 2.1 to 195.9 ± 3.4 samples/s (13.65% ± 1.34% slowdown).With responder threshold one and a 5 s receive timeout, it removes the failed worker and continues with six workers; the slowdown is close to the structural loss of approximately 14.3%.Source: Table V · p. 7, C. Results · p. 7
  • A2 TTD tracks the configured process-group timeout: 30.01 ± 0.01, 120.03 ± 0.01, and 600.03 ± 0.01 s, showing the long delay is a watchdog effect, not an architectural detection floor.Changing NCCL_TIMEOUT without changing the init_process_group argument did not change the approximately 600 s TTD.Source: Table VI · p. 7, C. Results · p. 7
  • A6 aggregate throughput rises 17.9% (85.4 to 100.7 samples/s), but loss reduction (0.130 vs. 0.126) shows no statistically detectable improvement over matched no-fault training.The similar transient loss increases—0.30 in fault runs and 0.32 in the baseline near t = 220 s—are attributed to gossip overlay dynamics, not recovery.Source: Figure 2 · p. 7, C. Results · p. 7

VII. RQ2: RESILIENCE–COST TRADE-OFFS · A. Motivation · B. Approach

RQ2 empirically maps the trade-off between steady-state overhead and recovery latency for fault-tolerance mechanisms on a fixed deployment.

A. Motivation

  • Choosing a mechanism requires balancing continuous steady-state overhead against recovery latency once a fault occurs.Overhead includes storage, network, and redundant compute; the comparison helps determine whether mechanisms are substitutes, complements, or Pareto-competitive.Source: A. Motivation · p. 8
  • Prior FT papers report favorable single points, but head-to-head comparisons across mechanism families on the same workload are uncommon,,,.RQ2 reconstructs this frontier empirically on a fixed deployment.Source: A. Motivation · p. 8

B. Approach

  • The controlled comparison uses ResNet-50 with micro-batch 32, fp32 SGD with momentum, and ∼205 MB per-rank state on 8×V100 across two nodes.The architecture is A2 Ring-AR, disk mechanisms use BeeGFS scratch, and the fault is a single SIGKILL at t = 30 s; all choices except the FT mechanism are held constant.Source: B. Approach · p. 8
  • Seed counts are n=6 for FT2, FT3, FT5, and FT6, and n=3 for FT1 and FT4; FT7 and FT8 are reactive-only and reported analytically.Seed count is scaled to wall-clock cost, with more seeds where variance is informative.Source: B. Approach · p. 8
  • The evaluation logs SS overhead, effective Trestore, per-save cost, and expected lost work, then places configurations on the (SS, Trestore) plane to identify the frontier.SS overhead is relative throughput loss versus the failure-free RQ1 baseline in Table V.Source: B. Approach · p. 8
  • FT1 and FT2 use the same 100-step full-state checkpoint but persist it to BeeGFS and peer pinned host memory via gloo, respectively.This tests whether in-memory replication (Gemini-style) dominates disk or forms a Pareto trade-off.Source: B. Approach · p. 8
  • The second experiment includes all remaining V configurations, reports FT7 and FT8 analytically, and summarizes both experiments in Table VII.The evaluated configurations are those marked V in Table II.Source: B. Approach · p. 8

C. Results · D. From Restore Time to End-to-End Recovery · VIII. RQ3: SENSITIVITY ACROSS DEPLOYMENTS

On A2, FT1, FT2, and FT4 occupy distinct Pareto positions because steady-state overhead, restore time, and lost work trade off. Adding multi-second process-group re-formation compresses restore-time ratios and makes steady-state cost more decisive.

C. Results

  • FT1 and FT2 checkpoint the same 204.7 MB state every 100 steps, while FT7 and FT8’s analytical recovery bounds are excluded from the frontier as non-comparable measurements.FT1 and FT2 are compared on A2 + F1; FT7 and FT8 require external reconfiguration that the harness does not execute.Source: C. Results · p. 8
  • FT1’s 0.48% SS overhead is lower than FT2’s 3.78%, while FT2’s ∼17 ms Trestore is faster than FT1’s ∼155 ms.FT1 saves from rank 0 without a barrier, while FT2’s coordinated ring communication puts all eight ranks on the critical path; expected completion time favors FT1 unless faults arrive every few seconds.Source: C. Results · p. 8
  • FT4’s emergency save completes in 884.7 ± 89.8 ms within a 2.0 s lead time and inherits FT2’s ∼17 ms Trestore.FT4 uses peer-pair in-memory storage with a reactive trigger, conceptually closest to Gupta et al.’s JIT-checkpoint mechanism,,. A warning shorter than ∼885 ms at this state size makes it no better than no-FT for that fault.Source: C. Results · pp. 8–9
  • Periodic mechanisms re-execute ∼50 steps, or ≈5.6 s per fault, under uniform arrival, whereas FT4 re-executes almost nothing when its warning arrives early enough.The periodic estimate uses the 100-step interval and A2’s ∼112 ms/step; FT4’s viability is bounded by its emergency-save cost.Source: C. Results · pp. 8–9
  • On the A2 frontier, FT1 disk, FT2 in-memory, and FT4 JIT are non-dominated: FT2 incurs 3.78% steady-state overhead, while FT4 does no periodic checkpoint work but has about 50× FT2’s restore time.FT4’s effective restore time includes its emergency save on the critical path. FT7 and FT8 are excluded because recovery requires external reconfiguration the harness does not execute.Source: C. Results · p. 8
  • FT5 incurs 372.30 ± 41.16% SS overhead at the same ∼17 ms Trestore as FT2, making it Pareto-dominated despite re-executing at most one step.FT2 incurs 3.78% SS overhead and can re-execute up to 99 steps at interval 100.Source: C. Results · p. 9

D. From Restore Time to End-to-End Recovery

  • With TPG varied from 2 to 10 s, FT2 ≺ FT1 ≺ FT4 remains the recovery ordering, but the FT1/FT2 ratio falls from ∼9× to 1.01–1.07×.The common process-group term preserves absolute gaps of 138 ms and ∼747 ms; once TPG dominates, SS overhead is more decisive: FT4 has no periodic checkpoint work, versus 0.48% for FT1 and 3.78% for FT2.Source: D. From Restore Time to End-to-End… · p. 9

A. Motivation · B. Approach · C. Results

The cross-architecture and multi-fault results show that fault-tolerance trade-offs shift with architecture, state size, and runtime recovery capability. FT2 offers low overhead and fast restore on small-state architectures, while FT4 avoids periodic checkpoint work but can have long restore times on large states.

A. Motivation

  • The study asks whether architecture, per-rank state size, or fault regime changes the fixed-A2 mechanism trade-off enough to make a single-cell recommendation unsafe.Source: A. Motivation · p. 9

B. Approach

  • The evaluation varies FT2 and FT4 across architectures under F1, uses a 2.6 GB A2 control, and tests multiple worker losses with F2 and F3.Runs use 8×V100; multi-fault cells are exploratory at n ≤2 except decision-framework cells, which are boosted to n=5.Source: B. Approach · p. 9

C. Results

  • FT2 warm-save throughput is ∼460–510 MB/s across A2–A5 and A7, while A1 is ∼290 MB/s; A4 and A5 overhead exceeds 100% because 2.5 GB state is large relative to 33 ms steps.Small-state viable FT2 cells A1–A3 cluster at 3.69–3.91% overhead; A6 is structurally infeasible because it lacks a long-lived full-rank ring process group.Source: C. Results · pp. 9–10, Table VIII · p. 10
  • FT2 costs 3.69–3.91% steady-state overhead for ∼17 ms restore on A1–A3, while FT4 has ∼0.9–1.5 s effective restore on small states; on large states FT2 reaches 14–176% overhead and FT4 9.9–10.8 s.The trade-off also shifts with larger state even when architecture is fixed, and only A6 consistently continues in the tested F2/F3 cells.Source: Table VIII · p. 10, Table IX · p. 10, C. Results · p. 11
  • FT4 has zero periodic steady-state checkpoint work on every architecture, but effective restore ranges from 889 ms on A3 to nearly 11 s on A5 and A7.At identical 204.7 MB state, A1 emergency save is 1479.5 ± 231.9 ms versus A2 884.7 ± 89.8, A3 871.8 ± 39.3, and A6 881.1 ± 103.5; all seven saves finish within the warning window.Source: C. Results · p. 10, Table IX · p. 10
  • With A2 held fixed, FT1 overhead rises from 0.48 ± 0.03% at 205 MB to 3.14 ± 0.24% at 2.6 GB, while FT2 fails to complete training at 2.6 GB.All three FT2 seeds timed out on a default-process-group ALLREDUCE before fault injection; A4, A5, and A7 complete FT2 at the same state size, but the proposed cause remains a hypothesis.Source: C. Results · p. 10, Table X · p. 11
  • Periodic FT2, FT3, and FT6 overhead varies by at most 1.1 percentage points across F1–F3 on A1–A5 and A7, whereas FT5 varies by 3–8 points.A6 is excluded from this comparison because its F2/F3 throughput was sampled before its gossip cadence stabilized.Source: C. Results · p. 10
  • Only A6 gossip recovers consistently among tested F2 configurations; A6+FT1 completes all 2000 steps under F3, while every tested non-A6 F3 configuration halts.A6+FT1 F2 recovery is 2537 ± 2272 ms; its F3 recovery is 3905 ± 2445 ms, as non-A6 runtimes cannot reform their process groups before the second fault.Source: C. Results · pp. 10–11

IX. DECISION FRAMEWORK

The framework selects among characterized fault-tolerance candidates by minimizing expected job completion time subject to steady-state and pause-on-fault constraints, with storage and memory treated as feasibility limits. The preferred mechanism depends on failure intervals, restore caps, storage tier, warning coverage, architecture, and state size.

IX. DECISION FRAMEWORK

  • Selection minimizes expected completion time using overhead and expected failure costs, subject to steady-state and pause-on-fault caps; storage, host memory, and device memory constrain feasibility.Recovery cost includes detection, restore, process-group reconstruction, and initialization; periodic mechanisms lose about half an interval of work, while covered reactive mechanisms lose near zero.Source: §IX · p. 11
  • On A2 at 205 MB, FT2 minimizes completion time only below M ≈ 4.2 s (9.7 s with cold-cache restores); otherwise FT1 minimizes it unless its restore cap binds.FT1 and FT2 lose the same work because they checkpoint at the same interval, while FT2 has 3.3 points more overhead and 138 ms less per-fault cost.Source: §IX · p. 11
  • With a warning whose lead time exceeds emergency-save cost, FT4 minimizes measured A2 completion time at every M; without covered warning, it is not a candidate.For large states, its effective restore is ∼10 s, compared with FT2’s restore under 230 ms.Source: §IX · p. 11
  • The table is read by architecture class and state, then failure regime and binding constraint; its selections cover characterized candidates, and absolute values should be re-derived on the target cluster.Evidence codes distinguish measured, extrapolated, and degradation-derived selections; transferable axes are per-rank state size and signal coverage. FT4 is measured across four small-state architectures with restores from 889 ms on A3 to 1497 ms on A1, while small-state FT1’s 0.48% overhead and 155 ms restore are measured on A2 only.Source: Table XII · p. 12, §IX · p. 12
  • FT2 is excluded for M2 because all three seeds deadlocked before fault injection, and no evaluated mechanism completes F2 or F3 end-to-end recovery without external orchestration.The pipeline/TP FT2 selection is extrapolated from Table VIII’s 100-step cell assuming SS overhead scales inversely with interval; A6 F2/F3 latency is dominated by native membership handling, so deployments should provision for several seconds of reconvergence rather than the mean alone. On halting architectures, recoverability is verified but completed recovery is not, making Trec a lower bound.Source: §IX · p. 12

X. THREATS TO VALIDITY · XI. CONCLUSION

FailBench finds that fault-tolerance outcomes and checkpoint trade-offs depend on architecture, while its recovery claims are bounded by the tested configurations, workloads, and fault model.

X. THREATS TO VALIDITY

  • Injector timing skew is below 20 ms, and the first 50 steps are discarded; sub-20 ms analytical restore bounds are not measured end-to-end recovery.The A1-sync/A1-async comparison changes both backend and synchronization policy; A1-async’s responder threshold of one makes graceful degradation under one fault close to guaranteed by construction, so the result characterizes that configuration.Source: §X · p. 12
  • On NCCL-primary architectures, Trestore is an analytical lower bound, with measured costs 20–120% higher; fixed fault victims limit generalization of position-dependent effects.SS overhead is amortized FT cost relative to the failure-free baseline; FT4’s zero SS entry reflects no periodic checkpoint work, with emergency save charged to effective Trestore.Source: §X · p. 12
  • On halting architectures, the runner establishes recoverability but not completed end-to-end recovery because it does not perform survivor election or process-group reconstruction.A6 reconfigures natively, while A1-async is native graceful continuation rather than FT recovery; FT7/FT8 have no state-preservation path and report only pre-fault SS overhead.Source: §X · p. 12
  • Exploratory F2/F3 cells use n≤2, while decision-framework cells use n=5; FT4–FT6 are simplified production proxies, and A5 omits Megatron’s full autograd wrappers.These proxies preserve targeted state-transfer and failure structures but are not end-to-end convergence evaluations; small-n FT experiments report mean ± half-range.Source: §X · p. 12
  • A6’s post-fault sample throughput rises 17.9% without detectable improvement in loss progress; loss reduction and time to target do not differ detectably (p = 0.67 and p = 0.83).The comparison uses a matched no-fault baseline over a 1123 s window from the fault instant (n = 7 fault, n = 3 baseline); real-workload validation remains open.Source: §X · p. 12, §XI · p. 13

XI. CONCLUSION

  • FailBench evaluates seven distributed-training architectures, eight FT mechanisms, and a no-FT baseline across 142 architecture–mechanism–trace tuples.The evaluation covers single, concurrent, and cascading fail-stop failures.Source: §XI · p. 13
  • Six configurations halt after one worker failure, while A1-async and A6 degrade gracefully; A2 checkpointing has a disk, in-memory, and JIT Pareto frontier, but operating points shift with architecture and state size.Some architecture–mechanism pairs are structurally infeasible rather than merely unevaluated.Source: §XI · p. 13
  • The decision framework selects among characterized candidates using failure rate, state size, checkpoint interval, signal availability, and service-level constraints.Future work should validate trade-offs on larger clusters, full Transformer workloads, and broader fault models.Source: §XI · p. 13

Related papers

Loading 2610.07688v1…