PaperlayerView exact revision on arXiv

Source-linked AI summary

Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs

John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, Guoqing Harry Xu

arXiv:2204.12013v1cs.DCcs.LG

TL;DR

Training large DNNs is costly, and preemptible instances’ frequent, unpredictable interruptions challenge reliable, efficient training. Bamboo has each pipeline node perform redundant computations over its own and its successor’s layer shard, enabling predecessor-based recovery after preemption. On EC2 spot clusters comprised of p3 instances, Bamboo delivers a 3.6× cost reduction versus on-demand instances and outperforms checkpointing by 3.7×.

  • Problem

    Training large DNNs is costly, and preemptible instances’ frequent, unpredictable interruptions challenge reliable, efficient training.

    Source: §1 · pp. 1–2

  • Method

    Bamboo has each pipeline node perform redundant computations over its own and its successor’s layer shard, enabling predecessor-based recovery after preemption.

    Source: §1 · p. 2

  • Results

    On EC2 spot clusters comprised of p3 instances, Bamboo delivers a 3.6× cost reduction versus on-demand instances and outperforms checkpointing by 3.7×.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across six representative models, Bamboo provides higher value than on-demand training and checkpointing/restart on spot instances.

    Source: §9 · p. 15

    One-node redundancy cannot recover from consecutive pipeline-node preemptions, while adding redundancy can make computation and communication overhead substantial.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

DNN models across many domains continue to grow in size, resulting in high resource requirements for effective training, and unpalatable (and often unaffordable) costs for organizations and research labs across scales. This paper aims to significantly reduce training costs with effective use of preemptible instances, i.e., those that can be obtained at a much cheaper price while idle, but may be preempted whenever requested by priority users. Doing so, however, requires new forms of resiliency and efficiency to cope with the possibility of frequent preemptions - a failure model that is drastically different from the occasional failures in normal cluster settings that existing checkpointing techniques target. We present Bamboo, a distributed system that tackles these challenges by introducing redundant computations into the training pipeline, i.e., whereby one node performs computations over not only its own layers but also over some layers in its neighbor. Our key insight is that training large models often requires pipeline parallelism where "pipeline bubbles" naturally exist. Bamboo carefully fills redundant computations into these bubbles, providing resilience at a low cost. Across a variety of widely used DNN models, Bamboo outperforms traditional checkpointing by 3.7x in training throughput, and reduces costs by 2.4x compared to a setting where on-demand instances are used.

1 Introduction

Bamboo targets the high cost of training growing DNNs and the inefficiency of checkpointing under frequent, unpredictable preemptions. It uses neighboring nodes’ redundant computations to keep training resilient, scheduling and managing that work to limit overhead.

1 Introduction

  • Growing DNNs raise training costs, while cheap preemptible instances bring unpredictable interruptions that make stateful training and checkpoint/restart inefficient.Checkpointing and restart spent 77% of training time on restarting and redoing work in the GPT-2 example.Source: §1 · pp. 1–2
  • Bamboo provides resilience for pipeline and pure data-parallel training by letting each node compute its own shard and its successor’s shard.It currently does not support model parallelism.Source: §1 · p. 2
  • Forward redundant computation runs eagerly in pipeline bubbles, while backward redundant computation runs lazily only after a preemption.The eager forward work can overlap normal computation; lazy backward work uses results already in memory to keep pauses short.Source: §1 · p. 2
  • Bamboo swaps forward redundant-computation intermediates to CPU memory during normal training and restores them to GPU memory only for backward recovery.The intermediates, rather than the redundant layers, are the main source of GPU-memory overhead.Source: §1 · p. 2
  • Zone-separated neighbors reduce consecutive-preemption risk at less than 5% overhead; Kubernetes reconfiguration adds instances, adjusts pipelines, and checkpoints periodically.Consecutive preemptions still require reconfiguration; if the cluster is exhausted and too few nodes remain, training suspends until instances are available.Source: §1 · pp. 2–3
  • On six representative DNN models trained on EC2 p3 spot clusters, Bamboo delivered a 3.6× cost reduction versus on-demand instances and outperformed a checkpointing approach by 3.7×.In BERT simulations across a wide range of preemption probabilities, Bamboo’s performance-per-dollar stayed constant and was 2.48× that of on-demand instances.Source: §1 · p. 3

2 Background

Pipeline parallelism partitions a model into layer shards and processes microbatches through stages, but scheduling and uneven partitions leave idle bubbles. Bamboo keeps synchronous training and uses those bubbles for redundant computation.

2 Background

  • Pipeline parallelism assigns layer shards to stages, sends each microbatch forward and backward through them, and allows different nodes to process different microbatches concurrently.The number of stages is the pipeline depth, and each worker communicates with the adjacent stages.Source: §2 · p. 3
  • GPipe schedules all forward passes before backward passes, while PipeDream’s 1F1B schedule interleaves them to reduce bubbles and peak memory use.The bubble is idle time in the pipeline schedule.Source: §2 · p. 3
  • Pipeline bubbles remain difficult to eliminate because equal-rate layer partitions are hard to find, and partitioning algorithms can be model- and hardware-specific and time-consuming.Those properties make such algorithms unsuitable when preemptions continually change the number of nodes.Source: §2 · p. 3
  • Bamboo uses synchronous microbatching to keep model state consistent and runs redundant computation in existing bubbles rather than trying to eliminate them.Asynchronous pipelining can use stale weights and introduce state inconsistencies during reconfiguration.Source: §2 · p. 3

3 Motivation

Real cloud traces show frequent, bulky, unpredictable preemptions, while checkpoint/restart and sample dropping have significant costs under frequent interruptions. Bamboo’s design exploits preemption patterns across zones; live migration also depends on idle targets that spot markets cannot guarantee.

3 Motivation

  • Across cloud and GPU-family traces, preemptions and additions are frequent, bulky, and unpredictable, making spot clusters highly dynamic.The traces covered 24-hour windows, with clusters of 64 instances except for one GCP cluster of 80.Source: §3 · p. 4, Figure 2 · p. 4
  • Most simultaneous preemptions in the analyzed traces came from a single zone, motivating Bamboo to place consecutive pipeline nodes in different zones.This best-effort placement can recover from many preemptions with one-node redundancy, though consecutive-node preemptions still require reconfiguration.Source: §3 · p. 4
  • In GPT-2 training across 64 EC2 p3.2xlarge GPU spot instances, restarting overheads for adapting checkpoints to new pipeline configurations plus wasted computations consumed 77% of training time; preemptions during checkpointing forced rollback.Source: §3 · p. 5
  • Sample dropping can lose enough samples under frequent preemptions to significantly affect accuracy; the experiment’s results understate the actual effect.The simulation randomly zeroed a selected pipeline’s gradients on a preemption event, whereas real dropped-sample rates can be higher.Source: §3 · p. 5
  • Live migration requires enough idle target nodes, but spot allocations cannot guarantee replacements for every preempted instance.The cited experiments always obtained fewer new allocations per preemption event than the number of preempted nodes.Source: §3 · pp. 5–6

4 Overview

Bamboo targets efficient, preemption-safe training on spot instances, using user-set pipeline parameters as upper bounds rather than optimizing cluster configuration. Its agents, workers, and runtime coordinate pipeline execution and recovery.

4 Overview

  • Bamboo asks users to set pipeline count D and depth P, and recommends about 1.5× more pipeline stages than an on-demand setup to avoid critical-path CPU/GPU swapping.The extra nodes hold redundant layers and accommodate pipeline adjustments; spot instances are cheaper, and preemptions can leave fewer active instances.Source: §4 · p. 6
  • P × D is the target cluster size and an upper bound; Bamboo passively adjusts pipelines after preemptions or new allocations rather than searching for a faster configuration.It requests instances to restore the target size when preemptions reduce pipeline depth or count.Source: §4 · p. 6
  • Each node runs an agent that monitors a DeepSpeed worker; workers and agents coordinate through etcd, while the runtime switches to a failover schedule after communication fails.Workers interpret statically generated instructions for computation and communication.Source: Figure 5 · p. 6, §4 · p. 6

5 Redundant Computation

Bamboo places replicas on neighboring pipeline nodes and uses pipeline bubbles to limit redundant-computation overhead. It runs forward redundancy eagerly, backward redundancy only after preemption, and reconfigures when failures exceed what one-node redundancy can handle.

5.1 Redundant Layers and Computation

  • Each node stores a replica of its layer shard on its predecessor, with the last node serving as predecessor to the first.A node performs normal computation on its own layers and redundant computation on its successor’s replicated layers.Source: §5.1 · p. 7
  • Eager forward redundancy stays local to the node, while backward redundancy would require extra inter-node communication, so Bamboo delays it until preemption.Eager forward computation also produces intermediate results that make recovery’s backward computation faster; lazy forward computation would lengthen recovery pauses.Source: §5.1 · p. 7
  • One-node redundancy cannot recover from consecutive pipeline-node preemptions; additional replicas may exceed bubble capacity and add inter-node dependencies.Bamboo instead separates consecutive nodes across zones to reduce the chance of adjacent losses; reported cross-zone transfer overhead is below 3%.Source: §5.1 · p. 8

5.2 Schedule Redundant Computation

  • The first four stages’ bubbles fit the full forward redundant computation; the last four cover about 60%, with the remainder overlapping regular forward computation.The measurements use BERT on four eight-stage pipelines with on-demand V100 GPUs.Source: Figure 10 · p. 8, §5.2 · pp. 8–9
  • Bamboo schedules forward redundancy in communication-wait bubbles and overlaps the portion that does not fit with regular forward computation on a different microbatch.The scheduling example describes bubbles arising when a faster stage waits at a communication barrier for a slower stage.Source: §5.2 · pp. 8–9
  • Bamboo swaps forward-redundancy outputs and intermediate results to CPU memory after each microbatch, while keeping redundant weights on the GPU for reuse.The swapped tensors are unnecessary during normal training because backward redundancy runs only after preemption.Source: §5.2 · pp. 8–9
  • After preemption, a failover schedule merges victim and shadow instructions, prioritizing communication and backward computation; multi-GPU nodes instead replicate all node layers as group replicas.Instructions formerly communicating between victim and shadow are removed, and backward computation runs early to free intermediate-result memory.Source: §5.2 · p. 9
  • For pure data parallelism, Bamboo replicates parameters and optimizer state on another node and over-provisions instances to reduce eager-redundancy overhead.Eager forward redundancy doubles batch size but takes about 1.5× computation time; over-provisioning by 1.5 × D keeps overhead below 10%.Source: §5.2 · p. 9

6 Pipeline Reconfiguration

Bamboo reconfigures pipelines when simultaneous adjacent failures occur or when end-of-step conditions make rebalancing urgent. It aims to restore the user-specified pipeline depth while managing spare nodes and redistributing redundant layers.

6 Pipeline Reconfiguration

  • Bamboo periodically checkpoints model state for rare fatal failures, because reconfiguration pauses training much longer than recovery using redundant computation.The checkpoints are not used except when training must restart after too many nodes are preempted to continue.Source: §6 · pp. 9–10
  • Reconfiguration triggers immediately after simultaneous consecutive preemptions or at an optimizer-step boundary when pipeline rebalancing is urgent.Urgency includes enough new nodes to reconstruct a pipeline or enough failures to approach a critical failure.Source: §6 · p. 10
  • Bamboo prioritizes restoring pipeline depth P, placing surplus arriving nodes in standby and reducing data-parallel pipeline count when too few nodes arrive for full pipelines.If arrivals and standby nodes can form another pipeline, Bamboo adds one; redundant layers are redistributed across the updated pipelines.Source: §6 · p. 10
  • At reconfiguration, the first node to reach the rendezvous selects the new cluster configuration and records it in etcd; nodes transfer model and optimizer state while reusing existing state where possible.Each node receives a new stage assignment, including layers, state, and redundancies.Source: §6 · p. 10

7 Evaluation

Bamboo was evaluated on six DNN models using EC2 spot instances and replayed preemption traces, alongside simulations and comparisons with other systems. Across the evaluation, Bamboo’s value exceeded on-demand training, while its resilience and overhead depended on preemption patterns and configuration.

7 Evaluation

  • The evaluation trained six representative vision and language models on EC2 p3 spot instances, targeting a validation accuracy for each model.The study also replayed collected traces and ran simulations to evaluate differing preemption conditions.Source: §7 · p. 10, §7.1 · p. 11

7.1 Training Throughput and Costs

  • Bamboo’s throughput at the 10% preemption rate averaged 15% below DeepSpeed on D×Pdemand instances.The authors attribute the reduction to fewer active instances, reconfiguration overhead averaging 7% of training time, and eager FRC increasing iteration time.Source: §7.1 · p. 11
  • 1.95× higher value overall than training with on-demand instances, with the benefit persisting for five models at a 33% preemption rate.The 33% rate was the worst-case segment of the collected trace.Source: §7.1 · p. 11

7.2 Different Failure Models

  • Across simulated preemption probabilities, Bamboo’s value remained stable and above the on-demand value of 1.1.Higher preemption probabilities reduced active training instances, while costs fell proportionally.Source: §7.2 · p. 12

7.3 Comparisons with Other Systems

  • On BERT, Bamboo-S exceeded Varuna by 2.5× and 2.7× in throughput at 10% and 16% preemption, respectively, and Varuna hung at 33%.Bamboo-S’s value advantages were 1.67× and 1.64× at 10% and 16%, respectively.Source: §7.3 · p. 13, Figure 13 · p. 13

7.4 Microbenchmarks of Redundant Computation

  • Eager-FRC-lazy-BRC, Bamboo’s RC setting, balances per-iteration overhead against pause time; lazy FRC reduces pause time by about 35%.Eager FRC incurs about 1.5× GPU-memory overhead, while lazy FRC incurs none.Source: §7.4 · p. 13

7.5 Cross-Zone Communication

  • Distributing nodes across availability zones usually changed throughput by less than 5% compared with placing them in one zone.The reported difference was low despite assigning consecutive pipeline nodes from different zones.Source: §7.5 · p. 14

7.6 Bamboo for Pure Data Parallelism

  • For pure data-parallel training, Bamboo outperformed Checkpoint by 1.64× in throughput and 1.22× in value.Both spot-instance approaches had higher value than on-demand training: 2× for Checkpoint and 1.79× for Bamboo.Source: §7.6 · p. 14, Table 6 · p. 14

8 Related Work

Related work covers parallel training, elastic resource management, and systems exploiting spot instances; Bamboo is positioned as complementary work focused on resilient DNN training with low-cost redundant computation.

8 Related Work

  • Data, model, and pipeline parallelism partition datasets, operators, and layers respectively; pipeline parallelism uses microbatches to saturate execution.The section also describes 3D parallelism as combining all three approaches.Source: §8 · p. 14
  • Related work includes elasticity and spot-instance approaches such as TorchElastic and Proteus, with Proteus exploiting dynamic pricing but not explicitly considering modern deep-learning workloads.The section also describes data-parallel weight synchronization through all-reduce or parameter servers.Source: §8 · p. 14
  • Elastic-training systems adapt to changing resources, while Varuna focuses on elasticity rather than quick recovery from preemptions [2].The section distinguishes Bamboo as specifically designed for frequent preemptions.Source: §8 · pp. 14–15
  • Spot-resource systems address cost, SLOs, or transient workloads, while GPU-scheduling work focuses on efficiency and throughput; these approaches are described as orthogonal to Bamboo.Bamboo instead aims to perform redundant computation at low cost.Source: §8 · p. 15

9 Conclusion

An evaluation across six representative models finds that Bamboo provides higher value than on-demand training and checkpointing/restart on spot instances.

9 Conclusion

  • Across 6 representative models, Bamboo provides much higher value than on-demand training and checkpointing/restart on spot instances.Source: §9 · p. 15

Related papers

Loading 2204.12013v1…