PaperlayerView exact revision on arXiv

Source-linked AI summary

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, Wei Lin

arXiv:2007.01045v1cs.DC

TL;DR

Training large DNNs on heterogeneous GPU systems requires efficient parallelization while preserving convergence and controlling memory without extra computation. DAPPLE combines synchronous data and pipeline parallelism with an automatic planner and an early-backward scheduler. It reports up to 3.23× planner speedup over PipeDream’s strategies and 1.6× runtime throughput versus GPipe with 12% lower memory consumption.

  • Problem

    Large DNN training on GPUs with diversified interconnects must improve efficiency and memory use while preserving convergence and avoiding additional computation.

  • Method

    DAPPLE combines synchronous data and pipeline parallelism with a planner for partitioning, replication, and placement and a scheduler that releases activation memory early.

  • Results

    DAPPLE’s planner reaches up to 3.23× speedup over PipeDream’s strategies under synchronous training, while its runtime achieves 1.6× GPipe throughput and 12% lower memory consumption.

  • Takeaways & Limitations

    DAPPLE provides a synchronous hybrid-parallel training approach that targets efficient large-model execution while reducing memory consumption without sacrificing training throughput.

  • Takeaways & Limitations

    GPipe-style synchronous scheduling requires activation memory proportional to the number of concurrently scheduled micro-batches, while recomputation adds approximately 20% computation.

Abstract

from arXiv · show

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23x under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6x speedup of training throughput and reduces the memory consumption of 12% at the same time.

I. INTRODUCTION

DAPPLE targets synchronous training of large DNNs on heterogeneous GPU systems by combining data and pipeline parallelism. Its planner and scheduler address strategy selection, convergence-compatible execution, memory consumption, and throughput.

  • Motivation: Large DNN models can require tens to hundreds of GBs of device memory, challenging training on GPUs with diversified interconnects.This motivates parallel training strategies that account for both model size and hardware connectivity.
  • Motivation: Synchronous data parallelism supports convergence and load balance but suffers gradient-synchronization overhead and cannot fit models exceeding one device’s memory.Aggressive data parallelism also requires large training batches, making model tuning harder.
  • Motivation: Pipeline parallelism partitions layers into stages and schedules micro-batches across interconnected devices, but synchronous execution can retain many intermediate activations.GPipe reduces this memory pressure through recomputation, which introduces approximately 20% additional execution-time overhead.
  • DAPPLE approach: DAPPLE automatically plans hybrid data/pipeline parallelization using model structure and hardware configurations, including partitioning, replication, placement, and communication-aware execution time.The planner targets the minimum theoretical execution time for a global step and addresses synchronous-training overheads absent from prior planning approaches.
  • DAPPLE approach: DAPPLE’s scheduler interleaves forward and backward stages deterministically to release activation memory earlier without storing multiple parameter versions or requiring recomputation.The framework combines synchronous training for convergence with a scheduling approach intended to balance parallelism, memory consumption, and execution efficiency.
  • Results: 1.6× training-throughput speedup and 12% lower memory consumption were reported for DAPPLE runtime versus GPipe, while the planner reached up to 3.23× speedup over PipeDream’s strategies under synchronous training.DAPPLE was evaluated across image classification, machine translation, and language modeling, and the planner also reached up to 2.32× speedup over the best data-parallel baseline.

B. Pipeline Planning

DAPPLE’s planner addresses synchronous pipeline-planning gaps by jointly optimizing stage partitioning, replication, and topology-aware device assignment for hybrid data/pipeline parallelism.

  • Pipeline planning determines stage partitioning and device mapping to maximize resource utilization and training throughput.
  • Synchronous planning must model parameter-update and replicated-stage AllReduce synchronization overheads to estimate end-to-end training time accurately.
  • With fixed micro-batches M and communication-overhead ratio α, fewer stages S yield higher pipeline efficiency.
  • DAPPLE’s planner uses profiling results to generate optimized hybrid parallelization plans for a given global batch size.The profiler measures layer execution time, activation sizes, and parameter sizes; planning is performed offline.
  • The planner minimizes one training iteration’s end-to-end execution time through stage partition, replication, and hardware-topology-aware device assignment.Replicating stages across devices enables hybrid data and pipeline parallelism that can exploit hierarchical interconnects.
  • DAPPLE’s runtime reduces recomputation needs while retaining reasonably low memory use and enough micro-batches to keep devices busy.

III. DAPPLE SCHEDULE

DAPPLE schedule limits activation memory by injecting micro-batches gradually and interleaving forward and backward tasks, while preserving pipeline efficiency relative to GPipe under matched configurations.

  • GPipe’s activation memory grows as O(M) with concurrently scheduled micro-batches and approximately 20% recomputation overhead is used to reduce it.
  • DAPPLE injects K micro-batches with K < M, then alternates one forward task with one backward task to release activations earlier.
  • Early backward scheduling frees each micro-batch’s forward activations after its corresponding backward task, allowing memory reuse by later micro-batches.
  • DAPPLE introduces the exact same bubble time as GPipe when stage partition, micro-batches, and device mapping are identical.
  • DAPPLE combines early backward scheduling with recomputation to further exploit memory usage.

IV. DAPPLE PLANNER

DAPPLE Planner generates an optimal hybrid parallelism plan from profiling, hardware, and batch-size inputs, minimizing synchronous pipeline latency across partition, replication, and placement choices.

  • DAPPLE Planner uses profiling results, hardware configurations, and global training batch size to generate an optimal hybrid parallelism execution plan.
  • Pipeline latency is the global-step execution time, optimized across all data- and pipeline-parallelism solution spaces.
  • The pivot stage has the least bubble overhead and dominates the steady phase used to calculate pipeline latency.
  • Synchronous pipeline execution comprises warmup, steady, and ending phases, whose durations sum to pipeline latency.
  • DAPPLE models inter-stage communication as independent stages and includes AllReduce overhead in the ending phase.
  • The latency objective approximates true latency by omitting internal bubbles in the pivot stage, while working well across reported benchmarks.
  • Device placement affects communication efficiency and utilization; DAPPLE permits arbitrary stage-to-device mappings unlike prior hierarchical placement.
  • Three placement policies—Fresh First, Append First, and Scatter First—reduce the search space to less than O(2S).

C. Planning Algorithm

DAPPLE selects the pivot stage by iteratively identifying the stage with the smallest bubble overhead, using steady-phase and intervening communication costs.

  • DAPPLE uses a heuristic to choose the pivot stage with minimum bubble overhead because that stage dominates the steady phase.
  • The initial pivot stage is S −1, and DAPPLE updates it iteratively toward stage 0.
  • The steady-phase duration for stage j is (M −1) × (Fj + Bj).
  • For a preceding stage s, DAPPLE changes the pivot when its steady-phase time exceeds the current pivot’s time plus intervening forward and backward costs.

2) Algorithm Formulation:

DAPPLE formulates planning as a dynamic-programming search over layer partitions, GPU allocations, replication, and placement, while extending prior planning beyond synchronous pipeline constraints.

  • 2) Algorithm Formulation:: The subproblem TP L(j, m, g) plans the first j layers on m GPUs with device set g, leaving remaining layers as a replicated final stage.
  • 2) Algorithm Formulation:: The full objective minimizes TP L(N, G, G) over partition boundary j, allocated GPU count m, and feasible device set g.
  • 2) Algorithm Formulation:: Each dynamic-programming transition adds a partition after layer j′, splitting the remaining stage into two stages with separate GPU allocations.
  • 2) Algorithm Formulation:: DAPPLE estimates each new subproblem’s latency while enumerating its three device-placement strategies.
  • 2) Algorithm Formulation:: The planner reuses the prior pivot stage when possible, avoiding iteration over all earlier stages during each transition.
  • 2) Algorithm Formulation:: Memorized search records the best split, replication, and placement for each point in the solution space.
  • D. Contributions over previous work: Unlike PipeDream’s asynchronous planning and GPipe’s block partitioning, DAPPLE explores uneven partitions and broader device assignments for synchronous pipelines.

1) Uneven Pipeline Partitioning with Fewer Stages:

DAPPLE improves synchronous pipeline planning by favoring fewer stages, slightly uneven partitions, and broader device assignments, then transforms planned graphs for replicated execution.

  • 1) Uneven Pipeline Partitioning with Fewer Stages:: Fewer pipeline stages reduce bubble overhead when the number of micro-batches is fixed.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: Slightly uneven model partitions can outperform perfectly even splits in synchronous pipeline training.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: DAPPLE’s device assignment strictly extends PipeDream’s hierarchical recursive partitioning and supports broader stage-placement choices.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: For layers with large activations relative to weights, Scatter First can replicate a layer across machines, using NVLink for activation communication and Ethernet for AllReduce.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: DAPPLE runtime converts the planned model graph into a pipelined parallel graph with stage-specific forward, backward, communication, and synchronous-update subgraphs.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: Stage 0 consuming twice stage 1’s time illustrates why replicated-stage approaches can differ in efficiency.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: Forward graphs are built stage by stage before backward graphs are constructed in reverse order to enforce dependencies across stage boundaries.
  • 1) Uneven Pipeline Partitioning with Fewer Stages:: Replicated stages can have different node counts on adjacent stages, requiring specialized split-concat operations.

2) Cross Stage Communication:

DAPPLE combines replicated pipeline stages with micro-batch scheduling and split-concat communication to balance utilization, communication, and memory. Its scheduler interleaves forward and backward work to release memory early while preserving execution efficiency.

  • Cross-stage communication: DAPPLE replicates pipeline stages across devices, splitting micro-batches for balanced stage execution rather than assigning whole micro-batches round robin.Replication supports hybrid data and pipeline parallelism; split-concat operations connect adjacent stages for one-to-many, many-to-one, and many-to-many communication.
  • Weight updates: Each device accumulates gradients across micro-batches, synchronizes replica gradients with AllReduce, and applies averaged gradients once.This differs from naive training because multiple micro-batches are injected concurrently and stage replication complicates weight updates.
  • Micro-batch scheduling: The number of concurrently scheduled forward micro-batches trades off pipeline parallelism against peak memory consumption.More concurrent micro-batches can improve utilization but increase memory use; Ki captures the stage-specific schedule size.
  • Micro-batch scheduling: DAPPLE schedules forward and backward stages in a deterministic round-robin order to release each pipeline task’s memory as early as possible.The schedule uses control dependencies to enforce micro-batch ordering and early backward execution.
  • Micro-batch scheduling: Policy A uses Ki = min(S − i, D) for low ACR, whereas Policy B uses Ki = min(2 ∗ (S − i) − 1, D) when communication needs more overlap.ACR is the ratio between cross-stage communication latency and average forward/backward computation time.

VI. EVALUATION

DAPPLE is evaluated across six DNN models, three GPU interconnect environments, and fixed-device planning scenarios. The best strategy depends on model structure, memory, communication topology, and computation-communication balance.

  • Evaluation setup: DAPPLE evaluates image classification, machine translation, and language modeling across hierarchical NVLink plus Ethernet and flat 25 Gbps or 10 Gbps Ethernet environments.The hardware study includes 16 available devices for planning comparisons.
  • Planning representation: DAPPLE’s plans encode pipeline stage replication and placement, with cross-server replication favored less on hierarchical interconnects because it adds inter-server communication.A P : Q plan denotes two stages replicated on P and Q devices, respectively.
  • Planning results: All evaluated models prefer data parallelism on a single Config-A server, where fast intra-server bandwidth handles gradient communication.The study reports bandwidth up to 130GB/s and gradient communication magnitudes up to 3.7GB.
  • Planning results: ResNet-50 consistently favors data parallelism because its 100MB model has high computation density, even on the 10 Gbps configuration.Gradient accumulation and computation/communication overlap allow DP to outperform pipelining in this case.
  • Planning results: VGG-19 uses data parallelism on Config-A and Config-B but favors a 15 : 1 pipeline on Config-C because its large final fully connected layer makes replicated gradient synchronization costly.The model has 548MB of parameters, with most weights concentrated in the final fully connected layer.

C. Performance Analysis

Across five models at fixed global batch size 128, DAPPLE’s hybrid plans outperform overlapped data parallelism across all three hardware configurations. Model-specific gains reflect weight, activation, layer, and network characteristics.

  • Overall performance: 1.71X, 1.37X, and 1.79X averaged speedups are achieved by DAPPLE hybrid plans over overlapped DP on Config-A, Config-B, and Config-C, respectively.The comparison fixes GBS = 128 across five models.
  • Model-specific analysis: 2.32X speedup is achieved for GNMT-16 on Config-C when DAPPLE’s hybrid approach is compared with the best data-parallel baseline.GNMT-16 uses a two-stage pipeline on faster networks and a straight pipeline at sufficiently large batch size on Config-C.
  • Model-specific analysis: VGG-19 benefits from splitting convolutional and fully connected layers because about 70% of weights reside in the final fully connected layer while activation size falls to 3MB.Replicating convolutional layers without the final fully connected layer reduces communication overhead on slow interconnects.
  • Model-specific analysis: BERT-48 and XLNet-36 favor DAPPLE across all configurations, with DAPPLE’s advantage over DP variants increasing as the network slows from 25 Gbps to 10 Gbps.The supplied analysis attributes this to communication and computation overlap opportunities and model memory differences.
  • Model-specific analysis: AmoebaNet-36 favors DAPPLE in all three configurations because its parameter and computation distributions require load-balanced staging while accounting for allreduce overhead.Its last third contains 73% of parameters, and lower-bandwidth networks shift the split toward larger layer IDs.

D. Scheduling Policy

DAPPLE’s scheduling policy selects the number of warm-up micro-batches using communication-to-computation characteristics, then combines early backward execution with optional recomputation. This improves throughput while reducing or preserving memory usage relative to GPipe.

  • Scheduling policy: 10% and 31% throughput improvements result from choosing Policy B over Policy A for VGG-19 and GNMT-16, whose ACR values are relatively high.For BERT-48 and XLNet-36, low ACR produces little policy difference, so the smaller-memory Policy A is preferred.
  • DAPPLE versus GPipe: Both DAPPLE and GPipe save about 19% averaged peak memory but lose 20% throughput when recomputation is applied with M = 2.This comparison keeps the micro-batch count fixed and isolates the recomputation trade-off.
  • DAPPLE versus GPipe: DAPPLE’s early backward scheduling keeps peak memory independent of the number of micro-batches, enabling larger M and reducing pipeline bubbles.The comparison uses BERT-48 with a two-stage pipeline on Config-B.
  • DAPPLE versus GPipe: 1.6× higher throughput and 0.88× averaged peak memory are achieved by DAPPLE without recomputation at M = 16 compared with GPipe, which supports only two micro-batches.The throughput gain is attributed mainly to the lower bubble proportion from using more micro-batches.
  • DAPPLE versus GPipe: 0.70× memory consumption is achieved by DAPPLE plus recomputation at M = 16 relative to baseline GPipe without recomputation.This combination can support larger micro-batch sizes or larger models.
  • Comparison with PipeDream: DAPPLE strategies consistently outperform PipeDream planner strategies by up to 3.23× under synchronous training.Both planners use the same configurations and profiling data, and their strategies run with DAPPLE Runtime.

H. Weak Scaling

DAPPLE scales large-model training across GPUs by combining pipeline and data parallelism with planner and scheduler support. On 8 V100s with NVLink, it supports BERT models up to 5.5B parameters, with linear model-size scaling in this setting.

  • Weak Scaling: 5.5B parameters: DAPPLE supports this BERT model size on 8 V100s with NVLink.The increase is achieved by varying layer counts under reasonable input size with re-computation.
  • Weak Scaling: Linear scaling: maximum BERT model size grows with layer count because encoder-layer parameters are balanced across devices.Longer pipelines introduce more bubbles and slightly reduce average GPU utilization.
  • Weak Scaling: 3.23× speedup: DAPPLE’s planner outperforms PipeDream’s planner under synchronous training scenarios.The planner jointly considers pipeline partition, replication, and placement to generate hybrid data/pipeline-parallel strategies.
  • Weak Scaling: 1.6× throughput speedup and 12% memory savings: DAPPLE’s scheduler outperforms GPipe on both training throughput and memory consumption.The scheduler avoids storing multiple parameter versions and reduces reliance on costly re-computation.
Loading 2007.01045v1…