PaperlayerView exact revision on arXiv

Source-linked AI summary

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, Wei Lin

arXiv:2007.01045v1cs.DC

TL;DR

Training large DNNs on heterogeneous GPU systems requires efficient parallelization while preserving convergence and controlling memory without extra computation. DAPPLE combines synchronous data and pipeline parallelism with an automatic planner and an early-backward scheduler. It reports up to 3.23× planner speedup over PipeDream’s strategies and 1.6× runtime throughput versus GPipe with 12% lower memory consumption.

  • Problem

    Large DNN training on GPUs with diversified interconnects must improve efficiency and memory use while preserving convergence and avoiding additional computation.

    Source: §I · p. 2, A. Limitations of GPipe Schedule · p. 4

  • Method

    DAPPLE combines synchronous data and pipeline parallelism with a planner for partitioning, replication, and placement and a scheduler that releases activation memory early.

    Source: §I · p. 2, B. Early backward scheduling · p. 4, §VIII · p. 13

  • Results

    DAPPLE’s planner reaches up to 3.23× speedup over PipeDream’s strategies under synchronous training, while its runtime achieves 1.6× GPipe throughput and 12% lower memory consumption.

    Source: §I · p. 2

  • Takeaways & Limitations

    DAPPLE provides a synchronous hybrid-parallel training approach that targets efficient large-model execution while reducing memory consumption without sacrificing training throughput.

    Source: §I · p. 2

    GPipe-style synchronous scheduling requires activation memory proportional to the number of concurrently scheduled micro-batches, while recomputation adds approximately 20% computation.

    Source: A. Limitations of GPipe Schedule · p. 4

Abstract

from arXiv · show

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23x under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6x speedup of training throughput and reduces the memory consumption of 12% at the same time.

I. INTRODUCTION

DAPPLE targets synchronous training of large DNNs on heterogeneous GPU systems by combining data and pipeline parallelism. Its planner and scheduler address strategy selection, convergence-compatible execution, memory consumption, and throughput.

Motivation

  • Large DNN models can require tens to hundreds of GBs of device memory, challenging training on GPUs with diversified interconnects.This motivates parallel training strategies that account for both model size and hardware connectivity.Source: §I · p. 1
  • Synchronous data parallelism supports convergence and load balance but suffers gradient-synchronization overhead and cannot fit models exceeding one device’s memory.Aggressive data parallelism also requires large training batches, making model tuning harder.Source: §I · p. 1
  • Pipeline parallelism partitions layers into stages and schedules micro-batches across interconnected devices, but synchronous execution can retain many intermediate activations.GPipe reduces this memory pressure through recomputation, which introduces approximately 20% additional execution-time overhead.Source: §I · pp. 1–2

DAPPLE approach

  • DAPPLE automatically plans hybrid data/pipeline parallelization using model structure and hardware configurations, including partitioning, replication, placement, and communication-aware execution time.The planner targets the minimum theoretical execution time for a global step and addresses synchronous-training overheads absent from prior planning approaches.Source: §I · p. 2
  • DAPPLE’s scheduler interleaves forward and backward stages deterministically to release activation memory earlier without storing multiple parameter versions or requiring recomputation.The framework combines synchronous training for convergence with a scheduling approach intended to balance parallelism, memory consumption, and execution efficiency.Source: §I · p. 2

Results

  • 1.6× training-throughput speedup and 12% lower memory consumption were reported for DAPPLE runtime versus GPipe, while the planner reached up to 3.23× speedup over PipeDream’s strategies under synchronous training.DAPPLE was evaluated across image classification, machine translation, and language modeling, and the planner also reached up to 2.32× speedup over the best data-parallel baseline.Source: §I · p. 2

B. Pipeline Planning

DAPPLE’s planner addresses synchronous pipeline-planning gaps by jointly optimizing stage partitioning, replication, and topology-aware device assignment for hybrid data/pipeline parallelism.

  • Pipeline planning determines stage partitioning and device mapping to maximize resource utilization and training throughput.Source: B. Pipeline Planning · p. 3
  • Synchronous planning must model parameter-update and replicated-stage AllReduce synchronization overheads to estimate end-to-end training time accurately.Source: B. Pipeline Planning · p. 3
  • With fixed micro-batches M and communication-overhead ratio α, fewer stages S yield higher pipeline efficiency.Source: B. Pipeline Planning · p. 3
  • DAPPLE’s planner uses profiling results to generate optimized hybrid parallelization plans for a given global batch size.The profiler measures layer execution time, activation sizes, and parameter sizes; planning is performed offline.Source: C. The DAPPLE Approach · p. 3
  • The planner minimizes one training iteration’s end-to-end execution time through stage partition, replication, and hardware-topology-aware device assignment.Replicating stages across devices enables hybrid data and pipeline parallelism that can exploit hierarchical interconnects.Source: C. The DAPPLE Approach · p. 3
  • DAPPLE’s runtime reduces recomputation needs while retaining reasonably low memory use and enough micro-batches to keep devices busy.Source: C. The DAPPLE Approach · pp. 3–4

III. DAPPLE SCHEDULE

DAPPLE schedule limits activation memory by injecting micro-batches gradually and interleaving forward and backward tasks, while preserving pipeline efficiency relative to GPipe under matched configurations.

IV. DAPPLE PLANNER

DAPPLE Planner generates an optimal hybrid parallelism plan from profiling, hardware, and batch-size inputs, minimizing synchronous pipeline latency across partition, replication, and placement choices.

  • DAPPLE Planner uses profiling results, hardware configurations, and global training batch size to generate an optimal hybrid parallelism execution plan.Source: §IV · p. 4
  • Pipeline latency is the global-step execution time, optimized across all data- and pipeline-parallelism solution spaces.Source: A. The Optimization Objective · p. 4
  • The pivot stage has the least bubble overhead and dominates the steady phase used to calculate pipeline latency.Source: A. The Optimization Objective · p. 5
  • Synchronous pipeline execution comprises warmup, steady, and ending phases, whose durations sum to pipeline latency.Source: A. The Optimization Objective · p. 5
  • DAPPLE models inter-stage communication as independent stages and includes AllReduce overhead in the ending phase.Source: A. The Optimization Objective · p. 5
  • The latency objective approximates true latency by omitting internal bubbles in the pivot stage, while working well across reported benchmarks.Source: A. The Optimization Objective · p. 5
  • Device placement affects communication efficiency and utilization; DAPPLE permits arbitrary stage-to-device mappings unlike prior hierarchical placement.Source: B. Device Assignment · p. 5
  • Three placement policies—Fresh First, Append First, and Scatter First—reduce the search space to less than O(2S).Source: B. Device Assignment · p. 5

C. Planning Algorithm

DAPPLE selects the pivot stage by iteratively identifying the stage with the smallest bubble overhead, using steady-phase and intervening communication costs.

2) Algorithm Formulation:

DAPPLE formulates planning as a dynamic-programming search over layer partitions, GPU allocations, replication, and placement, while extending prior planning beyond synchronous pipeline constraints.

2) Algorithm Formulation:

  • The subproblem TP L(j, m, g) plans the first j layers on m GPUs with device set g, leaving remaining layers as a replicated final stage.Source: 2) Algorithm Formulation · p. 6
  • The full objective minimizes TP L(N, G, G) over partition boundary j, allocated GPU count m, and feasible device set g.Source: 2) Algorithm Formulation · p. 6
  • Each dynamic-programming transition adds a partition after layer j′, splitting the remaining stage into two stages with separate GPU allocations.Source: 2) Algorithm Formulation · p. 6
  • DAPPLE estimates each new subproblem’s latency while enumerating its three device-placement strategies.Source: 2) Algorithm Formulation · p. 6
  • The planner reuses the prior pivot stage when possible, avoiding iteration over all earlier stages during each transition.Source: 2) Algorithm Formulation · p. 6
  • Memorized search records the best split, replication, and placement for each point in the solution space.Source: 2) Algorithm Formulation · p. 6

D. Contributions over previous work

  • Unlike PipeDream’s asynchronous planning and GPipe’s block partitioning, DAPPLE explores uneven partitions and broader device assignments for synchronous pipelines.Source: D. Contributions over previous work · p. 6

1) Uneven Pipeline Partitioning with Fewer Stages:

DAPPLE improves synchronous pipeline planning by favoring fewer stages, slightly uneven partitions, and broader device assignments, then transforms planned graphs for replicated execution.

1) Uneven Pipeline Partitioning with Fewer Stages:

2) Cross Stage Communication:

DAPPLE combines replicated pipeline stages with micro-batch scheduling and split-concat communication to balance utilization, communication, and memory. Its scheduler interleaves forward and backward work to release memory early while preserving execution efficiency.

Cross-stage communication

  • DAPPLE replicates pipeline stages across devices, splitting micro-batches for balanced stage execution rather than assigning whole micro-batches round robin.Replication supports hybrid data and pipeline parallelism; split-concat operations connect adjacent stages for one-to-many, many-to-one, and many-to-many communication.Source: 2) Cross Stage Communication · p. 7

Weight updates

Micro-batch scheduling

  • The number of concurrently scheduled forward micro-batches trades off pipeline parallelism against peak memory consumption.More concurrent micro-batches can improve utilization but increase memory use; Ki captures the stage-specific schedule size.Source: C. Micro-batch Unit Scheduling · p. 7, F0 GPU0 · p. 8
  • DAPPLE schedules forward and backward stages in a deterministic round-robin order to release each pipeline task’s memory as early as possible.The schedule uses control dependencies to enforce micro-batch ordering and early backward execution.Source: C. Micro-batch Unit Scheduling · p. 7, F0 GPU0 · p. 8
  • Policy A uses Ki = min(S − i, D) for low ACR, whereas Policy B uses Ki = min(2 ∗ (S − i) − 1, D) when communication needs more overlap.ACR is the ratio between cross-stage communication latency and average forward/backward computation time.Source: F0 GPU0 · p. 8

VI. EVALUATION

DAPPLE is evaluated across six DNN models, three GPU interconnect environments, and fixed-device planning scenarios. The best strategy depends on model structure, memory, communication topology, and computation-communication balance.

Evaluation setup

  • DAPPLE evaluates image classification, machine translation, and language modeling across hierarchical NVLink plus Ethernet and flat 25 Gbps or 10 Gbps Ethernet environments.The hardware study includes 16 available devices for planning comparisons.Source: A. Experimental Setup · p. 8, B. Planning Results · p. 8

Planning representation

  • DAPPLE’s plans encode pipeline stage replication and placement, with cross-server replication favored less on hierarchical interconnects because it adds inter-server communication.A P : Q plan denotes two stages replicated on P and Q devices, respectively.Source: B. Planning Results · pp. 8, 10

Planning results

  • All evaluated models prefer data parallelism on a single Config-A server, where fast intra-server bandwidth handles gradient communication.The study reports bandwidth up to 130GB/s and gradient communication magnitudes up to 3.7GB.Source: B. Planning Results · p. 10
  • ResNet-50 consistently favors data parallelism because its 100MB model has high computation density, even on the 10 Gbps configuration.Gradient accumulation and computation/communication overlap allow DP to outperform pipelining in this case.Source: B. Planning Results · p. 10
  • VGG-19 uses data parallelism on Config-A and Config-B but favors a 15 : 1 pipeline on Config-C because its large final fully connected layer makes replicated gradient synchronization costly.The model has 548MB of parameters, with most weights concentrated in the final fully connected layer.Source: B. Planning Results · p. 10

C. Performance Analysis

Across five models at fixed global batch size 128, DAPPLE’s hybrid plans outperform overlapped data parallelism across all three hardware configurations. Model-specific gains reflect weight, activation, layer, and network characteristics.

Overall performance

  • 1.71X, 1.37X, and 1.79X averaged speedups are achieved by DAPPLE hybrid plans over overlapped DP on Config-A, Config-B, and Config-C, respectively.The comparison fixes GBS = 128 across five models.Source: C. Performance Analysis · p. 11

Model-specific analysis

  • 2.32X speedup is achieved for GNMT-16 on Config-C when DAPPLE’s hybrid approach is compared with the best data-parallel baseline.GNMT-16 uses a two-stage pipeline on faster networks and a straight pipeline at sufficiently large batch size on Config-C.Source: C. Performance Analysis · p. 11
  • VGG-19 benefits from splitting convolutional and fully connected layers because about 70% of weights reside in the final fully connected layer while activation size falls to 3MB.Replicating convolutional layers without the final fully connected layer reduces communication overhead on slow interconnects.Source: C. Performance Analysis · p. 11
  • BERT-48 and XLNet-36 favor DAPPLE across all configurations, with DAPPLE’s advantage over DP variants increasing as the network slows from 25 Gbps to 10 Gbps.The supplied analysis attributes this to communication and computation overlap opportunities and model memory differences.Source: C. Performance Analysis · p. 11
  • AmoebaNet-36 favors DAPPLE in all three configurations because its parameter and computation distributions require load-balanced staging while accounting for allreduce overhead.Its last third contains 73% of parameters, and lower-bandwidth networks shift the split toward larger layer IDs.Source: C. Performance Analysis · p. 11

D. Scheduling Policy

DAPPLE’s scheduling policy selects the number of warm-up micro-batches using communication-to-computation characteristics, then combines early backward execution with optional recomputation. This improves throughput while reducing or preserving memory usage relative to GPipe.

Scheduling policy

  • 10% and 31% throughput improvements result from choosing Policy B over Policy A for VGG-19 and GNMT-16, whose ACR values are relatively high.For BERT-48 and XLNet-36, low ACR produces little policy difference, so the smaller-memory Policy A is preferred.Source: D. Scheduling Policy · p. 11

DAPPLE versus GPipe

  • Both DAPPLE and GPipe save about 19% averaged peak memory but lose 20% throughput when recomputation is applied with M = 2.This comparison keeps the micro-batch count fixed and isolates the recomputation trade-off.Source: E. Comparison with GPipe · p. 12
  • DAPPLE’s early backward scheduling keeps peak memory independent of the number of micro-batches, enabling larger M and reducing pipeline bubbles.The comparison uses BERT-48 with a two-stage pipeline on Config-B.Source: E. Comparison with GPipe · p. 12
  • 1.6× higher throughput and 0.88× averaged peak memory are achieved by DAPPLE without recomputation at M = 16 compared with GPipe, which supports only two micro-batches.The throughput gain is attributed mainly to the lower bubble proportion from using more micro-batches.Source: E. Comparison with GPipe · pp. 11–12
  • 0.70× memory consumption is achieved by DAPPLE plus recomputation at M = 16 relative to baseline GPipe without recomputation.This combination can support larger micro-batch sizes or larger models.Source: E. Comparison with GPipe · p. 12

Comparison with PipeDream

  • DAPPLE strategies consistently outperform PipeDream planner strategies by up to 3.23× under synchronous training.Both planners use the same configurations and profiling data, and their strategies run with DAPPLE Runtime.Source: F. Comparison with PipeDream · p. 12

H. Weak Scaling

DAPPLE scales large-model training across GPUs by combining pipeline and data parallelism with planner and scheduler support. On 8 V100s with NVLink, it supports BERT models up to 5.5B parameters, with linear model-size scaling in this setting.

Weak Scaling

  • 5.5B parameters: DAPPLE supports this BERT model size on 8 V100s with NVLink.The increase is achieved by varying layer counts under reasonable input size with re-computation.Source: H. Weak Scaling · p. 12
  • Linear scaling: maximum BERT model size grows with layer count because encoder-layer parameters are balanced across devices.Longer pipelines introduce more bubbles and slightly reduce average GPU utilization.Source: H. Weak Scaling · p. 12
  • 3.23× speedup: DAPPLE’s planner outperforms PipeDream’s planner under synchronous training scenarios.The planner jointly considers pipeline partition, replication, and placement to generate hybrid data/pipeline-parallel strategies.Source: §VIII · p. 13
  • 1.6× throughput speedup and 12% memory savings: DAPPLE’s scheduler outperforms GPipe on both training throughput and memory consumption.The scheduler avoids storing multiple parameter versions and reduces reliance on costly re-computation.Source: §VIII · p. 13

Related papers

Loading 2007.01045v1…