Source-linked AI summary
Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies
Felix Brakel, Uraz Odyurt, Ana-Lucia Varbanescu
TL;DR
Growing neural networks create compute and memory demands that increasingly exceed individual devices, motivating model parallelism. This survey synthesizes parallelisation types, implementation and search challenges, and modern Transformer use-cases. It identifies intra-operator and inter-operator parallelism as the core types, documents their distinct trade-offs, and finds their use in multi-billion-parameter Transformer systems, subject to sparse implementation evidence.
Problem
Neural networks increasingly exceed single-device compute and memory capabilities, creating a need to understand model-parallelism types, challenges, and modern uses.
Source: §1 · p. 1Method
The survey represents neural networks as operator graphs, reviews DNN auto-parallelisation, and snowballs literature on modern Transformer implementations.
Source: §3.1.3 · p. 3, §6 · p. 8Results
Model parallelism comprises intra-operator and inter-operator strategies, with hybrid combinations; multi-billion-parameter Transformers use these strategies in differing hardware configurations.
Source: §7 · p. 9Takeaways & Limitations
Intra-operator parallelism has high communication requirements, inter-operator parallelism can underutilise devices, and effective hybrid partitioning remains difficult.
Source: §7 · p. 9The modern-use-case analysis is constrained by scarce implementation details, and inter-operator training pipelines can suffer frequent stalls.
Source: §4 · pp. 4–5
Abstract
from arXiv · showhide
Neural networks have become a cornerstone of machine learning. As the trend for these to get more and more complex continues, so does the underlying hardware and software infrastructure for training and deployment. In this survey we answer three research questions: "What types of model parallelism exist?", "What are the challenges of model parallelism?", and "What is a modern use-case of model parallelism?" We answer the first question by looking at how neural networks can be parallelised and expressing these as operator graphs while exploring the available dimensions. The dimensions along which neural networks can be parallelised are intra-operator and inter-operator. We answer the second question by collecting and listing both implementation challenges for the types of parallelism, as well as the problem of optimally partitioning the operator graph. We answer the last question by collecting and listing how parallelism is applied in modern multi-billion parameter transformer networks, to the extend that this is possible with the limited information shared about these networks.
1 INTRODUCTION
The survey frames model parallelism as a response to neural networks’ growing compute and memory demands, while noting that distributing data-intensive workloads creates severe communication challenges. It investigates the types, challenges, and modern applications of model parallelism.
1 INTRODUCTION
- Neural networks’ increasing complexity raises both computational requirements and memory footprint for training and inference.More layers, neurons, parameters, and intermediate values increase operations and storage needs.Source: §1 · p. 1
- Model parallelism partitions a model across multiple devices to distribute workload, but large parameters and activations must cross device interconnects.Source: §1 · p. 1
- Interconnect bandwidth limits become especially severe beyond a single node, where Ethernet can be far slower than device memory or NVLink.Even NVLink bandwidth is reported as half the A100’s DRAM bandwidth.Source: §1 · p. 1
- The survey asks which model-parallelism types exist, what challenges they pose, and how they are used in modern systems.Source: §1 · p. 1
2 STUDY DESIGN
The study uses two snowballing-based literature reviews: one develops a theoretical framework from DNN auto-parallelisation, while the other examines model parallelism in modern Transformer networks. Sparse implementation details narrow the included use-case evidence.
2 STUDY DESIGN
- The first phase studies DNN auto-parallelisation to formulate model parallelism for search algorithms and answer which parallelism types exist.Source: §2 · p. 1
- Filtering the first-phase literature by code availability, maintenance, automation, and file-format compatibility reduced the collection to progressively smaller subsets.The reported counts are 22 papers with available code, 8 actively maintained, 7 fully automated, and 6 compatible with existing file formats.Source: §2 · p. 1
- The second phase snowballed from models in Figure 1 and Table 1 to study model parallelism in modern Transformer networks.Source: §2 · p. 2, Figure 1 · p. 2, Table 1 · p. 2
- Limited implementation detail reduced the second-phase evidence to four Megatron papers, one Gopher paper, and one PaLM paper.The initial collection also included two PaLM and two GPT papers, but papers were filtered for available model-parallelism details.Source: §2 · p. 2
3 MODEL PARALLELISM
The survey represents neural-network computation with operator graphs and distinguishes model parallelism by the dimensions exploited within or across operators. It connects these categories to hardware limits, workload dependencies, and strategy-selection methods.
3 MODEL PARALLELISM
- Model parallelism distributes model partitions across devices to address compute and memory limits, whereas data parallelism exploits parallelism in the data.Source: §3 · p. 2, §3.1.3 · p. 3
3.1.1 Scaling up networks.
- Scaling networks increases compute through more operations, memory through more parameters, and training cost through additional samples and passes.These effects make hardware limitations a barrier to continued scaling.Source: §3.1.1 · p. 2
3.1.2 Neural network workloads.
- An operator graph represents tensors as data and operators as computations, with edges describing tensor inputs to operator nodes.The survey uses this representation to model fully connected layers and forward and backward workloads.Source: §3.1.2 · p. 3
- Training requires forward and backward passes, while inference requires only forward passes; backward computation depends on forward outputs and stored activations.Source: §3.1.2 · p. 3
3.1.3 Pushing the limits of hardware.
- Checkpointing, algebraic transformations, lower precision, pruning, and distillation trade memory, computation, accuracy, or model size, and can complement model parallelism.Algebraic optimisations can potentially hurt parallelism, so these techniques are not mutually independent.Source: §3.1.3 · p. 3
3.2 Types of model parallelism
- Intra-operator parallelism partitions computation within an operator, while inter-operator parallelism partitions across multiple operator nodes; hybrid strategies combine them and may include data parallelism.Inter-operator strategies generally communicate at sub-graph boundaries, whereas intra-operator strategies are operator-specific.Source: §3.2 · p. 3, Figure 3 · p. 4
- Choosing a strategy depends on model architecture and device topology, motivating both hardware-specific ad hoc methods and general methods for varied models and devices.Intra-operator and inter-operator parallelism are also called tensor and pipeline parallelism, respectively, in some work.Source: §3.2 · p. 4
4 CHALLENGES
Model-parallel training faces search and implementation challenges: finding good strategies is difficult, while inter-operator and intra-operator approaches impose different utilization and communication costs.
Inter-operator parallelism
- Inter-operator parallelism can suffer low device utilization because dependencies between partitions create pipeline stalls during training.Backward computation also depends on the completed forward pass, contributing to pipeline bubbles.Source: §4 · p. 4
Intra-operator parallelism
- Intra-operator parallelism requires scattering each input tensor across devices and gathering outputs for every batch, creating extreme communication requirements.Source: §4 · p. 4
Combining parallelism types
- Hybrid strategies combine parallelism types because each form has limitations, but exploring these strategies remains difficult when frameworks provide simple or suboptimal support.Higher-level programming models and fully automated frameworks are alternative ways to express or discover parallelization strategies [34] [11].Source: §4 · pp. 4–5
Search-space
- Defining the search space requires covering many parallelizable dimensions while excluding illegal or suboptimal strategies.Source: §4 · p. 5
Strategy evaluation
Search method
- Selecting effective methods to traverse the model-parallel search space is itself challenging because proposed approaches have diverged across many directions.Source: §4 · p. 5
5 USE-CASES
The survey reviews model-parallel implementations for modern Transformer models, emphasizing how intra-operator, inter-operator, and combined strategies address scale, memory, and communication constraints. Its case studies span Megatron, Gopher, PaLM, and GPT, although implementation details are unevenly available.
5.1 Megatron
- Megatron partitions MLP weights by columns or rows, selecting the strategy that avoids reconstructing the nonlinear GeLU(XA) operation.Row partitioning requires communicating a partial product before GeLU, whereas the selected approach computes GeLU without reconstructing the full intermediate.Source: §5.1 · p. 6
- An 8.9B GPT-2 Transformer reaches 77% throughput scaling on eight GPUs by distributing self-attention Q, K, and V matrices over columns.The cited design exploits the self-attention block’s inherent parallelism.Source: §5.1 · p. 6
- Up to 1T-parameter models are fitted on A100 GPUs by combining 8-way intra-layer and up to 64-way inter-layer parallelism, with data parallelism achieving 52% hardware utilisation.The analysis recommends tensor parallelism within multi-GPU servers, pipeline parallelism across servers, and choosing model-parallel size so parameters and metadata fit GPU memory.Source: §5.1 · p. 6
- A combined NVIDIA-Microsoft system uses data, inter-layer, and intra-layer parallelism to train a 540B-parameter model on 420 DGX A100 servers.The implementation combines Megatron and DeepSpeed frameworks.Source: §5.1 · p. 6
- Four Transformer models spanning 22B to 1T parameters achieve 41.5% to 56.3% hardware utilisation on 8 and 512 GPUs, respectively, without data parallelism.The approach focuses on reducing activation memory and increasing parallelism through tensor parallelism and partitioning activations along the sequence dimension.Source: §5.1 · pp. 6–7
5.2 Gopher
- Gopher uses four-way inter-layer parallelism across four TPU pods, plus unreported intra-layer and data parallelism, to train a model with up to 280B parameters.Its hardware and software are custom TPU and JAX systems.Source: §5.2 · p. 7
5.3 PaLM
- PaLM trains a 540B-parameter model without inter-layer parallelism, using up to 12-way intra-layer and 256-way data parallelism within TPU pods.Avoiding inter-layer parallelism avoids the pipeline bubble problem during training.Source: §5.3 · p. 7
- Details for PaLM 2, GPT-3, and GPT-4 infrastructure are unavailable, limiting the survey’s comparison of modern model-parallel implementations.The survey includes these models for completeness but reports insufficient implementation information.Source: Table 2 · p. 6, §5.3 · p. 7, §5.4 · p. 7
6 DISCUSSION
The discussion surveys auto-parallelisation strategies and modern model-parallel systems, while emphasizing difficult comparisons caused by divergent search spaces, hardware, and evaluation practices.
Modern Transformer use-cases
- Intra-operator parallelism requires high-speed interconnects, with Megatron using 8-way NVLink parallelism and Gopher/PaLM hardware supporting up to 12-way parallelism.NVLink-based Megatron configurations are limited to a single compute node, whereas specialized clusters support wider intra-operator strategies.Source: §6 · p. 8
Evaluation practices
- Reported metrics remain difficult to interpret because hardware FLOPs can increase through memory-saving techniques without improving throughput, motivating MFU over HFU.MFU compares actual Transformer throughput with the system’s theoretical maximum.Source: §6 · p. 8
Modern Transformer use-cases
- At least two parallelism forms are used in Megatron-family systems, while the largest GPU deployments employ all three forms.Google’s TPU-based systems reduce inter-operator parallelism but still rely heavily on data parallelism when scaling to thousands of TPUs.Source: §6 · p. 8
Framework comparisons
- Alpa matches Megatron-LM and outperforms DeepSpeed, although comparisons with FlexFlow and Tofu were unavailable because of operator and code-release constraints.The comparison is therefore informative but incomplete across general-purpose frameworks.Source: §6 · p. 8
Comparison limitations
- Standardised testing is lacking, making framework metrics and search methods difficult to compare because each approach defines its search space differently.The survey consequently supplements quantitative comparison with qualitative analysis of partitioning and search strategies.Source: §6 · p. 8
Partitioning strategies
- FTPipe Mixed-pipe fits a 3B-parameter model on eight RTX 2080-Ti GPUs, whereas PipeDream produced no valid strategy on that setup.The strategies were calibrated using isolated operator profiling and a cost model.Source: §6 · p. 8
Auto-parallelisation frameworks
- Alpa uses integer linear programming for intra-layer strategies and dynamic programming to partition the operator graph across a device mesh.Its hierarchical search reflects distinct granularities and communication requirements for intra-layer and inter-layer parallelism.Source: §6 · p. 8
- FlexFlow searches SOAP strategies across Sample, Operator, Attribute, and Parameter dimensions, using a simulator and Markov Chain Monte Carlo search.Its simulator combines device and operator graphs with profiling and communication estimates to predict total execution time.Source: §6 · pp. 8–9
7 CONCLUSION
The conclusion identifies intra-operator and inter-operator parallelism, their hybrid combinations and trade-offs, and their use in multi-billion-parameter Transformer systems. It also calls for standardisation to make auto-parallelisation methods more comparable.
Types of model parallelism
- Model parallelism comprises intra-operator and inter-operator parallelism, often combined as hybrid parallelism with data parallelism.Source: §7 · p. 9
Challenges
- Intra-operator parallelism has extremely high communication requirements, while inter-operator parallelism can suffer low device utilisation during training.Source: §7 · p. 9
- Hybrid parallelism is difficult to optimise because operator graphs and device graphs may not map adequately onto each other.Source: §7 · p. 9
Modern use-case
- Multi-billion-parameter Transformers use model parallelism in training and inference, with Megatron combining intra-operator, inter-operator, and data parallelism across nodes.Source: §7 · p. 9
- PaLM uses specialised hardware to address intra-operator communication and does not use inter-operator parallelism.Source: §7 · p. 9
Future work
- The survey proposes standardised strategy, device, and model representations, plus fully explored search-space datasets, to enable broader comparison without expensive hardware.Source: §7 · p. 10