Source-linked AI summary
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, Xin Liu
TL;DR
Training LLMs on more than 10,000 GPUs requires high efficiency and stability despite substantial communication, failures, and stragglers. MegaScale uses full-stack algorithm-system co-design and in-depth observability, achieving 55.2% MFU on a 175B model across 12,288 GPUs, 1.34× over Megatron-LM.
Problem
Training LLMs at tens of thousands of GPUs creates major efficiency and stability challenges because jobs are long-running and failures or stragglers can affect the entire job.
Method
MegaScale combines algorithm-system co-design across the training stack with in-depth observability, automated fault recovery, and diagnostic tools.
Results
55.2% MFU was achieved for a 175B LLM on 12,288 GPUs, a 1.34× improvement over Megatron-LM.
Takeaways & Limitations
MegaScale demonstrates production LLM training at more than 10,000 GPUs with high efficiency, fault recovery, and monitoring for large-scale anomalies.
Takeaways & Limitations
Some probabilistic hardware anomalies remain undetectable by automated self-checks and require custom monitoring and case-by-case analysis.
Abstract
from arXiv · showhide
We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.
1 Introduction
MegaScale addresses the efficiency and stability challenges of training LLMs across more than 10,000 GPUs through algorithm-system co-design and in-depth observability. It reaches 55.2% MFU on a 175B model using 12,288 GPUs, a 1.34× improvement over Megatron-LM.
- Challenges: Training LLMs at tens of thousands of GPUs creates unprecedented efficiency and stability challenges because communication, failures, and stragglers affect long-running jobs.MFU captures training efficiency, while failures and stragglers can be expensive and slow the entire job.
- Approach: MegaScale applies algorithm-system co-design and in-depth observability across the training stack.Its full-stack approach spans model and optimizer design, parallelism, communication overlap, data loading, networking, and diagnosis tools.
- Reliability: MegaScale uses diagnosis tools to identify root causes, detect anomalies, localize faults, and support recovery from failures and stragglers.The tools include heartbeat-based monitoring, diagnostic tests, checkpointing and recovery improvements, and distributed CUDA-event analysis.
- Results: 55.2% MFU was achieved when training a 175B transformer model on 12,288 GPUs, improving MFU by 1.34× over Megatron-LM.The system was deployed in datacenters and also recovered a production training process over 100 times during failures.
2 Background
LLM training distributes computation and model state across devices using data, pipeline, tensor, and combined 3D parallelism. These strategies trade memory and computation distribution against communication and pipeline scheduling requirements.
- Data parallelism: Data parallelism replicates model and optimizer states while dividing data across devices, whereas ZeRO2 shards optimizer states and gradients across data-parallel processes.ZeRO replaces gradient all-reduce with reduce-scatter and all-gather operations.
- Pipeline parallelism: Pipeline parallelism distributes model layers across devices and subdivides batches into micro-batches for pipelined execution.Megatron-LM uses interleaved 1F1B scheduling, which divides each pipeline stage into virtual stages called model chunks.
- Tensor parallelism: Tensor parallelism partitions individual operators across devices, requiring communication to split inputs and merge outputs.GEMMs in MLP and self-attention blocks can be split across GPUs to use more computational units.
- Combined parallelism: Data, pipeline, and tensor parallelism can be combined into 3D parallelism to scale LLM training across many GPUs.Tensor-parallel communication is preferably confined within a node, while data and pipeline parallelism are more suitable for inter-node communication.
3 Efficient Training at Scale
MegaScale improves large-scale LLM training efficiency through algorithm-system co-design, overlapping communication with computation, operator and data-pipeline optimization, and network tuning. These techniques address computation, communication, initialization, and pipeline inefficiencies at scale.
- Algorithmic Optimizations: Attention and MLP computation can execute in parallel with a parallel transformer block, reducing computation time without degrading model quality.The paper also adopts sliding window attention, whose O(s × w) complexity is lower than full attention's O(s×s) when w ≪s.
- Algorithmic Optimizations: 87.5% of pipeline bubbles are reduced by using LAMB to scale the batch size 4× without accuracy loss.With interleaved pipeline parallelism, training one step at 4× batch size has fewer bubbles than training four steps at 1× batch size.
- Communication Overlapping in 3D Parallelism: Communication overhead is hidden by overlapping all-gather, reduce-scatter, and pipeline send/receive operations with computation across 3D parallelism.Pipeline communication is decoupled and launched asynchronously, while tensor/sequence-parallel communication is optimized around the parallel transformer block.
- Data Pipeline: Data preprocessing is overlapped with gradient synchronization, while redundant dataloaders are eliminated to reduce GPU idle time and disk-bandwidth contention.These optimizations target preprocessing overhead that otherwise creates non-negligible GPU idle time at each training step.
- Network Performance Tuning: Initialization time falls below 5 seconds on 2048 GPUs and below 30 seconds on more than 10,000 GPUs after reducing global-barrier complexity from O(n^2) to O(n).The optimization redesigns communication-group initialization to minimize global barriers.
- Network Performance Tuning: MegaScale analyzes dependencies across 3D parallelism and tunes network performance to improve large-scale training efficiency.The system applies a full-stack algorithm-system co-design approach spanning model, communication, operators, data, and networking.
4 Fault Tolerance
MegaScale provides fault tolerance for large-scale LLM training through deep observability, automated diagnosis, checkpoint optimization, and recovery mechanisms. The system targets inevitable faults, stragglers, and long recovery paths in clusters exceeding 10,000 GPUs.
- Fault Tolerance: Automatic fault identification and fast recovery provide fault tolerance with minimal human intervention and negligible impact on ongoing training tasks.The robust training framework is designed for software and hardware faults that become virtually inevitable at very large scale.
- Data Collection and Analysis: Heartbeat messages combine executor, process, log, and RDMA information for real-time anomaly detection and early warnings.Abnormal RDMA traffic patterns can reveal issues that do not appear as explicit training errors.
- Fault Diagnosis: Lightweight self-check diagnostics cover a broad range of hardware and software faults while balancing execution time against false positives.The paper identifies a trade-off between diagnostic accuracy and preserving effective training time.
- Fast Checkpointing and Recovery: Two-stage checkpointing lets GPU workers resume almost immediately while asynchronous HDFS writes proceed in the background.Pinned memory and optimized serialization reduce the first stage to several seconds.
- Fast Checkpointing and Recovery: A single worker per data-parallel group reads each shared checkpoint partition from HDFS and broadcasts it to peers, reducing recovery-time bandwidth pressure.The strategy reduces repeated HDFS reads for workers sharing the same state partition.
5 Training Troubleshooting
MegaScale diagnoses large-scale training instability and efficiency variation with observability tools that expose slow machines, distributed dependencies, and faulty workers. These tools support targeted identification of stragglers and communication-related faults, while some probabilistic hardware anomalies still require custom monitoring.
- Hardware anomalies: Probabilistic hardware anomalies can evade machine self-checks and degrade training efficiency despite automatic handling of most common faults.MegaScale adds custom monitoring and analysis tools for case-by-case anomaly detection.
- Performance diagnosis: The performance heat-map aggregates forward and backward computation latency across devices and steps to reveal machine-level differences.Its color encodes code-segment running time on a rank.
- Performance diagnosis: Approximately 0.5% of machines are substantially slower, and the slowest machine determines overall training efficiency.Excluding these outliers makes peak MFU consistent across runs.
- Performance diagnosis: The unified trace aggregates events from distributed views onto one timeline, exposing execution order, pipeline bubbles, synchronization, and event dependencies.Selecting an event reveals its dependencies.
- Fault diagnosis: The 3D visualization maps workers across tensor, pipeline, and data parallelism, showing data flow, communication operations, and error messages for selected workers.This helps identify faulty nodes when communication blocking causes cascading timeouts.
6 Experience
MegaScale combines algorithm-system co-design, communication and computation optimizations, and deep observability to improve LLM training efficiency and stability at production scale. Experiments report higher MFU, sustained convergence, automated recovery, and diagnoses of stragglers and network faults.
- Training Performance: Up to 6.1% higher MFU and near-linear scalability came from MegaScale’s 3D-parallel communication overlapping on the 530B model.Megatron-LM’s MFU decreased by 1.6% with increasing stragglers and communication.
- Training Performance: 1.34× speedup over Megatron-LM was achieved for 175B-model strong scaling, with 55.2% MFU on 12,288 GPUs.MFU decreased from 59.1% to 55.2% as GPU count increased, while MegaScale still exceeded Megatron-LM by 14% MFU at the largest scale.
- Training Performance: 17.6% MFU improvement over the original Megatron-LM baseline was decomposed across algorithmic, communication, operator, data-pipeline, code, and optimizer optimizations.The baseline was 47.7% MFU on the 175B model with 256 GPUs.
- Model Convergence: Algorithmic techniques produced comparable convergence in microbenchmarks, and LAMB with four times the batch size matched ADAM after around 250B tokens.These observations motivated enabling all algorithmic optimizations in production training.
- Model Convergence and Stability: A production run on more than 10,000 GPUs trained for several weeks, restarted over 100 times, and continued to converge.Over 90% of software and hardware faults were automatically identified and fixed.
- Fault Tolerance: Fault diagnosis took less than 10 minutes, recovery reached pre-crash progress within 15 minutes, and effective training time remained above 90%.The effective training time rate is defined using iteration count, iteration training time, and total training time.
- Problems Discovered and Fixed: Removing problematic hosts that took approximately 10% longer on forward computations improved MFU by about 0.7%.The repeated behavior across experiments indicated a machine-specific rather than software-specific issue.
- Problems Discovered and Fixed: Explicitly increasing NCCL timeout thresholds and improving network-component signal quality reduced training stalls from network-interface flapping.The issue involved links among network cards, AOC cables, and switches.
7 Related Work
Related work covers LLM pre-training and efficiency optimization, datacenter diagnosis, and distributed fault tolerance. MegaScale addresses an identified gap by emphasizing the system infrastructure and operational experience behind large-scale LLM training.
- LLM Training: Existing LLM technical reports predominantly compare model performance rather than detailing the infrastructure enabling large-scale training.The paper positions its system-design and deployment account as filling this gap.
- LLM Training: Fine-tuning generally requires substantially less computation and data than pre-training, with quantization and low-rank adaptation reducing resource needs further.The passage contrasts downstream adaptation with large-scale pre-training.
- LLM Optimizations: Other LLM-efficiency research explores sparse or linear attention and architectures such as RWKV and RetNet to reduce memory or improve efficiency.
- Diagnosis Tools in Datacenters: Datacenter diagnosis systems use active probing and network-wide analysis to detect problems such as packet black holes and silent drops.Examples include Pingmesh, EverFlow, LossRadar, and NetBouncer.
- Fault Tolerance: Distributed fault-tolerance work includes reactive techniques such as retry, replication, checkpointing, and message logging, alongside approaches such as preemptive migration and load balancing.The cited approaches target different failure and deployment scenarios.
- Fault Tolerance: Preemptive migration and load balancing often assume predictable failures, whereas real large-scale systems make failure prediction difficult because of system complexity.
8 Conclusion
MegaScale is a production-grade LLM training system for more than 10,000 GPUs that combines algorithm-system co-design with deep observability. It reports 55.2% MFU on a 175B model using 12,288 GPUs, a 1.34× improvement over Megatron-LM, alongside automated fault handling and diagnostic tools.
- Conclusion: MegaScale provides a production-grade system for training LLMs at a scale exceeding 10,000 GPUs.
- Conclusion: 55.2% MFU on a 175B LLM model using 12,288 GPUs represented a 1.34× improvement over Megatron-LM.
- Conclusion: The system combines algorithm-system co-design, tailored automatic fault recovery, and monitoring tools for deep observability and root-cause identification.