Source-linked AI summary
Characterization of Large Language Model Development in the Datacenter
Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, Tianwei Zhang
TL;DR
Efficient LLM development on large GPU clusters is difficult because failures, complex workloads, and imbalanced resource use challenge existing DL-workload assumptions. The paper characterizes six months of Acme traces, then develops fault-tolerant pretraining and decoupled evaluation scheduling. It reports severe workload imbalance and improvements of up to 1.8× in evaluation makespan.
Problem
LLM development requires costly large-scale GPU infrastructure, while prior task-specific DL workload findings do not fully capture its failures, resource imbalance, and distinctive execution patterns.
Method
The paper analyzes six months of Acme workload, monitoring, failure, and profiling traces and introduces fault-tolerant pretraining plus decoupled scheduling for evaluation.
Results
Pretraining uses 94.0% of Kalos GPU time despite representing 3.2% of jobs, while decoupled evaluation reduces evaluation makespan by up to 1.8×.
Takeaways & Limitations
LLM-specific characterization exposes resource and reliability challenges and supports systems tailored to pretraining robustness and timely evaluation feedback.
Takeaways & Limitations
The analysis excludes serving workloads, emphasizes GPU jobs, and primarily characterizes transformer-based decoder-only models.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization.
1 Introduction
LLM development requires large GPU infrastructure but remains difficult to operate efficiently because of failures, complex parallelization, unstable training, and queuing delays. Acme’s six-month characterization identifies distinctive workload imbalances and motivates fault-tolerant pretraining and decoupled evaluation scheduling.
- LLM development requires thousands of accelerators and faces hardware failures, intricate parallelization, unstable training, and long queuing delays.
- Existing analyses of task-specific DL workloads do not fully apply because LLMs self-supervise on broad data to build foundation models before downstream adaptation.
- Acme’s study analyzes six months of scheduler, monitoring, failure, and profiling traces from two LLM clusters containing 4,704 A100 GPUs.
- Characterization findings: Pretraining jobs comprise 3.2% of Kalos jobs but consume 94.0% of GPU time, while evaluation comprises 92.9% of jobs but uses 0.8%.
- System efforts: The paper develops fault-tolerant pretraining and decoupled evaluation scheduling to address failures, storage contention, GPU idleness, and delayed model-quality feedback.
- System efforts: Checkpoint acceleration reaches 3.6∼58.7×, while decoupled evaluation reduces evaluation makespan by up to 1.8×.
2 Background
The paper frames LLM development as a multi-stage, resource-intensive pipeline and introduces Acme’s dedicated GPU clusters, scheduling infrastructure, and six-month trace methodology. Its background contrasts LLM workloads with prior heterogeneous DL traces and defines the operational setting for characterization.
- LLM development pipeline: LLMs self-supervise on broad data to form foundation models and later adapt them to diverse downstream tasks.
- LLM development pipeline: Pretraining consumes most resources in the development workflow and requires system innovations such as state-sharding optimizers and careful model placement.
- Acme overview: Acme’s Seren and Kalos clusters contain 2,288 and 2,416 GPUs, respectively, using NVIDIA A100 GPUs and high-speed interconnects.
- Acme overview: Scheduling is built on Slurm for Seren and Kubernetes for Kalos, with resource isolation, quota reservation, and best-effort jobs.
- Trace methodology: Prior trace-analysis datacenters mix general DL workloads, whereas Acme is dedicated to LLM development.
- Trace methodology: The study analyzes six months of traces from two LLM clusters, including scheduler job logs and execution information.
3 Datacenter Characterization
The characterization contrasts LLM-development clusters with prior DL workloads, finding shorter jobs, polarized utilization, highly skewed resource demand, and substantial underutilization outside GPUs. Evaluation jobs create an unusual queuing pattern, while GPUs dominate datacenter power consumption.
- Scope: Acme analysis compares workload distributions, job types, resource utilization, and environmental impacts across LLM-development clusters.The study covers Seren and Kalos and examines both workload behavior and infrastructure consumption.
- LLMs versus prior DL workloads: 2 minutes is the median job duration in both Seren and Kalos, 1.7–7.2× shorter than median durations in other clusters.The comparison uses GPU job runtime excluding queuing delay.
- LLMs versus prior DL workloads: 97% and 99% are the median GPU utilizations in Seren and Kalos, versus 48% and 4% in Philly and PAI.Acme workloads show a polarized distribution concentrated near 0% and 100% utilization.
- Workload categories: Pretraining jobs consume 69.5% and 94.0% of total GPU time in Seren and Kalos while representing only 0.9% and 3.2% of job counts.Evaluation jobs are numerous but consume only 0.8% of resources in Kalos.
- Workload categories: Evaluation jobs have the longest queuing delays despite their lowest GPU demands and shortest durations.Most resources are reserved for pretraining, leaving evaluation jobs to lower-priority scheduling on limited spare capacity.
- Infrastructure: CPU memory utilization remains below 50%, CPUs are typically underutilized, and NICs remain idle for over 60% of the time.The analysis attributes these patterns to the infrastructure surrounding compute-intensive LLM workloads.
- Environmental impact: Around 30% of GPUs are idle yet consume 60W, while GPUs account for approximately two-thirds of GPU-server power consumption.GPU servers consume 5× the power of CPU servers on average.
4 Workload Profiling
Fine-grained profiling focuses on pretraining and evaluation, the most resource-intensive or numerous workload types. The analysis examines evolving pretraining strategies, memory and pipeline behavior, and evaluation bottlenecks that delay model-quality feedback.
- Scope: The profiling analyzes pretraining and evaluation because they are the most resource-intensive or quantity-intensive workloads.The section combines fine-grained resource-utilization analysis with representative-task profiling.
- Pretraining: InternEvo’s pretraining framework continuously refines parallelism and memory-management strategies to improve training efficiency.The initial version uses 3D parallelism and a hierarchical ZeRO mechanism.
- Pretraining: Communication on the critical path limits 3D-parallelism utilization, while a refined strategy achieves around 16% acceleration.Pipeline bubbles from hybrid parallelism contribute to the lower utilization.
- Pretraining: Activation recomputation is used to improve efficiency across different pipeline-parallel ranks.The profiling considers rank-level behavior under pipeline parallelism.
- Evaluation: Evaluation jobs dominate job count because they compute metrics across different LLM benchmark datasets and are needed to guide checkpoint evolution.The workflow aims to provide swift feedback for tracking training progress and selecting checkpoints.
5 Failure Analysis
The failure analysis examines runtime and hardware-monitor logs across Acme’s clusters, showing that infrastructure failures dominate GPU-time impact while overheating and auxiliary services create additional disruptions. Manual recovery remains costly because failures can roll training back to checkpoints and require developer intervention.
- Failure impact: Infrastructure failures account for over 82% of GPU time despite representing only 11% of failed jobs.These failures commonly affect long-running pretraining jobs through GPU, NVLink, and network errors, and require substantial diagnostic and restart effort.
- Failure causes: Training 7B models in Kalos can overheat GPUs, contributing to NVLinkError or ECCError incidents.The cluster server-room temperature increased by approximately 5°C during these workloads, and failure frequency fell after cooling improvements.
- Failure causes: Auxiliary services for metrics, logging, monitoring, and alerting induce ConnectionError and NetworkError incidents that can slow or disrupt training.These services are vulnerable to network instability and can introduce timeouts into the pretraining process.
- Failure impact: Only 6.7% of evaluation tasks encounter errors in Kalos, with no recorded GPU or NVLink failures.Their short duration may reduce stress on GPUs and NVLink connections.
- Recovery burden: Jobs are restarted after job errors, loss spikes, or stuck training, reverting to the last checkpoint and losing intervening progress.Because existing LLM frameworks lack automatic recovery, developers commonly restart interrupted jobs manually and remain on call.
- Recovery burden: The 123B training run became more stable after smaller checkpoint intervals and graceful termination, but interrupted jobs still required rapid restarts.Compared with the earlier 104B run, the improved process incurred fewer rollback losses while preserving current results before termination.
6 Deployed LLM Systems
The deployed systems target two operational bottlenecks in LLM development: unreliable pretraining and delayed evaluation feedback. Fault-tolerant pretraining combines checkpointing, diagnosis, and recovery, while decoupled evaluation scheduling decomposes trials to improve utilization and completion time.
- 6.1 Fault-tolerant Pretraining: Fault-tolerant pretraining combines frequent checkpointing, failure diagnosis, and automatic recovery from properly saved checkpoints.The design uses asynchronous checkpointing, heuristic rules with LLM assistance, and a detection toolkit for fault-node identification and restart.
- 6.1 Fault-tolerant Pretraining: Asynchronous checkpointing separates model-state saving from training by buffering states in CPU memory and writing them to remote storage on a separate thread.This reduces checkpointing overhead for TB-scale model states, which can otherwise slow training by up to 43%.
- 6.1 Fault-tolerant Pretraining: The diagnosis pipeline combines rule-based analysis with LLM-based log processing to identify failure causes automatically.The system compresses real-time logs, isolates critical errors, and applies predefined rules before escalating unresolved cases to LLM-assisted diagnosis.
- 6.1 Fault-tolerant Pretraining: Real-time log compression dynamically updates filtering rules and reuses them across repetitive or similar tasks.A Log Agent identifies fixed log patterns, writes regular expressions, and forwards error messages for diagnosis; self-consistency improves result robustness.
- 6.2 Decoupled Scheduling for Evaluation: Decoupled evaluation scheduling decomposes trials, balances GPU workloads with prior information, and prioritizes lengthy CPU metric computations.The coordinator uses round-robin allocation over sorted queues to increase GPU occupancy, overlap computation, and reduce trial-switch overhead.
- 6.2 Decoupled Scheduling for Evaluation: The trial coordinator reduces evaluation makespan by 1.3× on one node and 1.8× on four nodes for a 7B model evaluated across 63 datasets.The measurements compare limited-resource and relatively ample-resource settings.
7 Discussion
The study’s scope is limited to selected LLM development workloads and architectures in Acme. It excludes serving, emphasizes GPU jobs, and mainly characterizes transformer-based decoder-only models.
- Scope Limitations: The analysis excludes serving workloads and provides limited coverage of CPU jobs.Its focus is the developmental process before model serving, with predominantly GPU-based analysis.
- Scope Limitations: The characterization primarily covers transformer-based, decoder-only architectures such as GPT-3 and LLaMA 2.Mixture-of-Experts receives only a simple appendix characterization, while multimodal LLMs fall outside the analysis scope.
- Scope Limitations: The systems may not suffice for future workloads involving long-sequence pretraining, MoE pretraining, and efficient RLHF.The authors describe ongoing refinement to accommodate these advancing workload demands.
8 Conclusion
The paper characterizes LLM workloads and resource utilization in Acme, identifies distinctive development challenges, and reports system efforts for pretraining and evaluation. It presents these observations and lessons as resources for subsequent LLM-system and GPU-cluster research.
- Conclusion: The study reveals distinctive LLM-development features and challenges, including resource inefficiencies and substantial failure impacts.It uses the characterization to identify opportunities for systems tailored to LLM workloads.
- Conclusion: The paper introduces efforts addressing both pretraining and evaluation workloads.These efforts target development robustness and timely performance feedback within the studied operational setting.
- Conclusion: The authors make their observations, insights, system code, and models available to support subsequent research and GPU-cluster management.The conclusion frames the lessons as broadly applicable rather than limited to a single operational incident.
A.1 Job Final Statuses
Only approximately 20–30% of resources are consumed by jobs that ultimately complete, while failures account for about 40% of jobs and 10% of GPU resources.
- Approximately 20–30% of resources are consumed by jobs that finally complete.
- About 40% of jobs fail while consuming 10% of GPU resources.Failures predominantly occur during the early stages of execution.
A.2 Host Memory
Pretraining nodes use a modest share of total host memory, but memory demand varies substantially with data loading and checkpointing configurations. Other system components also consume notable memory.
- 123GB of the 1TB host-memory capacity is used in the illustrated pretraining configuration.
- Megatron-LM can require substantially more dataloader memory because it loads metadata for the entire dataset.On-the-fly data loading is more memory-efficient without obviously impacting throughput.
- Asynchronous checkpointing memory requirements depend largely on model size and training configuration.
- TensorBoard uses 6.5GB, while the distributed-file-system client and critical components use 45.3GB.
A.3 Carbon Emission
Acme combines moderate carbon-free energy use with measured datacenter efficiency and emissions, including 673 MWh consumed by Seren in May 2023.
- 673 MWh of electricity consumption by Seren in May 2023 corresponds to 321.7 tCO2e of effective emissions.
- Acme has a PUE of 1.25 and obtains approximately 30.61% of its energy from carbon-free sources.The carbon-free share includes renewable sources such as solar and wind.
- GPU memory temperatures are generally higher than GPU core temperatures, with some heavily loaded GPUs exceeding 65°C.The paper identifies cooling-system enhancement as necessary to address elevated temperatures.
A.6 MoE Model
The MoE model has much lower GPU utilization than the dense model because frequent all-to-all communication is poorly handled by the single-IB-NIC server configuration.
- The MoE model presents much lower GPU utilization than the dense model.
- Frequent all-to-all communication and insufficient internode communication capacity explain the MoE utilization gap.The single IB NIC server cannot efficiently handle this communication pattern.
- The MoE comparison uses the official training configuration released by Mistral.