Source-linked AI summary
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, Fan Yang
TL;DR
Shared GPU clusters must manage coarse-grained resources and gang-scheduled DNN jobs while preserving locality and handling failures. This paper analyzes a two-month production trace of multi-tenant training workloads, finding low effective GPU utilization and substantial unsuccessful execution, and proposes scheduler guidelines based on these findings.
Problem
There was no systematic study of multi-tenant GPU clusters used to train machine-learning models, despite their growing enterprise use and distinctive sharing and scheduling constraints.
Method
The paper correlates scheduler, GPU-utilization, and individual-job logs from a two-month trace to characterize queuing, utilization, and failures in Philly.
Results
GPU hardware utilization averages 52% among GPUs in use, while around 30% of jobs are killed or finish unsuccessfully; locality, colocation, and stack-wide failures shape these outcomes.
Takeaways & Limitations
Schedulers should balance locality against queueing, isolate interfering jobs while defragmenting resources, detect failures early, and adapt retry policies at runtime.
Takeaways & Limitations
Aggregate-workload utilization lacks a clear baseline because model types and batch sizes vary across jobs.
Abstract
from arXiv · showhide
With widespread advances in machine learning, a number of large enterprises are beginning to incorporate machine learning models across a number of products. These models are typically trained on shared, multi-tenant GPU clusters. Similar to existing cluster computing workloads, scheduling frameworks aim to provide features like high efficiency, resource isolation, fair sharing across users, etc. However Deep Neural Network (DNN) based workloads, predominantly trained on GPUs, differ in two significant ways from traditional big data analytics workloads. First, from a cluster utilization perspective, GPUs represent a monolithic resource that cannot be shared at a fine granularity across users. Second, from a workload perspective, deep learning frameworks require gang scheduling reducing the flexibility of scheduling and making the jobs themselves inelastic to failures at runtime. In this paper we present a detailed workload characterization of a two-month long trace from a multi-tenant GPU cluster in a large enterprise. By correlating scheduler logs with logs from individual jobs, we study three distinct issues that affect cluster utilization for DNN training workloads on multi-tenant clusters: (1) the effect of gang scheduling and locality constraints on queuing, (2) the effect of locality on GPU utilization, and (3) failures during training. Based on our experience running a large-scale operation, we provide design guidelines pertaining to next-generation cluster schedulers for DNN training workloads.
1 Introduction
DNN training on shared GPU clusters introduces coarse-grained GPU allocation and gang scheduling, creating distinctive utilization and queuing challenges. A two-month production study characterizes these effects, failures, and scheduler guidelines.
- Challenges: GPUs lack fine-grained sharing support, while multi-GPU training frameworks require simultaneous allocation, increasing fragmentation and low utilization.Software sharing mechanisms often impose high overhead, and gang scheduling reduces scheduling flexibility.
- Study scope: The study analyzes a two-month trace of around 100,000 jobs from hundreds of users by combining scheduler, utilization, and job logs.The analysis examines queuing, execution, and failure behavior in a production multi-tenant cluster.
- Queuing: Relaxing locality constraints reduces queueing delays, especially for jobs using many GPUs.Philly waits for suitable GPU availability because locality improves distributed-training performance through faster interconnects.
- Utilization: 52% average hardware utilization among GPUs in use reflects synchronization overhead from poor placement and interference from colocated jobs.Cluster allocation alone can overstate effective GPU utilization.
- Failures: 30% of jobs are killed or finish unsuccessfully, with programming errors appearing early and infrastructure failures such as HDFS failures occurring later.The paper characterizes failures across the software and systems stack.
- Guidelines: Schedulers should trade queueing delay for locality, isolate jobs on dedicated servers, detect errors early, and adapt retries using runtime failure analysis.Migration can help defragment resources, while small dedicated pools or single-GPU runs can expose simple multi-GPU errors.
2 Philly: System Overview
Philly is a centralized system for supervised machine-learning jobs that schedules GPU resources with gang and locality awareness. Its workflow spans submission, placement, execution, progress tracking, and completion across shared GPU clusters.
- Workloads: Philly supports supervised-learning jobs using frameworks including TensorFlow, CNTK, Caffe, and PyTorch, with architectures such as CNNs, LSTMs, and RNNs.Jobs learn models from labeled training data and use iterative optimization methods such as SGD.
- Workloads: Distributed training replicates the model across workers, which process data subsets and synchronize gradients after each iteration.Synchronization uses parameter servers or high-performance networking mechanisms.
- Cluster architecture: The cluster grew 5× in one year and expanded from 2-GPU to 8-GPU servers.Philly operates large GPU clusters shared across multiple company groups.
- Cluster architecture: Workers are placed with awareness of GPU and network locality, favoring same-machine or same-RDMA-domain communication over slower cross-rack traffic.Within-rack links use 100-Gbps RDMA, while cross-rack traffic uses Ethernet.
- Scheduling workflow: Jobs request one or more GPUs, are queued until allocation, and run one container per allocated machine under a YARN-based resource manager.Virtual clusters support multiple production groups, and preemption begins when 90% of GPUs are used.
- Scheduling workflow: Gang scheduling requires all GPUs simultaneously, while locality-aware packing reduces parameter-synchronization time but can increase queueing.Unfulfilled requests relinquish partial resources and retry after a two-minute back-off; locality constraints are eventually relaxed.
- Placement and utilization: Packing smaller jobs reduces fragmentation, but colocating jobs can lower GPU utilization through contention for shared resources such as the PCIe bus.Running jobs receive dedicated GPUs to maintain consistent worker performance and avoid stragglers.
- Completion: Jobs finish as passed, killed, or unsuccessful, and failed jobs are retried a fixed number of times.Successful jobs typically run a user-set number of iterations to improve model quality incrementally.
3 Impact of Locality Awareness
Locality-aware scheduling creates a trade-off between queueing delay and GPU efficiency in distributed DNN training. The analysis finds that fragmentation dominates waiting time, while distribution and colocation reduce utilization and training performance.
- Queueing delays: 25% of VC2 jobs using >4 GPUs experience at least 10 minutes of queueing delay, versus 10% of 1-GPU jobs.Jobs using more GPUs have a longer queueing-delay tail.
- Queueing delays: Jobs distributed across more servers start sooner but sacrifice locality, confirming the scheduler trades locality for lower scheduling delay.Jobs running on 16 servers begin execution much sooner than those running on 2 or 4 servers.
- Queueing delays: Fragmentation delay accounts for 74.2% of occurrences for 5–8-GPU jobs and around 80% of total waiting time across jobs.Fair-share delays are easier to mitigate with preemption, whereas fragmentation delays are harder to overcome.
- Queueing delays: VC5 has a higher fair-share-delay proportion of 37% because it often oversubscribes its quota.Queueing-delay fractions vary across virtual clusters.
- GPU utilization: ResNet-50 GPU utilization falls from 57.7% on one server to 49.6% across two servers, while intra-server interference lowers utilization by as much as 12.1%.The controlled experiments connect distributed placement and colocation with lower utilization and slower training performance.
- GPU utilization: GPU cycles are utilized 56.9% of the time for 8-GPU jobs but only 34.3% for 16-GPU jobs distributed across two servers.The median utilization for 8-GPU jobs is 73.12%, or 1.67x the 16-GPU median.
4 Training Progress and Completion
Training jobs often consume substantial resources despite failure or marginal convergence gains. Failures are frequently user-driven and early, while infrastructure failures are less common but can emerge much later and impose disproportionate costs on large jobs.
- 30.7% of jobs were killed or unsuccessful, consuming around 55% of total GPU time during the trace.
- Around 80% of passed jobs required all epochs to reach their lowest training loss, while around 75% reached within 0.1% of that loss using only 40% of epochs.
- Reaching within 0.1% of the lowest loss accounted for 62% of GPU time for passed jobs and 56% for killed jobs.
- Jobs using more than 4 GPUs retried more often and finished unsuccessfully at higher rates.
- Programming errors dominated failures and often occurred within 10 minutes, whereas infrastructure failures represented 6.2% of trials and generally had longer runtimes to failure.
- Semantic-error failures increased from 9.22% by runtime-to-failure analysis to 17.06% when GPU demand was included.
5 Design Implications for Future Schedulers
The authors recommend scheduler policies that prioritize locality, mitigate interference, and improve failure handling. These policies trade queueing, fragmentation, or migration overhead against training efficiency and failure prevention.
- Prioritizing locality: Schedulers should wait longer for locality because reduced locality can lower utilization and extend long-running jobs by multiple hours.
- Mitigating interference: Dedicated placement can reduce interference among small jobs, but it increases fragmentation and may lengthen waits for larger jobs.
- Mitigating interference: Job migration can defragment the cluster, reduce interference for small jobs, and improve intra-job locality for large jobs.
- Improving failure handling: Syntax checking, first-iteration execution, and cheaper VM pre-runs could prevent or capture many user and runtime errors before full training.
- Improving failure handling: Predictive failure mitigation could classify log errors in real time and adapt retry settings or blacklist problematic input data.
6 Related Work
Related work covers failure analysis in shared data-analytics clusters, machine-learning scheduling and runtime systems, GPU sharing and memory management, and approximate data processing. This paper distinguishes itself through a systematic production-cluster study focused on deep-learning job lifecycles.
- Failure analysis: Prior data-analytics studies emphasize fault tolerance and reliable execution, whereas this paper analyzes failures specific to deep-learning platforms.
- Scheduler and runtime: SLAQ and Optimus use quality improvement or convergence curves to guide machine-learning resource allocation and completion-time scheduling.
- The paper contributes a systematic production study covering queuing, execution, and failure across deep-learning jobs.
- GPU resource management: Baymax, Prophet, and Gandiva investigate GPU sharing, performance prediction, or time-sharing to improve machine-learning cluster efficiency.
- GPU resource management: Virtualized and fine-grained memory controls address networks that do not fit within available GPU memory.
- Approximate data processing: Approximate data processing frames machine-learning training as a setting for trading accuracy against earlier completion.
7 Conclusion
The paper identifies locality, colocation interference, and failures as major factors affecting GPU-cluster utilization. It also releases a production scheduler trace to support further research.
- The analysis links locality and colocated-job interference to lower GPU utilization in distributed training.
- The study characterizes how errors across the software and infrastructure stack contribute to training-job failures.
- The authors provide scheduler guidelines for future machine-learning systems based on data analysis and operational experience.
- The released trace contains job arrivals, sizes, placements, and runtimes from production deep-learning training jobs.