Source-linked AI summary
Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters
Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, Tianwei Zhang
TL;DR
GPU-cluster management needs workload evidence that reflects diverse and evolving deep learning jobs rather than relying on a single older trace. The paper analyzes SenseTime’s Helios traces and builds a prediction-based resource-management framework, whose case studies improve job completion time and node utilization. Its scope includes production GPU clusters, while broader scheduling reliability and alternative energy techniques remain future work.
Problem
Existing GPU schedulers largely rely on the 2017 Philly trace, whose single-dataset basis and age may limit workload generality amid rapid DL development.
Method
The paper analyzes Helios traces across clusters, jobs, and users, then uses historical data and machine-learning predictions to manage resources and schedule DL jobs.
Results
6.5× improvement in average JCT is reported for QSSF, while CES improves node utilization by up to 13% and conserves over 1.65 million kilowatt hours annually.
Takeaways & Limitations
The traces broaden DL workload evidence, and the framework provides extensible services deployable as plugins atop existing GPU-cluster management systems.
Takeaways & Limitations
Implementing advanced scheduling algorithms in production clusters with better reliability and robustness remains future work.
Abstract
from arXiv · showhide
Modern GPU datacenters are critical for delivering Deep Learning (DL) models and services in both the research community and industry. When operating a datacenter, optimization of resource scheduling and management can bring significant financial benefits. Achieving this goal requires a deep understanding of the job features and user behaviors. We present a comprehensive study about the characteristics of DL jobs and resource management. First, we perform a large-scale analysis of real-world job traces from SenseTime. We uncover some interesting conclusions from the perspectives of clusters, jobs and users, which can facilitate the cluster system designs. Second, we introduce a general-purpose framework, which manages resources based on historical data. As case studies, we design: a Quasi-Shortest-Service-First scheduling service, which can minimize the cluster-wide average job completion time by up to 6.5x; and a Cluster Energy Saving service, which improves overall cluster utilization by up to 13%.
1 INTRODUCTION
Large-scale GPU clusters support increasingly demanding deep learning workloads, but effective management depends on understanding their distinctive job and user behaviors. The paper addresses limited and aging workload evidence with SenseTime traces and a prediction-based management framework, demonstrating improved scheduling and utilization.
- Motivation: Training production deep learning models can require large GPU resources for thousands of petaflops of operations.Multi-tenant GPU clusters therefore regulate resources and access for different groups and users.
- Workload Characteristics: DL jobs combine iterative training, resource heterogeneity, placement sensitivity, and exploratory cancellations that distinguish them from conventional workloads.Multi-GPU communication depends on interconnect topology, while users may cancel jobs after early feedback from alternative configurations.
- Research Gap: Analysis based primarily on the 2017 Philly trace may not generalize to newer DL workloads or avoid trace-specific overfitting.The paper notes that rapid DL development and changing demands may make that trace outdated.
- Dataset: Helios contains over 3 million jobs collected for six months in 2020 across four independent GPU clusters, spanning diverse job types, applications, and purposes.The traces are intended as an alternative that can increase workload diversity alongside Philly.
- Contributions: The study analyzes clusters, jobs, and users, identifying seven implications for GPU-cluster design and management.Prior work had focused mainly on job-level characterization.
- Case Studies: 6.5× improvement in overall job completion time is achieved by the Quasi-Shortest-Service-First scheduling service.The service prioritizes jobs using historical information; the paired energy-saving service improves node utilization by up to 13%.
2 BACKGROUND
This section introduces Helios, its DL workloads and traces, and the terminology used to analyze GPU datacenter behavior. It motivates broader workload characterization by comparing Helios with Philly and highlighting differences in scale, job mix, GPU demands, duration, and wasted GPU time.
- 2.1 Helios Datacenter: Helios is a private, multi-tenant SenseTime datacenter with four representative clusters: Venus, Earth, Saturn, and Uranus.Saturn mixes NVIDIA Pascal and Volta GPUs; the other clusters use homogeneous GPU types.
- 2.2 Workloads in Helios: Helios supports training, inference, preprocessing, quantization, and other workloads submitted by product and research groups across multiple DL domains.Most GPU jobs use iterative training, while preprocessing and postprocessing jobs also form part of the development pipeline.
- 2.3 DL Job Traces from Helios: The traces span six months and contain 3.36 million jobs across 802 nodes and 6,416 GPUs.They combine Slurm job logs, daily virtual-cluster configurations, and inferred cluster timing information.
- 2.3.2 Comparisons with the Philly Trace: Helios contains 32.6× more jobs than Philly, around half CPU jobs, and GPU jobs requesting over twice Philly’s average GPU count.The maximum request is 2,048 GPUs, an order of magnitude higher than Philly’s.
- 2.3.2 Comparisons with the Philly Trace: Failed jobs account for over one-third of Philly’s GPU time and 9.3% in Helios.The comparison uses GPU time, calculated from execution time multiplied by GPU count, and groups timeout and node-fail jobs as failed.
3 CHARACTERIZATION OF DL JOBS
The paper extends prior DL workload studies by analyzing clusters, jobs, and users using traces that capture varied production and exploratory behaviors. This broader perspective supports conclusions intended to generalize across organizations and clusters.
- 3 CHARACTERIZATION OF DL JOBS: Prior studies focused mainly on job-level DL workload characterization, whereas this study analyzes clusters, jobs, and users.The authors position the analysis as more comprehensive than earlier DL trace studies.
- 3 CHARACTERIZATION OF DL JOBS: The traces include long-term production jobs and short-term exploratory jobs, with users sometimes stopping jobs based on expected performance.The assessment is statistical, and the authors state that the conclusions are general for other organizations and clusters.
3.1 Cluster Characterization
Cluster activity varies across daily, monthly, and virtual-cluster scales, with utilization shaped by job duration, GPU demand, and resource allocation. These patterns support prediction and expose trade-offs between static fairness partitions, queuing delay, and utilization.
- Daily Trends of Cluster Usage: 65%–90% hourly utilization varies across clusters, with Saturn highest and Venus and Earth relatively underutilized.Hourly utilization decreases by 5–8% at night; Saturn’s standard deviation is 7%, compared with 10–12% in other clusters.
- Daily Trends of Cluster Usage: Daily GPU submissions fall at night and dip around lunch and dinner, while Earth sustains roughly 100 jobs per hour but has the lowest utilization.Earth’s lower utilization is attributed to its shorter GPU jobs, since utilization depends on both job count and running time.
- Monthly Trends of Cluster Usage: Stable multi-GPU submissions and critical resource use make multi-GPU jobs more useful to predict than fluctuating single-GPU submissions.Single-GPU jobs have weak impact on monthly cluster usage, whereas multi-GPU jobs exhibit more stable monthly patterns and are critical to utilization.
- Virtual Cluster Behaviors: VC utilization correlates positively with average GPU demand, while queuing delay is approximately proportional to average job duration.Large-job VCs such as vc6YE and vcLJZ usually exceed 90% utilization, whereas small-job VCs such as vchbv and vccaA are generally below 65%.
- Virtual Cluster Behaviors: Static VC partitioning creates imbalanced resource allocation, allowing long queuing delays and resource underutilization to coexist.The paper identifies fairness-aware scheduling and appropriate VC regulation as important responses, while production implementation of advanced schedulers remains future work.
3.2 Job Characterization
DL jobs differ substantially in duration, GPU demand, and final status. Although single-GPU jobs dominate job counts, larger jobs consume most GPU time, while short debugging failures and early cancellations create scheduling and efficiency concerns.
- Job Execution Time: GPU jobs average 6,652s, 10.6× longer than CPU jobs at 629s, with GPU durations spanning seconds to weeks.More than 50% of CPU jobs run under 2s, whereas the median GPU execution time is 206s.
- Job Size: Over 50% of jobs are single-GPU in every cluster, but they consume only 3–12% of GPU time; jobs using ≥8 GPUs consume around 60%.Large jobs represent fewer than 10% of jobs but account for most computing resources.
- Job Final Status: 37.6% of GPU jobs are unsuccessful versus 9.1% of CPU jobs, partly because users cancel training after convergence or stop poor-performing jobs.The paper treats these user-terminated training jobs as successful cancellations rather than ordinary failures.
- Job Final Status: For jobs requesting 64 or more GPUs, fewer than a quarter complete successfully and cancellations reach roughly 70%.Long runtimes increase users’ opportunities to stop large jobs early to save time and resources.
- Job Final Status: Many failed jobs are short debugging runs that can wait longer than they execute, motivating a dedicated debugging VC with a short-term limit.Most failures stem from user errors such as script configuration or syntax and semantic errors.
3.3 User Characterization
User-level behavior is uneven across resource consumption, queuing, and completion. Most users run GPU training jobs, but a small group consumes much GPU time and a few marquee users bear most queuing time.
- Resource Consumption: Only about 25% of users conduct CPU tasks, while nearly every user runs GPU training jobs.The top 5% of users consume 45–60% of GPU time, indicating substantially greater concentration for GPU resources than for participation.
- Queuing Delay: Most users avoid severe queuing, but Uranus’s top 1%—only three users—bear over 70% of queuing time.These users are termed “marquee users” and are not necessarily among the top 10 resource consumers.
- Completion Rates: Users’ GPU job completion rates are generally low, so high unsuccessful-job fractions reflect broad user behavior rather than a few individuals.The paper recommends user-aware scheduling, including temporary priority adjustments and appropriate VC regulation.
4 A PREDICTION-BASED FRAMEWORK
The paper uses historical cluster behavior to motivate a prediction-based framework for improving GPU resource usage efficiency and workload performance.
- Framework Overview: Historical traces make cluster behaviors such as job duration and node states predictable enough to support prediction-based GPU resource management.The framework leverages these predictions to improve resource usage efficiency and workload performance.
4.1 Framework Overview
The framework is a centralized, plug-and-play manager that uses machine-learning predictions to support independently selectable resource-management services. It combines model updating with orchestration so operators can adapt resource decisions to cluster demands.
- The framework is a centralized manager atop each GPU cluster that supports independently integrated resource-management services.Operators can select services according to their demands, and Figure 10 presents the framework overview.
- Each service trains a machine-learning model to predict job behavior or cluster state, then uses those predictions to determine resource-management operations.
- The Model Update Engine regularly fetches runtime data or responds to events, periodically fine-tuning models as cluster conditions change.
- The framework provides extensibility through services such as Quasi-Shortest-Service-First Scheduling and Cluster Energy Saving.Services can be integrated as plugins atop existing systems including Slurm, YARN, and Kube-scheduler, typically with millisecond prediction and operation latency.
4.2 Quasi-Shortest-Service-First Scheduling
QSSF is a non-preemptive scheduler that predicts incoming jobs’ GPU time and prioritizes them to reduce head-of-line blocking in GPU clusters. Across Helios clusters, it substantially improves completion and queuing performance over FIFO while approaching idealized baselines.
- Motivation: FIFO scheduling can leave even short-term jobs with long queuing delays because it is unaware of runtime.
- Motivation: SJF and SRTF reduce average JCT but require perfect job-duration information, while SRTF additionally assumes jobs can be preempted and restored.
- Service Design: QSSF uses non-preemptive scheduling and ranks jobs by predicted GPU time, addressing gang-scheduling blockage from large GPU jobs.
- Service Design: QSSF combines rolling estimates from similar historical jobs with a GBDT machine-learning estimate to assign priorities.Historical matching uses attributes such as user and job-name similarity, while the algorithm weights the rolling and machine-learning estimates.
- Evaluation: Short-term jobs improve by at least 9.2× over SJF, while long-term jobs improve by 2.0~4.8× in Helios.The reduction in queued jobs benefits large jobs as well as short jobs.
- Evaluation: 1.5~6.5× improvement in average JCT and 4.8~20.2× improvement in average queuing delay are achieved over FIFO across four Helios clusters.QSSF performs comparably with SJF and SRTF without their unrealistic assumptions; queued jobs decrease by 37~82%.
4.3 Cluster Energy Saving
CES uses historical and predicted node-utilization trends to power down idle servers while preserving resources for arriving jobs. In Helios simulations, it improves utilization and sharply reduces wake-up operations and affected jobs.
- Motivation: Idle GPU nodes waste substantial energy, motivating Cluster Energy Saving to reduce consumption while satisfying user demands.The service applies Dynamic Resource Sleep to selected idle servers.
- Service Design: CES forecasts future running-node counts from time-related features and selects GBDT after comparing it with ARIMA, Prophet, and LSTM.The GBDT model achieves around 3.6% SMAPE error in Earth.
- Service Design: When requested resources exceed active nodes, CES immediately wakes nodes using the resource gap plus a buffer; periodic checks power down nodes when recent and predicted reductions exceed thresholds.The periodic procedure reduces active nodes toward the current running count while retaining the buffer.
- Scope: The evaluation covers DRS-based savings; DVFS-based techniques are identified as future work rather than evaluated here.
- Evaluation: CES predictions closely reflect actual node trends in Earth, allowing many idle nodes to be powered off while retaining enough active nodes for user demand.The evaluation uses three-week Helios traces and prior records for model training.
- Evaluation: 13.0% utilization improvement in Earth and 12.6% in Uranus are achieved, while CES invokes node wake-up only 1.1~2.6 times daily per cluster.Vanilla DRS averages 34.1 wake-up operations per day and affects nearly 6k jobs, compared with 251 of 198k jobs for CES over 21 days.
- Evaluation: Over 1.65 million kilowatt hours of electricity can be saved annually across the four clusters under the paper’s estimate.The estimate uses approximately 800 watts for an idle DGX-1 server and cooling consumption twice server energy.
5 RELATED WORK
Prior work analyzed CPU, HPC, private-cloud, and selected DL workloads, but this paper emphasizes broader characterization and prediction for large-scale GPU clusters. It extends workload analysis across clusters, jobs, and users and uses prediction for resource management.
- Cluster characterization: Earlier trace studies examined job characteristics, workload heterogeneity, cluster utilization, and user behavior in HPC and private-cloud systems.
- Cluster characterization: DL workload studies have examined training-job locality, GPU utilization, failure reasons, resource requirements, and performance bottlenecks.
- Cluster characterization: This work provides more comprehensive analysis across clusters, jobs, and user behaviors, with greater diversity reflecting newer DL algorithms and technologies.
- Prediction-based scheduling: Prior prediction-based schedulers use intra-job predictability, convergence prediction, duration distributions, or online profiling to improve DL scheduling.
6 DISCUSSIONS
The framework is intended to apply across cluster sizes, but prediction accuracy depends mainly on historical-data volume and cluster-specific workload characteristics. Future work targets broader service coverage, operation with limited job information, finer-grained resource features, and production-scale evaluation.
- 6 DISCUSSIONS: The framework can also apply to small clusters because scaling active jobs with resources preserves job and user behaviors.Independent virtual clusters with tens to hundreds of GPUs can be regarded as small clusters.
- 6 DISCUSSIONS: Prediction accuracy depends mainly on training-set size and cluster characteristics, with QSSF requiring diverse job distributions and CES relying on seasonal utilization trends.More historical data is expected to provide more comprehensive and generalized information for higher model accuracy.
- 6.2 Future Works: Future work will add services, support limited job information, incorporate finer-grained resource features, and evaluate the framework in production clusters at scale.Proposed features include GPU memory, computation-unit, and CPU utilization.
7 CONCLUSION
The paper analyzes real-world DL traces from four clusters, derives implications for GPU cluster design, and introduces historical-data-based management services. Its case studies improve average job completion time and conserve substantial annual power, while the Helios traces are publicly available for further research.
- 7 CONCLUSION: The study characterizes clusters, jobs, and users from real-world DL traces across four datacenter clusters and identifies seven design implications.The analysis extends beyond job characterization to guide more efficient GPU cluster systems.
- 7 CONCLUSION: A historical-data-based management framework predicts future job and cluster behavior to improve resource utilization and job performance.The framework is justified by the observed predictability of job and cluster behaviors.
- 7 CONCLUSION: 6.5× improvement in average JCT is achieved by the QSSF service, while CES conserves over 1.65 million kilowatt hours annually.These are the reported outcomes of the two case studies.
- 7 CONCLUSION: Helios traces are publicly available to support research on GPU datacenter system design.The dataset is released through the paper’s stated GitHub repository.