Source-linked AI summary
Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, Eric P. Xing
TL;DR
DL schedulers must configure resources and training parameters that interact and change during training, but existing approaches typically optimize only one side. Pollux jointly models and adapts both sides using goodput, reducing completion times and improving fairness relative to recent schedulers. Its throughput model remains limited in heterogeneous or specialized settings.
Problem
Resource allocation, batch size, and learning rate are difficult to configure jointly because their optimal values depend on the job, architecture, hardware, and changing cluster conditions.
Method
Pollux monitors jobs, models goodput from throughput and statistical behavior, and jointly tunes resources, batch sizes, learning rates, and related training parameters.
Results
Pollux achieves shorter job completion times than recent DL schedulers and is more fair, including when users configure jobs well.
Takeaways & Limitations
Goodput maximization provides a basis for coordinating cluster-wide resource allocation with per-job training optimization in shared DL clusters.
Takeaways & Limitations
Pollux’s throughput model may diverge for heterogeneous or specialized hardware, alternative synchronization or parallelization strategies, larger scales, and hidden non-network contention.
Abstract
from arXiv · showhide
Pollux improves scheduling performance in deep learning (DL) clusters by adaptively co-optimizing inter-dependent factors both at the per-job level and at the cluster-wide level. Most existing schedulers expect users to specify the number of resources for each job, often leading to inefficient resource use. Some recent schedulers choose job resources for users, but do so without awareness of how DL training can be re-optimized to better utilize the provided resources. Pollux simultaneously considers both aspects. By monitoring the status of each job during training, Pollux models how their goodput (a novel metric we introduce that combines system throughput with statistical efficiency) would change by adding or removing resources. Leveraging these information, Pollux dynamically (re-)assigns resources to improve cluster-wide goodput, while respecting fairness and continually optimizing each DL job to better utilize those resources. In experiments with real DL jobs and with trace-driven simulations, Pollux reduces average job completion times by 37-50% relative to state-of-the-art DL schedulers, even when they are provided with ideal resource and training configurations for every job. Pollux promotes fairness among DL jobs competing for resources based on a more meaningful measure of useful job progress, and reveals a new opportunity for reducing DL cost in cloud environments. Pollux is implemented and publicly available as part of an open-source project at https://github.com/petuum/adaptdl.
1 Introduction
DL cluster scheduling is difficult because resource allocations and training parameters are interdependent, dynamic, and costly to configure manually. Pollux addresses this by jointly adapting resources and training settings to maximize goodput, balancing throughput with statistical efficiency.
- Manual GPU allocation can cause queuing and inefficient usage when too many resources are assigned, or long runtimes when too few are assigned.
- Resource allocation, batch size, and learning rate are interdependent, with optimal values varying across tasks, architectures, and changing cluster conditions.
- System throughput generally rises with batch size, but statistical efficiency often falls even after retuning the learning rate.
- 10x larger batch sizes may be tolerated later in training than earlier without degrading statistical efficiency.
- Pollux jointly tunes GPU allocation, worker co-location, batch size, gradient accumulation, and learning-rate scaling while optimizing cluster-wide allocations.
2 Background: Distributed DL Training
Distributed DL training balances computation, synchronization, throughput, and statistical efficiency, while existing schedulers generally optimize resource allocation without co-adapting training parameters. Pollux targets this gap, subject to model and hardware scope limitations.
- Distributed execution: Distributed data-parallel training replicates model parameters across GPUs, averages local gradient estimates, and applies synchronized updates.
- System throughput: Iteration runtime combines gradient-computation time and synchronization time, with synchronization affected by gradients, network performance, and GPU co-location.
- Throughput and efficiency: Larger batch sizes can improve scalability by increasing the ratio of computation to synchronization, but may reduce statistical efficiency or final validation performance.
- Training parameters: Learning-rate scaling rules include linear scaling, square-root scaling, and adaptive approaches such as AdaScale.
- Existing schedulers: Existing schedulers either require fixed user-specified GPU allocations or choose resources without modeling statistical efficiency and its dependence on training parameters.
- Pollux: Pollux co-adapts resource decisions and training parameters to improve goodput, while its current throughput model excludes accelerator heterogeneity.
3 The Goodput of DL Training and Pollux
Pollux defines goodput as a joint measure of system throughput and statistical efficiency, then models and tunes the interdependent resource and training configurations that determine it. Its measurements show that statistical efficiency varies with batch size and training progress, while its throughput model fits observed data across tasks with at most 10% average error.
- Goodput definition: Goodput is the product of system throughput and statistical efficiency at iteration t.
- Goodput definition: Pollux represents a job configuration using GPU allocation, per-GPU batch size, and gradient accumulation steps, with total batch size summed across GPUs.
- Pollux’s approach: Pollux profiles each job to fit throughput and efficiency models, then periodically retunes its configuration according to cluster-wide resource availability and performance.
- Statistical efficiency: Statistical efficiency is measured relative to the initial batch size and learning rate, with Pollux considering batch sizes M ≥ M0.
- Statistical efficiency: Validation curves across evaluated batch sizes reached similar best values, within ±1% relative difference for all tasks except DeepSpeech2 at ±4%.
- Statistical efficiency: Larger batch sizes generally have lower statistical efficiency early in training, while efficiency changes over training and can increase sharply when learning rates are annealed.
- System throughput: Pollux models throughput using GPU allocation and co-location, batch size, and gradient accumulation; across all tasks, fitted-model error was at most 10%.
4 Pollux Design and Architecture
Pollux co-adapts DL training at both job and cluster-wide granularities: agents tune each job to its allocation, while the scheduler reallocates resources using goodput, fairness, and system constraints.
- Architecture: Pollux jointly adapts per-job training configurations and cluster-wide resource allocations.Job-level tuning covers batch size and learning rate; cluster-level decisions reallocate resources across jobs.
- PolluxAgent: PolluxAgent fits each job’s efficiency and throughput functions, then periodically reports its goodput function to PolluxSched.The agent uses observed training metrics to model current job behavior and support subsequent optimization.
- PolluxAgent: PolluxAgent continually measures gradient noise scale and throughput to select efficient batch sizes and adapt learning rates for current allocations.Learning-rate adaptation uses plug-in rules such as AdaScale for SGD or square-root scaling for Adam.
- PolluxSched: PolluxSched maximizes a generalized mean of per-job speedups to choose efficient cluster-wide allocations.A job’s speedup compares goodput under its allocation with goodput under a fair-resource allocation.
- PolluxSched: The fairness parameter p controls the trade-off between aggregate goodput and equal speedup across jobs.As p approaches −∞, the objective approaches the minimum speedup; the authors report p = −1 as achieving most goodput improvements with reasonable fairness.
- PolluxSched: PolluxSched accounts for reallocation delays and avoids network interference by preventing multiple distributed jobs from sharing a node.Reallocation penalties increase with a job’s historical reallocation rate, while node sharing can cause substantial slowdowns.
5 Evaluation
Pollux dynamically co-optimizes resource allocations and DL training configurations as cluster conditions and training progress change. It improves completion-time and fairness outcomes relative to state-of-the-art schedulers in testbed experiments and simulations.
- Testbed results: 37–50% lower average job completion times were achieved relative to state-of-the-art DL schedulers, even with well-tuned baseline configurations.The comparison used a 64-GPU testbed and intentionally supplied baseline schedulers with favorable job configurations.
- Testbed results: Pollux trades between high-throughput/low-efficiency and low-throughput/high-efficiency training modes according to cluster state and training progress.This adaptation occurs dynamically for each job rather than relying on a fixed training configuration.
- Simulation results: 1.5–5.4× better finish-time fairness than baseline DL schedulers was obtained using Pollux’s fairness knob.The fairness setting changes the scheduling objective to support different trade-offs among jobs.
- Simulation results: 25% lower training cost for large models such as ImageNet was shown as a potential outcome of Pollux-based cloud auto-scaling.This result comes from cluster-simulator evaluation and is presented as a potential cost reduction opportunity.
5.1 Experimental Setup
The evaluation combines a 64-GPU Kubernetes testbed with a synthetic workload sampled from Microsoft’s deep learning cluster traces. Baseline configurations were manually tuned under assumptions favorable to competing schedulers.
- Testbed: The testbed contained 16 nodes and 64 NVIDIA T4 GPUs, with four GPUs per AWS EC2 g4dn.12xlarge node.Each node also provided 48 vCPUs, 192GB memory, and a 900GB SSD.
- Workload: The synthetic workload randomly sampled 160 jobs from the busiest 8-hour range of Microsoft deep learning cluster traces.The trace included submission time, GPU count, and duration, while missing model-architecture information was supplemented through workload construction.
- Workload: Jobs were categorized by total GPU-time into Small, Medium, Large, and XLarge groups before matching them with training jobs.The categories span 0–1, 1–10, 10–100, and 100–1000 GPU-hours, respectively.
- Baselines: Baseline GPU counts and batch sizes were manually tuned using measured iteration times and model scalability.The setup used the number of GPUs and batch sizes specified for the synthetic workload when configuring baseline schedulers.
- Baselines: The baseline setup assumed highly rational and knowledgeable users, deliberately favoring the competing schedulers.The authors describe this uniformly sophisticated-user assumption as unrealistic and intended only for comparison.
- Pollux configuration: Pollux used a 60s scheduling interval, 30s reallocation-delay estimate, and default fairness setting p = −1.PolluxAgent reported updated throughput parameters and gradient statistics every 30s.
5.2 Testbed Macrobenchmark Experiments
Pollux co-adapts cluster resource allocation with per-job training configurations, improving goodput under changing contention. It substantially reduces completion times against baselines even when those baselines receive tuned configurations.
- 50% and 37% shorter average JCT was achieved versus Optimus+Oracle+TunedJobs and Tiresias+TunedJobs, respectively.These comparisons gave baselines well-tuned job configurations, favoring them in the evaluation.
- Pollux jointly adapts GPU allocations, co-location, batch size, gradient accumulation, and learning-rate scaling to optimize DL training goodput.
- During low contention, Pollux uses more GPUs and larger batches despite lower statistical efficiency when this raises overall goodput.During high contention, it shifts toward fewer GPUs and smaller batches, increasing statistical efficiency.
- 72% and 73% shorter average JCT was achieved versus Optimus+Oracle and Tiresias with more realistic job configurations, respectively.Pollux also reduced tail JCT by 50% and 56% and makespan by 43% and 48% against those baselines.
- Figure 6 tracks co-adaptation through active jobs, allocated GPUs, batch size, and statistical efficiency for ImageNet and YOLOv3 workloads.The figure exposes how job configurations change as cluster contention and training progress evolve.
5.3 Simulator Experiments
Pollux’s simulator replays measured throughput and gradient statistics across resource and batch-size configurations to evaluate scheduling under broader workloads. Across simulated traces, Pollux improves completion time and fairness while maintaining relative gains across workload intensities.
- Simulator construction: The discrete-time simulator replays measured system throughput and statistical efficiency for each model under varied resource and batch-size configurations.Measurements cover 146 GPU allocations and placements on a 16-node, 64-GPU testbed cluster.
- Scheduling performance: 48% and 32%: Pollux reduces average JCT relative to Optimus+Oracle+TunedJobs and Tiresias+TunedJobs, respectively.These simulator improvements are similar in scale to the testbed results.
- Fairness: 99%: Pollux with p = −1 achieves finish-time fairness ρ<2 while still providing significant performance improvements.Pollux with p = 1 has a long tail of jobs with ρ>4, whereas p = −1 provides the best fairness among the evaluated Pollux settings.
- Fairness: p = −10 produces slightly worse fairness because PolluxSched performs more reallocations while ignoring their cost to equalize speedups continuously.This contrasts with the fairness achieved at p = −1.
- Sensitivity to job load: Across increasing job-submission rates, all policies experience longer average JCT and makespan, while Pollux maintains similar relative improvements over the baselines.The comparison uses Pollux, Optimus+Oracle+TunedJobs, and Tiresias+TunedJobs.
- Interference avoidance: With zero interference slowdown, PolluxSched performs similarly whether or not interference avoidance is enabled.This indicates that it can find efficient cluster allocations while obeying the interference-avoidance constraint.
5.4 More Applications of Pollux
Pollux’s goodput-based scheduling supports adaptive cloud provisioning and can be applied to hyper-parameter optimization workloads. The section presents an auto-scaling comparison and a preliminary Bayesian-optimization experiment, while noting that broader HPO evaluation remains future work.
- Cloud auto-scaling: Goodput-driven scheduling can make provisioning more resources and using larger batch sizes later in training more cost-effective when statistical efficiency increases.Cloud users pay for the duration of held resources, motivating resource changes based on training progress.
- Cloud auto-scaling: Throughput-based auto-scaling quickly scales to more nodes and a larger batch size, which then remains constant because system throughput does not change with training progress.This is the comparison baseline for Pollux’s auto-scaler in ImageNet training.
- Hyper-parameter optimization: HPO algorithms submit training trials to evaluate hyper-parameter choices against objectives such as model accuracy or energy efficiency.Bayesian optimization launches a few jobs at a time, whereas bandit-based methods may launch many and early-stop unpromising trials.
- Hyper-parameter optimization: Pollux’s effects across different HPO algorithm types remain unevaluated; the section reports a TPE experiment with four concurrent and 100 total trials.The experiment tunes ResNet18 on CIFAR10 across learning rate, annealing, momentum, weight decay, and network width.
5.5 Artifact
Pollux is accompanied by an artifact containing its implementation, benchmark models, experiment scripts, simulator results, and raw testbed logs. The artifact also provides analysis scripts for reproducing the reported testbed experiments.
- Artifact contents: The artifact includes the full Pollux implementation and benchmark model implementations.The benchmark models correspond to Table 1.
- Artifact contents: The artifact includes testbed experiment scripts, the cluster simulator implementation, and simulator results.These materials cover the experiments described in Sections 5.2 and 5.3.
- Reproducibility: Raw testbed experiment logs and analysis scripts are provided separately for the reported results.The passage gives public repository locations for both artifact and results materials.
6 Additional Related Work
Prior work adapts training configurations or supports custom adaptation policies, but generally assumes extra resources are available. Pollux instead adapts batch size and learning rate alongside currently available cluster resources.
- Adaptive batch size training: AdaBatch increases batch size at predetermined training iterations while linearly scaling the learning rate.
- Adaptive batch size training: Smith et al. propose increasing batch size instead of decaying the learning rate during training.
- Adaptive batch size training: CABS adaptively tunes batch size and learning rate during training using gradient statistics similar to Pollux.
- Adaptive batch size training: Pollux complements adaptive batch-size strategies by coordinating batch size and learning rate with resources available in shared environments.
- Adaptive training infrastructure: KungFu enables custom adaptation policies and efficient monitoring, offering tools for implementing PolluxAgent's adaptive policies.
7 Conclusion
Pollux jointly allocates cluster resources and tunes each DL training job to maximize goodput. It outperforms and is fairer than recent DL schedulers, including when users configure jobs well.
- Pollux co-adaptively allocates resources while tuning each training job to use those resources effectively.
- Pollux formulates goodput by combining system throughput and statistical efficiency for distributed DL training.
- Pollux jointly tunes resource allocations, batch sizes, and learning rates, which are difficult for users to configure manually.
- Pollux outperforms and is fairer than recent DL schedulers, even when users configure their jobs well.