DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters

• Existing schedulers either allocate resources statically or use heuristics tightly coupled to ML frameworks and workload models.Static allocation can underutilize resources, while framework changes may invalidate analytical models.Source: §1 · p. 1
• DL2 uses a black-box, end-to-end learning approach to dynamically adjust resource allocation without expert heuristics or explicit workload modeling.Its policy network maps the current system state to resource-allocation decisions for current jobs.Source: §1 · pp. 1–2
• DL2 bootstraps its policy with historical scheduling decisions, then learns from live feedback to minimize average job completion time.This avoids extensive offline training over simulations built around explicit performance models.Source: §1 · p. 2
• DL2 integrates with Kubernetes and enables dynamic scaling of MXNet jobs; evaluation uses testbed experiments and controlled simulations driven by production job patterns.The evaluation also examines scaling overhead and generality.Source: §1 · p. 2
• 44.1% improvement in average job completion time over the widely adopted DRF scheduler.Source: §1 · p. 2

2.1 The Parameter Server Framework

• Workers compute gradients on data mini-batches, while parameter servers maintain and update shared model parameters in synchronized training.A worker starts the next iteration after receiving updated parameters; processing the full dataset once completes an epoch.Source: §2.1 · p. 2

2.2 Motivations

• Adding workers and parameter servers yields diminishing training-speed returns because communication overhead increases.The testbed comparison used ResNet-50, VGG-16, and Seq2Seq, increasing workers and PSs together.Source: §2.2 · p. 3
• With 12 total workers and PSs, Seq2Seq runs fastest at a 4:8 PS-to-worker ratio, while VGG-16 runs fastest at 6:6.Source: §2.2 · p. 3
• Production-cluster GPU utilization varies over time, creating opportunities to scale jobs as cluster load changes.Fixed allocations may leave GPUs idle after a job completes while other jobs remain active.Source: §2.2 · p. 3
• Explicit performance models omit multi-tenant interference; across 898 production jobs, average completion-time variation is 27.3%.Variation exceeds 100% for 3.5% of jobs, and modeling interference is difficult.Source: §2.2 · p. 3

2.3 Deep Reinforcement Learning

• Existing DRL schedulers commonly train offline using simulated traces based on explicit performance models, which may be inaccurate under interference.Historical traces alone may not provide feedback for the DRL policy's many possible resource-allocation decisions.Source: §2.3 · p. 4
• DL2 advocates online learning from live allocation feedback, with offline supervised learning to initialize its policy from an existing scheduler.The warm-up is intended to avoid poor initial decisions and support a smooth transition.Source: §2.3 · p. 4

3 DL2 Overview

• DL2 aims to find a live-cluster resource allocation policy that minimizes average completion time across concurrent jobs.Source: §3 · p. 4

3.1 DL Cluster

• Users specify each job’s worker and parameter-server resource demands and total training epochs, which can be estimated from expertise or job history.Source: §3.1 · p. 4
• When worker counts change, synchronous jobs adjust per-worker mini-batches to preserve total batch size, while asynchronous jobs keep mini-batch size unchanged.Source: §3.1 · p. 4

3.2 DL2 Scheduler

• DL2 warms its policy network on historical traces to match existing-scheduler decisions, then uses job progress as reward for online updates in time slots.The supervised warm-up is described as necessary because applying online reinforcement learning directly performs poorly.Source: §3.2 · p. 4

4.1 Policy Neural Network

• The policy network represents concurrent job types, run duration, remaining epochs, allocated resources, and current worker and parameter-server counts in its input state.Source: §4.1 · p. 5
• Rather than choose a full allocation at once, the network repeatedly selects one of 3 × J + 1 actions: add workers, parameter servers, both, or stop allocating.After each action, DL2 updates the state and infers again until resources are used or a void action is produced.Source: §4.1 · pp. 5–6

4.2 Offline Supervised Learning

• Offline supervised learning uses stochastic gradient descent on cross-entropy to train the network to reproduce resource-allocation decisions in existing-scheduler traces.Source: §4.2 · p. 6

4.3 Online Reinforcement Learning

• Online reinforcement learning rewards each time slot by the sum of concurrent jobs’ normalized epoch progress, with normalization intended to avoid bias toward large jobs.The paper connects greater per-slot epoch progress with fewer completion time slots.Source: §4.3 · p. 6
• DL2 uses REINFORCE for discounted cumulative reward and an actor-critic advantage over a value baseline to reduce gradient variance and stabilize policy learning.The value network shares the policy network’s structure except for its linear output layer.Source: §4.3 · p. 6
• Job-aware exploration replaces outputs in identified poor states with specified actions, while experience replay samples recent transitions across time slots to alleviate sample correlation.The poor-state cases include worker/parameter-server imbalance; replay is used to address correlated training samples.Source: §4.3 · p. 7

5 Dynamic Scaling

• Checkpointing and restarting a job to change worker or parameter-server counts adds delay; one DSSM example takes 1 minute to checkpoint and stop, then 5 minutes to restore.The paper proposes hot scaling in MXNet as an alternative that resizes resources without terminating training.Source: §5 · p. 7
• Scaling must preserve a consistent global model and correct worker-to-parameter-server updates while minimizing training interruption and balancing parameter-server load.Source: §5 · p. 7
• For parameter-server addition, the coordinator assigns parameters with a best-fit algorithm that balances parameter counts while minimizing movement, then uses version counters and a scaling clock to coordinate migration.Parameter servers move parameters when their counters reach the clock; the coordinator then notifies workers to resume training.Source: §5 · pp. 7–8
• Worker addition registers connections and copies the training dataset before starting; worker removal updates node lists and adjusts mini-batches to preserve total batch size.Source: §5 · p. 8

6.1 DL2 Implementation

• The Kubernetes scheduler maps cluster and job states to decisions in under 3 ms; container agents monitor training status and start scripts once job containers are ready.Jobs use MXNet, with workers and parameter servers running in Docker containers.Source: §6.1 · p. 8

6.2 Methodology

• Evaluation uses patterns from a 75-day production-cluster trace, eight MXNet tutorial model categories, 30 testbed jobs, and simulations with 500 servers and 200 jobs.The source job code and model details were unavailable, so the authors selected tutorial models for evaluation.Source: §6.2 · pp. 8–9, Table 1 · p. 9

6.3 Performance

• DL2 improves average job completion time by 44.1% over DRF, achieves 17.5% higher performance than Optimus, and outperforms OfflineRL by 37.9%.The comparison uses the testbed evaluation.Source: §6.3 · pp. 9–10, Figure 9 · p. 10
• DL2 scaling suspends workers for tens of milliseconds, versus tens of seconds for checkpoint-based scaling; larger models spend more time moving parameters.Only step 4 blocks worker training; parameter movement takes place in step 3 and may overlap with step 4.Source: §6.3 · p. 10, Figure 11 · p. 10, Figure 12 · p. 10

6.4 Generality

• With 20% total-epoch estimation error, DL2 still beats DRF by 28%; Optimus is more sensitive to training-speed variation that can break its performance-model convexity assumption.DL2’s average job completion time increases slightly as estimation error grows.Source: §6.4 · pp. 10–11, Figure 14 · p. 11, Figure 13 · p. 11
• DL2 gradually matches the ideal baseline on new job types, while its supervised-learning strategy also exceeds its existing scheduler, including a 41.3% speedup over SRTF.The evaluation introduces two new model categories during online RL and tests FIFO and SRTF as alternative supervised-learning strategies.Source: §6.4 · p. 11, Figure 15 · p. 11, Figure 16 · p. 11
• Scheduling jobs in batches when concurrency exceeds J can hurt performance if J is too small; a J large enough for peak concurrency performs better.The evaluation observed around 40 as the maximal number of uncompleted jobs.Source: §6.4 · p. 11, Figure 17 · p. 11

6.5 Training Design

• Training ablations find worse performance without job-aware exploration, experience replay, or a value network; federated training remains stable as cluster count increases and converges faster.The reported degradations are 28.8% without exploration, 39.6% without experience replay, and 21.1% without the value network.Source: §6.5 · pp. 11–12, Table 2 · p. 11, Figure 18 · p. 11

7 Discussion and Future Directions

• Fairness can be added as a quantified reward term, and all-reduce jobs can be supported by modifying the neural network’s input state and action space.The paper focuses on parameter-server jobs but notes that all-reduce support requires, for example, removing parameter-server elements.Source: §7 · p. 12
• Learning placement together with resource allocation is challenging because it substantially enlarges the action space; hierarchical neural networks may help.The paper uses the cluster’s default placement policy in this work.Source: §7 · p. 12
• Practical deployment may need to address adversarial inputs that fool the neural network and monitoring for exceptional scheduling.The paper identifies both as directions for further exploration.Source: §7 · p. 12

8 Related Work

• Prior DRL schedulers for cloud jobs and graph-parallel jobs train offline and use static resource allocation or do not adjust resources during execution.Other DRL work covers computation-graph placement, traffic routing, and video streaming, generally using offline data from models or simulators.Source: §8 · p. 12
• Unlike offline DRL approaches trained with model- or simulator-generated data, DL2 prepares its policy with supervised learning and improves it through online RL.The related-work discussion contrasts DL2’s approach with prior offline DRL training.Source: §8 · p. 12
• ML-cluster schedulers span convex-loss scheduling, utilization-fairness optimization, model-based resource adjustment, and GPU time-slicing, with different workload and allocation scopes.The cited work includes SLAQ, Dorm, Optimus, Bao et al., and Gandiva.Source: §8 · p. 12

9 Conclusions

• DL2 bootstraps scheduling with offline supervised learning, then improves its policy through reinforcement learning using live cluster feedback.The conclusion reports that the testbed and trace-driven simulations verify low scaling overhead, generality, and outperformance over hand-crafted heuristics.Source: §9 · pp. 12–13