PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepShare: Assurance-Driven Deep Learning Job Scheduling for Multi-Tenant Clusters

Jinghao Wang, Yihang Zhou, Xiao Zhou, Xinlei Zheng, Xiaoyang Sun, Tianyu Wo, Chunming Hu, Renyu Yang

arXiv:2609.16682v1cs.DC

TL;DR

Multi-tenant GPU clusters can remain underutilized while jobs queue because quota, scheduling, preemption, and sharing decisions are not coordinated. DeepShare uses the continuous QAD signal to coordinate elastic borrowing, assurance-aware ordering, preemption, and interference-aware colocation. Across trace-driven simulations and a Kubernetes testbed, it improves utilization and queueing or completion outcomes while maintaining guaranteed-tenant QoS.

  • Problem

    Quota, queueing, preemption, and sharing decisions are often driven by separate local signals, leaving GPUs underutilized while tenants experience long delays.

  • Method

    DeepShare coordinates elastic borrowing, runtime-aware scheduling, cost-sensitive preemption, and interference-aware colocation through the continuous Quota Assurance Degree.

  • Results

    70.58% average GPU utilization is achieved, 29.5% above Lucid, with 46% lower queueing delay in trace-driven evaluation and 34% lower job completion time on a 16-GPU Kubernetes testbed.

  • Takeaways & Limitations

    A shared tenant-assurance control loop coordinates utilization and QoS decisions more effectively than independently optimizing quotas, scheduling, and resource sharing.

  • Takeaways & Limitations

    The physical testbed is limited to 16 GPUs, with results most directly applicable to departmental-scale clusters, and heterogeneous accelerators require per-device interference-model retraining.

Abstract

from arXiv · show

Multi-tenant GPU clusters frequently remain underutilized even when tenants experience long queueing delays, because quota control, queue ordering, preemption, and GPU sharing are driven by different local signals. We present DeepShare, a scheduler that uses a continuous tenant-assurance signal to coordinate these decisions at runtime. DeepShare combines elastic quota borrowing, tenant-specific runtime prediction, cost-aware best-effort preemption, and interference-aware MPS colocation, while using the same assurance signal to decide when borrowed capacity should be reclaimed and when sharing should become more conservative. In trace-driven experiments on 23,859 Venus jobs and 3,200 internal jobs, DeepShare achieves an average GPU utilization of 70.58%, a 29.5% improvement over the strongest non-intrusive sharing baseline, while reducing average queueing delay by 46%. On a 16-GPU Kubernetes testbed, it reduces the average job completion time by 34% and maintains 93% QoS compliance for guaranteed tenants. These results show that treating tenant assurance as a runtime control loop achieves a more advantageous utilization-QoS trade-off than optimizing quotas, scheduling, and resource sharing independently.

I. INTRODUCTION

DeepShare addresses underutilization and queueing delays caused by quota, ordering, preemption, and sharing decisions that lack runtime coordination. It introduces QAD to coordinate elastic borrowing, assurance-aware scheduling, cost-sensitive preemption, and interference-aware colocation.

  • I. INTRODUCTION: Fixed quotas and locally optimized decisions leave GPUs idle while jobs queue, because quota recovery, queue ordering, and sharing are not coordinated.Borrowed quota must be reclaimable, short-job priority must not delay under-served tenants, and low-interference colocation can still impede recovery.
  • I. INTRODUCTION: DeepShare coordinates quota regulation, scheduling, preemption, and GPU sharing through the continuous Quota Assurance Degree (QAD).QAD distinguishes transient quota fluctuations from persistent under-service and provides a shared runtime signal for these decisions.
  • I. INTRODUCTION: 70.58% average GPU utilization is achieved in trace-driven simulations, 29.5% above Lucid, while average queueing delay falls by 46%.The evaluation uses trace-driven simulations of multi-tenant GPU workloads.
  • I. INTRODUCTION: QAD-aware scheduling applies tenant recovery priority before runtime prediction, so short-job optimization cannot override under-service.Runtime prediction refines ordering only after assurance priority is established.
  • I. INTRODUCTION: The framework combines reclaimable best-effort borrowing, cost-sensitive preemption, and QAD-aware MPS colocation in a Kubernetes-native scheduler.Best-effort jobs use surplus capacity and can be reclaimed when guaranteed demand is underserved.

B. Empirical Study of a Multi-Tenant GPU Cluster

The cluster study identifies severe underutilization, workload heterogeneity, and short-job starvation under fixed quotas and FIFO scheduling. These observations motivate DeepShare’s assurance-driven control loop for borrowing, ordering, reclamation, and colocation.

  • B. Empirical Study of a Multi-Tenant GPU Cluster: 19.8% average GPU compute utilization and 18.5% average memory utilization persist over seven days despite bursty daytime demand and quota-induced queueing.Burst submissions consume group quotas, after which allocations remain idle until the next active window.
  • B. Empirical Study of a Multi-Tenant GPU Cluster: 62% of submissions are short exploratory jobs using 11% of GPU-hours, while jobs over four hours consume 73% of GPU-hours.This duration heterogeneity creates an opportunity for runtime-aware scheduling.
  • B. Empirical Study of a Multi-Tenant GPU Cluster: 41% of simultaneously running job pairs have complementary compute and memory behavior, indicating colocation potential under an 80% combined-utilization cap.Such pairs could safely share a GPU under the current exclusive-assignment policy.
  • B. Empirical Study of a Multi-Tenant GPU Cluster: 47 minutes of median queueing delay for short jobs exceeds their median runtime of less than 30 minutes under FIFO scheduling.Per-user runtime distributions are sufficiently stable for data-driven runtime prediction, with 78% of users repeatedly submitting similar jobs.
  • B. Empirical Study of a Multi-Tenant GPU Cluster: DeepShare uses QAD to decide when idle quota can be borrowed or reclaimed, how jobs are ordered, and when colocation should become conservative.DRA exposes idle capacity to best-effort work, while assurance-aware scheduling and sharing protect tenant recovery.
  • B. Empirical Study of a Multi-Tenant GPU Cluster: Guaranteed jobs precede best-effort jobs, and QAD priority precedes runtime prediction so prediction errors cannot elevate well-served tenants above under-served ones.Best-effort allocations do not inflate QAD and are reclaimed when guaranteed demand is underserved.

B. Predictive Scheduling and Preemption

DeepShare resolves head-of-line blocking without letting runtime predictions override tenant-assurance priority. Its scheduler orders guaranteed work by QAD and predicted remaining time, then uses cost-aware preemption that protects guaranteed jobs and limits repeated or wasteful interruption.

  • B. Predictive Scheduling and Preemption: Runtime prediction addresses FIFO head-of-line blocking while preserving tenant-level guarantees.The design favors short jobs only after tenant recovery priority is established, reducing unnecessary preemption of best-effort jobs near completion.
  • B. Predictive Scheduling and Preemption: DeepShare orders guaranteed jobs before best-effort jobs and uses hierarchical tenant and cycle-local placement queues.Cycle-local usage accounting constrains promoted guaranteed work to quota and best-effort work to a configurable multiple of quota.
  • B. Predictive Scheduling and Preemption: QAD is the primary scheduling key, while predicted remaining time refines ordering only among similarly served tenants.Prediction errors cannot override quota assurance, elevate a well-served tenant above an under-served one, or trigger preemption of guaranteed jobs.
  • B. Predictive Scheduling and Preemption: Guaranteed jobs are never selected as preemption victims, and reclaiming a shared GPU preempts its complete resident set.A candidate device must remain feasible for the incoming job after eviction under GPU-memory and node-level CPU/memory constraints.
  • B. Predictive Scheduling and Preemption: DeepShare estimates remaining runtime with per-tenant gradient boosting models and a cluster-wide fallback for cold-start tenants.The estimates support both job ordering and cost-aware preemption decisions.
  • B. Predictive Scheduling and Preemption: Preemption cost favors long-running victims with limited prior disruption and penalizes fragmented sets, especially near-completion jobs.The default weighting parameters are α = 0.5 and β = 0.3.

8 PlaceJobs(QG, true)

The guaranteed-job placement stage appends a job to the best-effort queue before placement.

  • 16 PlaceJobs(QB, false): The scheduling procedure appends the job to QB before best-effort placement.

16 PlaceJobs(QB, false)

Best-effort placement considers increasingly disruptive ways to fit work, from direct assignment through sharing and reclamation to preemption.

  • 16 PlaceJobs(QB, false): The placement algorithm atomically assigns jobs, generates feasible resident-device candidates, colocates when allowed, and updates node headroom after reclamation.
  • 16 PlaceJobs(QB, false): When necessary, the scheduler preempts a selected victim set and binds the incoming job to the reserved device.
  • 16 PlaceJobs(QB, false): Placement escalates from exclusive allocation to interference-aware colocation, CPU/memory reclamation, and finally GPU preemption.This ordering increases the disruption only when less disruptive placement options are unavailable.

C. Interference-Aware Job Colocation

DeepShare treats GPU colocation as assurance-aware admission control rather than unconditional utilization maximization. It combines throughput-retention prediction with dynamic contention and tenant-state thresholds, validates decisions online, and limits candidate evaluation to scheduling-latency budgets.

  • C. Interference-Aware Job Colocation: Colocation is admitted only when predicted throughput retention is high enough for both tenants under current load and assurance states.Pairs must fit GPU memory and node-level CPU/memory headroom, include at least one best-effort job, and satisfy both retention thresholds.
  • C. Interference-Aware Job Colocation: The retention ratio ρ compares colocated throughput with exclusive throughput: ρ = tshared/texcl.The predictor uses DCGM telemetry and an offline-trained Random Forest to estimate pairwise interference with sub-millisecond inference latency.
  • C. Interference-Aware Job Colocation: Contention pressure rises with pending demand and free-GPU scarcity, while γ = 0.5 makes that growth sub-linear so short bursts do not disable all colocation.The tolerance calculation uses pending GPU demand Gp, free GPUs Gf, and a small ϵ to avoid division by zero.
  • C. Interference-Aware Job Colocation: Admission becomes more conservative under heavy load or urgent tenant recovery, and more permissive when pressure is low and both tenants are protected.The default maximum is k = 2 jobs colocated on one GPU, and the tolerance is clamped at one.
  • C. Interference-Aware Job Colocation: Online retention monitoring revokes degraded colocation after w consecutive below-threshold samples and preempts the best-effort partner while the guaranteed job continues.
  • C. Interference-Aware Job Colocation: Candidate generation and prediction cost O(m), sorting costs O(m log m), and node scoring caps evaluated candidates per job.These controls keep colocation admission within the reported scheduling-latency budget.

IV. SYSTEM IMPLEMENTATION

DeepShare is implemented as a Kubernetes-native scheduler plugin with quota control, real-time assurance state, interference-aware sharing, and runtime protection managed at the cluster layer.

  • System architecture: DeepShare packages its scheduler plugin, quota controller, node-local services, runtime estimators, and interference models without requiring user-code or framework changes.The prototype uses Go, Python, Helm/RBAC manifests, and lightweight quota CRDs.
  • Kubernetes integration: TenantQuota objects and job annotations configure quotas and service classes, while a lightweight controller reconciles them into tenant metadata without making placement decisions.The scheduler plugin retains the real-time control loop for queue ordering, colocation admission, and preemption.
  • State management: The scheduler maintains instantaneous and smoothed QAD state in memory, reconstructing it after failover without additional storage writes.An elected leader warm-starts the exponential moving average from the first cycle’s instantaneous QAD.
  • Scheduling path: Kubernetes extension points filter feasible nodes, score interference and assurance constraints, and reserve device claims and tenant reservations atomically.Filtering accounts for CPU, memory, GPU-memory, and best-effort-cap constraints.
  • Sharing and protection: NVIDIA MPS provides spatial sharing, but DeepShare treats admission as interference control because MPS does not isolate SM or memory-bandwidth contention.A DCGM poller revokes degraded pairs and preempts the best-effort partner after three consecutive low-retention windows.
  • Resource resizing: GPU allocation cannot be modified after Pod admission, so in-place resizing uses Kubernetes’ Pod resize subresource while preserving CPU and memory headroom.The control plane enables InPlacePodVerticalScaling and maintains headroom above VPA recommendations.

C. Scheduling Latency and Fault Tolerance

The evaluation combines trace-driven replay with profiling and deployment on a real Kubernetes testbed, while the implementation keeps per-job scheduling latency below 50 ms and supports fault-tolerant operation.

  • C. Scheduling Latency and Fault Tolerance: End-to-end scheduling latency stays below 50 ms per job, with feature extraction, queue bookkeeping, and capped interference scoring taking less than 25 ms.The latency envelope remains comparable to Lucid, with Kubernetes bind round trips dominating total latency.
  • C. Scheduling Latency and Fault Tolerance: Leader election enables a replacement replica to resume scheduling within one lease-renewal period, while idempotent Pod-deletion preemption avoids bespoke mid-cycle compensation.This design supports recovery during leadership changes.
  • Experiment setup: The prototype is evaluated using a 58-node, 219-GPU production trace and a complete 16-A100 Kubernetes deployment running 50 jobs.The production cluster supports large-scale replay, while the testbed validates end-to-end behavior.
  • Experiment setup: The study profiles diverse single-GPU training workloads and uses synthetic inputs to remove data-loading and storage-I/O variability.The workload suite is summarized in Table II.
  • Experiment setup: Trace replay reproduces DeepShare’s two-level queues, DRA, preemption, and interference-aware colocation alongside FIFO, SJF, QSSF, Tiresias, and Lucid.Venus quotas are synthesized across 12 virtual clusters using historical GPU-hour shares ranging from 4 to 32 GPUs.
  • Experiment setup: The evaluation measures completion time, queueing delay, makespan, GPU and memory utilization, and QAD to compare scheduling and assurance outcomes.The Venus and internal traces provide distinct workload and quota-management settings.

B. Job Execution Time Prediction Accuracy

DeepShare improves execution-time prediction and combines that capability with interference-aware colocation to reduce queueing and idle capacity, while its largest practical gains appear in responsiveness rather than execution speed.

  • B. Job Execution Time Prediction Accuracy: DeepShare reduces prediction MAPE from 68.72% to 31.84% and raises R2 from 0.6413 to 0.7286 on Venus.Accuracy is highest for users with at least 50 historical submissions and degrades gracefully for cold-start users through a cluster-wide fallback.
  • C. Colocation Strategy Performance: GPU utilization improves by 29.5% over Lucid because more acceptable pairs are admitted, dynamic tolerance adapts to low contention, and DRA supplies additional best-effort candidates.Memory utilization gains are limited by GPU memory as a hard sharing constraint.
  • C. Colocation Strategy Performance: 46% lower average queueing delay than Lucid is obtained, from 1,976 s to 1,068 s, while average JCT improves by 6.3%.Execution time remains broadly workload-dependent, so the larger benefit is reduced waiting rather than faster running jobs.
  • C. Colocation Strategy Performance: 45.9% lower average queueing delay than Lucid is achieved by full DeepShare, while removing runtime prediction and interference awareness increases delay by 18.4% and 30.1%.The ablation identifies colocation awareness as the larger contributor.
  • C. Colocation Strategy Performance: 31% further queueing-delay reduction occurs when interference-aware colocation is added to DRA, demonstrating their reported synergy.QAD remains the primary control signal in these combined decisions.

D. Multi-Tenant Quota Management

DeepShare uses QAD-driven elastic quota borrowing to fill idle capacity while preserving reclaimability, improving the utilization–assurance trade-off across simulation and physical deployment.

  • D. Multi-Tenant Quota Management: DeepShare is the only evaluated strategy achieving average Q_i(t) = 1.0 while maintaining 1,168 s average queueing delay.Fixed-quota strategies achieve the same average assurance only with delays 36–84× higher, whereas Tiresias reaches 0.15 assurance.
  • D. Multi-Tenant Quota Management: DRA lets tenants borrow idle capacity while QAD preserves reclaimability when guaranteed demand becomes under-served.This separates quota compliance from utilization in the scheduler’s control loop.
  • D. Multi-Tenant Quota Management: DeepShare maintains fewer idle GPUs over time, confirming that elastic allocation fills resource gaps.The result is shown in the idle-GPU time series.
  • D. Multi-Tenant Quota Management: 93% of tenant-cycle pairs meet the smoothed-QAD compliance threshold of 0.95 in physical deployment.Violations cluster during burst arrivals and transient recovery after preemption events.
  • D. Multi-Tenant Quota Management: On the 16-GPU testbed, the full system reduces average JCT by 34% and queueing delay by 66% versus Hard+Colocate.It processes the same 50-job batch in 67.6% of Hard-baseline time, corresponding to approximately 1.48× throughput.

F. Sensitivity Analysis

DeepShare remains robust across key tolerance and preemption parameters, balancing utilization, job impact, and reclamation quality. Its implementation also has near-linear candidate-ranking costs, though heterogeneous accelerators require new interference-model training.

  • Dynamic tolerance baseline (ρmin): 70.58% utilization with <8% average degradation at ρmin = 0.7 provides the stated balance between cluster efficiency and individual job impact.Aggressive sharing raises utilization to 71.2% but reaches 18% degradation and harms JCT, while conservative sharing lowers utilization to 55.3%.
  • Preemption cost weight (α): α ∈ [0.3, 0.8] is consistently near-optimal across both datasets, with default α = 0.5; β is similarly robust across [0.1, 0.6].Higher α values reduce repeated preemptions of the same job but can cause suboptimal resource reclamation.
  • Scalability and limitations: Candidate ranking costs O(n log n) for victims and O(m log m) for colocation, while interference inference costs O(1) per candidate pair.These costs characterize the scheduling and colocation decision overheads.
  • Scalability and limitations: Extending DeepShare to heterogeneous accelerators requires retraining the interference model on per-device profiling data.The limitation concerns adaptation across accelerator types rather than the reported parameter robustness.
Loading 2609.16682v1…