PaperlayerView exact revision on arXiv

Source-linked AI summary

Mosaic: GPU Sharing with Latency Guarantees through Kernel-Level Interference Prediction

Foteini Strati, Ethan Graham, Leo Stephan, Paul Elvinger, Ana Klimovic

arXiv:2610.07504v1cs.DC

TL;DR

GPU underutilization motivates colocation, but resource contention complicates latency guarantees and current prediction approaches have important limits. Mosaic models kernel interference using analytical and lightweight learned methods, and MosaicSched applies its predictions to admission control and colocation choices. Mosaic reduces prediction error across four GPU types, while MosaicSched keeps p99 latency below or very close to target SLOs across workloads.

  • Problem

    GPU colocation can improve utilization but contention degrades performance, and existing heuristics and predictors do not capture GPU interference accurately enough for latency guarantees.

    Source: §1 · pp. 1–2

  • Method

    Mosaic models kernel-level interference by mechanism with analytical and lightweight learned models, and MosaicSched uses predictions for online admission control and colocation-mode selection.

    Source: §1 · p. 2

  • Results

    Mosaic reduces prediction error by up to an order of magnitude across four GPU types, and MosaicSched keeps p99 latency below or very close to target SLOs across workloads.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MosaicSched admits best-effort work whenever headroom allows while maintaining explicit user-defined performance guarantees for high-priority jobs.

    Source: §8 · p. 12

    Mosaic's thread-block placement predictor assumes equal CUDA stream priorities; supporting stream priorities is future work.

    Source: §7.1 · p. 11

Abstract

from arXiv · show

GPUs are increasingly in demand for AI workloads, yet often remain substantially underutilized, motivating workload colocation. However, colocation introduces interference that can degrade latency-critical workloads. Existing approaches mitigate interference using either heuristic-based scheduling or interference predictors. Heuristics rely on coarse-grained metrics that overlook important interference sources, while predictors often depend on simulator-specific or similarly coarse metrics. However, GPU interference is complex and arises from multiple mechanisms, including thread-block placement, memory hierarchy contention, and intra-SM resource contention. We present Mosaic, a kernel-level interference predictor that explicitly models these mechanisms using a combination of analytical and lightweight learned models. Across four GPU architectures, Mosaic reduces prediction error by up to an order of magnitude compared to prior predictors. We integrate Mosaic into a scheduler, MosaicSched, that performs online kernel admission control and selects between full-GPU colocation and SM partitioning to maximize best-effort throughput while satisfying latency SLOs. Across all workloads, MosaicSched keeps the p99 latency below or very close to the target SLO.

1 Introduction

GPU underutilization motivates colocation, but contention makes latency guarantees difficult for existing schedulers. Mosaic predicts kernel interference from its constituent mechanisms, while MosaicSched uses those predictions to admit best-effort work and select a colocation mode subject to latency SLOs.

1 Introduction

  • GPUs can remain substantially underutilized despite high demand, creating cost and energy inefficiencies.A production ads inference service study reported 20%-40% utilization, and Microsoft reported less than or equal to 50% across 400 jobs.Source: §1 · p. 1
  • Heuristic schedulers use coarse resource metrics, yet neither Orion nor Green-Context-based scheduling preserves high-priority performance in every colocation scenario.For ResNet50, p99 latency can reach 3.4× the ideal no-colocation scenario.Source: §1 · p. 1, Figure 1 · p. 1
  • Existing predictors rely on simulator-specific or coarse hardware metrics, although interference involves block placement, memory hierarchy contention, and intra-SM resource contention.Mosaic models interference sources separately using analytical and lightweight learned models.Source: §1 · p. 2
  • Across four GPU types, Mosaic reduces prediction error by up to an order of magnitude compared to prior predictors.The evaluated GPUs are H100, A100, V100, and RTX-3090.Source: §1 · p. 2
  • Across all workloads and SLO targets, MosaicSched keeps p99 latency below or very close to the target SLO.It performs online admission control for best-effort kernels and chooses between full-GPU colocation and SM partitioning.Source: §1 · p. 2
  • Neither colocation mode is universally optimal: full-GPU sharing can increase interference, while SM partitioning limits best-effort throughput.Sharing can also lose its cost advantage when best-effort work must be throttled extensively to meet the high-priority latency target.Source: §1 · p. 2

2 Background

The background outlines NVIDIA GPU organization and CUDA execution, including how SM resources, streams, blocks, warps, and memory systems shape concurrent kernel execution.

GPU architecture

  • NVIDIA GPUs group Streaming Multiprocessors (SMs) into Texture Processing Clusters, while SMs share L2 cache and off-chip memory.Each SM contains subpartitions with warp schedulers, registers, and execution pipelines, plus configurable L1/shared memory.Source: §2 · p. 2

GPU programming model

  • CUDA streams order operations within a stream but allow operations from different streams to execute concurrently; Green Contexts restrict a stream to a subset of SMs.Kernel launches specify grid and block sizes, and by default blocks can execute on any available SM.Source: §2 · p. 2

Thread scheduling

  • Block residency per SM is constrained by threads, registers, and shared memory, so kernels with more blocks execute in successive waves.Blocks remain on their assigned SM until completion.Source: §2 · p. 2

GPU colocation schedulers

  • The scheduler comparison indicates that no listed scheduler captures all main sources of GPU interference.The table notes that TPC partitioning, as in LithOS, makes intra-SM interference irrelevant.Source: Table 1 · p. 3

Thread scheduling

  • Warp schedulers select eligible warps for execution, while warps can stall on dependencies, barriers, or execution-pipeline contention.Warps are typically 32 threads and are assigned to SM subpartitions after their block is scheduled.Source: §2 · p. 3

Memory subsystem and access

  • Shared memory is private to each SM, whereas global-memory accesses use per-SM L1 caches and a shared L2 cache.Shared-memory bank conflicts serialize accesses, and global-memory requests are served in 32-byte sectors.Source: §2 · p. 3

3 Why existing GPU schedulers are not enough

Existing GPU colocation approaches do not adequately model interference: heuristics use limited metrics, and predictors face generality or measurement limitations.

3 Why existing GPU schedulers are not enough

  • Colocation can increase execution times through contention for shared GPU resources, making accurate kernel-interference modeling important for effective scheduling.The section motivates colocation by persistent underutilization and associated cost and energy inefficiencies.Source: §3 · p. 3

3.1 Heuristic-based GPU schedulers

  • Heuristics consider selected metrics such as occupancy, bandwidth, or SM demand, but miss interference sources and cannot predict scenario-specific slowdown reliably.GEMV slowdown from inter-SM interference ranges from 1.02× to 2.15× across the described RTX-3090 colocation cases.Source: §3.1 · p. 3

3.2 Existing interference predictors

  • Existing predictors are limited by workload-specific features, simulator-only metrics, or coarse counters, while CPU techniques do not directly translate to GPUs.The section motivates a hardware-based kernel-level predictor that generalizes across applications and captures diverse resource contention.Source: §3.2 · pp. 3–4

4 Mosaic: GPU kernel interference predictor

Mosaic predicts kernel slowdown under colocation from isolated kernel profiles, combining separate models for placement, block packing, inter-SM interference, and intra-SM interference. It applies these components according to whether kernels use strict SM separation or full-GPU execution.

4 Mosaic: GPU kernel interference predictor

  • Mosaic takes two kernels and their isolated profiles to predict each kernel’s slowdown, combining four interference models with analytical and lightweight learned components.Its supported modes are strict SM separation and full-GPU execution.Source: §4 · p. 4

4.1 Modeling thread block placement

  • Mosaic models block placement by scheduling A first, then placing B as resources become available under SM capacity and carveout constraints.It estimates variation in A’s block completion times to predict when SM resources are released.Source: §4.1 · p. 4

4.2 Modeling block packing

  • Block-packing slowdown depends on B’s resident-block density and total active blocks, with kernel sensitivity varying by compute and memory characteristics.Fewer active blocks can reduce memory-system contention for memory-bandwidth-bound kernels.Source: §4.2 · p. 5

4.3 Modeling Inter-SM interference

  • Inter-SM prediction uses lightweight XGBoost to combine read/write traffic, L2 and HBM bandwidth, and latency-tolerance features because aggregate throughput alone misses interference.The model includes eligible warps, stall behavior, and relative compute- and LSU-pipeline utilization.Source: §4.3 · p. 5

4.4 Modeling Intra-SM interference

  • Intra-SM slowdown combines analytical pipeline and warp-scheduler estimates with an XGBoost residual for complex shared-memory and cache effects.The analytical estimate takes the maximum of pipeline and IPC-based slowdown; the learned residual is used when both kernels are non-compute-bound.Source: §4.4 · p. 6

4.5 Putting everything together

  • Mosaic reconstructs execution timelines differently by mode: separated kernels overlap from launch, while full-GPU execution includes placement-dependent non-overlap and overlap phases.The full-GPU timeline accounts for the interference slowdowns and, for B, block-packing slowdown.Source: §4.5 · pp. 6–7

5 MosaicSched

MosaicSched uses Mosaic predictions to admit best-effort kernels under a high-priority p99 slowdown target, then selects full-GPU sharing or SM separation to maximize best-effort throughput.

5 MosaicSched

  • MosaicSched targets maximum best-effort throughput while keeping the high-priority job's p99 slowdown within a user-specified tolerance.It takes the jobs, slowdown tolerance, and profiling information as inputs.Source: §5 · p. 7
  • Built on Orion, MosaicSched retains runtime per-kernel interception, replaces Orion's scheduling policy with Mosaic-based admission control, and can integrate with schedulers operating at other granularities.It tracks kernel completion with CUDA events and uses Green Contexts for SM separation.Source: §5 · p. 7

5.1 GPU scheduling with Mosaic-enabled predictions

  • A best-effort kernel is admitted only if Mosaic's predicted high-priority completion time stays within the slowdown budget, accounting for completed, in-flight, overlapping, and remaining work.Under strict SM separation, the scheduler separately accounts for slowdown from restricting the high-priority workload to its assigned SMs.Source: §5.1 · pp. 7–8

5.2 Selecting the GPU execution mode

  • Offline profiling selects the full-GPU or SM-separated configuration with the highest best-effort throughput, testing SM allocations in multiples of eight and keeping allocations fixed across consecutive high-priority kernels.The fixed allocation avoids overhead from frequent Green Context transitions.Source: §5.2 · p. 8

5.3 Minimizing prediction overheads

  • Prediction latency is approximately 1 μs for inter-SM prediction and 7–8 μs for full-GPU prediction.MosaicSched also caches predictions for unique kernel pairs because ML workloads repeat kernels.Source: §5.3 · p. 8

6 Evaluation

The evaluation tests MosaicSched’s SLO and throughput behavior, colocation cost benefits, predictor accuracy, and predictor-component ablations. Predictor evaluation spans four GPU types, while MosaicSched is evaluated on H100.

6.1 Methodology

  • Predictor evaluation spans H100, RTX-3090, A100, and V100, while MosaicSched is evaluated on H100 across inference workloads and predictor baselines KACE and GPUPool.The evaluated workloads include ResNet50, MobileNet-v3, DLRM, BERT, and LLAMA-3-8B.Source: §6.1 · p. 8

6.2 MosaicSched evaluation

  • The preferred mode depends on workload pair and SLO: MosaicSched chooses full-GPU sharing for all Figure 6 workloads at p=1.1, but favors SM separation for DLRM.For DLRM, it allocates 64–96 SMs to the HP job; SM allocation trades reduced interference against resources available to BE kernels.Source: §6.2.3 · pp. 9–10
  • Across workloads, MosaicSched keeps p99 latency at or near the specified tolerance, with at most 1.0% of requests exceeding the target except MobileNet at p=1.6, where 1.2% exceed it.The evaluation measures p99 SLO violations as the fraction of requests exceeding the user-specified slowdown tolerance.Source: §6.2.1 · p. 9, Figure 6 · p. 10
  • At p=1.25 and p=1.6, MosaicSched achieves higher BE throughput than Orion while maintaining lower HP p99 latency; at p=1.6, all workloads have mean cost savings above 1.At p=1.1, tight latency tolerance often leaves little or no cost benefit; BE throughput rises with greater SLO slack.Source: §6.2.2 · p. 9, Figure 6 · p. 10, Table 3 · p. 10

6.3 Mosaic Predictor accuracy

  • With separate SMs, Mosaic cuts p50 error 3.2–6.3× versus KACE and 2.2–4.0× versus GPUPool, and p95 error 2.1–3.5× and 1.45–3.0×, respectively.These are average absolute percentage errors on unseen kernels.Source: §6.3.1 · p. 10, Table 4 · p. 11
  • With shared SMs, Mosaic reduces p50 and p95 errors by 7–82× across GPUs by modeling thread-block scheduling, block packing, and intra- and inter-SM interference.The comparison is on unseen kernels under default CUDA streams.Source: Table 5 · p. 11, §6.3.2 · p. 11

6.4 Predictor ablation study

  • In H100 full-GPU ablations, adding block packing reduces p50 error from 15% to 8.7%; the full predictor reaches 8.0% p50 and 48.2% p95 error.Accurate thread-block placement alone also performs substantially better than KACE and GPUPool.Source: Table 6 · p. 11, §6.4 · p. 11
  • Intra-SM components are complementary: removing the pipeline/warp model raises compute-heavy p95 error from 46.8% to 80.0%, while memory/L2-only inter-SM features raise H100 p95 error from 23% to 44%.The XGBoost memory model matters most for global- and shared-memory-heavy kernels.Source: §6.4 · p. 11, Table 7 · p. 11

7 Discussion

The discussion identifies limitations in power-throttling modeling, stream-priority support, and per-GPU calibration, and considers adapting Mosaic to AMD GPUs.

7.1 Limitations of the Mosaic predictor

  • Mosaic does not explicitly model power throttling, which can reduce GPU frequency and increase kernel execution time.Throttling is relatively infrequent but varies across GPU generations and systems, and further experimentation is needed to model it.Source: §7.1 · p. 11
  • The thread-block placement predictor assumes equal CUDA stream priorities, while priorities can alter block scheduling and execution time.Supporting stream priorities is future work.Source: §7.1 · p. 11
  • Inter- and intra-SM XGBoost models and analytical parameters are calibrated per GPU, so new GPU generations require retraining.Reducing this calibration overhead is future work.Source: §7.1 · p. 11

7.2 Applicability to AMD GPUs

  • Several components transfer to AMD GPUs, but adapting Mosaic requires accounting for AMD’s greedy thread-block scheduler, which can mix blocks from different kernels.L2-cache and memory interference can be modeled accurately using XGBoost and existing features.Source: §7.2 · p. 12

8 Conclusion

Mosaic combines analytical and learned models in a lightweight kernel-level GPU interference predictor, and MosaicSched uses it to provide performance guarantees while admitting best-effort work when headroom allows.

8 Conclusion

  • Mosaic achieves up to an order of magnitude lower p50 and p95 error than prior predictors, with per-prediction latency in the microsecond range.MosaicSched enables user-defined performance guarantees for high-priority jobs while admitting best-effort work whenever headroom allows.Source: §8 · p. 12

A Appendix

The appendix describes a diverse dataset of kernels used to train and test Mosaic, spanning four GPU types and four kernel categories.

A Appendix

  • The dataset covers H100, A100, RTX-3090, and V100 GPUs and memory-bandwidth-bound, compute-bound, global-memory-latency-intensive, and shared-memory-intensive kernels.The kernels vary in resource requirements, size, and execution time.Source: A Appendix · p. 14
  • Training and testing kernels come from, CUDA Samples, cuDNN, cuBLAS, FlashInfer [41], and custom microbenchmarks.Input sizes and parameters are varied to change kernels’ resource requirements and sensitivity to interference.Source: A Appendix · p. 14
  • Table 8 lists the different kernels used for training and testing the Mosaic predictor.Its caption names kernels from, cuBLAS, cuDNN, FlashInfer [41], and custom microbenchmarks.Source: A Appendix · p. 14, Table 8 · p. 15
Loading 2610.07504v1…