MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning

1 INTRODUCTION

• Workloads have different resource bottlenecks and performance sensitivities, so making individual GPUs more powerful does not ensure efficient resource utilization.The paper identifies GPU resource utilization as a bottleneck despite advances in GPU technology.Source: §1 · p. 1
• Finding an effective GPU partition is challenging because configuration is cumbersome, exploration is time-consuming, and exploration incurs overhead.The authors describe optimal partitioning as non-trivial and potentially disruptive to jobs.Source: §1 · p. 1
• MISO uses interference-prone co-location to estimate near-optimal interference-free MIG partitions, avoiding extensive exploration of MIG configurations.The paper characterizes MPS and MIG as offering different partition granularity and performance isolation.Source: §1 · p. 1
• 49%, 15%, and 23% lower than the unpartitioned GPU scheme for job completion time, makespan, and system throughput, respectively; all three metrics are within 10% of Oracle.The evaluation uses representative production environments and emerging BERT and GNN workloads.Source: §1 · p. 2

2 BACKGROUND

• NVIDIA A100 resource partitioning serves as a vehicle for demonstrating MISO's core ideas, with expected relevance to future vendor capabilities.The authors describe the technology as relevant to current and future MIG-enabled cloud and HPC centers.Source: §2 · p. 2

2.1 GPU Resource Sharing

• MPS shares GPU compute units while cache and memory remain shared; MIG partitions additional resources and provides stronger isolation, but offers fixed slice sizes and requires stopping applications to reconfigure.MPS allows a new application to launch when sufficient memory exists; MIG slices provide exclusive access to their assigned resources.Source: §2.1 · p. 2

2.2 NVIDIA A100 Tensor Core GPUs and MIG Capability

• An A100 has 18 supported MIG hardware configurations, while assigning jobs to the resulting slices creates additional partition configurations.MIG slice profiles specify slice types and maximum counts; hardware restrictions rule out some combinations.Source: §2.2 · pp. 2–3

2.3 System Throughput, Job Completion Time, and Makespan

• System throughput compares jobs' combined progress under co-location with their exclusive execution rates.The paper uses STP for system throughput.Source: §2.3 · p. 3
• Average job completion time includes queue waiting and execution, while makespan spans from the first job's start to the last job's completion.The paper uses all three measures to evaluate system performance.Source: §2.3 · p. 3

3 MOTIVATION

• Emerging workloads often underuse GPU resources, motivating co-location as a way to improve utilization.The paper attributes underutilization to workload bottlenecks such as memory latency and bandwidth.Source: §3 · p. 3
• MIG sharing delivered higher system throughput than proportionally matched MPS sharing in the example, while a poorly chosen MIG partition could underperform MPS.The example attributes MIG’s advantage to performance isolation and resource exclusivity; not all MIG configurations outperform MPS.Source: §3 · p. 3, Figure 3 · p. 3
• MIG partition performance varies by job mix, and the ordering of two configurations can reverse, so the best partition is mix-specific.The two mixes are (CNN, EMB, MLP) and (MLP, DeepSpeech, GNN).Source: §3 · p. 4, Figure 4 · p. 4
• Exhaustively testing MIG configurations is costly because reconfiguration resets GPUs and requires checkpointing and restarting co-located jobs.The reported reconfiguration takes about 4 seconds, while checkpoint and restart overhead can range from seconds to minutes.Source: §3 · p. 4
• 8% to 14% lower system throughput resulted from heuristic partitions based on job memory, power consumption, or SM utilization compared with the optimal partition.The heuristics select a MIG partition by matching slice GPC counts to job characteristics using cosine similarity.Source: §3 · p. 4, Figure 5 · p. 4
• MPS profiling lets MISO estimate MIG configurations without exhaustively evaluating them, supporting selection of a near-optimal partition for higher throughput and lower job completion time.MPS is more flexible to explore, while MIG provides interference-free resource partitions.Source: §3 · p. 4

4 MISO: THE MIG SOLUTION

• MISO combines a central controller and per-GPU server APIs with a predictor that estimates job performance on MIG slices from MPS profiling, then optimizes the partition.The controller monitors the job queue and uses server status updates to make scheduling decisions.Source: §4 · p. 4

4.1 MISO Performance Predictor

• MISO’s learning-based predictor estimates per-job performance on MIG slices from co-located MPS runs, avoiding repeated MIG profiling that can require pausing other jobs.MIG-based profiling incurs up to 8× more overhead than MISO’s MPS profiling for similar scheduling quality; MISO’s profiling cost remains near-constant as job count grows.Source: §4.1 · p. 5
• The predictor uses a U-Net to translate a 3×7 MPS input matrix into a 3×7 matrix of normalized speedups on 7g, 4g, and 3g MIG slices.The model is trained on 14,000 augmented job-mix data points and reaches validation MAE 0.017 after 50 epochs.Source: §4.1 · pp. 5–6

4.2 MISO Scheduling Optimizer

• The single-GPU optimizer selects feasible MIG slices for each job to maximize summed execution speeds normalized to each job’s maximum speed.It re-partitions when a job starts or finishes and uses predicted speed information to search without experimentally reconfiguring the GPU.Source: §4.2 · p. 6, Figure 9 · p. 6
• MISO solves partitioning locally per GPU rather than globally, because the cluster configuration space grows exponentially and job migration adds checkpointing overhead.Each A100 has 18 MIG configurations and supports at most 7 jobs; the optimizer’s maximum runtime in experiments was 0.5 ms.Source: §4.2 · p. 7

4.3 Miscellaneous Design Considerations

• For job placement, MISO uses an FCFS queue, starts new jobs on the least-crowded GPU, and re-profiles a running job after a significant execution-speed change.Configurable thresholds and historical data balance the cost of re-invocation against the performance benefit of repartitioning.Source: §4.3 · p. 7

4.4 Implementation

• MISO’s implementation uses MPS and MIG APIs, a central controller, and per-GPU server APIs that report job and partition status; users need not change their job code.MISO sets job environment variables to control MPS and MIG assignments.Source: §4.4 · p. 8

• The real-system testbed has eight A100-PCIe-40GB GPUs and 56 MIG slices, allowing it to serve up to 56 jobs at once.Source: §5 · p. 8
• The evaluation models production behavior with a Helios-derived 100-job test mix and uses 1,000-job mixes across 1,000 randomized simulation trials.Test jobs are limited to two hours and arrive with a Poisson rate parameter of 60 seconds; simulation uses 10 seconds.Source: §5 · p. 8
• Deep-learning workloads span computer vision, language modeling, speech recognition, and scientific computing, with models and batch sizes sampled from Table 2.Source: §5 · p. 8, Table 2 · p. 8
• NoPart leaves GPUs unpartitioned, while OptSta applies one exhaustively selected static MIG configuration across GPUs and job mixes.Source: §5 · pp. 8–9
• Oracle selects a different best dynamic MIG configuration for each mix using offline job-profile information, and comparisons track average JCT, makespan, and system throughput.OptSta and Oracle omit profiling and switching overhead, whereas MISO includes it.Source: §5 · p. 9

6.1 Real System Evaluation and Analysis

• MISO reduces average JCT by 16% versus OptSta, shortens makespan by 23%, and raises system throughput by 35%; it is within 10% of Oracle on makespan and 7% on throughput.Source: §6.1 · p. 9
• For individual jobs, 50% of MISO jobs stay within 1.5× ideal JCT, compared with less than 30% for NoPart and OptSta.Ideal JCT is relative to isolated, interference-free execution on the full GPU.Source: §6.1 · p. 9, Figure 11 · p. 9
• NoPart jobs spend over 60% of their time queued and OptSta jobs about 5%, while MISO and Oracle eliminate queue time; MPS profiling accounts for 12% of MISO job time.MPS-based exploration keeps checkpointing to 3%, versus more than 20% for exhaustive MIG profiling.Source: Figure 12 · p. 9, §6.1 · p. 10
• As scheduled jobs increase from one to ten on a GPU, MISO’s advantage over NoPart widens, although OptSta can occasionally outperform MISO for particular mixes.MISO and Oracle results overlap for most job mixes.Source: §6.1 · p. 10, Figure 13 · p. 10
• Halving MPS profiling time increases prediction error, while extending it to 1.5× can degrade JCT by 4% because longer profiling brings diminishing accuracy gains.Source: §6.1 · p. 10, Figure 14 · p. 10
• MISO improves average JCT by 35% over an MPS-only baseline, and 80% of its jobs have less than 2× ideal JCT degradation versus 30% on MPS-only.Source: §6.1 · p. 10, Figure 15 · p. 10

6.2 Simulation Evaluation and Analysis

• In 40-GPU, 1,000-job simulations, MISO’s median improvements over NoPart are 70% for JCT, 20% for makespan, and 30% for system throughput, remaining close to Oracle across runs.Source: §6.2 · p. 11, Figure 16 · p. 11
• Doubling checkpoint overhead leaves gains intact, prediction errors of 1.7%–9% retain comparable gains, and across arrival rates MISO improves JCT 30%–50%, makespan over 15%, and throughput over 25% versus NoPart.Source: §6.2 · p. 11, Figure 17 · p. 11, Figure 18 · p. 11

• CPU-server co-location methods do not address GPU-specific challenges such as architecture, sharing granularity, and allowable partitions.Source: §7 · p. 11
• GPU-sharing work includes CUDA-stream scheduling, time sharing, and MPS space sharing, but these approaches do not tackle MIG-enabled sharing’s specific challenges.Source: §7 · p. 11
• Abacus and Zahaf et al. use naive static MIG partitions, while MISO is presented as the first work addressing MIG-enabled datacenter operation to improve JCT and throughput.Source: §7 · pp. 11–12

• Partitioning support may extend beyond A100: Hopper retains MIG support, while AMD and Intel are also pursuing GPU-resource partitioning approaches.Source: §8 · p. 12
• Future GPUs may exceed today’s 18 MIG combinations, potentially affecting prediction accuracy and optimizer work; the optimizer took under 80 ms with 180 combinations.The paper reports that it scales linearly with combinations for a given co-location degree and finishes within a second at a 100× increase.Source: §8 · p. 12
• Cloud and HPC operators could expose heterogeneous sub-GPU partitions as job units, while users could use MISO’s predictor to estimate performance and providers could price those units.Source: §8 · p. 12

9 CONCLUSION

• 49% lower average job completion time than the unpartitioned GPU scheme, and within 10% of the Oracle technique.Source: §9 · p. 12
• MISO dynamically partitions NVIDIA A100 GPU resources among co-located jobs.Source: §9 · p. 12
• A learning-based method quickly finds the optimal MIG partition for a given job mix running in MPS.Source: §9 · p. 12

APPENDIX

• Figure 20 shows all 18 possible MIG configurations in an A100 GPU.Source: APPENDIX · p. 12, Figure 20 · p. 12
• Each row represents one configuration; the second row, for example, represents (4g.20gb, 2g.10gb, 1g.5gb).Source: APPENDIX · p. 12