Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

1 Introduction

• Exclusive GPU allocation can leave compute or memory unused when a job needs only a fraction, while temporal or concurrent sharing could recover that capacity.MPS and MIG offer sharing mechanisms, but their support involves tradeoffs in allocation flexibility, performance isolation, and memory management.Source: §1 · pp. 1–2
• GPU memory constrains sharing because combined job demands must be managed; device allocations are not automatically pageable, while Unified Memory oversubscription can incur page-fault and migration costs.These constraints motivate examining when resources are unused, how long those intervals last, and which workloads could share effectively.Source: §1 · p. 2
• 100 ms monitoring across a university cluster addresses the limited fine-grained utilization insight provided by prior monitoring at seconds-to-minutes intervals.The tool ran on 89 nodes and 319 GPUs for several months; the paper analyzes a 10-day period covering 7,399 jobs.Source: §1 · p. 2
• Over 70% of jobs used less than 20% of GPU memory, and 50% averaged below 5% SM utilization; about 80% had a major requested-versus-actual duration mismatch.The examined multi-GPU jobs also showed highly periodic communication patterns.Source: §1 · p. 2
• Fine-grained monitoring insights could inform systems-layer scheduling and multitasking that improve utilization while preserving performance isolation, complementing hardware and application optimization.The paper describes this systems-layer opportunity as “plenty of room in the middle.”Source: §1 · p. 2

2 Monitoring Framework and Environment

• The tool was deployed on a university research cluster comprising 89 nodes: 20 high-performance nodes with two NVIDIA A100 PCIe 40 GB GPUs each, and 69 standard nodes with four NVIDIA A100 SXM4 80 GB GPUs each.The study was conducted in fall 2024.Source: §2 · p. 2, §2.1 · p. 3

2.1 GPU Cluster Overview

• The cluster’s 20 high-performance nodes each had two A100 PCIe 40 GB GPUs, while 69 standard nodes each had four A100 SXM4 80 GB GPUs.All nodes had a 100 Gb/s InfiniBand NIC and a 1 Gb/s Ethernet NIC for out-of-band traffic.Source: §2.1 · p. 3

2.2 Design Goals and Challenges

• DCGM’s SM Activity metric provides a more precise measure of compute utilization than nvidia-smi, which can report 100% utilization when one of an A100’s 108 SMs is used.The comparison illustrates how nvidia-smi can miss fractional SM usage and distort utilization analysis for multitasking.Source: §2.2 · p. 3, Figure 1 · p. 4

2.3 Design, Implementation and Deployment

• The host-level tool queries DCGM and /proc, buffers measurements for database writes, and collects scheduler job information separately at a lower frequency.At a 100 ms sample period, a four-GPU host incurs about 0.9% overhead on one CPU core; GPU overhead is negligible.Source: §2.3 · pp. 3–4
• Per-host SQLite databases avoid synchronization across hosts and make the tool immune to host failures, although analysis requires offline aggregation.The tool is a standalone executable written in a few hundred lines of C.Source: §2.3 · p. 4

3 Job Characteristics

• During the 10-day period, 7,399 jobs came from 150 users across 12 departments, with Chemistry and Chemical & Biological Engineering dominating reserved GPU time.Table 1 summarizes jobs and reserved GPU time by department, omitting departments with at most 0.5% of overall GPU time.Source: Job Distribution by Department · p. 4, Table 1 · p. 5

Job Duration and Categorization

• Each of the four elapsed-time categories represents roughly 25% of jobs, and the CDF’s discrete jumps suggest jobs often use specific common time limits.The categories are short (<600 seconds), medium (600–7200 seconds), long (7200–28800 seconds), and very long (>28800 seconds).Source: Job Duration and Categorization · p. 5, Figure 2 · p. 5

GPU Resource Requests

• 95% of jobs requested one GPU, while multi-GPU and multi-node jobs used a disproportionate share of GPU-time and faced longer queues.The median queue time was nearly 2 hours for single-GPU jobs and 2 days for multi-GPU requests.Source: GPU Resource Requests · p. 6, Figure 3 · p. 6, §4 · p. 6

4 Our Findings

• 29% GPU compute utilization and 16% GPU-memory utilization were the cluster-wide averages over the 10-day sample.CPU resources also remained largely unused.Source: §4 · p. 6, Figure 4 · p. 7

4.1 Significant Mismatch between Requested vs. Actual Durations

• About 80% of jobs had requested durations far from their actual runtimes; only about 20% were within a factor of two.Over-requesting can increase queue delays because schedulers allocate larger time slots.Source: §4.1 · p. 7, Figure 5 · p. 7
• 34.17% of jobs reached their requested time limit, while 60.7% completed successfully.Some timeouts are intentional, such as interactive or checkpointed jobs that run until their limit.Source: §4.1 · p. 7, Table 2 · p. 8
• Fully idle jobs consumed 2.82% of reserved GPU-time, including 2.15% allocated to jobs idle for more than 8 hours.Of 1,043 fully idle jobs lasting under a minute, 1,041 completed successfully, likely reflecting tiny reservations or configuration errors.Source: §4.1 · pp. 7–8, Table 3 · p. 8

4.2 Great Opportunities for Multitasking

• Fixed-duration, whole-GPU reservations can lock away underused resources, motivating examination of GPU sharing and multitasking.The paper notes that many reservations are substantially underutilized.Source: §4.2 · p. 9, Low SM and Memory Utilizations · p. 9
• 50% of jobs averaged below 5% SM utilization, while 75% had peak memory usage below 20% and average SM usage below 20%.The analysis covers single-GPU jobs and stratifies utilization metrics by job duration.Source: Low SM and Memory Utilizations · p. 9, Figure 6 · p. 9
• Only 5% of jobs ever reached 75% of GPU memory capacity, and many jobs maintained stable memory usage for extended periods.Among jobs running over 8 hours, just 10% exceeded 90% of memory capacity; stable usage may reduce memory churn in multitasking.Source: Low SM and Memory Utilizations · p. 9, Figure 7 · p. 10

Fine Temporal Granularity

• 100 ms sampling captures SM-utilization peaks and troughs that 1-second and 10-second sampling can mask.The comparison covers SM usage and NVLink throughput.Source: Fine Temporal Granularity · p. 10, Figure 8 · p. 11, Figure 9 · p. 11
• NVLink traffic shows bursty patterns at fine-grained timescales, while coarser sampling can mask fluctuations.Source: Fine Temporal Granularity · p. 10, Figure 8 · p. 11
• Fine-grained utilization data can let a scheduler enqueue additional tasks onto a GPU before the current job finishes.Source: Fine Temporal Granularity · p. 10

Implications for Multitasking

• Over 70% of jobs consume less than 20% of SMs and memory, motivating dynamic multitasking to reclaim underused GPU resources.Source: Implications for Multitasking · p. 11
• Single-GPU jobs are candidates for multiplexing because they are not gang-scheduled and incur no inter-GPU traffic.They account for a substantial fraction of aggregate cluster usage.Source: Implications for Multitasking · p. 11
• Fine-grained preemptive time slices of 100–200 ms and mechanisms such as MIG, MPS, and UVM offer opportunities to explore multitasking.The paper describes these mechanisms as advancing toward greater maturity.Source: Implications for Multitasking · pp. 11–12

4.3 Highly Periodic, and Predictable, Communication Patterns

• Parallel workloads accounted for 26% of allocated GPU-time, and synchronization can make unintended interference compound into straggler effects.The communication-pattern analysis presents three representative examples.Source: §4.3 · p. 12

Single-Node Patterns

• A 4-GPU neuroscience job repeats a roughly 17-minute communication phase with bursts about once per minute, while a psychology job shows bursts at approximately 10- and 60-second intervals.The psychology workload’s denser regions still show the two distinct periodic intervals.Source: Neuroscience Job · p. 12, Psychology Job · pp. 12–13, Figure 10 · p. 12
• ≈100 Gb/s network saturation over 2-second windows bottlenecks the 4-node, 16-GPU workload, constraining both NVLink and PCIe to the network limit.The resource pattern repeats for 12 hours with almost no noise; the authors identify the job as likely doing AI training.Source: Comprehensive Picture · p. 13, Figure 12 · p. 14
• PCIe TX readings align with node network TX, while the selected GPU has no NVLink TX; its inbound PCIe traffic indicates network data is split among multiple GPUs.The selected GPU’s NVLink RX changes as it first reads at the network limit and later receives some data directly from the NIC via RDMA.Source: Comprehensive Picture · p. 13
• For every 7-second period, the selected GPU spends 1 second idle; across 16 GPUs and 12 hours, those troughs imply around 21 hours of available GPU computation.The workload’s GPU memory remains flat at 50%, leaving space for other jobs’ data.Source: Comprehensive Picture · p. 13

Implications for Distributed Jobs

• Distributed jobs can be network-bandwidth bottlenecked, leaving high-bandwidth NVLinks unsaturated and causing SM idle periods; single-node jobs can instead fully saturate NVLink during blocking communication.Insufficient interconnect performance can degrade throughput during communication-intensive phases.Source: Implications for Distributed Jobs · p. 14
• Repetitive communication may support traffic prediction and out-of-phase scheduling to help minimize contention and maintain consistent throughput.The authors also suggest transparent multitasking alongside long-running distributed jobs to improve utilization, especially in bandwidth-limited environments.Source: Implications for Distributed Jobs · p. 14
• The general challenge is minimizing interference during bursty communication periods.The comparison to prior work is identified by Table 4, but its caption alone provides no specific comparison values.Source: Implications for Distributed Jobs · p. 14, Table 4 · p. 15

5 Related Work

• DCGM supports 100–200 ms telemetry but is rarely used at scale; the study is among the first to collect DCGM SM-level metrics at 100 ms across a multidisciplinary research cluster.The authors note that the MIT SuperCloud dataset samples every 100 ms using nvidia-smi, whose utilization metric can overstate SM usage by orders of magnitude; Ganglia typically relies on coarse-grained NVML data.Source: §5 · p. 15
• The described dcgm_exporter and Prometheus setup targets Kubernetes and is non-trivial to configure, whereas this tool runs on bare-metal Nvidia data-center GPU servers without dependencies and can be started or stopped from the command line.The comparison concerns deployment and operation, not measured monitoring overhead.Source: §5 · p. 16
• Philly and Helios used 100-second NVML sampling and did not measure fractional SM utilization, while Weng et al. [2022] studied time-multiplexed sharing.The authors argue that sub-second analysis reveals internal idle phases that standard job-level metrics cannot readily detect.Source: §5 · p. 16
• Gandiva and AntMan explore GPU sharing, but AntMan’s roughly 1-second NVML monitoring may miss quickly shifting utilization patterns.The authors report that sub-second SM measurements show jobs often leave unused capacity for co-location.Source: §5 · p. 16
• Orion relies on offline profiles, while QSSF [Hu et al. 2021] assumes stable job behavior and accessible training data; the authors suggest adaptive scheduling using fine-grained metrics.The stated QSSF assumptions may not hold in research environments where development and debugging are common.Source: §5 · pp. 16–17

6 Limitations

• 100 ms sampling may smooth short-lived network and interconnect spikes, while microsecond-to-millisecond kernels require profiling tools that are not applicable cluster-wide due to job degradation.The tool supports rates 10–100× faster, but DCGM’s highest available sampling frequency was 10 Hz.Source: §6 · p. 17
• Precise Job ID-to-GPU mappings were available only for the final 10 days, so the analysis uses that clean, fully instrumented dataset rather than inferring longer-term mappings.Source: §6 · p. 17
• The dataset contains 343 multi-GPU jobs but only 7 multi-node jobs, limiting insight into domain-specific workload behavior; the analysis aggregates departments.Source: §6 · p. 17
• The analysis combines A100 PCIe 40 GB and A100 SXM4 80 GB GPUs despite their differing interconnect bandwidth; the mostly single-node workload made the impact on overall findings small.The authors identify heterogeneous hardware and network capabilities as subjects for future investigation.Source: §6 · p. 17

• Over 70% of jobs used less than 20% of GPU memory, while 50% averaged below 5% SM utilization.Source: §7 · p. 17
• 80% of jobs had a mismatch between requested and actual durations, limiting scheduler effectiveness.Source: §7 · p. 17
• Multi-GPU jobs showed highly periodic communication patterns, suggesting opportunities for predictive resource allocation and compatible multiplexing.Source: §7 · p. 17
• Emerging technologies including MIG, MPS, and UVM indicate potential to improve cluster efficiency through dynamic resource sharing and intelligent scheduling.Source: §7 · p. 17