PaperlayerView exact revision on arXiv

Source-linked AI summary

Measurement-Driven Diagnosis and Mitigation of Host-CPU Co-location Interference in Single-GPU LLM Serving on a Multi-GPU Server

Guanjie Cheng, Guowei Li, Yingying Wen, Xinkui Zhao, Zhe Liu, Shuiguang Deng

arXiv:2609.05425v1cs.DC

TL;DR

External CPU workloads can disrupt the CPU-side serving path of single-GPU LLM inference, while existing work offers limited guidance on locating this interference and selecting protections. The paper combines macro metrics with NVTX/Nsight tracing, introduces CPTI and CTS, and builds CoTail to diagnose tails and validate OS-level protections. Its measurements show that service-stage tails before GPU submission are stronger degradation signals than CUDA/GPU metrics, motivating a same-platform, workload-aware deployment procedure.

  • Problem

    Existing studies provide limited guidance on how realistic external CPU workloads degrade LLM serving, where the degradation arises, and which protections operators should select.

  • Method

    The paper combines macro serving metrics with NVTX/Nsight tracing, introduces CPTI and CTS, and uses CoTail to screen workloads, diagnose tails, select protections, and validate decode SLOs.

  • Results

    Service-stage NVTX tails provide stronger diagnostic signals than CUDA/GPU metrics, with top correlations exceeding |ρ| > 0.98 versus CUDA/Nsight correlations of |ρ| ≤0.470.

  • Takeaways & Limitations

    CoTail provides a same-platform deployment procedure that selects OS-level protections and validates them against a common-baseline decode SLO.

  • Takeaways & Limitations

    CoTail thresholds are calibrated on the primary platform and should be recalibrated for different CPU topologies, interconnects, kernels, inference engines, or workload mixes.

Abstract

from arXiv · show

Host CPUs in GPU servers are often under-used during LLM inference. Co-locating CPU workloads can improve resource use, but it can also seriously hurt serving quality. Existing work mainly improves LLM serving engines or studies CPU-GPU boundary delays. It gives limited guidance on how external CPU workloads affect the serving path and how operators should choose protection policies. This paper studies host-CPU co-location interference in single-GPU LLM serving. We show that the main observed problem is not slower GPU kernels. Instead, CPU workloads amplify long tails in CPU-side serving stages before GPU work is submitted. To capture this effect, we introduce the Core Path Tail Index (CPTI) and Core Tail Suppression (CTS). Based on these metrics, we build CoTail, a measurement-driven diagnostic procedure that screens workload risk, profiles serving-stage tails, selects OS-level protections, and validates decode SLO compliance. In our primary setup, unprotected nginx co-location reduces throughput by 78.8%, increases TTFT by 429.5%, and increases TPOT by 362.4%. CoTail-guided protections improve nginx throughput by up to 4.4x and reduce TPOT by 4.5x. Under a common-baseline deployment SLO, CoTail satisfies all 12 oracle-feasible held-out cases, compared with 10/12 for Always-rt and 11/12 for Macro-only. It also reduces RT usage from 28 to 22 cases and lowers mean co-tenant slowdown from 56.65% to 51.21%.

1 Introduction

The paper studies how external CPU workloads disrupt single-GPU LLM serving and develops CoTail to diagnose service-stage tails and select protections. Its central finding is that degradation often arises before GPU work is submitted, rather than from slower GPU kernels.

  • Motivation: CPU co-location can improve use of under-used host CPUs but severely degrade throughput, TTFT, and TPOT.The problem concerns interference in the CPU-side serving path of GPU-based LLM inference.
  • Research gap: Existing studies provide limited guidance on degradation severity, its location in the serving path, and protection selection for realistic external CPU workloads.The paper frames these as unresolved operator-facing questions.
  • Research questions: The paper asks how severe co-location interference is, where it arises, and how operators should choose protections.These questions cover performance recovery, diagnosis, and deployment decisions.
  • Key finding: Severe degradation is often caused by CPU-side service-stage tail amplification before GPU submission, not slower GPU kernels.Affected stages include scheduler, batch construction, execution-wrapper, and model-forward tails.
  • CoTail: CoTail profiles workloads, screens risk, diagnoses service-stage tails, selects OS-level protections, and validates them against a decode SLO.It is presented as a measurement-driven operator procedure for the studied server setting.
  • Metrics: The paper introduces CPTI and CTS to summarize service-stage tail amplification and protection-mediated suppression.These metrics link macro-level degradation, NVTX-observed serving stages, and protection effects.

2 Background and Experimental Methodology

The methodology combines CPU-side serving-path instrumentation with macro-level serving measurements across workloads, systems, and OS protections. Experiments isolate how scheduling, cache, memory, I/O, and NUMA effects can disrupt GPU-serving control paths.

  • Serving path: LLM serving repeatedly schedules requests, constructs batches, invokes execution, and submits GPU work through a CPU-side loop.The study instruments four core stages: scheduler.step, batch.construct, model.execute, and model.forward.
  • Experimental setup: The experiments use single-GPU serving on an 8× RTX 4090 server with a dual-socket AMD EPYC 7T83 NUMA CPU.The primary system uses vLLM with DeepSeek-R1-Distill-Qwen-7B; held-out systems include other models and SGLang.
  • Protections: The study compares no protection with nice, cgroup CPU weighting, EngineCore-targeted real-time scheduling, and NUMA isolation.Real-time scheduling targets the framework-specific EngineCore thread, while NUMA isolation separates local and remote sockets.
  • Measurement: Primary macro results average 10 completed serving runs and use 95% bootstrap confidence intervals with 10,000 resamples.Baseline and co-location runs are resampled within each protection mode before recomputing degradation.
  • Cost accounting: Co-tenant slowdown is measured as percentage utility loss relative to the CPU-alone workload baseline.The analysis avoids Nsight Systems during this cost-aware utility measurement to prevent profiler-induced interference.

3 Macro Impact of CPU Co-location

CPU co-location produces workload- and metric-dependent serving degradation, with nginx showing the most severe unprotected impact. Real-time scheduling mainly protects throughput and TPOT, while NUMA isolation is often better for TTFT and locality-sensitive workloads.

  • Workload dependence: CPU co-location risk depends strongly on workload type, with stress-ng, openssl, kernel_build, and memcached each reducing throughput by more than 45% unprotected.7zip produces the mildest throughput degradation at 5.4%.
  • Protection effects: Real-time scheduling gives the best throughput and TPOT for six of eight workloads, keeping losses below 7% and TPOT increases below 6% in those cases.The pattern indicates sensitivity of steady-state decoding to timely CPU-side scheduling and runtime-control paths.
  • Protection effects: NUMA isolation achieves the lowest TTFT increase for six workloads and reduces ffmpeg TTFT increase to 9.4%, versus 103.6% unprotected and 58.4% under rt.It also provides the best throughput and TPOT for ffmpeg.
  • Combined protection: The combined rt+numa policy keeps throughput loss below 6% and TPOT increase below 8% for ffmpeg, kernel_build, and nginx.This targeted result addresses workloads sensitive to both scheduling and topology.

4 Diagnosing the Dominant Service-Path Bottleneck

CPU co-location interference is primarily localized to long tails in the CPU-side serving path before GPU submission, which CUDA-runtime and GPU-execution metrics only partially capture. CPTI summarizes this tail amplification across core serving stages to diagnose macro-level degradation.

  • CUDA/Nsight signals provide only partial explanations because they mainly observe events near or after kernel submission, missing upstream scheduling and batching delays.
  • |ρ| ≤0.470 for the strongest CUDA/Nsight correlations, versus |ρ| > 0.98 for top service-stage correlations with macro-level degradation.GPU execution P99 can even correlate negatively with throughput drop and TPOT increase when fewer or smaller batches reach the GPU.
  • 78.8% throughput drop and 362.4% TPOT increase occur in unprotected nginx despite only an 8.0% increase in total TKLQT.Upstream serving-loop stalls can reduce continuously in-flight kernels, leaving launch and queuing metrics weakly changed.
  • The diagnostic procedure therefore inspects the service-stage path upstream of CUDA submission rather than relying only on kernel-level metrics.Cross-workload correlations and intervention checks support the service-control path as the observed localization of macro-level degradation.
  • CPTI measures positive P95/P99 tail amplification across scheduler.step, batch.construct, model.execute, and model.forward.The score uses normalized stage latencies against the same-protection no-interference baseline; improvements do not contribute negative amplification.
  • CPTI ranges from 0.12 for 7zip to 12.50 for nginx, with batch.construct dominating high-degradation cases.Nginx reaches CPTI = 12.50, while memcached and kernel_build reach 7.06 and 6.88; ffmpeg instead is dominated by model.forward.

5 From Service-Stage Diagnosis to Deployment

CoTail converts service-stage tail diagnosis into a same-platform deployment procedure that selects and validates OS protections under a common-baseline decode SLO. Held-out evaluation shows stronger SLO coverage and lower recurring protection costs than simple decision rules, with explicit platform and causal-scope limitations.

  • 5.1 Core Tail Suppression: CTS compares protected and unprotected-co-location CPTI to quantify suppression of the diagnosed core-path tail.Larger CTS indicates stronger suppression; CTS<0 indicates worsening, and CTS is used only after high-CPTI diagnosis.
  • 5.1 Core Tail Suppression: 98.1% CTS reduces nginx CPTI from 12.50 to 0.24 under rt, while NUMA can improve TTFT or throughput despite modest or negative CTS.CoTail therefore combines CTS with workload-only cache/memory signals and macro-SLO validation.
  • 5.3 CoTail Decision Procedure: CoTail screens workload-only risk, diagnoses service-stage tails, selects rt, numa, or rt+numa, and accepts policies only after common-baseline SLO validation.The SLO requires throughput loss ≤10% and TPOT increase ≤10% relative to the unprotected, no-interference LLM-alone baseline; CTS is additionally required for service-tail-dominated cases.
  • 5.4 Held-out Validation of CoTail: Frozen thresholds, workload labels, SLO budgets, and selection rules are evaluated on held-out models, frameworks, and four unseen CPU workloads against an oracle from the same candidate set.This tests same-platform transfer rather than cross-platform generalization.
  • 5.4 Held-out Validation of CoTail: 12/12 oracle-feasible held-out cases pass with CoTail, compared with 10/12 for Always-rt, while CoTail achieves the lowest mean TPOT regret.Under the amortized cost objective, CoTail uses RT in 22/28 rather than 28/28 cases, lowers mean co-tenant slowdown from 56.65% to 51.21%, and remains lowest-cost across evaluated reuse horizons.
  • Deployment Scope: CoTail is a same-platform diagnostic procedure, not a universal cluster scheduler, and its thresholds require recalibration across hardware, software, and workload environments.The authors also state that CPTI and CTS provide intervention-consistent diagnostic evidence, not formal causal proof of all interference channels.

6 Related Work

Prior work improves internal LLM serving, studies CPU–GPU interference, and provides OS controls and tracing tools. This paper instead examines how external CPU workloads perturb the host-side control path that submits GPU work.

  • LLM serving systems: Prior LLM serving systems optimize batching, KV-cache management, scheduling, and related internal serving functions.The paper studies external CPU workloads perturbing the host-side control path rather than improving the internal serving stack.
  • CPU–GPU interference and co-location: Prior CPU–GPU interference studies examine CPU sharing, synchronization delays, coupled inference, and CPU-induced slowdowns in GPU-cloud or multi-GPU settings.The paper complements this work by identifying upstream service-stage tail amplification and showing CUDA-runtime and GPU-execution signals are insufficient for realistic external CPU co-location.
  • OS resource control and tracing: Linux scheduling, cgroup, real-time, NUMA, Nsight Systems, and NVTX provide standard mechanisms for resource control and CPU–GPU timeline attribution.These mechanisms support the paper’s OS-protection comparisons and macro–micro tracing approach.

7 Conclusion

The study concludes that CPU co-location can severely degrade single-GPU LLM serving through CPU-side service-stage tails, even without GPU memory oversubscription. CoTail provides a same-platform deployment procedure, but its thresholds and scope remain platform-specific.

  • Conclusion: CPU co-location can severely degrade single-GPU LLM serving without GPU memory oversubscription, with the strongest signal in CPU-side service-stage tails before GPU submission.The study primarily evaluates one multi-GPU server and single-GPU serving instances.
  • Conclusion: CoTail converts macro–micro diagnosis into a deployment procedure that screens workload risk, selects OS-level protections, and validates decode SLO compliance.Its experiments span multiple workloads, models, frameworks, and protection strategies in the studied environment.
  • Limitations: Thresholds should be recalibrated for different CPU topologies, GPU interconnects, kernels, inference engines, and workload mixes.NVTX-based diagnosis also requires framework instrumentation, and co-tenant utility is only a deployment-cost estimate rather than a full fairness analysis.

B Workload-Only Hardware Profiles

CoTail’s first-stage screening uses workload-only hardware profiles collected without launching an LLM service. The profiles summarize CPU activity, cache and memory pressure, context switching, and loopback I/O, while explicitly limiting interpretation of the memory proxy.

  • Profile construction: Workload-only profiling derives compact screening features from detailed CPU, cache, scheduling, and loopback-I/O counters.Each workload is profiled in 20 rounds using paired pre-workload baselines.
  • Profile construction: 20-round medians are reported, while IQR is used only for quality control and does not automatically exclude runs.Additive counters are workload-induced deltas over paired baselines; ratio metrics are reported from workload windows.
  • Interpretation limits: The mem/cache proxy is a cache-miss-derived GB/s-equivalent relative signal, not measured memory-controller bandwidth.Direct memory-controller counters were unavailable under the unprivileged shared-server setup.
  • Interpretation limits: These counters support first-stage risk screening, while serving degradation and protection effects are evaluated separately at the macro level.The screening profile is not itself the end-to-end serving evaluation.

C Full Macro Results

The macro evaluation separates strategy-specific degradation analysis from common-baseline deployment acceptance. CoTail uses frozen, calibrated rules and held-out validation to select protections while accounting for service quality and deployment costs.

  • CoTail decision procedure: The selector analysis reports protection choices by workload and macro metric, including targeted combined-policy validation for mixed-risk workloads.The main text uses the combined-policy result to justify the rt+numa branch.
  • CoTail decision procedure: The decision procedure defines TTFT-dominated cases by comparing TTFT increase with throughput drop and TPOT increase, while execution-path tails cover model.execute and model.forward.This operational definition is used in Algorithm 1.
  • Evaluation protocol: CoTail thresholds and policy rules are calibrated on the primary vLLM/DeepSeek setup and fixed before held-out evaluation.The held-out validation spans other models, frameworks, and four unseen CPU workloads.
  • Evaluation protocol: Strategy-specific relative changes isolate marginal co-location degradation, whereas deployment SLO acceptance uses the common unprotected no-interference baseline.The common baseline includes clean protection overhead in deployed service quality.
  • Cost accounting: Co-tenant slowdown is utility loss relative to the CPU-alone workload baseline and is measured without Nsight Systems for deployment-cost accounting.Utility units match each workload’s natural throughput unit and are not used for CPTI/CTS diagnosis.
  • Cost accounting: Recurring deployment costs include TPOT regret, RT exposure, and co-tenant slowdown, while profiling and canary exposure are amortized over reuse horizon H.This separates repeated deployment effects from one-time diagnostic costs.

I Diagnostic Aggregation Robustness Checks

The robustness checks treat CPTI as an interpretable aggregation of upstream service-stage tails rather than a uniquely optimal metric. Its same-data correlations support diagnostic usefulness, but macro-SLO validation remains required.

  • CPTI design: CPTI averages uniform four-stage P95/P99 tail measures to retain directly relevant service stages without fitting stage weights to the workload set.The design goal is compact diagnostic aggregation, not proof of a uniquely optimal metric.
  • Robustness check: Spearman ρ = 0.976 for both throughput drop and TPOT increase across eight original workloads in the unprotected same-data check.This check complements service-stage correlations and CUDA/Nsight-level analysis.
  • Interpretation limits: CPTI summarizes the upstream service-stage tail path, while CUDA/Nsight rows capture near-kernel signals from different feature sets and sample sizes.The paper therefore treats the rows as serving different purposes rather than as a matched statistical comparison.
  • Interpretation limits: Macro-SLO validation is still required before accepting a deployment, even when CPTI provides an interpretable diagnostic score.The aggregation is not used as a standalone deployment acceptance criterion.

J Representative Nsight Timeline Evidence

The representative timeline localizes nginx co-location slowdown to an expanded EngineCore synchronization wait and delayed serving activity before GPU submission, rather than slower individual GPU kernels.

  • Timeline localization: 24.9×: the highlighted sem_wait interval expands from 1.514 s in none-none to 37.642 s in none-nginx.The traces share a similar early execution pattern before diverging later.
  • Timeline localization: The co-location trace shows a later, more concentrated NVTX activity window after the extended wait.This provides qualitative localization evidence for delay in the serving-control path.
  • Interpretation: The slowdown is not primarily attributed to individual GPU kernels becoming slower, but to shifted and clustered service-stage execution before GPU work is submitted.The trace is qualitative evidence; the broader diagnostic claim relies on service-stage correlation and robustness analyses.
  • Macro impact: 78.8%: throughput drops in the nginx case, while TTFT increases by 429.5% and TPOT increases by 362.4%.These macro-level symptoms accompany the trace-level delay in the same case.

K.2 Correlation Robustness and Leakage Checks

Robustness and intervention checks support service-stage tail amplification as an intervention-consistent explanation for co-location degradation, while preserving limits on causal and universal claims.

  • Robustness checks: Correlation is treated as diagnostic rather than causal, with checks covering all cases, unprotected-only cases, leave-one-workload-out robustness, and bootstrap intervals.These checks address leakage from pooled workload–policy combinations.
  • Protection-specific evidence: Under rt, the strongest remaining signal shifts toward batch.construct, consistent with real-time scheduling suppressing part of the scheduler-side wait.Per-protection checks are also strong for none, nice, cgroup, and numa.
  • Metric comparison: CUDA/Nsight-level correlations are much weaker than service-stage NVTX correlations, so CUDA-level metrics provide only partial explanations for degradation.Representative cases also show severe macro degradation without large CUDA/Nsight-level changes.
  • Decode-path interpretation: Throughput drop and TPOT increase are dominated by tails in scheduler.step, model.execute, and model.forward on the steady-state decode path.TTFT correlates less strongly with these stages and also depends on admission, batching, cache, memory, and topology interactions.
  • Intervention checks: Controlled schedstat measurements, service-stage delay injection, and cross-workload trace summaries provide intervention-consistent support rather than formal proof of a single interference channel.Delay injection reproduces throughput loss and TPOT increase when core serving stages are delayed.
  • CPTI and CTS: 96–100%: rt reduces CPTI for stress-ng, openssl, redis, kernel_build, memcached, and nginx, whereas nice and cgroup often leave CPTI nearly unchanged or increase it.numa is most effective for 7zip in CTS and can improve ffmpeg TTFT through effects not captured by CPU-side tail suppression.
  • Macro recovery: CTS is most strongly associated with TPOT reduction overall and with throughput and TPOT recovery under rt, where the protection directly targets CPU-side service-stage tails.Under numa, macro recovery can also arise from locality and topology effects.
Loading 2609.05425v1…