PaperlayerView exact revision on arXiv

Source-linked AI summary

MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs

Youssef Ennouri, Soonhoi Ha

arXiv:2609.02109v1cs.DCcs.AI

TL;DR

Concurrent heterogeneous AI inference on shared GPUs creates interference, while fully joint surrogate profiling becomes combinatorial as co-running model count increases. The paper introduces a MeanField surrogate using local configurations and aggregate GPU state, integrates it into a GA scheduler, and reports accurate, scalable prediction with near-exhaustive scheduling quality.

  • Problem

    Shared-GPU interference complicates heterogeneous AI inference scheduling, and joint surrogate profiling grows combinatorially with the number of co-running models.

  • Method

    The paper trains a MeanField surrogate from local model configurations and aggregate GPU state, using stratified samples, and integrates it into a GA-based runtime scheduler.

  • Results

    The surrogate achieves R2 ≈ 0.96 across N ∈ {2, 3, 4, 5, 6}; at N = 5, GA scheduling is within 0.10% of exhaustive search with zero SLA violations across eight dynamic scenarios.

  • Takeaways & Limitations

    MeanField provides profiling-efficient near-optimal scheduling for concurrent heterogeneous AI workloads on shared GPUs.

Abstract

from arXiv · show

Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.

I. INTRODUCTION

Concurrent heterogeneous workloads create nonlinear shared-GPU interference, while joint surrogate profiling scales combinatorially with the number of models. The paper proposes a MeanField surrogate and GA scheduler to provide scalable prediction and runtime scheduling.

  • Shared-GPU co-execution causes nonlinear interference across SM occupancy, VRAM capacity, and memory bandwidth.
  • Existing surrogate approaches require a profiling budget that grows combinatorially with the number of co-running models.
  • The MeanField surrogate predicts contention-aware performance from local model configurations and aggregate GPU state.
  • n* ≈ 20N samples are sufficient to reach R2 ≥ 0.95 over N ∈ {2, . . . , 6} under stratified sampling.
  • The surrogate is integrated into a genetic-algorithm scheduler and validated under dynamic workload scenarios.

II. RUNTIME SCHEDULING MODEL

The scheduler selects one discrete configuration per heterogeneous model using runtime workload and aggregate GPU state. Its utility balances throughput-weighted quality against reconfiguration costs and SLA or VRAM constraints.

  • Configuration space: Each model’s discrete configuration combines a variant with runtime parameters such as concurrency, skip rate, and image size.LLM variants are FP16 and AWQ; YOLO11 variants are n, s, and m.
  • Observed state: The runtime state includes workload level, current VRAM usage, and aggregate GPU utilization.
  • Scheduling objective: The scheduler chooses λ* = arg max over λ ∈ Λ of utility J(λ, st).The utility rewards throughput-weighted quality while penalizing reconfigurations and SLA violations.
  • Scheduling objective: Throughput-weighted quality rewards configurations only when they are both fast and accurate.Per-variant min–max normalization prevents faster variants from dominating slower variants’ scores.
  • Constraints: SLA and VRAM constraints exclude infeasible configurations, while throughput-floor violations receive J = −∞.

A. Offline Profiling

Offline profiling measures concurrent configurations under controlled GPU conditions and reveals substantial, workload-dependent interference between heterogeneous models.

  • Profiling setup: 648 profiling measurements cover 162 baseline N=2 configurations across four workload regimes.Each configuration is evaluated under low, medium, high, and burst workload levels.
  • Profiling setup: Each profiling sample is one sampled joint configuration under one workload regime, with repeated measurements averaged into one training row.For N > 2, configurations are sampled stratified by model type, variant, and workload level instead of fully enumerated.
  • Observed interference: AWQ LLM variants lose 8.4% average throughput under co-execution, while vision models lose 17–24% FPS depending on model size.ANOVA identifies workload intensity as the dominant factor for vision degradation and skip rate as the dominant factor for LLM slowdown.
  • Observed interference: Aggregate resource pressure, rather than specific model pairings, primarily drives the observed interference.This finding motivates replacing explicit pairwise interaction modeling with the MeanField approximation.

B. MeanField Surrogate

The MeanField surrogate predicts each model’s throughput from its local configuration and observed aggregate GPU state, then feeds normalized predictions into scheduling objectives and penalties.

  • Surrogate formulation: MeanField trains one lightweight predictor per model using local configuration and an aggregate field summarizing co-runner resource pressure.The aggregate field is the observed GPU state rather than the identities of individual co-runners.
  • Surrogate formulation: The predictor maps local configuration and observed state to raw throughput, which is normalized before entering the throughput-quality reward and SLA penalty.This connects surrogate inference directly to the scheduler’s objective and constraint handling.
  • Runtime use: Runtime workload level, VRAM usage, and aggregate utilization are held fixed during GA search, so candidates are ranked for the current state.The surrogate does not predict future GPU-state evolution.
  • Runtime use: Profiling cost grows as O(N · k) in sampled workload states rather than combinatorially, while aggregate GPU variables retain high predictive accuracy across evaluated mixtures.The surrogate is trained directly from concurrent N-model executions.
  • Model implementation: Each predictor is a 64–32 MLP with BatchNorm, ReLU, Dropout(0.1), and Sigmoid output, providing sub-millisecond local inference.The architecture was selected by ablation for the best accuracy–latency trade-off at N = 3 and n = 80.

C. Scalability Challenge

Fully joint surrogate modeling becomes impractical as the number of concurrent models grows because both profiling complexity and input dimensionality increase rapidly.

  • Joint-modeling limitation: A fully joint predictor requires O(k^N) profiling complexity and input dimensionality that grows with N.Here, k denotes the number of configurations per model.
  • Evaluation scope: MeanField changes profiling cost to O(N · k) rather than O(k^N) while maintaining high prediction accuracy.This formulation directly targets the scalability bottleneck of joint modeling.
  • Evaluation scope: MeanField is evaluated across N ∈ {2, 3, 4, 5, 6} using stratified cross-validation to measure accuracy and samples needed for convergence.The evaluation characterizes the trade-off between scalability and predictive performance.

A. Sample-Complexity Scaling

The MeanField surrogate reaches its target accuracy with an empirical profiling budget that grows approximately linearly with the number of concurrent models, contrasting with joint-surrogate profiling cost.

  • A. Sample-Complexity Scaling: The convergence criterion is the minimum training-set size n* needed for mean R^2 to reach 0.95.Results are averaged over 25 evaluations using five random seeds and five-fold stratified cross-validation.
  • A. Sample-Complexity Scaling: R^2 ≥0.95 is reached at every N with a modest sample budget that increases approximately linearly as N grows.The learning curves plateau near R^2 ≈0.95–0.97 with low inter-seed and inter-fold variability.
  • A. Sample-Complexity Scaling: n*≈20N samples are required for convergence across N ∈ {2, 3, 4, 5, 6}.The reported convergence counts are approximately 40, 60, 80, 100, and 120 samples, respectively.
  • A. Sample-Complexity Scaling: The near-linear budget contrasts with the combinatorial O(k^N) profiling cost of a joint surrogate.This scaling is presented as the basis for improved profiling efficiency as N increases.

B. Prediction Accuracy

MeanField maintains high prediction accuracy through N = 6 while supporting runtime scheduling over a large feasible configuration space and dynamic workload scenarios.

  • B. Prediction Accuracy: R^2 > 0.95 is maintained up to N = 6, staying within a narrow band of the N = 2 Joint MLP reference.The accuracy comparison uses mean R^2 ± standard deviation over 25 evaluations.
  • B. Prediction Accuracy: 78,732 VRAM-feasible configurations are evaluated at N = 5 across three LLMs and two YOLO-family models.The 7B LLM is AWQ-only under the 24 GB memory limit, and per-model SLA floors use empirical P25.
  • B. Prediction Accuracy: 26 ms is the median end-to-end GA decision latency, compared with 131 ms for exhaustive surrogate search.MeanField evaluation takes 8.5 ms and GA search takes 17 ms; the resulting GA is about 5× faster.
  • B. Prediction Accuracy: Scheduling results are reported over eight dynamic scenarios, with exhaustive search defined as a per-step argmax over the same surrogate-predicted landscape.This comparison isolates search quality from surrogate prediction differences.
  • B. Prediction Accuracy: The GA remains within 0.10% of exhaustive search with zero SLA violations.StaticFull is also near-optimal, while StaticBudgetK loses 10.51% utility at K = 500 and 5.22% at K = 5000.

V. CONCLUSION

The paper presents MeanField as a scalable surrogate for scheduling concurrent heterogeneous AI workloads on shared GPUs. Across N = 2–6 it achieves high accuracy with approximately linear sample-complexity growth, and at N = 5 it enables near-exhaustive scheduling with low latency and no SLA violations.

  • V. CONCLUSION: MeanField achieves R^2 ≈0.96 across N ∈ {2, 3, 4, 5, 6} with approximately linear empirical sample-complexity growth.
  • V. CONCLUSION: At N = 5, the GA handles 78,732 feasible configurations within 0.10% of exhaustive search and with zero SLA violations across eight dynamic scenarios.
  • V. CONCLUSION: The median end-to-end decision latency is 26 ms.
  • V. CONCLUSION: Future work will extend the LLM workload model to capture prefill/decode phases and KV-cache occupancy.
Loading 2609.02109v1…