PaperlayerView exact revision on arXiv

Source-linked AI summary

MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs

Youssef Ennouri, Soonhoi Ha

arXiv:2609.02109v1cs.DCcs.AI

TL;DR

Concurrent heterogeneous AI inference on shared GPUs creates interference, while fully joint surrogate profiling becomes combinatorial as co-running model count increases. The paper introduces a MeanField surrogate using local configurations and aggregate GPU state, integrates it into a GA scheduler, and reports accurate, scalable prediction with near-exhaustive scheduling quality.

Abstract

from arXiv · show

Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.

I. INTRODUCTION

Concurrent heterogeneous workloads create nonlinear shared-GPU interference, while joint surrogate profiling scales combinatorially with the number of models. The paper proposes a MeanField surrogate and GA scheduler to provide scalable prediction and runtime scheduling.

  • Shared-GPU co-execution causes nonlinear interference across SM occupancy, VRAM capacity, and memory bandwidth.Source: §I · p. 1
  • Existing surrogate approaches require a profiling budget that grows combinatorially with the number of co-running models.Source: §I · p. 1
  • The MeanField surrogate predicts contention-aware performance from local model configurations and aggregate GPU state.Source: §I · p. 1
  • n* ≈ 20N samples are sufficient to reach R2 ≥ 0.95 over N ∈ {2, . . . , 6} under stratified sampling.Source: §I · p. 1
  • The surrogate is integrated into a genetic-algorithm scheduler and validated under dynamic workload scenarios.Source: §I · p. 1

II. RUNTIME SCHEDULING MODEL

The scheduler selects one discrete configuration per heterogeneous model using runtime workload and aggregate GPU state. Its utility balances throughput-weighted quality against reconfiguration costs and SLA or VRAM constraints.

Configuration space

  • Each model’s discrete configuration combines a variant with runtime parameters such as concurrency, skip rate, and image size.LLM variants are FP16 and AWQ; YOLO11 variants are n, s, and m.Source: §II · p. 2

Observed state

  • The runtime state includes workload level, current VRAM usage, and aggregate GPU utilization.Source: §II · p. 2

Scheduling objective

  • The scheduler chooses λ* = arg max over λ ∈ Λ of utility J(λ, st).The utility rewards throughput-weighted quality while penalizing reconfigurations and SLA violations.Source: §II · p. 2
  • Throughput-weighted quality rewards configurations only when they are both fast and accurate.Per-variant min–max normalization prevents faster variants from dominating slower variants’ scores.Source: §II · p. 2

Constraints

  • SLA and VRAM constraints exclude infeasible configurations, while throughput-floor violations receive J = −∞.Source: §II · p. 2

A. Offline Profiling

Offline profiling measures concurrent configurations under controlled GPU conditions and reveals substantial, workload-dependent interference between heterogeneous models.

Profiling setup

  • 648 profiling measurements cover 162 baseline N=2 configurations across four workload regimes.Each configuration is evaluated under low, medium, high, and burst workload levels.Source: A. Offline Profiling · p. 2
  • Each profiling sample is one sampled joint configuration under one workload regime, with repeated measurements averaged into one training row.For N > 2, configurations are sampled stratified by model type, variant, and workload level instead of fully enumerated.Source: A. Offline Profiling · p. 2

Observed interference

  • AWQ LLM variants lose 8.4% average throughput under co-execution, while vision models lose 17–24% FPS depending on model size.ANOVA identifies workload intensity as the dominant factor for vision degradation and skip rate as the dominant factor for LLM slowdown.Source: A. Offline Profiling · p. 2
  • Aggregate resource pressure, rather than specific model pairings, primarily drives the observed interference.This finding motivates replacing explicit pairwise interaction modeling with the MeanField approximation.Source: A. Offline Profiling · p. 2

B. MeanField Surrogate

The MeanField surrogate predicts each model’s throughput from its local configuration and observed aggregate GPU state, then feeds normalized predictions into scheduling objectives and penalties.

Surrogate formulation

  • MeanField trains one lightweight predictor per model using local configuration and an aggregate field summarizing co-runner resource pressure.The aggregate field is the observed GPU state rather than the identities of individual co-runners.Source: B. MeanField Surrogate · p. 3
  • The predictor maps local configuration and observed state to raw throughput, which is normalized before entering the throughput-quality reward and SLA penalty.This connects surrogate inference directly to the scheduler’s objective and constraint handling.Source: B. MeanField Surrogate · p. 3

Runtime use

  • Runtime workload level, VRAM usage, and aggregate utilization are held fixed during GA search, so candidates are ranked for the current state.The surrogate does not predict future GPU-state evolution.Source: B. MeanField Surrogate · p. 3
  • Profiling cost grows as O(N · k) in sampled workload states rather than combinatorially, while aggregate GPU variables retain high predictive accuracy across evaluated mixtures.The surrogate is trained directly from concurrent N-model executions.Source: B. MeanField Surrogate · p. 3

Model implementation

  • Each predictor is a 64–32 MLP with BatchNorm, ReLU, Dropout(0.1), and Sigmoid output, providing sub-millisecond local inference.The architecture was selected by ablation for the best accuracy–latency trade-off at N = 3 and n = 80.Source: B. MeanField Surrogate · p. 3

C. Scalability Challenge

Fully joint surrogate modeling becomes impractical as the number of concurrent models grows because both profiling complexity and input dimensionality increase rapidly.

Joint-modeling limitation

  • A fully joint predictor requires O(k^N) profiling complexity and input dimensionality that grows with N.Here, k denotes the number of configurations per model.Source: C. Scalability Challenge · p. 3

Evaluation scope

  • MeanField changes profiling cost to O(N · k) rather than O(k^N) while maintaining high prediction accuracy.This formulation directly targets the scalability bottleneck of joint modeling.Source: C. Scalability Challenge · p. 3
  • MeanField is evaluated across N ∈ {2, 3, 4, 5, 6} using stratified cross-validation to measure accuracy and samples needed for convergence.The evaluation characterizes the trade-off between scalability and predictive performance.Source: C. Scalability Challenge · p. 3

A. Sample-Complexity Scaling

The MeanField surrogate reaches its target accuracy with an empirical profiling budget that grows approximately linearly with the number of concurrent models, contrasting with joint-surrogate profiling cost.

A. Sample-Complexity Scaling

  • The convergence criterion is the minimum training-set size n* needed for mean R^2 to reach 0.95.Results are averaged over 25 evaluations using five random seeds and five-fold stratified cross-validation.Source: A. Sample-Complexity Scaling · p. 3
  • R^2 ≥0.95 is reached at every N with a modest sample budget that increases approximately linearly as N grows.The learning curves plateau near R^2 ≈0.95–0.97 with low inter-seed and inter-fold variability.Source: A. Sample-Complexity Scaling · p. 3, Figure 2 · p. 3
  • n*≈20N samples are required for convergence across N ∈ {2, 3, 4, 5, 6}.The reported convergence counts are approximately 40, 60, 80, 100, and 120 samples, respectively.Source: Figure 3 · p. 3, A. Sample-Complexity Scaling · p. 4
  • The near-linear budget contrasts with the combinatorial O(k^N) profiling cost of a joint surrogate.This scaling is presented as the basis for improved profiling efficiency as N increases.Source: A. Sample-Complexity Scaling · p. 4

B. Prediction Accuracy

MeanField maintains high prediction accuracy through N = 6 while supporting runtime scheduling over a large feasible configuration space and dynamic workload scenarios.

B. Prediction Accuracy

  • R^2 > 0.95 is maintained up to N = 6, staying within a narrow band of the N = 2 Joint MLP reference.The accuracy comparison uses mean R^2 ± standard deviation over 25 evaluations.Source: B. Prediction Accuracy · p. 4, Table I · p. 4
  • 78,732 VRAM-feasible configurations are evaluated at N = 5 across three LLMs and two YOLO-family models.The 7B LLM is AWQ-only under the 24 GB memory limit, and per-model SLA floors use empirical P25.Source: C. Runtime Scheduling Validation · p. 4
  • 26 ms is the median end-to-end GA decision latency, compared with 131 ms for exhaustive surrogate search.MeanField evaluation takes 8.5 ms and GA search takes 17 ms; the resulting GA is about 5× faster.Source: C. Runtime Scheduling Validation · p. 4
  • Scheduling results are reported over eight dynamic scenarios, with exhaustive search defined as a per-step argmax over the same surrogate-predicted landscape.This comparison isolates search quality from surrogate prediction differences.Source: C. Runtime Scheduling Validation · p. 4, Table II · p. 4
  • The GA remains within 0.10% of exhaustive search with zero SLA violations.StaticFull is also near-optimal, while StaticBudgetK loses 10.51% utility at K = 500 and 5.22% at K = 5000.Source: C. Runtime Scheduling Validation · p. 4

V. CONCLUSION

The paper presents MeanField as a scalable surrogate for scheduling concurrent heterogeneous AI workloads on shared GPUs. Across N = 2–6 it achieves high accuracy with approximately linear sample-complexity growth, and at N = 5 it enables near-exhaustive scheduling with low latency and no SLA violations.

V. CONCLUSION

  • MeanField achieves R^2 ≈0.96 across N ∈ {2, 3, 4, 5, 6} with approximately linear empirical sample-complexity growth.Source: §V · p. 4
  • At N = 5, the GA handles 78,732 feasible configurations within 0.10% of exhaustive search and with zero SLA violations across eight dynamic scenarios.Source: §V · p. 4
  • The median end-to-end decision latency is 26 ms.Source: §V · p. 4
  • Future work will extend the LLM workload model to capture prefill/decode phases and KV-cache occupancy.Source: §V · p. 4

Related papers

Loading 2609.02109v1…