Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving

Weinan Liu; Zeyuan Ding; Dian Ding; Chengcheng Wan; Lu Tang; Guangtao Xue; Jiwu Shu; Yiming Zhang

Source-linked AI summary

Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving

Weinan Liu, Zeyuan Ding, Dian Ding, Chengcheng Wan, Lu Tang, Guangtao Xue, Jiwu Shu, Yiming Zhang

arXiv:2608.15762v1Aug 16, 2026cs.OScs.DCcs.LG

TL;DR

Multi-tenant model serving leaves fine-grained GPU idle slices underused because container-level scheduling is too heavyweight for short-lived opportunities. SliceScheduler uses cluster-wide operator state and simulation-based placement to schedule operators dynamically, improving token throughput 1.10×–2.29× over existing approaches while keeping SLA violations within 9%.

Problem
Container-level scheduling is too heavyweight to exploit short-lived, fragmented GPU idle slices in multi-tenant model serving under SLA constraints.
Method
SliceScheduler combines the Global Mapping Graph, global simulation, simulation-based scheduling, and operator execution to make dynamic cluster-wide operator placements.
Results
1.10×–2.29× token throughput improvement over baselines is reported, with Service Level Objective attainment improving by ∼9%.
Takeaways & Limitations
SliceScheduler demonstrates that dynamic operator-level scheduling can reclaim fine-grained idle slices while preserving online SLAs in multi-tenant AI clusters.
Takeaways & Limitations
Container-level and GPU-sharing approaches remain poorly suited to fragmented idle slices across multiple GPUs and containers.

Abstract

from arXiv · show

Container-granularity scheduling leaves abundant short-lived idle slices within containers unexploited. Reallocating containers is too heavyweight to utilize such fine-grained opportunities under SLA constraints, and operator-level scheduling requires reasoning about dependencies, memory safety, and cluster-wide execution dynamics in real time. In this paper, we present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving. The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions. SliceScheduler consists of four key components. First, we introduce the Global Mapping Graph (GMG), a unified abstraction that captures operator dependencies, tensor shapes, resource mappings, and execution states, providing a real-time, cluster-wide view with explicit resource semantics. Second, we build a global simulator on top of GMG to predict operator-level execution and memory evolution under candidate placements. Third, we design an incremental, simulation-based scheduling module that selects placements to exploit fragmented idle slices while avoiding memory violations and preserving SLA. Finally, we develop an operator executor that materializes scheduling decisions on GPUs and coordinates computation and cross-accelerator transfers. We implement SliceScheduler as a PyTorch backend and evaluate it using production trace replay. Experimental results show that SliceScheduler improves token throughput by 1.10--2.29$\times$ compared to existing approaches, while maintaining SLA violations within 9\%. SliceScheduler demonstrates that operator-level scheduling is a practical and effective approach to improving GPU utilization for multi-tenant LLM serving.

1 Introduction

SliceScheduler addresses GPU underutilization in multi-tenant LLM serving by scheduling operators into fragmented idle slices rather than reallocating containers. It combines a cluster-wide Global Mapping Graph, global what-if simulation, dynamic placement, and GPU execution coordination, improving throughput while preserving SLA constraints.

Motivation: Production traces show many short, fine-grained idle slices that container-level scheduling cannot exploit under strict SLA requirements.Observed idle-slice durations are p50 2551.11 ms, p90 10887.64 ms, and p99 17790.31 ms.
Motivation: Operator-level scheduling uses kernels, ATen ops, and LLM layers as finer-grained units that can fit idle slices and spread execution across the cluster.This removes an inference service’s dependence on a fixed set of containers or GPUs.
System Design: SliceScheduler’s Global Mapping Graph captures operator dependencies, tensor shapes, resource mappings, HBM occupancy, tensor residency, and evolving execution state.Tensor shapes inferred before execution provide explicit operator resource requirements for memory-safe scheduling.
System Design: The scheduler evaluates candidate placements through parallel what-if simulations, selecting policies that avoid OOM while optimizing predicted completion time and resource efficiency.The global simulator predicts cluster-wide operator execution and tensor-transfer evolution under each scheduling decision.
Evaluation: 1.10×–2.29× higher token throughput and ∼9% SLO attainment improvement are achieved over the online-only and GPU-sharing baselines in production trace replay.The simulator sustains ∼176 simulated operators per millisecond, about four orders of magnitude higher throughput than prior approaches.

2 Background & Motivation

Multi-tenant serving workloads leave substantial, fine-grained GPU idle slices that container-level scheduling cannot exploit efficiently. Using these slices requires cluster-wide, operator-level coordination that accounts for fragmented resources, memory pressure, dependencies, and execution dynamics.

Idle-slice characterization: Most 10-minute windows have model-instance compute time below 600 s, while demand remains uneven across traces even during peak periods.This indicates that instances are far from fully utilized in most windows and that aggregate headroom is fragmented across workloads.
Idle-slice characterization: 64.0% average GPU utilization still leaves substantial idle slices, whose median, 90th-percentile, and 99th-percentile durations are 2551.11 ms, 10887.64 ms, and 17790.31 ms.Idle slices shorter than the 99th-percentile duration account for 95.6% of total idle time.
Idle-slice characterization: 1.25× token throughput improvement is achieved by placing low-priority requests into high-priority online instances’ idle gaps while maintaining SLA.The co-location experiment uses two Llama3 8B instances on one A16 GPU.
Limitations of existing scheduling: Container-level control is too heavyweight for fine-grained gaps because it migrates and reconstructs entire application runtimes rather than moving individual kernels or small HBM regions.Elasticity additionally incurs checkpointing, rescheduling, gang-scheduling delays, image pulling, environment initialization, and checkpoint loading.
Challenges: Static GPU sharing misses transient idle slices, while dynamic sharing faces resident LLM state, memory pressure, interference, and poorly aligned idle times across GPUs.Across four replayed traces, no more than 20% of idle time overlaps, making multi-GPU exploitation difficult.
Required system capabilities: The proposed scheduling layer combines a Global Mapping Graph, global simulator, scheduling module, and operator executor to provide cluster-wide state abstraction and operator-level control.The design targets operator shapes and cluster-wide execution state rather than container-level resource ownership.

3 SliceScheduler Design

SliceScheduler is an operator-level scheduler that dynamically places low-priority operators into idle slices of high-priority operators. This design aims to improve GPU utilization while preserving SLA constraints.

3 SliceScheduler Design: SliceScheduler dynamically schedules low-priority operators into high-priority operators’ idle slices to improve GPU utilization without violating SLA constraints.It is presented as an efficient operator-level scheduling approach.

3.1 System Overview

SliceScheduler organizes cluster-scale operator scheduling into four coupled layers—state abstraction, prediction, decision, and execution—centered on the Global Mapping Graph (GMG). GMG provides a cluster-wide view that supports what-if placement simulation, scheduling decisions, accelerator execution, and runtime updates.

State abstraction: The Global Mapping Graph unifies operators, dependencies, tensor shapes, accelerator placements, and runtime state for coherent cluster-wide scheduling.GMG serves as the system’s global state abstraction.
Runtime coordination: Runtime feedback continuously updates GMG and estimators, enabling a closed-loop, prediction-driven scheduling process.The system combines a centralized master maintaining GMG with accelerators that execute operators.
Prediction: A Global Simulator evaluates candidate operator placements through what-if simulation of execution dynamics.The scheduler constructs global simulators from the evolving GMG state.
Decision and execution: The Scheduling Module selects operator placements, then updates GMG mappings, marks operators issued, and dispatches operators with required transfers.Scheduling and issuing connect simulation-based decisions to execution on accelerators.

3.2 Global Mapping Graph

The Global Mapping Graph (GMG) unifies computation structure, accelerator placement, execution state, and scheduling metadata at operator granularity. It provides a continuously evolving, cluster-wide view that supports temporal prediction and incremental scheduling.

Graph structure: GMG combines a Shape DAG, whose nodes represent shaped operators and whose edges represent data dependencies, with an Accelerator Set for operator placements.Each operator also carries executor state indicating whether it is unscheduled, issued, or done.
Execution lifecycle: GMG continuously records operator placements, issuing information, completion, and resource release as execution progresses.This makes GMG a dynamic cluster-wide view rather than a static task description.
Scheduling metadata: Global timestamps, dependency information, and output tensor shapes provide inputs for predicting execution time, HBM occupancy, and cross-accelerator transfer latency.Each node stores timestamps for issuance and completion, while output shapes support resource and transfer estimates.
Scheduling metadata: Checkpoint metadata identifies persistently retained intermediate results, enabling new graph instances to attach incrementally to retained accelerator states.Retained results may be located in accelerator HBM.

3.3 Global Simulator

The global simulator uses GMG snapshots and lightweight online estimators to reconstruct cluster execution and HBM state, then advances a unified event-driven timeline. It enables what-if evaluation of candidate operator placements without deploying them on the real cluster.

Estimator construction: Lightweight linear estimators predict tensor-transfer latency and operator execution time from transfer characteristics and symbolic, shape-derived features.Estimators receive online performance feedback from the operator executor, improving accuracy and supplying latency inputs to the simulator.
State reconstruction: GMG lets the simulator reconstruct each accelerator’s queued operators, dependencies, execution state, and HBM occupancy from a consistent cluster snapshot.Ready times depend on local predecessor completion and transferred outputs, while tensor residency follows ready and executing operators.
Simulation outputs: At operator granularity, the simulator predicts both cluster-wide execution timelines and HBM-usage evolution over time.Completion events trigger subsequent operator starts and estimator-based completion predictions.
Timeline simulation: The simulator advances a unified timeline by repeatedly processing the earliest-complete operator, starting an issued ready operator, and predicting its completion.This event-driven process propagates execution effects across accelerator queues.
Placement evaluation: SliceScheduler evaluates candidate placements through what-if simulation by copying GMG, applying a placement, and scoring simulated completion-time statistics without real deployment.The resulting placement score is derived from the simulated completion times of relevant operators.

3.4 Scheduling Module

SliceScheduler uses lightweight, incremental scheduling to select small connected operator batches and place them through simulation over candidate accelerator partitions. The method maintains GMG scheduling frontiers, accounts for transfer-induced delays and SLA priorities, then commits placements and updates execution state.

Scheduling rounds: Each scheduling round uses the current GMG, unscheduled frontier, and issued-operator budget to produce a small batch with placements while limiting overhead.The module is invoked for new graph instances and runtime feedback, so responsiveness is a central constraint.
Scheduling boundary: The scheduled and unscheduled frontiers define the current GMG scheduling boundary between issued or completed checkpoints and operators ready for scheduling.The scheduled frontier contains issued or done operators with unscheduled successors; the unscheduled frontier contains operators whose predecessors are absent or already issued or done.
Batch collection: SliceScheduler first selects the highest-priority unscheduled operator and follows successors to collect a capped, connected batch focused on one high-priority model request.Batch collection exploits dependency chains across graph instances while keeping scheduling overhead low.
Simulation-based placement: For each batch, the scheduler evaluates candidate placements on predefined accelerator partitions by applying them to a cloned GMG and simulating the resulting execution.Regularly decomposable inference graphs allow each partition’s internal operator-to-accelerator mapping to be predefined.
Decision materialization: After selecting a placement, SliceScheduler writes it to GMG, issues the corresponding operators to the executor, marks them issued, and updates both frontiers.This closes the incremental scheduling loop by materializing the decision and advancing the scheduling boundary.
Placement objectives: Explicitly simulated cross-accelerator transfers increase predicted downstream completion times, disfavoring placements that break locality while protecting high-priority SLAs.The scheduling algorithm captures both data locality and SLA priorities through its predicted timeline.

3.5 Operator Executor

SliceScheduler executes dependency-driven operators independently on accelerator-local queues while coordinating cross-accelerator transfers centrally. Its arbitration-managed protocol enforces safe transfer ordering and selects conflict-free transfers to avoid deadlocks and enable parallel communication.

Local operator execution: Accelerator-local queues execute dependency-ready operators independently, partitioning computation graphs across devices while running at most one operator per queue.Operators become ready when all input slots are ready; completion propagates readiness to successors.
Cross-accelerator transfers: Cross-accelerator dependencies require explicit tensor movement, but autonomous point-to-point transfers risk blocking, non-preemptible execution and circular waits.SliceScheduler therefore prevents queues from autonomously starting cross-accelerator transfers.
Cross-accelerator transfers: Each transfer uses Intent, Recv and Send operators, with Recv allocating the destination buffer before Send initiates data movement.This ordering delays communication until the destination buffer is successfully prepared.
Arbitration algorithm: A separate accelerator communication graph tracks pending transfer intents and admits only conflict-free transfers that can proceed concurrently.Conflicts may arise from shared sources, destinations or communication links.
Arbitration algorithm: Event-driven arbitration reacts to new intents and transfer completions, activates destination-side Recv operations, releases resources and avoids deadlocks while enabling parallel point-to-point transfers.The communication graph is re-examined at each trigger to select newly activatable edges.

4 Implementation

SliceScheduler is implemented in three layers: a PyTorch serving interface, a host-CPU master for cluster-wide scheduling and transfer coordination, and an accelerator-side operator executor.

Architecture: SliceScheduler is implemented in three layers: the serving application interface, the SliceScheduler master, and the operator executor.The architecture separates application integration, scheduling control, and accelerator execution.
PyTorch Integration: TorchDynamo captures each module’s computation graph on first execution, after which SliceScheduler registers and instantiates it as a graph template.This integration supports existing PyTorch serving applications with minimal application-code changes.
Runtime Components: The SliceScheduler master runs on the host CPU to perform cluster-wide scheduling and coordinate tensor transfers.The master provides centralized control over scheduling and transfer coordination.
Runtime Components: The operator executor runs on accelerators using LibTorch to execute operators and carry out tensor transfers.This separates accelerator-side execution and communication from host-side scheduling.

5 Evaluation

SliceScheduler is evaluated through trace replay on an eight-GPU server using layer-level scheduling, and it improves throughput, utilization, and SLO attainment over GPU-sharing baselines. Its simulator is accurate and fast, while scheduling quality must be balanced against simulation cost for scalability.

Evaluation setup: Evaluation replays four one-day Llama3-8B online traces from 19:00–19:30 alongside one offline request per second with 2048-token inputs and 15-token outputs.The system runs on a single server with eight NVIDIA A16 GPUs, each providing 64 GB of GPU memory.
Overall results: SliceScheduler improves SLO attainment by about 9%, request throughput by 9%–129%, token throughput by 10%–129%, and GPU utilization by 17%–38% over all baselines.The experiment’s relatively light trace segment and intentionally lightweight offline workload keep GPU utilization generally low.
Baseline comparison: Compared with Full-Share, SliceScheduler improves request throughput by 49% through automatic load balancing that extends overlapping idle slices across multiple GPUs.Compared with Subset-Share and Dual-Share, it improves throughput by about 9%–37% by exploiting idle slices across a larger, cluster-wide resource pool.
Execution overhead: SliceScheduler’s execution time is almost identical to Direct Execution, showing that operator scheduling adds negligible runtime overhead.It is also 8% faster than Eager Execution, mainly because it avoids Python interpreter overhead in eager mode.
Simulator accuracy: The global simulator has 1.83% average error versus SimAI’s 20.58%, while runtime estimators achieve 0.32% operator and 0.92% transfer mean absolute percentage errors.Operator and transfer sample coefficients of variation are 0.62% and 1.63%, respectively, supporting stable prediction from runtime feedback.
Simulation performance and scalability: The global simulator processes ∼176.8 operators per millisecond versus SimAI’s ∼0.065, achieving about four orders of magnitude higher simulation throughput.Longer prediction horizons improve placement quality by exposing fine-grained imbalance but increase simulation cost and can make centralized scheduling a bottleneck; the analysis suggests scaling to about 64 GPUs.

6 Related Work

Existing scheduling, GPU-sharing, simulation, LLM-serving, and graph-based systems generally operate at coarse, local, or heavyweight abstractions. SliceScheduler instead combines cluster-wide coordination with fine-grained scheduling of executable GPU operators.

Job scheduling and elasticity: Cluster schedulers operate on processes or containers, making their scheduling units too heavyweight to exploit transient idle slices; elasticity mechanisms reassign resources as demand changes.Examples include Kubernetes, Ray, and Firmament.
GPU sharing: GPU-sharing systems improve utilization but primarily coordinate within one GPU or a fixed subset, leaving complementary idle slices distributed across the cluster underexploited.SliceScheduler addresses this limitation through global scheduling.
Simulation: Microarchitectural simulators model processors and kernels at cycle level, while distributed simulators model large clusters; neither provides sufficiently fast simulation for cluster-scale scheduling.The passage states that realistic distributed-workload simulation can take hours.
LLM Serving: LLM-serving systems commonly separate local SLA-oriented request scheduling from global load balancing, but global schedulers use coarse granularity and low-frequency utilization signals.SliceScheduler bridges these levels with cluster-wide coordination and fine-grained execution control.
Graph-based scheduling: MapReduce and Spark schedule DAG tasks using data dependencies, whereas SliceScheduler applies graph structure to GPU-executable operators and reasons about accelerator placement.The passage contrasts coarse-grained analytics abstractions with fine-grained GPU execution in AI clusters.

7 Conclusion

SliceScheduler reclaims fine-grained idle slices through operator-level scheduling in multi-tenant AI clusters. Its integrated graph, simulator, scheduler, and executor improve utilization and throughput over container-level and GPU-sharing baselines while preserving online SLAs.

Conclusion: SliceScheduler is an operator-level scheduling system for reclaiming fine-grained idle slices in multi-tenant AI clusters.It targets idle opportunities that container-level scheduling cannot directly exploit.
Conclusion: The system combines the Global Mapping Graph, an event-driven global simulator, simulation-based scheduling, and an operator executor.Together, these components support scheduling decisions and their execution across the cluster.
Conclusion: SliceScheduler achieves higher throughput and utilization than container-level and GPU-sharing baselines while preserving online SLAs.The passage characterizes the resulting run-time overhead as negligible, but the available text is truncated after that claim.