Source-linked AI summary
Spatial LLM Workload Shifting Needs Foresight: Model Commitment for AI Data Center Operation under Power Grid Constraints
Bojun Du, Hongyang Jia, Tonghui Li, Qingchun Hou, Ze Wang, Ershun Du, Ning Zhang
TL;DR
Existing spatial workload shifting may send LLM requests to sites without the required model replica, leaving demand unserved and latency requirements unmet. MC jointly schedules replica commitment and request routing under grid constraints. Case studies report full service and lower operating cost.
Problem
Existing shifting methods may assume computing capacity alone lets a destination immediately serve requests, although model residency and latency requirements constrain feasible LLM service.
Source: §I · p. 1Method
MC jointly models intertemporal replica commitment and spatial request routing, with serving capacity constrained by LLM latency requirements.
Source: §I · p. 1, B. Intertemporal Model Commitment and… · p. 2Results
100% request-service rate and 29.0% reduction in total operating cost are reported for MC.
Source: §IV · p. 3, §III · p. 3Takeaways & Limitations
Explicitly modeling replica commitment and latency limits supports feasible LLM workload shifting under time-varying grid conditions.
Source: §IV · p. 3
Abstract
from arXiv · showhide
AI data centers may face power supply shortages during certain periods, requiring operators to shift large language model (LLM) inference workloads spatially to maintain service rates. However, existing workload-shifting methods typically assume that any data center with sufficient computing resources can immediately serve shifted requests, which may lead to infeasible transfers and unserved demand. This letter proposes model commitment (MC), a mixed-integer linear programming framework that jointly schedules model deployment and cross-site request routing under power constraints and electricity-price signals. First, MC formulates the intertemporal coupling introduced by model replica loading. Second, it translates prefill and decode latency requirements into the amount of demand that each replica can serve. Case studies based on real-world data show that MC enables AI data center operators to achieve a 100% service rate under time-varying grid conditions and reduce total operating cost by 29.0%.
I. INTRODUCTION
Existing spatial-shifting methods may route LLM requests to data centers without the required model replica, while replica loading and latency requirements constrain feasible service. The proposed model commitment framework coordinates replica deployment and request routing under power and price signals.
I. INTRODUCTION
- Existing shifting methods assume computing resources alone let a data center immediately serve requests, although the destination must already have the requested model replica.Ignoring replica residency may produce infeasible transfers and unserved requests.Source: §I · p. 1
- Replica loading and warm-up take substantial time, so operators must decide deployment in advance while coordinating computing resources under model-specific demand and power constraints.Once loaded, replicas must also meet prefill TTFT and decode TPOT requirements that determine how much demand they can serve.Source: §I · p. 1
- The letter proposes model commitment (MC), coupling intertemporal replica loading with spatial request routing and prefill and decode SLO constraints on replica serving capacity.Its numerical results report a 100% request-service rate and 29.0% lower operating cost.Source: §I · p. 1
- MC's ablations show that neglecting replica loading time or SLO limits leads to substantially lower service rates.These omissions can leave demand unserved.Source: §I · p. 1
II. GRID-RESPONSIVE LLM INFERENCE SCHEDULING
MC schedules model residency and spatial request routing together, using demand, power limits, and prices as inputs at 15-minute resolution.
II. GRID-RESPONSIVE LLM INFERENCE SCHEDULING
- At a 15-minute scheduling resolution, MC takes request demand, power supply limits, and electricity prices as inputs.The operator's objective is to improve service rate and reduce total cost.Source: §II · p. 1
- MC determines the resident state of model replicas at each data center.Replica residency is scheduled as part of the operator's decisions.Source: §II · p. 1
- MC also determines how requests are routed spatially across data-center sites.Routing is coordinated with model residency decisions.Source: §II · p. 1
A. Scheduling Objective
The scheduling objective allocates model-specific token demand between routed service and unserved volume, balancing unserved-token losses against electricity costs.
A. Scheduling Objective
- Each model-specific requested token volume is either routed to a data center or left unserved.The routing and unserved quantities are nonnegative.Source: A. Scheduling Objective · p. 1
- The routing variable records tokens sent from demand region r to data center n for model m in period t, while the unserved variable records tokens not served.The demand balance equates these quantities to model-specific token demand.Source: A. Scheduling Objective · p. 1
- MC minimizes the economic loss of unserved tokens together with total electricity cost.The unserved-token loss is model-specific, and electricity cost uses nodal prices and facility power.Source: A. Scheduling Objective · pp. 1–2
B. Intertemporal Model Commitment and Resource Constraints
MC commits model replicas over time and constrains serving capacity and data-center resources so request routing respects replica readiness and operational limits.
B. Intertemporal Model Commitment and Resource Constraints
- MC models replica loading as an intertemporal commitment process, with ready inventory updated after a loading lead time and unloaded replicas removed.The effective service-ready count also credits a fraction of the final loading period; a 10-minute readiness time contributes 1/3 in a 15-minute period.Source: B. Intertemporal Model Commitment and… · p. 2
- Loading and unloading cannot occur simultaneously, and replicas that are unavailable cannot be unloaded.The loading-mode indicator and a sufficiently large constant support these constraints.Source: B. Intertemporal Model Commitment and… · p. 2
- Replicas undergoing loading are excluded from the service-ready count but still occupy GPU resources.The model counts replicas undergoing loading during each period.Source: B. Intertemporal Model Commitment and… · p. 2
- GPU and memory constraints account for GPUs per replica, available GPU count, model-weight memory, reserved KV-cache memory, and available GPU memory.These limits constrain replica deployment and resource use.Source: B. Intertemporal Model Commitment and… · p. 2
- MC also imposes route-bandwidth and data-center I/O constraints on request routing and model loading or unloading.The corresponding formulations are omitted for brevity.Source: B. Intertemporal Model Commitment and… · p. 2
- Data-center power load is derived from resources used for model residency and request processing, including idle, inference, and replica-loading power.The formulation also includes per-replica base power, incremental energy per output token, loading and unloading energy, power usage effectiveness, and demand-response power limits.Source: B. Intertemporal Model Commitment and… · p. 2
C. SLO-Constrained Serving Capacity Modeling
The serving-capacity model translates prefill and decode latency requirements into per-replica demand limits. It uses mean queueing latency for TTFT and a throughput fraction for TPOT.
C. SLO-Constrained Serving Capacity Modeling
- TTFT capacity is constrained by mean per-replica queueing latency, using the assigned request arrival rate and mean prefill service time.The resulting SLO-constrained request rate is used to limit serving capacity.Source: C. SLO-Constrained Serving Capacity… · p. 2
- Average output length is used to derive the implied per-replica request arrival rate, and the final TTFT constraint is imposed in MC based on Eqs. (9)-(10).Source: C. SLO-Constrained Serving Capacity… · p. 2
- The TPOT limit constrains each replica to part of maximum throughput, with the SLO-compliant fraction approximately proportional to the TPOT limit in the practical operating range.The formulation uses an experimentally calibrated coefficient.Source: C. SLO-Constrained Serving Capacity… · p. 2
III. CASE STUDIES
In case studies using real-world grid-price profiles and request traces, MC achieves full service while reducing total cost against operational baselines. The results also show how replica commitment supports shifting under power and price changes.
III. CASE STUDIES
- The case study uses three data centers, nodal electricity-price profiles from three U.S. regions, and real-world OpenAI workload traces for three representative LLMs.The selected workloads are Qwen2.5-32B, Llama-3.3-70B, and DeepSeek-V3.Source: §III · p. 3
- MC raises service rate to 100% and reduces total cost versus B1-Myopic and B2-Static; B4 retains full service at higher cost, while B3, B5, and B6 have substantially lower service rates.MC has higher average electricity cost than B3 and B5, primarily because it serves much more demand; full service requires capacity at higher-priced data centers.Source: §III · p. 3, Table I · p. 3
- MC retains some resident capacity as demand declines rather than repeatedly cycling replicas, producing a persistent base load.Unloading reduces immediate power use, but restoring released capacity requires lead time and adds switching costs.Source: §III · p. 3
- MC begins reconfiguring around 21:30 ahead of demand peaking near 23:00, loading replicas at DC1 and DC3 before unloading them at DC2.The supply shortage and rising electricity price at DC2 drive this anticipatory reallocation.Source: §III · p. 3
IV. CONCLUSION
MC coordinates intertemporal model commitment and request routing under LLM-specific latency limits. It achieves a 100% service rate with a 29.0% reduction in total operating cost.
IV. CONCLUSION
- MC coordinates intertemporal model commitment and request routing under LLM-specific latency limits, achieving a 100% service rate and a 29.0% reduction in total operating cost.Source: §IV · p. 3