PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting the Flexibility of LLM Inference for Frequency Regulation Service

Yimeng Sun, Zhaohao Ding

arXiv:2610.23532v1eess.SYcs.SY

TL;DR

LLM inference’s growing power demand and flexible execution motivate using inference clusters for frequency regulation, but existing workload scheduling does not capture token-level execution for grid tracking. The paper combines hierarchical request dispatch and token allocation with reward redistribution and mean-field coordination. In simulations, it reports frequency tracking with measured regulation scores and inference latencies.

  • Problem

    Existing workload-level scheduling does not capture token-level LLM execution for frequency regulation, while fine-grained inference studies have not scheduled aggregate cluster power to track grid signals.

    Source: §I · p. 2

  • Method

    The approach combines hierarchical request dispatch and replica-level token allocation with reward redistribution for delayed feedback and mean-field approximation for replica coordination.

    Source: A. Problem Description · p. 3, §III · p. 4, B. Mean-Field Based Hierarchical… · p. 6

  • Results

    The proposed method achieves an average tracking error of 1.17%, a mean PJM performance score of 0.926, and a minimum 5-min score of 0.910, with average TTFT of 0.94 s and average E2E latency of 6.36 s.

    Source: B. LLM Inference Scheduling for… · p. 8

  • Takeaways & Limitations

    The reported response maintains consistent regulation delivery over evaluated intervals despite pointwise fluctuations from stochastic arrivals, discrete workloads, and SLO constraints.

    Source: B. LLM Inference Scheduling for… · p. 8

Abstract

from arXiv · show

While the widespread deployment of Large Language Models (LLMs) has significantly driven up power demands in computing clusters, the unique execution characteristics of LLM inference offer a novel opportunity to utilize these clusters as controllable loads. In this paper, we investigate the potential of LLM inference cluster to provide frequency regulation services through inference request scheduling. First, we model the problem as a hierarchical Markov decision process that includes request-to-replica dispatching and token-level resource allocation. Then, we introduce a reward redistribution network to address the temporal asynchrony between millisecond-level scheduling decisions and second-level frequency regulation signals. Furthermore, we develop an optimization framework based on a mean-field approximation to enable coordination among a large number of replicas under high request concurrency. Simulations using real-world data demonstrate that the proposed approach accurately tracks frequency regulation signals without compromising service-level objectives.

NOMENCLATURE

LLM inference’s temporal flexibility motivates using inference clusters as controllable loads, but existing scheduling work does not address token-level frequency regulation. The paper proposes hierarchical scheduling with reward redistribution and mean-field coordination to address this gap.

I. INTRODUCTION

  • Inference clusters may provide flexible demand for frequency regulation because prefill and decode have temporal flexibility and some applications tolerate seconds-long latency.The paper distinguishes software-level workload scheduling from hardware controls such as DVFS and GPU power capping.Source: §I · p. 2
  • Existing workload scheduling is generally too coarse for LLM inference, while fine-grained inference management has not targeted aggregate-power tracking of grid regulation signals.Token-level scheduling across prefill and decode is needed to exploit inference flexibility for regulation.Source: §I · p. 2
  • The paper identifies asynchronous feedback across millisecond scheduling and two-to-four-second regulation intervals, alongside coordination costs at cluster scale, as key challenges.The cited introduction notes that modern clusters may handle tens of thousands of requests per second.Source: §I · p. 2
  • A hierarchical MDP assigns request dispatching to an upper layer and token-level resource allocation to lower-layer replica schedulers.The notation distinguishes replicas, requests, fine-grained decision steps, regulation intervals, and their corresponding index sets.Source: NOMENCLATURE · p. 1, §I · p. 3
  • A cross-attention reward redistribution mechanism assigns delayed, coupled rewards to decision steps and is reported to improve scheduling stability and regulation performance.The paper also states that the reconstructed MDP remains approximately equivalent to the original formulation.Source: §I · p. 3
  • The proposed mean-field reinforcement learning algorithm targets scalable coordination among replicas and includes a theoretical convergence proof.The paper describes its overall framework as addressing the challenges through hierarchical reinforcement learning, reward redistribution, and mean-field approximation.Source: §I · pp. 2–3

II. PROBLEM FORMULATION

The problem formulation treats cluster regulation as two-stage inference scheduling: dispatch requests to replicas, then allocate tokens locally while meeting inference SLOs. Its objective combines regulation tracking with latency and completion considerations.

A. Problem Description

  • The cluster adjusts power through two-stage scheduling to track a target regulation signal while maintaining inference SLOs.The study focuses on real-time response to AGC-based secondary frequency regulation after capacity has been assigned.Source: A. Problem Description · p. 3, Figure 1 · p. 3
  • The cluster scheduler dispatches queued requests to replicas according to replica utilization, after which each replica performs fine-grained token allocation.Replicas are parallel copies of the same model running on GPU servers.Source: A. Problem Description · p. 3, Figure 2 · p. 3
  • Token scheduling changes batch composition and processing rate, which affect replica power; prefill and decode are modeled separately because their power characteristics differ.Their execution delays are bounded by TTFT and E2E latency requirements, respectively.Source: A. Problem Description · p. 3
  • The objective is to minimize frequency-signal tracking error while satisfying inference SLOs, using average aggregate cluster power over each regulation interval as the effective response.The cluster’s aggregate power is formed from the power consumption of its replicas.Source: A. Problem Description · p. 3, B. Hierarchical MDP Formulation · p. 4

B. Hierarchical MDP Formulation

  • The hierarchical MDP separates the cluster scheduler as upper-layer agent from N replica schedulers as lower-layer agents, with T fine-grained decisions grouped into K regulation intervals.Each interval contains D decision steps, so T = KD.Source: B. Hierarchical MDP Formulation · p. 3
  • The upper-layer state represents replica status and cluster-queue features, and its discrete action selects a replica for the current request.Queue features include a request embedding and the current request’s prefill and decode token demands.Source: B. Hierarchical MDP Formulation · pp. 3–4
  • Each lower-layer replica allocates a bounded token budget among local requests, with a null action for unassigned tokens; dispatch and allocation update queues, progress, replica states, and violation metrics.The token capacity is limited by the replica’s maximum tokens per decision step.Source: B. Hierarchical MDP Formulation · p. 4
  • The reward combines interval-delayed regulation tracking, penalties for TTFT and E2E violations, and a positive reward for completing requests before deadlines.The tracking reward aggregates the scheduling decisions within an interval, so an individual decision’s contribution is not directly observable at the time.Source: B. Hierarchical MDP Formulation · p. 4

III. SOLUTION METHOD

The solution addresses delayed feedback and replica-scale complexity by redistributing rewards across decision steps and using mean-field approximation for coordination.

III. SOLUTION METHOD

  • The MDP’s main challenges are asynchronous decision feedback and exponential growth in problem scale as the number of replicas increases.The solution assigns coupled, delayed rewards to corresponding decisions and then uses mean-field approximation for replica coordination.Source: §III · pp. 4–5

A. Reward Redistribution under Asynchronous Decision-Making and Feedback

The reward redistribution network assigns interval-level regulation feedback to individual scheduling steps, producing denser feedback for policy optimization. Segment-level return conservation preserves the original MDP’s optimal policies, while learned redistribution’s value gap decreases with training.

A. Reward Redistribution under Asynchronous Decision-Making and Feedback

  • If each segment preserves its discounted return, the redistributed MDP shares the original MDP’s set of optimal policies.This constrains discounted return over each segment without prescribing a unique reward for each decision step.Source: A. Reward Redistribution under… · p. 5
  • The network maps each segment’s state-action sequence and cumulative reward to stepwise redistributed rewards using cross-attention.The cumulative segment reward conditions the cross-attention module, which models contextual relationships with individual state-action representations.Source: A. Reward Redistribution under… · p. 5, Figure 3 · p. 5
  • The network replaces transition rewards with redistributed stepwise rewards while leaving states, actions, and next states unchanged; both policies train on the resulting buffer.The segment-sum constraint is learned, so small deviations are possible.Source: A. Reward Redistribution under… · p. 5
  • Theorem 1 states that the value-function gap between the original and redistributed MDPs decreases with training and becomes negligible with sufficient iterations.The result assumes the empirical redistribution loss decreases at O(n^-1/2).Source: A. Reward Redistribution under… · p. 5

B. Mean-Field Based Hierarchical Optimization

The framework separates request dispatching from replica-level token allocation, then uses mean-field approximation to make lower-layer coordination scalable. Reward redistribution and a two-time-scale update scheme address delayed regulation feedback and coupled optimization.

B. Mean-Field Based Hierarchical Optimization

  • The upper-layer DQN dispatches incoming requests, while each lower-layer replica scheduler performs fine-grained token-level resource allocation.The upper-layer policy is optimized to maximize expected return.Source: B. Mean-Field Based Hierarchical… · p. 6
  • The centralized lower-layer action-value function is computationally intractable at scale because the joint action space grows combinatorially with replica count.The resulting high-dimensional critic input also makes coordinated exploration and value estimation more difficult.Source: B. Mean-Field Based Hierarchical… · pp. 5–6
  • The mean-field approximation replaces dependence on the full joint action with other replicas’ mean action, reducing interaction complexity from exponential to linear in replica count.The mean action serves as a proxy for aggregate resource-allocation behavior and power dynamics.Source: B. Mean-Field Based Hierarchical… · p. 6
  • Lower-layer replicas optimize token allocation faster than the upper-layer scheduler, which updates against an approximately stationary lower-level environment.The framework’s coupled iterates asymptotically converge to a stationary solution under Robbins–Monro learning rates and αu,q/αl,q → 0.Source: B. Mean-Field Based Hierarchical… · p. 6
  • After training converges, online scheduling requires only forward inference through the upper- and lower-layer networks, without reward redistribution.The complete training and update procedure is given in Algorithm 1.Source: B. Mean-Field Based Hierarchical… · p. 7

A. Experimental Setting

The evaluation simulates an LLM cluster with 10,000 H100 GPUs and 312 inference replicas, using real-world request traces and PJM RegD signals. Regulation performance is assessed through average tracking error and a PJM-based score.

A. Experimental Setting

  • The simulated cluster has 10,000 H100 GPUs and 312 inference replicas, with execution modeled using VIDUR.VIDUR models response latency and model FLOPs utilization under different scheduling policies.Source: A. Experimental Setting · p. 7
  • Real-world request traces average 10,000 arrivals per second; baseline cluster power is 60% of rated capacity and regulation capacity is 10%.PJM regulation service is the representative case, using signals from the PJM RegD dataset.Source: A. Experimental Setting · p. 7
  • Regulation performance is evaluated using average tracking error and a PJM-based score calculated for each 5-minute settlement interval.Each regulation interval lasts 2 seconds, and each settlement interval contains 30 scoring intervals.Source: A. Experimental Setting · p. 7

B. LLM Inference Scheduling for Frequency Regulation

The proposed scheduler tracks regulation signals while keeping inference service constraints in view, and reward redistribution improves the stability and quality of regulation learning. Its interval-level reward assignment accommodates local deviations, while comparisons show the proposed method outperforming alternative reward strategies.

B. LLM Inference Scheduling for Frequency Regulation

  • 1.17% average tracking error, 0.94 s average TTFT, 6.36 s average E2E latency, and PJM mean and minimum 5-min scores of 0.926 and 0.910 under 10% regulation capacity.Power generally follows the target, with pointwise fluctuations from stochastic arrivals and heterogeneous workloads.Source: Figure 5 · p. 7, B. LLM Inference Scheduling for… · p. 8
  • During an up-regulation interval, the proposed strategy defers requests and limits processed tokens to reduce power, trading a slight latency increase for signal tracking while keeping SLOs within acceptable limits.The comparison covers five requests from one replica over 40 steps.Source: B. LLM Inference Scheduling for… · p. 8, Figure 6 · p. 8

C. Impact of Reward Redistribution

  • Without reward redistribution, tracking is unstable and the minimum score is 0.739, below the 0.75 PJM qualification threshold; the proposed method achieves mean and minimum scores above 0.90.The comparison reports regulation performance with and without reward redistribution.Source: C. Impact of Reward Redistribution · p. 8, Figure 7 · p. 8, Table II · p. 8
  • Within a regulation interval, instantaneous tracking errors can reach 90% while cumulative error is negligible by interval end; a step correcting accumulated error can receive positive reward despite a large local deviation.This reward assignment prioritizes interval-level consistency over immediate deviations.Source: C. Impact of Reward Redistribution · p. 8, Figure 8 · p. 8
  • Compared with no redistribution, uniform redistribution, and RUDDER, the proposed method accounts for long-term action impacts and temporal regulation dynamics, improving tracking while maintaining stable SLOs.Uniform rewards fail to distinguish action contributions, while RUDDER is less effective at capturing coupled scheduling effects under changing requests and signals.Source: C. Impact of Reward Redistribution · pp. 8–9, Figure 9 · p. 9
  • The proposed reward redistribution method improves steadily and has reduced training variance relative to baselines, whereas standard approaches face convergence challenges under sparse feedback.The training comparison is presented in Fig. 10.Source: C. Impact of Reward Redistribution · p. 9

D. Scalability Performance

The mean-field approach maintains stable performance as replica counts grow, balancing coordination and computational cost, while increasing regulation capacity exposes trade-offs with latency and SLOs. The paper concludes that inference scheduling can provide regulation service, and identifies extensions for future work.

D. Scalability Performance

  • With few replicas, centralized training provides the best tracking accuracy and lowest latency; mean-field achieves comparable performance with substantially shorter training time and sometimes outperforms it.The decentralized method converges fastest but has limited coordination and degraded performance.Source: D. Scalability Performance · p. 9
  • At 50 replicas, the centralized method fails to converge under the same hardware configuration, while the proposed mean-field method maintains stable performance and moderate training time across scales.Its accuracy decreases slightly in large settings, with the degradation described as acceptable.Source: D. Scalability Performance · pp. 9–10

E. Regulation Capacities and Performance Deterioration

  • As regulation capacity increases, upward regulation requires deferring more inference requests to reduce power, but SLO limits make perfect tracking infeasible at high capacities; request arrivals constrain prolonged downward regulation, further increasing tracking deviations.Source: E. Regulation Capacities and… · p. 10
  • All evaluated 5-min performance scores remain above 0.75, but capacities of 30% and above bring increased tracking errors and SLO violations alongside longer TTFT and E2E latency.The authors recommend balancing regulation performance against inference service quality when committing to capacity.Source: E. Regulation Capacities and… · p. 10, Table IV · p. 10

V. CONCLUSION

  • The conclusion describes a hierarchical MDP for request dispatching and token-level allocation, reward redistribution for dense feedback, and mean-field coordination of large replica populations.Real-world-data experiments demonstrate high tracking accuracy while maintaining SLOs.Source: §V · p. 10
  • Future work includes adapting the framework to sub-second products, optimizing capacity bids and adaptive baselines under changing conditions, and coordinating workload scheduling with GPU power capping or DVFS.These directions are proposed to exploit complementary flexibility for more accurate and reliable regulation.Source: §V · p. 10

APPENDIX A

APPENDIX A shows that redistributing rewards across a non-overlapping partition preserves the MDP’s value and optimal policies. APPENDIX B relates redistribution error to the value-function gap under a sublinear loss-decay assumption.

APPENDIX A

  • The trajectory is partitioned into non-overlapping segments that jointly cover all decision steps.Each segment contains a positive number of decision steps.Source: Appendix A · p. 10
  • The original and redistributed MDPs have the same value for every policy and identical optimal policy sets.The proof assumes trajectory segments form a non-overlapping partition.Source: Appendix A · p. 10

APPENDIX B

  • The value-function gap is bounded using the discounted return-conservation errors across reward segments.The proof defines a segment-wise discounted return-conservation error and relates the original and redistributed trajectory returns to those errors.Source: Appendix B · pp. 10–11
  • The analysis assumes the average segment-wise redistribution loss decays sublinearly with parameter updates.The assumption is motivated by convergence behavior commonly considered for stochastic first-order optimizers,.Source: Appendix B · p. 11

APPENDIX C

APPENDIX C analyzes lower-layer mean-field PPO: its critic approaches an asymptotically accurate advantage estimate near a stationary mean-field equilibrium, and the actor uses clipped PPO updates.

APPENDIX C

  • Given a converged critic, the actor is updated using the clipped PPO rule.The actor update uses sampled-trajectory stochastic gradients and learning rates satisfying the Robbins–Monro conditions.Source: Appendix C · p. 11
  • The critic’s stochastic updates track a limiting ODE under bounded gradient estimates and Robbins–Monro learning rates.A first-order expansion around the equilibrium and a negative-definite symmetric Jacobian part support the stability analysis.Source: Appendix C · p. 11
  • The lower-layer critic is locally exponentially stable near its mean-field Nash equilibrium, yielding an asymptotically accurate advantage estimate as learning approaches equilibrium.The stability claim holds in a sufficiently small neighborhood of the equilibrium.Source: Appendix C · p. 11

APPENDIX D

APPENDIX D analyzes the coupled upper- and lower-layer updates on different timescales. Under the stated learning-rate conditions, the coupled iterates converge asymptotically to a stationary solution.

APPENDIX D

  • The coupled upper-layer DQN and lower-layer mean-field PPO iterates asymptotically converge to a stationary solution when their learning rates satisfy Robbins–Monro conditions and the upper-to-lower rate ratio tends to zero.The upper layer evolves more slowly, allowing the lower layer to converge for a fixed upper-layer parameter.Source: Appendix D · p. 11
  • The upper-layer parameters track an ODE with a locally stable equilibrium, while a composite Lyapunov argument establishes convergence of both layers.The argument accounts for cross-layer coupling and uses the learning-rate separation.Source: Appendix D · p. 12
Loading 2610.23532v1…