PaperlayerView exact revision on arXiv

Source-linked AI summary

PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response

Yueying Li, Jiayang Chen, Yuanfan Chen, Leo Han, Haoran Qiu, Esha Choukse, Rodrigo Fonseca, Udit Gupta

arXiv:2608.21719v1cs.DCcs.AI

TL;DR

LLM inference clusters must satisfy instantaneous, time-varying power caps, but existing approaches do not account for phase-specific performance sensitivity or bounded user latency slack. PowerSlider combines a Flex SLO contract, prefill–think–answer disaggregation, stage-aware controls, and online KKT allocation to steer power where it costs least performance. On production SGLang traces, it achieves 78.3% goodput at a 30% cap reduction versus 47.6% for the best baseline and sustains at least 98% goodput through a replayed CAISO emergency day.

  • Problem

    Instantaneous demand-response caps create a serving constraint that static energy optimization and fixed-priority power shedding do not address across non-reasoning and reasoning workloads.

  • Method

    PowerSlider combines a Flex SLO contract with prefill–think–answer disaggregation and an online KKT solver controlling stage allocation, DVFS, and KV-cache partitioning.

  • Results

    78.3% online goodput versus 47.6% for the best baseline at a 30% cap reduction, with at least 98% goodput across a replayed CAISO grid-emergency day.

  • Takeaways & Limitations

    Phase asymmetry and Flex slack let PowerSlider reallocate scarce power toward configurations where shedding watts costs less goodput.

Abstract

from arXiv · show

AI inference clusters are increasingly constrained by instantaneous power, not just energy: grid operators condition new capacity on demand response, imposing time-varying power caps. Existing LLM serving systems optimize a static energy objective or shed fixed priority tiers under load; either way, goodput collapses when the power envelope moves. An LLM pipeline is not a uniform load: compute-bound prefill loses throughput almost linearly with GPU frequency, memory-bound answer decode sustains it down to $0.57\times$ nominal, and reasoning's thinking phase couples KV-cache capacity to scheduling -- so a cap should be steered to where each watt costs the least performance. PowerSlider does so with a new Flex SLO contract that turns bounded user slack into an optimization constraint, prefill--think--answer disaggregation exposing per-stage frequency and KV control, and a Karush--Kuhn--Tucker (KKT) online solver re-solving within 7.7 ms of every cap change, backed by a consolidated fail-safe that power-gates drained instances when DVFS bottoms out on static power. On SGLang with production traces, \sys{} sustains 78.3\% online goodput at a 30\% cap reduction versus 47.6\% for the best of five baselines ($1.64\times$), holds latency-critical tails within $1.3\times$ of nominal (baselines: $2.3$--$6\times$, up to $12\times$), and delivers 92\% mean goodput through a replayed CAISO grid-emergency day bottoming at $0.41\times$ (54\% at the trough; every baseline below 7\%).

I. INTRODUCTION

PowerSlider addresses instantaneous, time-varying power caps by exploiting phase asymmetry and bounded latency slack across LLM serving. Its Flex-aware, stage-aware online runtime sustains goodput under deep caps where static energy optimization and fixed-priority shedding fall short.

  • Motivation: Prefill is compute-bound, answer decode is memory-bandwidth-bound, and reasoning’s thinking phase couples accumulated KV-cache state to scheduling.These phase differences make uniform power capping inefficient and require stage-aware control.
  • Motivation: Latency slack in a Flex tier can be converted into power headroom by allowing bounded degradation during demand-response windows without shedding an entire service class.The paper grounds this contract in existing commercial API tiers with higher variable latency and lower cost.
  • Design: POWERSLIDER combines Flex SLO constraints, prefill–think–answer disaggregation, per-stage DVFS and KV-cache controls, and a KKT-based online solver for changing caps.The runtime jointly reallocates power across pipeline stages, service classes, and memory footprints.
  • Results: At a 30% cap reduction, POWERSLIDER achieves 78.3% online goodput versus 47.6% for the best baseline, a 1.64× improvement.It also holds latency-critical tails within 1.3× of nominal, while baselines reach 2.3–6× and up to 12×.
  • Motivation: Data-center demand response imposes an instantaneous power cap pmax(t), which differs fundamentally from an energy objective because the constraint binds at every instant.Real-time markets can dispatch and settle every 5 minutes, making runtime cap compliance an operational requirement.

B. LLM Serving Phases and Their Metrics

LLM serving phases respond differently to frequency scaling, while reasoning adds long, memory-intensive thinking; therefore, dynamic caps require stage-aware online control rather than uniform or offline policies.

  • Phase asymmetry: Prefill is compute-bound, decode is memory-bandwidth-bound, and reasoning adds a long thinking phase with substantial KV-cache demand.Thinking sequences can be 10–100× answer length and introduce TTFAT as an additional latency metric.
  • Flexible slack: Controlled latency degradation can reduce peak power, but rigid tiers instead shed best-effort traffic and eventually violate latency-critical targets.A Flex tier preserves both BE service and higher total goodput under tightening caps.
  • Flexible slack: The Flex contract maps bounded latency inflation into an optimization constraint under changing caps.It specifies a maximum latency multiplier and the fraction of event time for which degradation is allowed.
  • DVFS asymmetry: At small batch sizes, throughput saturates above approximately 810 MHz, whereas large batches scale nearly linearly through 1,410 MHz.This reflects memory-bandwidth-bound versus compute-bound regimes on A100-80GB.
  • Online control: Because the saturation knee shifts with batch size and allocation, frequency, batch size, and GPU allocation must be jointly reoptimized at every cap change.The resulting nine-group configuration space contains (10 × 9)^9 combinations, making dense offline profiling impractical.

C. Observation 3: Reasoning Workloads Break PD Disaggregation and Prediction-Based Routing

Reasoning workloads stress shared decode resources through long-lived KV state and unpredictable lengths, motivating PTA disaggregation, adaptive admission, and Flex-aware scheduling.

  • Observation 3: Power caps push reasoning workloads toward saturation, where both PD disaggregation and prediction-based routing become fragile.Long thinking chains create memory pressure and make length prediction unreliable.
  • PTA disaggregation: PTA separates prefill, think, and answer into dedicated GPU pools, exposing independent stage controls for reasoning workloads.The pipeline gives the optimizer per-stage frequency, chunk-size, and allocation variables.
  • PTA disaggregation: PTA gives think instances 44% higher batch capacity than PD at typical parameters by allowing stage-specific chunk sizes.It also decouples the frequency and chunk size of the two decode-like stages.
  • Flex scheduling: The Flex contract permits latency up to αcLc for at most a ρc fraction of an event, while BE requests may be deferred within 24 hours.For example, αc=2 and ρc=0.4 allow 2× latency for at most 40% of the event.
  • Flex scheduling: The contract becomes a smooth per-stage impact bound derived from overload probability and throughput capacity.This bound supplies the optimization interface for admission and resource allocation.

C. PSOpt Engine: Formulation and KKT Solver

PSOpt converts dynamic caps and Flex impact limits into a joint allocation problem, then uses convex relaxation and KKT structure to choose low-cost degradation and actuate it across timescales.

  • C. PSOpt Engine: Formulation and KKT Solver: PSOpt models each stage-class using GPU count, frequency, chunk size, compute-bound fraction, and KV-cache-limited capacity.Effective capacity is the minimum of compute-limited and memory-limited throughput.
  • C. PSOpt Engine: Formulation and KKT Solver: Its power model combines cubic dynamic power with frequency-independent static leakage and board/HBM idle power.Hardware-specific coefficients are profiled offline and can be refit online when observed latency drifts.
  • C. PSOpt Engine: Formulation and KKT Solver: At each epoch, PSOpt jointly selects GPU allocation, frequency, and chunk size under the power cap, impact bounds, and global GPU budget.The objective weights user-visible impact and allocation churn, prioritizing LC over Flex and BE.
  • C. PSOpt Engine: Formulation and KKT Solver: A convex relaxation replaces the roughly 10^18 discrete configurations with continuous GPU counts, frequencies, and chunk sizes suitable for KKT solution.Convex impact and churn terms make the relaxed problem tractable.
  • C. PSOpt Engine: Formulation and KKT Solver: PSOpt lowers frequency in ascending impact ratio, reclaiming watts first from memory-bound stages and classes with available slack.This ordering operationalizes the measured phase asymmetry and Flex tolerance.
  • C. PSOpt Engine: Formulation and KKT Solver: Under a deep cap, answer decode drops onto its throughput plateau, thinking settles higher, and LC prefill remains nominal instead of being uniformly slowed.The relaxed solution is projected onto discrete GPU counts and supported DVFS states.
  • D. PSSched and PSRoute: Actuating the Solution: PSSched and PSRoute actuate PSOpt’s target through pool resizing, frequency control, and adaptive admission based on observed lengths.Together they close the loop across event, minute, and millisecond timescales.

V. IMPLEMENTATION

The implementation realizes PTA through coordinated per-request DVFS, drain-before-reassign migration, persistent KV transfer, and hardware-enforced power limits.

  • Deployment: POWERSLIDER builds on SGLang with disaggregated prefill, think, and answer pools that exchange KV cache over TCP or RDMA.The deployment separates control components by execution cadence and critical-path role.
  • Pool management: GPU reassignment uses drain-before-reassign: instances stop admitting work, finish or migrate in-flight KV state, then join the new pool.This bounds migration cost by residual in-flight work but makes pool movement slower than DVFS response.
  • DVFS: The vote-commit daemon aggregates per-request stage targets and changes each GPU’s frequency every 50–100 ms with one clock-lock call.It also reasserts the NVML power limit as an independent backstop.
  • KV transfer: Persistent reference-counted worker sessions and sliding-window flow control support PTA’s extra think-to-answer KV transfer.The RDMA backend uses one-sided writes with the same framing.
  • Power enforcement: NVML power limits enforce the cap even when PSOpt’s model is stale or mispredicted, apart from a bounded subsecond transient.The optimizer chooses an operating point; hardware enforcement determines whether the cap holds.
  • Power enforcement: When a cap is too deep for all active GPUs at fmin, PSOpt consolidates load onto fewer GPUs during the next drain-bounded reallocation.Until consolidation completes, the NVML backstop maintains the cap.
  • Admission control: PSRoute prevents starvation by shedding BE first, using Flex slack next, and promoting overdue Flex requests to LC-equivalent priority.If LC becomes infeasible, it uses admission control rather than silently violating targets.

VI. EXPERIMENTAL METHODOLOGY

The evaluation combines real-system and simulator studies across reasoning and non-reasoning workloads, power-cap scenarios, latency and goodput metrics, and multiple baselines. POWERSLIDER sustains substantially higher goodput under severe reductions, including 100% non-reasoning goodput through a 60% reduction and 78.3% online goodput at a 30% reasoning-workload reduction.

  • Experimental Setup: The study uses DGX-A100 and GH200 instances with SGLang and Mooncake, plus a 64–512-GPU SplitwiseSim simulator.The real-system evaluation uses fewer than eight GPU nodes; the simulator models three-stage disaggregation, KV transfer, DVFS, and runtime control.
  • Experimental Setup: The workload suite mixes reasoning traces with Azure coding and chat traces under dynamic, bursty arrivals and varying traffic mixes.The default mix is 30% LC, 30% Flex, and 40% BE at saturation throughput.
  • Experimental Setup: The evaluation sweeps static 0–60% cap reductions and a 24-hour dynamic demand-response trace, including regimes where consolidation is required below the DVFS static-power floor.Metrics include per-class goodput, P90/P99 TTFAT and TTLT, and normalized BE throughput.
  • Experimental Setup: The comparison spans five baselines across architecture, DVFS strategy, and multi-SLO resource-management axes.The design questions cover goodput, tail latency, Flex sensitivity, component contributions, adaptation speed, overheads, hardware generality, and grid participation.
  • Goodput Results: 100% goodput is maintained for all three non-reasoning classes through a 60% reduction, while POLCA reaches 67.7% online goodput and other baselines fall below 15% beyond 40%.The result uses the 0.57× decode-frequency knee and Flex slack to absorb the remaining reduction.
  • Goodput Results: 1.64× more goodput is sustained than the best baseline under a 30% reduction in the high-load reasoning setting.POWERSLIDER retains 78.3% online and 54% BE goodput, versus 47.6% online and zero BE for SLOs-Serve+.
  • Goodput Results: 95–100% online goodput is maintained as the Flex share drifts from 5% to 80%, compared with 10% for POLCA.This time-varying-mix result appears in the Appendix F study.

B. Q2: Tail Latency under Power-Cap Reductions

POWERSLIDER keeps latency tails bounded under deep caps by combining stage isolation, Flex-aware admission, and online power allocation. The same mechanisms support continuous cap tracking, fast solver re-solves, and robust operation through a replayed grid-emergency day.

  • Tail Latency: At a 60% reduction, POWERSLIDER keeps LC TTLT at 89 s and TTFAT at 54 s, only 1.1× and 1.3× above uncapped latency.POLCA reaches 220 s LC TTLT and 182 s Flex TTFAT, while three other baselines exceed 750 s LC and 900 s Flex latency.
  • Component Ablation: The full design minimizes P90 latency across all regimes, while removing any one of KKT allocation, PTA disaggregation, or Flex raises latency substantially.At a 30% reduction, ablations increase LC TTLT by 74–100% and LC TTFAT by 59–101%.
  • Flex Sensitivity: At a 60% reduction, POWERSLIDER achieves 1.6–1.85× POLCA’s goodput across a broad Flex-parameter range.Increasing α from 2 to 10 yields little additional benefit compared with increasing it from 1.1 to 2.
  • Flex Sensitivity: The Flex contract remains within its ρc budget at every cap depth, with a worst observed violation mass of 27.5% at a 60% reduction.The residual mass is dominated by requests shed through admission control rather than slow completions.
  • Dynamic Adaptation: POWERSLIDER holds at least 98% online goodput across a replayed emergency day, while baselines fall to 0–6.6% near the 0.41× trough.The baselines do not recover within the trace because the day ends still capped at 0.77×.
  • Dynamic Adaptation: Delivered power stays under the binding cap throughout the replayed emergency day, including the trough where DVFS reaches the static-power floor.The consolidation fail-safe power-gates drained instances to shed the remaining power, with an NVML backstop during the steepest descent.
  • Dynamic Adaptation: PTA avoids the decode queueing that appears in PD under capped reasoning workloads, yielding 12.5× lower end-to-end latency.At nominal power, PD and PTA have similar latency because the additional think-to-answer handoff is hidden by pipelining.
  • Dynamic Adaptation: A 7.7 ms solve time in the nine-group setting and 63 ms at 81 groups keep PSOpt within a 100 ms DVFS interval.The solver cost is independent of cluster size because group GPU counts enter the capacity model rather than expanding the decision space.

G. Q7: Generality and the Economics of Participation

The paper positions POWERSLIDER as a grid-facing layer for architecture-aware, multi-class power allocation, distinct from fixed-envelope and static-priority approaches. Its evaluation reports positive demand-response economics, while the discussion identifies hardware-control and telemetry limits that constrain faster or finer-grained participation.

  • Generality: A cubic power model fitted on A100 also fits public H100 curves, requiring only one refitted power cubic per hardware type.This supports hardware portability across the evaluated GPU families.
  • Economics: POWERSLIDER remains net-positive throughout the ERCOT-ERS scenario, while baseline revenue loss exceeds the demand-response payment at 27–34% depth.No baseline reaches 90% goodput beyond a 30% cap.
  • Hardware Limits: Current whole-GPU frequency control takes tens to hundreds of milliseconds, forcing vote–commit batching and limiting participation in fast grid programs.The paper identifies per-partition frequency domains with microsecond transitions as a route to faster control.
  • Hardware Limits: Opaque firmware-mediated enforcement limits trustworthy cap guarantees and motivates on-package telemetry with bounded response time and per-SM/HBM attribution.Such hardware would support contractual exceedance bounds and compute-granularity power management.
  • Positioning: POWERSLIDER extends multi-class QoS and runtime power-control ideas to GPU LLM inference by combining contractual slack with stage-aware allocation under varying caps.The paper contrasts this with CPU request-level slack and fixed-priority GPU systems.
  • Positioning: Unlike fixed-envelope power-management systems, POWERSLIDER decides which stages, service classes, and memory configurations absorb a moving cap.The related systems described include PowerWeave, Power Sloshing, and PowerGrad.
  • Conclusion: Dynamic power caps make LLM serving an architecture-aware online control problem because stages differ in compute intensity, memory pressure, DVFS sensitivity, and SLO slack.The proposed response is the Flex contract, PTA disaggregation, and a KKT-guided solver.

APPENDIX

The appendix models transformer-layer compute intensity and shows why prefill is compute-bound while decode, especially with long contexts, is memory-bound and frequency-reduction friendly.

  • KKT interpretation: Stage-classes with SLO headroom have η_i = 0 and are degraded first, while η_i becomes positive when the impact bound is tight.The multiplier therefore identifies which stage-class constraints are active.
  • Hardware balance: The hardware balance point is I⋆ = C/W, separating compute-bound layers with AI ≥ I⋆ from memory-bound layers with AI ≪ I⋆.C denotes peak FLOP/s and W denotes HBM bandwidth.
  • Arithmetic intensity: Prefill with large batch or prompt length has high arithmetic intensity and approaches a fully compute-bound fraction γ_i.The model states that prefill uses large B, causing AI to grow with B and γ_i to approach 1.
  • Arithmetic intensity: Decode with small batch is memory-bound because its arithmetic intensity is far below the hardware balance point.For decode, AI ≈ 2B/bw ≪ I⋆, so γ_i ≈ 0.
  • Arithmetic intensity: Longer context increases the KV-cache memory term, further reducing γ_i for decode and reasoning traces.This supports reducing decode frequency with minimal throughput loss.

3) Throughput, Power, Latency, and Energy Models:

The serving models fit latency, throughput, power, and energy as frequency-dependent functions, with distinct behavior for compute-bound and memory-bound stages.

  • Latency: Per-batch latency is piecewise linear in frequency, with a batch-size-dependent saturation-frequency knee f⋆(BS).The two segments use slopes b1 and b2 on either side of the knee.
  • Latency: Memory-bound stages have b2 ≪ b1, so lowering frequency beyond f⋆ reduces power with negligible latency cost.Compute-bound stages instead may remain on a single linear segment across the frequency range.
  • Throughput: Small-batch throughput saturates above approximately 810 MHz, whereas large-batch throughput scales nearly linearly up to 1,410 MHz.These regimes match the two latency-slope behaviors.
  • Power: Per-GPU power follows P(f) = c2f^3 + c1f + c0, and aggregate stage power is k_iP(f_i).The fitted power model has R2 above 0.95 across all profiled points.
  • Energy: Energy per token has an intermediate-frequency minimum in compute-bound regimes but decreases monotonically with lower frequency in memory-bound regimes.This justifies aggressive frequency reduction for answer decode.

C. From Static Provisioning to Dynamic Runtime Routing

PowerSlider combines static stage-pool provisioning with per-request dispatch and on-demand pool borrowing, allowing runtime capacity to follow workload skew between optimization epochs.

  • Static provisioning: The allocator provisions Prefill, Think, and Answer pools at startup, while the dispatcher routes requests across stage-tagged instances.Requests are classified into two-phase or three-phase paths according to whether they emit reasoning traces.
  • Dynamic dispatch: Dispatch selects the least-loaded eligible instance using memory reservations, pending-token counts, or a weighted combination.Reservations are updated before dispatch returns, and KV-cache transfers use the destination’s inbound bandwidth.
  • Online pool rebalancing: When one stage saturates while another is idle, the dispatcher borrows an underutilized sibling instance and places it in a shared mixed pool.The instance returns to its home pool after its in-flight work drains.
  • Online pool rebalancing: Online rebalancing softens the static P/T/A boundary per request, while PSOpt performs coarse reprovisioning at epoch boundaries.Class-visibility filters preserve the SLO-tier ordering during rebalancing.
  • Runtime structure: Algorithm 2 separates one-shot PROVISION at epoch boundaries from per-request DISPATCH with on-demand REBALANCE.The algorithm exposes the two-layer runtime structure.

D. Experimental Setup Details

The evaluation section defines workload and baseline setup, then examines routing errors and the gap between predicted and phase-aware routing across workload distributions.

  • Evaluation setup: Table IV lists the evaluated workloads and their latency targets, while Table V summarizes five baselines using the same cluster and power-aware routing.The tables establish the workload requirements and comparison configurations.
  • Routing analysis: The length classifier has a 37.1% overall misprediction rate and sends 26.6% of Medium and 16.3% of Long requests to the Short pool.These misroutes inflate Short-pool load by 25%.
  • Routing analysis: Medium requests are hardest to classify, with only 49.9% routed correctly after class-size recalibration.Figure 13 details the resulting misrouting cascade.
  • Workload sensitivity: Figure 14 sweeps think-to-answer ratio and prefill length to compare PD and PTA across workload-distribution space.The sweep supports the Observation 3 analysis.

1) Per-Request Transient Behavior:

Across transient load, workload-mix, power-cap, and economic tests, POWERSLIDER sustains service more consistently than baselines by adapting to changing demand and power constraints. Its benefits persist across workload distributions and hardware-related scope boundaries, though full H100 evaluation remains future work.

  • 49%–71% TBAT reduction is achieved as the T/A ratio rises from 0.5:1 to 16:1 at high load.The advantage grows because longer thinking chains create more batch interference under PD and colocated serving, which PTA eliminates by isolating answer decode.
  • A full H100 evaluation is left to future work despite the cubic DVFS model fitting public H100 power after coefficient refitting.The simulator carries an H100 profile, but the reported expectation of improved deep-cap retention is not a full evaluation.
  • POWERSLIDER completes the most requests across LC, Flex, and BE classes with stable response times after a 90%–60% cap step.Baselines either shed BE traffic or accumulate unbounded queueing.
  • Up to 1.85× the goodput of POLCA is delivered at a 60% power-cap reduction across a wide range of Flex SLO parameters.The result is reported while varying relaxation ratio α and violation-budget fraction ρ.
  • 0.34–0.44 J/token is maintained at 97.9–100% goodput across cap depths, including 0.42 J/token with 97.9% online load served at a 60% reduction.Stage-aware DVFS slows memory-bound pools where throughput is nearly frequency-insensitive.
  • $103 per 2 h event is reached at a 60% reduction under scarcity-priced events, while every baseline’s revenue loss overtakes DR payment at moderate depths.POWERSLIDER’s revenue loss never exceeds $4 per event because it holds online goodput at 97.9–100%.
  • 7.7 ms solve time plus a 100 ms DVFS interval supports sub-second responses for caps up to approximately 15–20%, while deeper responses use one 5 min reallocation epoch.These actuation paths fit AGC frequency regulation, spinning-reserve, and emergency-DR windows.
  • 95–100% online goodput is sustained across a 5%→80% Flex workload drift at a 30% power-cap reduction.At 80% Flex, goodput is 95.3%; when full service is infeasible, POWERSLIDER sheds BE admissions.
Loading 2608.21719v1…