PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

Jae Gon Kim, Donghoon Yoo, Hanyul Ryu, Sungho Ha, Juyeon Lee, Soojung Ryu

arXiv:2609.11133v1cs.LGcs.DC

TL;DR

LLM-serving power is a capacity constraint, while a shipped vendor profile provides modest, model-dependent efficiency gains with latency costs and phase-blind control. The paper introduces phase-decoupled, per-stack calibration with latency-gated operating modes, achieving larger efficiency gains while preserving tail-SLO compliance within the evaluated MoE-serving scope.

  • Problem

    GPU power constrains LLM-serving capacity, while fixed vendor profiles offer modest, model-dependent gains, latency costs, and one setting for prefill and decode despite their different hardware regimes.

  • Method

    A phase-decoupled controller calibrates prefill clock windows and decode power caps automatically for each model, quantization, engine, and hardware combination, using latency-gated acceptance criteria.

  • Results

    1.5-3.4× the vendor profile’s efficiency gain was achieved, with every latency-gated mode holding the tail SLO in repetitions where static vendor profiles missed it; the controller Pareto-dominated Max-Q on agentic workloads.

  • Takeaways & Limitations

    Phase-aware, per-stack calibration can convert serving latency headroom into higher energy efficiency and more SLO-compliant capacity than a fixed vendor profile in the evaluated MoE workloads.

  • Takeaways & Limitations

    The claims are scoped to MoE serving, and the runtime guard was not triggered in recorded runs, leaving direct validation of runtime intervention outstanding.

Abstract

from arXiv · show

Datacenter GPU power is the binding constraint on LLM serving capacity, and production serving has shifted to prefill/decode (PD) disaggregation. Deploying NVIDIA's Max-Q inference profile on a disaggregated B200 system, we found its realized gain modest (+8.6% tokens/J), model-dependent, and carrying a mean end-to-end latency cost (+5.2%) that throughput-only evaluation does not surface; the profile also applies one setting to prefill and decode GPUs that operate in opposite hardware regimes. We hypothesize that the optimal power setting is a property of the deployed (model, quantization, engine, hardware) combination rather than of the GPU class, that each lane warrants its own profile, and that converting SLO headroom into energy safely requires latency-gated calibration under a runtime SLO guard rather than a fixed recipe. We present a phase-decoupled, model-calibrated controller: the prefill lane runs under an SM-clock window whose floor is a latency guarantee by construction, and the decode lane under a power cap placed by automatic calibration just above a measured throughput/latency cliff. Because a disaggregated decode lane draws flat, memory-bound power, the cap binds continuously, the reactive-overshoot weakness that led POLCA to reject capping is absent, and the GPU's own power manager retains throughput under the cap. On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e versus +8.6% at +5.2% for Max-Q, a Pareto improvement on both axes. On Qwen3-235B-A22B (NVFP4) every operating mode meets the ITL-p99 SLO in every repetition; both vendor profiles miss it. A decode-actuator A/B shows the calibrated cap beats static clock locks, and a three-day sustained run saves 32.3% of a lane pair's electricity. Both models are MoE; a dense model recovers roughly 5x less, so we scope our claims to MoE serving.

1 Introduction

The paper argues that GPU power is the binding resource in LLM serving, while fixed, phase-blind vendor profiles leave model- and stack-specific efficiency and latency headroom unused. It proposes phase-specific, per-stack calibration with tail-latency gating and reports higher efficiency than Max-Q within a scoped MoE-serving evaluation.

  • Introduction: The motivation is that Max-Q delivered only +8.6% tokens/J while increasing mean end-to-end latency by +5.2% on a 480B agentic workload.Throughput-oriented validation did not surface the latency cost.
  • Introduction: The controller calibrates distinct actuators and settings for prefill and decode per model, quantization, engine stack, and hardware combination.The design assigns mechanisms by lane rather than applying one GPU-class recipe.
  • Introduction: Latency-gated operating modes hold the tail SLO in every repetition where both static vendor profiles miss it in some runs.The recorded runtime guard never fired, so this contrast is attributed to latency-gated calibration rather than runtime intervention.
  • Scope: The paper scopes its claims to single-node steady-state MoE serving because a comparable dense model recovered roughly 5× less efficiency.Training and cluster-level oversubscription are outside the stated scope.

2 Background and Motivation

PD disaggregation makes prefill and decode persistent, spatially separated regimes with different bottlenecks, exposing why one power actuator should not govern both. The paper therefore contrasts clock control for compute-bound prefill with adaptive power capping for memory-bound decode near a throughput cliff.

  • Background: Prefill is compute-bound and power-hungry, whereas decode is memory-bandwidth-bound with flat power below TDP.Disaggregation places the phases on separate GPU pools, making lane identity sufficient to identify the bottleneck regime.
  • GPU power knobs: A decode lane has no prefill spike, so power capping becomes steady-state control of a flat signal rather than reactive overshoot control.This removes the overshoot scenario that motivated POLCA’s rejection of capping.
  • Mechanism split: For prefill, β≈1 makes clock changes track execution time, while decode with β≪1 reaches a bandwidth-saturation cliff below which throughput collapses.The model motivates a clock window for prefill and a cliff-aware actuator for decode.
  • Mechanism split: A calibrated cap delegates frequency allocation to the GPU power manager, which continuously adapts the SM/memory split instead of pinning one clock.The software-predicted adaptive-frequency alternative is not evaluated in the supplied evidence.
  • Calibration motivation: Power and decode cliffs vary with model, quantization, and engine configuration, motivating fingerprinted calibration rather than transferable fixed settings.Measured engine changes shifted natural decode power and inter-token latency.

3 Design

The design uses separate per-lane actuators, a three-mode operating ladder, automatic calibration, and an SLO guard. Prefill receives a calibrated SM-clock window, while decode receives a calibrated power cap accepted only when throughput and tail latency remain within bounds.

  • 3 Design: The controller maximizes tokens/J subject to an ITL-p99 SLO using per-lane settings drawn from a calibration artifact indexed by model, quantization, engine, and hardware.The artifact supplies distinct decode caps and prefill clock windows across PERF, BAL, and EFF modes.
  • Phase-decoupled actuation: Decode uses a calibrated power cap whose GPU-managed DVFS preserves throughput while reducing power, outperforming static clock locks in the reported A/B.The cap delegates the SM/memory frequency split to the on-device power manager.
  • Phase-decoupled actuation: Prefill uses a calibrated SM-clock window whose floor guarantees the latency bound by construction, with runtime adjustment inside the window.The measured results do not separate the floor’s contribution from the active in-window policy.
  • Operating modes: The three operating modes spend progressively more calibrated headroom, with BAL recommended as the default because it captures most EFF gain at lower latency cost.All modes share one artifact and the same runtime SLO guard.
  • Automatic calibration: The calibration gate accepts a setpoint only when throughput remains within tolerance of reference and ITL-p99 stays within the SLO, then records the engine fingerprint.Any fingerprint change invalidates the artifact and triggers recalibration.
  • Automatic calibration: A latency-only correction moved the accepted 235B decode cap from near 65% to 80% of natural draw and restored 100% SLO compliance at +22.4% tokens/J.Throughput-only acceptance preserved throughput but violated ITL-p99 in most repeated runs.

4 Experimental Setup

The evaluation uses controlled, closed-loop workloads and measures energy across the full 8-GPU node alongside mean and tail latency. It compares a shipped Max-Q baseline with controlled alternatives under reproducible, fixed-concurrency conditions.

  • Hardware and serving stack: The study evaluates Qwen3-Coder-480B FP8 and Qwen3-235B-A22B NVFP4 on a single 8× B200 node using NVIDIA Dynamo with PD-disaggregated serving.The 480B model uses separate TP4 prefill and decode GPU groups, while the 235B model uses TP1 PD across GPUs 0–1.
  • Latency metrics: Latency evaluation includes mean end-to-end time, TTFT p95/p99, and ITL p95/p99 against a fixed ITL-p99 SLO.Each head-to-head arm and workload cell uses three repetitions, with dispersion reported as ±1σ where applicable.
  • Workloads and load control: Both workloads use fixed-concurrency closed-loop streams, keeping one request in flight per client and maintaining stationary lane operating points.The agentic workload uses four tool-calling turns per client with growing context, while standard requests are drawn from ShareGPT.
  • Latency interpretation: Mean end-to-end latency complements throughput because fixed closed-loop concurrency makes it proportional to inverse completion rate, exposing degradation that aggregate tokens/s can conceal.Identical request streams normalize token counts across arms, while tail ITL p99 protects the streaming experience.
  • Baseline: Every Max-Q comparison uses the shipped profile, which saves 12.5–12.9% of baseline power versus 4.0–4.1% for a uniform SM-clock reconstruction.The same hardware block is used for the comparison, avoiding an understated reconstructed baseline.
  • Energy measurement: The evaluation reports tokens/J from client-observed tokens divided by energy integrated from NVML power samples across all eight GPUs.Power is sampled every 100 ms, and the denominator includes idle and other lanes rather than only actively controlled GPUs.

5 Evaluation

Across workloads and models, the calibrated phase-specific controller improves the efficiency–latency tradeoff over Max-Q while preserving tail-latency compliance, with power caps outperforming static clock locks on decode.

  • 5.1 Head-to-head vs NVIDIA Max-Q: +20.4% tokens/J at +3.5% mean-e2e makes BAL a Pareto improvement over Max-Q’s +8.6% at +5.2% on 480B agentic serving.EFF reaches +29.3% tokens/J at +4.9% mean-e2e, or 3.4× Max-Q’s efficiency gain.
  • 5.1 Head-to-head vs NVIDIA Max-Q: On repeated 235B workloads, EFF reaches +22.4±2.4% tokens/J at +8.9% mean-e2e, while BAL exceeds Max-Q outside the reported dispersion on both workloads.Every operating mode meets the ITL-p99 SLO in every repetition, whereas both vendor profiles miss it in a fraction of agentic repetitions.
  • 5.2 The operating-mode ladder is monotone and SLO-guarded: Efficiency and mean power savings increase monotonically from PERF to EFF, while every ladder rung remains within the ITL-p99 SLO and Max-Q falls inside the ladder.On standard traffic, efficiency rises from +7.6% to +22.4% and mean power saved from +10.8% to +29.5%.
  • 5.3 Decode-actuator A/B: The calibrated decode cap preserves throughput and beats static locks: throughput-optimal locks cut decode power only 5–9%, whereas aggressive locks reduce throughput by 31–33%.The cap exploits GPU self-DVFS, bounding peak power while retaining adaptive clock allocation.
  • 5.6 Load dependence: the advantage across the concurrency range: Decode-cap savings track binding fraction, reaching 94–99% under standard traffic but only 13–24% under high-concurrency agentic traffic.The paper reports this as an association between realized load and cap engagement, not as an independently manipulated variable.
  • Limitations: The study’s evidence is scoped to steady-state single-node MoE serving, while dense-model recovery is roughly five times lower and factorial lane attribution remains unmeasured.The telemetry attributes power reductions by lane but cannot isolate efficiency contributions from prefill-window-only and decode-cap-only ablations.

6 Discussion

The controller’s efficiency gains translate into higher facility capacity and predictable per-GPU power bounds, but the evidence remains scoped to single-node MoE serving and several operational questions remain open.

  • Scope and limitations: The evaluation is limited to single-node B200 serving of two Qwen3-family MoE models under one engine stack, with external validity under native trace arrivals untested.Cross-node power shifting and cluster-level scheduling are treated as higher layers rather than evaluated here.
  • Scope and limitations: The study makes no dense-model claims because cap-only recovery was roughly 5× smaller on a comparable dense model than on the evaluated MoE models.The reported comparison is +5.7±8.0% versus +26.0±6.4% cap-only tokens/J.
  • Datacenter implications: The cap-based decode lane deterministically bounds per-GPU peak power by the commanded cap, unlike scheduling-based savings alone.This bound is the quantity used in provisioning and oversubscription planning.

7 Related Work

Prior work largely uses phase-aware frequency control or cluster-level scheduling, while this paper selects actuators per lane, calibrates them per serving stack, and evaluates against shipped vendor profiles on latency and energy.

  • The calibrated phase-decoupled ladder Pareto-dominates NVIDIA’s in-flight vendor profile on the agentic workload.The paper also notes that NVIDIA’s multi-knob recipes outperform frequency-only scaling by 7-9% in its own data.
  • Cluster-level systems such as TAPAS and RAPID provide placement, routing, and role-assignment levers that are orthogonal to per-lane actuation.The paper presents these approaches as composable scheduling layers rather than direct competitors.
  • Prior DVFS systems use frequency actuation, whereas this work replaces decode’s calibrated static frequency setpoint with a calibrated power cap.The paper contrasts this mechanism with VoltanaLLM, DualScale, and other frequency-based controllers.
  • Prior power-capping work targets monolithic serving, while this paper argues that disaggregated decode restores cap viability through cap placement.This reconciles its result with prior claims that clock locking dominates decode power control.
  • Training-energy systems motivate exclusion because training lacks the spatial phase lanes used by this serving design.The paper treats training methods as an analogue rather than a competing serving approach.
  • The paper positions per-lane mechanism selection, stack-tracking calibration, and latency-aware evaluation as gaps beyond phase awareness alone.Its comparison targets the mean-e2e and tail-ITL plane rather than throughput alone.

8 Conclusion

The paper addresses the gap between reported and realized vendor-profile efficiency by combining phase-decoupled actuation with per-stack calibration. Its calibrated ladder improves efficiency while preserving tail-SLO compliance in the reported comparisons.

  • The calibrated operating-mode ladder achieves 1.5-3.4× the vendor profile’s efficiency gain and Pareto-dominates it on agentic workloads.Every latency-gated operating point holds the tail SLO in repetitions where static vendor profiles miss it in a fraction of runs.
  • The two lanes require different mechanisms, not merely different settings, with calibrated decode power capping outperforming static frequency locking.The paper attributes this to disaggregation turning phase management from temporal to spatial.
  • The paper concludes that throughput-only validation and intuition-based power-control conclusions do not persist once mean end-to-end latency is evaluated.

Future Work

Future work targets untested adaptive control, runtime-guard behavior, lane-specific ablations, and more realistic arrival processes, while deferring training-side comparison.

  • Future work includes testing an adaptive per-phase DVFS controller under the paper’s SLO gate as a fourth decode A/B arm.
  • A fault-injection study is proposed to force the runtime guard to trip and measure recovery time and cost.
  • The planned factorial per-lane ablation includes a floor-only prefill arm to separate the in-window policy’s contribution.
  • Future evaluation will use native trace arrival processes, while training-side comparison remains deferred because training lacks spatial phase separation.

Reproducibility

The paper discloses measured configurations, calibration and runtime-guard contracts, source data, and measurement protocols, but two control components remain specified only behaviorally and one contribution is not isolated.

  • The reproducibility package discloses every measured configuration, calibration acceptance contract, runtime-guard contract, source data, and measurement protocol.Setpoints are reported relative to locally measurable natural draw and maximum clock anchors.
  • The setpoint-search heuristic and in-window clock policy are described only through behavioral contracts rather than implementation details.The paper states that any search passing the acceptance gate produces valid setpoints.
  • The in-window policy’s contribution to prefill-lane savings is not separated because it was active in every measured arm.A floor-only reproduction is identified as the experiment needed to isolate that contribution.

A The actuator-selection campaign

The actuator-selection campaign explored phase-specific control across models, quantizations, workload shapes, and serving topologies to determine the target design.

  • A The actuator-selection campaign: The exploratory campaign covered four base models, three quantizations, three workload shapes, and both colocated and disaggregated serving configurations.This campaign preceded the main evaluation and informed the per-lane mechanism assignment.
  • A The actuator-selection campaign: The campaign focused on why disaggregated serving, power caps, MoE models, and phase-decoupled actuation are appropriate targets.The appendix organizes this evidence around target selection, cap suitability, MoE benefit, and run-granularity behavior.

A.1 Phase signatures are model-specific but topology-invariant

Prefill and decode form distinct activity signatures for every tested model, but the separation magnitude varies widely while remaining invariant to serving topology.

  • A.1 Phase signatures are model-specific but topology-invariant: 100% held-out classification accuracy separates prefill and decode using tensor-pipe and DRAM activity counters across every tested model.The counters were sampled at 100 ms and produced disjoint phase clusters.
  • A.1 Phase signatures are model-specific but topology-invariant: The phase-separation gap spans two orders of magnitude, so classification thresholds cannot be hard-coded across models.Figure A1 reports a 2.7× to 141× range.
  • A.1 Phase signatures are model-specific but topology-invariant: The same 70B model has statistically indistinguishable signatures when measured colocated versus PD-disaggregated.Its prefill counter ratios are 4.51 versus 4.48, while decode is 0.032 in both settings.
  • A.1 Phase signatures are model-specific but topology-invariant: Phase behavior is therefore treated as a property of the model-quantization pair, while disaggregation makes each lane’s phase static and removes runtime phase detection.The counters also quantify distance from compute saturation and inform reclaimable efficiency.

A.2 Why disaggregated serving is the target

Disaggregation separates prefill and decode regimes so each lane can use its own actuator, avoiding colocated trade-offs and enabling larger, latency-safe efficiency gains.

  • A.2 Why disaggregated serving is the target: Disaggregation removes the colocated zero-sum trade-off by assigning separate mechanisms and setpoints to the statically isolated prefill and decode lanes.The rightmost configuration is best in both efficiency and prefill tail-latency panels.
  • A.2 Why disaggregated serving is the target: +38.8% tokens/J arrives with a 5.8% TTFT p95 improvement and full ITL-p99 SLO compliance in the disaggregated configuration.This result is the strongest reported illustration of simultaneous efficiency and latency improvement.
  • A.2 Why disaggregated serving is the target: A colocated cap-plus-clock-lock gain of +44% tokens/J fell to +4% after redeployment to PD-disaggregated serving, requiring mechanism validation on the target topology.The comparison shows that colocated actuator conclusions do not transfer directly.
  • A.2 Why disaggregated serving is the target: The final lane assignment uses a clock window for compute-bound prefill and a power cap for memory-bound decode because decode clocks exhibit a flat-then-cliff response.The prefill window bounds latency impact by construction, while the decode cap uses self-DVFS adaptivity.
  • A.2 Why disaggregated serving is the target: +30.8% tokens/J with no inter-token-latency cost and full SLO compliance made cap-only control Pareto-dominant on the colocated 480B MoE model.The cap-only arm was the best configuration in that per-configuration sweep.
  • A.2 Why disaggregated serving is the target: +26.0±6.4% tokens/J was recovered on the 480B MoE versus +5.7±8.0% on the 405B dense model, showing architecture-dependent cap headroom.MoE models leave more slack below saturation and combine it with a flat, memory-bound decode profile.
Loading 2609.11133v1…