ePACT: Energy-Performance-Aware Commitment Tracking for LLM Serving

1 Introduction

• With positive costs in both directions, a serving decision's value depends on its effect on the gap between realized and committed energy, not just energy saved.The rates and settlement periods are contract-specific.Source: §1 · pp. 1–2
• Workload uncertainty and two-sided costs require decisions to account for cumulative energy, remaining commitment, backlog, and service quality.The useful direction of adjustment depends on the remaining commitment and workload.Source: §1 · p. 2
• A global planner updates interval targets from measured consumption while a local decision maker predicts SLOs and ranks admitted configurations by asymmetric deviation cost.Coarse-to-fine search reuses promising configurations asynchronously while serving continues.Source: §1 · p. 2
• ePACT formulates serving as two-sided commitment tracking: minimize asymmetric overuse and underuse costs subject to a deadline-miss tolerance, without knowing future arrivals.Energy accounting includes active, draining, and standby replicas.Source: §1 · p. 2
• The evaluation includes single-hour comparisons, controller ablations, and 24-hour trace simulations for H20 and H200 pools.The ablations characterize global feedback and local configuration adaptation.Source: §1 · pp. 2–3

LLM serving under SLOs

• SLO-aware serving work improves scheduling, memory management, or phase separation, while latency and scaled SLOs express request-level service requirements.Examples include vLLM [21], Sarathi-Serve [2], and DistServe [46].Source: §2 · p. 3

Electricity energy commitment

• Carbon-aware systems shift flexible computation in time, while this work minimizes deviation costs for a supplied hourly commitment under service requirements.A power cap limits instantaneous demand; an energy commitment concerns cumulative consumption over a settlement horizon.Source: §2 · p. 3

Energy-aware LLM Inference service

• Energy-aware inference systems adapt instances, parallelism, or GPU frequency to reduce energy under SLOs, whereas ePACT targets deviation from an interval commitment.The related systems include DynamoLLM [36], throttLL’eM, GreenLLM [24], VoltanaLLM, and DualScale [3].Source: §2 · p. 3

Online scheduling and simulation

• Online scheduling and simulation systems advance runtime scheduling and performance estimation, while ePACT uses snapshot-based prediction to admit configurations by deadlines and rank their interval-energy contribution.Examples include FastServe, Llumnix [37], Vidur [1], and LLMCompass.Source: §2 · p. 3

3 System Overview

• The overview couples global interval-target updates with local configuration search and selection for a co-located serving pool.Figure 1 assigns remaining commitment and cumulative-energy tracking to the global planner, and adaptive search, prediction, SLO checking, and application to the local decision maker.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Control Components

• The global planner allocates the remaining hourly commitment across remaining intervals and recomputes the current target from measured consumption at each decision boundary.This lets later decisions account for earlier overuse or underuse rather than follow a fixed allocation.Source: §3.1 · p. 4
• The local decision maker predicts candidate configurations' interval energy and request completion times, then selects the lowest-cost option among those passing the predicted SLO check.Configurations vary active-replica count and GPU clock, using a snapshot of current requests and replica states.Source: §3.1 · p. 4

3.2 Serving and Feedback

• Serving continues under the current configuration during search, and energy accounting includes active, draining, and standby replicas.Arrivals and execution progress update the captured live state; measurements feed the next global target update.Source: §3.2 · p. 4

4.1 Problem Formulation

• The formulation minimizes hourly asymmetric deviation costs subject to request-level service requirements, accounting for energy across active, draining, and standby replicas.The authors do not claim an optimal online policy or guaranteed feasibility under arbitrary arrivals.Source: §4.1 · p. 5

4.2 Global Planner: Dynamic Interval Targets

• The global planner rescales each interval’s nominal allocation using measured consumption and the nominal allocation remaining, so the target reflects the remaining commitment.Targets are recomputed from updated measurements rather than fixed in advance.Source: §4.2 · p. 6

4.3 Local Decision Maker: SLO Checks and Target Matching

• For each candidate, snapshot-based simulation predicts interval energy and request completion times, then checks unfinished requests against their original deadlines.Energy prediction covers one interval, while completion prediction continues until captured requests finish; future arrivals are not used.Source: §4.3 · pp. 6–7
• Among SLO-admitted candidates, the controller minimizes asymmetric target-deviation cost; if none pass, it prioritizes fewer predicted misses.If no candidate has a valid prediction, ePACT retains the current configuration.Source: §4.3 · p. 7

4.4 Adaptive Action Search

• Adaptive search starts from coarse candidates and warm starts, refines around the best configuration, and evaluates remaining configurations if none has a valid prediction.The catalog contains at most 40 configurations and search ends after the best candidate’s one-step neighbors are evaluated.Source: §4.4 · p. 7
• Candidate evaluations run asynchronously with serving, in parallel when proposed together, and each configuration is evaluated at most once per decision.With n_eval configurations and per-candidate simulation cost T_sim, simulation work is O(n_eval T_sim).Source: §4.4 · p. 7

5.1 Experimental Setup

• Physical single-hour runs and GPU-profiled 24-hour simulations used 32-GPU H20 and H200 pools and Azure-trace requests; full-day commitments were frozen and shared across methods.The full-day replay used 969,781 requests.Source: §5.1 · p. 8

5.2 Single-Hour Service-Energy Comparison

• Across both GPUs and commitment settings, ePACT stayed within 0.09 percentage points of vLLM’s attainment and had the smallest absolute deviation among five methods.H200 energy fell from 15.27 to 13.81 MJ while attainment remained above 99.63%. vLLM overused fixed commitments by 17.65% on both GPUs; evaluated DynamoLLM had lower attainment: 71.88%-75.59% on H20 and 76.34%-87.12% on H200.Source: Figure 2 · p. 9, §5.2 · pp. 9–10

5.3 Controller Ablation

• Global feedback reduced accumulated deviation, while local selection preserved higher attainment than Fixed-Config; local selection’s tracking benefit depended on the setting.For H200 at 3× under decreasing commitments, feedback reduced deviation from +9.43% to −0.05%. Under decreasing H20 commitments at 3× and 3.5×, local selection reduced overuse to 18.35% and 20.11%, from 19.36% and 27.32%.Source: §5.3 · p. 10

5.4 24-Hour Commitment Tracking

• 73.76% and 75.74%: ePACT reduced daily deviation cost versus vLLM on H20 and H200, with attainment within 0.03 percentage points of vLLM on both.Attainment was 99.4892% on H20 and 99.9371% on H200.Source: Table 1 · p. 10, §5.4 · p. 10
• 2.1630% on H20 and 2.3092% on H200: ePACT’s mean absolute hourly deviation, versus vLLM’s 11.4304%; 22 and 21 of 24 hours were within ±5%.The within-band counts are for H20 and H200, respectively.Source: §5.4 · p. 10
• H200 exceeded its commitment by 5.80% in hour 7 and 5.26% in hours 21 and 23, despite attainment above 99.9% in each hour.Hourly deviations show residual errors that aggregate daily measures can hide.Source: §5.4 · p. 11

5.5 Prediction Accuracy and Planning Overhead

• Held-out single-replica tests yielded latency WAPE of 0.44%-2.66% and energy WAPE of 0.19%-2.89%; a 97-request trace slice had latency WAPE of 4.15%.In the trace slice, latency relative error’s 95th percentile was 14.74%, 84 of 97 predictions were within ±5%, and signed energy error over 120 seconds was +0.76%.Source: §5.5 · p. 11
• 60 asynchronous decisions averaged 23.35 seconds of search per 60-second interval; serving continued and the controller checked result validity before applying it.The measurement does not establish a search speedup or zero serving overhead.Source: §5.5 · p. 11

6 Conclusion

• ePACT is an LLM serving system for preset hourly energy commitments with positive two-sided deviation costs.Source: §6 · p. 11
• In 24-hour H20 and H200 trace simulations, ePACT reduced asymmetric deviation cost by 73.8% and 75.7% relative to vLLM, with near-vLLM attainment.Source: §6 · p. 11
• Mean absolute hourly deviation was at most 2.31% in those simulations.The conclusion describes the results as reducing modeled deviation exposure while preserving service quality.Source: §6 · p. 11

AI use statement

• Generative AI tools aided and polished the paper’s writing, including language refinement, grammar checking, typo correction, and clarity and readability improvements.Source: AI use statement · p. 12
• The authors state that generative AI tools were not used to generate research ideas, design the methodology, or conduct experiments.Source: AI use statement · p. 12
• The authors reviewed and revised AI-assisted text and take responsibility for the final content.Source: AI use statement · p. 12

• Measured GPU energy updates global targets, while arrivals and execution progress refresh the local decision maker’s live state.Future targets are recomputed after observations; the local controller uses a captured state snapshot.Source: A Additional Implementation Details · p. 16, Figure 3 · p. 16
• Deactivated replicas drain running requests, and reactivated replicas use model weights retained in GPU memory rather than reloading them.Queued requests that have not begun execution may be redistributed when the active-replica count changes; running requests stay on their replicas.Source: A Additional Implementation Details · p. 16
• Search runs against a decision-boundary snapshot while serving continues, and a candidate is applied only if still valid for the live state.The snapshot’s one-interval energy prediction ranks candidates; it is not a record of energy actually consumed after the snapshot.Source: A Additional Implementation Details · p. 16
• The simulator preserves queue, cache, progress-feedback, and pending-step semantics; pending steps finish before a clock transition.Its calibration combines phase-specific duration components, a decode plateau model, and empirical power corrections.Source: A Additional Implementation Details · p. 16
• Figure 4 illustrates that an SLO-failing configuration is rejected, while a farther-from-target candidate can have lower weighted deviation cost than another.The caption identifies A as failing the SLO check and B as having lower weighted deviation cost than C despite being farther from the target.Source: Figure 4 · p. 17

B Scope and Limitations

• The hourly model assumes positive costs for overuse and underuse, so lower physical energy use does not necessarily mean lower cost.The modeled exposure is not universal: contracts differ in settlement periods, tolerance bands, and netting rules, and the score excludes fixed procurement payments.Source: B Scope and Limitations · p. 17
• A predicted SLO pass does not guarantee actual deadlines or target reachability, which can be constrained by standby power, insufficient work, or heavy load.ePACT accelerates useful requests rather than generating work; procurement, storage dispatch, and tariff-band optimization are outside the evaluated controller.Source: B Scope and Limitations · p. 17

B.1 GPU Energy and Facility-Level Commitments

• The energy metric covers GPU-pool device energy, including GPU memory and active, draining, and standby consumption, but excludes CPU, networking, cooling, and facility energy.The 24-hour simulator estimates this same device-level quantity from calibrated power and execution duration, rather than a facility meter.Source: §B.1 · p. 17
• Mapping a facility commitment to a GPU target requires expected PUE and other IT energy; dividing by PUE alone approximates a GPU budget only when other IT use is negligible or separately budgeted.The experiments report GPU-pool results without a PUE multiplier and do not claim measured facility-level tracking; alignment depends on estimates holding for the relevant load and hour.Source: §B.1 · p. 18

C.1 Execution and Configuration

• Single-hour comparisons match workloads, SLO definitions, and commitments across five policies, while 24-hour studies use CPU simulation with GPU-specific profiles.Policy comparisons do not independently establish prediction accuracy; the available single-replica validation is scoped separately.Source: §C.1 · p. 18, C Evaluation Protocol and… · p. 18
• All methods use tensor-parallel degree one; H200 evaluations cap the maximum clock at 1710 MHz after preliminary 1980 MHz runs triggered thermal throttling.The model fits on one GPU, and Fixed-Config also uses 1710 MHz on H200.Source: §C.1 · p. 18

C.2 Commitment Settings and Calibration

• For full-day H20 calibration, the preceding same-weekday workload supplies the historical reference, using ordinary full-capacity vLLM energy for each hour.The same-weekday assumption is similar hourly demand patterns, not identical arrivals.Source: §C.2 · p. 18
• The hourly reduction fraction is guided by the lowest measured GPU-pool energy among evaluated configurations meeting the prescribed SLO-attainment requirement.This is a minimum over evaluated configurations, not a global lower bound or a guarantee for another day’s workload.Source: §C.2 · p. 18
• The H20 evaluation target is frozen before online tracking, and all compared policies receive the same preset hourly target without access to future evaluation-day arrivals.A 20 MJ historical reference with γ_h = 0.8 yields a 16 MJ reservation; workload changes can make exact tracking infeasible.Source: §C.2 · p. 19
• Full-day fractions range from 75% to 100%; H200 reuses the historical fractions but scales them by evaluation-hour vLLM energy from an offline simulation.Single-hour settings use specified fixed or decreasing fractions across SLO scales; H20 decreasing-target deviations for Fixed-Config reflect one execution trajectory rescored across those scales.Source: §C.2 · p. 19

C.3 Measurement and Reporting

• Single-hour attainment includes arrivals’ full completion tails, but energy settlement covers only the first 3,600 seconds; single-run attainment differences do not establish statistical significance.The full-day replay carries requests, deadlines, and serving state across hourly boundaries; SLO outcomes belong to arrival hours, while energy belongs to consumption hours.Source: §C.3 · p. 19
• Hourly deviation cost is a nonmonetary, energy-equivalent score with weights of 2 for overuse and 1 for underuse, rather than physical consumption or a currency charge.The weights encode an experimental asymmetry; actual tariffs may require different weights or settlement functions, and fixed procurement payments are excluded.Source: §C.3 · pp. 19–20
• H20 commitments total 496.1070 MJ, while vLLM, ePACT, and DynamoLLM consume 538.5299, 506.9002, and 453.9654 MJ, respectively; DynamoLLM has a larger hourly deviation score and lower SLO attainment than ePACT.Table 1 sums hourly costs rather than netting daily deviations; opposite deviations cannot cancel.Source: §C.3 · p. 20

C.4 Supplementary Figures and Tables

• Figures 5 and 6 show arrivals, GPU energy, signed commitment deviation, and SLO attainment by arrival hour for the 24-hour H20 and H200 comparisons.Tables 4 and 5 report ePACT hourly results at SLO scale 3 on the May 11, 2024 trace; their target fractions use different vLLM references for H20 and H200.Source: §C.4 · pp. 20, 23, 24, Figure 5 · p. 21, Figure 6 · p. 22

C.5 Prediction Accuracy and Control Overhead

• 0.44%–2.66% class-level end-to-end latency WAPE and 0.19%–2.89% energy WAPE were measured on five held-out workload classes.The largest request-level relative latency error among 18 requests was 3.12%, and the worst evaluated energy cell had 3.28% error; models were frozen before evaluation.Source: §C.5 · p. 25
• 4.15% end-to-end latency WAPE and +0.76% signed energy error were measured over a continuous-arrival 120-second window.The 95th percentile per-request relative latency prediction error was 14.74%, and 84 of 97 predictions were within ±5%; these single-replica checks establish no error bound for every 32-replica configuration or full-day trajectory.Source: §C.5 · p. 25
• 60 asynchronous decisions had a mean search wall time of 23.35 seconds, and serving continued during search without establishing that planning has no latency impact.Batch sizes 1, 2, 3, and 4 occurred in 265,295, 113,992, 38,503, and 1,199 steps, respectively.Source: §C.5 · p. 25