PaperlayerView exact revision on arXiv

Source-linked AI summary

ECO: Energy-Oriented Configuration Optimization for Attention FFN Disaggregated LLM Serving

Zou Qingyun, Bin Gao, Zhuobin Huang, Weng-Fai Wong, Tulika Mitra, Bingsheng He

arXiv:2610.08373v1cs.AR

TL;DR

Energy-efficient LLM serving must meet latency and throughput SLOs, but AFD configurations are coupled through the pipeline and costly to evaluate. ECO combines a structure-aware energy prior with residual learning and cost-aware constrained search. Across 16 scenarios, its frozen configurations reduce energy and raise output token rate relative to Default while meeting SLOs.

  • Problem

    Selecting low-energy AFD configurations that meet SLOs is difficult because deployment constraints and pipeline effects interact while physical measurements are costly and sparse.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    ECO jointly searches admissible AFD structures and controls using a calibrated stage-and-pipeline energy prior, GP residual correction, and cost-aware constrained Bayesian optimization.

    Source: §1 · pp. 2–3, §4.1 · p. 5, §4.2 · p. 6

  • Results

    40.5% lower serving energy and 20.7% higher output token rate on average than Default across 16 held-out scenarios, while meeting SLOs.

    Source: §1 · p. 3, §5.1 · p. 7

  • Takeaways & Limitations

    ECO's selected feasible energy averages 33.1% below BO and 25.8% below GA across the eight A6000 scenarios.

    Source: §5.1 · p. 7

    Both eight-GPU Qwen selections have less than 1% throughput headroom above the SLO floor, and feasibility on the fixed search workload does not establish robustness to workload shifts or higher demand.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Energy-efficient LLM serving requires minimizing serving GPU energy while meeting latency and throughput service-level objectives (SLOs). Attention--FFN disaggregation (AFD) enables separate resource allocation and operating controls for attention and expert computation, but their energy effects remain coupled through the execution pipeline. Realizing its energy-saving potential therefore requires navigating a hierarchical configuration space in which deployment structures constrain admissible controls and shape their end-to-end effects. Finding low-energy configurations that meet SLOs is challenging because physical evaluations are costly and only a small fraction of candidates can be measured. We present Energy-Oriented Configuration Optimization (ECO), which jointly searches deployment structures and their admissible operating controls under a limited measurement budget. ECO constructs a structure-aware energy prior from calibrated stage behavior and pipeline dependencies, then learns residual prediction errors with a Gaussian process. Its cost-aware constrained Bayesian optimization prioritizes measurements according to expected energy improvement while accounting for SLO feasibility, execution success, and evaluation cost, and returns the lowest-energy measured feasible configuration. Across all 16 scenarios on A6000 and A100 with Qwen and DeepSeek, ECO's frozen configurations, evaluated on disjoint requests, reduce serving energy by 40.5\% and increase output token rate by 20.7\% on average relative to baselines while meeting target SLOs. Across the 8 A6000 scenarios, its selected feasible energy averages 33.1\% below generic constrained Bayesian optimization and 25.8\% below genetic search.

1 INTRODUCTION

ECO addresses the challenge of finding low-energy AFD configurations that meet fixed service objectives despite pipeline coupling, hierarchical constraints, model- and load-dependent responses, and costly measurements. It combines structure-aware energy modeling with budgeted search and reports lower energy and higher output-token rate on held-out requests.

Challenge

  • Pipeline delays can extend the time other GPUs consume power, so a local power reduction need not reduce total serving energy.The effect depends on how the selected deployment structure permits overlap and propagates delays.Source: §1 · p. 1, Figure 1 · p. 2
  • Static configuration dependencies restrict valid settings, but static legality alone does not ensure successful execution or SLO feasibility.The appropriate deployment and controls can also differ by model and offered load.Source: §1 · p. 2

Structure-guided optimization

  • ECO combines a calibrated stage-and-pipeline energy prior, Gaussian-process residual correction, and cost-aware constrained Bayesian optimization to select physical measurements.Search returns the lowest-energy measured configuration that passes execution and SLO checks.Source: §1 · pp. 2–3

Empirical evaluation

  • 40.5% lower serving energy and 20.7% higher output token rate on average than Default across 16 held-out scenarios, while meeting fixed SLOs.The frozen configurations were evaluated on disjoint requests, with scenarios weighted equally.Source: §1 · p. 3

Deployment insights

  • ECO examines configuration reuse, asymmetric GPU allocation, and service headroom in the tested scenarios.Its configuration characterization also quantifies search-space coverage under the measurement budget.Source: §1 · p. 3

2 AFD CONFIGURATION LANDSCAPE

The AFD configuration landscape couples pipeline behavior with a hierarchical search space, and selected configurations vary with model and load. These properties motivate using structure and target-workload measurements to guide limited physical evaluations.

Motivation 1: Structure–control coupling

  • Raising the FFN clock target from 2010 to 2100 MHz reverses its energy effect across DBO thresholds (32, 512) and (64, 1024).The matched Qwen measurements at 16 requests/s show increased energy under (32, 512) and decreased energy under (64, 1024); each cell contains one observation.Source: §2 · p. 3

Motivation 2: Hierarchical configuration space

  • Allocation constrains role parallelism, while static checks leave 2,700–22,950 legal A6000 candidates and 16 attempts cover only 0.070–0.593%.Static legality does not establish execution success or SLO feasibility, which still require measurement.Source: §2 · pp. 3–4

Motivation for ECO

  • ECO's workflow calibrates an energy prior, updates predictive models from trial outcomes, selects on a fixed search workload, then freezes the choice for held-out replay.The held-out replay supplies the main results.Source: Figure 3 · p. 4

Motivation 3: Model and load dependence

  • With an eight-GPU budget, higher load raises Qwen's clocks but shifts DeepSeek's allocation; these are best feasible findings under limited search, not global optima.Qwen retains 4A+2F as load rises from 8 to 16 requests/s, while DeepSeek changes from 4A+1F to 5A+1F.Source: §2 · p. 4

Motivation for ECO

  • Pipeline coupling motivates a stage-and-pipeline prior, configuration hierarchy motivates structure-conditioned search, and sparse coverage motivates residual correction as measurements arrive.Calibration is performed on the target workload because model and load affect configuration selection.Source: §2 · p. 4

3 PROBLEM FORMULATION

The formulation searches static deployment structures and their permitted operating controls for the target workload, subject to execution, SLO, and measurement-budget constraints. It returns the lowest-energy measured configuration that passes eligibility checks and freezes it for independent replay.

Configuration space

  • A configuration pairs a deployment structure specifying allocation, tensor parallelism, and DBO settings with per-role frequency and power-cap controls.DP and EP are derived under backend rules; placement and other serving settings remain fixed.Source: §3 · p. 4
  • The search domain contains statically legal structures and only the operating controls permitted by each structure's hardware and constraints.Static legality does not guarantee execution success or SLO feasibility.Source: §3 · p. 4

Energy and feasibility

  • Feasibility requires successful execution, TTFT and TPOT within thresholds, and output token rate at least Rmin; applicable scenarios also constrain p90 TTBT.Thresholds remain fixed throughout search, and unmeasured energy and service metrics must be estimated.Source: §3 · pp. 4–5

Measurement budget

  • Search permits at most K attempts within a GPU-hour budget, charging deployment, warmup, replay, telemetry, cleanup, and failed attempts.The experiments use 16 search attempts plus one separately charged preparation evaluation.Source: §3 · p. 5

Final selection

  • The selected configuration is the lowest-energy measured eligible candidate; if none is feasible, search reports that no feasible configuration was found.The choice is frozen and evaluated on disjoint held-out requests, which do not affect search or selection.Source: §3 · p. 5

4 ECO METHODS

ECO combines structure-specific stage and pipeline energy estimates with Gaussian-process residual correction, then uses cost-aware constrained Bayesian optimization to guide measurements and select a feasible configuration.

4 ECO METHODS

  • ECO calibrates structure-specific stage responses, propagates predicted times through pipeline dependencies, and returns the lowest-energy measured feasible configuration.Calibration uses the search budget, and held-out replay measurements do not update search or selection.Source: §4 · p. 5, §4.1 · p. 5

4.1 PREDICTIVE MODELS

  • Within each deployment structure, ECO fits stage durations from token counts and control probes, then simulates pipeline dependencies and resource contention to estimate capacity and energy.The FIFO recurrence starts an operation after its predecessor completes and its resource becomes available; effective processing rate is limited by arrival rate or pipeline capacity.Source: §4.1 · pp. 5–6
  • A Gaussian process learns residual energy errors, while separate service models predict SLO outcomes and an execution model estimates trial success probability.Failed runs update execution-success estimates but do not provide energy or latency targets.Source: §4.1 · p. 6

4.2 ACQUISITION FUNCTION

  • C2BO weights expected energy improvement by SLO feasibility and execution success per predicted GPU-hour, switching to feasibility per cost until a feasible configuration is found.Candidates predicted to exceed the remaining budget are excluded from the next trial.Source: §4.2 · p. 6

4.3 CANDIDATE FILTERING AND SELECTION

  • C2BO filters statically illegal configurations, interleaves acquisition choices with structure and control probes, and periodically reconsiders deferred candidates before freezing its measured feasible selection.The final configuration is evaluated on independent held-out requests, which do not update search.Source: §4.3 · pp. 6–7

5 EXPERIMENTAL EVALUATION

The evaluation compares frozen configurations against Default on disjoint held-out requests across A6000 and A100 scenarios, with SLO thresholds fixed before search. ECO reduces energy and raises output token rate on average, while ablations and key findings examine search behavior, tuning choices, and selected configurations.

5.1 RESULTS

  • 40.5% lower serving energy and 20.7% higher output token rate on average versus Default across 16 scenarios, while meeting SLOs.The selected configurations were frozen before evaluation on disjoint held-out requests, with scenarios weighted equally.Source: §5.1 · p. 7, §5 · p. 7
  • ECO reaches GA’s 30.93 kJ feasible-energy target after 4 attempts, versus 10 for BO and 9 for GA.The lowest feasible energies were 17.39, 26.50, and 30.93 kJ for ECO, BO, and GA, respectively.Source: §5.1 · p. 8

5.2 ABLATION STUDY

  • At budget 16, Full ECO uses 10.1% less energy than Constant prior + GP, while Simplified pipeline + GP remains 9.2% above Full ECO.Full ECO finds the lowest feasible energy at every budget; the ablation trends suggest the prior guides early search, residual learning corrects persistent error, and pipeline scheduling improves estimates.Source: §5.2 · p. 8
  • 40.2% and 18.1% joint-tuning savings at 4 and 8 requests/s exceed sequential tuning’s 30.7% and 14.6%.The additional energy reduction over sequential tuning narrows from 13.7% to 4.1% at higher load.Source: §5.2 · p. 8
  • At 4 requests/s, cost-aware ECO uses 21.1% and 10.8% less energy at budgets of 2.0 and 2.5 GPU-hours, respectively.It matches No-cost after six attempts; No-cost removes acquisition cost normalization.Source: §5.2 · p. 8

5.3 KEY FINDINGS

  • On four A6000 GPUs, both models retain selected GPU counts and clocks when the tested request rate doubles.Power caps may still require adjustment within the retained structure.Source: §5.3 · p. 8
  • Under the eight-GPU budget, both models select Attention-heavy deployments; higher-load DeepSeek adds Attention capacity while retaining its FFN pool.The selections motivate scaling GPU groups independently, but do not isolate the benefit of any single configuration choice.Source: §5.3 · pp. 8–9
  • Both eight-GPU Qwen selections have less than 1% throughput headroom above the SLO floor.Adding service margin changes the feasible set and can change the selected configuration and energy savings; fixed-workload feasibility does not establish robustness to workload shifts or higher demand.Source: §5.3 · p. 9

6 RELATED WORK

Related work covers coordinated resource provisioning and frequency control, runtime energy management, and automatic configuration search. ECO’s stated focus is measurement-budgeted selection of AFD configurations using hierarchy and pipeline information.

6 RELATED WORK

  • AFD systems study resource partitioning and communication-aware execution (Zhu et al., 2025; Wu et al., 2026), while DynamoLLM coordinates instance count, parallelism, and frequency.POLCA supports cluster power oversubscription, and DualScale coordinates placement and frequency control in prefill/decode disaggregation.Source: §6 · p. 9
  • AFlex combines AFD provisioning, operator profiling, and GPU frequency control (Hu et al., 2026); runtime approaches include iteration-level prediction and separate prefill optimization and decode feedback control.The runtime examples are throttLL’eM (Kakolyris et al., 2024) and GreenLLM (Liu et al., 2025).Source: §6 · p. 9
  • CherryPick applies BO to cloud configuration selection, while SCOOT combines BO, constraint handling, and parallel suggestions for LLM inference (Cheng et al., 2025).ECO uses constrained BO for unknown feasibility (Gelbart et al., 2014) and cost-aware BO for variable evaluation duration (Snoek et al., 2012), with an AFD-specific energy prior.Source: §6 · p. 9

7 CONCLUSION

ECO jointly searches AFD deployment structures and operating controls under a limited measurement budget, combining a calibrated stage-and-pipeline energy prior with residual learning and cost-aware constrained Bayesian optimization. Its frozen configurations improve energy and output token rate over Default across the tested scenarios while meeting SLOs.

7 CONCLUSION

  • ECO combines a calibrated stage-and-pipeline energy prior, residual learning, and cost-aware constrained Bayesian optimization to jointly select AFD structures and operating controls under a limited measurement budget.The framework returns measured static configurations for the target workload.Source: §7 · p. 9
  • Across 16 A6000 and A100 scenarios, ECO’s frozen configurations reduce serving energy by 40.5% and increase output token rate by 20.7% on average relative to Default while meeting SLOs.Evaluation used disjoint requests.Source: §7 · p. 9
  • Future work should evaluate separate adaptation schedules end-to-end and quantify their mechanisms through controlled attribution and stage-waiting measurements.Source: §7 · p. 9

ETHICS STATEMENT

The study evaluates serving efficiency by considering energy consumption and service quality together. Its experiments use real text prompts from replay trace files.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The appendix documents the fixed replay configurations, workload construction, and client-side service metrics used to evaluate the configurations. Workloads use trace-derived requests, and SLO thresholds remain fixed during search and held-out replay.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement directs readers to the experimental setup and appendices for the reported hardware, software, workload, measurement, and search settings.Static configurations fix deployment structure and operating controls during each replay; A and F denote attention and FFN, and E denotes expert GPUs.Source: REPRODUCIBILITY STATEMENT · p. 10, A APPENDIX · p. 13

A.1 WORKLOADS AND DEPLOYMENT SETTINGS

  • The 16 scenarios combine two models, two GPU platforms, and four GPU-budget/load pairs, with GPU allocation bounded by the available budget.Replay requests and token lengths come from the Splitwise Azure trace; each run freezes its selected configuration before a separate held-out replay with matched arrival schedules.Source: §A.1 · p. 13, Table 2 · p. 13
  • Operating controls are specified per GPU and role, while parallelism and DBO settings must satisfy model, backend, and control-grid constraints.DBO splits batches only when its decode or prefill threshold is reached; enabling it does not mean every batch is split.Source: §A.1 · p. 13

A.2 LATENCY, THROUGHPUT, AND SLOS

  • Client-side TTFT is first-token arrival minus submission, and TPOT is post-first-token response duration divided by completed output tokens beyond the first.The section computes p90 latency values across requests, with TPOT calculated per request before taking the percentile.Source: §A.2 · pp. 13–14
  • The service metrics include p90 token-arrival intervals for TTBT and aggregate output token rate over the interval from first submission to last completion.Fixed thresholds are set from historical Default measurements; all applicable latency and throughput thresholds, including TTBT where required, apply throughout search and held-out replay.Source: §A.2 · p. 14

A.3 ENERGY ACCOUNTING AND RESULT NORMALIZATION

The appendix specifies how serving energy and held-out changes are measured, then details ECO’s eligibility rules, surrogate updates, and measurement allocation alongside the BO and GA comparison procedures.

A.3 ENERGY ACCOUNTING AND RESULT NORMALIZATION

  • Serving energy integrates measured GPU power over the serving interval, including idle and waiting periods on allocated GPUs but excluding GPUs outside that allocation.The serving window runs from first request submission to last response completion; loading and warmup precede it.Source: §A.3 · p. 14
  • Held-out percentage changes compare each frozen selection with a Default replay on the same held-out requests, while historical Default measurements set the fixed SLO thresholds.Comparisons are averaged across independent search seeds and then equally across 16 scenarios.Source: §A.3 · p. 14
  • 20.7% is the mean output-token-rate increase across 16 equally weighted scenarios, with each four- and eight-GPU scenario contributing once.Scenario-level percentage changes are averaged over independent search seeds before the across-scenario summary.Source: §A.3 · p. 14

B.1 MEASUREMENT OUTCOMES AND CONFIGURATION ELIGIBILITY

  • A configuration is eligible only if every attempt succeeds and its averaged service metrics meet all applicable SLOs; ECO returns the eligible configuration with lowest mean energy.Failed launches, incomplete responses, and invalid measurements consume search attempts but cannot qualify as low-energy results.Source: §A.3 · p. 14, §B.1 · p. 15, Table 3 · p. 15

B.2 SURROGATE IMPLEMENTATION DETAILS

  • ECO fits structure-specific stage-time models from calibration records, applies local control responses, and propagates stage times through the pipeline before residual-GP correction.Routing imbalance adjusts FFN time, transferred bytes scale dispatch and combine times, and unidentified effects or extrapolation increase prior uncertainty.Source: §B.2 · p. 15

B.3 EXPECTED IMPROVEMENT IN ENERGY UNITS

  • The energy GP’s log-normal posterior yields expected improvement in energy units, which ECO weights by predicted SLO and execution probabilities and divides by predicted GPU-hours.Before a feasible incumbent exists, selection uses feasibility per unit cost; successful SLO-violating runs update residual targets, but execution failures do not.Source: §B.3 · p. 16

B.4 MEASUREMENT ALLOCATION AND SCREENING DEFAULTS

  • ECO interleaves acquisition choices with new-structure and single-control probes, while repeats usually wait until the final quarter of the attempt allowance.A structure probe supplies calibration records at one operating point; a single-control probe changes one role’s frequency or power cap relative to a measured reference.Source: §B.4 · p. 16

B.5 BO AND GA BASELINE CONFIGURATIONS

  • The BO baseline uses independent GPs and energy-unit expected improvement weighted by SLO and execution probabilities and divided by predicted GPU-hours.BO models trial-cost corrections; GA instead uses base cost estimates and a steady-state population with lexicographic fitness, repairing offspring to untried affordable legal candidates.Source: §B.5 · pp. 17–18

C.1 STATISTICS FOR THE SEARCH TRAJECTORIES

Search-trajectory statistics record measured attempts, feasible observations, and selected energies, while selected A6000 settings show how ECO’s chosen deployment and controls vary across scenarios.

C.1 STATISTICS FOR THE SEARCH TRAJECTORIES

  • ECO first reaches the DeepSeek four-A6000, 4-request/s target energy at attempt 4, versus attempt 10 for BO and attempt 9 for GA.The target is GA’s best feasible energy of 30.9306 kJ; attempt counts include failures and do not estimate elapsed search time.Source: §C.1 · p. 18, Table 5 · p. 18
  • For Qwen, ECO selects lower energy with 10 feasible attempts than BO’s 15, while an even lower-energy ECO attempt is excluded for violating TTFT.DeepSeek GA’s attempts 2, 4, and 11 fail execution and count toward its 16-attempt budget.Source: §C.1 · p. 18, Table 5 · p. 18

C.2 SELECTED A6000 DEPLOYMENT SETTINGS

  • For Qwen under an eight-GPU budget, both loads select 4A+2E; increasing load raises attention frequency from 1560 to 2100 MHz and expert frequency from 1560 to 1920 MHz.The listed configurations use TPA = TPF = 1 and enable DBO; the joint selection does not isolate the energy effect of any one choice.Source: §C.2 · p. 19, Table 6 · p. 19

D CONFIGURATION CHARACTERIZATION DETAILS

The characterization audits show that only a small fraction of legal configurations are measured, and that observed energy interactions can reverse frequency preferences. Separate microbenchmarks identify sampled operator energy minima, but these are not universal optima for end-to-end serving.

D CONFIGURATION CHARACTERIZATION DETAILS

  • Sixteen search attempts covered 0.593%, 0.103%, and 0.070% of the audited legal candidate spaces, which contained 2,700, 15,525, and 22,950 candidates, respectively.The candidate spaces correspond to Qwen on four A6000 GPUs, Qwen on eight GPUs, and DeepSeek on eight GPUs.Source: D CONFIGURATION CHARACTERIZATION DETAILS · p. 19
  • Static checks reduced Qwen’s four-GPU A6000 enumeration from 6,075 candidates to 2,700 legally retained candidates.The eight-GPU enumeration had 31,725 candidates per model before static legality checks.Source: D CONFIGURATION CHARACTERIZATION DETAILS · p. 19
  • Of completed attempts, feasible counts were 9/14 for four-GPU Qwen, 4/16 and 2/15 for eight-GPU Qwen, and 13/16 and 12/16 for eight-GPU DeepSeek.Unmeasured candidates’ execution and SLO outcomes remain unknown.Source: D CONFIGURATION CHARACTERIZATION DETAILS · p. 19
  • One fully observed quartet showed an interaction contrast of −1302.58 J (−2.414% of E(a0, b0)) and a reversal in FFN-frequency preference across DBO settings.Three cells failed the throughput SLO; the single-observation quartet supports no confidence interval or general interaction-strength ranking.Source: D CONFIGURATION CHARACTERIZATION DETAILS · pp. 19–20
  • The quartet records end-to-end metrics but no stage-timing or waiting measurements, so it cannot locate a migrating bottleneck.The quartet varied only DBO thresholds and the FFN clock target.Source: D CONFIGURATION CHARACTERIZATION DETAILS · p. 20

E MOTIVATION MICROBENCHMARK DETAILS

  • At a fixed 1410 MHz, TP1×EP4, TP2×EP2, and TP4×EP1 averaged 0.1976, 0.3735, and 0.3327 J per iteration, respectively.The microbenchmark used four GPUs, 6144 global tokens, one microbatch, and five repetitions per configuration.Source: E MOTIVATION MICROBENCHMARK DETAILS · p. 20
  • In separate batch-one, 128-token prefill sweeps from 510–1410 MHz, sampled minima were 690 MHz for attention and 870 MHz for FFN, not universal optima.End-to-end selection also evaluates communication, waiting, and service constraints.Source: E MOTIVATION MICROBENCHMARK DETAILS · pp. 20–21

Related papers

Loading 2610.08373v1…