PaperlayerView exact revision on arXiv

Source-linked AI summary

EcoServe: Designing Carbon-Aware AI Inference Systems

Yueying Li, Zhanqiu Hu, Esha Choukse, Rodrigo Fonseca, G. Edward Suh, Udit Gupta

arXiv:2502.05043v2cs.DC

TL;DR

Existing LLM inference approaches prioritize operational metrics while overlooking embodied carbon and workload and hardware heterogeneity. EcoServe co-designs provisioning, resource allocation, and scheduling around four carbon-focused principles, with reported savings of up to 2× total carbon across offline and online serving.

  • Problem

    Existing approaches optimize operational metrics while overlooking embodied emissions and fail to jointly leverage workload and hardware heterogeneity across provisioning, allocation, and scheduling.

    Source: §1 · p. 1

  • Method

    EcoServe co-designs capacity planning, resource allocation, and runtime scheduling using Reuse, Rightsize, Reduce, and Recycle principles.

    Source: §1 · p. 1

  • Results

    Up to 2× total carbon savings across offline and online serving, with individual strategies saving 29–41% versus performance-optimized baselines while maintaining SLOs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Hardware-software co-design across software provision, scheduling, and hardware provision can reduce total carbon while maintaining performance.

    Source: §8 · p. 13

Abstract

from arXiv · show

The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources but also significant energy, yielding large operational and embodied carbon emissions. In this work, we present three main observations based on modeling and traces from the production deployment of two Generative AI services in a major cloud service provider. First, while GPUs dominate operational carbon, host processing systems (e.g., CPUs, memory, storage) dominate embodied carbon. Second, offline, batch inference accounts for a significant portion (up to 55\%) of serving capacity. Third, there are different levels of heterogeneity across hardware and workloads for LLM inference. Based on these observations, we design EcoServe, a carbon-aware resource provision and scheduling framework for LLM serving systems. It is based on four principles - Reduce, Reuse, Rightsize, and Recycle (4R). With a cross-stack ILP formulation and design, we demonstrate that EcoServe can lower carbon emissions by up to 47\%, compared to performance, energy, and cost-optimized design points, while maintaining performance targets and SLOs.

1 INTRODUCTION

LLM growth increases infrastructure energy demands, while prevailing inference systems focus on operational efficiency and leave embodied emissions and workload-hardware heterogeneity under-addressed. EcoServe co-designs provisioning and scheduling around four carbon-reduction strategies, reporting lower total carbon while maintaining performance targets.

1 INTRODUCTION

  • LLM scale growth has sharply increased infrastructure demands, with model size rising over 100× from GPT-2 to PaLM between 2019 and 2022.The passage also cites a projection that AI-driven data centers could account for up to 40% of power in local grids such as Virginia and Oregon.Source: §1 · p. 1
  • Host systems contribute disproportionately to embodied carbon, whereas GPUs dominate operational power consumption.The stated host-system embodied overheads include memory capacity, storage, and motherboard components.Source: §1 · p. 1
  • Existing approaches do not leverage all workload and hardware heterogeneity dimensions or co-optimize capacity planning, resource allocation, and runtime scheduling.Relevant dimensions include model size, prompt length, decoding patterns, workload phases, and SLO requirements.Source: §1 · p. 1
  • EcoServe co-designs capacity planning, resource allocation, and runtime scheduling so long-term provisioning can be adapted to workload variation.Capacity planning targets embodied carbon, while runtime scheduling adapts provisions without increasing embodied or operational overhead.Source: §1 · p. 1
  • EcoServe’s 4R strategies are Reuse, Rightsize, Reduce, and Recycle, addressing resource use across system operation.Reuse exploits underutilized host CPUs for offline inference; Rightsize adjusts CPU and GPU provisioning; Reduce removes unnecessary host memory and storage; Recycle extends host-system lifetime while selectively upgrading accelerators.Source: §1 · pp. 1–2
  • 1.4–2.2× reduction in total carbon emissions is reported across diverse LLM deployment scenarios while maintaining performance targets.The reported evaluation includes open-source datasets and production traces; individual strategies save 29–41% versus performance-optimized baselines.Source: §1 · p. 2

2 BACKGROUND

The background distinguishes operational emissions from hardware-embodied emissions and reviews inference systems’ resource-management techniques. It motivates carbon modeling and optimization that account for more than energy or cost alone.

2 BACKGROUND

  • Operational carbon comes from energy consumed during operation, while embodied carbon includes hardware manufacturing, transportation, and disposal emissions.Memory and storage are identified as carbon-intensive in manufacturing, and embodied carbon can exceed 50% of typical data-center hardware lifetime emissions.Source: §2 · p. 2
  • EcoServe extends prior carbon models with finer-grained GPU-versus-host footprints and embodied-carbon estimates for memory, storage, and power-related components.Its DRAM comparison plots bit density and embodied-carbon footprint across technologies and three manufacturers.Source: §2 · p. 2, Figure 2 · p. 3, Figure 3 · p. 3
  • LLM serving spans compute-bound prompt processing and memory-bound decoding, with online low-latency and offline high-throughput SLOs.Large models and KV-cache storage also make workloads memory-capacity-bounded.Source: §2 · p. 2
  • Inference systems combine parallelization, memory management, and scheduling techniques to improve utilization and meet latency constraints.Examples include data, tensor, and pipeline parallelism; activation checkpointing, offloading, and paged attention; and dynamic batching and chunked prefill.Source: §2 · p. 2
  • Carbon optimization differs from energy or cost optimization because it also depends on grid carbon intensity, lifetime hardware utilization, provisioning, and infrastructure impacts.The passage notes that recent inference work has considered energy and cost, but these objectives do not directly optimize carbon impact.Source: §2 · p. 2

3 UNDERSTANDING AI SYSTEMS’ CARBON FOOTPRINT

The section models operational and embodied carbon in AI inference systems, then characterizes how GPU, host-system, and energy-source differences shape their total footprint.

3 UNDERSTANDING AI SYSTEMS’ CARBON FOOTPRINT

  • The accounting combines operational emissions from host and GPU power over runtime with embodied carbon, using measured power and grid carbon intensity.Host power is measured using Intel RAPL; GPU power using NVML, while embodied-carbon estimates draw on product reports, life-cycle analyses, and modeling tools.Source: §3 · p. 3

3.1 Modeling Embodied Carbon of AI Systems

  • The embodied-carbon model accounts for host and GPU components, including processors, memory, storage, boards, power delivery, and cooling.Application-processor estimates use process technology and die area; product environmental reports supply estimates for components lacking specific technology information.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1 Integrated circuits (Application Processors).

  • Memory estimates scale wafer-level DRAM data by technology bit density; newer nodes have lower embodied carbon per GB, while SSDs are estimated at 0.110 kg CO2e per GB.The work estimates DDR4/LPDDR5, GDDR6, HBM2, and HBM3e at 0.29, 0.36, 0.28, and 0.24 kg CO2e per GB, respectively.Source: §3.1.2 · p. 3

3.1.2 Memory and Storage.

  • The PWB estimate scales with board area: Total PCB CF = 0.048 × AreaPCB.The reference Dell R740 PWB area is 1925 square centimeters, corresponding to 176 kg CO2.Source: §3.1.3 · p. 4

3.2 Characterizing Total Carbon

  • Embodied carbon rises with GPU compute and memory bandwidth, alongside power capacity, as GPU generations improve in performance.An NVIDIA L4 has 3× lower embodied carbon than an NVIDIA H100.Source: Figure 4 · p. 3, §3.2 · p. 4
  • Host systems contribute heavily to end-to-end embodied carbon, largely through mainboards, memory, and storage, despite being underutilized relative to GPUs.The figure caption reports that host-processing systems account for over half of embodied carbon.Source: §3.2 · p. 4, Figure 5 · p. 4
  • As power sources become cleaner, operational carbon falls and embodied carbon becomes more significant; GPU power dominates operational carbon while hosts dominate embodied carbon.In high-carbon regions, GPU operational carbon dominates.Source: §3.2 · p. 4, Figure 6 · p. 4
  • In profiled vLLM and SGLang workloads, CPU utilization averaged around 6% of a 16-core processor, though multimodal, encoder-based, or agentic systems may use CPUs more.The measurements covered several models and request rates from 2–32 requests per second using ShareGPT requests.Source: §3.2 · p. 4

4 ECOSERVE DESIGN

EcoServe combines carbon-aware provisioning and runtime scheduling under the 4R principles, coordinating reuse, rightsizing, reduction, and recycling across inference workloads.

4.1 Design Principles

  • 3.67× speedup over baseline Llama.cpp comes from layered KV-cache transfers to CPUs for decoding, while prompt computation remains GPU-favorable.Low-arithmetic-intensity operations such as attention scoring and KV-cache decoding are suited to CPU execution.Source: §4.1.1 · p. 5
  • 1.32× less offline GPU provisioning at peak with reallocation every 4 hours; larger CPU batches allow up to 45% offline capacity reduction.Offline demand reaches 21% and 45% of capacity on average for Services A and B, and 27% and 55% at peak, respectively.Source: Figure 11 · p. 6, §4.1.1 · p. 6
  • For Gemma-27B, A100 is preferable for smaller prompt inputs and batches and for decode, while H100 becomes more efficient as prompt inputs and batches grow.EcoServe profiles workload slices and SLOs to match workloads to heterogeneous GPUs rather than provisioning only H100 for prompting.Source: §4.1.2 · p. 6
  • Memory and storage comprise 36% of embodied emissions in Azure A100 offerings; EcoServe sizes DRAM around inference needs and SSDs near GPU-memory capacity.Remote storage with GPUDirect is proposed as a way to reduce over-provisioned storage.Source: §4.1.3 · p. 7
  • Host components can be retained longer: DRAM and SSD bandwidth are rarely inference bottlenecks, and CPU aging is 0.8 years after five years at 20% utilization.Under the same utilization assumption, SSD age increases by one year over five years.Source: §4.1.4 · pp. 7–8

4.2 Putting it all together

  • EcoServe provisions server configurations for different carbon-performance trade-offs, then uses a runtime load balancer to place workloads and adjust resource allocation.The framework maintains online, mixed, and offline pools; pool sizes adjust through periodically triggered ILP based on demand and carbon intensity.Source: §4.2 · p. 8, §4.2.1 · p. 8

4.2.1 Offline profiling and online adaptation.

  • Workload slicing divides prompt and decode request rates into fine-grained slices, enabling allocation based on each slice’s expected latency and resource needs.Requests are grouped into histogram buckets and divided into slices according to a configurable slice factor.Source: §4.2.2 · p. 8

4.2.2 ILP for co-design scheduling and allocation.

  • The ILP jointly assigns workload slices to GPUs and allocates CPU cores and memory, minimizing weighted carbon and hardware cost subject to capacity and SLO constraints.The SLO constraints cover time to first token and time per output token; the weighting variable balances carbon and cost.Source: §4.2.2 · p. 8

5 IMPLEMENTATION AND EXPERIMENTAL SETUP

The evaluation uses heterogeneous CPU/GPU hardware, several models and workload traces, and a software stack spanning inference engines, simulators, profiling models, and ILP allocation.

Hardware Systems

  • The testbed combines PCIe NVIDIA H100, A100, A6000, L4, and A40 GPUs with a dual-socket Intel Sapphire Rapids CPU.The CPU frequency governor is set to performance mode with turbo boost disabled.Source: §5 · p. 9

Models, Workloads and Load Generator

  • Evaluation workloads use scaled Azure Functions and production offline traces, alongside sampled public datasets and models ranging from Gemma-2-2B to Bloom-176B.The model/workload set includes ShareGPT, AFT, and LongBench examples for online and offline settings.Source: §5 · p. 9

Software

  • Deployments use vLLM and Llama.cpp; profiling models, the Splitwise simulator, and CVXpy-based ILP allocation support comparisons across hardware, workloads, and carbon intensities.Carbon-intensity inputs are sampled from WattTime and GreenSKU.Source: §5 · p. 9

6 EVALUATING ECOSERVE

EcoServe achieves up to 47% carbon savings across online and offline workloads compared with performance-optimized configurations, with performance within 3% except for reuse. The savings vary by workload: online gains come from reducing resources and rightsizing, while offline gains also rely on CPU reuse.

6.1 End-to-end Evaluation

  • 47% average carbon savings are reported for EcoServe, with comparable performance against performance-, energy-, and cost-optimized designs.The reported performance is close to perf-opt within 3% except for reuse; Figure 15 assumes iso-throughput.Source: §6.1 · p. 9, §6.1.1 · p. 9, Figure 15 · p. 10

6.1.1 Performance Analysis.

  • Reuse is the performance exception: its TPOT is around 5% worse because longer-context offline decoding is offloaded to CPUs.The passage notes that improved interconnects or batched KV-cache transfers could recover this drop.Source: §6.1.1 · p. 9

6.1.2 Breakdown Analysis.

  • For online workloads, Reduce saves approximately 17–40% carbon by reducing DRAM and SSD capacity, while rightsizing further limits high-end hardware overprovisioning.Online workloads seldom use CPU resources, so adding Reuse maintains rather than increases savings.Source: §6.1.2 · p. 10
  • Offline savings rely more on CPU reuse and rightsizing than on Reduce, whose conservative DRAM sizing must match the model size.Longer offline workloads yield greater benefits.Source: §6.1.2 · p. 10
  • Mixed production workload A benefits less from aggressive online Reduce because it has less offline demand, yet overall carbon savings remain similar across the two mixes.The authors describe this as evidence of generality across the analyzed workload types.Source: §6.1.2 · p. 10

6.2 Sensitivity and Scalability Study

EcoServe’s carbon savings vary with carbon intensity, request rate, and workload, while its control-plane overhead scales sub-linearly with cluster size. The sensitivity evaluation reports carbon reductions against SplitWise and describes how EcoServe allocates resources under different conditions.

6.2.1 Varying Carbon Intensity.

  • 26.5% average carbon savings on the evaluated trace followed EcoServe’s software provisioning choice to allocate shorter prompts to A100s.The experiment used low, medium, and high carbon-intensity grids of 17, 261, and 501 gCO2/kWh, respectively.Source: §6.2.1 · p. 10
  • 38% embodied and 36% operational carbon reductions are reported for large and medium models under lower load while matching throughput with 30 H100s and 17 A100s.In the stated 40 H100 power-equivalent setup, SplitWise used 35 Prompt and 8 Token machines; EcoServe adaptively placed longer requests on A100s.Source: §6.2.1 · p. 10
  • Reuse is most often selected for longer requests at lower carbon intensity, whereas rightsizing and Reduce save more for online workloads at higher carbon intensity.Offline carbon savings are most significant in the observed reuse configurations.Source: §6.2.1 · p. 10
  • EcoServe consistently outperforms SplitWise across the tested carbon intensities and loads, with a larger gap at lower request rates and higher carbon intensity.Figure 17 compares Bloom-176B and Llama-70B on iso-power deployments.Source: Figure 17 · p. 11

6.2.2 Scalability of the control plane.

  • 2.1–2.7× control-plane overhead growth from 10 to 160 nodes remained sub-2 seconds, reaching 1.315 seconds for offline planning and 0.585 seconds for online decisions.The table reports overhead in seconds across cluster sizes, workloads, and system loads; the authors attribute favorable scaling to clustering similar workloads and hardware in the ILP formulation.Source: Table 3 · p. 10, §6.2.2 · p. 11

6.3 Reuse Strategy Evaluation

EcoServe’s reuse evaluation combines CPU kernel optimization with workload- and load-aware CPU reuse for decoding. It reports throughput and embodied-carbon benefits, alongside conditions where CPU reuse can add operational carbon or perform poorly.

6.3 Reuse Strategy Evaluation

  • 1.34× average CPU decoding performance improvement over llama.cpp is reported across batch sizes and parallelism dimensions on Intel SPR.The figure reports up to 4.03× improvement; the evaluation varies core counts and workload sizes.Source: Figure 18 · p. 11, §6.3 · p. 11
  • 3.51× average embodied-carbon savings versus a non-optimized baseline are reported across the evaluated workloads.EcoServe CPU decoding achieves up to 66% relative embodied-carbon savings versus naive llama.cpp; naive CPU reuse can have worse embodied carbon than GPU non-reuse for Gemma-27B and short-context Llama-8B cases.Source: §6.3 · p. 11, Figure 19 · p. 11
  • On a 56-core SPR CPU attached to A100, CPU reuse achieves 0.53–2.29× GPU throughput, although CPU energy efficiency is generally worse, especially for larger models when both processors are compute-bound.Across the reported workloads, EcoServe achieves 3.51× average embodied-carbon savings versus a non-optimized baseline; longer execution can still add operational-carbon cost.Source: §6.3 · p. 11
  • Reuse may not work well for operational carbon in short-context, larger-model decoding, while low-carbon-intensity regions allow more CPU offloading than high-intensity regions.The strategy’s total carbon benefit reflects the net effect of embodied and operational carbon.Source: §6.3 · p. 11

6.4 Right-sizing Strategy Evaluation

EcoServe’s right-sizing strategy allocates resources by workload phase and demand under SLO constraints. It outperforms Melange and single-hardware baselines on carbon and energy in the evaluated online and offline settings.

6.4 Right-sizing Strategy Evaluation

  • The evaluation compares right-sizing with Melange’s cost-optimal GPU allocation and a single-hardware baseline that disregards SLO slack and hardware energy/carbon implications.The workload traces were randomly sampled from open-source traces; each request-generator run lasted five minutes, with the first and last 5% discarded for warm-up.Source: §6.4 · pp. 11–12
  • Over 2.56× better carbon and 2.25× better energy than Melange are reported for online workloads at lower request rates.EcoServe separates allocation by LLM phase, selecting hardware for each workload slice, while Melange uses perf-per-cost hardware and can overprovision decode resources.Source: §6.4 · p. 12, Figure 20 · p. 12
  • EcoServe also performs better on carbon and energy than the tested single-hardware choices, with L4 the closest baseline.The passage attributes the flexibility to allocating hardware combinations based on workload slices and SLOs.Source: §6.4 · p. 12
  • Up to 1.6× improvement in both energy and carbon over the next-best baseline is reported for offline workloads.Under the stated US-Central/South carbon intensity, operational carbon is around 0.64× embodied carbon.Source: §6.4 · p. 12

6.5 Recycle Strategy Evaluation

EcoServe’s asymmetric CPU/GPU upgrade schedule balances embodied and operational carbon savings, achieving approximately 16% cumulative carbon savings over 10 years versus fixed four-year upgrades.

6.5 Recycle Strategy Evaluation

  • EcoServe upgrades hosts every nine years and GPUs every three years, compared with baseline four-year upgrades for both.The strategy reflects that GPU energy efficiency is estimated to double every 3.5 years on average.Source: Figure 21 · p. 12
  • Approximately 16% cumulative carbon savings over 10 years versus the baseline fixed strategy come from EcoServe’s asymmetric upgrade strategy.The comparison is fixed four-year upgrades for both host and GPU versus host upgrades every nine years and GPU upgrades every three years.Source: §6.5 · p. 12, Figure 21 · p. 12
  • Less frequent host upgrades save embodied carbon, while more frequent GPU upgrades enable earlier adoption of energy-efficient hardware and operational carbon savings.The strategy can incur negative operational carbon savings for hosts, which have less impact on overall operational emissions to AI inference than GPUs.Source: §6.5 · p. 12

7 RELATED WORK

Prior work addresses CPU inference, heterogeneous hardware, offloading, and serving optimization, but the cited systems leave gaps in carbon-aware resource provisioning and scheduling. EcoServe focuses on co-designing provisioning and request scheduling with carbon as an objective.

7 RELATED WORK

  • CPU inference work uses techniques such as quantization and memory-mapped weights, while this work builds on llama.cpp and adds co-design of provisioning and scheduling.The cited CPU-efficiency efforts also include sparsity-aware techniques, quantization optimizations, offloading, and library enhancements.Source: §7 · p. 12
  • Mobius addresses heterogeneous interconnect bandwidth through pipeline parallelism for training, whereas this work focuses on request scheduling, resource provisioning, and carbon.The passage distinguishes prior work on model partition and placement from EcoServe’s objectives.Source: §7 · pp. 12–13
  • Prior offloading systems target particular workloads or online settings, but the cited proposals do not consider resource provisioning and carbon impact together.The passage cautions that naive offloading without phase, request-length, and capacity planning can increase embodied and operational carbon through GPU overprovisioning and PCIe overhead.Source: §7 · p. 13
  • General serving systems often overlook LLM serving phases and autoregressive behavior, while cited systems are not carbon-aware and do not consider CPU’s role.The paper reports that carbon-optimized and energy-optimized designs differ, motivating joint resource provisioning and runtime scheduling.Source: §7 · p. 13

8 CONCLUSION

EcoServe presents a quantitative framework for reducing AI infrastructure carbon through software and hardware co-design while maintaining performance. It offers a foundation for carbon-aware system design in sustainable AI infrastructure.

8 CONCLUSION

  • EcoServe holistically optimizes AI infrastructure’s carbon footprint through co-design of software provisioning, scheduling, and hardware provisioning while maintaining performance.The authors describe the framework as a foundation intended to inspire future research in sustainable AI infrastructure.Source: §8 · p. 13

Related papers

Loading 2502.05043v2…