PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane

Twinkll Sisodia

arXiv:2609.05565v1cs.DCcs.LG

TL;DR

Production LLM sustainability depends on serving conditions as well as model choice, and existing systems research studies these factors separately. This paper synthesizes those results into an llm-d control-plane proposal and evaluation plan; its central proposal remains a testable engineering hypothesis, not a measured improvement.

  • Problem

    Production inference energy and carbon depend on workload, cache, phase, hardware, and deployment conditions, while relevant systems results are often discussed separately.

    Source: §1 · pp. 1–2

  • Method

    The paper synthesizes recurring patterns in prior work into an SICP design sketch for llm-d and a proposed evaluation plan.

    Source: §1 · p. 2, §12 · p. 11

  • Results

    The paper proposes treating an inference request path as an optimization target while holding quality and SLOs constant, but reports no measured SICP improvement.

    Source: §12 · p. 11

  • Takeaways & Limitations

    The synthesis treats sustainability as an orchestration problem across workload, phase, cache, hardware, and placement, with energy, carbon, cache-recompute cost, and power headroom considered alongside SLO signals.

    Source: §12 · p. 11

    The paper does not remeasure cited results or establish that their gains transfer unchanged to llm-d; carbon accounting also depends on location, time, PUE, and grid-emissions assumptions.

    Source: §11 · p. 11

Abstract

from arXiv · show

Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Reported performance, energy, carbon, and cost improvements remain the results of the cited papers and systems. I group the literature into recurring design patterns and use those patterns to sketch a Sustainable Inference Control Plane (SICP) for llm-d. The proposed control plane would consider latency, energy, carbon, cache reuse, serving cost, and quality when routing and scaling, while keeping TTFT/TPOT SLOs as hard constraints. I also outline an evaluation framework based on SLO-satisfied goodput per joule and per gram CO2e, together with a reproducible experimental plan. The main observation from connecting the literature is that sustainable LLM inference is unlikely to come from one "green" model or one accelerator; it is more naturally treated as a control problem across model, phase, cache, hardware, replica, region, and time.

1 Introduction

The paper frames LLM sustainability as a production-serving problem and asks how existing research can inform an llm-d control plane. It synthesizes prior work into a taxonomy, recurring patterns, an SICP design sketch, and an evaluation plan, while distinguishing proposals from reported results.

1 Introduction

  • Production inference energy depends on workload shape, software stack, accelerator, batching, decoding, and parallelism, so model size alone cannot characterize its impact [Fernandez et al., 2025].The paper also notes that serving systems have reduced energy, power, carbon, and cost while meeting latency SLOs through dynamic reconfiguration [Stojkovic et al., 2025b, Li et al., 2025b, Stojkovic et al., 2025a].Source: §1 · p. 1
  • llm-d provides a potential sustainability insertion point because its control plane already coordinates cache-aware routing, phase placement, autoscaling, and heterogeneous engines [llm-d Maintainers, 2026a, Mitra et al., 2026, Guy et al., 2026, Liu and Kumar, 2026, llm-d Maintainers, 2026b].The proposed decision extends endpoint selection toward feasible choices that satisfy SLOs while considering incremental energy, carbon, power pressure, and recomputation.Source: §1 · p. 2
  • The paper contributes a cross-layer taxonomy, recurring engineering patterns, an llm-d SICP design sketch, and a measurement and evaluation plan.The taxonomy spans request through lifecycle layers; the design connects patterns to routing, KV placement, power management, model selection, autoscaling, and regional placement.Source: §1 · p. 2

2 Scope and Evidence Selection

  • The review selects directly relevant serving and sustainability work from 2023 through September 3, 2026, using training-only sustainability work as background rather than serving evidence.It is not an exhaustive PRISMA-style systematic review; the proposed equations, design choices, and roadmap are synthesis proposals, not implemented or experimentally validated results.Source: §2 · p. 2

3 What the Recent Literature Converges On

Across the reviewed systems, sustainability-relevant decisions span workload, phase, cache, model, hardware, and deployment choices. The literature therefore points toward control above individual kernels or model servers, while leaving the proposed control-plane implications to be tested.

3.1 Inference energy is workload-dependent, not a model constant

  • Energy per query varies with workload geometry and serving configuration: decode may dominate generation-heavy workloads, speculative decoding can save energy at small batches but increase it at high batches, and tensor parallelism can reduce latency while increasing total energy.The best optimization changes with workload geometry, so static choices such as always maximizing parallelism are hard to justify.Source: §3.1 · p. 3

3.2 Prefill and decode should be treated as different resources

3.3 KV state is both a performance asset and an energy asset

  • KV reuse can avoid recomputing long prefixes, but cache retention and transfer also consume resources, motivating an explicit transfer-versus-recompute model.Prior work spans paged memory management, distributed cache tiers, retaining or evicting state during tool-call pauses, and cross-endpoint KV sharing [Kwon et al., 2023, Qin et al., 2024, Li et al., 2025a, Guy et al., 2026].Source: §3.3 · p. 3

3.4 Efficiency is a model–software–hardware co-design problem

  • Efficiency depends on model–software–hardware co-design: quantization alone does not ensure performance, framework and kernel choices affect energy, and a lower-power device may suit decode despite lower peak speed [Lin et al., 2024, Fernandez et al., 2025, Patel et al., 2024].The proposed implication is to choose hardware by phase and workload rather than by one benchmark score.Source: §3.4 · pp. 3–4

3.5 Quality-aware model routing is a sustainability lever

  • Routing among models of different capability and cost can maintain a target quality level, but cost is not a perfect proxy for energy or carbon [Chen et al., 2023, Ding et al., 2024, Ong et al., 2024, Narasimhan and Menon, 2025].The synthesis proposes that future routers learn quality and sustainability jointly.Source: §3.5 · p. 4

3.6 Carbon, power, and embodied impact require control above the model server

4 A Taxonomy of Sustainable Inference Levers

The taxonomy makes interacting control variables visible across eight layers, while allowing decisions to operate at different timescales rather than requiring every layer to be optimized on every request.

4 A Taxonomy of Sustainable Inference Levers

  • The taxonomy spans request, model, and phase/operator layers, covering request shape and reuse expectations, model configuration, and operators such as prefill, reasoning, and decode.The boundaries are not strict; optimizing one layer in isolation can worsen another.Source: §4 · p. 5
  • State, hardware, replica/cluster, datacenter/region, and lifecycle layers cover cache and recomputation, accelerators and power states, routing and autoscaling, regional conditions, and embodied impact.The proposed hierarchy separates millisecond routing, seconds-to-minutes autoscaling and power control, and slower regional capacity and lifecycle decisions.Source: §4 · p. 6
  • A sustainable control plane need not optimize all eight layers for every request; it should expose fast routing, slower autoscaling and power control, and still slower regional and lifecycle decisions.These timescales connect request-level choices to broader infrastructure decisions.Source: §4 · pp. 5–6

5 Why llm-d Is a Natural Sustainability Control Point

llm-d operates above serving engines and coordinates distributed placement and orchestration. Its routing, cache-sharing, and tracing capabilities offer potential integration points for sustainability signals across endpoints and regions.

5 Why llm-d Is a Natural Sustainability Control Point

  • llm-d is designed above engines such as vLLM and SGLang, with its architecture evolving toward a general production control layer [llm-d Maintainers, 2026a, llm-d Maintainers, 2026b].The project describes itself as an inference control plane and has expanded capabilities including high availability, GPU-utilization-aware routing, flow control, and autoscaling foundations.Source: §5 · p. 6
  • Its predicted-latency scheduler uses prompt length, prefix-cache hit rate, running requests, queue depth, and KV utilization to estimate TTFT and TPOT on each endpoint.The paper proposes testing whether energy counters and hardware identity could support incremental-joule prediction in the same routing architecture.Source: §5 · p. 6
  • Peer-to-peer KV sharing compares copying cached state with recomputing a prefix, while end-to-end tracing spans gateway, endpoint picker, KV cache, phase proxy, and model server [Guy et al., 2026, Liu and Kumar, 2026].The paper proposes enriching the transfer crossover with energy and reuse factors, and attaching measured or estimated energy and carbon to request traces.Source: §5 · p. 6
  • Because llm-d can compare replicas and endpoints, the paper argues sustainability orchestration is better placed at the control plane than inside one model server.That placement preserves the ability to compare heterogeneous endpoints or regions.Source: §5 · p. 6

6 Proposed Architecture: Sustainable Inference Control Plane

The proposed Sustainable Inference Control Plane (SICP) extends llm-d’s decision loop with sustainability signals while retaining performance signals and quality/SLO constraints. It combines fast request routing with slower controls over serving resources and placement.

6 Proposed Architecture: Sustainable Inference Control Plane

  • SICP is a proposed, incremental llm-d extension: sustainability signals join existing performance signals as scorers, constraints, and autoscaling inputs.It is a design proposal, not a description of functionality already implemented in llm-d.Source: §6 · p. 6
  • The architecture pairs fast routing that predicts SLO feasibility, cache reuse, energy, and carbon with slower controls that adjust replicas, phases, power states, models, and regions.Telemetry closes the control loop.Source: Figure 1 · p. 7

6.1 Fast-path request decision

  • Fast-path estimates cover latency, energy, carbon, and serving cost for feasible endpoint plans, with quality and TTFT/TPOT SLOs constraining selection.The estimates use request properties and endpoint state, while carbon includes time-varying regional grid intensity.Source: §6.1 · p. 7

6.2 Cache action as an explicit decision

  • Cache decisions include local hit, route-to-owner, peer transfer, and recomputation; agentic workloads add retaining state for a TTL versus eviction or offload.This makes cache policy an explicit, measurable energy-optimization decision.Source: §6.2 · pp. 7–8

6.3 Phase-aware power and hardware control

  • Phase control assigns prefill and decode across hardware classes and power states, using workload variants such as compute-dense prefill and memory-bandwidth-efficient decode.Mixed or aggregated execution remains an option when network transfer costs outweigh disaggregation benefits.Source: §6.3 · p. 8

6.4 Model and reasoning routing

  • Model and reasoning routing would make quality-constrained sustainability choices, while measuring router energy because routing overhead can erase savings from avoiding large-model calls.The proposal extends model routing to an explicit sustainability decision.Source: §6.4 · p. 8

6.5 Regional and temporal scheduling

  • The slow path can defer or geographically shift deadline-flexible batch work, while keeping latency-sensitive sessions pinned for cache locality and user experience.The proposal treats interactive traffic as having less geographic and temporal flexibility than batch or offline traffic.Source: §6.5 · p. 8

7 Metrics: From Tokens per Second to Sustainable Goodput

The metrics section proposes evaluating sustainable serving through a vector of SLO- and quality-constrained measures, rather than throughput or energy alone. It pairs goodput efficiency with latency, quality, utilization, and resource reporting so operators can see trade-offs.

7 Metrics: From Tokens per Second to Sustainable Goodput

  • Performance-only measures can reward waste, while energy-only measures can reward unacceptable latency or low-quality models.The section therefore calls for SLO- and quality-constrained metrics.Source: §7 · p. 8
  • SLO-goodput counts useful output tokens or completed requests that meet required latency and quality constraints over an interval.The proposed efficiency measures include SLO-goodput per joule, per gram CO2e, per liter of water, and per dollar.Source: §7 · p. 8
  • The proposed reports pair efficiency metrics with TTFT, TPOT, latency percentiles, success rate, model quality, cache hit rate, throughput, utilization, and power.The cited “intelligence per watt” direction also argues for relating capability to energy rather than energy alone.Source: §7 · p. 8
  • A metric vector makes trade-offs explicit because operational carbon, embodied carbon, water, cost, and latency are not interchangeable.A single weighted score can embed value judgments and hide regressions; a Pareto frontier can support different operator policies.Source: §7 · p. 9

8 Experimental Roadmap

The experimental roadmap outlines a reproducible evaluation plan for validating SICP, rather than treating the design sketch as evidence. It spans testbed, workload, baseline, ablation, and measurement choices.

8 Experimental Roadmap

  • A later implementation study should validate SICP with controlled, reproducible experiments; the roadmap is a practical starting point, not evidence for the design.Source: §8 · p. 9

8.1 Testbed

  • The minimum testbed uses two accelerator classes and 8–16 endpoints, with serving variants deployed through llm-d and energy telemetry collected alongside tracing and metrics.A stronger testbed includes NVIDIA H100/H200/B200-class hardware plus an alternative supported accelerator family.Source: §8.1 · p. 9

8.2 Workloads

  • The workload matrix spans short chat, long-context/RAG, code, reasoning, agentic, batch/offline, and supported multimodal inference, sweeping offered load and prefix sharing.Source: §8.2 · p. 9

8.3 Baselines

  • Baselines include round-robin, least-loaded, cache-aware, predicted-latency, energy-only, carbon-only, and joint SICP routing, alongside alternative phase and KV policies.Phase comparisons cover aggregated, fixed disaggregation, and adaptive aggregation/disaggregation; KV comparisons cover recompute, owner-affinity, offload, and P2P transfer.Source: §8.3 · p. 9

8.4 Ablations

  • Ablations remove energy prediction, carbon intensity, cache energy, model routing, DVFS, and embodied-carbon accounting to separate sustainability gains from generic load balancing.Source: §8.4 · p. 9

8.5 Measurement discipline

  • Measurement discipline calls for high-resolution energy attribution to requests or batches, material host and network/storage energy, and explicit carbon and embodied-carbon assumptions.Report PUE, carbon-data source, average versus marginal carbon values, and lifecycle and utilization assumptions for embodied carbon.Source: §8.5 · p. 9

9 Research Questions and Testable Hypotheses

The paper frames six testable research questions about extending inference control across energy prediction, cache actions, phase allocation, geographic placement, model choice, and workload types. Each is presented with a hypothesis for evaluation.

9 Research Questions and Testable Hypotheses

  • RQ1 asks whether latency-routing features plus hardware and power state can support useful short-horizon incremental-energy prediction without expensive profiling.Source: §9 · p. 10
  • RQ2 hypothesizes that peer-to-peer transfer can reduce TTFT and energy for long shared prefixes on high-bandwidth fabrics, with the crossover depending on system conditions.The stated conditions include prefix length, network type, memory tier, and endpoint utilization.Source: §9 · p. 10
  • RQ3 asks whether adaptive prefill/decode disaggregation can outperform fixed disaggregation by selecting aggregation, chunked prefill, or disaggregation based on SLO headroom and transfer cost.Source: §9 · p. 10
  • RQ4 hypothesizes that regional shifting can reduce operational carbon for batch and stateless traffic, while shifting multi-turn sessions may increase recomputation enough to offset gains.The proposed comparison favors session pinning with carbon-aware placement of new sessions over naive global routing.Source: §9 · p. 10
  • RQ5 compares sustainability gains from model routing and infrastructure routing, hypothesizing that their relative value depends on query difficulty and fixed-model quality or compliance requirements.The hypothesis is that the best system composes both.Source: §9 · p. 10
  • RQ6 asks whether a graph or phase representation can extend control-plane sustainability decisions from two-stage text prefill/decode to agentic and multimodal workloads.The proposed inspiration is composite serving systems such as M* [Jha et al., 2026].Source: §9 · p. 10

10 Future of llm-d: Toward a Resource-Intelligence Control Plane

The proposed next step for llm-d is to make resource intelligence a first-class control-plane concern, extending current orchestration toward quality- and SLO-constrained useful work per resource.

10 Future of llm-d: Toward a Resource-Intelligence Control Plane

  • llm-d is evolving toward a common decision layer above changing engines, with expanded cache management, heterogeneous-engine support, multimodal serving, and routing.The paper presents resource intelligence as a proposed next step, not an existing capability.Source: §10 · p. 10
  • Proposed Endpoint Picker and autoscaling extensions would add incremental joules, power headroom, carbon intensity, and demand-response signals under availability guardrails.These are proposed extensions, not current llm-d features.Source: §10 · p. 10
  • Phase-specific DVFS and accelerator selection would map prefill and decode to energy-efficient operating points, while KV orchestration prices hits, transfers, offload, and recomputation.The KV alternatives include local hits, owner routing, peer-to-peer copies, and DRAM or storage offload.Source: §10 · p. 10
  • Quality-aware model routing, sustainability traces, and cross-cluster federation would add quality targets, per-request resource telemetry, and flexible placement for deadline-tolerant batches.The proposal would protect cache-local interactive sessions while routing flexible work across regions or renewable-powered sites.Source: §10 · p. 11
  • The proposed target is quality- and SLO-satisfied useful work per constrained resource, rather than tokens per second per GPU.The paper argues that this is a more durable abstraction as accelerators, models, and energy systems change.Source: §10 · p. 11

11 Limitations and Responsible Claims

The paper distinguishes prior systems’ measured results from its own unvalidated proposal and cautions that environmental estimates and optimization interactions depend on their settings.

11 Limitations and Responsible Claims

  • The paper does not remeasure cited gains or claim unchanged transfer to llm-d; reported improvements are not additive, and interactions can be antagonistic.For example, cache retention may avoid recomputation while increasing memory pressure, and geographic carbon routing may degrade cache locality.Source: §11 · p. 11
  • Carbon accounting depends on location, time, PUE, and average versus marginal grid emissions, so environmental metrics should be reported alongside raw energy and system behavior.Water and embodied-carbon estimates have additional site, cooling, hardware-lifetime, and utilization dependencies.Source: §11 · p. 11
  • Model routing and reasoning-budget control should meet explicit quality requirements and be evaluated on application-relevant tasks.The paper cautions against sustainability gains that conceal quality degradation from users.Source: §11 · p. 11

12 Conclusion

The conclusion connects recurring serving-system control variables to an orchestration problem and proposes a testable llm-d control-plane experiment, not a measured gain.

12 Conclusion

  • Across the reviewed work, recurring control variables include workload shape, phase behavior, cache locality, hardware choice, power state, SLO headroom, and placement.The paper interprets this pattern as evidence that sustainability is also an orchestration problem, not only a model- or kernel-level one.Source: §12 · p. 11
  • A sustainability-aware llm-d experiment could test energy, carbon, cache-recompute cost, and power headroom alongside TTFT, TPOT, queue depth, and cache locality.The experiment can build on existing serving engines rather than replacing them.Source: §12 · p. 11
  • The proposal treats each inference request path as an optimization target while holding quality and SLOs constant, pending implementation and measurement.Until then, SICP is an engineering hypothesis derived from prior research, not a measured improvement.Source: §12 · p. 11

Related papers

Loading 2609.05565v1…