PaperlayerView exact revision on arXiv

Source-linked AI summary

Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

Xiaoyang Zhang, Yuejun Yan, Qingang Zhang, Yonggang Wen

arXiv:2610.35230v1cs.DCcs.AIcs.CY

TL;DR

Carbon-aware LLM routers have treated conversation turns independently, despite the carbon cost of recomputing history when a turn moves away from its cached deployment. CarbonRouter prices that recomputation and uses a gate to avoid likely reversed switches while jointly choosing a model, GPU, and region. It reduces CO2 by 16.1% on average relative to FairCarbon, with larger savings for longer sessions, while improving predicted quality and estimated latency.

  • Problem

    Existing carbon-aware LLM routers treat conversation turns independently and do not price the history recomputation incurred when a turn moves to a deployment without its cache.

    Source: §1 · p. 1

  • Method

    CarbonRouter jointly selects a model, GPU type, and region for each query, pricing uncached-history recomputation and using a lightweight gate to avoid switches likely to be reversed.

    Source: §1 · p. 1

  • Results

    CarbonRouter reduces CO2 by 16.1% on average relative to FairCarbon and by 23.1% for sessions longer than 30 turns, while improving predicted quality and estimated latency.

    Source: §7 · p. 9

  • Takeaways & Limitations

    A cache miss can more than triple emissions for a query with a long history, making cache locality consequential to carbon-aware routing.

    Source: §1 · p. 1, §7 · p. 8

    The evaluation is trace-driven simulation: energy and prefill latency use analytical estimates, and answer quality is predicted rather than judged.

    Source: §6 · p. 8

Abstract

from arXiv · show

The sustainability of the Web has drawn growing attention, as large language model (LLM)-based web services leave a substantial carbon footprint. These services typically run in geo-distributed data centers whose grid carbon intensity varies across regions and over time, so routing each query to a cleaner region or a smaller model can reduce emissions. A conversation, however, is a stateful web session: each turn reuses the key--value (KV) cache of its history, which resides only where earlier turns were served. Serving a turn on another deployment requires recomputing this history, whose carbon cost grows with the conversation and can outweigh the saving of the move. Existing carbon-aware LLM routers treat a conversation's queries as independent and ignore this cost. We present CarbonRouter, an online framework that jointly selects the model, GPU type, and region for each query to minimize carbon emissions. CarbonRouter introduces a cache-aware carbon cost model, which charges history recomputation to every candidate deployment using cached-token counts from the serving backends, and uses a lightweight gate that keeps a session on its cached deployment when it would likely move back at the next turn and rebuild its history again. We evaluate CarbonRouter in trace-driven experiments on two real-world conversation datasets with five LLMs, three GPU types, and four regions, where it achieves the lowest carbon of all policies. CarbonRouter reduces CO$_2$ by 16.1\% on average relative to FairCarbon, a state-of-the-art carbon-aware router with the same action space, while also improving predicted answer quality and estimated latency. Moreover, the saving rises further to 23.1\% for sessions longer than 30 turns.

1 INTRODUCTION

Carbon-aware routing can reduce emissions by choosing cleaner regions or more efficient models and GPUs, but moving a turn can require costly recomputation of its conversation history. CarbonRouter prices that recomputation and jointly selects deployments for each query; its evaluation reports lower carbon than prior policies.

1 INTRODUCTION

  • LLM web services can reduce operational carbon by using smaller or more efficient models and GPUs, or regions with cleaner grids whose carbon intensity varies over time.These services route requests among geo-distributed data centers.Source: §1 · p. 1
  • A deployment switch requires recomputing uncached conversation history, and long histories raise the carbon advantage needed for switching; existing carbon-aware routers treat turns independently and omit this cost.For 4K history tokens, the new region must be 68% cleaner than the current one for a switch to pay off.Source: §1 · p. 1
  • CarbonRouter jointly selects model, GPU type, and region, charging candidate deployments for history recomputation; its CCAR algorithm also uses a lightweight gate to avoid switches likely to be reversed next turn.The framework operates at the routing layer and leaves models, inference engines, and clients unchanged.Source: §1 · pp. 1–2
  • CarbonRouter achieves the lowest carbon of all evaluated policies, reducing CO2 by 16.1% on average relative to FairCarbon and 23.1% for sessions longer than 30 turns.It also increases mean predicted quality; the saving grows with session length.Source: §1 · p. 2

2 CARBON AND KV-CACHE ANALYSIS

The analysis examines regional and temporal grid-carbon variation alongside the emissions cost of rebuilding conversation history after a KV-cache miss. It asks when those factors make routing a session to another deployment worthwhile.

2.2 Carbon Cost of a KV-Cache Miss

  • 40.3% of queries carry at least 4K history tokens, and cache misses raise median emissions by 126%, 231%, and 632% across increasing history ranges, adding up to 2.5 s of prefill latency.The emissions increases correspond to 2K–4K, 4K–8K, and more than 8K history tokens, respectively.Source: Figure 2 · p. 2, §2.2 · p. 3

2.3 When Does Switching Pay Off?

  • With 4K history, destination B must be 68% cleaner than cached region A; as histories grow, the cleaner destination must be cleaner still.The threshold assumes default energy ratios and the single-turn analysis shows a trend, not an exact threshold.Source: Figure 2 · p. 2, §2.3 · p. 3

2.1 Carbon Intensity across Regions and Time

  • Grid intensity spans 20.4× annually and varies 2.7–5.9× within each evaluated region; no region is cleanest consistently, supporting online, per-query routing.The comparison uses 2024 annual means across 21 countries and areas and five-minute data from four evaluation regions in 2025.Source: §2.1 · p. 2

3 CARBONROUTER DESIGN

CarbonRouter jointly routes each turn across models, GPU types, and regions using cache state and deployment profiles. Its design combines quality filtering, cache-aware cost estimates, latency constraints, and a gate that can keep sessions on their cached deployment.

3.1 Overview

  • CarbonRouter estimates each candidate deployment’s energy, latency, and emissions, then selects the feasible deployment with the lowest estimated carbon using current regional carbon intensity and cached-history state.Profiles can be supplied by operators, and the framework can use measured profiles or another carbon-intensity provider.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2 System Model

  • The system models sequential conversation turns across available deployments; each backend retains its KV cache until its time-to-live expires or memory pressure triggers eviction.The backend reports cache state to the router.Source: §3.2 · p. 3

3.3 Quality Prediction and Model Filtering

  • A quality predictor estimates candidate-model quality on a 0–1 scale, and model filtering admits smaller models only when their predicted quality is near the best candidate.The quality threshold is defined as a required fraction of the best candidate’s predicted quality.Source: §3.3 · p. 3

3.4 Carbon Cost Model

  • The carbon cost model accounts for cached and uncached history separately, charging cached tokens at the cached-input energy rate and recomputing the remainder.A cache miss incurs a recomputation penalty; the cost estimate uses regional carbon intensity at dispatch for the request.Source: §3.4 · p. 4

3.5 Latency Model

  • TTFT is estimated from prefill execution for the uncached history, making it a compute-side estimate rather than an end-to-end user-latency measure.The model also neglects throughput variation with context length and concurrent load.Source: §3.5 · p. 4

3.6 Constraints and Optimization Goal

  • The online router uses information available at dispatch, enforces quality and TTFT constraints where possible, and serves best-effort on the shortest-estimated-TTFT deployments when only the TTFT bound cannot be met.It relaxes the TTFT bound rather than rejecting the turn in that case.Source: §3.6 · p. 4

3.7 Routing Algorithm

  • CCAR proposes the feasible deployment with the lowest estimated carbon, charging candidates for uncached-history recomputation, while a learned gate can keep a session on its cached deployment.The gate acts when a proposed switch would leave a feasible deployment that still caches the history and predicts a likely reversal at the next turn.Source: §3.7 · p. 5

4 EVALUATION

The evaluation uses trace-driven simulation across real conversation workloads and compares CarbonRouter with carbon-aware and cache-aware baselines. CarbonRouter reduces carbon overall, with larger savings for longer sessions and savings across the tested sensitivity settings.

4.1 Experimental Setup

  • Trace-driven evaluation uses ShareChat and ShareGPT, five instruction-tuned LLMs, three GPU types, four regions, analytical energy and latency profiles, and emulated cache availability.The profiles use published data rather than measurements on the evaluated deployments.Source: §4 · p. 5, §4.1 · p. 5

4.2 Overall Performance

  • CarbonRouter achieves the lowest CO2 on both datasets, reducing emissions 16.1–54.3% versus four baselines, including 16.1% versus FairCarbon.The overall comparison is normalized to Session-Affinity = 1; P95 TTFT is estimated from prefill time.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Effect of Session Turns

  • Savings over FairCarbon rise from 15.3% for sessions longer than 5 turns to 23.1% for sessions longer than 30; in the latter group, recomputation energy per query falls 70%.That reduction is from 205 to 61 mWh.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4 Ablation Study

  • Without pricing, CO2 rises 8.9%; without the gate, CO2 rises 3.5% overall while estimated P95 TTFT rises 71.4%.The authors attribute more of the carbon saving to pricing and more of the latency benefit to the gate.Source: Table 2 · p. 7, §4.4 · p. 7
  • For sessions longer than 30 turns, removing the gate raises CO2 by 7.3% and estimated P95 TTFT reaches 1.47 s, while the gate blocks 4.1 switches per session.Across all sessions, the gate blocks 0.7 switches per session.Source: Figure 5 · p. 7, §4.4 · p. 7

4.5 Sensitivity Analysis

  • CarbonRouter saves 9.4–18.8% over FairCarbon in every tested setting; at βQ=0.85, savings fall from 15.1% at c=0.5 to 9.4% at c=0.25.The authors attribute the reduction under tighter capacity to the cleanest region filling earlier.Source: Table 3 · p. 8, Table 4 · p. 8, §4.5 · p. 8

5 RELATED WORK

Prior carbon-aware computing exploits spatial and temporal grid variation, while prior carbon-aware LLM routers treat conversation turns independently and do not price history recomputation. Related work also includes cache-aware LLM serving and offline task migration.

  • Spatial and temporal carbon shifting exploit grid-intensity variation, but these systems assume workloads are deferrable or stateless, so moving them costs little.Simple geographical load-balancing policies capture much of the benefit of spatial shifting.Source: §5 · p. 8
  • AgentDecarbonizer [41] prices lost-cache recomputation for offline agent tasks, while cache-aware LLM serving schedules queries to cache-holding instances without carbon routing.GreenCache [32] sizes one deployment’s KV cache by carbon but does not move requests.Source: §5 · p. 8
  • Carbon-aware LLM routers select models, GPUs, or regions but treat turns independently, unlike CarbonRouter’s cache-miss pricing.The cited routers include GAR [29], CEDAR, and FairCarbon.Source: §5 · p. 8

6 LIMITATIONS

The evaluation and quality conclusions rely on simulation, analytical cost models, and predicted rather than judged answers.

  • The evaluation is trace-driven simulation, not measurement on the authors’ own hardware.Energy and prefill latency come from an analytical model built from published data.Source: §6 · p. 8
  • Answer quality is predicted rather than judged.Source: §6 · p. 8
  • Production validation of measured hit and miss energy remains future work, as do models of network latency, queueing, and cross-region KV-cache transfer.Source: §6 · p. 8

7 CONCLUSION

CarbonRouter accounts for history recomputation when routing multi-turn conversations. It achieves lower reported CO2 than FairCarbon, with larger savings for sessions longer than 30 turns.

7 CONCLUSION

  • A cache miss can more than triple the emissions of a query with a long history.Source: §7 · p. 8
  • 16.1% lower CO2 on average relative to FairCarbon, rising to 23.1% for sessions longer than 30 turns.CarbonRouter also improves predicted answer quality and estimated latency relative to FairCarbon.Source: §7 · p. 9
  • CarbonRouter prices history recomputation for every candidate deployment and uses a lightweight gate to avoid switches likely to be reversed.Source: §7 · p. 9

A ENERGY AND LATENCY PROFILES

The appendix describes analytical energy and latency profiles for prefill and decoding, along with deployment and GPU assumptions. Table 6 presents model–GPU energy profiles relative to a Llama-3.1-70B A100 reference.

A ENERGY AND LATENCY PROFILES

  • An uncached input token for a model with P_m parameters costs 2P_m FLOPs in the prefill model.The model also specifies prefill utilization, peak FP16 rate, GPU power, and GPU count as inputs to its energy and throughput estimates.Source: A ENERGY AND LATENCY PROFILES · p. 11
  • The energy profile uses a Llama-3.1-70B reference energy of 4.24 × 10^-9 kWh per token and GPU factors calibrated to relative energy values.The stated relative energies are 1.433, 1, and 0.8 for A10, A100, and H100, respectively.Source: A ENERGY AND LATENCY PROFILES · p. 11
  • For batch-one decoding, the model treats each step as memory-bound and estimates throughput from GPU bandwidth and model size.It uses batch-one decoding time because a memory-bound step changes little up to the effective batch size of about 5–17 sequences.Source: A ENERGY AND LATENCY PROFILES · p. 11
  • The 70B model uses eight A10, eight A100-40GB, or four H100 GPUs, leaving 32–147 GB for KV cache under the stated memory assumptions.Table 6 lists energy relative to Llama-3.1-70B on A100 and throughput in tokens/s.Source: A ENERGY AND LATENCY PROFILES · p. 11, Table 6 · p. 11

Related papers

Loading 2610.35230v1…