PaperlayerView exact revision on arXiv

Source-linked AI summary

HeatCache: Thermal-aware Energy-efficient LLM Inference Scheduling for Chassis-level Liquid Cooling in Sustainable Edge Server Rooms

Rui Lu, Huanghuang Liang, Kaiqi Guan, Dan Wang

arXiv:2609.12449v1cs.DCcs.PFcs.SYeess.SY

TL;DR

Institution-scale LLM inference faces energy and thermal constraints as ambient setpoints rise, while scheduling must balance service objectives with thermal safety. HeatCache uses AIO heat buffering, heat-demand prediction, and online scheduling to manage that balance. Its reported evaluation shows reduced SLO violations, while the conclusion presents thermal-aware scheduling as a practical lever for reliable, energy-efficient edge serving.

  • Problem

    Warm-setpoint edge LLM serving needs scheduling that accounts for AIO thermal headroom while meeting latency SLOs and avoiding thermal limits.

    Source: §1 · pp. 1–2

  • Method

    HeatCache estimates each GPU’s safe heat budget, predicts candidate actions’ heat demand, and selects energy-efficient actions subject to thermal checks and latency SLOs.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    HeatCache reduces SLO violations by up to 82.5% compared with vLLM-A/W and achieves the lowest violations among evaluated systems.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    Thermal-aware scheduling is presented as a practical lever for reliable, energy-efficient edge LLM serving without infrastructure retrofits.

    Source: §6 · pp. 8–9

    HeatCache targets single-server, institution-scale multi-GPU systems with chassis-level AIO cooling, not facility-plumbed or immersion cooling or cross-server coordination.

    Source: C. Limitations and Discussions · p. 13

Abstract

from arXiv · show

LLM inference is increasingly deployed at institution-scale edges to meet service requirements. However, multi-GPU inference consumes a large amount of electricity and produces substantial heat. To improve sustainability, operators and regulations often demand raising the ambient setpoint to reduce cooling electricity. This can increase thermal throttling and hardware aging, leading to Service-Level Objective violations. In this paper, we present HeatCache, a thermal-aware, energy-efficient LLM inference scheduler for commercial chassis-level AIO liquid-cooled GPUs at sustainable ambient temperatures. HeatCache treats AIO loops as a temporary heat buffer, measured by heat budget and schedules requests to minimize energy subject to thermal safety and SLO constraints, based on an electrical-informed heat-demand estimation from HeatiTS. We implement HeatCache atop vLLM and show that it reduces computing energy by up to 18.0%, decreases thermal-throttle exposure by 81.7%, and maintains SLO violation rates below 0.9% even up to $48~^{\circ}\mathrm{C}$.

1. Introduction

Institution-scale LLM serving faces rising energy and thermal constraints as operators raise room temperatures, while existing inference schedulers often ignore thermal dynamics. HeatCache uses AIO thermal buffering to schedule inference for energy efficiency subject to thermal safety and SLOs.

1. Introduction

  • Raising ambient setpoints can reduce cooling electricity but shrinks thermal headroom, while thermally unaware scheduling can misalign heat generation and dissipation and risk throttling or SLO violations.The paper notes prior estimates of 2–5% cooling-electricity savings per 1 °C increase.Source: §1 · p. 1
  • Chassis-level AIO loops offer short-term thermal capacitance that can buffer heat, while fitting existing server chassis and racks for institutional edge deployments.The paper frames this as an opportunity to manage heat generation without facility-level liquid-cooling infrastructure.Source: §1 · pp. 1–2
  • HeatCache’s online controller, using heat-budget and workload heat-demand models, reduces computing energy by up to 18.0%, decreases thermal-throttle exposure by 81.7%, and maintains SLO violation rates below 0.9% up to 48 °C.Source: §1 · p. 2
  • HeatCache combines a per-GPU heat-budget model, HeatiTS heat-demand forecasts, and an online controller that matches candidate scheduling actions to thermal capacity while enforcing SLO and temperature limits.HeatiTS maps candidate actions to predicted heat demand and includes an electricity regularizer.Source: §1 · p. 2

2. Background and Motivation

Dense GPU workloads and warmer ambient setpoints create thermal risks for edge LLM serving, while chassis-level AIO cooling can provide thermal buffering. Motivational experiments show that TTFT violations vary with ambient temperature and cooling configuration.

2.1. Heat and Cooling Challenges in Edge LLM Serving

  • High GPU power and uneven thermal profiles can trigger clock throttling, degrading throughput and tail latency; warm ambient conditions reduce AIO thermal headroom and raise throttling risk.AIO cold plates and sealed loops reduce thermal resistance and provide capacitance that smooths power spikes.Source: §2.1 · p. 2

2.2. LLM Scheduling, SLOs, and Thermal Motivation

  • AIO cooling kept TTFT violations within a few percent in the motivational experiment, while air cooling reached up to 34% as ambient temperature increased.Warmer coolant increased violations, whereas larger coolant volumes provided more buffering and reduced them.Source: §2.2 · p. 3, Figure 2 · p. 3
  • Existing edge LLM schedulers largely ignore temperature, although batching and routing affect heat distribution and throttling in warm AIO-cooled environments.The authors use the temperature-dependent TTFT results to motivate thermal-aware scheduling.Source: §2.2 · p. 3

3. Design of HeatCache

HeatCache coordinates LLM serving, GPU power management, and AIO-loop thermal behavior to minimize IT energy while meeting latency SLOs and thermal constraints. It combines a per-GPU heat budget, job-level power and heat forecasts, and an online controller that selects feasible scheduling actions.

3.1. Design Overview

  • The design coordinates LLM batching and routing, GPU frequency, and AIO-loop thermal behavior to minimize IT energy while preserving latency SLOs and avoiding GPU throttling.It targets a multi-GPU edge server operating at a warmer ambient setpoint.Source: §3.1 · p. 3

3.2. Physics-based Heat Budget Model

  • A reduced-order RC model tracks each GPU and AIO loop’s thermal state under measured power and ambient conditions, estimating remaining heat storage before throttling.The model uses thermal capacitances and resistances across the junction, package/cold-plate interface, coolant, and radiator.Source: §3.2 · pp. 3–4

3.3. HeatiTS: Job-Level Electrical-regularized Power and Heat Demand Model

  • HeatiTS forecasts job-level power and heat demand from GPU job history and candidate actions, using electrical regularization and a high-quantile heat-demand estimate.Its forecasts support both heat-budget checks and expected IT-energy estimation in the controller.Source: §3.3 · pp. 4–5

3.4. Energy-Efficient SLO-Aware Online Controller

  • HeatCache re-ranks candidate serving actions using heat budgets and HeatiTS power/time predictions, selecting energy-efficient options that satisfy thermal and latency constraints.The controller can permit short, high-throughput bursts when heat budget is available and adjust workload or GPU frequency as the budget approaches exhaustion.Source: §3.1 · p. 3, §3.4 · pp. 5–6

4. Implementation and Evaluation

HeatCache combines an AIO heat-budget model, job-level heat-demand forecasting, and an SLO-aware controller, implemented on vLLM and evaluated on multi-GPU AIO-cooled testbeds. At a 48±2°C setpoint, it reduces computing energy and SLO violations while limiting thermal-throttle exposure; ablations show the value of its heat-budget, forecasting, and joint-control components.

4.1. Implementation

  • The vLLM plugin combines a physics-based heat-budget monitor, HeatiTS heat-demand forecaster, and SLO-aware controller, with online micro-batch tuning and safe DVFS states.The prototype uses Kubernetes, KServe, NVIDIA DCGM, and an offline-calibrated per-GPU lumped RC model.Source: §4.1 · p. 6

4.2. Evaluation Setup

  • Evaluation used four water-cooled RTX 4090 GPUs in an enclosure with an 18°C–48°C setpoint, across Chat, Code, and Sum traces and multiple serving and DVFS baselines.The metrics were per-job GPU computing energy, TTFT/TPOT SLO violations, and throttle exposure; the coolant loop’s pump and fans use about 25 W per loop, included in total system energy but excluded from GPU computing-energy comparisons.Source: §4.2 · pp. 6–7, §4.1 · p. 6

4.3. Evaluation Results

  • 18.0%: HeatCache reduces computing energy by up to 18.0% through AIO thermal buffering, consistently outperforming the compared baselines.The comparison is under Chat, Code, and Sum at 48±2°C; Figure 5c is relative to vLLM-A@18°C.Source: §4.3 · p. 7, Figure 5 · p. 7
  • 82.5%: HeatCache reduces SLO violations by up to 82.5% versus vLLM-A/W and achieves 0.8%–2.8% throttle exposure.It has the lowest throttle exposure and SLO violations among the compared approaches.Source: §4.3 · p. 7

4.4. Ablation Studies

  • 80.50%/86.07%: Compared with w/o RC, full reduces TTFT/TPOT SLO violations by 80.50%/86.07% and throttle exposure by 81.69%.The largest gap is on Sum; w/o PFCV has comparable throttle exposure but 7% lower latency than full, which incurs verification overhead.Source: Figure 6 · p. 7, §4.4 · p. 8
  • Physics-only estimates in w/o HeatiTS miss prompt-length and burst-dependent heat dynamics, producing higher SLO violations and computing energy than full.Under mixed prompt lengths and bursty arrivals, estimates may be unsafe or overly conservative, while HeatiTS supports headroom allocation that maintains thermal safety.Source: §4.4 · p. 8

5. Related Works

Prior thermal-aware GPU scheduling and energy-management approaches often assume air cooling or treat thermal behavior as external, while cooling controllers commonly optimize independently of workload placement. HeatCache instead models chassis-level AIO loop inertia and coordinates workload heat demand with available thermal budgets for warm edge environments.

Thermal Management in GPU Computing

  • Earlier GPU thermal-management work addresses throttling through workload migration or balanced scheduling, but assumes air cooling and ignores cooling dynamics and LLM scheduling specifics.HeatCache models water-cooling loops as thermal buffers and coordinates job heat demand to prevent throttling at high ambient temperatures.Source: §5 · p. 8

Datacenter Cooling and Energy Efficiency

  • Cooling optimization often treats workloads as exogenous, and cluster thermal placement approaches such as TAPAS and TAWS do not model per-device capacitance.HeatCache coordinates heat demand and budgets while modeling AIO loop inertia at the edge.Source: §5 · p. 8

LLM Inference Scheduling and SLO Management

  • LLM serving schedulers prioritize throughput and SLOs, while energy-aware DVFS systems generally ignore cooling dynamics relevant to warm edge environments.HeatCache integrates heat prediction and thermal-aware scheduling to meet SLOs and prevent throttling.Source: §5 · p. 8

6. Conclusion

HeatCache uses thermal-aware scheduling for institution-scale LLM inference under rising ambient setpoints, exploiting transient buffering in chassis-level AIO cooling. The prototype avoids throttling while maintaining service quality without infrastructure retrofits.

  • HeatCache is a thermal-aware scheduling controller for institution-scale LLM inference under rising ambient setpoints.Source: §6 · p. 8
  • It exploits transient thermal buffering from chassis-level AIO liquid cooling in the serving stack.Source: §6 · p. 8
  • Thermal-aware scheduling proactively avoids throttling while maintaining service quality, without infrastructure retrofits.Source: §6 · p. 9

A.1. RC network and continuous-time dynamics

The appendix models each AIO-cooled GPU with a lumped thermal network that tracks heat flow through hardware and coolant toward room air. Its dynamics use room temperature as an external boundary condition.

A.1. RC network and continuous-time dynamics

  • The model represents a single AIO-cooled GPU with a lumped RC thermal network.Source: Figure 7 · p. 11, §A.1 · p. 11
  • The thermal states include package/cold-plate, coolant, and radiator temperatures, alongside the GPU junction state.Source: §A.1 · p. 11
  • Room air is treated as an external boundary at the setpoint, and the network's continuous-time dynamics follow energy balance.Source: §A.1 · p. 11

A.2. Discretization and thermal divider ratio

The appendix discretizes the GPU thermal model and defines a resistance-divider reference profile. It calibrates per-GPU RC parameters from heating and cooling experiments, then uses live telemetry to monitor heat budget online.

A.2. Discretization and thermal divider ratio

  • The continuous thermal model is converted to the discrete state update Tg,j+1 = AdTg,j + BdPg,j + EdTroom.Source: §A.2 · p. 12
  • The thermal resistance divider ratio defines a reference temperature profile for a target junction temperature under room-temperature boundary conditions.Source: §A.2 · p. 12

A.3. Calibration of RC parameters

  • Per-GPU RC parameters are identified offline with controlled step-load experiments that capture both heating and cooling transients.The experiments use idle, high-power, and low-power plateaus; parameter fitting minimizes junction-temperature trajectory error.Source: §A.3 · p. 12
  • After calibration, the online heat-budget monitor integrates discrete dynamics using live power and junction-temperature telemetry and the ambient setpoint.Source: §A.3 · p. 12

B. Instantiating ˆLr(a) for Common LLM SLO Metrics

The appendix maps the controller's generic latency estimate to standard LLM serving SLO metrics. It estimates waiting time by replaying predicted job durations and applies metric-specific bounds to candidate actions.

B. Instantiating ˆLr(a) for Common LLM SLO Metrics

C. Limitations and Discussions

HeatCache is scoped to single-server, chassis-level AIO systems and leaves other cooling architectures and facility-level coordination outside its design. Future work targets easier calibration, cluster-scale scheduling, and joint facility control.

C. Limitations and Discussions

  • HeatCache targets institution-scale, multi-GPU servers with chassis-level AIO cooling, not facility-plumbed direct-to-chip/CDU or immersion systems.Those alternatives have substantially different thermal states and control knobs.Source: C. Limitations and Discussions · p. 13
  • The controller operates within one server, assumes a given ambient setpoint, and does not model fine-grained airflow or detailed fan and radiator dynamics.It uses a reduced-order lumped RC thermal model and does not coordinate cross-server placement, migration, or multi-tenant policies under shared power caps.Source: C. Limitations and Discussions · p. 13
  • Future work proposes self-calibration and online drift detection to generalize headroom tracking across heterogeneous GPUs and AIO designs.The stated aim is to reduce per-platform setup effort while remaining safe under aging and maintenance.Source: C. Limitations and Discussions · pp. 13–14
  • Future work also proposes cluster-scheduler integration and facility-controller co-design, including setpoint, airflow, carbon, and price-aware objectives.Source: C. Limitations and Discussions · p. 14
Loading 2609.12449v1…