Source-linked AI summary
ThermE: Predictive Management of Shared Thermal Headroom for Sustained LLM Inference on Thermally Constrained Edge SoCs
Rui Lu, Yuheng Wang, Bozheng Liu
TL;DR
Sustained LLM inference on thermally constrained edge SoCs draws on shared, time-varying headroom, but existing controls lack visibility into request state and future headroom. ThermE predicts headroom and coordinates actions; on the Stress workload at 65°C, it reduced latency metrics relative to vLLM and lowered SLO violations relative to the strongest baseline.
Problem
Edge SoCs couple CPU, GPU, and RAM through shared cooling, while existing controls do not account for how current actions affect future thermal headroom.
Source: §1 · pp. 1–2Method
ThermE combines LLM-to-heat compilation, physics-constrained headroom prediction, and uncertainty-aware scheduling to manage shared thermal headroom.
Source: §1 · p. 2, §4.3.1 · p. 6Results
40.55% and 12.17%: under Stress at 65°C, ThermE reduced TTFT and TPOT, respectively, relative to vLLM.
Source: §1 · p. 2Takeaways & Limitations
ThermE provides a predictive thermal resource management approach for sustained inference on compact platforms where limited cooling constrains continuous workloads.
Source: §8 · p. 12
Abstract
from arXiv · showhide
Compact edge system-on-chip (SoC) platforms increasingly run sustained LLM inference under thermal constraints, while their CPU, GPU, and RAM share a cooling path. Prefill and decode therefore consume shared, time-varying thermal headroom, yet vendor governors react only near hardware throttling thresholds without knowledge of request state or upcoming work. A control decision that improves current performance can thus consume headroom too quickly and degrade subsequent service. In this paper, we present ThermE, a runtime system that predicts and jointly manages shared thermal headroom for sustained LLM inference on edge SoCs. Its Fast LLM-to-Heat Compiler maps the model, requests, runtime state, and candidate actions to domain heats without executing LLMs. A partial differential equation (PDE)-constrained Headroom Predictor uses ThermPINN for offline thermal identification and a Reduced Headroom Predictor (RHP) for low-overhead online uncertainty-calibrated headroom forecasts. An Uncertainty-Aware Action Scheduler then selects actions that balance serving quality and future headroom. We implement ThermE atop vLLM and evaluate it across four LLM inference workloads. The results show that ThermE reduces TTFT and TPOT by 40.55% and 12.17%, respectively, relative to vLLM. It achieves a 5.70% SLO violation rate, compared with 12.30% for the strongest baseline, while its predictor obtains a 1.94 $^\circ$C MAE with 11.50 ms overhead.
1 Introduction
Sustained LLM inference on edge SoCs consumes shared, time-varying thermal headroom, while hardware governors react without knowing request state or upcoming work. ThermE predicts headroom and coordinates inference actions; on the Stress workload at 65°C, it improved serving metrics relative to vLLM.
1 Introduction
- CPU, GPU, and RAM share a cooling path, so heat generated in one domain reduces the headroom available to the others over time.Deployment conditions set initial headroom, while workload and cooling conditions shape its later availability.Source: §1 · p. 1, Figure 2 · p. 2
- Vendor governors react to measured temperature, current, or power rather than LLM phase, request state, remaining tokens, or upcoming work.Existing controls adjust instantaneous power or latency without accounting for future headroom, so a performance-improving decision may consume it too quickly.Source: §1 · pp. 1–2
- Motivational experiments find that throttling follows substantial headroom consumption, thermal zones heat and cool together, and hardware profiles trade off TTFT, TPOT, and headroom use.Together, these observations characterize headroom as a shared, time-varying resource to manage proactively according to serving demand and thermal state.Source: §1 · p. 2
- ThermE predicts and manages shared thermal headroom for sustained LLM inference on thermally constrained edge SoCs.Its compiler maps candidate workloads to CPU, GPU, and RAM heat without executing the LLM; its predictor and scheduler support uncertainty-aware action selection.Source: §1 · p. 2
- 40.55%, 12.17%, and 13.01%: under Stress at 65°C, ThermE reduced TTFT, TPOT, and p95 end-to-end latency relative to vLLM.Its SLO violation rate was 5.70%, compared with 12.30% for the strongest baseline.Source: §1 · p. 2
2 Background
Edge SoCs couple compute and memory temperatures through shared cooling, and retained heat makes current headroom depend on recent activity. LLM serving distributes work across CPU, GPU, and shared RAM, while hardware thermal controls respond to zone temperatures and trip points.
2.1 Thermal Characteristics of Edge SoC Platforms
- Shared cooling couples CPU, GPU, and memory thermal behavior; ambient temperature and cooling state set initial headroom, while retained heat from recent activity affects current margin.The heat spreader, heat sink, and fan form a common path across these domains.Source: §2.1 · pp. 2–3
2.2 LLM Inference Execution on Edges
- Prefill processes the prompt and builds the KV cache, while decode generates tokens one at a time; prefill largely determines TTFT and decode is characterized by TPOT and sustained throughput.vLLM maps request processing, scheduling, and kernel launches to the CPU, model kernels to the integrated GPU, and weights and KV cache to shared LPDDR RAM.Source: §2.2 · p. 3, Figure 3 · p. 3
2.3 Hardware Thermal Headroom Management
- Hardware thermal control uses sensor readings and zone trip points to adjust power or cooling, with temperatures able to change effective CPU and GPU frequencies.On Jetson platforms, frequencies are reduced as measured temperatures cross throttling thresholds.Source: §2.3 · p. 3, Figure 4 · p. 3
3 Motivation
On a Jetson AGX Orin, three experiments show that built-in throttling reacts after headroom is largely consumed, thermal zones heat and cool together, and hardware profiles trade serving performance against headroom.
Setups
- The experiments used a Jetson AGX Orin 64GB in a thermostatic chamber controlling temperatures from 48°C to 65°C.The study collected thermal-zone temperatures, CPU and GPU frequencies, power, TTFT, and TPOT as applicable.Source: §3 · p. 4
Observation 1: Built-In Thermal Throttling Is Reactive and Preset
- After 22.25 minutes at 98.56°C, GPU frequency fell 6.7%, then CPU frequency fell 3.5% 1.53 minutes later; prolonged throttling raised TTFT to 2.43× its sampling-region mean.After prolonged throttling, TPOT reached 1.11× its sampling-region mean from 45–150 minutes; frequency recovered only after the testbed was restarted.Source: §3 · p. 4
Observation 2: Shared Cooling Couples SoC Thermal Domains
- Each 1°C CPU rise accompanied 0.72°C GPU and 0.73°C SoC rises; off-reference cooling coefficients remained between 0.46 and 0.95.The authors attribute the zones’ linked heating and recovery to their shared package and cooling path, so activity in one domain changes headroom for the whole package.Source: §3 · p. 4
Observation 3: Hardware Profiles Trade Off Serving Performance and Thermal Headroom
- S3 reduced power by 35.57% and increased t_zone by 128.46%, but raised TPOT by 16.97%; S2 reduced TTFT by 10.13%.All metrics are normalized to S1, and t_zone measures time to entry into the throttling zone.Source: §3 · p. 4, Figure 6 · p. 4
- S4 and S4(H) reduced power by 43.37% and 42.59%, respectively, while increasing TTFT by 23.17% and 37.41% and TPOT by 45.26% and 45.30%.The results show that minimizing power alone does not preserve serving performance.Source: §3 · p. 5
4 Design of ThermE
ThermE coordinates workload-to-heat estimation, thermal-headroom prediction, and action scheduling for sustained LLM inference on shared-cooling edge SoCs. Its components estimate candidate-action heat and headroom before the scheduler selects hardware profiles and scheduled-token caps.
4.1 Design Overview
- ThermE selects actions pairing hardware profiles with scheduled-token caps to balance serving performance and heat generation under limited shared headroom.Its asynchronous controller evaluates candidate sequences outside the inference critical path.Source: §4.1 · p. 5
4.2 Fast LLM-to-Heat Compiler
- The Fast LLM-to-Heat Compiler converts model, request, runtime, and candidate-action information into CPU, GPU, and RAM heat inputs without executing the LLM.It separates static workload compilation, resource-pressure encoding, and frequency-aware power estimation.Source: §4.1 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 6
4.3 PDE-Constrained Headroom Predictor
- The PDE-constrained predictor uses ThermPINN offline to identify coupled thermal dynamics, then the RHP forecasts uncertainty-calibrated headroom online with low computational overhead.The stated online overhead is approximately 11.5 ms.Source: §4.3 · p. 6
4.3.1 Offline Thermal Identification and RHP Con-
- ThermPINN reconstructs temperature fields from sensor, power, and cooling histories, while POD and an operator ensemble reduce those fields to compact online state updates.The ensemble is conditioned on cooling state and uses distinct bootstrap weights to capture variation in identified dynamics.Source: §4.3.1 · pp. 6–7
4.4 Uncertainty-Aware Action Scheduler
- Beam search selects the highest-scoring feasible action sequence using predicted service metrics and thermal headroom, while excluding sequences that cross a hard temperature boundary.Only the first selected action is applied; the scheduler updates its decision after the next observation.Source: §4.4 · pp. 7–8
5 Implementation
The ThermE prototype integrates an asynchronous controller with vLLM and thermal-platform telemetry. Its implementation also trains the thermal model on multi-testbed data and runs reduced prediction and scheduling periodically.
Prototype and integration
- ThermE is implemented as an asynchronous controller atop vLLM, with serving and platform adapters publishing synchronized, timestamped snapshots.The prototype samples platform telemetry every 50 ms and constructs actions from supported hardware profiles and scheduled-token caps.Source: §5 · p. 8
ThermPINN implementation
- ThermPINN training uses 465,300 measurements spanning about 267 device hours, four cooling conditions, 20 workload classes, and three testbeds.Complete trajectories are split into training, validation, and test sets at 74.7%, 15.3%, and 10.0%.Source: §5 · p. 8
Online prediction and scheduling
- A dedicated worker runs the 1.5 MB RHP bundle every 2 seconds for a 64-second forecast and searches three control intervals; failures trigger conservative fallback.The testbed is an Nvidia Jetson AGX Orin in a temperature-controlled chamber.Source: §5 · p. 8, Figure 10 · p. 8
6 Evaluation
ThermE is evaluated as a runtime controller for sustained LLM serving under limited shared-package headroom, across controlled workloads and a sustained vehicle-trunk case. The evaluation reports serving, thermal, prediction, sensitivity, and ablation results.
6.1 Evaluation Setup
- Evaluation used four models and chat, code, summarization, and long-decode workloads on a Jetson AGX Orin; it measured serving, thermal, power, prediction, and runtime-overhead outcomes.The default setup used the Stress task, a 65°C chamber temperature, and 60% fan cooling.Source: §6 · p. 8, §6.1 · p. 9, Table 1 · p. 9
6.2 Overall Performance
- ThermE reduced TTFT by 40.55%, TPOT by 12.17%, and end-to-end latency by 13.01% relative to vLLM; its SLO violation rate was 5.70% versus 12.30% for the strongest baseline.Serving-quality results are shown in Figure 11.Source: §6.2 · p. 9, Figure 11 · p. 10
- ThermE first throttled after 29.70 s, versus 63.10 s for vLLM, while achieving 5.70% versus 31.40% SLO violations and 26.68 s versus 30.67 s end-to-end latency.Figure 12 places ThermE on the non-dominated frontier, with better serving quality but higher thermal exposure.Source: §6.2 · p. 9, Figure 12 · p. 10
- The predictor achieved 1.94°C temperature MAE and 2.82°C RMSE across thermal zones; residual calibration adjusts exposed headroom for prediction uncertainty.Figure 14 reports prediction error.Source: Figure 14 · p. 10, §6.2 · p. 10
6.3 Sensitivity Analysis
- Across Chat, Code, and Summary, ThermE reduced TPOT versus the strongest baseline by 28.56%, 5.46%, and 41.33%, respectively.For Summary, TTFT was up to 88.95% lower across baselines but remained 11.06% higher than the best baseline.Source: Table 2 · p. 10, §6.3 · p. 11
- At 512-token prompts, TTFT rose from 4.36 s at 30°C to 6.69 s at 70°C; at 30°C, increasing prompts from 128 to 1,024 tokens raised TTFT from 1.10 s to 7.55 s.TPOT also increased with ambient temperature and prompt length; longer prompts require more prefill work and expand the context accessed during decoding.Source: Figure 15 · p. 10, §6.3 · pp. 10–11
6.4 Ablation Studies
- Removing the compiler, thermal coupling, or uncertainty margin increased TTFT, TPOT, and SLO violations by 11.02–14.32%; Greedy raised TTFT and TPOT by 13.18% and 13.19%.Greedy increased SLO violations by 15.63%; the full numerical solver increased TTFT by 8.70%, TPOT by 26.68%, and SLO violations by 48.20%.Source: Table 3 · p. 10, §6.4 · p. 11
7 Related Work
Prior work optimizes inference across attention, placement, and hardware, while separate lines of work model thermal behavior and manage thermal resources. ThermE is positioned as predictive thermal control for sustained inference on shared-cooling edge SoCs.
Related inference optimization
- Inference optimizations include PagedAttention, prefill/decode overlap, heterogeneous GPU placement, mobile NPU execution, GPU/NPU coordination, and CPU/GPU/memory frequency control.ThermE complements these approaches with predictive thermal control for sustained serving under limited shared headroom.Source: §7 · p. 11
Thermal Modeling and Headroom Prediction
- Thermal-modeling work spans transient and heat-sink models, power-to-temperature mapping, neural-operator prediction, adaptive-sampling PINNs, thermal-history estimation, and accelerated wafer-scale simulation.ThermE translates uncertainty-calibrated forecasts into shared headroom for low-overhead serving decisions.Source: §7 · pp. 11–12
Thermal-Aware Resource Management
- Thermal-management systems include environment-aware frequency control, cloud placement and reconfiguration, and wafer-scale scheduling; ThermE jointly controls hardware profiles and scheduled-token caps on shared-cooling edge SoCs.Source: §7 · p. 12
8 Conclusion
The paper presents ThermE for managing shared thermal headroom during sustained LLM inference on edge SoCs. Its evaluation reports improved serving quality under thermal constraints, and the authors frame it as a foundation for onboard intelligence on compact platforms.
Conclusion
- ThermE combines LLM-to-heat compilation, physics-constrained headroom prediction, and uncertainty-aware scheduling to coordinate hardware profiles and scheduled-token caps.Source: §8 · p. 12
- The evaluation demonstrated improved serving quality under thermal constraints.Source: §8 · p. 12
- The authors describe ThermE as a foundation for sustained onboard intelligence in satellites, smart AI mobile devices, and other compact platforms with limited cooling.Source: §8 · p. 12