Source-linked AI summary
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas, Ricardo Bianchini
TL;DR
Cloud LLM inference creates thermal and power-management challenges because workloads have distinct phases, configuration trade-offs, and differing SaaS/IaaS flexibility. TAPAS coordinates placement, routing, and SaaS reconfiguration using historical infrastructure data; in large-scale evaluation, it reduced temperature, row power, and throttling while preserving inference performance. The framework also supports oversubscription and cooling or power failures within its evaluated scope.
Problem
Cloud LLM inference challenges thermal and power management because its phases and configurations have distinct performance, thermal, power, and quality characteristics, while SaaS and IaaS workloads differ in flexibility.
Method
TAPAS uses historical temperature, power, and load data for thermal- and power-aware VM placement, request routing, and SaaS instance reconfiguration.
Results
TAPAS maintains P99 inference tail latency while reducing maximum temperature by 17%, peak row power by 23%, and thermal and power throttling events by 97% and 99%, respectively.
Takeaways & Limitations
TAPAS supports cooling and power oversubscription, up to 40% additional capacity, and operation during cooling or power failures while maintaining performance and quality.
Takeaways & Limitations
The evaluation focuses on air cooling, although many insights may apply to other cooling technologies, and targets clouds co-locating SaaS and IaaS GPU workloads.
Abstract
from arXiv · showhide
The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques often are inadequate for LLM inference due to the fine-grained, millisecond-scale execution phases, each with distinct performance, thermal, and power profiles. Additionally, LLM inference workloads are sensitive to various configuration parameters (e.g., model parallelism, size, and quantization) that involve trade-offs between performance, temperature, power, and output quality. Moreover, clouds often co-locate SaaS and IaaS workloads, each with different levels of visibility and flexibility. We propose TAPAS, a thermal- and power-aware framework designed for LLM inference clusters in the cloud. TAPAS enhances cooling and power oversubscription capabilities, reducing the total cost of ownership (TCO) while effectively handling emergencies (e.g., cooling and power failures). The system leverages historical temperature and power data, along with the adaptability of SaaS workloads, to: (1) efficiently place new GPU workload VMs within cooling and power constraints, (2) route LLM inference requests across SaaS VMs, and (3) reconfigure SaaS VMs to manage load spikes and emergency situations. Our evaluation on a large GPU cluster demonstrates significant reductions in thermal and power throttling events, boosting system efficiency.
1 Introduction
TAPAS addresses the thermal and power challenges of cloud LLM inference by coordinating workload placement, request routing, and SaaS reconfiguration. In large-scale production-trace simulations, it substantially reduced throttling while preserving inference performance and enabling greater capacity.
- Motivation: LLM inference stresses datacenter cooling and power because modern GPUs consume substantial power and share constrained infrastructure.The paper notes that A100 and H100 GPUs have TDPs of 6.5 kW and 10.2 kW, respectively.
- Motivation: Traditional thermal and power-management techniques are sub-optimal for LLM inference workloads with distinct execution phases and configuration trade-offs.Clouds also co-locate provider-controlled SaaS VMs with opaque IaaS VMs, limiting which workloads can be reconfigured.
- TAPAS: TAPAS combines thermal- and power-aware GPU VM placement, request routing, and SaaS instance reconfiguration.It uses historical temperature, power, and service-load data to respond to load spikes and cooling or power failures.
- Results: TAPAS maintains P99 inference tail latency while reducing maximum temperature by 17% and peak row power by 23%.These reductions create opportunities for up to 40% additional capacity and lower datacenter TCO.
- Results: TAPAS reduces thermal and power throttling events by 97% and 99%, respectively, compared with other practical policies.The evaluation uses traces from hundreds of production racks across a subset of datacenters and also examines cooling and power failures.
2 Characterizing Challenges in Thermal and Power Infrastructure for GPUs
GPU datacenters exhibit substantial spatial and temporal heterogeneity in cooling and power demand, making infrastructure-aware management essential. Measurements show temperature variation across locations, workloads, and GPUs, while row-level power imbalance creates opportunities and risks for oversubscription.
- Cooling: Temperature varies by up to 1°C across rows and 2°C across racks within a row, while rack height has a minor impact.The observed spatial differences reflect airflow and construction patterns that are difficult to estimate without empirical measurement or expensive simulation.
- Cooling: GPU temperature is sensitive to both inlet temperature and GPU load, and the regression model predicts it with mean absolute error below 1°C.The model also captures inlet-temperature increases caused by power leakage.
- Cooling: GPU temperatures can differ by up to 10°C within one server despite identical inlet temperatures and utilization, due to layout and manufacturing variation.This heterogeneity complicates thermal management because individual GPUs may approach thresholds at different rates.
- Cooling: Thermal modeling must account for temporal variation from outside temperature and load, spatial variation from datacenter and server layouts, and airflow requirements.The paper introduces datacenter-scale models for inlet and GPU temperatures to capture these effects.
- Power: Power demand is heavily imbalanced across rows: 50%, 75%, and 90% of rows draw 28%, 18%, and 10% less P99 power than the most power-hungry row.High-power rows create hotspots and force provisioning that can leave capacity in lower-demand rows unused.
- Power: GPU clusters offer strong power-oversubscription potential, but safe oversubscription requires managing tail-end rows that generate hotspots.The observed row imbalance makes hotspot control central to realizing this opportunity.
3 Characterizing Opportunities in Thermal and Power Properties of GPU Workloads
The characterization identifies workload placement, predictability, SaaS flexibility, and LLM configuration as opportunities for managing thermal and power behavior. These observations motivate thermal- and power-aware scheduling for cloud GPU workloads.
- 80 VMs across two rows produced maximum temperatures above 85°C under worst-case placement, versus around 72°C for typical placement.
- More than 60% of GPU VMs run longer than two weeks, while VM and row power workloads exhibit periodic diurnal patterns.
- Historical power templates predict most row hours with less than 10% error, while customer information predicts over 75% of VM hours with errors below 10%.
- SaaS workloads provide thermal and power flexibility because cloud providers manage their VMs and can route requests across endpoint instances.
- LLM inference separates into prefill and decode phases, and configuration parameters trade off temperature, power, performance, and result quality.
4 TAPAS Design
TAPAS combines thermal- and power-aware VM placement, request routing, and SaaS instance reconfiguration for cloud LLM clusters. It uses historical profiles and workload-specific controls to manage normal operation, load spikes, oversubscription, and infrastructure failures.
- TAPAS targets cooling and power oversubscription while minimizing IaaS impact and maintaining SaaS performance and accuracy.
- The architecture extends VM allocation and load balancing, adds a per-SaaS VM Instance Configurator, and maintains multiple workload profiles.
- VM placement: The VM Allocator filters aisles and rows by peak airflow and power, then prefers cooler servers for IaaS workloads and warmer servers for SaaS workloads.
- Request routing: The Load Balancer routes requests using VM load together with aisle airflow, row power, and server temperature constraints.
- Instance configuration: The Instance Configurator selects GPU frequency, batch size, model parallelism, quantization, and model size to maximize goodput without quality impact under calculated limits.
- Oversubscription and failures: During failures, TAPAS recalculates available airflow, row power, and server inlet temperature, redirects requests, reduces instance load, and finally applies IaaS power capping if needed.
5 Evaluation
TAPAS is evaluated through real-cluster experiments, large-scale simulation, ablations, workload-mix sensitivity, oversubscription, and infrastructure-failure scenarios. Across these settings, it reduces thermal and power peaks while preserving quality and service objectives, with selective quality trade-offs during failures.
- Evaluation setup: The evaluation combines a scaled real-cluster experiment, a discrete-time simulator, and production-trace workloads spanning IaaS and SaaS inference.The real cluster emulates two rows of 80 servers, while the simulator models datacenter load, inference execution, cooling, power, and capping events.
- Real-cluster evaluation: 20% reduction in peak utilization compared to Baseline was achieved in the real-cluster experiment while maintaining latency SLOs and result quality.The experiment used two rows of 80 servers measured at one-minute intervals.
- Real-cluster evaluation: 4% absolute error compared to the simulation validated the accuracy of the simulator.This comparison was reported for the real-cluster experiment.
- Large-scale simulation: 15% reduction in maximum temperature and 24% reduction in peak power were achieved over one week of large-scale simulation without hurting result quality.The comparison was against Baseline using five-minute intervals.
- Ablation and workload mix: For a 50/50 IaaS/SaaS mix, each individual TAPAS policy reduced both temperature and power by up to 12% compared to Baseline without quality or normal-condition SLO violations.The policies evaluated VM placement, request routing, and instance configuration.
- Ablation and workload mix: TAPAS achieved maximum reductions of 23% in temperature and 28% in power for an entirely SaaS workload, while its effectiveness was limited to VM placement for entirely IaaS workloads.The reported flexibility came from the adaptability of SaaS workloads.
- Emergencies: During thermal or power failures, TAPAS managed reduced capacity through selective actions, including routing requests to smaller models, with up to 12% quality impact.The failures reduced capacity limits to 90% for cooling and 75% for power.
6 Related work
Related work addresses datacenter cooling, power utilization, and LLM serving through adaptive infrastructure control, power prediction, placement, scheduling, and serving optimizations. TAPAS focuses these concerns on power utilization in LLM inference clusters.
- Datacenter cooling management: Prior cooling systems target thermal hotspots and cooling efficiency using adaptive control and technologies such as warm-water, immersion, and free cooling.The cited systems also include optimization for under-provisioned cooling.
- Datacenter power management: Prior power-management systems use offline placement, load shedding, power prediction, workload-aware overclocking, or service spreading to improve datacenter power utilization.These approaches include Flex, SmartOClock, and SmoothOperator.
- LLM serving: LLM-serving research addresses power oversubscription, model parallelism, predictive scheduling, cache management, batching, scheduling, and autoscaling, while TAPAS focuses on power utilization in LLM inference clusters.The cited systems include POLCA and μ-Serve, alongside orthogonal serving optimizations.
7 Conclusions
TAPAS provides thermal- and power-aware scheduling for LLM inference in GPU datacenters through placement, routing, and instance configuration. It reduces thermal and power peaks, supports oversubscription and failures, and maintains performance and quality with minimal quality impact.
- Contribution: TAPAS combines VM placement, request routing, and instance configuration for thermal- and power-aware scheduling of LLM inference.The system is designed for GPU datacenters.
- Conclusion: TAPAS reduces thermal and power peaks while maintaining performance and quality with minimal quality impact.This conclusion summarizes the system’s reported efficiency and quality objectives.
- Conclusion: TAPAS supports cooling and power oversubscription and handles thermal and power failures in LLM inference clusters.These capabilities are part of the system’s stated contribution.