PaperlayerView exact revision on arXiv

Source-linked AI summary

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, Esha Choukse

arXiv:2408.00741v1cs.AIcs.ARcs.DC

TL;DR

LLM inference clusters consume substantial energy while meeting demanding performance SLOs, but energy management remains underexplored. DynamoLLM dynamically selects energy-efficient configurations, reducing energy by 53%, operational carbon emissions by 38%, and customer cost by 61% while meeting latency SLOs.

  • Problem

    Energy consumption in LLM inference clusters remains underexplored despite power-hungry GPUs and the need to maintain high performance while reducing carbon emissions.

  • Method

    DynamoLLM uses energy-performance profiles to dynamically select configurations by scaling instances, model parallelism, and GPU frequency under performance SLOs.

  • Results

    53% less energy, 38% lower operational carbon emissions, and 61% lower customer cost were achieved while meeting latency SLOs.

  • Takeaways & Limitations

    DynamoLLM demonstrates that dynamically configuring LLM inference clusters can reduce energy, carbon emissions, and customer cost while preserving performance guarantees.

Abstract

from arXiv · show

The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.

I. INTRODUCTION

DynamoLLM is an energy-management framework that dynamically selects efficient LLM inference configurations using workload heterogeneity and fluctuations while meeting performance SLOs. On production-level traces, it reduces energy, operational carbon emissions, and customer cost.

  • I. INTRODUCTION: LLM inference energy management is challenging because request lengths, model compute properties, and service SLOs create heterogeneous, fluctuating workloads.These variations produce a dynamic environment unlike traditional datacenter energy-management settings.
  • I. INTRODUCTION: DynamoLLM automatically and dynamically selects energy-efficient configurations from energy-performance profiles while meeting performance SLOs.The framework exploits workload properties and multiple configuration knobs to reduce inference energy consumption.
  • I. INTRODUCTION: DynamoLLM maintains differently configured instance pools for request types and dynamically resizes them as request distributions change.Requests with fewer input and output tokens can use fewer GPUs at lower frequency, while input-heavy requests are more frequency-sensitive.
  • I. INTRODUCTION: A hierarchical controller structure manages configuration knobs with lower complexity and fewer centralized bottlenecks, while transition techniques reduce or hide reconfiguration overheads.Controllers operate under upper-level constraints and forward further constraints downward to coordinate resource management.
  • I. INTRODUCTION: 53% energy, 38% operational carbon emissions, and 61% customer cost reductions were achieved while meeting latency SLOs.The evaluation used a large GPU cluster running production-level traces from a major cloud provider.

II. BACKGROUND · III. OPPORTUNITIES FOR ENERGY EFFICIENCY · A. Heterogeneous Energy-Performance Profiles

LLM inference has distinct prefill and decode phases, creating heterogeneous energy-performance profiles across request lengths, loads, models, and service SLOs. DynamoLLM exploits this heterogeneity by selecting request-specific and workload-adaptive configurations while predicting unknown output lengths.

  • II. BACKGROUND: Prefill processes input tokens in parallel, whereas decode serially generates output tokens, producing distinct compute behaviors that affect inference performance and energy.Prefill scales with input-token count, while decode generates output tokens sequentially; tensor parallelism aggregates communication across GPUs and is considered across 2, 4, and 8 GPUs.
  • A. Heterogeneous Energy-Performance Profiles: Request types spanning short, medium, and long input/output lengths exhibit different TTFT/TBT requirements and energy-performance characteristics.Requests are categorized into nine input/output-length buckets, including SS, MM, and LL, to capture this heterogeneity.
  • A. Heterogeneous Energy-Performance Profiles: The least-energy configuration is TP2 at 1.2 GHz for SS requests, whereas LL requests require TP8 and minimize energy at 1.6 GHz.The lowest-power feasible configuration is not always energy-optimal because longer execution time can increase total energy; combining all requests would follow the most constrained SLO.
  • A. Heterogeneous Energy-Performance Profiles: Energy-efficient request routing requires separating requests by input/output lengths and assigning different server configurations, but output length must be predicted because it is unknown at arrival.DynamoLLM relies on prior output-length prediction methods,, and includes a mechanism to mitigate occasional mispredictions.
  • A. Heterogeneous Energy-Performance Profiles: 650 TPS low load permits TP4 at 1.2 GHz as the lowest-energy configuration, while 4K TPS high load requires TP4 or TP8 at 2 GHz.Low load provides SLO slack for low-frequency operation, whereas high load reduces slack and requires higher-frequency configurations.
  • A. Heterogeneous Energy-Performance Profiles: Compute-bound large models are more sensitive to frequency and parallelism and often require high-energy modes, whereas sparse smaller models tolerate lower-performance configurations.The requested model therefore directly influences its energy profile and feasible operating point.

B. Dynamic LLM Inference Workloads

LLM inference workloads vary substantially in request composition and load over time, making static energy-optimal configurations quickly suboptimal. Effective energy management therefore requires dynamically adapting per-request-type resource pools while balancing tuning precision against fragmentation.

  • Changing Request-Length Distribution: Request-length distributions differ between Coding and Conversation, with Conversation typically having longer outputs and shorter inputs than Coding.Both services contain a significant fraction of each request type, and request-type popularity changes over time.
  • Configuration Implications: Different request types require different energy-optimal configurations, so resources should be split into individually configured pools that adapt to the current request distribution.Classifying requests into too few classes limits energy fine-tuning, whereas too many classes can cause fragmentation and hurt efficiency.
  • Workload Load Variation: Conversation peaks at 1.7× its average load and 3.3× its valley load, while Coding peaks at 2.8× and 34.6×, creating slack for energy-saving operation.Servers can use less performant, energy-optimized configurations during low-load periods and switch to more performant modes as load builds, while maintaining SLOs.
  • Key Insight: Highly dynamic workloads can quickly make an energy-optimal configuration suboptimal, necessitating automatic and user-transparent configuration selection.The large configuration search space makes manual or static selection impractical.

C. Reconfiguration Overheads

DynamoLLM must quickly transition among configurations as LLM workloads change, but reconfiguring instance count, model parallelism, and GPU frequency incurs overheads. GPU-frequency adjustments alone can substantially increase inference-step latency.

  • C. Reconfiguration Overheads: Workload adaptation requires reconfiguring the number of inference server instances, model parallelism, and GPU frequency.The framework must transition quickly between configurations to capture fast changes in LLM inference workloads.
  • C. Reconfiguration Overheads: GPU-frequency adjustments take around 50–80 ms, while one LLM decode iteration takes 20–30 ms.The adjustment overhead can therefore significantly affect inference performance.
  • C. Reconfiguration Overheads: Resetting GPU frequency on every inference iteration can potentially double the latency of individual inference steps.Figure 3 compares throughput with a constant 1980 MHz frequency against background frequency resets to 1980 MHz.

IV. DYNAMOLLM: AN ENERGY MANAGEMENT FRAMEWORK FOR LLM INFERENCE CLUSTERS

DynamoLLM is an energy-management framework for LLM inference environments that integrates with existing inference platforms while meeting performance SLOs. It dynamically manages pools, model instances, parallelization, request scheduling, and GPU frequency to reduce energy consumption.

  • DynamoLLM is the first energy-management framework for LLM inference environments, integrating with existing inference platforms to operate workloads energy-efficiently and cost-effectively while meeting performance SLOs.
  • DynamoLLM uses a hierarchy of controllers with cluster resources split into per-request-type pools.
  • The Pool Manager dynamically adjusts pool and model-instance counts, schedules requests for lower per-pool energy consumption, and re-shards instances into more energy-efficient parallelizations.
  • The Instance Manager schedules requests to the inference engine and adjusts each instance’s GPU frequency when needed.

A. Configuring Instances for Energy-Efficiency

DynamoLLM profiles each deployed model across request lengths, model parallelisms, and GPU frequencies, then selects energy-minimizing configurations that satisfy resource, load, and performance constraints. The exact solver guarantees optimality but incurs roughly 100s of milliseconds of overhead, motivating hierarchical approximation for fine-grained decisions.

  • Generating LLM profiles: The system profiles models using varied request lengths, tensor parallelisms TP2, TP4, and TP8, and GPU frequencies from 800–1980MHz in 200MHz steps.It profiles several load levels up to maximum throughput and extrapolates the resulting energy-performance profile.
  • Generating LLM profiles: Profiles can be reused across services and cached locally, reducing repeated profiling overhead when multiple services share a model.DynamoLLM stores profiles in a global repository and caches them in cluster-local storage upon service deployment.
  • Selecting the energy-optimized configuration: DynamoLLM selects the energy-optimal configuration by minimizing total energy while satisfying GPU-budget, load-assignment, and performance-SLO constraints.The solver limits total GPUs to N, distributes expected load L across instances, and requires aggregate expected performance to meet the SLO.
  • Selecting the energy-optimized configuration: The exact optimization guarantees energy optimality but incurs roughly 100s of milliseconds of solver overhead, preventing configuration decisions every few seconds.DynamoLLM therefore introduces hierarchical subtasks and an approximation heuristic to reduce computation complexity.

B. Hierarchical Control for Dynamic Load · C. Reduced Overheads for Smooth Reconfiguration

DynamoLLM uses hierarchical, locally optimized controllers operating at different timescales to adapt cluster resources while accounting for reconfiguration costs and downtime. It reduces reconfiguration overhead through cached model weights, efficient re-sharding, continued service during transitions, and faster GPU-frequency changes.

  • B. Hierarchical Control for Dynamic Load: DynamoLLM decomposes optimization across hierarchical controllers, assigning each knob a locally optimal value under upper-level constraints while operating at different timescales.This avoids globally searching the configuration space and assumes lower-level controllers use their highest-performance configuration.
  • B. Hierarchical Control for Dynamic Load: Every epoch, the cluster manager predicts each request type’s load and selects the minimal number of highest-performance instances needed to support its peak.Consolidating work into fewer nodes reduces user cost and idle energy on lightly loaded GPUs.
  • B. Hierarchical Control for Dynamic Load: DynamoLLM mitigates fragmentation by assigning one fewer instance to a pool and forwarding leftover load to the next larger request-type pool.This prevents accumulated overprovisioning across pools from reducing energy efficiency.
  • B. Hierarchical Control for Dynamic Load: Controllers use an Overhead Table to compare energy savings against reconfiguration costs and downtime before changing scale, sharding, or frequency.Reconfiguration is staggered across instances, prioritizing those with higher potential energy savings while others continue serving workloads.
  • C. Reduced Overheads for Smooth Reconfiguration: DynamoLLM reduces server scale-in/out overhead by caching model weights locally within the cluster and applying additional instance-creation strategies.The supplied passage introduces these strategies but does not provide their complete list.
  • C. Reduced Overheads for Smooth Reconfiguration: Re-sharding parallelism transfers only the weights needed by the target layout: TP4→TP2 takes around 100ms because parallel transfers move 1/4 of the model weights.The transfer overhead is defined using T as the time to move 1/8 of the model, with T = 50ms in the stated setup.
  • C. Reduced Overheads for Smooth Reconfiguration: During TP4→TP8 transitions, the old instance can preserve throughput while sending weights, whereas TP8→TP4 transitions require operating at lower throughput.DynamoLLM also lets the old instance serve requests while the new instance synchronizes, avoiding downtime during synchronization.
  • C. Reduced Overheads for Smooth Reconfiguration: GPU-frequency changes are accelerated by keeping nvidia-smi loaded in memory and running the controller in privileged mode to avoid OS-user interaction overhead.These techniques support rapid frequency adjustments.

D. Predictive Scheduling for Request Heterogeneity

DynamoLLM predicts each request’s output length to route it among heterogeneous instance pools, while emergency handling escalates from deadline-aware queue reordering to GPU-frequency increases, request re-steering, and squashing overdue requests.

  • Request heterogeneity mapping: Output-length prediction classifies requests as short, medium, or long and routes them to the corresponding instance pool using predicted output and known input length.The predictor serves as a proxy model, while the cluster manager forwards requests to the pool manager responsible for each request type.
  • Emergency handling: When queues build because arrivals exceed processing, instance managers prioritize requests nearing deadline before applying stronger emergency actions.This escalation begins with reordering queued requests, then raises GPU frequency if deadlines remain at risk.
  • Emergency handling: If backlog persists, the system re-steers unstarted requests to another instance within the pool and eventually squashes requests waiting beyond a threshold, prompting users to retry.These actions escalate only after queue reordering and maximum GPU-frequency increases prove insufficient.

E. DynamoLLM Implementation · V. EVALUATION · A. Evaluation Setup

DynamoLLM is implemented as a modular, lightweight control system integrated with LLM inference platforms and evaluated under production-level workloads across multiple baselines and configurations.

  • E. DynamoLLM Implementation: DynamoLLM is built on vLLM but can integrate with TensorRT-LLM without modifications.This demonstrates platform-level modularity in the implementation.
  • E. DynamoLLM Implementation: Lightweight gRPC controllers coordinate management while keeping memory and compute requirements low.Cluster and pool managers run in a dedicated VM, while instance managers are co-located with inference engines to reduce communication overhead.
  • A. Evaluation Setup: Experiments use servers equipped with 8 H100 GPUs and report Llama2-70B [67] results, with Mixtral, Falcon, and BLOOM [59] showing the same trends.The reported evaluation therefore centers on Llama2-70B while covering additional model families qualitatively.
  • A. Evaluation Setup: The workload uses production-level traces consisting of 1-hour open-source traces plus 1-day and 1-week Coding and Conversation traces from the fleet.These traces provide both open-source and fleet-derived workload durations for evaluation.
  • A. Evaluation Setup: DynamoLLM is compared with five systems, including SinglePool, a state-of-practice baseline that routes requests to a common TP8 instance pool.The provided setup describes SinglePool as a baseline using tensor parallelism across eight GPUs.
  • A. Evaluation Setup: Evaluation summarizes latency and power for systems running Llama2-70B [67] on 1-hour open-source production traces [50].Figure 7 reports latency summaries, while Figure 8 reports power summaries for the same workload and model.
  • A. Evaluation Setup: The comparison varies instance count, model parallelism, and GPU frequency to match configurations to current workload demand.These controls correspond respectively to scaling pool instances, adjusting parallelism, and changing GPU frequency.

B. Cluster-Level Experiments · C. Sensitivity Studies

Cluster-level experiments evaluate DynamoLLM on production traces against fixed-resource baselines, showing lower power consumption while adapting resources to load. Sensitivity studies find that moderate prediction errors have limited impact on energy and performance because DynamoLLM detects mispredictions and reconfigures its controls.

  • B. Cluster-Level Experiments: DynamoLLM scales the number of GPU servers with current load, whereas baselines provision 12 H100 servers for peak demand on one-hour Conversation-service production traces.
  • B. Cluster-Level Experiments: MultiPool increases energy consumption by 20% over SinglePool because it allocates more resources while always operating them.
  • B. Cluster-Level Experiments: DynamoLLM varies GPU frequency and the number of GPUs across TP2, TP4, and TP8 sharding configurations over time while serving Llama2-70B on production traces.
  • B. Cluster-Level Experiments: 43% and 9% lower P50 and P99 power consumption, respectively, are achieved by DynamoLLM over the baseline through energy-efficient operating modes.
  • B. Cluster-Level Experiments: MultiPool lowers P99 TTFT and TBT latencies by 5.3% and 11.1% but raises P50 TTFT and TBT latencies by 11.4% and 7.6% over SinglePool.
  • C. Sensitivity Studies: Sensitivity analysis evaluates how prediction-model accuracy affects system efficiency by introducing bounded output-length misclassification errors under medium load.
  • C. Sensitivity Studies: Energy is also evaluated across different load levels for the Llama2-70B model.
  • C. Sensitivity Studies: 13% higher energy consumption and 7.3% higher TTFT result from a 40% prediction error compared with no error, indicating a modest impact on energy and performance.DynamoLLM promptly detects mispredictions and reconfigures its control knobs, explaining its robustness.

D. Long Cluster-Level Experiments · E. Large-Scale Simulations · F. Cost and Carbon Emission

Across long-term experiments and large-scale simulations, DynamoLLM reduces energy consumption under both peak and low-utilization conditions and across Conversation and Coding services. These energy savings also reduce GPU, energy, and operational carbon costs.

  • D. Long Cluster-Level Experiments: The long-cluster experiment uses one-day Conversation traces covering all invocations of a subset of service instances during a typical work day.The experiment runs for 24 hours on 11 H100 servers.
  • D. Long Cluster-Level Experiments: During 24-hour Conversation-service experiments on 11 H100 servers, DynamoLLM reduces energy relative to SinglePool during peak hours and low-utilization periods.The experiment scales instance counts with load, while SinglePool uses a fixed pool configuration.
  • E. Large-Scale Simulations: DynamoLLM significantly reduces normalized energy consumption for both Conversation and Coding services in week-long production-trace simulations.The simulator evaluates five systems using one-week traces; DynamoLLM operates in more energy-efficient modes for Conversation because its inputs are typically shorter.
  • E. Large-Scale Simulations: The large-scale study uses a discrete-time simulator to model energy consumption for different systems from production traces.Figure 14 reports normalized energy consumption for week-long Conversation and Coding traces.
  • F. Cost and Carbon Emission: 38.5% cost reduction: DynamoLLM lowers average GPU servers from 40 to 24.6 during week-long experiments, saving $1362.7/hour at current GPU VM pricing.DynamoLLM minimizes GPU count while optimizing energy efficiency.
  • F. Cost and Carbon Emission: 56%: DynamoLLM reduces energy costs by up to this amount, translating to $4.4/hour savings because energy costs are substantially lower than GPU costs.
  • F. Cost and Carbon Emission: DynamoLLM maps week-long carbon-intensity traces from multiple grids onto energy consumption over time to compare operational emissions with SinglePool.The comparison uses Conversation traces and includes the CAISO grid.

VI. RELATED WORK · VII. CONCLUSION

Prior work improves SLO-constrained resource efficiency through sharing, allocation, hardware reconfiguration, power management, and efficient LLM-serving techniques. DynamoLLM instead targets energy-optimal LLM inference-cluster configuration under performance guarantees, reducing energy, carbon emissions, and customer cost.

  • VI. RELATED WORK: Earlier work improves SLO-constrained efficiency for latency-sensitive workloads through resource sharing,,, dynamic allocation, and hardware reconfiguration.
  • VI. RELATED WORK: Other studies enable safe power management and oversubscription,, by leveraging workload characteristics,.
  • VI. RELATED WORK: Recent LLM-serving work addresses heterogeneous resources, platforms, memory and key-value caches,, and node- or cluster-level scheduling,,,,,,.
  • VI. RELATED WORK: Unlike studies focused on throughput or latency, DynamoLLM shows that LLM-inference energy optimization has distinct performance, energy-consumption, and overhead trade-offs.
  • VII. CONCLUSION: DynamoLLM reduces energy by 53%, carbon emissions by 38%, and customer cost by 61% while operating under performance guarantees.It automatically and dynamically configures cluster organization, including instance count, model parallelism, and GPU frequency.
  • VII. CONCLUSION: DynamoLLM exploits heterogeneity in inference compute properties and fluctuations in inference workloads to save energy.
  • VII. CONCLUSION: The framework automatically selects an energy-optimal cluster organization using instance count, model parallelism, and GPU frequency.
Loading 2408.00741v1…