Source-linked AI summary
Where Does the Energy Go? Profiling LLM Agent Inference on Blackwell GPUs
Qi Luo, Kunlin Li, Ziwen Wang, Yun Chen
TL;DR
The energy use of full LLM-agent sessions remains poorly characterized despite their growing-context, tool-interleaved execution. This study profiles three workloads using GPU, CPU, DRAM, and whole-system measurements on Blackwell GPUs, finding that GPU telemetry misses a substantial share of system energy and that energy use varies across workload regimes.
Problem
Prior work has not provided fine-grained, full-stack energy attribution across complete agent sessions, whose growing context and tool execution differ from conventional inference.
Source: §1 · pp. 1–2Method
Results
GPU-only measurements omit 41–45% of system energy, while energy per output token is 63× higher for sequential agents than saturated serving in this setup.
Source: §1 · p. 2Takeaways & Limitations
Context and token management are promising targets for reducing agent energy use, and GPU telemetry alone is insufficient for full-system energy accounting.
Source: §1 · p. 2, §6 · p. 8The study evaluates one dual-GPU server and one 27B model, leaving generalization to other platforms and scales uncertain.
Source: §5 · p. 8
Abstract
from arXiv · showhide
LLM agents that iteratively reason, plan, and invoke tools create workload profiles fundamentally different from single-pass inference, yet how their energy consumption is distributed across hardware components and workload phases remains poorly understood. Characterizing these workloads therefore requires simultaneous visibility into both component-level power and phase-level execution. We conduct a full-stack energy profiling study combining NVML GPU counters, Intel RAPL CPU/DRAM counters, and Intelligent Platform Management Interface (IPMI) system-level sensors on 2x NVIDIA RTX PRO 6000 Blackwell GPUs, and profile three representative workloads with Qwen3.8-27B. For mathematical reasoning, we compare thinking-enabled and thinking-disabled modes. Our measurements reveal that GPU-only telemetry misses 41-45% of system energy across all three workloads, with non-GPU components accounting for the remainder. In our setup, the sequential agent workload consumes 63x more system energy per output token than saturated serving, reflecting the absence of batching, context growth across turns, and tool-induced idle periods. Extended thinking generates 21-75% more tokens per problem, while per-token energy differs by less than 1% between modes within each dataset, indicating that the resulting increase in energy is driven by output volume rather than a change in per-token efficiency. Continuous batching improves system-level energy efficiency by 3.2x from 1 to 16 requests per second, as GPU power plateaus while throughput continues to increase with batching depth. For the memory-bandwidth-bound reasoning workload, throughput remains unchanged under per-GPU power caps of 400-600 W but drops sharply at 300 W. These findings suggest that context-management techniques such as summarization and selective retrieval may help reduce energy consumption in agent deployments.
1 Introduction
Agent workloads combine growing, tool-interleaved trajectories with variable reasoning, but their full-session energy has been poorly characterized. This study profiles energy across hardware components and workload regimes, finding substantial non-GPU energy and large efficiency differences between sequential agents and batched serving.
Motivation
- Agent sessions accumulate context and alternate between GPU-intensive inference and CPU-bound tool execution, unlike conventional single-pass or batched inference.The paper identifies full-stack energy attribution across complete agent sessions as an evidence gap.Source: §1 · pp. 1–2
Findings
- 63× more system energy per output token is consumed by the sequential agent workload than by saturated serving in this setup.Continuous batching improves system-level energy efficiency by 3.2× from 1 to 16 requests per second; for reasoning workloads, throughput is unchanged under 400–600 W caps but drops sharply at 300 W.Source: §1 · p. 2
Full-stack profiling
- GPU-only measurements omit 41–45% of total system energy across the workloads studied.The study combines NVML, Intel RAPL, and IPMI to measure GPU, CPU, DRAM, and whole-system power.Source: §1 · p. 2
Thinking modes
- Extended thinking generates 21–75% more tokens per problem, while per-token energy differs by less than 1% between modes within each dataset.The authors attribute the additional energy in their experiments to output-token volume rather than a change in per-token efficiency.Source: §1 · p. 2
2 Related Work
Prior energy studies measure different system boundaries and commonly use predetermined requests or replayed traces. This study profiles tool-interleaved agent trajectories on Blackwell GPUs with GPU-level and whole-system measurements, alongside a continuous-serving baseline.
Measurement scope
- Prior energy studies characterize accelerator, component, or request-level energy, but do not capture complete tool-interleaved trajectories or energy between an agent’s model calls.The review notes studies using GPU telemetry, software estimates, RAPL, and request-level attribution.Source: §2 · p. 2
Serving systems and reasoning workloads
- Serving-system and reasoning studies use controlled queries or replayed traces rather than tool-interleaved workloads where tool output determines the next model request.The reviewed serving work includes DynamoLLM [Stojkovic et al., 2025] and DistServe [Zhong et al., 2024]; reasoning work includes Jin et al. [2025].Source: §2 · p. 3
Agent workloads
- AgentStop [Pham et al., 2026] measures CPU and GPU energy on an Apple M1 Max, while this study profiles complete agent trajectories on Blackwell GPUs with GPU-level and whole-system measurements.The authors state that prior work had not evaluated tool-interleaved trajectories with both measurement scopes alongside continuous serving on the same platform.Source: §2 · p. 3
3 Methodology
The methodology profiles three workload types using a dual-GPU Blackwell server and measures component and whole-system energy. It also specifies the workload configurations and energy metrics used for evaluation.
3.1 Platform and Serving
- The experiments use two NVIDIA RTX PRO 6000 Blackwell GPUs, two Intel Xeon Platinum 8558 CPUs, and 256 GB DDR5 ECC memory.Qwen3.8-27B is served via vLLM with tensor parallelism across both GPUs, BF16 precision, chunked prefill, and a 131k-token maximum context length.Source: §3.1 · p. 3
3.2 Energy Profiling
- NVML measures per-GPU energy, RAPL measures CPU package and DRAM energy, and IPMI reports component and total-system power.The measurements are collected within a common timing loop; RAPL CPU package power and IPMI CPU rail readings differ by 0.1–0.6% across evaluated workloads.Source: §3.2 · p. 4
- The residual Pothers is defined as system power minus GPU, CPU, and DRAM power, and includes estimated and unmeasured component contributions.These include fans, VRM conversion losses, PSU overhead, the baseboard, SSDs, and the BMC.Source: §3.2 · p. 4
3.3 Workloads
- The study evaluates SWE-bench Verified agentic coding, AIME 2022–2024 and MATH-500 reasoning, and ShareGPT continuous serving.Mathematical problems are tested with thinking enabled and disabled; ShareGPT prompts arrive at 1–16 requests per second under continuous batching.Source: §3.3 · p. 4
3.4 Metrics
- Reported metrics are energy per token, component- and system-level power, and energy per task; energy metrics use total system energy unless marked GPU-only.Energy per token is total system energy divided by output tokens; energy per task is total system energy to complete one problem or request.Source: §3.4 · p. 4
4 Experiments
The experiments profile agentic coding, mathematical reasoning, and continuous serving, compare their energy profiles, and evaluate GPU power capping.
4.1 Agentic Coding
- GPU-only telemetry misses approximately 44% of system energy during SWE-bench agentic coding.Shell commands account for 46.7% of tool calls and often leave GPUs largely idle while CPUs stay active; per-turn energy and duration rise as context accumulates.Source: §4.1 · pp. 5–6, Figure 2 · p. 5, Figure 4 · p. 6
4.2 Mathematical Reasoning
- Per-token energy differs by less than 0.4% between thinking modes, while thinking-enabled runs generate 75% more tokens per AIME problem and 21% more per MATH-500 problem.The paper attributes the corresponding increase in total energy per problem primarily to greater output volume.Source: §4.2 · p. 6, Table 1 · p. 6
4.3 Continuous Serving
- System energy per token falls from 2,955 to 920 mJ/tok as serving increases from 1 to 16 requests per second, a 3.2× efficiency improvement.GPU power largely plateaus by 4 requests per second while throughput continues to increase at higher rates.Source: §4.3 · pp. 6–7, Table 2 · p. 7, Figure 5 · p. 7
4.4 Cross-Workload Comparison
- 58,383 mJ per output token for the agent workload is approximately 63× saturated serving at 16 requests per second (920 mJ/tok); GPU-to-system power ratio stays 55–59% across workloads.The paper attributes the gap to lack of batching, context growth, and tool-interleaved execution.Source: §4.4 · p. 7
4.5 Power Capping
- At 400–600 W, reasoning throughput remains approximately 48 tokens per second, but 300 W cuts it by 3.1×; at serving 4 requests per second, 300 W lowers energy to 1,167 vs. 1,215 mJ/tok at 400 W.For reasoning, the 300 W cap also raises system energy per token to 2.5× its value at 400 W.Source: §4.5 · pp. 7–8, Figure 6 · p. 8
5 Discussion
Energy use varies with output volume, context growth, batching, and power limits, while GPU telemetry alone misses a substantial share of system energy. The findings are bounded by measurements from one dual-GPU server and one model.
5 Discussion
- 63× more system energy per output token is used by the sequential agent workload than saturated serving in this setup.Within agent trajectories, per-turn energy tracks execution duration as context grows.Source: §5 · p. 8
- 21–75% more output tokens per problem accompany thinking-enabled modes, while per-token energy differs by less than 1% within each dataset.The added energy therefore mainly reflects token volume; early stopping and budget-aware decoding may reduce this overhead, subject to accuracy tradeoffs.Source: §5 · p. 8
- 41–45% of total system energy is unreported by GPU telemetry in this setup.The passage recommends full-system measurements or platform-specific corrections for carbon estimates.Source: §5 · p. 8
- 3.2× higher energy efficiency is achieved with continuous batching from 1 to 16 requests per second as GPU power largely plateaus.For reasoning, 500 W and 600 W caps are non-binding, 400 W reduces GPU power without affecting throughput, and 300 W causes a sharp performance drop.Source: §5 · p. 8
- The study covers one dual-GPU server and one 27B model, leaving generalization to other platforms and scales uncertain.There are no external PDU measurements for AC validation, and the “Others” category includes estimated contributions.Source: §5 · p. 8
6 Conclusion
The conclusion summarizes a full-stack energy profiling study across agentic coding, mathematical reasoning, and continuous serving on Blackwell GPUs. It reports that energy use varies with workload features and that GPU telemetry alone is insufficient for full-system accounting.
6 Conclusion
- GPU telemetry alone is insufficient for full-system energy accounting across the profiled workload types.The study profiles agentic coding, mathematical reasoning, and continuous serving on Blackwell GPUs.Source: §6 · p. 8
- Context growth and output-token volume shape energy use across the profiled workload types.Source: §6 · p. 8
- Batching and power limits also shape energy use across the profiled workload types.Source: §6 · p. 8