PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective

Jiin Kim, Byeongjun Shin, Jinha Chung, Minsoo Rhu

arXiv:2506.04301v2cs.LGcs.AR

TL;DR

AI agents extend LLM inference with dynamic, tool-using workflows, creating system-level cost and sustainability concerns beyond static single-turn inference. This paper quantitatively characterizes those costs across agent designs and workloads, finding substantial overheads, latency penalties, and sharply diminishing accuracy returns from test-time scaling.

  • Problem

    System-level deployment costs of dynamic, iterative AI-agent reasoning remain underexplored despite its compute, latency, memory, and infrastructure demands.

    Source: C. Test-Time Scaling in AI Agents · p. 3, D. Motivation · p. 3

  • Method

    The paper quantitatively evaluates resource usage, serving behavior, and infrastructure costs across diverse agent workflows, tasks, and test-time scaling designs.

    Source: §I · p. 2, §V · p. 8

  • Results

    AI agents incur substantial energy overheads, latency penalties, and infrastructure costs, while test-time scaling yields sharply diminishing accuracy returns.

    Source: §IX · p. 13

  • Takeaways & Limitations

    Agent systems should use compute-aware reasoning and workflow designs that optimize accuracy per unit cost under deployment constraints.

    Source: §IX · p. 13

Abstract

from arXiv · show

Large-language-model (LLM)-based AI agents have recently showcased impressive versatility by employing dynamic reasoning, an adaptive, multi-step process that coordinates with external tools. This shift from static, single-turn inference to agentic, multi-turn workflows broadens task generalization and behavioral flexibility, but it also introduces serious concerns about system-level cost, efficiency, and sustainability. This paper presents the first comprehensive system-level analysis of AI agents, quantifying their resource usage, latency behavior, energy consumption, and datacenter-wide power consumption demands across diverse agent designs and test-time scaling strategies. We further characterize how AI agent design choices, such as few-shot prompting, reflection depth, and parallel reasoning, impact accuracy-cost tradeoffs. Our findings reveal that while agents improve accuracy with increased compute, they suffer from rapidly diminishing returns, widening latency variance, and unsustainable infrastructure costs. Through detailed evaluation of representative agents, we highlight the profound computational demands introduced by AI agent workflows, uncovering a looming sustainability crisis. These results call for a paradigm shift in agent design toward compute-efficient reasoning, balancing performance with deployability under real-world constraints.

I. INTRODUCTION

The paper examines how test-time scaling and dynamic AI-agent reasoning extend LLM inference while increasing computational and infrastructure demands. It frames AI agents as iterative systems that use external tools and argues for system-level evaluation to support sustainable deployment.

I. INTRODUCTION

  • Test-time scaling improves LLM reasoning by allocating additional inference computation without changing model parameters.Source: §I · p. 1
  • Existing AI infrastructure already faces substantial energy, power, cooling, and capital demands from large-scale LLM deployment.Source: §I · p. 1
  • AI agents extend static LLM inference through iterative reasoning, tool calls, observations, and decision refinement.Source: Figure 1 · p. 1, §I · p. 1
  • Dynamic reasoning can require dozens of inference calls per request, potentially increasing computational costs by orders of magnitude.Source: §I · pp. 1–2
  • This paper provides a quantitative system-level characterization of dynamic reasoning costs across agent workflows and infrastructure behavior.Source: §I · p. 2

A. Definition of AI Agents

B. Core Components and Workflows of AI Agents

AI-agent workflows coordinate reasoning, memory, planning, and tools through alternating inference and tool-use phases. The paper motivates system-level analysis because these dynamic control flows create compute, memory, latency, and deployment challenges that static-LLM optimizations may miss.

B. Core Components and Workflows of AI Agents

C. Test-Time Scaling in AI Agents

D. Motivation

  • Agentic execution introduces compute overhead, memory pressure, and unpredictable latency and resource-use patterns compared with single-pass inference.Source: D. Motivation · p. 3
  • Because prior work emphasized task success and qualitative behavior, this paper measures deployment costs across diverse agent designs and tasks.Source: D. Motivation · p. 3

III. METHODOLOGY

The methodology evaluates representative agent workflows across diverse benchmarks, tools, reasoning strategies, and serving configurations.

Agent workflows

  • The study evaluates five agents spanning internal reasoning, tool use, reflection, tree search, and structured planning.The agents are CoT, ReAct, Reflexion, LATS, and LLMCompiler.Source: §III · pp. 3–4

Benchmarks

  • The benchmarks cover multi-hop question answering, web shopping, mathematics, programming, and conventional single-turn dialogue.HotpotQA, WebShop, MATH, HumanEval, and ShareGPT use task-specific tools where applicable.Source: §III · p. 4

Experimental scope

  • Agent–benchmark pairs are omitted when the workflow lacks the capabilities required by the target task.CoT is excluded from WebShop, while LLMCompiler is excluded from MATH and HumanEval.Source: §III · p. 4

Infrastructure

  • Experiments use vLLM with prefix caching and Llama-3.1-8B-Instruct as the default backend, with a 70B model for model-size analysis.The experiments run on A100 GPUs, using one GPU for the 8B model and eight for the 70B model.Source: §III · p. 4

IV. DEMYSTIFYING AI AGENTS

The paper analyzes agent execution from individual workflows to concurrent serving, focusing on invocation structure, latency, and deployment bottlenecks.

Scope

  • The analysis proceeds from single-request execution to LLM and tool-call characteristics, then to concurrent serving and scalability issues.The serving analysis targets system-level bottlenecks that emerge when multiple agent requests are handled concurrently.Source: §IV · pp. 4–5

Measurement dimensions

  • The study examines average LLM and tool invocations per request and agents’ latency breakdowns alongside end-to-end latency for processing a single request, including overlapping execution phases.In LLMCompiler, pink bars indicate overlapping LLM and tool execution latencies because tools execute asynchronously during plan generation.Source: Figure 4 · p. 5, Figure 5 · p. 5

A. Overall Workflow of AI Agents

Agent workflows incur substantial execution overhead because iterative reasoning and tool interaction increase calls, latency, context size, and GPU underutilization. Prefix caching mitigates repeated-prefill costs and memory pressure, especially for long-context or parallel workloads.

A. Overall Workflow of AI Agents

  • 71.0 LLM calls per request is the highest invocation count, observed for LATS because tree search expands multiple reasoning branches.Tool-augmented agents average 9.2 times more LLM calls than CoT.Source: A. Overall Workflow of AI Agents · p. 5
  • 69.4% of total latency comes from LLM inference and 30.2% from tool execution, while LLMCompiler overlaps only 18.2% through asynchronous planning.Sequential dependencies make broad overlap difficult because tool selection depends on LLM output and later inference depends on tool observations.Source: A. Overall Workflow of AI Agents · p. 5
  • Up to 54.5% of execution time can be GPU-idle when tools run on CPUs or external systems, while decode occupies 74.1% of GPU execution time.Tool latency and GPU usage determine idle periods across workloads.Source: A. Overall Workflow of AI Agents · p. 5
  • Agent latency has a broader, heavier-tailed distribution than single-turn ShareGPT because reasoning steps and tool calls vary across requests.Most ShareGPT responses complete within 9.7 seconds under the described measurement setting.Source: A. Overall Workflow of AI Agents · p. 6, Figure 7 · p. 6

B. LLM Inference and Tool-Calling Characteristics

  • HotpotQA inputs grow from roughly 1,000 tokens to 3–4× their initial size as histories accumulate, increasing KV-cache use and prefix overlap.Tool histories are larger for HotpotQA and WebShop, whereas LLM histories dominate in MATH and HumanEval.Source: B. LLM Inference and Tool-Calling… · p. 6, Figure 8 · p. 6
  • Prefix caching reduces prefill latency by 60.1% and end-to-end agentic inference latency by 15.7% on average.The benefit is smaller for decoding-dominant CoT workloads and larger where iterative calls reuse long prefixes.Source: B. LLM Inference and Tool-Calling… · p. 7
  • Tool-augmented agents use 3.0× more KV-cache memory per request on average and up to 5.4× more in the worst case than CoT.The overhead comes from appending intermediate reasoning and tool responses to successive contexts.Source: B. LLM Inference and Tool-Calling… · p. 7
  • Prefix caching reduces LATS memory requirements by 64.8% on average by reusing shared prefixes across parallel child-node calls.For sequential agents, within-request sharing is unavailable, but caching can still help across concurrent requests.Source: B. LLM Inference and Tool-Calling… · p. 7

C. AI Agent Serving Characteristics

Concurrent scheduling and prefix caching improve agent serving efficiency, but multi-step reasoning still produces lower throughput and greater system pressure than conventional single-turn serving.

C. AI Agent Serving Characteristics

  • 25× and 6.2× throughput gains result from concurrent execution for HotpotQA and WebShop, respectively, while average latency increases 2.1×.Concurrency uses GPU idle intervals during tool execution to process other requests.Source: C. AI Agent Serving Characteristics · p. 7
  • ShareGPT sustains 6.4 QPS, whereas ReAct supports 2.6 QPS on HotpotQA and 1.2 QPS on WebShop even with prefix caching.The lower peak throughput reflects repeated LLM calls and tool interactions in multi-step reasoning.Source: C. AI Agent Serving Characteristics · pp. 7–8, Figure 11 · p. 8
  • Prefix caching increases ReAct throughput by 5.62× on average but improves ShareGPT throughput only 1.03×.Repeated LLM calls amplify the benefit of avoiding redundant prefill operations in agent workloads.Source: C. AI Agent Serving Characteristics · p. 8
  • Long prefill stages can block decoding and create system-wide queuing delays, making repeated agent calls especially prone to inter-request contention.Prefix caching mitigates this interference by shortening redundant prefill work.Source: C. AI Agent Serving Characteristics · p. 8
  • Prefix caching reduces average and maximum KV-cache memory usage by 51.7% and 63.5%, respectively, under identical QPS conditions.The reduction comes from reusing shared-prefix key-value pairs across agent requests.Source: C. AI Agent Serving Characteristics · p. 8, Figure 12 · p. 8

V. DEMYSTIFYING TEST-TIME SCALING IN AI AGENTS

The evaluation measures agent accuracy and computation cost across benchmark-specific protocols using 50 sample questions per design point.

  • Accuracy and computation cost are averaged over 50 sample questions for each agent design point.HotpotQA and MATH use exact match, WebShop uses its task-specific score, and HumanEval uses unit-test success.Source: §V · p. 8

A. Analyzing Cost-Efficiency Across AI Agent Design Spaces

Agent configurations trade accuracy against latency, with increasing computation generally producing diminishing accuracy returns. The best settings therefore depend on task structure and practical latency or compute constraints.

A. Analyzing Cost-Efficiency Across AI Agent Design Spaces

  • LLMCompiler outperforms ReAct on HotpotQA in accuracy and cost-efficiency through parallel structured plans, but is less efficient than ReAct on interdependent WebShop tasks.DAG-style planning causes unnecessary tool invocations when WebShop actions depend on one another.Source: A. Analyzing Cost-Efficiency Across AI… · p. 9
  • As computation cost rises, accuracy improves with diminishing returns, making Pareto-frontier configurations preferable to accuracy-only optimization.Cost-efficiency is defined as accuracy divided by end-to-end latency; FLOPs as a cost proxy yield similar qualitative conclusions.Source: A. Analyzing Cost-Efficiency Across AI… · pp. 8–9
  • Higher iteration budgets initially improve accuracy, but average latency and accuracy saturate while 95th-percentile latency continues increasing linearly.Outlier tasks consume the full budget, worsening predictability and cost-efficiency.Source: A. Analyzing Cost-Efficiency Across AI… · p. 9, Figure 14 · p. 9
  • Few-shot examples initially improve accuracy and can reduce average latency, but excessive prompting produces diminishing or negative accuracy returns.Good examples can reduce reasoning steps enough to offset the additional per-token processing cost of longer prompts.Source: A. Analyzing Cost-Efficiency Across AI… · p. 9, Figure 15 · p. 9
  • Accuracy-to-latency ratio identifies the most cost-effective iteration and few-shot configurations under response-time or compute constraints.These configurations are marked by blue markers in Figures 14 and 15.Source: A. Analyzing Cost-Efficiency Across AI… · p. 9, Figure 14 · p. 9, Figure 15 · p. 9

B. Test-Time Scaling of AI Agents

Test-time scaling improves agent accuracy through sequential or parallel reasoning, but the two strategies impose different latency and resource trade-offs. Model size also changes how much reasoning is needed and how much energy is consumed.

B. Test-Time Scaling of AI Agents

  • Sequential Reflexion scaling raises latency from 16.9s to 25.6s for a 4% accuracy gain, while a later equal gain requires 269.5s, or 31× higher cost.Both Reflexion and LATS show diminishing accuracy returns as reflection steps increase.Source: B. Test-Time Scaling of AI Agents · p. 10
  • Increasing LATS child nodes from 1 to 16 improves accuracy by 14.4 percentage points while reducing average latency by 196.3s.Parallel path evaluation reaches high-quality answers faster, but increases concurrent requests and memory pressure.Source: B. Test-Time Scaling of AI Agents · p. 10, Figure 16 · p. 10
  • Parallel scaling suits latency-sensitive workloads, whereas sequential scaling lowers peak resource demand but produces higher step-by-step latency.The distinction follows from concurrent versus non-concurrent LLM calls.Source: B. Test-Time Scaling of AI Agents · p. 10
  • The 70B model reaches high accuracy with lower latency, while the 8B model uses more tokens to match performance but consumes less energy per request.The 70B model uses 8 A100 GPUs, whereas the 8B model uses one.Source: B. Test-Time Scaling of AI Agents · p. 10
  • LATS with parallel scaling allows the 8B model to approach 70B performance by exploring multiple paths and selecting the best one.This indicates that test-time strategy can partially compensate for a smaller backend model.Source: B. Test-Time Scaling of AI Agents · p. 10, Figure 17 · p. 10

VI. AI INFRASTRUCTURE IMPLICATIONS

Agentic test-time scaling imposes substantial GPU-energy and datacenter-power demands relative to conventional single-turn inference. These costs grow to city- or grid-scale levels under realistic traffic assumptions, motivating compute-aware workflows.

VI. AI INFRASTRUCTURE IMPLICATIONS

  • Agent-based test-time scaling consumes 62.1×–136.5× more GPU energy per query than conventional single-turn inference.Reflexion uses 41.53 Wh and 348.41 Wh per query on 8B and 70B backends, while LATS uses 22.76 Wh and 158.48 Wh.Source: §VI · p. 10, Table III · p. 11
  • Projected Reflexion energy use reaches 2.97 GWh daily for 8B and 24.89 GWh for 70B under conservative one-query-per-user assumptions.The estimate excludes CPU, memory, networking, storage, cooling, and batching overheads.Source: §VI · pp. 10–11
  • Even under modest assumptions, projected demand rivals the daily electricity consumption of Seattle and its surrounding area, estimated at 24.8 GWh.Source: §VI · p. 11
  • Under 71.4 million daily queries, 8B agents require 67.7–123.6 MW and 70B agents approach 1 GW, versus roughly 7.6 MW for single-turn ShareGPT at 70B.The power calculation converts per-query GPU energy into datacenter-wide demand.Source: §VI · p. 11, Table IV · p. 11
  • Scaling agent workloads to 13.7 billion daily searches would require nearly 200 GW for Reflexion with a 70B backend.This exceeds announced datacenter projects and many national-grid power budgets.Source: §VI · p. 11
  • Because accuracy saturates while compute and energy continue increasing, sustainable agents require compute-aware workflows rather than unconstrained reasoning depth.The paper connects this imbalance to practical limits such as Deep Research requests lasting up to 30 minutes and usage capped at 25 runs per 30 days for ChatGPT Plus.Source: §VI · p. 11

VII. DISCUSSION

Sustainable agent serving requires combining model-level efficiency, carbon-aware execution, and adaptive resource allocation, while recognizing that fixed-SLA analysis remains an open problem.

Future directions for sustainable AI agent serving

  • Quantization, distillation, sparse architectures, adaptive routing, and heterogeneous model selection can reduce agent serving cost and latency.The paper suggests assigning smaller or larger models according to agent role and task significance.Source: §VII · p. 12
  • Carbon-aware computing can provide environmental and economic benefits for agents that are not strictly latency-sensitive.Execution can migrate to instances with lower carbon intensity or electricity costs.Source: §VII · p. 12
  • Adaptive scaling can allocate GPU resources according to task difficulty and importance, avoiding over-provisioning while maintaining quality of service.Source: §VII · p. 12

Agent serving under SLA constraints

  • The analysis does not target a specific SLA because agentic systems lack established SLA standards, leaving efficient SLA-constrained serving for future work.Instead, it characterizes energy–efficiency trade-offs across diverse configurations and workloads.Source: §VII · p. 12

VIII. RELATED WORK

Prior work advances agent workflows, interfaces, component optimizations, and LLM inference techniques, but this paper emphasizes their broader infrastructure costs and efficiency trade-offs. Its conclusion argues for compute-aware agent design because test-time scaling can impose large overheads with diminishing accuracy returns.

VIII. RELATED WORK

  • AI agent workflows use iterative reasoning, tool execution, reflection, and multi-agent coordination to improve decision-making and behavioral flexibility.Source: §VIII · p. 12
  • Agent interfaces standardize tool invocation, context management, and multi-agent communication, whereas this work studies the infrastructure challenges of agentic workloads.Source: §VIII · p. 12
  • Prior system optimizations target scheduling, pipelined or parallel execution, coordination, and resource isolation, while this study characterizes broader behavior across agents at scale.Source: §VIII · p. 12
  • The paper focuses on fundamental LLM inference optimizations because applying the full range of recent techniques to agents remains unclear and beyond scope.Relevant directions include KV-cache reuse or compression, speculative decoding, and prefill–decode disaggregation for long-context workloads.Source: §VIII · pp. 12–13

IX. CONCLUSION

  • Agent workloads incur substantially higher energy and latency costs than conventional single-turn inference, while test-time scaling yields sharply diminishing accuracy returns.The conclusion therefore recommends compute-aware reasoning, smarter scheduling, caching, prompt engineering, and hybrid scaling adapted to deployment constraints.Source: §IX · p. 13

Related papers

Loading 2506.04301v2…