PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang, Ziming Miao, Yi Zhu, Hui Shen, Zhongwei Wan, Fan Yang, Mi Zhang

arXiv:2608.24004v1cs.CL

TL;DR

LLM-agent inference is costly, and existing speculative decoding methods degrade substantially at large batch sizes. AgentSpec analyzes these bottlenecks and combines structure-isolated drafting with redundancy-aware budget allocation. Experiments report superiority over state-of-the-art baselines, while the method’s speedup can depend on explicit semantic metadata and repeated generation patterns.

  • Problem

    LLM-agent serving has high inference cost, while existing speculative decoding methods suffer substantial speed degradation under large batch sizes.

  • Method

    AgentSpec combines structure-isolated drafting with redundancy-aware budget allocation to reduce rejected speculative paths and better use dynamically available token budgets.

  • Results

    AgentSpec demonstrates superiority over state-of-the-art baselines and achieves over 2.5× faster decoding than standard autoregressive decoding without thinking mode.

  • Takeaways & Limitations

    AgentSpec provides a speculative decoding approach tailored to batch inference of LLM agents.

  • Takeaways & Limitations

    AgentSpec may require minor serving-interface adaptation, benefits from explicit semantic block information, and can depend on repeated generation patterns when such metadata is unavailable.

Abstract

from arXiv · show

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.

1 Introduction

LLM agents face high inference costs, while speculative decoding loses effectiveness at the large batch sizes used by modern serving systems. AgentSpec targets these limitations through structure-isolated drafting and redundancy-aware budget allocation.

  • LLM-agent serving incurs high inference cost because agents generate long, multi-round outputs.
  • Large-batch serving exposes substantial speed degradation in existing speculative decoding methods, limiting their real-world effectiveness.
  • Existing methods face high speculative-token rejection rates, increasing verification overhead that can outweigh accepted-token savings as batch size grows.
  • AgentSpec uses structure-isolated drafting to restrict speculation to semantically coherent workflow segments and reduce irrelevant speculative paths.
  • AgentSpec adds redundancy-aware budget allocation to use dynamically available token budgets across requests more efficiently.
  • The evaluation compares AgentSpec with NGram, EAGLE-3, SuffixDecoding, and MTP across multiple models and agentic workloads in vLLM.

2 Related Works

Prior work improves LLM-agent efficiency through planning, action selection, caching, reuse, and speculative decoding. Speculative methods either use draft models or retrieve historical continuations, but existing designs can suffer high rejection rates and verification costs.

  • Agent-efficiency research reduces model calls through improved planning, task decomposition, and action selection.
  • Other approaches cache or reuse intermediate results across agent steps.
  • Speculative decoding accelerates lossless inference by generating multiple candidate tokens and verifying them with the target model.
  • Draft-model-based methods such as EAGLE-3 and MTP train lightweight draft models, whereas NGram and SuffixDecoding retrieve continuations from generation history.
  • These algorithm-level speculative designs often incur high rejection rates, increasing verification cost and degrading speed.

3 Speculative Decoding for Batch Inference of LLM Agents

Large-batch speculative decoding for LLM agents loses its speed advantage because rejection and token-budget utilization become major efficiency bottlenecks. The analysis measures these effects across batch sizes and shows why existing methods degrade in vLLM.

  • Overall Efficiency Analysis: At maximum batch size 1, EAGLE-3 exceeds 2.5× speedup and SuffixDecoding exceeds 1.9×, but speedup rapidly diminishes as batch size increases.When maximum batch size exceeds 32, speculative decoding becomes slower than standard autoregressive decoding.
  • Efficiency Model: The analysis models execution-time savings using drafted-token count, rejection rate, acceptance savings, drafting cost, and verification cost.The formulation isolates rejection rate and drafted-token count as the dominant efficiency factors when drafting cost is small.
  • Efficiency Model: Lower rejection rates and higher draft-budget utilization produce larger speculative-decoding speedups.The draft-token budget is treated as a central determinant of effective acceleration.
  • Bottleneck Analysis: EAGLE-3 exceeds 50% rejection and NGram exceeds 85% across most batch sizes, limiting effective draft-budget utilization.The resulting verification overhead contributes directly to degraded throughput speedup at large batch sizes.
  • Bottleneck Analysis: When batch size exceeds 256 on an A100, FFN latency grows linearly and dominates decoding, causing verification cost to approach autoregressive decoding cost.This compute-bound regime reduces the time saved by accepting draft tokens.
  • Bottleneck Analysis: Existing methods severely under-utilize the available token budget, while AdaSpec’s acceptance-rate-based allocation does not improve utilization.The comparison examines actual drafted tokens against M(b) across batch sizes.

4 Method: AgentSpec

AgentSpec improves speculative decoding for LLM agents through structure-isolated drafting and redundancy-aware budget allocation. It organizes historical context by semantic structure, restricts draft retrieval to matched blocks, and allocates token budgets according to redundancy.

  • Structure-Isolated Drafting: AgentSpec requires an agentic structure identifier and organizes historical context by agent-specific execution structure.The identifier accompanies each request and supports semantic-block tracking and retrieval.
  • Structure-Isolated Drafting: Generation patterns are more consistent within semantic blocks than across blocks or user queries, motivating structure-isolated drafting.The analysis measures repeated segments while distinguishing semantic blocks and queries.
  • Structure-Isolated Drafting: AgentSpec retrieves speculative drafts only from the matched semantic block, avoiding irrelevant contexts and lowering rejection rates.The method maintains separate caches for semantic blocks; its rejection rate reaches 26%, over 2× lower than existing baselines.
  • Redundancy-Aware Budget Allocation: Redundancy-aware allocation assigns more draft tokens to requests with higher acceptance potential while ensuring each request can draft its most confident prefix.The redundancy signal combines candidate consensus with a saturation term that downweights estimates supported by limited history.
  • Redundancy-Aware Budget Allocation: AgentSpec allocates a batch-level speculative budget using batch size and an overall budget factor, then adjusts each request's draft length by redundancy.The method computes per-request redundancy from continuations within the same semantic block and user query.
  • Redundancy-Aware Budget Allocation: AgentSpec keeps total drafted tokens below the maximum budget and approaches that budget as batch size increases.This indicates effective utilization of the available speculative budget.

5 Experiments

The experiments evaluate AgentSpec across diverse agentic workloads, models, batch sizes, thinking modes, and baselines. AgentSpec consistently improves efficiency and latency, while its components jointly contribute to performance.

  • Experimental Setup: AgentSpec is evaluated against draft-model and draft-model-free baselines on four models spanning four LLM families and four agentic workloads.The workloads include workflow-based Code Generation and Deep Research, plus model-based SWE-Bench and GAIA.
  • End-to-End Comparison: AgentSpec achieves up to 104% higher goodput than baseline speculative decoding and up to a 2.02× speedup over autoregressive decoding.Most baseline methods provide lower goodput than standard autoregressive decoding in agent workloads, whereas AgentSpec improves goodput across workloads and model settings.
  • Different Execution Patterns: AgentSpec achieves up to a 2.2× speedup over autoregressive decoding across USACO difficulty subsets and LLMs.Its speedup remains better than other speculative decoding methods from Bronze through Platinum subsets.
  • Non-Agentic Benchmark: AgentSpec remains more efficient than speculative decoding baselines and autoregressive decoding across all evaluated Spec-Bench subsets.Spec-Bench contains fewer repeated historical generations and lacks the explicit semantic structures common in agentic workloads.
  • Latency: AgentSpec reduces tail latency by up to 1.47× at P90 and 1.39× at P99 relative to standard autoregressive decoding.The analysis measures end-to-end latency from query submission to final output and compares AgentSpec with baseline speculative decoding methods.
  • Ablation Studies: AgentSpec consistently outperforms its single-component variants, and structure-isolated drafting contributes more speedup than redundancy-aware budget allocation.The variants are AgentSpec (S), using structure-isolated drafting, and AgentSpec (R), using SuffixDecoding with redundancy-aware allocation.
  • Sensitivity Studies: AgentSpec maintains superior efficiency across maximum batch sizes and thinking modes, exceeding 2.5× autoregressive speed under the w/o think setting.Some baselines degrade below autoregressive decoding as maximum batch size increases, while AgentSpec remains superior.

6 Conclusion

AgentSpec is a speculative decoding algorithm tailored to batch inference for LLM agents. It combines structure-isolated drafting with redundancy-aware budget allocation, and experiments report superiority over state-of-the-art baselines.

  • Conclusion: AgentSpec targets batch inference of LLM agents with structure-isolated drafting and redundancy-aware budget allocation.The two components constrain speculation to coherent workflow segments and exploit dynamically available token budgets using agent-level redundancy.
  • Conclusion: Experimental results demonstrate AgentSpec’s superiority over state-of-the-art speculative decoding baselines.

Limitation

AgentSpec may require minor practical adaptation and benefits from explicit semantic block information. Without that metadata, its speedup may depend on repeated generation patterns in the workload.

  • Limitation: AgentSpec’s serving interface may require minor adaptation, and its speedup without explicit semantic metadata may depend on repeated generation patterns.The structure-isolated drafting component benefits from semantic block information, although AgentSpec can operate without such metadata.

Ethical Considerations

AgentSpec is a serving-time acceleration method that preserves the underlying model’s generation setup but may lower barriers to misuse. Responsible deployment safeguards and non-sensitive workflow metadata are therefore emphasized.

  • Safety Scope: AgentSpec does not modify model parameters, training data, or the generation objective, so existing safety mechanisms should remain applicable.The paper names content moderation, tool-use control, and deployment restrictions as applicable mechanisms.
  • Responsible Deployment: Improved inference efficiency may lower barriers for misuse, motivating rate limiting, permission control, and monitoring.The paper cites automated spam generation and unsafe tool-use workflows as examples of misuse.
  • Data Considerations: AgentSpec requires lightweight semantic structure metadata describing workflow structure rather than private or sensitive user information.The experiments use publicly available benchmarks and do not require additional private user data.

A.1 Experiment Configuration Details

The experiments use vLLM with fixed serving conditions and report implementation details for AgentSpec, baselines, and GPU evaluation. AgentSpec accepts agent-level structure and request identifiers to guide speculation.

  • Experiment setup: Experiments use vLLM V1 with official baseline implementations, fixed maximum batch size and serving parameters, and five repeated measurements.The listed setup uses NVIDIA A100 80G GPUs unless otherwise specified.
  • Baseline configuration: NGram uses five speculative tokens and a lookup maximum of four, while SuffixDecoding uses 32 speculative tokens.
  • Inference configuration: The evaluation uses FlashAttention-2, vLLM’s default maximum batch size of 256, and the official MiMo-7B-RL MTP module.
  • AgentSpec interface: AgentSpec adapts speculation length to batch size and requires application-provided semantic structure and the current request’s query index.
  • AgentSpec interface: The extended vLLM interface accepts structure, query-id, and agent-id parameters, with structure encoding delimiters for code blocks, tool calls, and mathematical expressions.
  • Evaluation scope: Table 7 reports AgentSpec speedup across GPU platforms for GPT-OSS-20B and multiple agent workloads.

A.2 Algorithm Pseudocode

AgentSpec separates structure-isolated drafting from redundancy-aware budget allocation. Its drafting procedure tracks semantic blocks and retrieves matching historical continuations, while budget allocation distributes speculative lengths across requests.

  • Structure-Isolated Drafting: Structure-Isolated Drafting initializes a PDA, converts generated tokens to strings, and uses PDA state to identify the current semantic block.
  • Structure-Isolated Drafting: The drafting algorithm retrieves cache entries for the current request, query, and semantic block, returning no candidates when that cache is empty.
  • Redundancy-Aware Budget Allocation: Redundancy-Aware Budget Allocation computes a total speculative budget from α and batch size, then counts candidate continuations for each request.
  • Redundancy-Aware Budget Allocation: Requests without candidate continuations receive zero allocation, while other requests are evaluated using their most frequent continuation prefix.
  • Redundancy-Aware Budget Allocation: The algorithm assigns each request a draft length using the redundancy-based allocation and returns the resulting per-request lengths.
  • Complexity analysis: AgentSpec has time complexity T = O(K · L) + O(K), matching NGram asymptotically with PDA updates costing O(1) amortized per token and filtering plus scoring costing O(K) per step.
Loading 2608.24004v1…