PaperlayerView exact revision on arXiv

Source-linked AI summary

AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

Sumin Lee, Sukmin Cho, Suengjae Lim, Youngjin Kwon

arXiv:2610.01108v1cs.CL

TL;DR

Existing retrieval-based speculative decoding for coding agents can miss reusable session and repository text, mismatch emitted forms, and ignore changing acceptance across agents and turns. AGSPEC adds source- and emission-aware corpora plus offline and online draft-length control to existing retrieval engines. It achieves the highest throughput in most evaluated settings on SWE-bench Verified and TeamBench, and remains effective beyond repository-based multi-agent benchmarks.

  • Problem

    Existing retrieval-based speculative decoding misses some reusable session and repository text, mismatches stored files with emitted forms, and ignores agent- and turn-dependent accept lengths.

    Source: §1 · p. 2, §2.2 · p. 4

  • Method

    AGSPEC layers session, workspace, and global corpora indexed for agent emission forms with offline per-agent draft caps and online verification-feedback adaptation on existing retrieval engines.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    AGSPEC achieves the highest throughput in most evaluated settings on SWE-bench Verified and TeamBench, reaching 2.27–4.37× autoregressive throughput at batch size 1.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support following the agent pipeline by indexing session-produced text in emitted forms and drafting according to how much each agent can reuse.

    Source: §5 · p. 10

    Session and workspace corpora may contain sensitive prompts, repository content, or tool outputs, so deployments should restrict access and discard them when sessions end.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent pipelines: much of the reusable text is missing from their corpora or stored in a form that differs from what the agent emits, and their draft lengths ignore that accept length varies across agents and drifts over turns. We present AgSpec, a framework that supplies the corpus and draft-length policies that existing retrieval engines lack in coding-agent pipelines. AgSpec retrieves from session, workspace, and global corpora, retaining the ongoing session trajectory and indexing opened files in the agent's emission format. It bounds each agent's draft length with an offline-profiled cap and adapts the length online from verification feedback. On two repository-level multi-agent coding benchmarks, AgSpec outperforms five retrieval-based drafters and EAGLE-3 in most evaluated settings, raising generation throughput over autoregressive decoding up to 4.37$\times$ at batch size 1 and 4.76$\times$ at batch size 16. AgSpec also remains effective on benchmarks without a repository or a multi-agent pipeline, showing that its gains generalize to coding agents broadly.

1 INTRODUCTION

Coding agents offer reusable session and repository text, but existing retrieval-based speculative decoding misses some of it and drafts without accounting for agent-specific acceptance. AGSPEC adapts corpus and draft-length policies to the pipeline and improves throughput across coding-agent settings.

1 INTRODUCTION

  • Retrieval-based speculative decoding copies continuations without a separate draft model, while repeated identifiers, code fragments, and repository conventions make coding-agent text reusable.Accepted tokens save sequential target-model steps, but rejected drafts add verification cost, especially at large batch sizes.Source: §1 · p. 1
  • Existing corpora miss session and repository text or store files in forms that do not match emitted patches; broader retrieval scope raises accept length.A truncated file or an earlier attempt may otherwise fall out of reach.Source: §1 · p. 2
  • Accept length differs across agents and shifts within and across turns, so fixed or match-derived draft lengths can misfit agent-specific contexts.The paper frames agent and context characteristics as guidance for draft length.Source: §1 · p. 2
  • AGSPEC leaves retrieval engines unchanged, organizes session, workspace, and global corpora, and combines per-agent offline draft caps with online adaptation from verification feedback.Workspace artifacts are indexed in forms the agent emits.Source: §1 · p. 2
  • 3.61× versus 2.72×: AGSPEC-SAM raises Gemma3-27B SWE-bench speedup over autoregressive decoding at batch size 16, and leads on single-agent and non-repository benchmarks.The broader evaluation reports the highest throughput in most evaluated settings.Source: §1 · pp. 2–3

2 BACKGROUND

Coding-agent sessions span multiple calls, tools, turns, and workspace artifacts, while retrieval-based speculative decoding depends on corpus scope and draft length. Existing approaches do not fully account for session and repository reuse, emission formats, or role- and turn-dependent acceptance.

2.1 CODING AGENTS

  • A coding-agent session can involve specialized roles and repeated turns, with a persistent workspace whose artifacts may be truncated from model prompts.Workspace artifacts differ from conversation history and can outlast a session.Source: §2.1 · p. 3
  • Under unified-diff editing, copied artifact lines gain a prefix, so their emitted token sequence differs from the file stored in the workspace.Other scaffolds also emit whole files, search-and-replace blocks, or line-addressed commands.Source: §2.1 · p. 3

2.2 RETRIEVAL-BASED SPECULATIVE DECODING

  • Retrieval-based speculative decoding matches a context suffix in a corpus and copies its continuation; corpus scope governs whether a useful match exists, while draft length governs rejection.The paper describes these as the two central design choices for retrieval-based drafting.Source: §2.2 · p. 3
  • Prior methods cover only some agent reuse sources, and their fixed or preset-factor draft lengths ignore verification outcomes, roles, and turns.AGSPEC addresses these gaps with source- and emission-aware corpora and adaptive draft length.Source: §2.2 · p. 4

3 AGSPEC DESIGN

AGSPEC adds coding-agent-aware corpus policies and verification-driven draft-length control on top of an existing retrieval engine. It retains session and workspace text in relevant forms and adapts drafts using agent-specific profiling and runtime feedback.

3 AGSPEC DESIGN

  • AGSPEC layers corpus organization and verification-driven draft-length control over an existing retrieval engine without modifying it.It applies to engines that search multiple corpora and accept a per-step draft length.Source: §3 · p. 4

3.1 CODING-AGENT-AWARE RETRIEVAL

  • The session corpus shares prompts, tool outputs, and generated tokens across calls; the workspace corpus holds opened artifacts, while the global corpus is fixed.Session and workspace corpora are live and discarded when the session ends.Source: §3.1 · pp. 4–5
  • Workspace artifacts are indexed as stored and in scaffold emission forms; the longer live match wins, while a global match must exceed it by margin δ.A unified-diff scaffold receives copies with kept-line and removed-line prefixes.Source: §3.1 · p. 5

3.2 AGENT-AWARE DRAFT-LENGTH CONTROL

  • At each step, AGSPEC sets draft length using the match length, the requesting agent’s online scale, and its offline cap, then limits it to available continuation tokens.If the retrieval engine has its own length policy, AGSPEC uses the smaller draft length.Source: §3.2 · p. 5
  • Offline profiling uses each agent’s historical reusable-token distribution to set a cap where another draft position no longer covers its verification cost.The cap provides an agent-specific bound.Source: §3.2 · pp. 5–6
  • Online verification feedback updates each agent’s scale to track runtime acceptance drift, targeting full draft acceptance with probability 1 − τ under the stated idealization.The scale is initialized at 1.0.Source: §3.2 · p. 6

4 EVALUATION

Across repository benchmarks, AGSPEC improves decoding throughput over autoregressive decoding and is highest or second-highest in every evaluated setting. Tests beyond repository-based, multi-agent workflows and ablations examine its generality and the effects of its corpus and draft-length policies.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses three models, two repository-level benchmarks, six prior speculative methods, and AR decoding, measuring throughput and accept length.SWE-bench Verified and TeamBench have multi-agent workflows; the implementation uses SAM-Decoding and SuffixDecoding engines.Source: §4.1 · p. 6

4.2 MAIN EVALUATION

  • AGSPEC reaches 2.27–4.37× autoregressive throughput at batch size 1 and 1.08–4.76× at batch size 16, ranking first or second in every setting.It has the highest AL in most settings.Source: §4.2 · p. 7, Figure 4 · p. 7
  • With AGSPEC, throughput rises 16.5% on average for SuffixDecoding and 64.6% for SAM-Decoding; AGSPEC-SAM beats SuffixDecoding in nine of twelve settings.AL rises by 18.5% with SuffixDecoding and nearly doubles with SAM-Decoding.Source: §4.2 · p. 7

4.3 NON-REPOSITORY AND IMPLICIT-AGENT TESTS

  • On LiveCodeBench and Terminal-Bench, AGSPEC improves throughput over prior speculative methods without repository context or separate explicit agent roles.LiveCodeBench throughput reaches 4.87–7.94× AR; on Terminal-Bench it is 9.3–27.2% above the fastest prior speculative method.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4.1 IMPACT OF CODING-AGENT-AWARE RETRIEVAL

  • Replacing per-call text with session history raises AL by 1.6–2.1×; adding workspace text yields 2.9–4.8× the AL of the global corpus alone.The workspace corpus adds 26–47% on its own and 2.2–7.0% beyond the session corpus.Source: §4.4.1 · pp. 8–9, Figure 6 · p. 8
  • Emission-form workspace indexing raises coder AL by 13–39% in turn one, with gains falling to 3% or less by turn five.By later turns, the session corpus already contains the coder's patches in emitted form.Source: Figure 6 · p. 8, §4.4.1 · p. 9

4.4.2 IMPACT OF AGENT-AWARE DRAFT-LENGTH CONTROL

  • AGSPEC is fastest at both batch sizes, 25% and 16% above K = 8; fixed K = 32 is 14% faster than K = 8 at batch 1 but 30% slower at batch 16.At batch 1, K = 32 accepts AL 6.0 versus 5.5 for AGSPEC; at batch 16, its AL is 10.3 versus 10.4.Source: §4.4.2 · p. 9, Figure 7 · p. 9
  • At batch size 16, online scaling raises throughput 66% and cuts rejected-token share from 78% to 50%.At batch size 1, AGSPEC rejects at most 6 tokens per step while accepting 83–89% as many as K = 32; its drafts grow from about 10 tokens early to 21 in turn five.Source: §4.4.2 · p. 9, Figure 7 · p. 9

5 CONCLUSION

AGSPEC adapts retrieval-based speculative decoding to coding-agent pipelines by making reusable text available in agents’ emitted forms and adjusting draft lengths to each agent. It achieves the highest throughput in most evaluated settings on SWE-bench Verified and TeamBench, and remains effective without a repository or explicit agent roles.

5 CONCLUSION

  • AGSPEC makes session-produced text available for retrieval in the forms agents emit and adapts draft length using agent-specific profiles and verification feedback.The framework is orthogonal to the retrieval engine and runs on existing engines without modification.Source: §5 · p. 9
  • AGSPEC achieves the highest throughput in most evaluated settings on SWE-bench Verified and TeamBench.It reaches 2.27–4.37× autoregressive decoding throughput at batch size 1.Source: §5 · p. 10
  • AGSPEC remains effective without a repository or explicit agent roles.The reported results support its use beyond repository-based, multi-agent benchmarks.Source: §5 · p. 10

USE OF GENERATIVE AI TOOLS

The authors disclose using generative AI tools for manuscript work and figure and experiment presentation, while retaining responsibility for the final content.

USE OF GENERATIVE AI TOOLS

  • Generative AI tools assisted with translating, drafting, and editing manuscript text.The authors reviewed the suggestions and take responsibility for the final content.Source: USE OF GENERATIVE AI TOOLS · p. 10
  • The tools also provided feedback on figures and the presentation of methods and experiments.The authors state they reviewed AI-assisted suggestions.Source: USE OF GENERATIVE AI TOOLS · p. 10
  • AI advice was used to justify the online adaptation discussed in §G.1.The authors take responsibility for claims and artifacts produced with AI assistance.Source: USE OF GENERATIVE AI TOOLS · p. 10

ETHICS STATEMENT

AGSPEC preserves the target model’s output distribution for each model call, while its session and workspace corpora raise data-confidentiality concerns that deployments should address.

ETHICS STATEMENT

  • AGSPEC verifies drafts with the target model, preserving its output distribution for each model call.The decoding method therefore introduces no new output-level behavior relative to ordinary target-model decoding.Source: ETHICS STATEMENT · p. 10
  • Session and workspace corpora may contain sensitive prompts, repository content, or tool outputs.These materials create a disclosure concern distinct from output-level behavior.Source: ETHICS STATEMENT · p. 10
  • Deployments should restrict access to each session’s corpus, prevent disclosure across sessions, and discard the corpus when the session ends.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility protocol fixes evaluation and serving conditions, controls corpus construction and session resets, and uses consistent decoding and seeds across runs.

REPRODUCIBILITY STATEMENT

A IMPLEMENTATION DETAILS

The implementation fixes baseline draft policies before measurement and specifies corpus and draft-length settings for the reported runs.

A IMPLEMENTATION DETAILS

  • Baseline draft lengths and engine settings were fixed before measurement, with tree-based REST and FastCoder reported using a chain approximation rather than mapped tree depth.SuffixDecoding uses the vLLM default, EAGLE-3 uses a depth-3 chain, and the autoregressive baseline follows the proposer path without drafting.Source: A IMPLEMENTATION DETAILS · p. 14
  • The session corpus admits each prompt’s trailing 2,048 tokens and resets between sessions, while the workspace corpus is session-isolated.The preference margin is 5 tokens; the draft-length scale starts at 1.0, uses η = 0.5, is clipped to [0.05, 12.0], and persists across sessions.Source: A IMPLEMENTATION DETAILS · p. 14
  • The global corpus uses instances disjoint from evaluation: 70 build and 30 evaluation instances for SWE-bench Verified, and 213 and 30 for TeamBench.Table 3 reports final-run break-even estimates alongside the τ and per-agent caps used; SWE-bench Verified models use the λ measured with Devstral.Source: A IMPLEMENTATION DETAILS · p. 14, Table 3 · p. 15

B ADDITIONAL CORPUS ABLATIONS

The corpus ablations favor retaining session history, while the preference-margin and continuation-selection analyses show that corpus sources benefit from different retrieval choices.

B ADDITIONAL CORPUS ABLATIONS

  • Restricting the session corpus to the current call sharply reduces AL, while sharing it across agents yields a further, smaller gain.The table reports AL changes relative to the reported configuration on SWE-bench Verified at batch size 1.Source: B ADDITIONAL CORPUS ABLATIONS · p. 15, Table 4 · p. 15
  • Growing the workspace corpus across sessions brings no meaningful benefit, though the authors leave open whether this changes when many sessions share a repository.The workspace artifacts from the current session already capture most useful workspace reuse.Source: B ADDITIONAL CORPUS ABLATIONS · p. 15
  • Pooling the workspace corpus into the global corpus performs on par with keeping them separate, but the authors retain separation for serving manageability across users’ repositories.Table 4 reports the corpus-control AL changes relative to the reported configuration.Source: B ADDITIONAL CORPUS ABLATIONS · p. 15, Table 4 · p. 15

C CORPUS PREFERENCE MARGIN

  • Agreement stays high across a broad range of preference margins, with δ = 2 highest across all three models; the reported δ = 5 is not sharply tuned.The sensitivity table reports oracle-normalized continuation agreement on SWE-bench Verified at batch size 1.Source: Table 5 · p. 15, C CORPUS PREFERENCE MARGIN · p. 16

D CORPUS-SPECIFIC CONTINUATION SELECTION

  • The global corpus benefits from frequency-based selection, the workspace corpus from its earliest occurrence, and the session corpus shows only small differences across rules.REST-style selection indicates additional headroom for structured candidate ranking in the global corpus.Source: D CORPUS-SPECIFIC CONTINUATION SELECTION · p. 16, Table 6 · p. 16

E DRAFT TOKEN COMPOSITION

Draft rejection and speed-up vary across settings, while the adaptive controller uses verification feedback to track normalized acceptance lengths and agent-specific behavior.

E DRAFT TOKEN COMPOSITION

  • 12.0–39.8-point lower rejection rates on the suffix automaton accompany drafting on 98–99% of request steps, versus 50–56% for SAM-Decoding.The reported reduction reflects shorter, better-placed drafts rather than less frequent drafting; on the suffix tree, rejection rates are similar on TeamBench and 1.9–10.0 points lower on SWE-bench Verified.Source: E DRAFT TOKEN COMPOSITION · p. 16

F SPEED-UP BY TURN AND AGENT

G.1 INTERPRETING THE SCALE UPDATE

  • Under the stated idealization, the scale update is a stochastic subgradient step targeting the τ-quantile of normalized unbounded acceptance length.The scale fluctuates around and tracks this target as the session corpus grows and the distribution shifts; updates skip fully accepted drafts truncated by the cap.Source: §G.1 · pp. 17–18

G.2 ADAPTIVE DRAFT LENGTH BEYOND RETRIEVAL

  • Unlike model-based drafters, retrieval-based SD lacks token probabilities, entropy, or hidden states, so AGSPEC combines per-agent offline caps with online scaling from verification feedback.The online scale sizes drafts from match length, using the issuing agent as an additional signal.Source: §G.2 · p. 18

G.3 OVERHEAD OF AGSPEC

The overhead analysis separates global-corpus memory and construction costs from the per-step retrieval work included in throughput.

G.3 OVERHEAD OF AGSPEC

  • The global corpus occupies 2.5–4.3 GB of host memory and takes 9–15 seconds to load; it uses no GPU memory and does not reduce KV cache.Session and workspace corpora are session-local and released afterward.Source: §G.3 · p. 18, Table 7 · p. 18
  • Building the global corpus from recorded trajectories takes 18–26 seconds on CPU and requires no GPU.Workspace corpus construction is per session, while the session corpus grows during decoding; both costs are included in reported throughput.Source: §G.3 · p. 18
  • At batch size 16, AGSPEC accounts for 1.1–5.5% of a 47–182 ms step, with the remainder attributed to the target model and engine.The reported proposer time includes corpus queries, draft selection, and appending step tokens; the engine’s asynchronous forward pass prevents a complete CPU-side split.Source: §G.3 · pp. 18–19, Table 8 · p. 19
Loading 2610.01108v1…