PaperlayerView exact revision on arXiv

Source-linked AI summary

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

arXiv:2608.26004v1cs.AIcs.CL

TL;DR

As agentic pipelines accumulate context, forward-pass latency becomes the dominant inference bottleneck, while standard speculative decoding cannot exploit differing drafter–verifier context views. ASYMSPEC lets the drafter use full context and the verifier use compressed context, recovering approximately 90% of full-context performance at lower compute with higher throughput.

  • Problem

    Growing retrieval, tool-use, and interaction contexts make forward-pass latency the primary inference bottleneck, while standard speculative decoding requires identical drafter and verifier inputs.

  • Method

    ASYMSPEC uses a compressed-context verifier and full-context drafter, combining same-model cross-context δ-fusion with a parameter-free Context-Divergence Acceptance gate.

  • Results

    Approximately 90% of full-context accuracy is recovered across four agentic capabilities and two end-to-end benchmarks, with 1.3–1.7× throughput and 0.2–0.3× compute on text tasks.

  • Takeaways & Limitations

    Asymmetric context access provides a near-ceiling accuracy and efficiency operating point when compression discards critical reasoning signals.

  • Takeaways & Limitations

    ASYMSPEC requires verifier logits and therefore does not apply to proprietary APIs exposing only generated text.

Abstract

from arXiv · show

Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $δ$-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$--$1.7\times$ throughput speedups at $0.2$--$0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

1 Introduction

Agentic pipelines accumulate context that makes inference costly, while compression lowers cost by discarding details needed for accuracy. ASYMSPEC separates drafter and verifier context access to recover accuracy at compressed verification cost.

  • Growing retrieval, tool-use, dialogue, memory, and multimodal contexts make the forward pass the dominant production latency bottleneck.
  • Compression reduces serving cost but discards finegrained details critical for task accuracy, creating an accuracy–overhead tradeoff.
  • Standard speculative decoding accelerates generation losslessly but requires drafter and verifier to process identical input tokens.
  • ASYMSPEC lets the verifier use compressed context while the drafter reads full context, exploiting verifier-dominated latency and drafter efficiency.
  • Across four agentic capabilities and two end-to-end benchmarks, ASYMSPEC recovers ≈90% of full-context accuracy at 0.2–0.3× compute and 1.3–1.7× throughput.

2 Related Work

Prior work improves speculative decoding, context compression, contrastive decoding, and agent-level acceleration, but generally preserves shared context or targets different operating points. ASYMSPEC relaxes shared context access to recover information lost through compression.

  • Speculative decoding: Standard speculative decoding and long-context variants feed drafter and verifier the same input, limiting exploitation of long-context compute asymmetry.
  • Context compression: Context-compression methods reduce long-context overhead through token pruning, compact latents, or KV-cache operations, while treating accuracy loss as an unavoidable cost.
  • Contrastive decoding: Contrastive decoding and SCD subtract logits to address model-capacity differences, but both operate on a single shared context.
  • Asymmetric configurations: RAPID reverses ASYMSPEC’s context assignment by using a truncated drafter and full-context verifier, preserving full-context latency rather than compressed-verifier efficiency.
  • Agent acceleration: Agent-oriented methods accelerate decisions by speculating over actions, tools, or plans, whereas ASYMSPEC changes token-generation context access.

3 Method

ASYMSPEC uses a full-context drafter to steer a compressed-context verifier. Contrastive context-gain fusion supplies discarded information, while divergence-aware acceptance stabilizes draft verification.

  • Asymmetric context access: The verifier reads compressed context for low latency, while the lightweight drafter reads full context to reconstruct discarded information.
  • Speculation step: Each speculation step runs augmented-drafter, base-drafter, and verifier passes, producing logits a, b, and t plus K draft tokens.
  • Contrastive δ-fusion: The context-gain signal δ is formed by subtracting base-drafter logits from augmented-drafter logits, isolating the shift induced by additional context.
  • Contrastive δ-fusion: After rejection, βδ is fused into verifier logits, shifting prediction toward the full-context result without requiring the verifier to attend to full context.
  • Context-divergence acceptance (CDA): The CDA gate relaxes acceptance as full and compressed views diverge, using JSD so γeff remains within [γ/2, γ].
  • Context-divergence acceptance (CDA): Upon rejection, the method emits the δ-fused arg max; unlike standard speculative decoding, it does not preserve a strict target distribution and is calibrated for greedy emission.
  • Cross-modal extension: Cross-modal use is supported because δ and γeff are computed on the shared output vocabulary, allowing a vision-language drafter to steer a text-only verifier.

4 Experiments

Experiments evaluate ASYMSPEC across isolated agentic capabilities and end-to-end agents under asymmetric compression, with accuracy and efficiency measured against full- and compressed-context references.

  • Benchmarks: The evaluation covers long-context multi-hop QA, multi-turn instruction following, tool use, multimodal reasoning, GAIA, and SimpleQA.
  • End-to-end setup: GAIA uses the smolagents CodeAgent ReAct loop with cached DuckDuckGo search and webpage visits as the only tools.
  • Decoding procedure: The algorithm samples K drafts from the full-context drafter, computes compressed-context drafter and verifier logits, then accepts drafts or emits fused arg-max tokens.
  • Model variants: The end-to-end comparison includes text and cross-modal drafter variants, while the verifier receives pre-computed captions in the image-aware setting.
  • Configuration: Headline runs use Qwen3-32B verification, greedy decoding, K = 2, β = 1.0, and γ = 0.5.
  • Metrics: Speedup measures wall-clock throughput over the full-context Ceiling, whereas FLOPs measure normalized per-step prefill compute.
  • Baselines: References include compressed-context Floor, full-context Ceiling, standard shared-context SD, SCD, and inverse-asymmetric RAPID.

5 Results

Across isolated capabilities, multimodal reasoning, and live agent loops, ASYMSPEC recovers much of compressed-context accuracy while retaining verifier-side efficiency. Recovery is strongest when compression discards critical signals, with stable acceptance rates and identifiable structural limits.

  • Isolated agentic capabilities: 87–99% of full-context performance is reached, leaving a 0.9–9.7-point residual gap because compressed verifiers cannot fully reconstruct complex reasoning chains or tool dependencies.The limitation is attributed to verifier input constraints and logit steering alone.
  • Isolated agentic capabilities: 59–94% of the Floor–Ceiling gap is closed on long-context multi-hop QA and tool use at 0.23× the Ceiling’s compute.Speedup and FLOPs are averaged across the three text benchmarks.
  • Isolated agentic capabilities: Recovery activates with compression severity: ASYMSPEC restores over two-thirds of the Floor–Ceiling gap at 500 tokens, while gains vanish near the 12k-token ceiling.Acceptance remains stable at 0.851–0.855 across the truncation sweep.
  • Multimodal reasoning: 53.9% overall accuracy on MathVista exceeds symmetric SD by 10.1 points, with the largest gains on VQA and FQA.The drafter supplies visual grounding while the verifier supplies causal logic; the text-only verifier remains below the VL-drafter reference.
  • End-to-end agentic loops: 24.2% on GAIA and 65.0% on SimpleQA match or exceed available full-context references without degradation as context accumulates.Draft acceptance remains stable at 0.88–0.90 in live loops.
  • End-to-end agentic loops: Compute falls to 0.78× and 0.80× full-context FLOPs for GAIA and SimpleQA, respectively, as compression ratios reach 1.91× and 1.33×.The reported relationship tracks verifier prefill reduction with context shortening.
  • End-to-end agentic loops: With a Qwen3-VL-2B drafter, ASYMSPEC reaches 23.0% on the full n=165 split, but falls below the Floor on the file-attachment subset.The web subset gains +7.1 pp over Floor, while limited text capacity constrains file-attachment recovery.

6 Ablations

The ablations examine speculation depth, mechanism components, compute efficiency, and robustness across compressors, hyperparameters, drafter capacities, and model families. Results support K=2 as the default, confirm the necessity of CDA and δ-fusion, and show that recovery depends on compression severity and adequate drafter capacity.

  • 6.1 Speculation depth (K): K=2 provides the most stable accuracy–efficiency trade-off for text benchmarks, while MathVista is the exception where K=4 performs better.MultiChallenge degrades at K=4, API-Bank is flat across K, and LongBench gains only marginally before saturating.
  • 6.2 Mechanism ablation: Replacing same-model δ=a−b with raw augmented logits costs 2.8 points, while the two-model SCD-style contrast loses 11.7 points.These comparisons identify the same-model cross-context construction as the informative δ source.
  • 6.2 Mechanism ablation: δ-fusion redirects API-Bank outputs toward schema details present only in the full specification, including seconds and structured health_data fields.The CDA gate accepts ordinary draft tokens, while δ-fusion acts at rejection points where the compressed verifier diverges from the schema.
  • 6.4 Robustness: Performance remains flat across β ∈[1.0, 2.0] and γ ∈[0.4, 0.7], requiring no per-dataset calibration.Across compressors, ASYMSPEC yields 63–70% Floor–Ceiling recovery; models ≤0.6B fail to extract reliable signals, while ≥1.7B is the practical minimum.
  • 6.4 Robustness: A Qwen3-4B drafter raises the Llama-3.3-70B compressed-context Floor from 50.6 to 58.4 F1 in a cross-family pairing.Cross-family evaluation uses vocabulary and logit-space alignment over shared string-identical tokens.

7 Conclusion

ASYMSPEC breaks standard speculative decoding’s shared-context constraint by letting a full-context drafter steer a compressed-context verifier through contrastive δ-fusion. It recovers near-ceiling accuracy at a fraction of compute, with recovery increasing as compression becomes more severe.

  • 7 Conclusion: ASYMSPEC lets a lightweight full-context drafter steer a compressed-context verifier via contrastive δ-fusion.This breaks the symmetric context constraint of standard speculative decoding.
  • 7 Conclusion: ASYMSPEC recovers approximately 90% of full-context performance using only 0.2–0.3× the FLOPs on text tasks.The reported recovery is near-ceiling accuracy at a fraction of compute cost.
  • 7 Conclusion: Accuracy gain scales monotonically with compression-loss severity, providing a criterion for when asymmetric steering is warranted in agentic pipelines.The conclusion links greater compression loss with greater recovery benefit.

Limitations

ASYMSPEC’s recovery is bounded by the information retained after compression and by drafter capacity. Its broader deployment also depends on vocabulary alignment, verifier-logit access, deterministic decoding, and complementary system optimizations.

  • Recovery is bounded by information retained in the compressed view and the drafter’s capacity to extract it.
  • Cross-modal recovery is constrained by modality-translation fidelity, such as image-to-caption quality.The paper identifies richer multimodal drafters that process raw pixels as future work.
  • Cross-family δ-fusion requires explicit vocabulary and logit-space alignment, and recovery varies across model pairs.
  • ASYMSPEC requires verifier logits and therefore does not apply to proprietary APIs exposing only generated text.
  • Evaluation focuses on deterministic decoding because stochastic sampling degrades reproducible, parsable agent outputs such as JSON and tool calls.
  • End-to-end wall-clock latency also includes tool execution and network I/O, so system-level optimizations remain complementary.

B Design space and benchmark details

The evaluation fixes model and decoding choices while varying benchmark-specific compression, harnesses, drafter configurations, and cross-family alignment. It also documents multimodal and agentic setups and implementation checks.

  • Models and hyperparameters: The verifier is Qwen3-32B, with Qwen3-0.6B/1.7B/4B text drafters and Qwen3-VL-2B-Instruct for multimodal reasoning.Primary runs use bf16, greedy decoding (τ=0), disabled thinking mode, and default K=2.
  • Per-benchmark compression: Compression is applied only to verifier inputs, using benchmark-native summaries, retained turns, API signatures, or captions.Reported token reductions include 8.1× for LongBench, 7.6× for MultiChallenge, and 7.4× for API-Bank.
  • Agentic setup: GAIA and SimpleQA use deterministic smolagents ReAct harnesses with cached tools; GAIA includes web-only and file/image-attachment configurations.
  • Agentic setup: The VL-drafter rerun keeps tools, prompts, step bounds, compression, K=2, and β=1.0 unchanged while routing attachments into the vision tower.
  • Benchmark details: API-Bank evaluates 200 Method A instances using api-acc, the fraction of calls matching the gold API name and argument set.
  • Cross-family alignment: Heterogeneous model pairs map drafter-vocabulary δ through 109,566 string-identical tokens, while committed tokens remain restricted to the shared set and paired special tokens.
  • Implementation validation: The MathVista image result collapses to 30.5% without patches, compared with 53.0% with patches.

F Throughput Table

Throughput results show that ASYMSPEC targets a lower-cost operating point with near-Ceiling accuracy, while realized speed depends on hardware, acceptance, and cross-modal overhead.

  • Throughput mechanism: Fixed-γ throughput exceeds the compressed baseline when successful drafting amortizes the third forward over K tokens and skips verifier autoregressive steps.
  • Operating points: ASYMSPEC reaches near-Ceiling accuracy at approximately 0.2–0.7× compute, whereas SD-full reaches Ceiling accuracy at full-context cost.Throughputs are comparable across LongBench, MultiChallenge, and API-Bank, differentiating the methods primarily by cost regime.
  • Cross-modal overhead: MathVista is the slowest regime because each request includes a vision-tower forward, adding necessary cross-modal overhead.
  • Hardware effects: 1.3–1.7× throughput aligns with memory-bandwidth-bound decoding, so FLOPs reduction does not translate linearly into wall-clock speed.FLOPs measure compute/energy savings, whereas throughput measures realized latency gains.
  • Acceptance dynamics: Acceptance remains high under asymmetric context, with AR 0.78–0.92 and MAL 2.6–2.8 across available K+1=3 positions.These levels persist through the multi-turn GAIA loop with online recompression.

G Additional ablation grids

Additional ablations examine CDA’s divergence choice and speculation depth, supporting the default settings while identifying benchmark-specific exceptions.

  • Divergence choice: JSD and KL produce identical API-Bank decisions, while LongBench differs by at most 1.5 F1; JSD is chosen for its universal bound.API-Bank records 339/534 accepted at K=2 and 334/534 at K=4 for both gates.

H Extended drafter-size sweep

The extended sweep finds that CDA’s main advantage is robust, tuning-free behavior, while its accuracy gains depend on task compression and drafter size. On near-lossless MultiChallenge, AsymSpec provides no improvement and is mildly below the compressed floor.

  • CDA requires ≥1.7B drafters to reliably extract context-gain signals.The sweep identifies drafter capacity as a reliability threshold for recovering information from the full context.
  • CDA remains flat across the tested γ threshold, while β=1.0 is a stable midpoint for LongBench.LongBench F1 is flat for β ∈{1.0, 2.0}, whereas β=0.5 underperforms by 3–3.6 F1 at K=2; acceptance rate and mean accepted length remain flat.
  • At 4B, CDA matches or exceeds tuned-λ performance on judge-independent metrics.It is identical on API-Bank and improves LongBench F1 by +0.7/+1.8 at K=2/4, respectively.
  • On API-Bank, tuned-λ and CDA are numerically identical across all six sweep cells.At K=2, the best relation is inverted-U with 1.7B as the sweet spot; at K=4, the best cell is 4B with both methods at 63.7.
  • On MultiChallenge, every method difference stays within the ≤3 pp headroom, and AsymSpec sits mildly below the compressed floor.The task is near-lossless, so the sweep finds no consistent ordering or accuracy improvement.
Loading 2608.26004v1…