PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu, Zhijiang Guo

arXiv:2610.21463v1cs.CLcs.AIcs.LG

TL;DR

Long tool-use trajectories make strong-model decoding costly, while many steps could be drafted by a smaller model and existing collaboration methods still leave the strong model generating each step. SpecAgent has a weak model draft grounded segments for semantic review by the strong model, which commits valid prefixes and repairs rejected steps. Across the reported benchmarks, it retains most strong-model quality while reducing memory traffic and wall time.

  • Problem

    Existing sharing approaches keep the Strong model generating steps, even though many routine steps could be produced by a smaller model and long trajectories make Strong decoding costly.

    Source: §1 · pp. 1–2

  • Method

    SpecAgent has the Weak model execute read-only calls while drafting and hold the first state-changing action for Strong-model review of the longest semantically valid prefix.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across τ2-Bench, AutomationBench, VitaBench, and MCP-Atlas, SpecAgent retains most Strong-only quality while reducing memory traffic, wall time, or Strong-model token generation.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results suggest a Strong model can review grounded drafts rather than author every agent step, retaining most quality while cutting latency and memory traffic.

    Source: §6 · p. 11

    The latency benefit requires Weak drafting steps to be faster than Strong steps, and decode-bound assumptions may fail at larger batches.

    Source: §3.3 · p. 6, §5 · p. 11

Abstract

from arXiv · show

Large language model agents solve tasks through long sequences of tool calls, and the prevailing design has a single strong model generate every step, although many steps are routine enough for a much smaller model. Existing ways of sharing this work keep the strong model as the generator: routers and collaborative agents decide which model handles each query or step, and speculative methods check a small model's drafts one step at a time, usually against the strong model's own action. We introduce SpecAgent, a collaboration protocol that turns the strong model from the author of each step into a reviewer of steps drafted by a weak model. The weak model executes read-only tool calls as it drafts, so that every step is grounded in real observations, and holds the first action that would modify the environment. The strong model then verifies the drafted segment in a single prefill pass, commits the longest prefix it judges semantically valid, and repairs the first rejected step. Because verification replaces autoregressive decoding with parallel prefill, the strong model decodes only short verdicts and occasional repairs rather than every step. On $\tau^2$-Bench, SpecAgent comes within one point of the strong model's success rate while roughly halving memory traffic and wall time, and on AutomationBench and VitaBench it recovers 44-75% of the quality gap while lowering both. In an edge-cloud deployment on MCP-Atlas with three cloud models, it closes 53-75% of the weak-strong quality gap while the strong model generates at most half as many tokens. These results suggest that a strong model need not author every step of an agent: reviewing grounded drafts retains most of its quality while cutting both latency and memory traffic.

1. Introduction

Long tool-use trajectories make strong-model decoding costly, even though many steps are routine enough for a smaller model. SpecAgent lets a weak model draft grounded steps for strong-model review, retaining most strong-model quality while reducing memory traffic and wall time.

1. Introduction

  • Long trajectories make Strong-model token-by-token decoding a major serving cost, although routine lookups and reads could often be produced by a much smaller model.Representative frontier-agent tasks span tens of turns in customer service to as many as a hundred in everyday-life services.Source: §1 · p. 1, Table 1 · p. 1
  • Routers, collaborative agents, and most speculative agents still leave the Strong model generating every step or checking drafts against its own action.Exact matching can discard valid progress because different queries, lookup orders, or arguments may reach a valid solution.Source: §1 · p. 2
  • SpecAgent has the Weak model draft grounded steps while the Strong model commits the longest semantically valid prefix and repairs the first rejected step.The Weak model executes read-only calls but holds the first state-changing action for review.Source: §1 · p. 2
  • The evaluation compares complete quality–resource operating points against single-model agents and reports compute, memory traffic, and wall time as separate axes.The contribution also includes a cost model and analysis of how acceptance affects savings.Source: §1 · p. 2
  • On τ2-Bench, SpecAgent comes within one point of Strong-only success, with memory traffic at 43–57% and wall time at 41–54% of Strong-only; it also recovers 44% and 75% of the quality gap on AutomationBench and VitaBench.On MCP-Atlas, it closes 53–75% of the Weak–Strong quality gap while the Strong model generates at most half as many tokens.Source: §1 · p. 2

2. Related Work

Prior agent-sharing methods generally keep a model generating each step, while speculative approaches often retain drafts only when they match the Strong model’s own action. SpecAgent instead reviews grounded multi-step segments and exploits parallel prefill to reduce Strong-model decoding.

2. Related Work

  • Speculative decoding drafts tokens for parallel verification while preserving the model’s output distribution, and it can complement SpecAgent by accelerating each model’s own decoding.The cited works are Chen et al. (2023a) and Leviathan et al. (2023).Source: §2 · pp. 2–3
  • Unlike prior agent speculation that keeps drafts agreeing with the large model’s own step, SpecAgent verifies grounded multi-step segments and checks before environment-changing actions.Prior agent methods include Guan et al. (2025), Hooper et al. (2026), Hua et al. (2024), Ji et al. (2026), Nichols et al. (2025), and Ye et al. (2026).Source: §2 · p. 3, Figure 1 · p. 3
  • Routers and trajectory-level handoffs still let the model in control generate each step; SpecAgent instead lets the Strong model decide how much drafted progress to keep and generate repairs only.Prefill processes many tokens per weight read, whereas decoding rereads weights for each token.Source: §2 · p. 3

3. SpecAgent

SpecAgent separates drafting from committing: the Weak model can execute reversible reads, while the Strong model reviews drafts at the boundary where an error could change the environment. It accepts valid prefixes semantically, repairs rejected steps, and uses parallel prefill to reduce Strong-model decoding costs.

3.1. Problem Formulation

  • Reads leave the environment unchanged and can be removed from history, whereas state-changing errors cannot be undone; review is therefore placed at the irreversible boundary.The task is modeled as partially observable, with actions producing observations and policies conditioning on task and history.Source: §3.1 · p. 4

3.2. Speculative Execution Protocol

  • The Weak model executes read-only drafts using real observations, holds the first state-changing action, and may commit a chunk of reads without review when it reaches the chunk limit.The Strong model reviews the history and chunk in a single pass, accepting the longest semantically valid prefix rather than requiring agreement with its own proposed action.Source: §3.2 · pp. 4–5
  • A rejection discards the suffix; the verifier can fuse a replacement into its verdict, with a separate Strong corrective turn reserved for repairs that cannot use that replacement.The next draft starts from committed history, and a Strong-authored repair is committed without another review.Source: §3.2 · p. 5

3.3. Efficiency Analysis

  • Prefill processes many tokens per weight read, while autoregressive decoding rereads active weights per token; the cost model assumes Strong serving remains decode-bound and models per-task acceptance independently across a chunk.The comparison also accounts for verdict cost, the rate of fused versus separate repairs, and the Weak model’s drafting cost.Source: §3.3 · pp. 5–6
  • Strong-model time and memory savings improve as acceptance rises, but end-to-end speedup requires Weak drafting steps to be cheaper than Strong-only steps.Read-only chunks committed without review can further lower the Strong cost ratio.Source: §3.3 · pp. 5–6

4. Experiments

The experiments evaluate SpecAgent against single-model and collaborative baselines across customer-service, business-workflow, delivery, and edge–cloud tool-use tasks. They assess quality alongside resource use, finding strong quality retention with lower memory traffic and wall time on several benchmarks, while edge–cloud gains are in remote usage and cost rather than latency.

4.1. Setup

  • Evaluation covers τ2-Bench Airline and Retail, AutomationBench, and VitaBench, using Pass@1 for quality and per-trajectory resource means that include all agent-model requests.The main comparison uses Qwen3.5-4B as Weak and Qwen3.5-397B-A17B as Strong; MCP-Atlas is reported separately.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2. Main Results

  • 86.0 vs. 86.0 Pass@1 on τ2 Airline and 87.7 vs. 88.6 on Retail for SpecAgent vs. Strong-only; it recovers 44% and 75% of the quality gap on AutomationBench and VitaBench.AutomationBench Pass@1 is 10.5 between Weak-only 6.8 and Strong-only 15.2; VitaBench Pass@1 is 44.5 between 34.0 and 48.0.Source: Table 2 · p. 7, §4.2 · p. 7
  • Against Strong-only, HBM-read falls to 43–45%/53–57%/78–84%/59–65% and wall time to 41%/54%/57%/69% across Airline/Retail/AutomationBench/VitaBench, while FLOPs rise.FLOPs rise by 30%, 38%, 38%, and 87%, respectively.Source: Table 2 · p. 7, §4.2 · p. 8
  • On τ2, SpecAgent matches or exceeds AgentCollab quality with HBM-read lower by 1.6–1.9× and wall time lower by 2.6–3.1×; it alone beats Strong-only on both resources across benchmarks.On VitaBench, AgentCollab reaches 50.0 versus Strong-only 48.0 and SpecAgent 44.5, but uses 1.4× SpecAgent’s HBM-read and wall time.Source: Table 2 · p. 7, §4.2 · p. 8

4.3. Edge–Cloud Deployment: MCP-Atlas

  • Across three MCP-Atlas cloud Strong models, SpecAgent recovers 53–75% of the Weak–Strong gap, uses 36–50% of Strong-only output tokens, and costs 59–81% as much, without reducing wall time.The local Weak model takes about as long per task as the hosted Strong models: 109 seconds versus 105–151 seconds.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5. Analysis and Discussion

The analysis relates measured Strong-model costs to a cost model, showing how verification changes decoding, compute, memory traffic, and time. It examines when quality is recovered, which design choices matter, and where the trade-off may not pay.

Why time falls while FLOPs rise

  • 81–85% of SpecAgent’s FLOPs come from Strong verification and repair, versus 39–66% of HBM-read; Weak drafting accounts for 15–19% of FLOPs and 34–61% of HBM-read.On VitaBench, about 28.4k uncached prefill tokens per task (12.5k for Strong-only) accompany about 0.21 PFLOP of removed decoding; Strong generation falls to 0.16–0.34 of Strong-only wall time.Source: §5 · p. 10, Figure 3 · p. 10

When verification recovers Strong quality

  • When Weak-only is unreliable, SpecAgent stays close to Strong-only on the small τ2 subsets; when Weak is much weaker, repairs grow and less of the quality gap is recovered.The reported examples are Airline 70.4 vs. 66.7 and Retail 74.2 vs. 75.8; the authors caution that these subsets are small.Source: §5 · p. 10

Strong decoding explains the saving

  • On τ2, Strong reviews about once every seven committed steps, decoded tokens fall to 17–30% of Strong-only, and observed HBM-read share matches prediction within 0.01.Table 4 reports the predicted and observed Strong decoding and generation-time ratios relative to Strong-only.Source: §5 · p. 9, Table 4 · p. 10

Design choices

  • On Retail, reviewing every chunk more than triples Strong calls and raises HBM-read by 63–79% and wall time to 2.5–3.0× past Strong-only; exact-step acceptance lowers Pass@1 to 82.5 and raises FLOPs by 70%.Without fused replacement, every rejection becomes a separate Strong turn and Pass@1 is 82.5; the authors attribute quality retention to semantic acceptance preserving valid Weak progress.Source: §5 · p. 11, Figure 4 · p. 11

Across Strong models

  • 50%, 46%, and 36% are SpecAgent’s Strong token shares for Strong-only trajectories of 3.6k, 4.6k, and 10.6k tokens, respectively, across the three MCP-Atlas Strong models.API cost falls less, to 59–81%, because each review resends context as input.Source: §5 · p. 11

Deployment regimes

  • The latency trade-off requires Weak steps to be faster than Strong steps; above about 37 sequences on H20, decoding becomes compute-bound and the 30–87% FLOP increase would begin to cost time.The edge–cloud deployment does not meet the Weak-faster-than-Strong condition.Source: §5 · p. 11

6. Conclusion

SpecAgent makes the Weak model the drafter and the Strong model the reviewer of grounded agent steps. Across the evaluated benchmarks, it retains most Strong-only quality while reducing memory traffic, wall time, or remote cost.

6. Conclusion

  • SpecAgent lets the Weak model draft grounded trajectories while the Strong model reviews them, keeping valid progress and stopping before state-changing actions.The conclusion says stronger Weak drafters and better-calibrated verifiers could yield further memory and latency savings without changing the protocol.Source: §6 · p. 11

A. Experimental Setup

The setup specifies benchmark subsets and trajectory counts, model-serving configurations, and the baselines used for comparison.

A. Experimental Setup

  • τ²-Bench uses Airline and Retail tasks, while AutomationBench and VitaBench use specified public-task subsets and multiple trajectories where noted.The benchmarks use 50 Airline tasks and 114 Retail tasks with three trajectories per task, 600 AutomationBench tasks with one trajectory each, and 100 VitaBench Delivery tasks with two trajectories each; the AutomationBench warm-up domain is excluded.Source: A. Experimental Setup · p. 15
  • MCP-Atlas evaluation uses 101 public tasks without private-key server requirements and excludes six tasks timed out by any method.The selected subset is somewhat easier than the full benchmark, so absolute scores are not comparable to the public leaderboard; methods are compared on the same tasks, and Gemini 3.1 Pro grades responses, counting a task solved at 75% graded-claim coverage.Source: A. Experimental Setup · p. 15
  • Qwen3.5-4B serves as Weak and Qwen3.5-397B-A17B as Strong on the three non-MCP benchmarks, with separate deployments.On MCP-Atlas, Weak runs locally on an RTX 5090 and Strong models are called through provider APIs; the passage also specifies concurrency, step limits, sampling, and thinking settings.Source: A. Experimental Setup · p. 15
  • Baselines include Weak-only and Strong-only, AgentCollab’s scheduled Strong/Weak control with progress checks and escalation, and single-model comparisons on MCP-Atlas.AgentCollab begins with two Strong steps, then uses Weak actions and TRUE/FALSE checks; a failed check triggers a Strong interval of min(Bmax, B0 + kl), with B0=2, k=2, and Bmax=6. On τ², checks are separate calls because agents lack a think tool; Speculative Actions is described in Appendix E.Source: A. Experimental Setup · p. 15

B. Metric Definitions

This section defines how the paper measures task quality and estimates model-request resources, then aggregates those measures across trajectories. It also specifies assumptions and exclusions that bound the resource estimates.

B.1. Task Quality

  • Paired Pass@1 differences are bootstrapped by resampling tasks with replacement, retaining all trials for each sampled task, and reporting the 2.5th and 97.5th percentiles.Failed or incomplete trials remain in the denominator; trial counts differ by benchmark.Source: B. Metric Definitions · p. 15, §B.1 · p. 16

B.2. Request Shape and Prefix Reuse

  • Each request is characterized by input and output tokens, reused input prefixes, newly prefilled tokens, decode forwards, and output logits.Prefix reuse is modeled within sessions that share task, trial, model, role, and serving replica; it does not cross sessions.Source: §B.2 · p. 16, Table 5 · p. 16

B.3. Estimated FLOPs

  • FLOPs are estimated separately for prefill, decode, and the output head, using architecture-specific attention and MLP terms.The accounting excludes several operations and inter-GPU communication; MCP-Atlas models with different attention use adjusted attention terms.Source: §B.3 · pp. 16–17

B.4. Estimated HBM-Read

  • HBM-read estimates bytes read per request at batch size one, including weight reads and decoding cache, recurrent-state, and activation terms.It is an analytical estimate, not a hardware-counter measurement; a Weak-model spot check agreed with estimated weight and KV reads within 5% on one held-out request and 10% across short request shapes, while Strong was not profiled.Source: §B.4 · p. 18

B.5. Aggregation, GPU Time, Wall Time, and Calls

  • Per-method resource totals are averaged over official trajectories, while GPU-seconds charge each request its full duration on its serving GPUs as if it ran alone.That GPU-time measure is an upper bound on actual occupancy; wall time is measured per trajectory, and calls count successful generation requests split by model.Source: §B.5 · p. 18

B.6. Supplementary Results

The supplementary results report pass@3 and pass3 for τ²-Bench single-model baselines over the same three trials per task.

B.6. Supplementary Results

  • The supplementary table reports pass@3 and pass3 for τ²-Bench single-model baselines, using three trials per task.Pass@3 requires at least one success per task, whereas pass3 requires all three; scores are percentages.Source: §B.6 · p. 18, Table 8 · p. 19

C. Proofs for Section 3.3

The proofs analyze a cycle in which a Weak model drafts a chunk, a Strong model verifies it, and the cycle either commits the chunk or repairs after rejection. They derive expected progress and compare Strong cost and elapsed time with Strong-only execution.

C. Proofs for Section 3.3

  • A cycle commits at least k steps when its first k−1 drafted steps are accepted, giving Pr(S≥k)=p^(k−1) under Assumption 2.A repair is needed with probability 1−p^n when the first rejected step occurs before the chunk ends.Source: C. Proofs for Section 3.3 · p. 19
  • Expected Strong cost per committed step includes a verdict and a repair weighted by the probability of rejection, then compares that cost with Strong-only decoding.Across tasks, the ratio of total costs weights each task's ratio by its number of steps.Source: C. Proofs for Section 3.3 · p. 19
  • Expected cycle time includes Weak drafting time, verdict time, and expected repair time, divided by expected committed steps.The proof expresses the resulting time ratio across tasks using the same step-count weighting as the cost ratio.Source: C. Proofs for Section 3.3 · p. 19

D. Benchmark Adapters

Benchmark adapters bind shared protocol logic to each benchmark's tools and policies. Their choices specify which actions are held for review, what context the verifier receives, and how it retains valid progress and handles rejection.

D. Benchmark Adapters

  • Adapters accept a fused replacement only when it matches the tool schema and executes in the environment; otherwise, the Strong model takes a repair turn.The adapter binds benchmark-specific components to the shared protocol.Source: D. Benchmark Adapters · p. 19, Table 9 · p. 20
  • The protocol holds state-changing actions for review, but executed reads may leave external traces even when rejected; simulated-user replies are held only on AutomationBench and MCP-Atlas.On τ²-Bench, holding user replies more than triples Strong calls without improving quality; MCP-Atlas and VitaBench tool outputs may be truncated in the verifier's view.Source: D. Benchmark Adapters · p. 20

D.1. Prompts

  • Prompt instructions distinguish already-executed reads from pending mutations: reads are rejected only for invalid tools or required arguments, while pending mutations are judged as if they will run.The customer-service prompt also requires policy preconditions and explicit customer confirmation where applicable.Source: §D.1 · p. 21
  • The verifier retains the longest justified prefix, judging each action against evidence available before it and stopping at invalid tools, arguments, mutations, or incomplete replies.For partial acceptance or rejection, it supplies an evidence-based reason and may propose one schema-valid replacement tool action.Source: §D.1 · pp. 21–22
  • The Strong continuation receives the retained dialogue and review diagnosis, checks material claims against observations, and produces one useful next step.It should prefer a missing lookup or required confirmation over a speculative mutation and must follow the supplied schema.Source: §D.1 · p. 22

E. Speculative Actions on 𝜏2-Bench

The released Speculative Actions implementation was ported to 𝜏2-Bench, where it underperformed Weak-only on Airline and Retail and used more Strong calls than Strong-only. Its background calls run in the live environment without read-only filtering or validation against the real user message.

E. Speculative Actions on 𝜏2-Bench

  • Unlike Speculative Actions’ latency-focused approach, which predicts the user’s next query and needed API calls, this section reports task success for the ported implementation.Speculative Actions describes validating predictions when the real query arrives and restricting speculative side effects to idempotent, reversible, or sandboxed actions.Source: E. Speculative Actions on 𝜏2-Bench · p. 22
  • The port has the Strong model act at every step while a Weak model guesses the next user message and a Strong decider continues for up to five tool steps.Background tool calls execute in the live environment without a read-only filter, and their results are added without checking the guess against the real user message.Source: E. Speculative Actions on 𝜏2-Bench · pp. 22–23
  • 72.0% Pass@1 on Airline and 48.5% on Retail fell below Weak-only’s 80.0% and 80.4%, respectively.These results characterize the released implementation, not the paper’s guarded design.Source: E. Speculative Actions on 𝜏2-Bench · p. 23
  • 17.4 Strong calls per Airline trajectory and 17.1 per Retail trajectory exceeded Strong-only’s 13.8 and 13.1, respectively.The extra calls come from a background decider running on top of the Strong actor; the authors did not diagnose the quality drop further.Source: E. Speculative Actions on 𝜏2-Bench · p. 23

Related papers

Loading 2610.21463v1…