Source-linked AI summary
Speculative Macro Commit for Faster Tool-Using Agents
Zeyu Liu, Souvik Kundu, Peter A. Beerel
TL;DR
Tool-using agents suffer latency from serial action–observation turns that token-level speculation cannot eliminate. SMC uses a concurrent authoritative actor and speculative drafter to execute and reuse mined multi-action patterns, achieving substantial latency reductions while preserving quality on Telecom and accepting a small AppWorld completion tradeoff.
Problem
Serial tool-agent action–observation turns create wall-clock delays beyond model inference, while token-level speculation remains confined to individual model calls.
Method
SMC mines recurring multi-action patterns and commits pre-executed draft steps and observations after the actor confirms the first drafted action.
Results
SMC matches sequential accuracy on τ^2-Bench Telecom while reducing latency 10.23% versus SA and 18.59% versus sequential execution; on AppWorld it reduces wall time 7.64% versus SA and 44.93% versus sequential execution with a small completion drop.
Takeaways & Limitations
Multi-step speculative execution can reduce agent latency beyond single-step action speculation when patterns are predictable, pre-executed, and accepted by online checks.
Abstract
from arXiv · showhide
Tool-using LLM agents spend wall-clock time not only on model inference but also in serial action--observation turns, where each tool call, environment transition, and observation can delay subsequent decisions. We introduce \textbf{Speculative Macro Commit} (SMC), a runtime mechanism for a two-tier agent system: a large authoritative actor model produces the official trajectory, while a faster speculative drafter model continuously predicts and executes future action chains on an isolated environment snapshot. SMC mines recurring multi-action skeletons from training traces and stores them in a macro library used to match against action chains predicted by the drafter at runtime. When the actor's next tool call matches the first drafted action, SMC commits the remaining pre-executed draft steps, together with their observations, to the official trajectory. Using Qwen3.5-27B INT4 as the authoritative actor model and Qwen3.5-4B as the speculative drafter model, SMC matches the sequential agent's overall accuracy while reducing latency by 10.23\% over the Speculative Actions (SA) baseline and 18.59\% over sequential execution on the $τ^2$-Bench Telecom subset. On AppWorld, SMC reduces wall time by 7.7\% over SA baseline and 44.9\% over sequential execution, with a small reduction in task completion. Overall, SMC provides a practical way to reuse multi-step speculative execution and reduce agent latency beyond single-step speculative actions. Our code is publicly available \href{https://github.com/zeyuliu1037/speculative-macro-commit}{\textcolor{magenta}{here}}.
1. INTRODUCTION
Tool-using agents incur substantial wall-clock delay from serial action–observation turns and tool execution, which token-level speculation cannot remove. SMC addresses this gap by reusing guarded multi-action drafts, preserving quality while reducing latency.
- Motivation: Sequential action–observation turns create a wall-clock bottleneck beyond model throughput, with tasks waiting through many dependent steps.Tool execution can account for 35% to 61% of total latency in representative agents.
- Method: SMC mines recurring multi-action patterns and commits pre-executed draft steps after the authoritative actor confirms the first action.The committed steps include their returned observations and skip corresponding large-model calls and environment delays.
- Runtime: SMC runs an authoritative actor and faster speculative drafter concurrently throughout long-horizon tool-agent tasks.The executor measures wall-clock latency, lets the drafter run ahead, and falls back to ordinary execution when speculation cannot be used.
- Results: 10.23% lower latency than SA and 18.59% lower than sequential execution were achieved on the τ^2-Bench Telecom subset.SMC matched the single-model baseline’s reward in this evaluation.
- Results: On AppWorld, SMC reduced wall time by 7.64% versus SA, with a small accuracy tradeoff.The reported comparison is against the AppWorld SA configuration.
2. RELATED WORK
Prior work reduces latency within model calls, through tool speculation, serving improvements, or model-visible workflow composites. SMC instead performs guarded multi-step runtime commits while keeping mined workflows hidden from the actor model.
- Speculative execution: Token speculative decoding accelerates individual model calls, but does not remove dependencies between successive tool-agent invocations.Each tool result must return before the next prompt can be formed.
- Speculative execution: SA reuses speculative work one action at a time, whereas SMC commits several already executed draft steps after verifying an anchor action.SMC trades SA’s lossless one-step verification for a guarded multi-step approximation.
- Speculative tool execution systems: PASTE speculates tool calls, while ThunderAgent and KVFlow target program-aware serving and prefix or cache management without changing trajectory actions.These systems address complementary parts of the serial LLM–tool loop.
- Workflow compression and meta-tools: AWO exposes mined tool-call composites as model-visible meta-tools, whereas SMC keeps mined steps as hidden executor state.SMC commits them only after the actor’s anchor call is verified.
3. SPECULATIVE MACRO COMMIT
SMC augments a concurrent actor–drafter executor with mined macros, isolated draft execution, runtime alignment, and guarded multi-step commits. The executor commits matched draft suffixes only after the actor confirms the anchor call and online checks pass.
- Executor Roles and State: The executor maintains committed history H and live state E, while the drafter executes future action–observation chains in isolated draft state Eᵢ.The authoritative actor defines real execution; the drafter runs ahead from the committed history.
- Executor Roles and State: A draft chain Q contains drafted tool calls and their results, and macro commits append selected executed pairs to H while advancing E.Without a macro commit, the executor grows H one authoritative step at a time.
- Executor Roles and State: The SA-only baseline retains concurrent actor–drafter execution but commits only the first matching drafted action and result.It provides the matched runtime comparison for SMC.
- Macro Mining: Candidate macros are mined from successful traces, normalized with argument slots, and filtered using occurrence counts and conservative drafter reliability.Candidates must satisfy nₘ ≥ nₘᵢₙ and pₘ ≥ τ; filtering is separate from runtime commitment.
- Macro Commit Rule: A runtime alignment matches a macro’s history suffix to draft-prefix actions, then waits for the actor to match q₁ before committing q₂ through qₗ.The minimum skipped-depth condition requires ℓ−1 ≥ Lₘᵢₙ.
- Macro Commit Rule: Online checks discard stale work and enforce benchmark-specific safety before committed draft steps are appended to H and E advances.In AppWorld, irreversible or unknown API calls are rejected and forkable mutations require matching live-state replay.
- Macro Commit Rule: The actor never emits a macro call; macros remain runtime decisions over already executed drafter work rather than new model-visible actions.This distinguishes SMC from registered meta-tools.
4. EXPERIMENTAL RESULTS
Across two full tool-agent benchmarks, SMC reduces wall time by reusing pre-executed speculative action chains while preserving τ 2-Bench outcomes and incurring a small AppWorld completion trade-off.
- 4.1. Setup: SMC uses Qwen3.5-27B as the authoritative actor and Qwen3.5-4B as the speculative drafter in a three-GPU runtime.The actor, replica, and drafter share the speculative serving configuration; SA and SMC differ only in whether macro commit is enabled.
- 4.1. Setup: SMC commits matched, pre-executed draft suffixes after the actor agrees with the first action, skipping corresponding actor calls and tool-result waits.The executor discards draft work when the first action does not match and restarts speculation from the committed history.
- 4.2. Main Results: On τ 2-Bench, SMC lowers latency to 22.47 seconds per task, 18.59% below sequential execution and 10.23% below SA, with unchanged task outcomes.SMC matches the sequential baseline on all 2,285 tasks and fixes one task relative to SA without regressions.
- 4.2. Main Results: On AppWorld, SMC reaches 195.9 seconds per task, 44.93% below sequential execution and 7.64% below SA, while TGC falls from 70/168 to 68/168 tasks.This result contrasts with τ 2-Bench, where the additional speedup does not change task outcomes.
- 4.3. Analysis of the AppWorld Gain: SMC commits macros on 62.0% of AppWorld tasks and achieves 3.81% skip density, close to Telecom’s 3.91%, indicating similar skipped-work coverage.Controlled AppWorld slices show a 13.5% gain on same-outcome tasks and a 10.7% gain when neither run emits an NTC step.
5. MECHANISM ABLATIONS
The ablations show that hidden runtime reuse requires conservative online validation, and that useful speedups depend on sufficiently deep commits aligned with the critical path. Simpler interfaces either fail to trigger reuse reliably or reduce accuracy, while guarded SMC preserves committed outcomes in the audit.
- Ablation requirements: SMC requires hidden reuse, aggressive candidate filtering, and enough skipped actor calls to produce useful latency gains.The main results identify these three conditions as built into the commit rule.
- Interface ablations: Registered meta-tools increase latency and lower accuracy because the actor almost never selects mined routines.Exposing mined routines as ordinary tools creates an additional tool-selection problem instead of reliably skipping actor calls.
- Interface ablations: 11.34% latency reduction from passive committing comes with accuracy falling from 99.52% to 96.48%.The result shows that hidden reuse can skip actor calls, but requires anchoring committed steps to the authoritative trajectory.
- Commit precision: 90.4% precision follows the full staged rule, while all committed events preserve outcomes in the held-out audit.Precision rises from 34.6% for library matches alone to 70.6% after draft execution, 87.9% after anchor verification, and 90.4% with the depth guard.
- Critical-path depth: 1.64% slower than SA, the legacy runtime shows that raw macro count and skipped-step totals do not guarantee end-to-end speedup.The final runtime suppresses shallow opportunities and commits only sufficiently deep steps already executed on the draft branch, achieving 10.23% faster than SA.
6. CONCLUSION
SMC extends speculative action execution by committing several already executed draft steps after a verified anchor call. The conclusion reports quality-preserving latency gains on τ 2 Telecom and larger AppWorld wall-time gains with a small completion tradeoff, provided commits pass conservative runtime checks.
- Conclusion: SMC extends speculative action execution from single-step reuse to committing several already executed draft steps after a verified anchor call.The mechanism uses conservative offline mining, anchor verification, a minimum skipped depth, and online state and argument checks.
- Conclusion: On τ 2 Telecom, SMC provides a quality-preserving latency gain beyond sequential execution and the equal-hardware SA baseline.On AppWorld, it provides a larger wall-time gain with a small completion tradeoff and larger gains on same-accuracy slices.
- Conclusion: Workflow macros help only when they are predictable enough to mine, already executed by the drafter, and accepted by online checks.The conclusion frames hidden runtime state as useful only when these conditions hold.