SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management

Motivation and gap

• Prior tool-call speculation and retrieval-based drafting address efficiency, but TOOLSPEC's datastore is built once and frozen, leaving evolving cross-session recurrence unaddressed.The paper frames persistent tool-call memory as a lifecycle problem and evaluates write-back, partitioning, eviction, and execution gating.Source: §1 · pp. 1–2

Results

• +27% post-warmup acceptance gain over the frozen store and up to 1.74× wall-clock decode speedup over vanilla decoding are reported for SPECMEM.The acceptance gap reaches +46% by the final session; across benchmarks it ranges from +11% to +27% on τ 2-bench domains and is +34% on Seal-Tools.Source: §1 · p. 2
• The frozen store degrades across sessions while the live store holds acceptance, and an ablation attributes essentially the entire gain to online write-back.The gap scales with the share of novel calls arriving after the store is built.Source: §1 · p. 2
• A globally shared store exposes cross-user draft provenance, while local scoping adds almost nothing to acceptance and removes that measured leakage channel.Under a global store, 14–68% of a user's drafts came from other users' histories.Source: §1 · p. 2
• A retrieval-confidence threshold cannot ensure safe speculative execution, whereas gating on declared side effects removes irreversible mistakes by construction.The passage reports that a stale draft's similarity can be indistinguishable from a correct draft's.Source: §1 · p. 2

Speculative decoding for tool-calling agents

• Speculative decoding for agents covers next tool calls, observations, on-device use, concurrent I/O, and forked, beam, and multimodal variants, but discards cross-session memory.Retrieval drafting exploits recurring, templated calls; prior approaches also include token-match caches and online neural drafters.Source: §2 · pp. 2–3

The closest prior datastore, and its gap

• TOOLSPEC retrieves historical calls with embedding similarity and a schema-aware FSM, but its store is fixed and global, without write-back, eviction, or side-effect gating.SPECMEM targets the resulting gap by studying store lifecycle and cross-session dynamics.Source: §2 · p. 3

Agent memory, caching, and tool safety

• Agent-memory systems maintain bounded, updated, user-scoped stores for knowledge or skill recall, while semantic caches reuse responses and evict stale entries.Those systems do not store speculative drafts, and stale cache hits affect correctness rather than acceptance.Source: §2 · p. 3
• Prior tool-safety work documents graded risks, while speculative-execution engines buffer side effects; related systems nonetheless assume tools are safe to execute speculatively.The cited work on privacy gates addresses a complementary axis.Source: §2 · p. 3

3 Methodology

• The methodology combines multi-session benchmark construction, a managed tool-call memory behind a speculative decoder, and a gate controlling pre-verification execution.These are the three components of the study's methodology.Source: §3 · p. 3

3.1 Preliminaries

• Speculative decoding verifies a proposed draft in one target-model pass, accepts its longest correct token prefix, and re-decodes from the first mismatch without changing output.The study focuses on how the datastore is scoped, updated, bounded, and gated as behavior evolves.Source: §3.1 · p. 3

3.2 Extending benchmarks to the deployment scenario

• Benchmark tasks are assigned to simulated users and re-issued across ordered sessions with perturbed numeric arguments, while some tasks first appear after warmup.The construction extends single-shot pools, is instantiated on BFCL and Seal-Tools, and is identical across experimental arms.Source: §3.2 · p. 3

3.3 SPECMEM: live memory management

• SPECMEM retrieves each user's nearest stored call above similarity τ=0.35, then writes every verified call back and evicts the least-recently-used entry at capacity.When a user store is empty or too dissimilar, retrieval backs off to a shared population store and then a schema-only skeleton.Source: §3.3 · p. 4

3.4 Gating speculative execution of side-effecting tools

• A schema-level safe_to_speculate flag permits pre-verification execution only for safe calls; execution of other drafted calls waits for verification.The relevant property is whether execution can commit an unrecoverable, externally visible effect, not whether the operation is idempotent; a confidence gate is compared but deemed insufficient.Source: §3.4 · p. 4

4.1 Experimental setup

• The stream assigns disjoint benchmark tasks to simulated users, reissues them across ordered sessions with perturbed numeric arguments, and introduces some tasks after warmup.The setup uses BFCL v4; mean accepted tokens, accepted fraction, and exact-match rate score drafts against the served model’s greedy output.Source: §4.1 · pp. 4–5

4.2 Acceptance results

• +27% mean accepted tokens (19.96 vs. 15.70 MAT) for SPECMEM over the frozen store after warmup on BFCL; the no-memory baseline reaches 5.92 MAT.The comparison covers sessions 1–11 across three seeds.Source: §4.2 · p. 5, Table 1 · p. 6
• Across sessions, the frozen store falls from near 21 MAT to 13.5–14.9 while SPECMEM holds at 18.9–21.5, widening the gap to +46% by the final session.The no-memory baseline stays near 5.9 MAT; the figure reports per-session BFCL results across three seeds.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3 Wall-clock results

• The reimplementation’s speedups are 1.06–1.66×, below TOOLSPEC’s reported 3.5–4.2×; the paper cautions that absolute speedups are not comparable across papers.The stated differences include served models, serving stack, external-loop overhead, and BFCL version.Source: §4.3 · p. 6
• SPECMEM matches or exceeds reimplemented TOOLSPEC speedup on all four single-session benchmarks, with retrieval and write-back overhead included in timing.Under concurrent load, effective output throughput over vanilla decoding rises from 2.9× to 3.3× as concurrency grows from 1 to 32.Source: §4.3 · p. 6, Table 2 · p. 6

5.1 Ablation: attributing the gain

• Keeping the store live accounts for essentially the entire acceptance gain: the live global arm reached 20.06 MAT, while adding per-user partitioning changed MAT by −0.15.Eviction changed MAT by −0.05; capacity 48 cost 0.04–0.05 MAT versus unbounded storage, and global sharing exposed 14–68% cross-user draft provenance. Further sweeps linked the gap to post-warmup arrivals and found it shrank to ∼2% when warmup covered every task.Source: §5.1 · pp. 6–7

5.2 Side-effect gating for speculative execution

• Gating by declared side effects eliminated irreversible cost, while confidence gating at τc=0.7 still incurred cost 255 and committed 40 bad actions.The side-effect gate defers unsafe drafts until verification; the policy ranking also held under raw error counts and a three-tier severity map.Source: §5.2 · pp. 7–8, Table 4 · p. 8

5.3 Generalization across served models

• Across served models, the post-warmup gap was +27–29% and the final-session gap was +45–48%.The authors conclude that the repeated holds-versus-degrades pattern reflects the memory lifecycle rather than one target model.Source: §5.3 · p. 8

5.4 Seal-Tools and live multi-turn τ 2-bench traces

• +34% post-warmup gap on Seal-Tools, where the frozen store fell from 22.6 MAT to 13.2–14.6 while SPECMEM held at 19.9–22.3.The result replicated on a benchmark constructed independently of BFCL; the final-session gap was +54%.Source: §5.4 · p. 8
• +22% overall gap on live multi-turn τ 2-bench; its domain gaps were +27% for airline and retail and +11% for telecom.The gaps tracked the domains’ shares of novel calls: 82%/56% unique calls for airline/retail versus 14% for telecom.Source: §5.4 · p. 8

6 Conclusion

• Online write-back sustains acceptance as streams evolve, while local scope and eviction provide isolation and boundedness.The conclusion recommends keeping draft stores live and gating speculative execution on declared side effects rather than retrieval confidence.Source: §6 · p. 8

Limitations

• Synthetic users and sessions limit direct generalization to organic use; un-redacted multi-session agent telemetry remains the decisive test.The reported gap depends on user repetition and post-warmup novelty, although its direction is robust across seeds, drift rates, data constructions, and benchmarks.Source: Limitations · p. 9
• MAT scores one retrieved draft against greedy output and omits multi-turn-selection and irrelevance categories; its absolute values are not comparable to trained drafters.Wall-clock speedups use an external specula­tive loop with serving-stack timers rather than an engine-integrated hook.Source: Limitations · p. 9
• The retrieval threshold τ=0.35 and single embedder are fixed rather than swept, and dynamics are confirmed on three open-weight model families, not proprietary agents.Source: Limitations · p. 9
• The side-effect gate assumes correct per-tool safe_to_speculate annotations and binary safety labels; call-dependent tools motivate a per-call ternary refinement.Rollback, compensating actions, and learned idempotency inference remain open directions.Source: Limitations · p. 9

Ethical Considerations

• A global memory may expose one user’s interaction history through another user’s retrieved drafts, while incorrect speculative execution can trigger irreversible actions before verification.Source: Ethical Considerations · p. 9
• Per-user partitioning and deferring unsafe tools until verification are safeguards, but deployment should add access control, auditing, sandboxing, and human confirmation for high-impact actions.The experiments use public benchmarks and simulated users rather than real user logs.Source: Ethical Considerations · p. 9

A.1 Stream details

• 2,960 queries per seed and 8,880 across three seeds are generated; headline comparisons score sessions 1–11 after a shared cold-start session.After declined queries are excluded symmetrically, each arm scores 6,814 post-warmup BFCL queries and 7,592 Seal-Tools queries.Source: §A.1 · p. 12
• Warmup covers 8 of 15 tasks per user, and post-warmup queries weight novel tasks 3:1 over warmed tasks; the headline gap shrinks to about 2% when all tasks are warmed.Source: §A.1 · p. 12
• Declined queries are removed from every arm, so conclusions apply only to queries for which the served model issues a tool call.That share is 82% of unique queries for gpt-oss on BFCL and 95% on Seal-Tools.Source: §A.1 · p. 12
• The τ 2-bench replay uses one trace per task and served model, generated with a live GPT-4.1 user simulator and shared across memory arms.Its 278 tasks yield 1,753 post-warmup decision points; the reported seeds construct replay streams that reuse cached targets.Source: §A.1 · p. 12
• The disjoint-per-user BFCL design uses 600 examples from the simple, multiple, and parallel pool; it excludes multi-turn and irrelevance-detection categories.The protocol scores one drafted call against one target call, not dialogue-level tool selection.Source: §A.1 · p. 12

A.2 TOOLSPEC-style reimplementation

• The TOOLSPEC-style baseline is reimplemented with a fixed global store and shared models, session protocol, and timers, rather than run from its official release.It reproduces confidence-gated retrieval and a schema-aware FSM cold-miss surrogate; the authors describe using BFCL v4 and their served models as substitutions, not an exact reproduction. Retrieval and write-back cost about 9.4/9.5 ms per query inside the timed loop.Source: §A.2 · p. 12

B Reproducibility Details

• GPT-OSS-120B, Gemma-4-31B-IT, BFCL, Seal-Tools, ToolAlpaca, and ToolBench use Apache 2.0; Nemotron-3-Super-120B uses the NVIDIA Nemotron Open Model License.API-Bank and τ 2-bench use the MIT License.Source: §B.1 · pp. 12–13
• The planned release contains the experimental harness, stream-construction code, configurations, and derived results.Source: §B.1 · p. 13

• Three open-weight tool-calling models were served: gpt-oss-120b, gemma-4-31B-it, and Nemotron-3-Super-120B.Query embeddings used all-MiniLM-L6-v2 on CPU, and the live τ 2-bench user simulator was GPT-4.1 via the OpenAI API.Source: §B.2 · p. 13
• Experiments used NVIDIA H200 GPUs and required approximately 20 H200 GPU-hours of active compute.gpt-oss-120b and gemma-4-31B-it used one GPU each; Nemotron-3-Super-120B used four with tensor parallelism.Source: §B.2 · p. 13
• SPECMEM used top-1 cosine retrieval with τ = 0.35 and a per-user capacity of 48 entries under LRU eviction.Targets were decoded greedily; settings were fixed in pilot runs and held constant across benchmarks and models, with capacity also swept in §5.1.Source: §B.2 · p. 13
• The harness used Python 3.10, PyTorch 2.9.1, and Transformers 5.6.0, with models served through sglang or vLLM.The OpenAI-compatible endpoints used sglang 0.5.11 for gpt-oss-120b and gemma-4-31B-it, and vLLM 0.18 for Nemotron-3-Super-120B.Source: §B.2 · p. 13