SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

1 INTRODUCTION

• Autoregressive decoding serializes wrappers, argument keys, and values, adding latency for requests with many fields or multiple calls.Tool-call responses must also satisfy machine-readable interface schemas, and one response may contain different tools or repeated calls.Source: §1 · p. 1
• Direct parallel drafting faces unknown realized calls and fields, while independent values may differ from the target’s output under preceding fields and calls.Speculative verification keeps drafts provisional, accepting target matches and supplying a correction at the first mismatch.Source: §1 · p. 2
• SchemaFill forecasts possible call positions and slots, drafts future argument values concurrently, and verifies packed candidates under the actual output prefix before commitment.The shared runtime evaluates concurrent drafts in one target forward pass while isolating branch attention and reusing verified KV states.Source: §1 · p. 2
• Up to 4.05× mean end-to-end speedup over autoregressive decoding, with SchemaFill outperforming existing speculative-decoding baselines on GLAIVE and BFCL.The framework packs concurrent slot drafts with structural text for target verification.Source: §1 · p. 2

Tool Use, Function Calling, and Structured Generation

• Tool-use and grammar-constrained methods improve tool capability or structural validity; SchemaFill instead addresses inference cost while retaining target control over tools, fields, ordering, and values.Examples include Toolformer’s self-supervised API annotations (Schick et al., 2023), Gorilla’s retrieval-aware training (Patil et al., 2024), and ToolLLM/ToolBench’s real-world API scale (Qin et al., 2024).Source: §2 · p. 2

Speculative Decoding

• Existing speculative-decoding methods propose continuations of the current serialized prefix, whereas SchemaFill verifies candidates for possible fields and calls beyond that prefix.The related methods include draft-token verification (Leviathan et al., 2023; Chen et al., 2023), EAGLE (Li et al., 2024; 2025), Medusa (Cai et al., 2024), and Lookahead Decoding (Fu et al., 2024).Source: §2 · pp. 2–3

Schema-Aware Tool-Call Acceleration

• ToolSpec combines schema proposals, retrieval, and token recycling; SchemaFill prepares future argument values and forecast call positions, then verifies packed candidates under the actual prefix.Local branch acceptance does not commit text to the final response.Source: §2 · p. 3

Research Gap

• The open gap is target-verified speculation whose candidate units are noncontiguous slots in a structured multi-call response.SchemaFill packs possible argument slots across fields and calls, extending the response only with target-consistent tokens.Source: §2 · p. 3

3.1 STRUCTURED MULTI-CALL GENERATION

• The model receives a request, context, and tool catalog, then emits an ordered sequence of selected tools and argument objects for dispatch after generation.A response may contain multiple calls or repeat a tool.Source: §3.1 · p. 3
• Schemas expose possible fields, types, and required status, but the target chooses optional fields and values, which may depend on earlier fields or calls.Different calls may use different schemas.Source: §3.1 · p. 3
• Autoregressive decoding serializes tool names, argument keys, values, punctuation, and enclosing structure, stopping at the closed outer object or end-of-sequence token.Source: §3.1 · p. 3

3.2 SPECULATIVE DECODING ON SERIALIZED TOOL CALLS

• Speculative decoding proposes tokens and verifies them under the actual prefix, where each candidate score depends on that prefix and preceding candidate tokens.Prefixes and proposals can cross tool names, keys, values, or structural delimiters; schema constraints do not commit later fields or calls.Source: §3.2 · pp. 3–4
• At temperature 0, the runtime accepts the longest candidate prefix matching greedy target decisions, commits a target correction at the first mismatch, and resumes from the corrected prefix.The paper states that Appendix B proves consistency with greedy AR for this rule and gives a distributionally consistent stochastic rule.Source: §3.2 · p. 4

3.3 ACCUMULATED AR DEVIATION IN DIRECT PARALLEL SLOT GENERATION

• A BFCL diagnostic uses an offline greedy AR layout with marked argument values, then generates each value independently without seeing values from other slots.The oracle layout controls for tool and field selection in this diagnostic; SchemaFill does not receive it at inference.Source: §3.3 · p. 4
• 49.5% of the fixed eight-parameter cohort had a value differing from AR by slot eight; in the fixed three-call cohort, the fraction reached 43.3% by call three.These curves measure cumulative AR deviation, not execution success or the difficulty of an individual slot.Source: Figure 2 · p. 4, §3.3 · p. 4

4.1 METHOD OVERVIEW

• The forecast–draft–verify loop prepares later values before their serialized positions are reached, without committing forecasts to the final response.The full schema remains in the target prompt, and forecasting changes where draft computation is spent rather than what the target emits.Source: §4.1 · pp. 4–5

4.2 SLOT-PARALLEL SPECULATIVE DECODING

• SchemaFill forecasts possible argument slots from the request and tool catalog, while the target determines the realized tools, fields, order, and values.The default runtime ranks tools with BM25 over request and dialogue context against public tool metadata, then expands forecast tools into schema-key slots.Source: §4.2 · p. 5
• Future slot values remain provisional until packed with structural text and verified by the target under the actual output prefix.A packed proposal can span fields and tool-call boundaries; the target accepts matching tokens and supplies a correction at the first mismatch.Source: §4.2 · p. 6

4.3 KV-CACHE MANAGEMENT

• The shared target-KV arena stores public context, verified history, and scratch candidates while attention masks preserve branch-specific visibility.Accepted rows extend their owning paths; rejected suffixes are discarded, and a SIDE value must be verified again under the actual MAIN prefix before commitment.Source: §4.3 · pp. 6–7

5.1 EXPERIMENTAL SETUP

• The evaluation uses 650 sampled GLAIVE requests and all 450 requests from three BFCL v3 categories, with shared prompts and Qwen2.5-7B-Instruct as target.Methods are compared at temperature 0; end-to-end timing includes preparation, drafting, verification, and stopping.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 MAIN RESULTS

• 340.60 ± 2.96 TPS on GLAIVE and 375.71 ± 3.66 TPS on BFCL yield SchemaFill speedups of 3.73× and 4.05× over AR, respectively.It also exceeds EAGLE-3 by 1.51× and 1.87×, and ToolSpec by 1.75× and 1.56×, on GLAIVE and BFCL, respectively.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 ABLATION STUDY

• Relative to EAGLE-3, Stage B raises TPS by 48.07% and AL by 61.18% through SchemaFill’s schema-guided MAIN path without SIDE branches.The comparison combines the MAIN runtime, metadata-guided forecasting, and structural/slot concatenation.Source: §5.3 · p. 8, Table 2 · p. 8
• Within the shared runtime, enabling SIDE drafting in Stage C changes TPS by +11.04% and AL by +19.95% over Stage B.SIDE drafting prepares later slot values before MAIN reaches them.Source: §5.3 · pp. 8–9
• Cross-call packing in Stage D changes TPS by +13.69% and AL by +17.07% over Stage C by proposing prepared values and structural bridges together.Stage D is the final cumulative stage and combines candidates across call boundaries for one target verification.Source: §5.3 · pp. 8–9, Table 2 · p. 8

5.4 SCALING WITH CALL COUNT

• With forecast budget B = C, mean initialization-tree size grows from 61.78 nodes at one call to 151.98 at four, while later verification rounds use much smaller trees.The separate diagnostic samples 100 requests per call-count group from one to four calls.Source: §5.4 · p. 9, Table 3 · p. 9
• In the call-count diagnostic, throughput peaks around two to three calls at 438.62–443.54 TPS and AL 7.98–8.06, then falls to 410.72 TPS and AL 7.54 at four calls.At four calls, mean verification-tree size is 42.71 nodes; the authors associate the pattern with slot-drafter quality limiting further acceptance gains.Source: §5.4 · p. 9, Table 3 · p. 9

• Figure 4’s candidate-node counts peak at slot initialization and drop sharply in subsequent decoding rounds.Its curves average requests reaching each round in the Table 3 diagnostic and exclude prompt tokens.Source: Figure 4 · p. 9
• SchemaFill achieves 3.73× and 4.05× end-to-end throughput over AR on GLAIVE and BFCL, respectively, and outperforms EAGLE-3 and ToolSpec.Candidates are packed across fields and calls, and only tokens verified under the actual output prefix are committed.Source: §6 · p. 9
• Future argument values are drafted concurrently without assuming the realized call sequence or that arguments are independent.The target verifies every candidate before commitment, supplying target-consistent tokens.Source: §6 · p. 9

A IMPLEMENTATION DETAILS

• Main experiments use Qwen2.5-7B-Instruct as target and a locally trained EAGLE-3 head shared by EAGLE-3 and SchemaFill.The target uses temperature 0 and BF16 weights and computation; the drafter is trained on public tool-calling requests.Source: A IMPLEMENTATION DETAILS · p. 13
• The production runtime uses BM25-ranked tool metadata and a three-call forecast budget, with active speculative branches sharing a segmented draft tree and one target-tree verification.It reuses valid target KV allocation capacity and prompt prefixes; the measured draft budget is k = 3, committed-stream depth 2, branch depth 1, and total draft tokens 8.Source: A IMPLEMENTATION DETAILS · p. 13
• Experiments run batch size one on one NVIDIA RTX PRO 6000 Blackwell Workstation GPU, with model loading, graph setup, warmup, and trace serialization excluded from E2E time.GPU timing runs are serialized, and methods share SDPA, normalization, CUDA graphs, and target numerical settings.Source: A IMPLEMENTATION DETAILS · p. 13

B SPECULATIVE-SAMPLING CONSISTENCY IN THE TOOL-CALL SETTING

• Under identical target conditionals at each prefix, MAIN/SIDE masking, and shared stopping rules, SchemaFill emits exactly the greedy AR token sequence.Only the matching MAIN prefix and the target’s first correction are committed; SIDE tokens require fresh MAIN verification.Source: B SPECULATIVE-SAMPLING CONSISTENCY IN… · pp. 13–14
• With exact target conditionals, fresh independent acceptance draws, and common termination rules, speculative sampling yields the same distribution of stopped tool-call responses as serial target sampling.On rejection, a residual-distribution correction is committed and later proposal tokens are discarded.Source: B SPECULATIVE-SAMPLING CONSISTENCY IN… · p. 14

C.1 EVALUATION SETS AND REFERENCE WORKLOAD COUNTS

• Glaive650 contains 650 requests, 650 reference calls, and 1,255 top-level argument fields.Requests were sampled by source conversation and then by eligible decision point from 27,278 eligible decisions across 20,670 conversations.Source: §C.1 · p. 15
• BFCL450 contains 450 requests: 50 executable-parallel, 200 parallel, and 200 parallel-multiple examples, totaling 1,333 calls and 3,356 fields.Executable-parallel references were parsed from raw ground-truth expressions; other field counts use the first official allowed argument value, omitting fields denoting absence.Source: §C.1 · p. 15
• Reference counts are offline workload statistics, stored separately from inference inputs and not used during inference.The BFCL set includes all requests in the three selected source categories.Source: §C.1 · p. 15

C.2 PER-REQUEST KV-CACHE OVERHEAD

• Table 4 measures per-request KV-cache growth relative to greedy AR with the same target-KV layout.The comparison holds target and drafter weights fixed and measures memory increase from extra KV, using an EAGLE-3 drafter, BF16 weights, and batch size one.Source: Table 4 · p. 15
• Across BFCL450, effective KV averages 228.02 MiB for SchemaFill versus 182.93 MiB for AR.Effective KV counts retained target rows and drafter KV; the separate reusable public-prefix snapshot averages 100.68 MiB of process-level storage.Source: §C.2 · p. 15
• The effective-KV sum is a conservative upper bound because target and drafter peaks are measured separately.Reserved spare capacity is excluded.Source: §C.2 · p. 15

C.3 BFCL RUNTIME TIME BREAKDOWN

• Table 5 partitions BFCL runtime across BM25, drafting, target verification, and remaining generation time.The four phases sum to the profiled time; BM25 covers lexical ranking, drafting covers EAGLE-3 generation, and verification includes tree construction and result evaluation.Source: Table 5 · p. 15, §C.3 · p. 15
• The breakdown excludes prompt construction, tokenization, initialization, prefill, model loading, graph setup, and warmup.The instrumented run took 129.23 seconds versus 127.51 seconds uninstrumented; reported throughput uses uninstrumented runs.Source: §C.3 · p. 15

C.4 RUN-TO-RUN VARIANCE

• SchemaFill’s output-token and target-call counts were identical across three runs per dataset, yielding AL variance of 0.00.TPS variance for SchemaFill was 8.78 on Glaive and 13.40 on BFCL; other refreshed configurations used one launch each.Source: §C.4 · p. 16

C.5 EXPERIMENTS AT TEMPERATURE ONE

• At target temperature 1, the evaluation compares AR, EAGLE-3, ToolSpec, and SchemaFill on fixed cohorts of 650 Glaive and 450 BFCL requests.The ToolSpec adapter retains its candidate construction and uses categorical target draws to select accepted tree paths and corrections.Source: §C.5 · p. 16
• 3.58× on Glaive and 3.87× on BFCL are SchemaFill’s speedups over AR at target temperature 1.Table 6 reports end-to-end results on Glaive650 and BFCL450, with one run per method and dataset; learned drafters use temperature 0.Source: §C.5 · p. 16, Table 6 · p. 16

C.6 SLOT-ORDER DIAGNOSTIC MODEL

• Each additional parameter is associated with 1.25× the odds of at least one AR deviation, with a 95% interval of 1.14–1.38.This source- and call-count-adjusted response-level logistic model uses 450 BFCL responses and describes AR-selected layouts, not held-out end-to-end performance.Source: §C.6 · p. 16

D CROSS-TARGET EVALUATION

• The evaluation covers Qwen3-8B, Mistral-7B-Instruct-v0.3, Hermes-3-Llama-3.1-8B, and Qwen2.5, with a separately trained drafter for each target.Each target has its own EAGLE-3 head, shared by EAGLE-3 and SchemaFill.Source: D CROSS-TARGET EVALUATION · pp. 16–17
• All four methods use a forecast budget of three, k = 3, MAIN depth two, SIDE depth one, and total draft budget eight.Evaluations use target temperature 0 for greedy decoding and 1 for stochastic decoding, while learned drafters retain temperature 0.Source: D CROSS-TARGET EVALUATION · pp. 16–17
• Table 7’s single-run request times include initialization and failed attempts, but exclude model loading and one-time graph setup.These measurements are separate from the repeated-run estimates in Table 1.Source: D CROSS-TARGET EVALUATION · p. 17, Table 7 · p. 17
• At temperature 0, SchemaFill reaches 1.44×–1.55× ToolSpec’s throughput; at temperature 1, it reaches 1.36×–1.90× EAGLE-3’s and 3.57×–3.87× AR’s.Table 7 reports cross-target BFCL450 evaluation; its speedup is TPS relative to AR at the corresponding temperature.Source: Table 7 · p. 17, D CROSS-TARGET EVALUATION · p. 17