PaperlayerView exact revision on arXiv

Source-linked AI summary

SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decoding

Boyuan Yang, Wei Gao

arXiv:2610.17436v1cs.AI

TL;DR

Mobile LLM tool calls are slow because decoding is memory-bandwidth-bound. SchemaWeaver uses tool-schema structure to parallelize argument-value decoding; it achieves up to 3.9× tool-call latency speedup and reduces energy consumption by 2.1×.

  • Problem

    Mobile LLM tool calling has long latency because decode passes are memory-bandwidth-bound and mobile GPU or NPU use cannot help much under shared memory architecture.

    Source: §1 · pp. 1–2

  • Method

    SchemaWeaver uses fixed tool-schema positions to parallelize decoding of argument values, while fixed argument names can be loaded in a prefill pass.

    Source: §1 · p. 2

  • Results

    SchemaWeaver achieves up to 3.9× speedup in tool call latency without noticeable correctness loss compared to existing schemes, and reduces energy consumption by 2.1×.

    Source: §10 · pp. 12–13

  • Takeaways & Limitations

    SchemaWeaver's gain grows as memory bandwidth falls, with speedup rising from 3.0× to 3.9× when the memory bus is slowed by 4.7×.

    Source: §1 · p. 3

    SchemaWeaver runs on mobile CPU; it brings no gain on the phone’s GPU or NPU today.

    Source: §9 · p. 12

Abstract

from arXiv · show

Large language models (LLMs) on mobile devices can act as agents that call external software tools to complete complex tasks, but the major challenge of supporting on-device mobile agentic AI with LLMs is the long latency of LLM tool calling on mobile devices. In particular, the decode phase is the main contributor of such long latency, but is hard to speed up on mobile processors as it is bound by memory bandwidth rather than compute power. To address this limitation, in this paper we present SchemaWeaver, a new on-device mobile agentic AI technique that enforces schema-guided parallel decoding in the decode phase of LLM tool calling, to reduce its latency by producing multiple tokens in each decode pass. SchemaWeaver can be generically applied to the LLM inference engine on mobile devices without involving any training efforts or changes on LLMs, and our experiment results on major mobile SoCs and LLM families show that SchemaWeaver can reduce the LLM tool calling latency by up to 3.9 times, compared to the competitive baselines.

1 INTRODUCTION

LLM agents use machine-readable tool schemas to select tools and generate calls, but on-device tool-call decoding creates substantial user-facing latency. SchemaWeaver uses schema information to parallelize argument-value decoding, reducing tool-calling latency in the reported evaluations.

1 INTRODUCTION

  • Decode contributes up to 81% of tool-calling latency and is difficult to accelerate because mobile inference is limited by memory bandwidth.On the OnePlus 12R with Snapdragon 8 Gen 2, decode speed on mobile CPU is > 5× slower than prefill speed.Source: §1 · pp. 1–2
  • Speculative decoding remains bandwidth-bound by draft-model passes, while jump-forward decoding still depends on sequential decode passes.SchemaWeaver instead targets parallel generation of argument values using fixed positions described by the tool schema.Source: §1 · p. 2
  • SchemaWeaver builds a tool skeleton from fixed schema parts, batches argument values for parallel decoding, and uses padding, write-back, and early call emission.Fixed argument names can be loaded in a prefill pass rather than decoded.Source: §1 · p. 2
  • Optional argument values are generated sequentially after required values, with provisional calls for read-only tools emitted early and later replaceable.The paper notes that optional values cannot adapt to parallel decoding.Source: §1 · p. 2
  • SchemaWeaver is presented as an on-device tool-calling acceleration technique that requires no training efforts or changes to LLMs.The authors implemented it in llama.cpp and evaluated it with Qwen, Llama 3.2, and SmolLM2 families.Source: §1 · pp. 2–3
  • 2.13× end-to-end tool calling latency reduction over competitive baselines accompanies a drop from 44.3 to 13.1 forward passes per call.With memory bandwidth slowed by 4.7×, speedup rises from 3.0× to 3.9×; correctness costs <2%, and energy consumption falls 2.1× with 8.9 MB extra peak memory.Source: §1 · p. 3

2 BACKGROUND & MOTIVATION

Mobile tool calls spend much of their time in decoding, which is constrained by memory bandwidth; tool schemas expose fixed structure that creates an opportunity for parallel decoding.

2.1 Tool Calling Interfaces of LLM Agents

  • Tool schemas fix every character of a tool call except argument values, across JSON, Python-like, and special-token interfaces.Source: §2.1 · p. 3

2.2 Tool Call Latency Breakdown

  • Decode contributes up to 81% of end-to-end tool-calling latency; 54-63% of tool-call-turn decode tokens are fixed.Source: §2.2 · p. 3, Figure 3 · p. 3

2.3 Memory-Bound Tool Call Decoding on Mobile Processors

  • On mobile processors, decode throughput is much slower than prefill, and slowing memory-bus frequency reduces decode throughput by 70% versus 17% for prefill.Source: §2.3 · pp. 3–4, Table 1 · p. 3
  • A four-token decode pass increases pass latency by up to 20%, lowering average per-token decode latency by 70% and motivating parallel decoding.Source: §2.3 · p. 4, Table 2 · p. 4

3 OVERVIEW

SchemaWeaver uses the tool schema to locate argument values before decoding, then decodes required values in parallel and handles optional arguments separately.

3 OVERVIEW

  • SchemaWeaver assumes available tools have schemas listing argument names, types, and optionality, and operates inside the on-device inference engine.Source: §3 · p. 4

3.1 Tool Skeleton Builder

  • The Tool Skeleton Builder lays out fixed call text and required argument names so a prefill pass reveals each value's starting position.Source: §3.1 · p. 4

3.2 Parallel Value Decoder

  • The Parallel Value Decoder generates each argument value in a separate sequence, stops each at its type-specific boundary, and writes the results back as one call sequence.Source: §3.2 · p. 5

3.3 Optional Argument Resolver

  • The Optional Argument Resolver decides optional arguments serially after required values, while a provisional call with required arguments can start early only for read-only tools.Source: §3.3 · p. 5

4 PARALLEL DECODING OF ARGUMENT VALUES

This section details how SchemaWeaver builds a tool-call skeleton, decodes argument values in parallel, and reassembles the completed call.

4.1 Building the Tool Call Skeleton

  • The skeleton combines the tool name and fixed call syntax with argument names, placing all required argument sequences at the same position after one prefill pass.Source: §4.1 · p. 5

4.2 Parallel Token Sequences

  • 13.1 to 6.8 average decode passes yields 39% lower decoding latency; calls with at least two required arguments reach 47% latency reduction.Source: §4.2 · p. 6, Figure 7 · p. 6
  • Parallel sequences advance together and stop at type-specific boundaries; padding placeholder tokens can improve mobile batch efficiency when only two or three sequences remain unfinished.Source: §4.2 · p. 6
  • On BFCL with Qwen3-1.7B, parallel decoding's correctness impact is as low as 0.9%, although separated sequences can miss dependencies between argument values.Source: Figure 8 · p. 6

4.3 Assembling the Tool Call

  • Parallel decoding diverges the KV cache, so the reassembler joins argument text and writes back the complete call in one prefill pass.Source: §4.3 · pp. 6–7

5 HANDLING OPTIONAL ARGUMENTS

SchemaWeaver delays optional-argument decisions until required values are available, while provisional calls overlap some added work with tool execution.

Optional-argument decision timing

  • Deciding optional arguments early lowers their correctness 17% and tool-call correctness 13.5%; including every optional argument lowers argument correctness another 5.2%.Source: §5 · p. 7, Table 4 · p. 7

Provisional calls

  • Serially deciding optional arguments after required values raises argument correctness to 79% but adds 13.5 decode passes on average.Source: §5 · p. 7
  • A provisional call carries required arguments while optional decisions and write-back proceed during execution, but it is used only for read-only tools.Source: §5 · p. 7

6 IMPLEMENTATION

SchemaWeaver is integrated into llama.cpp through a library and a server hook that intercepts tool calls. The inference engine is otherwise unchanged, and the implementation is deployed on two Android phones.

6 IMPLEMENTATION

  • A 350-line server patch hooks tool-call decoding, while a 930-line C++ library uses llama.cpp’s public API.The hook returns a provisional call as soon as it is assembled, then feeds the completed call’s tokens back as if decoded serially.Source: §6 · p. 8
  • The implementation runs on a OnePlus 12R with Snapdragon 8 Gen 2 and 16 GB RAM, and a Pixel 7 with Tensor G2 and 8 GB RAM.Both phones run Android; the models use publicly released GGUF files in Q8_0 and Q4_0 quantizations.Source: §6 · p. 8

7 EVALUATION

The evaluation tests open-source LLMs from 0.6B to 4B parameters on BFCL and DroidCall, comparing SchemaWeaver with four decoding baselines. It measures latency, whole-call correctness, forward passes, memory, and energy under a controlled phone-CPU protocol.

7.1 Latency and Correctness of Tool Calling

  • On BFCL, Qwen3-1.7B reaches 2.13× speedup over constrained decoding, while whole-call correctness is 74.0% versus 76.0%.Jump-forward reaches 1.82×; speculative decoding is slower than constrained decoding in the reported Qwen3 settings, at 0.27× and 0.42×.Source: §7.1 · p. 9
  • 677 ms versus 1,771 ms for constrained decoding gives SchemaWeaver a 2.62× speedup on DroidCall with Qwen3-1.7B on OnePlus 12R.Across other settings, speedups are 2.67×–3.55×, with correctness loss against constrained decoding within 2%.Source: §7.1 · p. 9, Table 6 · p. 9

7.2 Different Model Sizes and Quantizations

  • Speedups over constrained decoding range from 1.76× to 2.36× on OnePlus 12R and 1.93× to 2.50× on Pixel 7.Figure 11 breaks latency into phases; Qwen3-4B does not fit Pixel 7’s 8 GB memory, yet SchemaWeaver still reaches 2.31× speedup there.Source: §7.2 · pp. 9–10, Figure 11 · p. 10

7.3 Impact of Memory Bandwidth

  • When Pixel 7’s memory bus slows 4.7×, SchemaWeaver’s speedup rises from 2.23× to 2.89×, versus 3.23× slower constrained decoding.The per-phase results show serial decoding slows more with the bus than skeleton prefill, which is less bandwidth-sensitive.Source: §7.3 · p. 10, Figure 13 · p. 10

7.4 Impact of the Number of Arguments

  • Calls with four or more arguments gain 2.5× speedup, compared with 1.9× for one-argument calls.Three-argument calls reach 1.9× versus 2.2× for two-argument calls, in part because optional arguments more often require write-back.Source: Figure 12 · p. 10, §7.4 · p. 11

7.5 Ablation Studies

  • Skipping fixed tokens and issuing a provisional call yields up to 1.92× speedup; parallel values and padding raise it to 2.13×.In one call, SchemaWeaver starts the tool 505 ms after generation begins, 3.97× sooner than constrained decoding, and completes in 981 ms with 9 passes instead of 35.Source: §7.5 · p. 11, Table 7 · p. 11, Table 8 · p. 11

7.6 Memory Overhead

  • SchemaWeaver adds only 8.9 MB of peak resident memory over constrained decoding on OnePlus 12R, compared with about 2.2 GB for EAGLE-3.The additional SchemaWeaver memory holds parallel-sequence KV-cache entries and type constraints.Source: §7.6 · p. 11, Table 9 · p. 11

7.7 Energy Consumption

  • 12.8 J to 6.1 J is a 2.10× energy reduction per tool call on Pixel 7; SchemaWeaver starts the call 2.23× sooner than constrained decoding.Jump-forward saves 1.97× energy, while SchemaWeaver saves 2.8× on memory rails; the figure separates CPU and memory rails.Source: §7.7 · pp. 11–12, Figure 14 · p. 12

8 RELATED WORK

The related work places SchemaWeaver among mobile agents, parallel text-generation methods, and speculative decoding. It distinguishes SchemaWeaver by its schema-based parallel decoding, training-free use of phone-ready models, and lack of a second model.

On-device LLM agents

  • Agent-X reuses unchanged prompt parts across turns, whereas SchemaWeaver shortens tool-call generation within a turn; the two can be combined.SPORK and PASTE start tools while a call is still being generated and discard results when guesses are wrong, inspiring SchemaWeaver’s optional-argument handling [5] [56].Source: §8 · p. 12

Parallel text generation

  • SimpleTool also parallelizes function names and arguments, but trains special tokens and evaluates on desktop GPUs; SchemaWeaver is training-free and runs quantized models on phones [55].HyFunc uses a trained cascade and template-filled syntax.Source: §8 · p. 12

Speculative decoding

  • Unlike speculative methods that rely on draft weights or workload history, SchemaWeaver’s saved passes are fixed by the schema and require no second model.The approaches can be combined: a draft model can shorten each value sequence while SchemaWeaver decodes the sequences together.Source: §8 · p. 12

9 DISCUSSIONS

SchemaWeaver currently runs on mobile CPUs; its schedule change is not inherently CPU-specific, but the evaluated GPU and NPU implementations do not provide gains. The implementation also depends on inference-engine features not currently available in other listed Android frameworks.

Mobile accelerators

  • On the GPU, SchemaWeaver is slower than constrained decoding at 0.39×; NPU decoding also pays for CPU round trips.The authors propose future work that lets the NPU run prefills while the CPU decodes values, if state transfer costs less than it saves.Source: §9 · p. 12

Other inference engines

  • The current design needs a schema-derived grammar and caller-supplied attention masks for batched decoding; only llama.cpp offers both on Android.The authors expect similar speedups on other inference frameworks, but identify the required features as currently missing there.Source: §9 · p. 12

Mobile accelerators

  • SchemaWeaver’s CPU-side phase shortening can be combined with MNN-AECS, which runs bandwidth-bound decoding on fewer, slower CPU cores to save energy.Source: §9 · p. 12

10 CONCLUSION

SchemaWeaver parallelizes tool-call argument generation to accelerate mobile LLM tool use, reporting lower latency and energy consumption without noticeable correctness loss.

10 CONCLUSION

  • SchemaWeaver accelerates mobile LLM tool use by parallelizing tool-call argument generation.The technique allows parallel generation of argument values for tool calls.Source: §10 · p. 12
  • Up to 3.9× speedup in tool call latency, without noticeable correctness loss.The comparison is described as being against existing schemes.Source: §10 · p. 12
  • 2.1× reduction in energy consumption.Source: §10 · p. 13

A THE CORE SET OF THE MEASUREMENT PROTOCOL

The measurement protocol selects and pins four performance cores at fixed clocks; the appendix compares core sets and explains the trade-offs behind that choice.

A THE CORE SET OF THE MEASUREMENT PROTOCOL

  • Table 10 compares serial decode-step, four-sequence-step, and prefill-token costs across core sets; its ratio is four-sequence over serial.The protocol’s selected core set is bolded.Source: Table 10 · p. 16, A THE CORE SET OF THE MEASUREMENT… · p. 16
  • The four-sequence step costs 1.22× the serial step on OnePlus 12R and 1.11× on Pixel 7 with the chosen cores.Adding more cores does not lower this overhead; the measured ratios range from 1.18–1.25× on 12R and are 1.19× on Pixel 7.Source: A THE CORE SET OF THE MEASUREMENT… · p. 16
  • Using all eight cores is 9% slower than four on OnePlus 12R and 27% slower on Pixel 7 because each pass waits for the slowest core.The four little Pixel 7 cores alone are two to three times slower.Source: A THE CORE SET OF THE MEASUREMENT… · p. 16
  • The X3 shortens every pass but is excluded because its clock is not sustainable under continuous load.It shortens the serial step by 20% and the prefill token by 43%, while shortening the baseline and SchemaWeaver alike.Source: A THE CORE SET OF THE MEASUREMENT… · p. 16
  • Two Pixel 7 X1 cores already saturate the memory bus, since two threads decode as fast as four.Source: A THE CORE SET OF THE MEASUREMENT… · p. 16

Related papers

Loading 2610.17436v1…