Source-linked AI summary
Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution
Kyudan Jung, Hyunsin Park, Yoonhyung Lee, Jinhwan Park, Jinhyeok Yang, KiHyun Nam, Jaegul Choo, Jinkyu Lee
TL;DR
Cascaded voice assistants incur external-tool latency only after speech recognition and LLM tool selection have finished. This work speculatively executes tools from partial ASR and injects cached results into the LLM prompt; median time-to-first-audio falls from 5.79 seconds to 4.60 seconds. The authors report that responses are faster, more accurate, and more stable, while noting that on-device LLM prediction may not finish before ASR finalization.
Problem
Cascaded assistants must wait until the user finishes speaking and the LLM identifies a tool request before invoking external tools, leaving remote-operation delays directly perceptible to users.
Source: §1 · p. 1Method
A rule-based Predictor launches likely tool requests from partial ASR, caches their results, and injects them into the LLM prompt while retaining ordinary LLM-directed tool calling as fallback.
Source: §1 · p. 1Results
Median waiting time from the end of speech to the response decreased from 5.79 seconds to 4.60 seconds, and tool-calling F1 increased from 44.8% to 60.6%.
Source: §1 · p. 2Takeaways & Limitations
On a commercial Android device, the method makes voice-agent responses faster, more accurate, and more stable; direct cache injection drives the accuracy improvement, while verified fallback increases cache reuse and reduces wasted calls.
Source: §5 · p. 4The offline neural-predictor results omit inference latency, and on-device LLM routing may not consistently finish before ASR finalization, limiting the overlap available for speculative execution.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls. We present speculative tool execution for on-device cascaded voice agents, which predicts tool requests from partial ASR hypotheses and initiates tool execution while speech is still being received, thereby reducing end-to-end response latency. Our approach introduces a Predictor module that anticipates tool calls during speech recognition, executes them speculatively, and caches the results. The cached outputs are then injected into the LLM prompt, enabling faster responses. Additionally, to mitigate errors caused by user self-corrections during speech, we employ a rule-based validation mechanism that selectively injects only valid cached results. As a final safeguard, the LLM retains the ability to issue tool calls directly, ensuring that the latency of our framework is upper-bounded by the baseline serial execution pipeline in the worst case. We evaluate our method using live measurements from a fully implemented Android voice assistant. Our approach reduces the median time-to-first-audio from 5.79,s to 4.60,s and decreases the standard deviation from 3.49,s to 2.81,s, resulting in more predictable response latency.
1. INTRODUCTION
Cascaded assistants wait for finalized speech and an LLM tool request before starting external tools, leaving remote-tool delays on the user-visible path. This work predicts tool requests from partial ASR, runs them early, and reuses cached results to reduce response latency.
1. INTRODUCTION
- Conventional cascaded assistants wait for the user to finish speaking and the LLM to identify a tool request, so remote operations can add user-perceived delay.The paper motivates reducing this latency for responsive mobile voice interactions.Source: §1 · p. 1
- A rule-based Predictor watches partial ASR hypotheses, launches likely long-latency tools, caches results, and injects them into the LLM prompt.The standard LLM tool-calling path remains available when no suitable speculative result exists.Source: §1 · p. 1, Figure 2 · p. 2
- Median waiting time fell from 5.79 seconds to 4.60 seconds, while tool-calling F1 increased from 44.8% to 60.6%.The authors also report reduced variance and more predictable response times.Source: §1 · p. 2
2. METHODOLOGY
The method overlaps tool execution with streaming ASR by predicting requests from partial transcripts, caching results, and making them available to the LLM. If a speculative result is unavailable, normal LLM-directed tool execution remains as a fallback.
2.1. Problem Formulation
- The timing formulation targets the interval from ASR finalization to tool completion, hiding tTool − tASR during ongoing ASR to reduce total response time.The pipeline timestamps run from utterance start through ASR, tool selection, tool completion, response generation, and TTS completion.Source: §2.1 · p. 2
2.2. Predicting Which Tool to Call
- The Predictor checks partial ASR after six words, every five additional words, or a pause longer than 300 ms, then analyzes clauses independently.Clause segmentation allows multiple information-seeking intents in one utterance to be dispatched in parallel.Source: §2.2 · p. 2
2.3. Direct Injection and Cache Matching
- At ASR finalization, completed speculative results are inserted into the LLM prompt; if the LLM issues a tool call, a matching cached result is reused or normal execution proceeds.The cache check verifies a match and avoids redundant retrieval when the speculative result is already available.Source: §2.3 · pp. 2–3
3. EXPERIMENTS
The evaluation uses live measurements from an Android voice assistant, a manually constructed set of spoken requests, and separate latency and tool-use metrics.
3.1. Experimental Setup
- The system runs streaming Conformer ASR and TTS on the CPU, while a quantized Llama-3.2-3B LLM runs on the NPU; CPU-side components handle prediction, cache management, and tool dispatch.The remote search tool uses the EXA web-search API.Source: §3.1 · p. 3
3.2. Evaluation Dataset
- The evaluation set contains 146 manually written English utterances, including normal requests, hard negatives, and adversarial cases involving corrections, retractions, late intents, or topic changes.The audio totals approximately 1.5 hours, and a complete run takes approximately 3 hours including TTS generation and live network calls.Source: §3.2 · p. 3
3.3. Metrics
- TTFA measures time from the end of input speech to first TTS audio, while tool-request performance is reported with precision, recall, F1, and exact accuracy.The evaluation also tracks false reads, wasted speculative calls, and cache-ready rate.Source: §3.3 · p. 3
4. RESULTS
The speculative system improves response speed and consistency over the cascaded baseline, with latency gains tied to overlapping tool execution with speech. Ablations and robustness tests examine cache use, intent changes, and the trade-offs of an LLM-based predictor.
4.1. Main Results
- 17 seconds to 8 seconds TTFA for a 15-second query requiring approximately 10 seconds of search illustrates the latency hidden by overlap.The comparison is between the cascaded baseline and the speculative system.Source: §4.1 · p. 3
- Tool-selection accuracy improves, with a higher F1 score and fewer unnecessary speculative tool calls.The authors hypothesize that task-relevant prompt context helps the small 3B language model select tools.Source: §4.1 · p. 3
- Shifting remote-tool latency into speaking time leaves post-speech latency dominated by more stable local inference, without increasing final-answer failure rate.The authors report a faster, more predictable interaction experience.Source: §4.1 · pp. 3–4
4.2. Ablation Study
- Speculative prefetch concurrent with on-device ASR adds no noticeable latency penalty; its mean and p95 latency are comparable to or below baseline.The variant retains the rest of the baseline pipeline, although its median latency rises slightly.Source: §4.2 · p. 4
- Verified fallback alone leaves latency and F1 close to baseline because tool selection still precedes cache lookup, exposing most selection latency.The additional lookup also contributes latency.Source: §4.2 · p. 4
- Direct cache injection reduced mean latency from 6.69 to 6.17 seconds and increased F1 score from 44.8% to 63.0%.The authors attribute the accuracy improvement primarily to direct cache injection.Source: §4.2 · p. 4
- The final method adds verified fallback to direct injection, achieving the lowest median and mean latency, highest Ready rate, and fewest wasted calls, despite slightly worse p95 latency and F1.The authors describe this as the best overall balance of response speed, cache utilization, and speculative-call efficiency.Source: §4.2 · p. 4
4.3. Hard Negatives and Adversarial Cases
- Hard negatives yield a 9.1% false read rate; adversarial examples yield 90.2% F1 and 12.3% false read, with 21 wasted calls.The adversarial cases include corrections, retractions, and topic changes; because speculation uses read-only tools, these errors do not revise user data.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper presents speculative tool execution for an on-device voice agent, starting tool calls during streaming ASR.
- The proposed on-device method starts tool calls during streaming ASR.Source: §5 · p. 4