Source-linked AI summary
Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents
Zhenyu Zhang, Kewei Fu, Yongqi Tong, Chenzhi Zheng, Wei Wu, Jiang-Ming Yang, Jianshe Li
TL;DR
Voice agents may leave users waiting after speech ends, although actionable information can arrive earlier and preparation during speech can be truncated by update-bounded approaches. Heartbeat Thinking lets successive Thinker invocations and tools continue across speech updates, passing completed preparation to a Responder. Across three benchmarks it improves aggregate speculative coverage or retrieval reuse over a windowed baseline, with benchmark-dependent task-quality effects; on FDBench v3 it also reduces final-response latency by 36.4%.
Problem
Cascaded pipelines defer preparation until speech ends, while chunk-bounded or interrupt-on-update approaches can truncate calls before their arguments are complete.
Source: §1 · p. 1Method
Heartbeat Thinking uses periodic Thinker invocations to process partial transcripts and completed preparation, letting ongoing reasoning and dispatched tools continue across speech updates before a Responder handles the full request.
Source: §1 · pp. 1–2Results
Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over SHANKS-style Window Thinking; on FDBench v3, mean final-response latency falls from 6.90 to 4.39 seconds (36.4%).
Source: §1 · p. 2Takeaways & Limitations
Across the tested benchmarks, Heartbeat retains more reusable preparation than a windowed baseline, but task-quality changes vary by benchmark.
Source: §5 · p. 5Latency analysis focuses on FDBench v3, and the Responder receives only a snapshot of completed preparation at context assembly, excluding results returned afterward.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request. Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete. This couples computation time to speech updates, limiting useful work completed in advance. We introduce Heartbeat Thinking, which handles updated input through new Thinker invocations while allowing ongoing invocations and dispatched tools to finish. Periodic invocations read the latest partial transcript and a shared record of completed reasoning and tool results. These results help resolve missing arguments and initiate dependent calls, allowing preparation to progress across speech updates. Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, with benchmark-dependent effects on final task quality. On FDBench v3, it increases tool coverage by 16 percentage points and reduces mean final-response latency by 36.4\%. The results show that heartbeat thinking paradigm preserving ongoing computation during speech can turn listening time into reusable preparation for faster and more capable voice-agent responses.
1 Introduction
Voice agents can begin preparing while a user speaks, but speech-bounded or interrupt-on-update approaches may leave useful work unfinished. Heartbeat Thinking lets successive invocations and tools continue across speech updates, then passes completed preparation to a Responder; experiments report higher speculative coverage or retrieval reuse than a windowed baseline, with benchmark-dependent task-quality effects.
1 Introduction
- Conventional cascaded pipelines defer preparation until speech ends, although actionable information may arrive earlier in a request.Prior work has shown reasoning and tool use can begin on partial speech (Hooper et al., 2026).Source: §1 · p. 1
- Chunk-bounded or interrupt-on-update generation can truncate incomplete tool calls, while evolving requests can make completed results outdated (Hooper et al., 2026).These constraints motivate letting computation finish across speech updates while later input determines whether results still apply.Source: §1 · p. 1
- Periodic Thinker invocations read the latest partial transcript and shared completed reasoning and tool results without interrupting ongoing work.Asynchronous tool results can resolve missing arguments or support dependent calls in later invocations.Source: §1 · pp. 1–2
- At speech end, a Responder checks completed preparation against the full request, reuses applicable results, and makes remaining calls without waiting for unfinished speculative jobs.The exchange of completed records and results also permits independently selected Thinker and Responder models.Source: §1 · p. 2
- On FDBench v3, speculative tool coverage rises from 34.67% to 50.67%, while mean final-response latency falls from 6.90 to 4.39 seconds (36.4%).The latency is also 45.7% below Pipecat, the fastest evaluated cascaded baseline.Source: §1 · p. 2
2 Related Work
Prior work prepares during speech through prefix-based reasoning, response drafting, or maintained reasoning state, and decoupled systems exchange preparation through different interfaces. Heartbeat connects independently scheduled invocations through completed reasoning records and tool results.
2 Related Work
- Early preparation spans prefix-based reasoning in LiveMind (Chen et al., 2024), response drafting in PredGen (Li and Grover, 2025), and maintained reasoning state in LTS-VoiceAgent (Zou et al., 2026).Figure 1 contrasts when reasoning and tool use occur in cascaded pipelines, SHANKS, and Heartbeat.Source: §2 · p. 2
- Decoupled systems exchange preparation through completed tool results in SHANKS, streamed outputs in DuplexOmni (Huang et al., 2026), or model-internal state in AsyncReasoning (Yakushev et al., 2025).Source: §2 · p. 2
- Heartbeat connects independently scheduled invocations through completed reasoning records and tool results.Appendix F discusses these interfaces and streaming computation policies in more detail.Source: §2 · p. 2
3 Heartbeat Thinking
Heartbeat schedules overlapping Thinker invocations during listening and stores completed reasoning and tool results for later use. At the detected end of speech, a separate Responder receives a snapshot of completed preparation and handles the complete request.
3 Heartbeat Thinking
- Heartbeat receives streaming ASR transcripts, dialogue history, and tool schemas; a periodic Thinker prepares reasoning and tool calls during listening, followed by an endpoint-triggered Responder.The roles can use different models, and shared state stores completed reasoning records, requested calls, and returned results.Source: §3 · p. 2
- Every Δ seconds, a new Thinker reads the latest transcript and completed preparation; at most K invocations run concurrently, with triggers skipped when all slots are occupied.New speech and tool results enter subsequent invocations without interrupting ongoing generation.Source: §3 · p. 2
- Completed reasoning is recorded before requested tools run asynchronously, allowing model and tool execution to overlap and later invocations to use returned results for missing arguments or dependent calls.The prompt directs the Thinker to avoid repeating pending or completed calls with identical names and arguments.Source: §3 · p. 2
- At the detected end of speech, the scheduler stops new invocations and cancels unfinished speculative tasks; the Responder uses a snapshot of completed preparation and begins without waiting for pending work.It checks preparation against the complete request, rejects interpretations contradicted by later speech, and executes required calls that lack available results.Source: §3 · pp. 2–3, Figure 2 · p. 3
4 Evaluation
The evaluation compares Heartbeat with Window Thinking and other voice-agent baselines across three benchmarks, measuring reusable speculative preparation, task quality, and latency. Heartbeat has higher speculative coverage on all three benchmarks, while task-quality effects vary; on FDBench v3 it also reduces final-response latency.
4.1 Setup and Metrics
- The evaluation covers function calling on SpeechFCEval, disfluent intent following on FDBench v3, and knowledge-grounded answering on AudioCRAG.Baselines include Speech-to-Speech, Pipecat, LiveKit, and SHANKS-style Window Thinking; STC measures completed speculative calls reused without repetition, with an alternate sample-based measure on AudioCRAG.Source: §4.1 · pp. 3–4
4.2 Results
- Heartbeat reduces FDBench mean final-response latency from Window Thinking’s 6.90 to 4.39 seconds (36.4%), also 45.7% below Pipecat.Correct tool calls begin at −4.37 seconds relative to speech end, versus −2.74 seconds for Window Thinking and 2.45–3.52 seconds for cascaded baselines.Source: Figure 4 · p. 4, Figure 3 · p. 4, §4.2 · pp. 4–5
- Heartbeat gains 10.56, 16.00, and 42.37 percentage points in STC over Window Thinking on SpeechFCEval, FDBench, and AudioCRAG, respectively.On FDBench, Quality rises from 72.0% to 77.0% and Pass@1 from 57.0% to 65.0%, but Pass@1 remains below Speech-to-Speech and LiveKit; task performance falls on SpeechFCEval and AudioCRAG.Source: §4.2 · p. 4
- Heartbeat retains higher STC across all three tested Responders, and later invocations can incorporate late constraints and explicit intent revisions before speech ends.In 4.0% of FDBench recordings, answers use erroneous or outdated results despite correct tool-call arguments.Source: §4.2 · p. 5
5 Conclusion
Heartbeat Thinking overlaps speculative computation across speech updates and passes completed preparation to a separate Responder. Across three benchmarks it retains more reusable preparation than a SHANKS-style windowed baseline, while task-quality changes vary by benchmark.
5 Conclusion
- Heartbeat overlaps speculative computation across speech updates and passes completed preparation to a separate Responder.Across three benchmarks, it retains more reusable preparation than a SHANKS-style windowed baseline; on FDBench it also reduces final-response latency, while task-quality changes vary by benchmark.Source: §5 · p. 5
Limitations
The evaluation uses fixed scheduling and bounded Thinker concurrency, and its latency analysis is limited to FDBench v3.
Limitations
- Heartbeat uses a fixed scheduling interval and bounded Thinker concurrency; latency evidence is limited to FDBench v3.The authors have not explored adaptive scheduling or evaluated other tool workloads for the trade-off among speculative coverage, inference cost, and response latency.Source: Limitations · p. 5
Ethical Considerations
Collection and use of speech and dialogue traces require applicable consent and access controls. Upstream dataset licenses do not establish permission to redistribute derived audio or identifiable dialogue traces.
Ethical Considerations
- Consent and access controls must govern trace collection and use; upstream licenses do not establish redistribution permission for derived audio or identifiable dialogue traces.The SpeechFCEval dataset card declares Apache-2.0 and asks users to respect upstream terms; the FDBench and original CRAG text-data repositories publish CC BY-NC 4.0 licenses.Source: Ethical Considerations · p. 5
A.1 Task and framework overview
The framework prepares for a complete spoken request using partial transcripts, dialogue history, and tool schemas. Periodic Thinker invocations build reusable preparation; at speech end, a Responder checks that work against the full request and completes the answer.
A.1 Task and framework overview
- Heartbeat frames the task as answering the complete request and executing its required tool calls using dialogue history, tool schemas, and the full utterance.Streaming ASR supplies partial transcripts during listening and a final transcript when the utterance ends.Source: §A.1 · p. 6
A.2 Periodic asynchronous Thinker
- At fixed intervals, a Thinker reads the latest partial transcript and completed preparation, while new speech and tool results enter later invocations rather than interrupting ongoing generation.A trigger is skipped when all K concurrent invocation slots are occupied.Source: §A.2 · p. 6
- Thinker outputs can be no action, reasoning only, or reasoning with structured tool requests, which are recorded before requested tools execute asynchronously.The shared state links each requested call to its returned result, distinguishing pending work from results available for further reasoning.Source: §A.2 · p. 6
- Later invocations avoid repeating pending or completed calls with identical tool names and arguments, and can use returned results to advance dependent calls.Preparation can progress even when the partial transcript has not changed.Source: §A.2 · pp. 6–7
A.3 Endpoint-triggered Responder
- At the detected end of speech, the Responder receives the final transcript and available preparation, checks it against the complete request, reuses applicable results, and issues remaining calls.Unfinished speculative tasks are cancelled at handoff; outputs arriving after context assembly are discarded.Source: §A.3 · p. 7
B Benchmark Setup
The evaluation measures speculative tool coverage or retrieval reuse alongside task performance across three speech benchmarks. The setup specifies benchmark composition and scoring, model and scheduling configurations, and latency measurement conventions.
B Benchmark Setup
- Experiments use NVIDIA H200 GPUs; the main task-quality comparison uses DeepSeek-V4-Flash 0731 as Responder, while cascaded pipelines pair Qwen3-ASR with a text reasoning model.Speculative tool coverage measures reference calls completed and reused without repetition, with benchmark-specific matching rules.Source: B Benchmark Setup · p. 7
B.1 SpeechFCEval
- SpeechFCEval evaluates function calling across five task groups using 1,914 synthesized and human-speech samples, averaging accuracy equally by group and STC by sample.The test set combines Speech-ACEBench, Speech-BFCL, and SmartInteract; runtime failures count as zero.Source: §B.1 · p. 7, B Benchmark Setup · p. 7
B.2 FDBench v3
- FDBench v3 evaluates spoken-intent following under disfluency across 100 recordings, with answer quality judged by DeepSeek-V4-Flash and latency measured from aligned speech end.Final Response Latency measures reply onset after the last tool call, or reply onset when no call is required; recordings receive equal weight in latency aggregation.Source: §B.2 · pp. 7–8
B.3 AudioCRAG
- Table 4 lists the speech-pipeline and heartbeat scheduling settings for the FDBench v3 latency evaluation.AudioCRAG evaluates knowledge-grounded spoken answering on 1,312 samples, using a 500 ms heartbeat interval and at most two concurrent Thinker invocations.Source: Table 4 · p. 8, §B.3 · p. 8
C.1 Task Results
Heartbeat increases speculative coverage or retrieval reuse over Window Thinking across the evaluated benchmarks, but final task-quality effects vary. Model comparisons likewise show that greater coverage does not consistently correspond to higher task performance.
C.1 Task Results
- 10.56 percentage points: SpeechFCEval STC increases from 33.13% with Window Thinking to 43.69% with Heartbeat, but overall function-calling accuracy is 66.85% versus 67.95%.Coverage rises in four of the five task groups, while SmartInteract coverage decreases slightly from 40.57% to 38.32%.Source: §C.1 · p. 8
- 16.00 percentage points: FDBench v3 STC rises from 34.67% with Window Thinking to 50.67% with Heartbeat, while response quality rises from 72.0% to 77.0% and Pass@1 from 57.0% to 65.0%.Heartbeat’s Pass@1 remains below Speech-to-Speech and LiveKit.Source: §C.1 · p. 8
- 42.37 percentage points: AudioCRAG STC rises over Window Thinking, but Heartbeat’s correct-answer rate is 55.49% versus 57.24%, and its missing-answer rate is 16.84% versus 14.33%.The supplied passage reports Heartbeat at 62.42% STC and Window Thinking at 20.05% STC.Source: §C.1 · p. 8
C.2 Model Comparisons
- On SpeechFCEval, Gemma has higher accuracy than the default Thinker, 68.58% versus 66.85%, despite slightly lower STC, 42.71% versus 43.69%.Across the three benchmarks, the passage reports nonzero STC for all four tested Thinkers.Source: §C.2 · p. 8
- On SpeechFCEval, Qwen3.8-Flash-Next has higher STC than Qwen3.5-397B-A17B, 55.27% versus 41.20%, but lower accuracy, 64.30% versus 69.70%.Heartbeat retains higher STC than Window Thinking with all three Responders on all three benchmarks.Source: §C.2 · p. 8
D Speculative Tool-Use Analysis Details
The section analyzes when speculative calls are initiated, how preparation is counted and handed to the Responder, and where speculative results can fail to support the final answer.
D Speculative Tool-Use Analysis Details
- Early and late groups measure calls with final-request-consistent arguments initiated before speech end, while revision cases measure whether an outdated call is followed by a corrected one.These initiation measures differ from speculative tool coverage, which requires completed results in the Responder context without response-stage repetition.Source: D Speculative Tool-Use Analysis Details · pp. 8–9
D.1 Timing and Case Studies
- 73.3% of early-information recordings, 49.2% of late-information recordings, and 70.6% of revision recordings had a correct or corrected call initiated before speech ended.Revision cases count corrected calls following an outdated request.Source: Table 10 · p. 10, §D.1 · p. 10
- In 4.0% of recordings, the final answer used erroneous or outdated speculative results despite a correctly specified request; a corrected search could not undo an earlier cart action.The cases distinguish timely speculative execution from correct use of its results.Source: §D.1 · p. 10
E.1 Overlapping jobs and the response boundary
- At the response boundary, only completed speculative work available at context assembly enters the initial Responder context; unfinished tasks are cancelled and later results are excluded.Completed reasoning may be available even when its requested tool result is not, and overlapping invocations can finish out of order.Source: §E.1 · p. 10
E.2 Interpreting speculative tool coverage
- Coverage counts each reference call at most once and removes a match when the Responder repeats the same tool name with exactly the same arguments.A two-call request can therefore have 1/2 non-repeated coverage either because of repetition or because a result arrives after context assembly.Source: §E.2 · pp. 10–11
F Extended Related Work
- Heartbeat overlaps successive Thinker invocations, unlike chunk-bounded or interrupt-on-update approaches, and connects them to asynchronous tools and an endpoint-triggered Responder through shared records.The related-work discussion also contrasts preparation ranging from reasoning context to reusable response text and audio.Source: F Extended Related Work · pp. 11–12