PaperlayerView exact revision on arXiv

Source-linked AI summary

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng

arXiv:2610.34653v1cs.CLeess.AS

TL;DR

Spoken agents and corpora lack a defined earliest audio-grounded point at which a tool and its arguments are ready, despite the latency of waiting for the turn to end. The paper constructs a gated, aligned synthetic corpus and trains a chunk-level omni-modal model to detect that point and supply call payloads. It reports tool-state macro-F1 of 0.48 and 0.47 on held-out and external data, with correct calls starting an estimated 1.8–2.0 seconds early in-domain.

  • Problem

    Existing spoken agents and spoken-dialogue corpora do not define the earliest instant when audio grounds a tool call and all its required arguments, leaving timing unscored against that point.

    Source: §1 · p. 1

  • Method

    The authors build a gated synthetic spoken-dialogue corpus with aligned earliest-actionable-point labels and train an omni-modal model to decide per chunk when to fire and emit a call or partial transcript.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    0.48 and 0.47 tool-state macro-F1 on held-out dialogues and audio-rendered τ 2-bench accompany an estimated 1.8–2.0 s mean lead for correct in-domain calls over a turn-end cascade using the same formatter.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The results provide a data-driven path toward spoken agents that act while listening, with the firing decision still a bottleneck.

    Source: §1 · p. 1

    The turn-end baseline assumes zero transcription latency, an ideal streaming recognizer and an assumption most favorable to the baseline.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $\tau^2$-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $\tau^2$-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.

1. INTRODUCTION

Spoken agents typically wait until a user finishes before deciding on a tool call and its arguments, even when the request may already be actionable. Existing systems and benchmarks do not define or evaluate the earliest audio-grounded instant when that call can be made; this paper frames that gap as both a data and modeling problem.

1. INTRODUCTION

  • The paper targets the delay before tool selection and argument filling, which begins only after the user finishes in existing cascaded or end-to-end voice agents.The authors distinguish this decision step from transcription speed and the tool’s round-trip time.Source: §1 · p. 1
  • Prior incremental systems shorten perceived waits or issue queries while listening, but their timing relies on proxies, LLM-judged offsets, or change-detection labels.The cited work does not define and verify the earliest point at which a call becomes decidable.Source: §1 · p. 1
  • Full-duplex, turn-taking, and incremental-dialogue systems address floor control or early meaning, while tool benchmarks evaluate calls from completed turns rather than when they become callable.The paper identifies the missing target as the earliest instant when the audio grounds both the tool and every required argument.Source: §1 · p. 1
  • The paper builds a synthetic corpus with stamped actionable instants and disjoint train–test argument values, then trains a chunk-level model to decide when to fire and what to emit.It evaluates on held-out dialogues and an audio rendering of τ 2-bench; correct calls start an estimated 1.8–2.0 s early, while the firing decision remains a bottleneck.Source: §1 · p. 1

2. METHODS: DATA PIPELINE AND MODELING

The method labels the earliest chunk at which speech grounds every required tool argument, then trains an omni-modal model to make chunk-level tool-state decisions and provide payloads for call completion.

2. METHODS: DATA PIPELINE AND MODELING

  • The earliest-actionable-point label, t_fire, marks when every required argument is grounded in the speech heard so far; the corresponding gold saving is turn end minus t_fire.READY commits to a specific call and UNSURE flags a likely but under-specified tool; either state counts as a fire.Source: §2 · pp. 1–2

2.1. Data pipeline

  • The pipeline instantiates 48 domains’ typed task templates with a deterministic entity sampler, simulates dialogues, synthesizes aligned speech, and annotates gated tool-call events.It retains 26,694 dialogues from 27,000 generated profiles; the figure depicts the stages and per-chunk labels.Source: §2.1 · p. 2, Figure 1 · p. 2
  • Three deterministic gates filter invalid calls, reject ungrounded or superseded arguments, and repair arguments or delay t_fire until they are grounded.The gates prune 140,595 proposed events to 53,492; because they can only delay t_fire, the label is an upper bound on the true earliest point.Source: §2.1 · p. 2
  • Task-group holdout yields zero train/test argument-value overlap, and the τ 2-bench rendering provides an external evaluation set.The τ 2 column in Table 1 denotes that external, eval-only rendering.Source: §2.1 · p. 2, Table 1 · p. 2

2.2. Modeling

  • The model keeps Qwen3-Omni-30B-A3B frozen, trains control-token embeddings and output rows, and attaches LoRA to attention and expert feed-forward projections.The control tokens cover voice activity, early-tool-call states, payload delimiters, and pending markers.Source: §2.2 · p. 2
  • For each 0.6 s audio chunk, the model predicts voice activity and, on speech, a tool state; READY or UNSURE opens one of three payload formats.ASR+JSON emits a self-contained call, ASR sends a partial transcript to a frozen formatter, and NONE requires external ASR near the fire.Source: §2.2 · p. 2

3. EXPERIMENTS

The experiments evaluate chunk-level tool decisions and call completion on held-out dialogues and audio-rendered τ²-bench, comparing three payload variants with turn-end baselines. Correct calls can fire before the turn ends, but decision precision and cross-domain payload quality remain challenges.

3.1. Setup

  • The setup trains payload variants with LoRA rank r=16 and evaluates streaming decisions on held-out and audio-rendered τ²-bench splits.Chunk-level VAD and ET states are scored against gold with exact alignment; event matching allows a same-state gold event within ±2 chunks.Source: §3.1 · pp. 2–3

3.2. Results

  • ASR achieves the best ID event F1 at 0.414 and the best ET and VAD state classification, while event recall is around 0.5 on ID and 0.56–0.58 on τ²-bench.The payload-free NONE variant has precision of only 0.123 on ID and 0.147 on τ²-bench.Source: Table 2 · p. 3, §3.2 · p. 3
  • One chunk of lateness tolerance lifts READY F1 by 0.06–0.10 and macro-F1 by 0.03–0.04, with little further gain at +3 chunks.False-positive READY fires are the precision bottleneck, while WAIT is nearly solved.Source: Table 3 · p. 3, §3.2 · p. 3
  • TURN reaches recall 0.894 but fires on 751–789 of 1,499 turns, about two-thirds without a gold READY; ASR+JSON has the highest precision at 0.409 after giving up 0.3 recall.Across the four systems, F1 is within 0.05; at this operating point, 59% of early fires are spurious.Source: §3.2 · pp. 3–4, Table 4
  • 1.83 s (ASR) and 2.01 s (ASR+JSON) are the mean ID savings for correct calls; on τ²-bench, early correct calls save a median 1.8 s.About half of correct ID calls fire before turn end; the distributions are right-skewed, so the means are carried by a long tail.Source: Figure 3 · p. 4, §3.2 · p. 4

4. ABLATION: PRODUCING THE CALL

The ablation compares ways to turn READY fires into calls, finding that a model-generated transcript passed to a formatter outperforms self-emitted JSON. Gold-text comparisons indicate that transcript fidelity remains a major source of call errors.

4. ABLATION: PRODUCING THE CALL

  • EM reaches 0.536 when a formatter builds calls from ASR+JSON’s own span, versus 0.173 from the model’s own JSON.The formatter reaches tool naming accuracy of 0.83 and argument-slot accuracy of 0.71, compared with 23% and 52% for the trained JSON head.Source: §4 · p. 4, Table 5 · p. 4
  • EM reaches 0.578 for the ASR variant, which was trained without a JSON head.The table compares call quality on each variant’s own READY fires, using the model’s span, Parakeet text, or gold transcript as applicable.Source: §4 · p. 4, Table 5 · p. 4
  • With gold text, tool-name ACC remains 0.84 while argument accuracy rises to 0.76 and EM to 0.63, pointing to span fidelity as the residual error.The 30B formatter loses at most 4.3 EM points.Source: §4 · p. 4, Table 5 · p. 4
  • NONE reaches Tool ACC of 0.87 but EM of 0.038 on ID, as external Parakeet ASR fills only half the argument slots.On τ 2, NONE reaches EM 0.090, still roughly 7× below the span variants; Parakeet entities in spoken form do not string-match the gold JSON.Source: §4 · p. 4, Table 5 · p. 4
  • TURN-ASR reaches EM 0.103 with Parakeet text and EM 0.605 with gold text; on τ 2, the corresponding values are 0.073 and 0.558.The passage attributes the Parakeet-arm shortfall to the same normalization failure, not timing.Source: §4 · p. 4, Table 5 · p. 4

5. CONCLUSION

The paper frames spoken tool calling as deciding when the audio first supports an executable call. Its pipeline and chunk-level model show that calls can be made early, while fire precision remains an open gap.

5. CONCLUSION

  • The model reaches tool-state macro-F1 of 0.48 on held-out dialogues and 0.47 on audio-rendered τ 2-bench, with correct early calls estimated 1.8–2.0 s ahead of a turn-end cascade in-domain.The pipeline labels 26,694 dialogues with disjoint train/test argument values; 59% of early fires are spurious, and the study uses one seed and one base model.Source: §5 · p. 4

Related papers

Loading 2610.34653v1…