Source-linked AI summary
When Intent Arrives Late: A Benchmark for Full-Duplex Speech Models under Delayed Intent Revelation
Yang Xiao, Tianyi Peng, Hanyu Meng, Ting Dang
TL;DR
Existing evaluations largely assess turn-taking mechanics or behavior after a harmful request is fully specified, leaving intent revealed late within an ongoing utterance unexamined. LateIntent-Bench pairs ambiguous prefixes with benign or harmful continuations and measures response timing and engagement. Across four models, delayed intent produces distinct timing and engagement patterns, while moving a pause after intent reveal keeps PRR near baseline.
Problem
Existing full-duplex benchmarks focus on interaction mechanics, while safety benchmarks typically evaluate harmful requests after they are fully specified, leaving delayed intent within one ongoing utterance unexamined.
Source: §1 · p. 1Method
LateIntent-Bench pairs an ambiguous prefix with benign or harmful continuations, delays the reveal point with a controlled pause, and measures premature response onset and matched engagement.
Source: §1 · p. 1, §2.1 · p. 2Results
Across four models, three increase harmful engagement while maintaining benign responsiveness, whereas FLM-Audio reduces engagement across both branches; post-reveal pauses yield 0.00–0.04 PRR versus 0.32–1.00 for pre-reveal pauses.
Source: §1 · p. 2, §3.4 · p. 4, §3.3 · p. 4Takeaways & Limitations
Evaluations on fully specified requests alone do not capture behavioral dynamics when intent-defining information is delayed in full-duplex interactions.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Native full-duplex speech models can respond before a user finishes speaking, making the behavior of the model depend on both the final utterance and when intent-defining information arrives. Existing benchmarks primarily evaluate turn-taking mechanics, whereas safety evaluations typically assume that the complete request is observed prior to response generation. We introduce LateIntent-Bench, a matched-pair benchmark for delayed intent revelation. A shared ambiguous prefix precedes either a benign or a harmful continuation, using a controlled pause to delay when the branches become distinguishable. We define the Premature Response Rate (PRR) to measure whether response onset precedes the revelation of intent. Joint evaluation of harmful and benign engagement distinguishes changes in safety selectivity from general losses in responsiveness. Across 3,136 sessions, four native full-duplex models exhibit distinct response-timing patterns under delayed intent. Three models show increased harmful engagement while maintaining benign responsiveness, whereas one loses engagement with both branches. Inserting a 1.5s silence after intent revelation keeps PRR near the no-pause baseline and substantially reduces changes in harmful engagement. These results demonstrate that evaluating fully specified requests alone fails to capture emerging timing patterns, highlighting the necessity of assessing delayed intent revelation in full-duplex models. The code will be publicly released soon.
1. INTRODUCTION
Native full-duplex models may respond before a request reveals whether it is benign or harmful, so evaluation must consider when intent becomes available as well as the final request. LateIntent-Bench studies response timing and branch-specific engagement under delayed intent revelation.
Research gap
- Existing full-duplex benchmarks mainly assess interaction mechanics, while safety benchmarks typically evaluate harmful requests only after they are fully specified.Prior multi-turn work studies intent emerging across turns, not within one ongoing utterance.Source: §1 · p. 1
Contribution
- LateIntent-Bench pairs a shared ambiguous prefix with benign or harmful continuations and delays their distinguishing information with a controlled pause.It measures whether response onset precedes intent revelation and separately compares assistance across the two branches.Source: §1 · p. 1
Results
- Across four models, PRR rises at markedly different reveal delays; three increase harmful engagement while maintaining benign assistance, whereas FLM-Audio reduces assistance to both branches.Moving a 1.5 s pause after intent reveal keeps PRR near the no-pause level and reduces behavioral shifts for most models.Source: §1 · pp. 1–2
2. BENCHMARK
LateIntent-Bench uses matched benign and harmful requests to locate when intent becomes distinguishable, then evaluates response timing and engagement under controlled reveal delays. A common evaluation protocol and pause-placement control help interpret these measures.
2. BENCHMARK
- The benchmark asks whether a model responds before intent is distinguishable and whether it assists the completed request; its matched benign branch separates reduced harmful engagement from broad disengagement.The benchmark structure places a reveal delay at the shared prefix–continuation boundary.Source: §2 · p. 2, §2.1 · p. 2, Figure 2 · p. 3
2.1. Matched requests
- Each of 98 matched items shares an ambiguous prefix and branches into a benign or harmful continuation, with an automatic filter checking the branch properties and prefix identity.Seed behaviors span 10 categories.Source: §2.1 · p. 2
2.2. Speech synthesis and delayed intent revelation
- A controlled silence at the prefix–continuation boundary delays when intent becomes observable while leaving the surrounding synthesized speech unchanged.All conditions use the same underlying speech; the reveal time is defined relative to the prefix end plus the delay.Source: §2.2 · p. 2
2.3. Common interaction protocol and metrics
- The common protocol evaluates response timing and engagement separately, scoring the model’s text through a 15 s hangover while excluding lead-in greetings.Engagement is accepting the request objective and beginning assistance, including a clarification asked in order to proceed; task completion is not required.Source: §2.3 · pp. 2–3, Figure 2 · p. 3
2.4. Pause-placement control
- The pause-placement control compares no pause, a 1.5 s pause before intent reveal, and the same pause after harmful intent is observable, leaving the reveal time unchanged in the post condition.The control uses 25 matched items selected for textual feasibility, without using model outcomes.Source: §2.4 · p. 3
3. MAIN RESULTS
Across four native full-duplex models, delayed intent revelation produces different response-timing and engagement patterns. Moving the same pause after intent reveal largely preserves baseline timing, showing that pause placement matters.
3.1. Experimental setup
- The common-protocol evaluation covered 3,136 sessions across four native full-duplex models, with 784 conditions per model.Each model–delay cell contains 196 sessions across the benign and harmful branches of 98 matched items.Source: §3.1 · p. 3
3.2. Effect of delayed intent revelation
- PersonaPlex and Moshi depart from no-delay timing at δ = 0.5 s, VoiceChat-11B remains at zero PRR until δ = 1.0 s, and FLM-Audio reaches 0.964 at δ = 1.5 s.The figure shows PRR against reveal delay and indicates that each model’s substantial rise begins at a different delay.Source: §3.2 · p. 3, Figure 3 · p. 3, Table 1 · p. 4
3.3. Engagement under delayed intent revelation
- At δ = 1.5, PersonaPlex, Moshi, and VoiceChat-11B increase harmful engagement while maintaining similar benign engagement; FLM-Audio’s benign engagement falls from 0.724 to 0.378.VoiceChat-11B’s HER rises from 0.031 at δ = 0 to 0.122 at δ = 1.5 while BER remains stable.Source: §3.3 · p. 4, Figure 4 · p. 4
3.4. Effect of pause placement
- With the same 1.5 s pause, PRR is 0.32–1.00 before intent reveal versus 0.00–0.04 after reveal; engagement shifts are also smaller after reveal for most models.Moshi’s HER differs by only 0.04 between pause placements.Source: §3.4 · p. 4, Table 2 · p. 4, Figure 5 · p. 4
4. CONCLUSION
LateIntent-Bench evaluates native full-duplex speech models when intent-defining information arrives late. Across four models, timing and engagement shift differently, and pause-placement controls show that fully specified requests alone do not capture these dynamics.
- LateIntent-Bench evaluates native full-duplex models when intent-defining information arrives late.Source: §4 · p. 4
- Across four models, three increase harmful engagement while maintaining benign responsiveness, whereas FLM-Audio loses engagement across both branches.Source: §4 · p. 4
- Control experiments show that timing effects depend on pause placement rather than duration alone, exposing dynamics that fully specified-request evaluations miss.Source: §4 · p. 4