Source-linked AI summary
FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech
Puneet Mathur, Dinesh Manocha
TL;DR
Full-duplex interaction requires distinguishing hesitation from completed intent using partial speech, but acoustic VAD lacks semantic information and ASR cascades add transcription dependence. FD-VAD makes causal audio-to-language CONTINUE/STOP decisions without intermediate transcription, and reaches 0.853 TurnBench dev EOT recall at FP ≤0.10 in zero-shot evaluation.
Problem
Acoustic VAD cannot determine whether a pause reflects hesitation or completed intent, and streaming endpoint decisions must be made from partial audio without future speech.
Source: §1 · p. 1Method
Results
FD-VAD achieves the highest TurnBench dev EOT recall among compared systems in zero-shot evaluation, at 0.853 with FP ≤0.10.
Source: §1 · p. 2Takeaways & Limitations
FD-VAD evaluations support direct semantic endpointing from streaming audio without intermediate transcription, while remaining modular with respect to the downstream voice agent.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic voice activity detection lacks this semantic information, while cascaded ASR-based endpointing introduces transcription dependence and additional processing stages. We formulate semantic endpoint detection as a causal audio-language reasoning task and introduce FD-VAD, an ASR-free streaming endpointer that maps bounded causal audio windows directly to Continue/Stop decisions. FD-VAD combines a frozen speech encoder with a lightweight modality adapter and a parameter-efficiently adapted language model, using a last-chunk training objective for streaming inference. We further introduce confidence-gated endpoint commitment to control interruption versus delay and boundary-focused hard-negative sampling to improve decisions around ambiguous turn boundaries. Across in-domain and conversational evaluations, FD-VAD outperforms strong streaming and non-streaming semantic turn classifiers, and achieves the highest EOT recall among qualifying systems on TurnBench dev set $0.853$ (at FP<=0.10) in a zero-shot setting. These results show that semantic endpointing can be performed directly from streaming audio without intermediate ASR or dialogue state tracking.
1. INTRODUCTION
Full-duplex agents must infer from partial speech whether a pause signals hesitation or a completed conversational intent, a distinction acoustic VAD alone cannot make. FD-VAD addresses this with causal audio-to-language endpoint decisions, confidence-gated commitment, and boundary-focused training.
1. INTRODUCTION
- Acoustic silence cannot distinguish a completed conversational act from a hesitation, while streaming decisions must use partial audio without future speech.Misjudging a pause can either interrupt a speaker or delay a response in full-duplex interaction.Source: §1 · p. 1
- FD-VAD maps bounded causal audio windows directly to CONTINUE/STOP decisions without transcript generation or a separate acoustic-to-semantic cascade.A frozen speech encoder and lightweight adapter provide audio representations to a language model, with supervision on the latest chunk.Source: §1 · pp. 1–2
- Confidence-gated endpoint commitment controls the interruption–latency trade-off, while boundary-focused hard-negative sampling emphasizes ambiguous regions near true endpoints.The gate uses predicted stop probability and consecutive decisions required before commitment.Source: §1 · p. 2
- FD-VAD achieves the highest TurnBench dev EOT recall among compared systems in zero-shot evaluation, at 0.853 with FP ≤0.10.It also matches a strong non-streaming turn classifier at utterance level, 0.965 vs. 0.966.Source: §1 · p. 2
2. RELATED WORK
Prior endpointing systems use acoustic boundaries, transcription or other intermediate signals, completed-utterance classification, or dialogue-coupled turn-taking. FD-VAD instead predicts semantic turn state directly from causal audio windows.
2. RELATED WORK
- FD-VAD removes transcription, completed-segment, and intermediate-signal dependencies by mapping observed speech directly to a semantic CONTINUE/STOP state.Unlike full-duplex models that jointly generate responses or dialogue-state controllers, it remains independent of the downstream voice agent.Source: §2 · p. 2
- FD-VAD specializes the audio encoder–adapter–LLM paradigm for causal endpoint prediction using bounded audio context and a last-chunk objective.This contrasts with related methods using ASR, CTC representations, silence-triggered inference, or time-to-next-speech supervision.Source: §2 · p. 2
- A causal audio window is encoded by frozen WavLM and projected into a LoRA-adapted Qwen2.5-0.5B model for CONTINUE/STOP prediction.Training supervises only the final chunk of each window.Source: Figure 2 · p. 2
3. METHOD
FD-VAD combines a frozen speech encoder, a modality adapter, and a parameter-efficiently adapted language model to predict semantic completeness from bounded causal audio windows. Its training objective supervises each window’s final chunk and boundary-focused sampling reallocates training toward ambiguous cases.
3. METHOD
- FD-VAD uses frozen WavLM features, a two-layer modality adapter, and Qwen2.5-0.5B-Instruct with jointly optimized adapter and LoRA parameters.The adapter downsamples 50 Hz features to 12.5 Hz audio tokens; 10.3M parameters, or 1.7% of the total, are trainable.Source: §3 · p. 2
- Every 320 ms, inference predicts from a causal window containing at most 2.56 s of left context.The window yields approximately 128 WavLM frames and 32 audio tokens after adapter downsampling.Source: §3 · p. 3
- Training supervises only the terminal chunk of each window with a CONTINUE or STOP target followed by an end-of-sequence token.This last-chunk objective trains the state at the most recent observed chunk.Source: §3 · p. 3
- Boundary-focused sampling reallocates training mass to examples within ±2 chunks of the endpoint while leaving inference unchanged.It targets ambiguous boundary cases rather than changing the inference architecture.Source: §3 · p. 3
4. DATA AND EXPERIMENTAL SETUP
The experiments train FD-VAD on conversational endpointing data and assess it on held-out utterances and zero-shot conversational dialogue using chunk-, utterance-, and event-level measures.
Training Data
- The English smart-turn-v3.1 training set contains 63,386 clips, with complete utterances labeled STOP from the acoustic offset and incomplete utterances labeled CONTINUE throughout.The corpus includes 42,529 human and 20,857 TTS recordings; incomplete examples are corpus-native rather than artificially truncated.Source: §4 · p. 3
Evaluation Data and Protocols
- Evaluation includes a held-out 1,000-clip in-domain test set, synthetic-to-human transfer, and zero-shot TurnBench dialogue evaluation.TurnBench uses real two-channel human dialogue and event-anchored EOT evaluation; FD-VAD receives no TurnBench training.Source: §4 · p. 3
- The held-out smart-turn-v3.1 evaluations report chunk-level semantic endpoint detection and utterance-level classification across complete and incomplete turns.Table 1 covers 1,000 utterances, and Table 2 reports 95% bootstrap confidence intervals over 2,000 samples.Source: Table 1 · p. 3, Table 2 · p. 3
Evaluation Metrics
- Chunk-level measures include precision, recall, F1, accuracy, and false-stop rate, while utterance-level measures include endpoint accuracy, strict sentence accuracy, and near-boundary F1.Confidence-gated evaluation pairs false-interruption rate with endpointing delay; TurnBench reports recall at FP ≤0.10 and median commit latency.Source: §4 · p. 3
5. RESULTS
FD-VAD achieves the strongest qualifying zero-shot TurnBench EOT recall, while in-domain results show balanced endpoint performance. The section also reports latency trade-offs, ablation trends, and synthetic-to-human transfer limits.
In-Domain Chunk and Utterance Performance
- FD-VAD performs consistently at utterance level on complete and incomplete turns (0.972/0.969), unlike acoustic VAD, which drops from 0.998 to 0.680 on incomplete speech.At chunk level, FD-VAD reaches F1=0.832 with a 0.6% false-stop rate on incomplete utterances.Source: §5 · p. 3
Zero-Shot Conversational EOT Transfer
- 0.853 EOT recall is the highest among qualifying systems on TurnBench dev at FP ≤0.10, with FD-VAD evaluated zero-shot.Its median commit latency is 1019 ms, above VAP’s 463 ms; gated FD-VAD benefits substantially from acoustic gating in continuous dialogue.Source: Table 3 · p. 4, §5 · p. 4, Figure 3 · p. 4
Robustness to Training Data Source
- Synthetic-only training retains similar per-chunk performance on held-out human speech, with a 1.1-point reduction, but degrades stricter all-chunk utterance metrics.The result emphasizes the importance of human training data for consistent endpoint decisions across complete utterances.Source: Table 4 · p. 4, §5 · p. 4
Streaming Latency Analysis
- 45.8 ms per 2.56 s causal window keeps FD-VAD’s processing cost constant with utterance duration, while its operating point yields 190 ms median delay and 2.0% false-interruption rate.The 320 ms inference stride updates endpoint state as speech arrives rather than waiting for utterance completion.Source: §5 · p. 4
Ablations and Stability
- Reducing stride from 320 ms to 160 ms slightly raises aggregate chunk accuracy but degrades complete-turn and near-boundary F1; freezing the adapter also degrades endpoint metrics.Boundary-focused sampling yields the strongest complete-turn, near-boundary, and strict-sentence performance; three-seed variation is small at 0.9804 ± 0.0005 chunk accuracy.Source: §5 · p. 4, Table 4 · p. 4
6. CONCLUSION
The conclusion presents FD-VAD as causal audio-language reasoning over partial speech for streaming semantic endpoint detection. Evaluations show transfer to continuous real-time dialogue, where it complements acoustic gating.
Conclusion
- FD-VAD reformulates turn completion as causal audio-language reasoning over partial speech.It maps bounded audio context directly to CONTINUE/STOP decisions without intermediate transcription.Source: §6 · p. 4
- The approach remains modular with respect to the downstream voice agent.The conclusion characterizes direct audio-to-language reasoning as a simple and portable foundation for turn-taking in full-duplex spoken agents.Source: §6 · p. 4
- Evaluations show FD-VAD transfers to continuous real-time dialogue, where semantic reasoning complements conventional acoustic gating.Source: §6 · p. 4