Source-linked AI summary
Voice Activity Projection: Self-supervised Learning of Turn-taking Events
Erik Ekstedt, Gabriel Skantze
TL;DR
Turn-taking is difficult to identify from dialog because pauses, overlaps, and interruptions make turn boundaries ambiguous. The paper proposes Voice Activity Projection, which models dependencies across a window of future voice activity and maps predictions to zero-shot event tasks. Its objective matches or exceeds prior-work performance on existing tasks and significantly outperforms alternatives on two new predictive tasks.
Problem
Method
Results
Takeaways & Limitations
Jointly modeling a VAP window provides outputs corresponding to specific future states and is favorable for more complex predictive turn-taking tasks.
Source: §6 · p. 4
Abstract
from arXiv · showhide
The modeling of turn-taking in dialog can be viewed as the modeling of the dynamics of voice activity of the interlocutors. We extend prior work and define the predictive task of Voice Activity Projection, a general, self-supervised objective, as a way to train turn-taking models without the need of labeled data. We highlight a theoretical weakness with prior approaches, arguing for the need of modeling the dependency of voice activity events in the projection window. We propose four zero-shot tasks, related to the prediction of upcoming turn-shifts and backchannels, and show that the proposed model outperforms prior work.
1. Introduction
Turn-taking is difficult to model from turn boundaries because pauses, overlaps, interruptions, and optional turns resist objective labeling. The paper motivates voice-activity modeling as a self-supervised alternative and extends prior work with a revised objective and new zero-shot tasks.
1. Introduction
- Pauses, overlaps, interruptions, and optional turns make objective turn identification difficult, while silence-threshold policies risk interruptions or sluggish responses.Source: §1 · p. 1
- Voice activity transitions identify SHIFTs, within-speaker silences identify HOLDs, and short isolated activity can correspond to backchannels.Source: §1 · p. 1
- Supervised turn-taking models use human-annotated labels, making data collection costly while giving their outputs high model utility.Source: §1 · p. 1
- Prior self-supervised work predicts future voice activity and uses its outputs for zero-shot SHIFT/HOLD and more complex turn-taking tasks.Source: §1 · p. 2
- The paper identifies a theoretical weakness in prior work, proposes an alternative, and reports better performance on two new zero-shot tasks with equal or greater performance on existing tasks.Source: §1 · p. 2
2. Voice Activity Projection
Voice Activity Projection predicts each interlocutor’s future voice activity in a discretized window. The paper’s proposed objective models the window’s bins jointly rather than independently, preserving dependencies among possible future activity patterns.
2. Voice Activity Projection
- Voice Activity Projection predicts the future voice activity of each interlocutor in a dialog.Source: §2 · p. 2
- The VAP window discretizes future voice activity into fixed bins, using a threshold to label bins active or inactive.Source: §2 · p. 2
- Independent bin probabilities do not statistically model the VAP window as a whole and can imply heavily out-of-distribution configurations.Source: §2 · p. 2
- The Discrete objective assigns probabilities to complete VAP-window states, preserving bin dependencies across 256 possible configurations.Source: §2 · p. 2
- The VAP window spans 2 seconds, with bin durations increasing farther into the future (200, 400, 600, and 800 ms), where prediction is expected to be harder and time resolution lower.Source: §2 · p. 2
- The four compared models are Discrete, Independent, Independent-40, and Comparative, which predicts the speakers’ voice-activity ratio as one scalar.Source: §2 · p. 2
3. Zero-shot classification of turn-taking events
The paper defines four zero-shot tasks for predicting turn-taking events by directly mapping VAP outputs to task classes. Two tasks are adopted from prior work, while SHIFT prediction and backchannel prediction are introduced here.
3. Zero-shot classification of turn-taking events
- Four zero-shot tasks map VAP predictions directly to event classes; SHIFT prediction and backchannel prediction are new, while two tasks come from prior work.Source: §3 · p. 3
- SHIFT/HOLD evaluates the next speaker during mutual silence, using surrounding single-speaker regions to restrict evaluation to proper turn shifts.Source: §3 · p. 3, Figure 3 · p. 3
- SHIFT prediction detects an upcoming shift while a speaker remains active, using 500 ms before a shift as positive samples.Source: §3 · p. 3
- Backchannel prediction targets short, isolated activity preceded by the other speaker, with 500 ms before a backchannel as positive samples.Source: §3 · p. 3
- For backchannel prediction, the Independent models compare early non-active-speaker activity with later activity, while the Comparative model is not evaluated.Source: §3 · p. 3
- SHORT/LONG distinguishes a backchannel onset from a proper-shift onset using the first 200 ms of the segment.Source: §3 · p. 3
4. Model & Training
The experiments compare VAP objectives using a shared speech-and-voice-activity encoder and causal transformer predictor. Models are trained and evaluated on splits of the Switchboard dataset.
4. Model & Training
- A shared base model uses frame-wise speech and voice-activity encoders followed by a sequence predictor to compare VAP objectives.Source: §4 · p. 4
- The speech encoder processes raw waveforms with pretrained CPC features, while voice-activity inputs include current frames and a longer-range activity-ratio history.Source: §4 · p. 4
- A causal decoder-only transformer feeds a VAP head, with cross-entropy for the Discrete model and binary cross-entropy for the other models.Source: §4 · p. 4
- Training and final evaluation use Switchboard: 2,205 dialogs for training and 135 held-out dialogs for testing after excluding dialogs with more than two speakers.Source: §4 · p. 4
5. Results
The proposed model significantly outperformed alternatives on the two prediction tasks, while showing no significant difference on SHIFT/HOLD or SHORT/LONG. Backchannel prediction benefited most from the model’s treatment of dependencies among future voice-activity events.
- Evaluation used a 3s minimum-context threshold, then test-set weighted F1 averaged across 11-kfold splits by model type.Source: §5 · p. 4, Table 1 · p. 4
- The proposed model significantly outperformed alternatives on SHIFT-prediction and Backchannel-prediction, while S/H and S/L differences were not significant.Source: Table 1 · p. 4, §5 · p. 4
- Backchannel prediction benefited most from the Discrete model, and it has the most complex assumptions about speakers’ future voice activity.Source: §5 · p. 4
6. Conclusion and Future Work
The paper models future voice activity jointly across a projection window rather than independently, and reports equal or better performance on prior zero-shot tasks and significant gains on two predictive tasks. It argues that this objective can benefit future work on more complex conversational dynamics and proposes exploring alternative window representations and prosodic information.
- The proposed objective matched or exceeded prior work on SHIFT/HOLD and SHORT/LONG, and was significantly better on SHIFT-prediction and Backchannel-prediction.Source: §6 · p. 4
- The objective models an entire voice-activity projection window jointly, producing outputs corresponding to specific future states.Source: §6 · p. 4
- Future work could test additional projection-window representations and information such as prosody for modeling conversational dynamics.Source: §6 · p. 4
Related papers
This paper cites
- Attention Is All You Need2017
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2018
- Language Models are Few-Shot Learners2020
- PyTorch: An Imperative Style, High-Performance Deep Learning Library2019
- A Simple Framework for Contrastive Learning of Visual Representations2020
- Representation Learning with Contrastive Predictive Coding2018
Cited by
- Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems2026
- VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model2026
- HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models2026
- Jarvis: A Proactive Speech Agent for Multi-Party Conversations2026
- XTurnix: Large-Scale Self-Supervised Turn Control through Two-State Binary Decisions2026
- Using Semantic Uncertainty to Estimate Transition Relevance in Turn-taking2026