Source-linked AI summary
End-to-end speaker segmentation for overlap-aware resegmentation
Hervé Bredin, Antoine Laurent
TL;DR
Speaker segmentation is usually assembled from separate voice activity, speaker change, and overlap detection tasks. This paper trains an end-to-end multi-label model with permutation-invariant training and applies it to short audio chunks, then uses its outputs for detection and overlap-aware resegmentation. The approach improves existing diarization pipelines, with relative DER gains over VBx of 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.
Problem
Speaker segmentation traditionally combines voice activity detection, speaker change detection, and overlapped speech detection rather than modeling them directly.
Method
The paper trains a multi-label end-to-end segmentation model with permutation-invariant training and uses its outputs for overlap-aware resegmentation of existing diarization pipelines.
Results
The approach consistently improves existing baselines on all datasets; relative DER improvement over VBx reaches 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.
Takeaways & Limitations
The model supports voice activity detection, overlapped speech detection, and overlap-aware resegmentation, including a best pipeline combining the model with VBx clustering.
Takeaways & Limitations
Overlapped speech detection remains unsolved and sometimes ill-defined, with missed detection and false alarms the main error sources across all three datasets.
Abstract
from arXiv · showhide
Speaker segmentation consists in partitioning a conversation between one or more speakers into speaker turns. Usually addressed as the late combination of three sub-tasks (voice activity detection, speaker change detection, and overlapped speech detection), we propose to train an end-to-end segmentation model that does it directly. Inspired by the original end-to-end neural speaker diarization approach (EEND), the task is modeled as a multi-label classification problem using permutation-invariant training. The main difference is that our model operates on short audio chunks (5 seconds) but at a much higher temporal resolution (every 16ms). Experiments on multiple speaker diarization datasets conclude that our model can be used with great success on both voice activity detection and overlapped speech detection. Our proposed model can also be used as a post-processing step, to detect and correctly assign overlapped speech regions. Relative diarization error rate improvement over the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.
1. Introduction
The paper replaces separate voice activity, speaker change, and overlap detection tasks with an end-to-end speaker segmentation model. It also demonstrates strong performance across datasets, overlap-aware resegmentation gains, and reproducible access to the model and outputs.
- The proposed model jointly encompasses voice activity detection, speaker change detection, and overlapped speech detection instead of treating them as separate tasks.
- The model performs at least as well as several voice activity detection baselines and sets a new state of the art for overlapped speech detection on AMI, DIHARD 3, and VoxConverse.
- The model consistently beats a heuristic overlap-aware resegmentation baseline and sets a new AMI state of the art when combined with VBx.
- The pretrained model and expected VAD, OSD, and resegmentation outputs are shared through the pyannote open-source library for reproducibility.
2. End-to-end speaker segmentation
The model predicts speaker activity at fine temporal resolution on short chunks, learns speaker-label assignments with permutation-invariant training, and derives task-specific outputs through post-processing. Sliding-window alignment enables overlap-aware resegmentation of existing diarization outputs.
- Model formulation: The model performs multi-label classification with permutation-invariant training on 5-second chunks at approximately 16ms temporal resolution.Short chunks reduce and stabilize the number of speakers relative to whole-conversation EEND.
- Model formulation: Each reference segmentation is encoded as Kmax-dimensional binary frames, where each dimension indicates whether a speaker is active at each frame.
- Permutation-invariant training: Because any permutation of speaker dimensions is valid, training minimizes binary cross-entropy over possible speaker assignments.The implementation computes pairwise losses and uses the Hungarian algorithm to select the minimum-loss permutation.
- Training: The model is trained with randomly cropped 5-second chunks, background-noise augmentation, and synthetic overlap created by mixing two chunks and merging their annotations.
- Post-processing: Speaker activations are converted into task outputs by thresholding and temporal post-processing; VAD uses the maximum activation, while OSD uses the second-highest activation.Segmentation and speaker-change detection can use a single 0.5 threshold or more advanced hysteresis-based processing.
- Overlap-aware resegmentation: For overlap-aware resegmentation, 5-second windows slide across an existing diarization output, align activations by permutation, aggregate them over time, and apply threshold-based post-processing.
3. Experiments
Experiments use one segmentation model trained on a composite multi-dataset corpus, with dataset- and task-specific threshold tuning. Evaluation covers voice activity detection, overlapped speech detection, and resegmentation using reported error metrics.
- DIHARD3 and VoxConverse development data were split into training and development subsets because neither dataset provides a training set.
- One segmentation model was trained on a 62-hour composite corpus and selected using a 24-hour composite development set.
- Detection thresholds were tuned separately for each dataset and task, with objectives based on detection error rate, F1-score, or diarization error rate.
- The model ingests 5-second, 16-kHz audio chunks and extracts frames every 16ms before recurrent processing.
- The protocols report voice activity detection, overlapped speech detection, and resegmentation metrics including false alarms, missed detections, speaker confusion, and DER.
4. Results and discussions
The segmentation model performs successfully on voice activity detection and overlapped speech detection, while overlap-aware resegmentation produces consistent gains over existing diarization baselines. Relative DER improvement over VBx reaches 17% on AMI and 13% on both DIHARD and VoxConverse.
- Voice activity detection: The segmentation model outperforms models trained specifically for voice activity detection across the reported comparisons, with an off-the-shelf-model caveat for silero vad.
- Overlapped speech detection: The model successfully performs overlapped speech detection despite not being trained specifically for that task, outperforming pyannote 1.1.
- Overlap-aware resegmentation: Resegmentation consistently improves all evaluated baselines on all datasets.
- Overlap-aware resegmentation: 17% relative DER improvement over VBx is reached on AMI, versus 13% on DIHARD and 13% on VoxConverse.
5. Conclusions
The best reported pipeline combines the proposed VAD, off-the-shelf VBx clustering, and overlap-aware resegmentation. Despite these gains, missed detection and false alarms remain the main error sources, leaving overlapped speech detection unresolved.
- DER reaches 19.9% on AMI, 19.3% on DIHARD 3, and 7.1% on VoxConverse with the best reported pipeline.With a 250ms forgiveness collar, VoxConverse DER reaches 3.4%.
- Missed detection and false alarms remain twice as high as speaker confusion across all three datasets, even with a forgiveness collar.
- The authors conclude that overlapped speech detection remains an unsolved and sometimes ill-defined problem.