Source-linked AI summary
Spot the conversation: speaker diarisation in the wild
Joon Son Chung, Jaesung Huh, Arsha Nagrani, Triantafyllos Afouras, Andrew Zisserman
TL;DR
Speaker diarisation for unconstrained online videos is difficult because of varied domains, many speakers, cross-talk, and degraded backgrounds, while available datasets are limited. The paper introduces an audio-visual diarisation method and semi-automatic annotation pipeline, then uses them to create VoxConverse. VoxConverse provides challenging multi-speaker video data with overlap and diverse backgrounds, and the method achieves lower DER than audio-only baselines.
Problem
Unconstrained online videos lack large-scale, freely available diarisation datasets despite their varied domains, speakers, cross-talk, and degraded backgrounds.
Method
The paper combines active speaker detection, face recognition, speech enhancement, and audio-only speaker recognition in a scalable semi-automatic annotation pipeline with human verification.
Results
The audio-visual method obtains much lower DER than audio-only state-of-the-art baselines on VoxConverse, while the dataset includes 3–4% overlapping speech and videos with up to 21 speakers.
Takeaways & Limitations
VoxConverse supplies a challenging and diverse audio-visual benchmark for diarisation in the wild, and integrating visual information yields large performance gains on this dataset.
Takeaways & Limitations
Annotation still encounters wrong-speaker assignments for non-visible speech, VAD false alarms, and missed overlapped speech.
Abstract
from arXiv · showhide
The goal of this paper is speaker diarisation of videos collected 'in the wild'. We make three key contributions. First, we propose an automatic audio-visual diarisation method for YouTube videos. Our method consists of active speaker detection using audio-visual methods and speaker verification using self-enrolled speaker models. Second, we integrate our method into a semi-automatic dataset creation pipeline which significantly reduces the number of hours required to annotate videos with diarisation labels. Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community. Our dataset consists of overlapping speech, a large and diverse speaker pool, and challenging background conditions.
1. Introduction
Speaker diarisation remains difficult for unconstrained online videos, motivating an audio-visual method and scalable annotation pipeline that produce VoxConverse.
- Motivation: Online-video diarisation is harder than diarisation in constrained domains because videos vary widely and contain off-screen speakers, cross-talk, and degraded backgrounds.Challenges include talk shows, news broadcasts, celebrity interviews, home vlogs, rapid exchanges, laughter, and applause.
- Motivation: Manual annotation is daunting, contributing to a shortage of large-scale, freely available datasets for unconstrained speech.Existing NIST-RTE evaluations use constrained audio-only datasets that are not freely available to the research community.
- Method: The paper proposes a scalable audio-visual diarisation method for web videos using active speaker detection, face and speaker verification, and speech enhancement.The method targets both on-screen and off-screen speaker identification under cross-talk and background noise.
- Contributions: The authors integrate automatic annotation with manual verification to reduce the hours required to annotate videos and curate VoxConverse.The resulting dataset is designed to be challenging and diverse, covering unconstrained audio-visual conversations.
2. Related works
Related work developed primarily around domain-specific diarisation datasets and challenges, while DIHARD broadened domains but remained limited in web-video scale and accessibility.
- Research landscape: Diarisation research has progressed unevenly because diverse, large-scale datasets and end-to-end models have been limited.Deep learning has therefore been applied less extensively to diarisation than to several other speech-processing tasks.
- Research landscape: NIST Rich Transcription challenges established diarisation error rate as the primary metric while focusing largely on meetings and domain-specific datasets.Broadcast news, telephone speech, and meetings developed largely separate datasets and research lines.
3. Dataset description
VoxConverse is a diverse, audio-visual dataset of challenging multi-speaker videos, featuring rapid speech, overlap, varied environments, and substantial speaker variation.
- Dataset scale: The development set contains 216 multi-speaker videos spanning 1,218 minutes and 8,268 annotated speaker turns, while the test set spans about 2,612 minutes across approximately 232 videos.These sets provide substantial annotated material for evaluating diarisation in the wild.
- Dataset conditions: Videos cover political debates, panel discussions, celebrity interviews, comedy news segments, and talk shows with dynamic noise, laughter, and applause.Face detections and tracks are included as part of the audio-visual annotation.
- Dataset conditions: Speech segments are quick and short, creating rapid-turn-taking conditions for diarisation.This characteristic accompanies the dataset's varied acoustic environments and overlapping speech.
- Dataset statistics: 91% of video time contains speech, and 3–4% of speech contains overlap, with one video reaching 29.8% overlap.Videos range from 22 seconds to 20 minutes and typically contain 4–6 speakers, with one containing 21.
4. Dataset collection
VoxConverse is built through a semi-automatic pipeline that combines automatic audio-visual processing with human verification. The pipeline detects, clusters, and labels visible and off-screen speakers while retaining full-video coverage for challenging conversations.
- Pipeline overview: The collection pipeline first generates diarisation annotations automatically, then has human annotators check and refine them.This two-stage process combines scalable automatic processing with manual correction.
- Video collection: Videos are collected from YouTube using keywords for multi-speaker discussions, with duplicate or highly similar content removed.The initial search targets videos containing alternating or simultaneous speech.
- Visual processing: Face tracks are created from shot-level face detections and clustered using face-recognition embeddings, while overlapping tracks are prevented from sharing an identity cluster.The face detector handles varied scales and poses, and embeddings are averaged across each track.
- Active speaker detection: Active speaker detection combines SyncNet audio–mouth-motion correlation with AVSE speech isolation followed by WebRTC voice activity detection.A face track is accepted as speaking only when both detectors agree, reducing false alarms from laughter and music.
- Off-screen speech: Off-screen speech is assigned by comparing speaker embeddings from unseen segments with speech segments linked to visible active speakers.The pipeline uses cosine distance and a threshold to verify likely speaker identities.
- Manual verification: Human verification is necessary because common errors include assigning non-visible speech to the wrong speaker, VAD false alarms, and missed overlapping speech.The authors use corrected outputs to identify failure modes and develop annotation guidelines.
5. Experiments
The experiments evaluate an audio-only DIHARD baseline against the proposed audio-visual method and ablations on VoxConverse. The audio-visual system achieves lower diarisation error, while combining two active speaker detectors reduces false alarms with only a small increase in missed speech.
- Evaluation setup: The evaluation compares the proposed audio-visual method with an audio-only DIHARD 2019 baseline and two ablations on the VoxConverse development set.The baseline uses VAD, x-vectors, PLDA scoring, and agglomerative hierarchical clustering; the proposed method is evaluated with and without speech enhancement.
- Evaluation protocol: DER combines missed speech, false alarm speech, and speaker confusion, with lower values indicating better performance.A 0.25-second forgiveness collar compensates for small annotation inconsistencies.
- Analysis: Identity-clustering errors directly cause speaker confusion, making the face-clustering threshold the most consequential tuned threshold in the audio-visual system.The system also tunes thresholds for active speaker detection and speaker identification.
- Results: The audio-visual method obtains substantially lower DER than the audio-only state-of-the-art baselines, demonstrating the value of visual information for diarisation on VoxConverse.The reported comparison is made on the development set, where lower metric values are better.
- Results: The combined active speaker detection method significantly decreases false alarm rate while causing only a small increase in missed speech.The ablations compare SyncNet-based ASD alone and AVSE-based ASD alone against the combined design.
- Difficulty: The DIHARD 2019 baseline obtains a DER of about 20% on VoxConverse, leaving substantial room for improvement.The authors note that the 26% DER on the DIHARD development set may differ partly because VoxConverse uses a 0.25-second forgiveness collar.
6. Conclusion
The paper presents a scalable audio-visual diarisation algorithm and uses it to create VoxConverse from in-the-wild videos. The dataset contains 70 hours of annotated video and is intended for free public release after its planned challenge use.
- The authors develop a high-performance audio-visual algorithm for automated diarisation and use it to generate VoxConverse from in-the-wild videos.
- The dataset-generation pipeline is described as fully scalable and effective across a range of domains.
- VoxConverse contains 70 hours of annotated video and is being scaled up for subsequent release to the research community free of charge.The data is planned for use in the second VoxCeleb Speaker Recognition Challenge before public release.