Source-linked AI summary
AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, Caroline Pantofaru
TL;DR
Active speaker detection lacked a large, diverse, carefully labeled audiovisual benchmark, limiting evaluation across environments and accuracy. The paper introduces the publicly released AVA-ActiveSpeaker dataset and a real-time audiovisual model trained from pixels and audio, reporting state-of-the-art benchmark results and performance analyses. The dataset also supports analysis across noise conditions, recurrent models, and multimodal designs.
Problem
Active speaker detection lacked a large, diverse, carefully labeled audiovisual dataset for comparing algorithms across environments and accuracy.
Method
The paper constructs AVA-ActiveSpeaker with dense face-track annotations and presents a real-time audiovisual model trained directly from pixels and audio without pre-trained networks.
Results
The paper reports state-of-the-art benchmark results for various models and analyzes performance under conditions relevant to downstream modeling choices.
Takeaways & Limitations
AVA-ActiveSpeaker provides a public benchmark and an AVA label ecosystem that support deeper cross-task and multimodal analysis.
Takeaways & Limitations
The face detection system is not perfect, and illumination and partial occlusion can cause missed detections.
Abstract
from arXiv · showhide
Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made comparisons and improvements difficult. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) that will be released publicly to facilitate algorithm development and enable comparisons. The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames or about 38.5 hours of face tracks, and the corresponding audio. We also present a new audio-visual approach for active speaker detection, and analyze its performance, demonstrating both its strength and the contributions of the dataset.
1. Introduction
Active speaker detection supports many video-analysis applications, but robust modeling has been limited by task difficulty and the lack of a large, diverse labeled dataset. The paper introduces AVA-ActiveSpeaker, a public benchmark with dense annotations, and a real-time audiovisual model with analyzed benchmark performance.
- Active speaker detection identifies which visible person is speaking at a given time and supports applications including speech analysis, video re-targeting, and human-robot interaction.
- Visual-only methods can confuse speech with other mouth or face motions, while audio-only methods require assumptions that may not generalize.
- A large, diverse dataset was needed because prior experimental data consisted of seven 30-minute presentations and part of one panel discussion.
- AVA-ActiveSpeaker provides about 3.65 million labeled frames, about 38 hours of face tracks, corresponding audio, and labels for speaking status and audibility.
- The paper presents a large-scale public benchmark with dense spatio-temporal labels and a real-time audiovisual model trained directly from pixels and audio without pre-trained networks.
- The study reports state-of-the-art benchmark results and analyzes model performance and conditions relevant to downstream application-specific choices.
2. Related work
Prior active speaker work often relied on heuristics or small, task-specific datasets, while related audiovisual corpora were constrained in diversity or recording conditions. AVA-ActiveSpeaker addresses these gaps with diverse, dense labels added to the broader AVA label ecosystem and supports multimodal model comparisons.
- Applications using active speaker detection: Application systems often use heuristics such as lip motion or single-face assumptions, which can bias active speaker detection toward precision and omit difficult conditions.
- Datasets: Direct active speaker detection studies evaluated on datasets shorter than one hour, unlike AVA-ActiveSpeaker’s diverse video and temporally dense face labels.
- Datasets: The dataset enriches AVA’s existing action and speech activity labels, enabling cross-task analysis alongside active speaker evaluation.
- Datasets: Earlier audiovisual corpora were often designed for frontal, high-resolution speech recognition or were limited in subject diversity and recording conditions.
- Datasets: AVA-ActiveSpeaker contains approximately 40K labeled face tracks totaling 38 hours, varied languages, task-agnostic labels, and unconstrained recording conditions.
- Multimodal approaches: The paper compares visual-only and audiovisual models trained from scratch and examines tradeoffs among accuracy, latency, computational complexity, and background noise.
3. Dataset Construction
AVA-ActiveSpeaker was constructed through video selection, label-vocabulary definition, face-track generation, and human annotation. The resulting public dataset provides dense spatio-temporal face-track labels in CSV form, with categories distinguishing speaking and audibility.
- Dataset construction comprised video selection, label-vocabulary definition, face-track detection, and human annotation.
- The released dataset contains dense spatio-temporal annotations and a CSV file recording face bounding boxes over time with corresponding temporal labels.
- Video selection: The selected material came from AVA v1.0 movie segments and was chosen for diversity in languages, recording conditions, and speaker demographics, with synchronized audio and video.
- Label Vocabulary Definition: The annotation vocabulary distinguishes Not Speaking, Speaking and Audible, and Speaking but not Audible.
- Face track generation: Face tracks were generated automatically from detected faces, linked over time using bounding-box overlap and similarity, with short gaps filled by smoothing.
- Active Speaker Annotation: Human annotators labeled each visible face using a video interface showing the face track and synchronized audio waveform over continuous time.
4. Labeled Dataset
The labeled dataset combines dense active-speaker annotations with analyses of face-track characteristics, annotation agreement, overlapping speakers, and relationships to existing AVA action and speech labels. These analyses expose challenging visual conditions, label disagreements, and cases where speech is not attributable to a visible speaker.
- Dataset characteristics: A significant portion of labeled faces are smaller than 100 pixels wide, while crowded scenes and lively conversations increase prediction difficulty.Face-width distributions and concurrent-face durations characterize challenging visual and conversational conditions.
- Annotation quality: Approximately 40K face tracks from 160 videos were labeled by three annotators, achieving Fleiss’ kappa of 0.72.Most disagreements occurred near temporal speech-segment boundaries and reflected perceptual differences.
- Temporal labels: Active-speaker segments are often short because movie shots can move away from the speaker, breaking the visible face track during an utterance.The dataset therefore captures speech that is temporally continuous while the corresponding visible face track is not.
- Overlapping speakers: Multiple co-occurring active speakers are uncommon, but single frames contain up to 9 audible-speaking faces and up to 22 inaudible-speaking faces in group contexts.The reported overlapping instances span approximately 3 minutes and include contexts such as choruses.
- Limitations: The face-detection system misses some candidate tracks under challenging illumination and partial occlusion.These missed detections constrain coverage of the labeled dataset in such conditions.
- Relationships to existing labels: Cross-checking AVA action labels reveals disagreements caused by boundary timing, harder negatives such as laughing or yawning, and inaudible or musical contexts.The comparisons use the co-occurrence of “talk-to” and “sing-to” labels with active-speaker labels.
- Relationships to existing labels: The active-speaker labels extend prior AVA speech-activity labels by attributing speech to visible faces, but cannot cover offscreen speakers or faces turned away from the camera.The overlap analysis distinguishes speech without a visible speaker from speech attributed to one, alongside speaker-without-speech and no-speech cases.
- Relationships to existing labels: Clean speech increases by 12% when a speaker is visible compared with speech without a visible speaker.The paper relates this pattern to shots that focus on the speaker.
5. Multimodal active speaker detection
The paper formulates active speaker detection as a joint mapping from face tracks and audio to frame-level speech probabilities. Its end-to-end architecture jointly trains audio, visual, and prediction networks from raw pixels and audio representations, with auxiliary modality losses.
- Task formulation: The model learns frame-level speech probabilities from a face track and corresponding audio signal.The target is a sequence of probabilities aligned with the face frames.
- Architecture: The end-to-end function is decomposed into jointly trained audio, visual, and prediction networks that fuse low-dimensional modality embeddings.The audio and visual networks produce embeddings consumed by a prediction network.
- Architecture: The networks are trained from scratch using face pixels for vision and Mel-spectrogram representations for audio, without pre-trained networks.The visual input is processed directly from pixels rather than pre-trained embeddings.
- Training objective: Training uses cross-entropy supervision with auxiliary audio-only and visual-only classification losses weighted by λa = λv = 0.4.The auxiliary terms encourage the prediction network to use both modality embeddings.
- Inputs: The visual input uses M consecutive 128 × 128 grayscale face thumbnails, while audio uses 0.5 seconds of Mel-spectrogram context.Varying M tests how temporal visual information affects predictions.
- Model variants: The study compares static prediction with recurrent prediction using GRUs, and examines tradeoffs among accuracy, latency, and computational complexity.The static model processes frame sets independently, whereas the recurrent model uses prior state.
6. Evaluation and Analysis
Evaluation on AVA-ActiveSpeaker compares visual-only and audiovisual models across temporal context, sound conditions, face sizes, and held-out challenge data. Audiovisual models reduce errors over visual-only models, while performance varies with background music and face size.
- Overall results: The evaluation reports overall results for visual-only and audiovisual models, using V, AV, and fM to denote modality and stacked-face count.Table 4 is the main results table for the experiments.
- Temporal context: Performance improves from one to two visual frames, while static models continue improving until roughly f10 or 0.5 seconds of visual information.GRU models saturate at f2, indicating that recurrent history can supply longer-term context.
- Audiovisual benefit: > 40% reduction in error is achieved by static audiovisual models over visual-only models, while GRU models show > 30% reduction.Audiovisual modeling addresses visual-only confusion from mouth motions that are not speech.
- Control comparison: The VV control remains within hundredths of visual-only models, supporting audio as the source of the audiovisual improvement rather than added model capacity.The comparison uses two independent visual towers to control for parameter and embedding dimensionality.
- Sound conditions: Audiovisual performance drops with background music but remains above visual-only performance, while both model types show resilience to background sound overall.Visual-only performance is described as similar across environments.
- Face size: Both visual-only and audiovisual models perform better with larger faces, but false-positive-rate behavior differs across modalities.At a fixed threshold, the false positive rate increases for visual-only models and decreases for audiovisual models as face size changes.
7. Conclusion
The paper introduces AVA-ActiveSpeaker as a large-scale benchmark with dense spatio-temporal speaking annotations and presents a joint audiovisual model that reduces visual-only errors by 36%.
- Dataset contribution: AVA-ActiveSpeaker provides dense spatio-temporal spoken-activity annotations across 15-minute clips from 160 AVA v1.0 videos.The dataset is presented as the first publicly available large-scale benchmark for active speaker detection.
- Model contribution: The joint audiovisual approach reduces errors in visual-only approaches by 36%.The paper also analyzes model performance across several conditions.
Appendix
The appendix adds supplementary information about dataset labels, the labeling process, and model performance.
- Section A: Section A provides additional information about dense spatio-temporal labels and the labeling process.
- Section B: Section B provides supplementary information related to model performance on the dataset.
A. Dataset Information
AVA-ActiveSpeaker uses human ratings of face tracks against synchronized audio to distinguish speaking, non-speaking, and audible speech. The dataset also exposes annotation mismatches, overlapping speakers, and face-track coverage limitations in challenging video conditions.
- Annotation: Raters label each visible face track as not speaking, speaking and audible, or speaking but not audible using synchronized video and audio.The interface overlays color-coded labels on the audio timeline, and raters can navigate and modify labels.
- Data format: The released annotations use face-track records with normalized bounding-box coordinates and comma-separated fields.Figure 9 illustrates the eight-value CSV format and the normalized top-left and bottom-right box coordinates.
- Dataset analysis: AVA-ActiveSpeaker identifies overlapping speakers with per-person labels and color codes speaking-audible, speaking-inaudible, and non-speaking instances.These dense labels can support analysis of overlapping speech and audiovisual speech separation.
- Coverage: Some face tracks are missed in crowded scenes, with small or partially occluded faces, difficult lighting, and lower-resolution video.The missed detections are not labeled, so coverage is constrained by the automated detection and tracking pipeline.
B. Supplementary Results
The supplementary evaluation compares visual-only and audiovisual static or recurrent models across noise, face size, and error cases. Audiovisual models generally outperform visual-only models, while recurrent models gain from short temporal context but still fail on small faces and competing mouth motions.
- Model comparison: V denotes visual-only, AV audiovisual, GRU gated recurrent unit, and fM the number of frames in the visual-network input stack.These abbreviations distinguish modality, recurrence, and temporal input size in the model comparisons.
- Model comparison: AV-GRU is ∼10% better TPR than V-GRU and ∼5% better TPR than AV-static at 10% FPR.The same general pattern holds for recurrent models, while static-model gains continue through M = 10.
- Temporal context: Performance improvements for recurrent models saturate at 2 frames, indicating that only a short amount of history is needed.Figure 15 presents ROC curves for static and recurrent visual-only and audiovisual models.
- Background sound: AV models slightly dip with overlapping music and noise but still outperform V models, while V models remain unaffected by background sound.Figure 16 partitions ROC curves by background sound conditions.
- Face size: At 10% FPR, AV-GRU improves TPR by ∼10% for small faces, ∼15% for medium faces, and ∼13% for large faces over V-GRU.The largest difference occurs for medium faces; visual information is harder to use for small faces and can close the gap for large faces.
- Error analysis: AV models correct visual-model errors using audio context, including cases involving profiles, motion around faces, and partial occlusions.AV-GRU errors remain concentrated in small faces with multiple moving faces and scenes where several people vocalize but only one speaks.