Source-linked AI summary
A Light Weight Model for Active Speaker Detection
Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, Liangyin Chen
TL;DR
Active speaker detection identifies who is speaking in video, but approaches that use multiple candidates or complex models can require substantial memory and computation. This paper proposes a lightweight architecture using a single candidate, split convolutions, and GRU cross-modal modeling. On AVA-ActiveSpeaker, it reports 94.1% mAP versus 94.2% for SPELL [22], with substantially fewer parameters and FLOPs.
Problem
Method
The framework uses one candidate face sequence, split convolutions for audio-visual feature extraction, and a GRU for cross-modal modeling.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors report single-frame inference times of 0.1–4.5 ms and describe the method as feasible for real-time applications.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Active speaker detection is a challenging task in audio-visual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23x) and FLOPs (0.6G vs. 2.6G, about 4x). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https://github.com/Junhua-Liao/Light-ASD.
1. Introduction
Active speaker detection identifies speakers in video and supports applications such as diarization, tracking, and video editing. Because existing approaches can demand substantial memory and computation, this section introduces a lightweight design and reports near-state-of-the-art benchmark performance at lower resource cost.
1. Introduction
- 94.1% mAP with 1.0M parameters and 0.6G FLOPs compares with SPELL [22] at 94.2% mAP, 22.5M parameters, and 2.6G FLOPs.The figure compares mAP with FLOPs for methods predicting one frame with three candidates; blob size represents parameter count.Source: Figure 1 · p. 1, §1 · p. 2
- Prior approaches use multiple candidate face sequences, 3D visual convolutions, and complex cross-modal attention, increasing memory and computational requirements.The authors identify real-time use with limited resources, including automatic video editing and live television, as a deployment challenge.Source: §1 · p. 1
- The framework uses one candidate face sequence with corresponding audio, split convolutions for feature extraction, and a GRU for cross-modal modeling.A novel loss function is also designed for training.Source: §1 · p. 2
- 95.6% fewer parameters and 76.9% fewer FLOPs than the state-of-the-art method are reported on AVA-ActiveSpeaker [32].The method is also reported to achieve state-of-the-art performance across scenarios with different numbers of people and face sizes.Source: §1 · p. 2
2. Related Work
Related work places active speaker detection within audio-visual learning and traces its development alongside the arrival of large-scale benchmark data. It reviews contextual and attention-based approaches before motivating lightweight models for resource-constrained deployment.
2. Related Work
- Active speaker detection is presented as an instance of audio-visual separation, within a broader audio-visual learning paradigm used for several video tasks [12] [17] [14] [31] [4] [35] [24].The passage frames audio-visual learning as fusing information sources to form joint representations.Source: §2 · p. 2
- The field's development was limited by a lack of large-scale benchmark data until AVA-ActiveSpeaker [32] was introduced, followed by new solutions.Earlier work explored audio-visual correlation and lip motion.Source: §2 · p. 2
- Prior methods model temporal and relational context [22], add spatial context [46], or aggregate audio and visual features with attention [36] [43] [44].Source: §2 · p. 2
- The review argues that resource-heavy inputs and models complicate deployment on constrained devices and in real-time television production.It cites automatic video editing on phones and tablets and live television shot selection as motivating applications.Source: §2 · p. 2
3. Method
The framework pairs visual and audio feature encoders with a detector that combines their features, models temporal context, and predicts whether the candidate is speaking. Its lightweight design uses split convolutions, a bidirectional GRU, and a main-plus-visual-auxiliary loss.
3.1. Visual Feature Encoder
- Three visual blocks split 3D convolution into spatial 2D and temporal 1D convolutions, using 3- and 5-kernel paths before 1-kernel integration.The first block’s 2D convolution reduces spatial dimensions; global max pooling produces the visual feature.Source: §3.1 · p. 3
3.2. Audio Feature Encoder
- The audio encoder splits 2D convolution into 1D convolutions over MFCC and temporal dimensions, then pools to align audio timing with visual features.It uses three blocks with different-receptive-field paths and global average pooling over the MFCC dimension.Source: §3.2 · pp. 3–4
3. Method
- Visual and audio features are added, passed through a bidirectional GRU for temporal modeling, and classified by a fully connected layer.The frontend encodes the candidate face sequence and corresponding audio; the backend predicts whether the candidate is speaking.Source: §3 · p. 3, §3.3 · p. 4
3.3. Detector
- A bidirectional GRU models temporal context in the summed visual-audio features before a fully connected layer predicts whether the candidate speaks.The detector operates on the multimodal features formed by summing visual and audio features.Source: §3.3 · p. 4
3.4. Loss Function
- The loss uses a main classifier and visual auxiliary classifier, omitting an audio auxiliary classifier because audio alone cannot identify which candidate is speaking.The loss is the main-classifier loss plus a visual auxiliary loss weighted by 0.5.Source: §3.4 · p. 4
- Candidate speaking scores are temperature-scaled before softmax, with temperature decreasing during training; framewise loss compares predicted probabilities with ground truth.The initial temperature is 1.3, with decay degree 0.02 per training epoch.Source: §3.4 · p. 4
4. Experiment
Experiments compare the lightweight framework on AVA-ActiveSpeaker and Columbia, then examine its components and performance across face-count and face-size scenarios. The results show competitive benchmark performance with substantially lower model costs than the state-of-the-art comparison.
4.1. Dataset
- AVA-ActiveSpeaker contains 262 Hollywood movies and 5.3 million labeled faces; evaluation uses its validation set because the test set is unavailable.The Columbia benchmark is an 87-minute panel discussion with five speakers and two to three visible at a time.Source: §4.1 · p. 5
4.2. Implementation Details
- Experiments use 112 × 112 faces, PyTorch, an NVIDIA RTX 3090, and Adam for 30 epochs with a 0.001 learning rate decayed by 0.05 each epoch.AVA-ActiveSpeaker is evaluated by mAP, Columbia by F1, and model size and complexity by parameters and FLOPs.Source: §4.2 · p. 5
4.3. Comparison with the state-of-the-art
- 94.1% mAP, versus SPELL’s 94.2%, uses 23 times fewer parameters and 4 times less computation; on Columbia, the method reaches 81.1% average F1 without fine-tuning.The AVA-ActiveSpeaker comparison is on the validation set; the Columbia comparison is against TalkNet and LoCoNet.Source: §4.3 · pp. 5–6
- Without fine-tuning, the method achieves an 81.1% average F1 score on Columbia, outperforming TalkNet and LoCoNet.This result is reported as state of the art on that dataset.Source: §4.3 · p. 6, Table 2 · p. 6
4.4. Ablation Studies
- 94.1% mAP is reached by combining kernel sizes across receptive fields; increasing kernel size from 5 to 7 raises model cost without improving performance.A kernel size of 3 gives 93.0% mAP with 0.5M parameters and 0.21G FLOPs.Source: §4.4 · p. 6
- The visual encoder reaches 92.8% with 13.68M parameters and 1.53G FLOPs, while 3D convolution doubles model costs without improving performance.For audio, ResNet-18 raises the model to 11.98M parameters without improving performance; splitting MFCC and temporal dimensions with 1D convolutions performs better than 2D convolution.Source: §4.4 · pp. 6–7
- Bidirectional GRU reaches 94.1% mAP, while forward GRU improves mAP by 4.6% over direct FC prediction; the proposed loss raises BCE’s 93.1% to 94.1%.The authors attribute the loss improvement in part to visual auxiliary supervision and temperature coefficients.Source: §4.4 · p. 7
4.5. Qualitative Analysis
- The method outperforms multi-input methods across detected-face counts, and its advantage grows as faces get smaller (+0.2%, +1.4%, and +3.2% mAP).It is the only method reported to keep mAP above 90% when candidate count is below three or face width exceeds 64 pixels.Source: §4.5 · p. 8, Figure 6 · p. 8
5. Conclusion
The paper presents a lightweight active speaker detection framework using a single candidate, split convolutions for feature extraction, and simple cross-modal modeling. On AVA-ActiveSpeaker, it reports substantially lower resource costs with mAP lagging state-of-the-art methods by only 0.1%, and reports good robustness.
5. Conclusion
- 95.6% fewer model parameters and 76.9% fewer FLOPs, with mAP lagging state-of-the-art methods by only 0.1%.Reported for AVA-ActiveSpeaker.Source: §5 · p. 8
- The framework uses a single candidate, split convolutions for visual and audio feature extraction, and simple modules for cross-modal modeling.Source: §5 · p. 8
- The authors report that their method shows good robustness.Source: §5 · p. 8