Source-linked AI summary
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, Haizhou Li
TL;DR
Active speaker detection needs reliable fine-grained interpretation of audio, visual, temporal, and synchronization information, while prior systems mainly use short-term features. TalkNet combines audio and visual temporal encoders with cross-attention and self-attention, achieving improvements over state-of-the-art systems on two ASD benchmarks.
Problem
ASD systems have not fully benefited from audio-visual temporal dynamics and inter-modality interaction, limiting their scope in challenging scenarios.
Method
TalkNet uses audio and visual temporal encoders, audio-visual cross-attention, self-attention, and audio augmentation for ASD.
Results
TalkNet outperforms state-of-the-art systems by 3.5% on AVA-ActiveSpeaker and 2.2% on Columbia ASD.
Takeaways & Limitations
Sentence-level audiovisual information provides the basis for exploring audio-visual relationships and synchronization in ASD.
Takeaways & Limitations
TalkNet is trained from scratch using only the AVA-ActiveSpeaker training set, without pre-training or additional post-processing.
Abstract
from arXiv · showhide
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
1 INTRODUCTION
Active speaker detection requires fine-grained decisions that integrate audio, visual, temporal, and synchronization cues. TalkNet addresses limitations of short-term ASD systems by modeling long-term context and audio-visual interaction.
- ASD detects who is speaking in a visual scene and supports audio-visual speech recognition, speech separation, diarization, and speaker tracking.
- Human judgments combine active voice, lip movement, and synchronization between voice and lips.
- Short-term ASD systems often use 200–600 ms segments, which may not reveal speaking activity or audio-visual synchronization.
- Longer utterance-level context is more reliable because short segments may not cover a complete word and single-modality cues can be misleading.
- TalkNet encodes temporal context from audio and visual frame sequences to improve ASD decisions without simply increasing fixed segment size.
- TalkNet introduces long-term audio-visual context, cross-attention, self-attention, and audio augmentation for noise robustness.
2 RELATED WORK
Prior ASD research combines audio and visual signals but largely relies on short-term features and has not fully explored long-term temporal context. Related audio-visual synchronization work supports using longer utterances to learn more representative cross-modal features.
- Single-modality ASD methods face ambiguity from background noise, overlapping speech, weak motion correlation, and non-speaking facial activities.
- Audio-visual ASD can avoid assuming that every detected speech signal belongs to an on-screen speaker, since cross talk and off-screen speech occur in practice.
- Most prior ASD systems make frame-level decisions from segments shorter than 600 ms, while long-term features remain underexplored.
- Utterance-level temporal modeling has shown benefits over frame-level modeling in audio tasks, and proposal-level temporal context benefits video object detection.
- Longer video utterances can extract more representative audio-visual synchronization features and improve performance.
- TalkNet is motivated by evidence that long-term context can benefit both intra-modal representation learning and inter-modal audio-visual relationships.
3 TALKNET
TalkNet combines temporal audio-visual representation with cross-attention and self-attention to classify speaking activity frame by frame. It also uses in-domain negative sampling to improve noise robustness.
- TalkNet: TalkNet encodes cropped face video and corresponding audio into temporal embeddings, then predicts whether the person speaks in each video frame.Its frontend contains separate audio and visual temporal encoders, while the backend performs attention-based classification.
- Visual Temporal Encoder: The visual encoder combines a spatial frontend with a temporal network to represent long-term facial-expression dynamics.The frontend uses 3D convolution and ResNet18, followed by a video temporal convolutional block.
- Audio Temporal Encoder: The audio encoder produces time-matched embeddings from MFCC sequences, using a 1,890 ms receptive field to capture long-term context.Dilated convolutions align audio and visual embedding time resolutions for subsequent attention.
- Audio-visual Cross-Attention: Cross-attention dynamically models audio-visual interaction by using each modality to generate attention features conditioned on the other.The outputs are concatenated along the temporal direction before self-attention processing.
- Self-Attention and Loss: Self-attention models utterance-level temporal information from the joint audio-visual feature to distinguish speaking and non-speaking frames.The final fully connected layer and softmax project the output to a frame-level ASD label sequence trained with cross-entropy loss.
- Audio Augmentation: Negative sampling augments training audio by replacing a video’s audio track with randomly selected audio from another video in the same batch.The augmented sample retains the original active- or inactive-speaker label and provides in-domain noise and interference speakers.
4 EXPERIMENTS
The experiments evaluate TalkNet on established ASD benchmarks and a newly constructed in-the-wild training resource. The datasets contain substantial synchronization, visibility, and annotation challenges relevant to active speaker detection.
- AVA-ActiveSpeaker: AVA-ActiveSpeaker contains 29,723 training, 8,015 validation, and 21,361 test video utterances evaluated with mean average precision.Its utterances range from 1 to 10 seconds and are provided as face tracks.
- AVA-ActiveSpeaker: AVA-ActiveSpeaker includes diverse languages, variable frame rates, blurry images, noisy audio, and dubbed dialogue that complicate audio-visual synchronization.These conditions make accurate alignment between audio and visual signals difficult.
- Columbia ASD: The Columbia ASD benchmark contains an 87-minute panel discussion with five speakers, while only two or three speakers are visible at a time.The common evaluation protocol uses F1 score, and the dataset lacks a standard train-test split.
- TalkSet: TalkSet defines real-world ASD conditions through active voice, visible lip movement, and audio-visual synchronization cues.The synchronization condition is considered only when both audio is active and lips are moving.
- Implementation: The experiments use consistent S3FD face tracking and apply visual augmentation, external audio augmentation for Columbia, and negative sampling for AVA.TalkNet uses MFCC audio features, 112 × 112 face inputs, 128-dimensional embeddings, and one eight-head transformer layer for each attention network.
5 RESULTS
TalkNet outperforms prior systems on AVA-ActiveSpeaker and Columbia ASD, while ablations show gains from long-term context, attention mechanisms, and negative sampling. Performance also remains stronger across challenging face-size and multi-person conditions.
- 5.1 Comparison with the State-of-the-art: 92.3% mAP lets TalkNet outperform MAAS-TAN by 3.5% on the AVA-ActiveSpeaker validation set.TalkNet also improves the best reported validation AUC by 3.6%.
- 5.1 Comparison with the State-of-the-art: 90.8% mAP lets TalkNet outperform the best prior work by 3.0% on the AVA-ActiveSpeaker test set.The test-set evaluation required assistance from the dataset organizer because ground-truth labels were not public.
- 5.1 Comparison with the State-of-the-art: 96.2% average F1 is TalkNet’s best result on Columbia ASD, improving over the best existing system by 2.2%.TalkNet performs best for three of the five speakers: Bell, Lieb, and Sick.
- 5.2.1 Long-term sequence-level temporal context: mAP improves from 75.2% to 89.4% as the fixed video duration increases from 0.2 to 4 seconds.No improvement appears from 50 to 100 frames because longer videos provide fewer training segments.
- 5.2.2 Short-term vs long-term features: TalkNet gains 4.8% mAP with longer input videos, whereas the prior short-term system loses 2.1%.The comparison suggests that longer inputs require long audio and visual receptive fields and an adequate attention mechanism.
- 5.2.3 Ablation study of TalkNet attention mechanism: Removing cross-attention or self-attention reduces validation mAP by 0.7% or 1.4%, while removing both reduces it by 2.3% to 90.0%.The ablation attributes these effects to learning inter-modality cues and long-term audio-visual temporal context.
- 5.2.4 Audio augmentation: Negative sampling outperforms traditional noise augmentation, while TalkNet without audio augmentation still outperforms the state of the art.Negative sampling uses in-domain noise and does not require external data.
- 5.2.5 Real-world conditions: TalkNet achieves the best results across face sizes and is more robust than competing methods as the number of visible faces increases.ASD performance decreases with smaller faces and becomes more challenging with one, two, or three visible faces.
6 CONCLUSION
TalkNet uses sentence-level audiovisual information to explore audio-visual relationships and synchronization for active speaker detection. It outperforms state-of-the-art systems on two mainstream ASD benchmarks.
- TalkNet uses sentence-level audiovisual video information to explore audio-visual relationships and synchronization.
- Figure 7 compares TalkNet with competing methods across various face sizes and specific face sizes within the same video frame.