Source-linked AI summary
Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration
Pu Wang, Hugo Van hamme
TL;DR
Crowded scenes and missing visual evidence make it difficult to match speech with visible faces or identify a speaker. ROAM-ASD-C reasons across visible candidates, while ROAM-ASD-CR adds enrollment-derived speaker representations. ROAM-ASD-C improves results across five benchmarks, including 96.8% mAP on AVA.
Problem
In crowded or unconstrained recordings, multiple candidates, interfering voices, and domain variation make speech-to-face correspondence ambiguous, while absent visual observations leave face-level ASD without an identity cue.
Source: §1 · p. 1Method
Results
ROAM-ASD-C improves ROAM-ASD across five benchmarks, achieving 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Active speaker detection (ASD) relies on audiovisual correspondence between speech and visible faces, yet this correspondence becomes unreliable in crowded scenes, or when visual evidence is unavailable. We introduce ROAM-ASD-C and ROAM-ASD-CR to extend ROAM-ASD with contextual and identity-aware reasoning. ROAM-ASD-C jointly reasons across simultaneously visible candidates, allowing other faces to provide complementary evidence when the target observation is ambiguous or incomplete. ROAM-ASD-CR incorporates registered-speaker representations derived from enrollment speech, allowing detection of whether a speaker is active even when that speaker is not visually observed. ROAM-ASD-C achieves state-of-the-art performance across five ASD benchmarks: 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW. It also improves ROAM-ASD by up to 13.3 mAP under 75% target-track visual removal.
1. INTRODUCTION
ROAM-ASD improves audiovisual fusion for challenging recordings, but visible-face ambiguity and missing visual evidence leave important gaps. ROAM-ASD-C adds reasoning across visible candidates, while ROAM-ASD-CR adds registered-speaker identity cues.
1. INTRODUCTION
- Crowded scenes, interfering voices, and domain variation make speech-to-face correspondence ambiguous, despite strong performance on conventional benchmarks.The introduction reports best baseline results of 93.7% mAP on WASD and 83.2% mAP on UniTalk.Source: §1 · p. 1
- ROAM-ASD had already improved cross-domain generalization and missing-observation robustness, reaching 98.8% mAP on WASD and 87.9% mAP on UniTalk.It combines localized visual cues with flexible joint self-attention over audio, face, and mouth streams, trained with modality dropout.Source: §1 · p. 1
- ROAM-ASD makes decisions independently for each visible face, leaving relationships among simultaneous candidates unused; without visual observations, face-level ASD lacks the identity cue needed to assign speech.The paper motivates contextual reasoning across candidates and speaker registration as complementary extensions.Source: §1 · p. 1
- ROAM-ASD-C jointly processes simultaneously visible candidates so other faces can provide complementary evidence when a target’s observations are ambiguous or incomplete.Its final face-level activity predictions remain independent binary decisions.Source: §1 · p. 1
- ROAM-ASD-CR adds enrollment-derived registered-speaker representations, enabling activity detection without visual observations and association with a face when one is visible.The overview shows ROAM-ASD-C adding inter-face context and ROAM-ASD-CR further adding registration tokens.Source: §1 · p. 1, Figure 1 · p. 2
- 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW: ROAM-ASD-C sets new state-of-the-art results across five benchmarks.It consistently improves ROAM-ASD, with larger gains in crowded and visually degraded conditions.Source: §1 · p. 2
2. MODEL
The model extends ROAM-ASD with joint reasoning across visible face candidates and speaker registration, so decisions can use inter-face context or enrolled-speaker identity. It retains ROAM-ASD’s audiovisual fusion and prediction head while adding contextual representations and registration-specific supervision.
ROAM-ASD-C
- The model retains ROAM-ASD’s audio, face, and mouth encoders, multimodal fusion, modality dropout, and per-face prediction head.Contextual reasoning is added to fused query representations before prediction.Source: §2 · p. 2
- Inter-face context jointly processes temporally aligned representations of overlapping face tracks, letting candidates exchange evidence before separate per-face predictions.Tracks are aligned to the longest track’s timestamps; scenes with more than six overlapping tracks are split into groups of at most six candidates.Source: §2 · pp. 2–3
ROAM-ASD-CR
- ROAM-ASD-CR appends enrollment-derived speaker representations to the visible candidate set, allowing registered speakers to participate in the same context block.The registration token is time invariant; with no registered speakers, the model reduces to ROAM-ASD-C.Source: §2 · p. 3
- A registered-speaker speak head predicts whether each enrollee is speaking, while an association head scores links between registered speakers and visible faces.The association head uses projected registration and face representations to compute a scaled dot-product score.Source: §2 · p. 3
- Registration training uses self-enrollment without additional speaker-identity annotation, with separate speaking and face-association objectives added to the original ASD objective.Speaking supervision labels matched active registrants positive and out-of-scene registrants negative; association supervision applies when a matched registrant is speaking and its face is visible.Source: §2 · p. 3
3. EXPERIMENTS
The experiments evaluate contextual reasoning and speaker registration across five ASD datasets using the ROAM-ASD setup. They test face-level performance, robustness to missing target-track visuals, and registered-speaker detection when visual inputs are removed.
Evaluation setup
- Evaluation uses AVA, ASW, Talkies, WASD, and UniTalk, following the ROAM-ASD setup.WASD includes five conditions, while UniTalk covers underrepresented languages, background noise, crowded scenes, and mixed conditions.Source: §3 · p. 3
Inter-face context experiments
- Contextual reasoning is tested across visible-face counts and scene conditions, and under target-track visual removal while other candidates remain mostly available.The protocol includes repeated 5%, 15%, or 30% holes, single 50% or 75% holes, and complete stream removal; mAP is measured on labeled frames within missing intervals.Source: §3 · p. 3
Speaker registration experiments
- Registration is tested with face and mouth inputs removed for all candidates, retaining scene audio and registration tokens.The experiments report registered-speaker AP, association accuracy, and false-alarm rate for out-of-scene registrants under over-registration.Source: §3 · p. 3
4. RESULTS
ROAM-ASD-C improves face-level ASD across five benchmarks, with larger gains in crowded and visually degraded conditions. ROAM-ASD-CR extends activity detection to registered speakers when visual observations are absent while preserving standard face-level performance.
Face-level ASD
- ROAM-ASD-C improves over ROAM-ASD across all five benchmarks, reaching 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW.Table 1 compares overall mAP with the best-performing prior methods and ROAM-ASD.Source: Table 1 · p. 4, §4 · p. 4
Missing target-track observations
- At 75% target-track visual removal, ROAM-ASD-C improves over ROAM-ASD by 3.1 mAP on AVA, 8.8 on WASD, 13.3 on UniTalk, and 10.7 on ASW.The evaluated missing segments remove face and mouth observations.Source: Figure 3 · p. 4, §4 · p. 4
Visible-face count
- On AVA, ROAM-ASD-C performs essentially the same as ROAM-ASD with one visible face, but its gain rises to 0.8 mAP with two faces and 2.2 mAP with three.On UniTalk, contextual reasoning improves all four conditions, with the largest gains under Noise and Crowded, both by 1.5 mAP.Source: §4 · p. 4
Speaker registration
- Under complete visual absence, registered-speaker AP exceeds face-ASD on ASW, AVA, WASD, and UniTalk, while ROAM-ASD-CR preserves standard face-level ASD performance.The registered-speaker branch retains 80.4%, 78.0%, 81.8%, and 60.3% AP on those datasets, respectively.Source: Table 3 · p. 4, §4 · p. 4
5. CONCLUSION
The paper extends ROAM-ASD with inter-face contextual reasoning and speaker registration. The extensions improve face-level ASD in crowded or visually degraded conditions and support registered-speaker detection without visual evidence.
Conclusion
- ROAM-ASD-C and ROAM-ASD-CR extend ROAM-ASD with inter-face contextual reasoning and speaker registration.Source: §5 · p. 4
- ROAM-ASD-C consistently improves face-level ASD, with larger gains in crowded and visually degraded conditions.Source: §5 · p. 4
- ROAM-ASD-CR enables registered-speaker activity detection when visual evidence is unavailable while preserving face-level ASD performance.Source: §5 · p. 4