Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

1 INTRODUCTION

• Manual video labeling is expensive and limits how much video data can be used for supervised training.The introduction contrasts this constraint with the vast numbers of videos uploaded to social websites.Source: §1 · p. 1
• Audio-visual co-occurrences provide free self-supervision for learning cross-modal representations from large-scale unlabeled video.The paper frames this as an alternative to relying on human labels.Source: §1 · p. 1
• Prior approaches overlook information exchange between modalities and struggle with heterogeneous scenes containing multiple sound sources.The paper notes that many earlier methods focused on speech and face scenes.Source: §1 · p. 2
• The proposed co-attention framework exchanges audio and visual information, using multi-head attention to associate sounds with discriminative visual regions.The pre-trained model is applied to sound source localization and action recognition.Source: §1 · p. 2

2.1 Audio-Visual Representation Learning

• Earlier audio-visual representation methods transferred supervision from one modality to another, but relied on established models trained with substantial data.Other work used audio-visual correspondence or temporal synchronization; the paper identifies missed information exchange as a difference from its framework.Source: §2.1 · p. 2

2.2 Audio-Visual Applications

• For sound-source localization, prior approaches included computational, hardware-based, and deep-learning methods, but mixed sounds challenge distinct-source recognition.The paper says its multi-head structure can localize each source in complicated scenes without supervision.Source: §2.2 · p. 2
• Action recognition models commonly use fully supervised training and substantial annotated data, while self-supervised alternatives exploit unlabeled videos.The paper positions its method as adding cross-modal interactions and offering competitive results with fewer model parameters.Source: §2.2 · p. 2
• Figure 2 depicts co-attention as cross-modal attention followed by self-attention, with audio-guided and visual-guided attention inside the cross-modal block.The modules may be cascaded in depth.Source: Figure 2 · p. 3

3 AUDIO-VISUAL SYNCHRONIZATION

• AVS is a binary task: predict whether audio and video are synchronized, using same-time clips as positives and different video slices as negatives.The synchronization labels come directly from the video clips, without human annotation.Source: §3 · p. 3

3.1 Network Architecture

• Audio and visual encoders feed co-attention modules whose attended features are fused to predict synchronization probability.The paper explores visual-guided attention, audio-guided attention, and cross-modal attention, followed by self-attention to model within-modality interactions.Source: §3 · p. 3, §3.1 · pp. 3–4
• The audio encoder produces 512D audio features, while the 3D-convolutional visual encoder averages frame features into 512D video features.The visual encoder is designed to capture motion information.Source: §3.1 · p. 3

3.2 Implementation Details

• Training uses about 240K unconstrained AudioSet clips without event annotations, with 4.2-second samples and 2.0–5.8-second audio shifts for negative examples.The authors use the same evaluation settings as prior work and compare against Multisensory and a retrained Multisensory* baseline.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 Results and Discussions

• 5.1% and 3.1%: the CMA model outperforms Multisensory* on Audioset-750K and Audioset-240K†, respectively, with 58.3% fewer parameters.Table 1 reports the AVS comparisons; † marks clips segmented by the authors, and * marks a retrained model.Source: Table 1 · p. 5, §3.3 · p. 5
• Without retraining, the CMA model reaches 60.8% accuracy on Kinetics AVS versus 57.5% for Multisensory*; speech categories rank highest, while nearly silent actions rank lowest.Figure 4 orders action categories from high to low accuracy.Source: §3.3 · p. 5, Figure 4 · p. 5
• Figure 5 contrasts single-source examples in its top three rows with multi-source examples in its bottom three; the model can localize multiple sound sources where baselines miss some.Examples include speech and Kinetics-Sounds actions such as chopping wood and singing.Source: Figure 5 · p. 6, §3.3 · p. 6
• AVS remains difficult when audio is silent, frames do not change, sources are mixed, or the sound-maker is off-screen; human accuracy was 66.6% ± 2.4% on longer shifted pairs.The human evaluation used 15-second pairs with audio shifted by five seconds.Source: §3.3 · p. 6

• Class Activation Maps, which use Global Average Pooling, visualize the regions the model associates with sound components.Source: §4 · p. 6
• On held-out speech examples, co-attention produces more precise and concentrated sound-source localization than the Multisensory baselines.All models respond to faces and mouth movements, while the authors report that co-attention benefits fine-grained audio and visual representations.Source: §4 · pp. 6–7
• On Kinetics-Sounds examples including chopping wood, playing organ, singing, dribbling basketball, and playing guitar, the models localize action regions.Source: §4 · p. 7
• Compared with baseline models, the co-attention model localizes multiple sound sources across video categories, including athletes and a coach making similar bouncing-ball sounds.The baselines localize the coach but miss the athletes; the authors attribute the result to multi-head attention capturing information about distinct sound sources.Source: §4 · p. 7

5 ACTION RECOGNITION

• For classification, the model concatenates the two co-attention output features, adds fully connected layers, and fine-tunes the pre-trained network with cross-entropy loss.The final layer has one output per dataset label; weights are initialized from the AVS-pre-trained model.Source: §5 · p. 7

5.1 Implementation Details

• At inference, 25 overlapping subclips are sampled at equal temporal spacing, and their outputs are averaged for a video-level prediction.Training samples 2.56-second clips from a randomly selected starting frame.Source: §5.1 · p. 7

5.2 Results and Discussions

• 2.0% on UCF101 and 1.3% on HMDB51: the co-attention model improves over AVTS pre-trained on Kinetics (240K examples).The comparison reports a remarkable boost against self-supervised methods pre-trained on the same dataset size; AVTS uses a visual subnetwork with more parameters.Source: Table 2 · p. 7, §5.2 · p. 7
• 17.3% on UCF101 and 13.4% on HMDB51: the full model improves when pre-trained on the pretext task rather than trained from scratch.Source: §5.2 · p. 7
• 4.5% on UCF101 and 6.3% on HMDB51: vision-only falls below the full model, with audio set to zero and co-attention changed to self-attention.The authors conclude that the audio subnetwork is important for action recognition and that the visual subnetwork can work independently.Source: §5.2 · p. 7
• Figure 6 presents localization ablations with one single-source example followed by examples containing mixed multiple sources.Source: Figure 6 · p. 8

• The study examines six model sizes using depth L=[1,2], hidden size H=512, and head counts A=[4,8,16].Source: §6 · p. 8
• Both AVS and action recognition benefit from shallower models, while properly increasing the number of heads can also improve performance.Source: §6 · p. 8
• In multi-source localization examples, greater co-attention depth concentrates localization, while more heads help identify more sound sources and correlated visual regions.The authors illustrate this with a singing scene where additional heads identify people in the bottom-left and middle-right.Source: §6 · p. 8

• The proposed self-supervised co-attention framework learns generic cross-modal representations through the AVS pretext task.Source: §7 · p. 8
• The co-attention model achieves state-of-the-art performance on AVS and learns semantic information used for sound-source localization and action recognition.Source: §7 · p. 8
• For efficiency, the networks are trained on an AudioSet subset; the authors leave using more pretraining data and applying the model to other audio-visual tasks for future work.Source: §7 · p. 8