Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

• Temporal co-occurrence across senses can signal a shared underlying event, offering a learning signal that traditional vision approaches may miss by ignoring non-visual modalities.The introduction illustrates this with the coincident taste, sound, sight, and feel of biting an apple.Source: §1 · p. 1
• A network learns fused audio-visual features without manual labels by detecting whether raw audio and video streams are aligned or synthetically shifted by a few seconds.The task encourages the network to combine visual motion with audio information.Source: §1 · p. 1
• The learned representation supports sound-source localization, visual and audio-visual action recognition, and separation of on-screen from off-screen sounds.Examples include visualizing an axe impact, a speaker’s mouth, or a musician’s hands, and isolating a visible speaker’s voice through visual masking.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The paper evaluates the representation qualitatively through sound-source visualization and quantitatively on action recognition, and proposes source separation that works on real-world footage such as television broadcasts.The authors describe the real-world video separation capability as the first of its kind.Source: §1 · p. 2

Evidence from psychophysics

• Psychophysical evidence shows multisensory integration: visual mouth motion changes interpretation of spoken sound in the McGurk effect, while sound timing affects whether moving objects seem to collide or overlap.Humans may fuse audio and visual signals at a fairly early processing stage and use them jointly in perceptual grouping. The McGurk effect is less effective after viewers watch unrelated audio and visuals, causing signals to become “unbound”.Source: §2 · pp. 2–3

Self-supervised learning

• The paper’s self-supervised alignment task uses frames and audio from the same video and requires motion analysis, unlike prior image-audio matching that can rely on scene semantics.Earlier methods learned sight-sound correlations from static images and ambient sound or modeled motion in limited domains; temporal-ordering methods use time as supervision, whereas this network analyzes audio-visual actions.Source: §2 · p. 3

Audio-visual alignment

• For speech-video alignment, Chung and Zisserman use a labeled face detector, tracking, and five mouth-cropped frames with a two-stream embedding-loss network.Their alignment goal permits a 2D CNN approach; they also propose image features for self-supervision, but report only preliminary results.Source: §2 · p. 3

Sound localization

• Earlier sound-localization work used mutual information, subspace methods, canonical correlations, and keypoints; this model associates motion and sound through self-supervision.Hershey et al. introduced localization using visual-motion/audio mutual information with a Gaussian process model.Source: §2 · p. 3

Audio-Visual Source Separation

• Prior audio-visual source separation methods use motion correlations, graphical models, priors, factorization, clustering, or beamforming.These methods make strong assumptions about sound-motion relationships and mostly use lab-recorded video; this paper combines its representation with a u-net to separate on- and off-screen sources in internet and broadcast videos.Source: §2 · pp. 3–4

Concurrent work

• Concurrent separation work addressed known-identity speech [44] [45], user-chosen speakers [47], or sounds of visible objects [48] [49], while attention-map methods localized sources in static images [50] [51] [52].The paper distinguishes its multiple-speaker separation as fine-grained motion analysis and its localization as responding to actions in videos.Source: §2 · p. 4

Aligning sight with sound

• The self-supervised objective trains a model to predict whether image and audio streams are synchronized, using temporally shifted audio as misaligned examples, similar to noise-contrastive estimation.The model maximizes the likelihood of synchronized and shifted stream pairs.Source: §3 · p. 4

Fused audio-visual network design

• The proposed 3D multisensory CNN uses early fusion, concatenating spatially tiled audio activations with video activations before further 3D convolutions and global average pooling.The network includes residual connections and resembles ResNet-18, using 3D convolutions following inflated-convolution work [56].Source: §3 · pp. 4–5, Figure 2 · p. 5

Training

• Training used approximately 750,000 AudioSet videos, with 4.2-second clips, audio shifts of 2.0–5.8 seconds, 125 full-frame-rate images, and 21 kHz stereo sound.The clips were sampled from 10-second videos; video inputs used random 224 × 224 crops and random horizontal flips.Source: §3 · p. 5

Task performance

• 59.9% accuracy on held-out videos (chance = 50%) compared with 66.6% ± 2.4% for humans given 15-second clips and 5-second shifts.The model was evaluated on its alignment task; participants were asked to identify the out-of-sync clip.Source: §3 · pp. 5–6
• On Kinetics categories [58], synchronization prediction was most successful for classes involving human speech, including news anchoring, answering questions, and testifying.The category evaluation is shown in Figure A1.Source: §3 · p. 6

Visualizations

• For non-speech Kinetics-Sounds videos, the strongest CAM responses show category examples after face-only activations were removed; training used no labeled data, and heat maps share a consistent scale.Source: Figure 5 · p. 7, §4 · p. 8

Visualizing the locations of sound sources

• The method applies CAM to the model’s final convolutional features and uses the magnitude of the alignment prediction as a measure of patch information and hypothesized sound-source likelihood.The alignment probability is computed from the features using the model’s final affine layer and a sigmoid.Source: §4 · p. 7
• Rather than estimate mutual information between modalities with a generative model, the approach selects patches and sounds that provide the most information about latent alignment y under the learned model.Source: §4 · pp. 7–8

Visualizations

• Top-ranked patches often highlight faces and moving mouths, while strong non-face responses include unusual salient stimuli; weakest-response videos contain relatively few faces.The authors interpret patch informativeness as a proxy for sound-source likelihood, while noting that “sound source” is ambiguous in in-the-wild videos and using it for motions correlated with sound production.Source: §4 · pp. 7–8
• CAM visualizations qualitatively suggest that the model’s attention varies with on-screen motion, unlike single-frame methods that largely attend to sound-making objects rather than actions.Source: §4 · p. 8

5 Action recognition

• For UCF-101 fine-tuning, the model started from alignment-task weights and trained on 2.56-second subsequences with random flips, crops, and audio shifts of up to one frame.At test time, it averaged outputs over 25 clips per video using a center 224 × 224 crop.Source: §5 · pp. 8–9
• On UCF-101 [64], ours scored 82.1% vs. Shuffle-and-Learn at 50.9% accuracy and O3N at 60.3%, near ImageNet-pretrained I3D [56] (84.2%) but below Kinetics-pretrained I3D [56] (94.5%).The model used both sound and vision; the compared ImageNet-pretrained I3D used vision only.Source: §5 · pp. 8–9
• Using ’s cross-video negative pairs instead of temporal misalignment yielded 78.7%, significantly worse than the proposed approach.The authors suggest ’s task can largely be solved without motion analysis.Source: §5 · p. 9
• Ablating audio reduced performance by 5%, training from scratch reduced it by 14%, and spectrogram inputs yielded similar performance.The audio ablation suggests sound contributes while visual features remain useful alone; the 14% drop was similar in magnitude to removing ImageNet pretraining from I3D.Source: §5 · p. 9

6.1 Source separation model

• The model separates synthetic on-screen and randomly paired off-screen audio by combining mixture spectrograms with multisensory video features in a u-net.Features from three temporal scales are interpolated, spatially pooled, tiled across frequency, and concatenated into the encoder; the model predicts log-spectrogram magnitude and phase and uses raw video without preprocessing or labels.Source: §6 · p. 9, Figure 6 · p. 10, §6.1 · pp. 10–11

6.2 Evaluation

• The audio-visual model trained with PIT outperformed the compared methods except on SAR, where the audio-only u-net PIT model was slightly better.It did particularly well when the mixture speakers were the same gender; on GRID transfer, audio-only methods did better on blind separation metrics, which the authors hypothesize reflects domain shift.Source: Table 2 · p. 11, §6.2 · p. 13
• On 200ms videos, the model significantly outperformed Hou et al. and Gabbay et al. [45] on on-screen SDR after audio was downsampled to 2 kHz.Only on-screen audio was evaluated for these baselines, which use 5-frame inputs and mouth-cropped video; the authors’ model uses raw video without face detection.Source: Table 3 · p. 11, §6.2 · pp. 13–14
• Using a single frame instead of motion features increased spectrogram loss from 11.4 to 14.8, with a particularly large drop for same-gender speaker mixtures.Removing the multisensory network’s audio stream also worsened performance, while converting audio to mono had no significant effect; transfer to GRID preserved a similar relative method ordering.Source: §6.2 · pp. 12–13

6.3 Qualitative results

• Qualitative examples show separation of concurrent voices in synthetic mixtures, an 8.3-second translation and interview, and a debate video where masking one speaker selects the other.The authors note that quantitative metrics do not fully convey predicted sound quality; a larger model was also trained on roughly 8× as many videos, with ambient AudioSet tracks sampled about 8% of the time.Source: §6.3 · p. 14

7 Discussion

• The temporal multisensory representation proved useful for action-recognition pretraining, visualizing sound-source locations, and on/off-screen source separation.These are the three downstream tasks demonstrated in the paper.Source: §7 · p. 14
• Future work could develop fused multisensory representations using learning signals beyond temporal misalignment, including ambient sound.The paper presents temporal misalignment as its method and ambient sound as one alternative signal.Source: §7 · p. 14
• A second direction is applying the representation to additional audio-understanding tasks that may benefit from vision and visual applications that may benefit from audio.Source: §7 · p. 14

Temporal alignment

• Temporal alignment training ran for 650,000 iterations using SGD, with authentic and shifted audio examples sharing video-subnetwork activations.Training began at 7.5 Hz for 30,000 iterations with batch size 45, then used batch size 15 across 3 GPUs; the learning rate started at 0.01 and halved every 200,000 iterations. Weights were randomly initialized using.Source: A1 Optimization · p. 18

Action recognition

• Action-recognition networks were fine-tuned for 30,000 iterations using Adam and cropping, left/right flipping, and small audio shifts.The batch size was 24 across 3 GPUs; the learning rate started at 10^-4 and dropped by a factor of 10 every 12,000 iterations, with audio shifts of up to 1 frame.Source: A1 Optimization · p. 18

Source separation

• Source-separation networks were fine-tuned with Adam for 160,000 iterations, while the model for 200ms sequences trained for 250,000 iterations.Both used batch size 18 across 3 GPUs and cropping and left/right flipping; the learning rate dropped after 120,000 iterations for the general model and after 150,000 iterations for the shorter-sequence model.Source: A1 Optimization · p. 19

A2 Baseline model details

• The spectrogram variant uses a 2D CNN adapted from ResNet-18, truncated after conv3_1, with temporal and frequency subsampling.Temporal subsampling follows conv2_2; frequency subsampling follows conv1, conv2_2, and conv3_1.Source: A2 Baseline model details · p. 19
• Its log-spectrograms use an 8 ms step and 24 ms window, then a 3 × F convolution with 512 channels before fusion.The window corresponds to 257 frequency bins, and subsequent network layers match Figure 2.Source: A2 Baseline model details · p. 19