PaperlayerView exact revision on arXiv

Source-linked AI summary

Look, Listen and Learn

Relja Arandjelović, Andrew Zisserman

arXiv:1705.08168v2cs.CVcs.LG

TL;DR

The paper asks what can be learned from correspondence between visual and audio streams in large collections of unlabelled videos. It trains visual and audio networks from scratch with AVC, and reports strong sound-classification and visual-transfer performance, localization, and fine-grained recognition.

  • Problem

    The paper investigates whether visual and audio semantic information can be learned from unlabelled videos without additional supervision.

  • Method

    AVC trains separate visual and audio networks from scratch to classify whether a video frame and short audio clip correspond, using naturally paired and mismatched examples.

  • Results

    The learned representations set state-of-the-art results on two sound-classification benchmarks, perform on par with state-of-the-art self-supervised ImageNet approaches, and support localization and fine-grained recognition.

  • Takeaways & Limitations

    Visual-audio correspondence in raw videos can provide supervision for learning useful semantic representations in both modalities.

  • Takeaways & Limitations

    AVC uses correlation rather than stronger temporal concurrency, and unconstrained videos can contain noisy, unrelated, occluded, or out-of-view audio sources.

Abstract

from arXiv · show

We consider the question: what can be learnt by looking at and listening to a large number of unlabelled videos? There is a valuable, but so far untapped, source of information contained in the video itself -- the correspondence between the visual and the audio streams, and we introduce a novel "Audio-Visual Correspondence" learning task that makes use of this. Training visual and audio networks from scratch, without any additional supervision other than the raw unconstrained videos themselves, is shown to successfully solve this task, and, more interestingly, result in good visual and audio representations. These features set the new state-of-the-art on two sound classification benchmarks, and perform on par with the state-of-the-art self-supervised approaches on ImageNet classification. We also demonstrate that the network is able to localize objects in both modalities, as well as perform fine-grained recognition tasks.

1. Introduction

The paper asks whether correspondence between visual and audio streams in unlabelled videos can train both modalities without additional supervision. It introduces AVC and reports strong transfer performance, localization, and fine-grained recognition.

  • Visual and audio events often co-occur because they share a common cause, providing a potential source of supervision in raw video.
  • AVC trains visual and audio networks to predict whether a video frame corresponds to a sound snippet, without requiring exact temporal synchrony.
  • The paper trains both networks from scratch on large numbers of unlabelled videos to learn visual and audio semantic information.
  • The audio network exceeds recent audio-recognition systems trained with visual supervision, while visual features perform similarly to purely visual self-supervised approaches.
  • Activation visualization localizes audio-event sources in video frames and corresponding regions of the sound source, alongside fine-grained distinctions such as different instruments.
  • Unlike related approaches that fix pretrained visual networks or target lip-speech synchronization, AVC trains both modalities from scratch for general semantic representations.

2. Audio-visual correspondence learning

The method learns audio-visual correspondence from naturally paired and mismatched frame-audio examples. L3-Net combines separate visual and audio feature extractors with a fusion classifier, despite substantial noise and modality mismatch in unconstrained videos.

  • The core learning signal is correspondence between visual and audio streams that appear together at the same time in a video.
  • AVC is a binary classification task distinguishing frame-audio pairs sampled simultaneously from one video from mismatched pairs drawn from different videos.
  • The task encourages semantic learning because solving it requires detecting concepts in both visual and audio domains.
  • Unconstrained videos make AVC difficult because audio may be noisy, unrelated to visible content, occluded, out of view, or introduced during editing.
  • Network architecture: L3-Net contains vision and audio subnetworks that extract features, plus a fusion network that predicts whether the signals correspond.
  • Network architecture: The fusion network concatenates two 512-D modality features into a 1024-D vector and produces a 2-way correspondence output through fully connected layers.

3. Results and discussion

The paper evaluates L3-Net on correspondence prediction, transfer classification, and qualitative analyses of learned representations. The supplied passage describes this evaluation scope but reports no specific results.

  • The evaluation measures AVC performance, transfer quality of visual and audio features, and qualitatively examines what the network learns.
  • Transfer experiments test the learned features on visual and audio classification tasks.
  • The qualitative analysis follows evaluation of the correspondence task and transfer-learning performance.

3.1. Datasets

The study uses two video datasets: a large unconstrained Flickr collection for representation learning and a smaller labelled Kinetics-Sounds subset for quantitative evaluation.

  • Flickr-SoundNet: Flickr-SoundNet contains over 2 million unconstrained Flickr videos; experiments use 500k videos split into 400k training, 50k validation, and 50k test clips.Only the first 10 seconds of each video are used.
  • Kinetics-Sounds: 19k Kinetics-Sounds clips form a labelled evaluation dataset, split into 15k training, 1.9k validation, and 1.9k test videos.The clips are 10 seconds long and cover 34 human action classes selected for potentially visual and aural manifestation.

3.2. Audio-visual correspondence

Audio-visual correspondence is evaluated as binary matching between video frames and one-second audio clips, with L3-Net compared against supervised baselines on Kinetics-Sounds.

  • Evaluation: The AVC evaluation uses balanced positive and negative pairs, so chance performance is 50%.Table 1 reports test-set accuracy for L3-Net and two supervised baselines trained on each dataset’s training split.
  • Results: 74% and 78% accuracy are achieved by L3-Net on the two datasets, compared with 50% chance performance.The task is difficult because inputs are local and videos are unconstrained; informal human tests were only a few percent better.
  • Baselines: The supervised-pretraining baseline performs on par with L3-Net, whereas supervised direct combination performs significantly worse.The direct-combination baseline combines independently supervised features without training the combined system for AVC.

3.3. Audio features

Audio representations learned from Flickr-SoundNet are evaluated on ESC-50 and DCASE, where L3-training establishes large gains over the previous sound-classification state of the art.

  • Evaluation: The audio subnetwork is trained on Flickr-SoundNet and its representations are evaluated on standard sound-classification benchmarks.The experimental setup follows the direct-comparison procedure used by Aytar et al.
  • Benchmarks: ESC-50 contains 2,000 five-second clips across 50 balanced classes, while DCASE scene classification contains 10 classes with 10 training and 100 test clips per class.The benchmarks cover environmental sounds and acoustic scenes, respectively.
  • Conclusion: L3-training achieves the new state of the art on both sound-classification benchmarks, despite learning both networks without supervision.The networks are trained using a random subset of SoundNet videos for efficiency, leaving possible gains from using all available data.
  • Results: 5.1% and 5% absolute improvements over SoundNet are obtained on ESC-50 and DCASE, respectively.The results reduce the gap to human performance on ESC-50 by 72% and reduce DCASE error by 42%.

3.4. Visual features

Visual features learned through AVC transfer effectively to ImageNet and develop semantic, fine-grained, and spatially localized visual concepts without sound input.

  • ImageNet transfer: 32.3% ImageNet accuracy is on par with state-of-the-art self-supervised methods and exceeds random, data-dependent, and Context Encoder initialization.The comparison uses linear classification of extracted features, though architectures differ across methods.
  • Learnt concepts: Visual units automatically recognize semantic entities and finer-grained categories, including instruments and activity-related concepts.The units are examined by ranking Kinetics-Sounds test images according to pool4 activation magnitude.
  • Spatial localization: Activation heatmaps localize objects despite clutter and occlusions by displaying the conv4 2 channel corresponding to a selected pool4 unit.The visual features used for these analyses do not take sound as input.
  • ImageNet transfer: AVC training produces a larger improvement over random initialization than all competing self-supervised methods.The comparison measures gains relative to each network’s random-initialization performance.
  • Training-data comparison: Flickr-video training differs from competing ImageNet-based self-supervision because video frames are typically off-centre, motion-blurred, and less likely to contain fine-grained categories seen during training.Despite these differences, the resulting visual features perform on par with ImageNet-trained self-supervised methods.

3.5. Qualitative analysis

Qualitative analyses show that AVC training learns semantic visual and audio features, localized event-relevant regions, and produces embeddings organized by action semantics rather than random structure.

  • Vision features: The vision subnetwork learns semantic entities, scene categories, and people-related concepts without explicit supervision.Examples include instruments, outdoor and concert scenes, water, crowds, text, babies, adults, and crowds.
  • Vision features: Visual activation heatmaps localize objects despite clutter and occlusion, sometimes focusing on interaction regions rather than complete objects.The playing-clarinet unit mainly activates at the interface between the player’s face and clarinet.
  • Audio features: The audio subnetwork learns semantic entities and fine-grained distinctions such as fingerpicking versus playing bass guitar.Audio heatmaps also reveal event-specific frequency and temporal patterns, including low-frequency bass-guitar responses and temporal steps for fingerpicking and tap dancing.
  • Versus random features: 512 pool4 units were examined because the authors considered whether semantic-looking results could arise by chance, then compared the trained network with random weights.The random network failed to show comparable correlations between units and action classes.
  • Versus random features: Trained L3-Net embeddings cluster by action semantics in t-SNE, unlike random representations, while similar classes still overlap without class-level supervision.Typing additionally separates according to whether hands appear with the keyboard, reflecting whether the sound is expected in the frame-sound pair.
  • Versus random features: 50-100%: L3-Net embeddings outperform the best random baselines on clustering quality.The evaluation uses k-means with 64 clusters and reports Normalized Mutual Information against Kinetics-Sounds action classes.

4. Discussion

The discussion reports strong performance across sound classification and visual self-supervision, alongside fine-grained recognition and localization abilities. It also identifies synchronized audiovisual concurrency as an unexplored stronger training signal.

  • Discussion: The AVC-trained network achieves superior sound-classification results to recent methods using fixed pretrained visual networks.The authors conjecture that a trainable visual network can better exploit the variety of visual information in video.
  • Discussion: Visual features are on par with state-of-the-art self-supervised approaches, while both modalities learn fine-grained distinctions such as bass versus acoustic guitar.The discussion also cites saxophone-versus-clarinet recognition as another example.
  • Discussion: Activation visualizations localize extended regions corresponding to instruments, rather than only the few moving pixels concurrent with sound.This contrasts the learned localization with the highlighted-pixel behavior described in earlier work.
  • Limitations and future work: Training for audiovisual concurrency could provide information beyond correlation but would require multiple video frames instead of a single frame.The paper leaves the additional benefit of this stronger condition for future exploration.

A. Kinetics-Sounds

Kinetics-Sounds is formed from 34 Kinetics action classes, while Figure 10 visualizes trained and random L3-Net representations for a subset of those classes.

  • Dataset: The Kinetics-Sounds dataset contains 34 action classes taken from Kinetics.The listed classes include activities involving instruments, speech, manual actions, sports, and household tasks.
  • Representation visualization: Figure 10 compares trained and random-weight L3-Net t-SNE embeddings for Kinetics-Sounds, showing only 13 of the 34 classes for clarity.The embeddings are colored with labels only for visualization, not training.
Loading 1705.08168v2…