PaperlayerView exact revision on arXiv

Source-linked AI summary

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, James Glass

arXiv:2006.09199v2cs.CVcs.CLcs.MMcs.SDeess.AS

TL;DR

Many visually grounded video-language methods require human captions or ASR transcripts, whose collection and language coverage are limited. AVLnet learns shared audio-visual representations from raw, randomly segmented video clips, then adds text branches for caption-based retrieval. The paper reports state-of-the-art image retrieval and establishes video retrieval tasks, while showing that learned representations also support downstream text retrieval.

  • Problem

    Many video-language approaches require human captions or ASR transcripts, which are expensive to collect and unavailable for most languages.

    Source: §1 · p. 1

  • Method

    AVLnet learns a shared audio-visual embedding space from raw audio and video in randomly segmented clips, with text branches added for caption-based retrieval.

    Source: §1 · p. 2

  • Results

    AVLnet achieves state-of-the-art image retrieval on Places Spoken Caption and establishes video retrieval tasks on YouCook2, CrossTask, and MSR-VTT.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    The learned audio-visual representations relate speech and natural sounds to visual objects, and can support downstream text retrieval with small amounts of captions.

    Source: §1 · p. 2, §8 · p. 13

Abstract

from arXiv · show

Current methods for learning visually grounded language from videos often rely on text annotation, such as human generated captions or machine generated automatic speech recognition (ASR) transcripts. In this work, we introduce the Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. To circumvent the need for text annotation, we learn audio-visual representations from randomly segmented video clips and their raw audio waveforms. We train AVLnet on HowTo100M, a large corpus of publicly available instructional videos, and evaluate on image retrieval and video retrieval tasks, achieving state-of-the-art performance. We perform analysis of AVLnet's learned representations, showing our model utilizes speech and natural sounds to learn audio-visual concepts. Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval. Our code, data, and trained models will be released at avlnet.csail.mit.edu

1 Introduction

The introduction motivates learning visually grounded language directly from video audio and vision, avoiding the caption and ASR annotations required by many existing approaches. It presents AVLnet, trained on randomly segmented HowTo100M clips, and extends it with text-processing models for downstream retrieval.

1 Introduction

  • Existing spoken-caption approaches require recorded captions, while video models often require human captions or ASR transcripts, limiting annotation scalability and language coverage.ASR can miss visually relevant non-speech sounds, make errors in noisy or accented audio, and is unavailable for 98% of languages.Source: §1 · pp. 1–2
  • Instructional videos provide abundant paired audio and vision, with narration that can relate spoken words to visual entities even when those entities do not produce the sound.This differs from audio-video work focused on identifying non-speech sounds and localizing their sources.Source: §1 · p. 1
  • AVLnet uses audio and video branches, non-linear feature gating, and the Masked Margin Softmax loss to learn a shared embedding space from raw clips.The loss simulates retrieval and trains against many negative samples.Source: §1 · p. 2
  • AVLnet learns audio-visual representations from raw video channels and achieves state-of-the-art speech-image retrieval while establishing video retrieval tasks on three datasets.The video datasets are YouCook2, CrossTask, and MSR-VTT.Source: §1 · p. 2
  • AVLnet-Text adds a text branch, and the authors report that audio representations trained without captions support downstream text retrieval in zero-shot and fine-tuned settings.The paper suggests these representations can be adapted using only a small amount of text captions.Source: §1 · p. 2

2 Related Work

Related work spans visually grounded speech learning, self-supervised audio-video learning, and text-video modeling from instructional videos. AVLnet builds on these lines by using naturally occurring video audio to relate speech to visual entities and by incorporating text when available.

Learning Visually-Grounded Speech

  • Visually grounded speech models learn audio-visual embeddings from spoken captions and images, but much prior work uses collected or synthetic speech captions rather than naturally occurring online videos.Prior work also explored speech retrieval, word-like unit discovery, and other paired visual and spoken data.Source: §2 · p. 3

Self-Supervised Audio-Video Learning

  • Self-supervised audio-video work includes proxy tasks, source separation, and spatial audio generation, while retrieval work either pairs video frames with surrounding speech or relates objects to their sounds.AVLnet instead uses entire video clips to relate objects to the speech describing them and trains directly for audio-video retrieval.Source: §2 · p. 3

Multi-Modal Learning from Instructional Videos

  • Instructional-video models commonly learn text-video embeddings without audio, and models that include audio still require text captions; AVLnet learns from raw video audio and can also use ASR or annotated text.This positions AVLnet as a raw-audio approach that can incorporate text when it is available.Source: §2 · p. 3

3 Technical Approach for Audio-Video Models

AVLnet pools local audio and visual features into modality vectors, gates their projections into a shared embedding space, and trains retrieval with a bidirectional contrastive loss. Random clip sampling avoids ASR-based segmentation and performs comparably to ASR-boundary sampling in the reported experiments.

3.1 Audio-Video Models

  • Parallel visual and audio branches extract local features and pool them into vectors representing each clip's overall content.The visual branch combines 2D and 3D CNN features into a 4096-dimensional vector; the audio branch mean-pools a temporal feature map into a 1024-dimensional vector and processes raw audio without pretraining.Source: §3.1 · pp. 3–4

3.2 Audio-Video Gated Embeddings

  • Non-linear feature gating projects the audio and visual vectors into a shared embedding space and recalibrates dimensions by learned importance.The gating is intended to encourage dimensions to activate in unison across modalities.Source: §3.2 · p. 4

3.3 Contrastive Loss for Audio-Video Retrieval

  • The Masked Margin Softmax loss contrasts true audio-visual pairs with impostor pairs and compares positives against a wider range of negatives.It applies retrieval losses in both directions, with negatives drawn from clips in the same video and from other videos.Source: §3.3 · p. 4

3.4 Video Clip Sampling

  • Randomly segmenting unlabeled instructional videos into clips avoids supervised ASR-boundary selection and supports languages without ASR.Despite the possibility of silent or non-salient clips, results are comparable between random clips and ASR-boundary clips.Source: §3.4 · pp. 4–5

4 Technical Approach for Text-Audio-Video Models

The text branch max-pools pretrained word embeddings, and AVLnet incorporates text through either independently gated tri-modal branches or audio-text fusion before gating.

4.1 Text Processing

  • Text captions are represented by max-pooled GoogleNews Word2Vec embeddings and integrated through either a separate text branch or audio-text fusion.Source: §4.1 · p. 5

4.2 Independent Tri-Modal Branch Architecture

  • AVLnet-Text-Tri keeps text, audio, and video branches separate, gates each independently, and jointly optimizes losses for all three modality pairs.Source: §4.2 · pp. 5–6

4.3 Audio-Text Fused Architecture

  • AVLnet-Text-Fused combines audio and text embeddings before gating, then trains their shared language embedding against video using the MMS loss.Audio and text captions from each clip are sampled together.Source: §4.3 · p. 6

5 Experimental Setup

The section specifies training and retrieval evaluations for audio-video and text-video models, including datasets, retrieval settings, and implementation choices.

5.1 Audio-Video Experiments

  • AVLnet is trained on HowTo100M using randomly segmented video and audio clips rather than ASR-defined segments to reduce supervision.Source: §5.1 · p. 6
  • After HowTo100M training, AVLnet is fine-tuned on Places Audio Caption, which pairs 400k images with 1,000 hours of spoken captions, for retrieval in both directions.Evaluation uses validation-set R@1, R@5, and R@10.Source: §5.1 · p. 6
  • Audio-video retrieval is evaluated on YouCook2, CrossTask, and MSR-VTT, including zero-shot, fine-tuned, and no-pretraining settings.Tasks retrieve video from audio and audio from video; YouCook2 and CrossTask are instructional datasets, while MSR-VTT contains general YouTube videos.Source: §5.1 · pp. 6–7

5.2 Text-Video Experiments

  • AVLnet-Text models train on ASR-segmented HowTo100M clips with ASR text, then fine-tune and evaluate on YouCook2, MSR-VTT, and LSMDC.Source: §5.2 · p. 7
  • Text-video evaluation uses AVLnet-Text-Tri for text-to-video retrieval with summed text-video and text-audio similarities, and AVLnet-Text-Fused for joint text-and-audio-to-video retrieval.Visual feature extractors are frozen during AVLnet training because of GPU memory limitations.Source: §5.2 · p. 7, §5.1 · p. 7

6 Audio-Video Experiments

Across the experiments, HowTo100M pretraining improves retrieval across video datasets, and AVLnet also uses speech and natural sounds to retrieve semantically related content. Ablations identify effective design choices, while visualization reveals audio-visual concept correspondences.

6.1 Comparison to State-of-the-art

  • HowTo100M pretraining improves retrieval across three datasets, especially in zero-shot and fine-tuned settings; AVLnet beats baselines, and Places fine-tuning yields large gains over prior results.Source: §6.1 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

6.2 Ablation Studies

  • Gating beats linear and nonlinear projection heads; MMS beats MIL-NCE, Binary Cross Entropy, Max-Margin Ranking, and InfoNCE, with MIL-NCE worst.Source: §6.2 · p. 9, Table 3 · p. 9
  • Using both 2D and 3D visual features improves performance, random and ASR-defined clips perform similarly, and 10s clips outperform 2.5, 5, and 20s clips.Source: §6.2 · p. 9, Table 3 · p. 9

6.3 Retrieving Speech versus Non-Speech Sounds

  • Retrieval is higher on Speech-241 than Sounds-241, while Sounds-241 performance remains far above the 4.1% chance level.Source: Table 4 · p. 9, §6.3 · p. 10

6.4 Qualitative Retrieval Results

  • Retrieval examples show semantic matches between spoken queries and videos, and sizzling-only audio retrieves frying scenes despite having no speech.Source: §6.4 · p. 10, Figure 3 · p. 10

6.5 Audio-Visual Concept Discovery

  • Unit visualization ranks dimensions by joint audio and visual purity; independently selected activations correspond, such as “oil” with pans of oil, suggesting learned audio-visual concepts.Source: §6.5 · pp. 10–11, Figure 4 · p. 11

7 Text-Video Experiments

AVLnet-Text performs competitively across text-video retrieval datasets, while a low-resource setup shows that speech-trained AVLnet can be adapted to text retrieval with little caption data.

7.1 Text-Video Retrieval Results

  • Both AVLnet-Text models outperform prior models on YouCook2; AVLnet-Text-Tri beats the prior audio-inclusive model on MSR-VTT and is on par on LSMDC, with higher R@1.The LSMDC audio is often silent because audio-description narration is inserted during dialogue breaks.Source: §7.1 · p. 11, Table 5 · p. 12
  • AVLnet-Text-Fused typically has higher recall than AVLnet-Text-Tri, but their retrieval setups differ: T+A→V versus T→A+V.Source: §7.1 · p. 11

7.2 Training with Text in a Low-Resource Scenario

  • After training on HowTo100M without text, AVLnet can retrieve with text in both zero-shot and fine-tuned conditions using small amounts of downstream captions.AVLnet-Text-Fused still achieves higher results, indicating that ASR text captions during HowTo100M training are beneficial.Source: §7.2 · pp. 11–12, Table 6 · p. 12
  • These low-resource results suggest AVLnet learns language representations from speech and can adapt audio representations to text with only a small amount of captions.Source: §7.2 · p. 12

8 Conclusion

The paper introduces self-supervised audio-video learning from instructional videos and reports retrieval results across image and video tasks. Its tri-modal extension adds text narration for text-to-video retrieval.

  • AVLnet learns audio-video representations directly from instructional videos, reducing the need for spoken or text annotations.Source: §8 · p. 13
  • The model establishes video-retrieval baselines on YouCook2, CrossTask, and MSR-VTT, and achieves state-of-the-art image retrieval on Places Spoken Caption.Source: §8 · p. 13
  • AVLnet relates speech and sound to visual objects, while AVLnet-Text adds narration to learn a multimodal embedding space for text-to-video retrieval.Source: §8 · p. 13

Broader Impact

The authors describe potential for learning speech-video correspondences across languages with naturally generated video content. The accompanying MMS illustration depicts similarity maximization for true pairs and minimization for imposters.

Broader Impact

  • Naturally generated video content could support learning speech-video correspondences in languages without ASR; the authors note ASR covers less than 2% of the world’s languages.They expect this may help scale speech-technology advances and broaden computer interaction.Source: Broader Impact · p. 13

A Illustration of the Masked Margin Softmax (MMS) Loss

B Video Datasets

The appendix documents dataset composition and availability for training and retrieval experiments, including video losses that affect reproducibility and evaluation splits.

  • YouTube dataset availability can shrink over time, complicating reproduction and comparison, so the paper reports downloaded video counts and experimental splits.Source: B Video Datasets · p. 16
  • HowTo100M originally contained 1,238,792 videos; 1,166,089 were available at download and used for training.The download occurred between December 2019 and March 2020.Source: B Video Datasets · p. 16
  • YouCook2 comprises cooking videos split into training, validation, and testing; the experiments use 9,586 training clips and 3,350 validation clips.Some videos were unavailable on YouTube.Source: B Video Datasets · p. 16
  • CrossTask videos are associated with tasks defined by step lists, and each video contains a subset of its task’s steps.The validation set has 360 videos and 2,852 clips; the training set has 18,067 clips.Source: B Video Datasets · p. 16
  • For MSR-VTT, the study uses the training-7k/test-1k-A split, trains on 6,783 clips with audio, and evaluates on 968 audio-containing test clips.The 32 test clips without audio count as retrieval mistakes.Source: B Video Datasets · p. 17
  • LSMDC consists of movie scenes with audio-description transcripts, but the recorded audio descriptions were unavailable and original movie audio is often silent.The experiments use 101,079 training clips and 1,000 testing clips.Source: B Video Datasets · p. 17

C Additional Concept Discovery Examples

AVLnet’s shared embedding dimensions align audio and visual features across spoken object words, action words, and natural sounds.

C Additional Concept Discovery Examples

D Additional Video and Language Retrieval Examples

AVLnet retrieves semantically similar video and audio clips across YouCook2 and CrossTask. The examples suggest this similarity can persist even when the correct result is not among the top five.

D Additional Video and Language Retrieval Examples

Loading 2006.09199v2…