PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Video Representations from Large Language Models

Yue Zhao, Ishan Misra, Philipp Krähenbühl, Rohit Girdhar

arXiv:2212.04501v1cs.CV

TL;DR

Video-language learning is limited by small or weakly aligned paired video-text datasets. LaViLa repurposes pretrained LLMs as visually conditioned narrators and uses their dense, diverse annotations for contrastive video-text pretraining. It outperforms prior methods across first- and third-person benchmarks, including reported gains of 10.1% on EGTEA and 5.9% on Epic-Kitchens-100 retrieval.

  • Problem

    Paired video-text corpora are much smaller than large image-text datasets, while existing video annotations can be sparse or poorly aligned.

  • Method

    LaViLa finetunes pretrained LLMs as visually conditioned narrators to generate dense annotations, then trains dual-encoder video-language representations with them.

  • Results

    LaViLa sets a new state-of-the-art across six first- and third-person video benchmarks, including 10.1% absolute gain on EGTEA mean accuracy.

  • Takeaways & Limitations

    LaViLa shows positive scaling with more training narrations, larger visual backbones, and stronger LLMs, and can outperform a full-data baseline using less than 50% of the data.

Abstract

from arXiv · show

We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs). We repurpose pre-trained LLMs to be conditioned on visual input, and finetune them to create automatic video narrators. Our auto-generated narrations offer a number of advantages, including dense coverage of long videos, better temporal synchronization of the visual information and text, and much higher diversity of text. The video-text embedding learned contrastively with these additional auto-generated narrations outperforms the previous state-of-the-art on multiple first-person and third-person video tasks, both in zero-shot and finetuned setups. Most notably, LaViLa obtains an absolute gain of 10.1% on EGTEA classification and 5.9% Epic-Kitchens-100 multi-instance retrieval benchmarks. Furthermore, LaViLa trained with only half the narrations from the Ego4D dataset outperforms baseline models trained on the full set, and shows positive scaling behavior on increasing pre-training data and model size.

1. Introduction

LaViLa addresses the limited scale of paired video-text data by using pretrained LLMs to generate dense video narrations, producing stronger representations across first- and third-person tasks.

  • Video-language learning is constrained by video-text corpora that are much smaller than billion-scale image-text datasets.
  • LaViLa sets a new state-of-the-art on six popular first- and third-person video benchmarks.
  • LaViLa repurposes pretrained LLMs as visually conditioned narrators and uses them to densely annotate long videos.The generated pseudo-supervision extends between and beyond existing annotated snippets.
  • Dense, well-aligned, and diverse generated text addresses sparse captions, weak speech alignment, and limited annotations in long-form videos.Prior automatically transcribed datasets can have visual-text alignment of ≤50%.
  • 10.1% absolute gain on EGTEA mean accuracy, 5.9% on Epic-Kitchens-100 mAP, and 5.9% on Ego4D intra-video accuracy are reported.The gains hold in both zero-shot transfer and fine-tuned settings.
  • LaViLa trained with 50% of human-labeled narrations can outperform a baseline trained with all narrations, while gains increase with more data and larger backbones.

2. Related Work

Prior video-language datasets remain limited in scale, scenario coverage, or alignment, motivating methods that can exploit larger sources of video data.

  • Vision-language representation learning commonly maps visual and textual embeddings into a common space using metric-learning techniques.
  • Conventional video-text datasets have limited scenarios or insufficient scale for learning generic video representations.
  • How-To100M scales video-text collection through automatic speech transcription, but ASR introduces textual noise and visual-text unalignment.
  • WebVid provides 10 million short videos with textual descriptions but remains several orders of magnitude smaller than large image-text datasets.

3. Preliminaries

Video models represent arbitrarily long videos as sets of short annotated clips and can learn aligned visual-text embeddings with dual encoders and contrastive objectives.

  • A long video is represented by N short clips, each defined by start and end frames and typically associated with an annotation.
  • Annotated clips often fail to densely cover the entire video because annotation is costly and visual content can be redundant.
  • Dual-encoder approaches encode video clips and free-form text separately, then project both modalities into a shared embedding space.
  • NARRATOR generates new action descriptions, while REPHRASER changes word order and diversifies narrations with related verbs or nouns.
  • InfoNCE contrastive learning associates corresponding video and text embeddings within a batch in both video-to-text and text-to-video directions.

4. LAVILA

LAVILA uses two LLM-based supervision mechanisms to expand video-text annotations: NARRATOR generates descriptions from visual clips, while REPHRASER diversifies existing narrations. These annotations train dual encoders contrastively, with cached generation preserving standard pre-training speed.

  • LLM supervision: LAVILA combines NARRATOR-generated descriptions and REPHRASER paraphrases with original annotations for dual-encoder training.The combined training set is (X, Y) ∪ (X′, Y′) ∪ (X, Y′′).
  • NARRATOR: NARRATOR conditions a pretrained language model on video embeddings using added cross-attention modules to autoregressively generate narrations.Visual embeddings are produced by a video encoder, while cross-attention lets text tokens attend to visual information.
  • NARRATOR: NARRATOR densely annotates unlabeled intervals by sampling clips from the remainder of long-form videos, alongside re-captioning existing clips.The method uses a stationary-process assumption and samples clips with duration based on the average ground-truth clip length.
  • REPHRASER: REPHRASER generates alternative narrations with synonym replacement and word-order changes, increasing text diversity for existing clips.It uses a text-to-text encoder-decoder model to autoregressively generate a new sentence from the original narration.
  • Dual-encoder training: The generated video-narration pairs are cached before contrastive pre-training, so LAVILA trains as fast as a standard dual-encoder contrastive model.Training uses symmetric cross-entropy over similarity scores for labeled and pseudo-captioned clips.

5. Experiments

LAVILA improves video-language representations across first- and third-person benchmarks, while remaining effective with limited narrations and complementary language supervision.

  • Main Results: LAVILA outperforms prior work across EK-100 MIR settings, metrics, and retrieval directions, with larger gains using a larger model.The best model achieves over 10% absolute gain zero-shot and 5.9–7.1% gain after fine-tuning.
  • Main Results: LAVILA surpasses previous EGTEA state-of-the-art by more than 10% absolute mean accuracy after fine-tuning the video encoder.The comparison includes methods using visual, auditory, and textual inputs, alongside a visual-only baseline with a comparable backbone.
  • Third-Person Video Pre-training: After HowTo100M pre-training, LAVILA outperforms prior methods on third-person UCF-101 and HMDB-51 action classification under linear probing.The experiments use a temporally aligned subset containing 3.3M sentences from 247k videos.
  • Semi-supervised Learning: LAVILA consistently outperforms the ground-truth-only baseline at 10%, 20%, 50%, and 100% narration availability.It can reach similar performance to the baseline with often less than 50% of the data and comparable performance to state-of-the-art with fewer annotations.
  • Language Supervision Ablations: REPHRASER and NARRATOR each improve the ground-truth baseline, while combining them yields a 3.5% average mAP improvement on EK-100 MIR.Dense pseudo-captioning further improves performance on 3 out of 6 reported metrics.

6. Conclusion and Future Work

The paper concludes that LAVILA improves video-language learning by automatically narrating long videos with LLMs and identifies scaling model size, visual backbones, and narrations as future directions.

  • LAVILA automatically narrates long videos with LLMs to improve video-language representation learning.
  • LAVILA sets new state-of-the-art results on six first- and third-person video understanding benchmark tasks.
  • Future Work: The method shows positive scaling behavior with more training narrations, larger visual backbones, and stronger LLMs.

C. Dataset Details

The datasets and evaluation setup combine narrated egocentric videos with action labels, textual annotations, and zero-shot or fine-tuned downstream protocols.

  • Ego4D: Ego4D contains 3,670 hours of egocentric video with temporally dense narrations, each paired with a timestamp and free-form sentence.
  • Annotations: Ego4D annotations include start and end timestamps, short textual narrations, and associated verb and noun classes.The action class can be uniquely determined by combining the verb and noun.
  • Evaluation: Zero-shot evaluation applies the pretrained model directly without downstream tuning, whereas fine-tuned evaluation performs end-to-end training on the target training split.

D.1. Pre-training on Ego4D

LAVILA uses a dual-encoder representation model and a visually conditioned GPT-2 narrator, with downstream evaluation spanning retrieval and action-recognition settings.

  • Architecture: Attention pooling converts variable-resolution visual features into a fixed-length representation for the text decoder’s cross-attention.
  • Narrator: GPT-2 XL is connected to the video encoder through learnable cross-attention modules with tanh gating.
  • Narrator: NARRATOR is trained on Ego4D video-narration pairs and selected using validation word-level accuracy and perplexity.
  • Downstream Evaluation: LAVILA fine-tuning evaluates EK-100 retrieval and classification, EGTEA action recognition, and additional video tasks using task-specific protocols.

D.7. LAVILA for Third-person Video Pre-training

The section specifies the pre-training and evaluation procedure used for third-person video experiments, including hard-negative sampling and frozen-encoder linear probing.

  • Each selected clip is paired with one additional hard negative clip sampled from the same video during batch construction.
  • Linear probing freezes the video encoder and trains a linear SVM on extracted video features.
  • Evaluation samples four evenly spaced 32-frame clips from each HMDB-51 or UCF-101 video and averages their prediction scores.

E. Additional Results

Additional EK-100 CLS results show that LaViLa achieves state-of-the-art top-1 action accuracy and retains its semi-supervised advantage with a TimeSformer-Large backbone.

  • LaViLa achieves state-of-the-art performance on EK-100 CLS in terms of top-1 action accuracy.
  • With a TimeSformer-Large backbone, LaViLa outperforms the ground-truth-only baseline at every evaluated narration-data fraction.

F. Additional Ablations

The additional ablations examine baseline design, semi-supervised data usage, initialization, batch size, projection dimension, and contrastive-loss temperature on EK-100 MIR.

  • Baseline comparisons: Zero-shot EK-100 MIR evaluation compares an improved Ego4D-pretrained baseline under video-input and initialization variants.
  • Baseline comparisons: Using video rather than extracted RGB frames increases the same EgoVLP+EgoNCE model’s average mAP by 5.7% and average nDCG by 4.3%.
  • Baseline comparisons: A TimeSformer-Base plus DistilBERT implementation with standard InfoNCE reaches 24.1% average mAP and 28.0% average nDCG.
  • Architecture and initialization: Replacing the spatial-temporal visual encoder with a CLIP-style ViT-Base reduces average mAP by 1.5% and average nDCG by 1.0%.
  • Training configuration: Increasing batch size from 512 to 1,024 improves performance, while gains diminish at 2,048; the default is 1,024.
  • Training configuration: Projection dimension 256 performs best compared with dimensions 128 and 512.
  • Training configuration: A fixed temperature of 0.07 for both NARRATOR and REPHRASER pairs outperforms the other tested temperature settings.

G. Qualitative Results

Qualitative examples illustrate that LaViLa’s NARRATOR generates diverse, viewpoint-aware descriptions, while REPHRASER varies existing narrations through paraphrasing.

  • Viewpoint coverage: NARRATOR can produce reasonable captions from different views, including activities of both the camera wearer and another person.
  • Figure examples: The qualitative figures use Ego4D video examples with associated video IDs and timestamps for visual inspection.
  • Generated narrations: NARRATOR generates descriptions that may focus on different objects or people involved in the observed action.
  • Generated narrations: REPHRASER changes narration word order and diversifies wording with related verbs or nouns.
Loading 2212.04501v1…