PaperlayerView exact revision on arXiv

Source-linked AI summary

MERLOT: Multimodal Neural Script Knowledge Models

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, Yejin Choi

arXiv:2106.02636v3cs.CVcs.CLcs.LG

TL;DR

MERLOT tackles the challenge of learning temporal commonsense beyond static images and literal captions by self-supervised pretraining on millions of unlabeled YouTube videos. Its multimodal spatial and temporal objectives produce strong video-task performance and transfer to static-image reasoning, while ablations identify benefits from video data, corpus diversity, and varied objectives.

  • Problem

    Learning temporal commonsense from static images and literal captions is limited, while enumerating the relevant facts, inferences, and counterfactuals is prohibitively difficult.

  • Method

    MERLOT uses self-supervised pretraining on 6M unlabeled YouTube videos to align frames with contextualized transcripts and contextualize representations over time.

  • Results

    MERLOT achieves a new state-of-the-art on 12 video tasks and transfers script-knowledge representations to static-image reasoning.

  • Takeaways & Limitations

    Learning full-stack visual reasoning and multimodal world knowledge from diverse video data is a promising path for future research.

  • Takeaways & Limitations

    Future work should explore finer-grained temporal objectives and multilingual learning from non-English videos and YouTube communities.

Abstract

from arXiv · show

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

1 Introduction

MERLOT addresses the difficulty of learning temporal commonsense from static images and literal captions by pretraining on unlabeled YouTube videos. It achieves strong video and image reasoning performance, while ablations highlight the value of video data, corpus diversity, and varied objectives.

  • Static images and literal captions struggle to capture the temporal inferences underlying everyday scenes, while enumerating such knowledge is prohibitively difficult.
  • MERLOT learns multimodal event representations through self-supervised pretraining on 6M unlabeled YouTube videos, matching frames with contextualized transcripts and modeling temporal structure.
  • MERLOT achieves a new state-of-the-art on 12 video tasks spanning recognition- and cognition-level reasoning across long and short timescales.
  • MERLOT transfers to static images, sorting scrambled visual stories into globally coherent narratives and attending across distant captions and frames.
  • Ablations find better pretraining with videos than still images, gains from diverse everyday-situation videos, and continued improvement with more pretraining.

2 Related Work

Earlier vision-language approaches primarily learn static image representations from human-annotated captions, while video work often targets specialized understanding tasks. MERLOT instead learns dynamic multimodal representations from video frames and transcripts without human annotation.

  • Prior joint text-image methods generally learn static-image representations using substantial human annotation, such as literal image descriptions.
  • MERLOT learns dynamic visual representations from video frames and transcripts without human annotation, contrasting with caption-based static-image approaches.
  • Web-video and ASR research has supported specialized tasks including object detection, action recognition, instruction alignment, captioning, and visual reference resolution.
  • Prior “what happens next” and video deshuffling work studies temporal structure, but MERLOT targets multimodal script knowledge using both visual and textual information.

3 MERLOT: Multimodal Event Representation Learning Over Time

MERLOT learns multimodal script knowledge from YouTube videos by jointly encoding frames and transcripts over time. Its architecture and objectives support frame-text matching, masked-word reconstruction, temporal reordering, and downstream vision-language tasks.

  • Architecture: The model jointly encodes video frames and transcript words with image, language, and vision-language Transformer components.Position embeddings distinguish images and captions across different timesteps, while pooled CLS representations support pretraining and downstream tasks.
  • Architecture: The image encoder is trained from random initialization with a grid-based ResNet-50/Vision Transformer design rather than supervised object-detector features.Its forward pass requires 40 gigaFLOPs, compared with 2 teraFLOPs for Faster-RCNN.
  • Pretraining Tasks and Objectives: Three pretraining objectives cover frame-level recognition and video-level reasoning: contrastive frame-transcript matching, masked language modeling, and temporal reordering.The objectives encourage matching corresponding frames and captions, reconstructing corrupted words, and ordering scrambled frames.
  • Results: MERLOT performs best on all metrics in the reported VCR and SIND visual-story comparisons against models using static image-caption training.The SIND setup requires arranging images temporally to match captions, while the VCR comparison uses models at the same base scale.

4 Experiments: Transferring MERLOT to Downstream Tasks

MERLOT is evaluated across static-image and video reasoning tasks, with analyses testing its temporal reasoning, pretraining objectives, context size, and data sources. It achieves state-of-the-art results across 12 video tasks and transfers to visual commonsense reasoning and zero-shot story ordering.

  • Image tasks: MERLOT transfers video-level understanding to VCR, where questions probe commonsense about events, intentions, and what may happen next.
  • Image tasks: MERLOT outperforms same-scale models trained on static images, captions, and supervised object detections across three VCR settings.
  • Qualitative examples: Without fine-tuning, MERLOT orders visual stories using cross-frame coreference and temporal duration cues, although it groups merry-go-round frames in a non-reference order.
  • Video reasoning: MERLOT sets a new state-of-the-art on 12 video reasoning tasks spanning short and long videos.
  • Ablations: The frame-transcript matching loss is crucial for downstream performance, while temporal ordering helps TVQA but slightly reduces VCR performance.
  • Ablations: Using diverse YT-Temporal-180M videos improves VCR performance by 6.5 points over instructional videos, with all data adding another 2.4-point boost.

5 Conclusion, Limitations, and Broader Impacts

MERLOT learns event-level multimodal representations from videos and static images, while the authors identify limitations and broader risks involving bias, privacy, dual use, and energy consumption.

  • Conclusion: MERLOT learns multimodal representations beyond single frames through self-supervised objectives on 6M YouTube videos.
  • Limitations: Future work should explore finer-grained temporal objectives and multilingual pretraining beyond English YouTube videos.
  • Broader Impacts: Local broadcast news comprised 30% of filtered data and may transmit racist patterns in crime coverage.
  • Broader Impacts: YouTube-based pretraining risks embedding social biases, including hegemonic and gendered perspectives and hateful content.
  • Broader Impacts: Video QA pretraining carries dual-use risks because downstream applications may include surveillance.
  • Broader Impacts: Pretraining was expensive upfront, and scaling data or compute could increase environmental cost.

Supplemental Material

The supplemental material documents the dataset, collection pipeline, privacy choices, filtering procedure, and experimental resources supporting MERLOT and YT-Temporal-180M.

  • Supplemental Material: The supplement includes data collection details, corpus exploration, qualitative analyses, experimental settings, and a dataset datasheet.
  • Data Collection: The collection pipeline finds relevant channels, downloads videos with English ASR, denoises transcripts, and aligns words with video frames.
  • Privacy and Related Data: The pipeline combines diverse YouTube videos with HowTo100M videos, while applying privacy-preserving collection and release decisions.
  • Data Collection: Channel discovery combines topic pages, VLOG data, viral how-to and cooking channels, object searches, and breadth-first expansion.
  • Filtering: Filtering removes videos without English ASR, videos over 20 minutes, gaming content, and visually unsuitable or static videos.
  • Filtering: Thumbnail-based filtering favors scenes containing multiple COCO objects and visual variation before full video download.

A.3 Denoising ASR Captions

MERLOT cleans noisy YouTube ASR transcripts, restores written-language conventions, estimates timing, and extracts aligned frame-text segments for pretraining.

  • Motivation: ASR transcripts can differ from written text, lack punctuation and capitalization, and contain mistranscriptions or translation errors.
  • Denoising: A GROVER-Large sequence-to-sequence model denoises ASR using synthetic corruptions derived from written news articles.
  • Denoising: The denoising model attempts to correct corruptions, add punctuation and capitalization, and estimate ASR quality for filtering.
  • Alignment: Clean transcript timing is estimated by aligning noisy and cleaned transcripts word by word with Dynamic Time Warping.
  • Segment Extraction: Aligned transcripts are segmented with buffers of at most L = 32 tokens, then paired with corresponding video frames.
  • Segment Extraction: Examples are formatted with 16 valid frames by merging short videos and splitting longer videos without sequence-level padding.

B Data Exploration

The corpus exploration evaluates curation heuristics and sampled content, finding broad topical coverage and a creator mix dominated by news studios, large YouTubers, and media companies.

  • Validity Check: The extraction pipeline relies on subjective heuristics, and its individual stages cannot feasibly be ablated exhaustively.
  • Validity Check: A validity check randomly sampled 100 videos and assessed language, English predominance, instructional status, creator type, music, and gaming commentary.
  • Validity Check: 86% of sampled videos were primarily in English, 84% were not instructional, and all contained language utterances.
  • Validity Check: News studios were the most common sampled creators at 29%, followed by big YouTubers at 26% and media companies at 24%.
  • Content Exploration: An LDA model over 55K sampled transcripts identified topics spanning sports, politics, fashion, construction, fantasy, and nail painting.
  • Content Exploration: The corpus showed broader topical coverage than comparably sized HowTo100M, and this diversity was helpful for downstream tasks.

C Qualitative Analysis of Model Representations

This section provides qualitative analysis of the representations learned by MERLOT.

  • The section examines MERLOT’s learned representations qualitatively.
  • The analysis focuses on representations learned by MERLOT.
  • MERLOT’s learned representations are assessed through qualitative analysis.

C.1 Analysis of the language-only encoder, and attention masking during pretraining

The language-only encoder identifies contextually informative transcript tokens for masking, encouraging multimodal event representations rather than easy local token recovery. Qualitative examples support this strategy, while attention patterns remain only partially interpretable and can include non-visual tokens.

  • Analysis of the language-only encoder, and attention masking during pretraining: Random BPE masking produced overly easy prediction examples, motivating a strategy that targets more informative visual words.Partially masking long words can allow missing tokens to be recovered from linguistic context alone.
  • Analysis of the language-only encoder, and attention masking during pretraining: Attention-guided masking selects tokens with high language-encoder attention weights instead of masking randomly.The strategy uses attention weights from a language-only encoder trained to match caption sequences with individual frames.
  • Analysis of the language-only encoder, and attention masking during pretraining: Around 1% downstream-task improvement over a SpanBERT baseline followed the attention-guided masking strategy.
  • Analysis of the language-only encoder, and attention masking during pretraining: High-attention tokens included visually meaningful words such as “scissors” and “toys,” even when scissors were absent from selected frames.The authors suggest such tokens can complete the event context, such as explaining how a toy package was opened.
  • Analysis of the language-only encoder, and attention masking during pretraining: The attention patterns also assign high weight to START and contraction-final tokens, whose visual significance remains unclear.START is not selected for masking, limiting its effect on the learning signal.
  • More qualitative examples for zero-shot story ordering: MERLOT produces reasonable temporal orderings for kayaking and tennis sequences, including serving followed by awaiting the return.
  • More qualitative examples for zero-shot story ordering: MERLOT attends globally across frames and captions, unlike CLIP’s independent image-caption matching.Qualitative examples show MERLOT connecting entities across frames and maintaining coherent groups of related images.
  • Analysis of the language-only encoder, and attention masking during pretraining: CLIP performs best in the reported zero-shot setup, although MERLOT outperforms the RSPNet baseline.The authors relate this result to object and background bias in activity-recognition benchmarks and the lower likelihood that speakers name obvious objects.

E Experimental setup and hyperparameters

The experimental setup combines implementation choices for pretraining, downstream finetuning, and unsupervised story ordering. Story-ordering evaluation uses pairwise temporal losses and handles the mismatch between pretraining and task formats with additional pretraining.

  • Experimental setup and hyperparameters: MERLOT uses AdamW with a 3e-4 learning rate, 0.1 weight decay, and β2=0.98 during the described training setup.
  • Experimental setup and hyperparameters: Clean or noisy ASR transcripts are randomly selected per iteration as data augmentation, avoiding within-video alignment inconsistencies.
  • Unsupervised Story Ordering: Story-ordering adaptation added 10 pretraining epochs with a 2e-5 peak learning rate and 384 x 384 resolution for five-segment stories.
  • Unsupervised Story Ordering: The temporal loss compares caption-caption, caption-frame, and frame-frame pairs, while caption-frame comparisons distinguish same-segment, before, after, and cross-video relations.
  • Unsupervised Story Ordering: The unified story-ordering score sums pairwise log-probabilities over caption and frame permutations.
  • Unsupervised Story Ordering: Randomly sampling and averaging two position-embedding orderings addressed sensitivity to the order of frame position tokens.Using more than two sampled orderings did not change performance.
  • Experimental setup and hyperparameters: Downstream inputs concatenate encoded images, word embeddings, and position embeddings before joint vision-language encoding.
  • Experimental setup and hyperparameters: Downstream finetuning used 384×704 resolution, standard dataset splits, AdamW, warmup followed by linear decay, weight decay, dropout, and early stopping.

E.3 Static Image Reasoning Tasks

Static-image reasoning evaluation includes VCR’s multiple-choice answer and rationale subtasks, with entity references visually highlighted. MERLOT is jointly finetuned on both subtasks, and the section situates this setup relative to prior VCR second-stage pretraining.

  • Static Image Reasoning Tasks: VCR contains 290k questions over 110k movie scenes across question answering and answer-justification subtasks.
  • Static Image Reasoning Tasks: The joint Q→AR metric requires selecting both the correct answer and the correct rationale.
  • Static Image Reasoning Tasks: Referenced VCR entities are highlighted with colored boxes using a consistent mapping between colors and entity names.
  • Static Image Reasoning Tasks: MERLOT is jointly finetuned on Q→A and QA→R using separate MLP heads and cross-entropy classification over four candidates.
  • Static Image Reasoning Tasks: Prior VCR second-stage pretraining applies masked-language modeling on VCR training data rather than directly answering the questions.UNITER reported a 2.8 % point performance boost from this second-stage pretraining.
  • Static Image Reasoning Tasks: Table 7 reports per-question-category results for MSRVTT-QA.

E.4 Video Reasoning Tasks

This section describes how MERLOT is adapted and finetuned for a range of video reasoning benchmarks, using uniformly sampled frames and task-specific answer formats.

  • MSRVTT-QA: MSRVTT-QA uses 244K questions over 10K videos, with five uniformly selected frames and a 1K-answer vocabulary.Out-of-vocabulary answers are automatically marked wrong.
  • TVQA: TVQA and TVQA+ use five-choice questions over video clips, with six uniformly selected frames per clip.TVQA+ additionally supplies bounding boxes linking depicted objects to visual concepts.
  • VLEP: VLEP evaluates which of two events is more likely to happen next from six uniformly selected video frames.The model predicts confidence for each candidate event with a classifier trained using cross-entropy loss.
  • DramaQA: DramaQA applies a five-choice question-answering setup to 17.9K questions over 23.9K video clips, using six uniformly selected frames.The classifier predicts confidence for each answer candidate with cross-entropy loss.
  • TGIF-QA: TGIF-QA includes action, transition, and FrameQA tasks requiring spatio-temporal reasoning, with five frames and candidate-answer classification.Action identifies repeated actions, Transition asks about states before or after another state, and FrameQA uses open-ended questions with dictionary-based answers.
  • ActivityNetQA: ActivityNetQA uses 58K questions over 5.8K videos, five uniformly selected frames, and a 1K-answer vocabulary.Answers are encoded one-hot and classified with binary cross-entropy loss.
  • Additional tasks: Several additional benchmarks use five candidate answers or captions paired with five uniformly selected frames, including FiTB, caption selection, and MSRVTT Multichoice.FiTB predicts a missing word, caption selection chooses the fitting caption, and MSRVTT Multichoice follows the LSMDC Multichoice objective.

F Datasheet for YT-Temporal-180M

The datasheet characterizes YT-Temporal-180M as a large pretraining corpus of YouTube videos paired with aligned ASR transcripts, including its construction, preprocessing, distribution, and known limitations.

  • Motivation: YT-Temporal-180M was created to investigate learning events from videos using frames and captions aligned over time.The dataset is intended for representation learning, potentially with or without ASR captions.
  • Contents: The dataset contains 6 million videos, 5 billion BPE transcript tokens, and 180 million extracted image frames.Each instance consists of raw video frames and text processed into temporally aligned segments.
  • Scope: The dataset is a sample without explicit relationships between videos and has no recommended splits because it was built for pretraining rather than evaluation.The authors expect its scale to reduce concerns about overfitting.
  • Limitations: The authors identify YouTube-related biases and note that YouTube ASR is often noisy despite correction procedures.Many transcript errors cannot be fixed by the described pipeline.
  • Distribution: The planned release distributes metadata and YouTube video links rather than the videos themselves, with privacy and copyright considerations cited.The authors intend to allow users to delete videos and avoid releasing the video files.
  • Collection and preprocessing: Collection used the YouTube API and youtube-dl, with data directly observable from YouTube.Preprocessing included ASR denoising, BPE tokenization, segmentation, and extraction of middle frames.
Loading 2106.02636v3…