MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

Rowan Zellers; Jiasen Lu; Ximing Lu; Youngjae Yu; Yanpeng Zhao; Mohammadreza Salehi; Aditya Kusupati; Jack Hessel; Ali Farhadi; Yejin Choi

Source-linked AI summary

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, Yejin Choi

arXiv:2201.02639v4Jan 7, 2022cs.CVcs.CLcs.LGcs.SDeess.AS

TL;DR

Multimodal video understanding needs representations that integrate vision, language, and sound over time. MERLOT Reserve addresses this with joint multimodal encoding and contrastive masked-span learning at scale, achieving strong finetuned and zero-shot results across video and vision-language benchmarks.

Problem
The paper asks whether time-locked correlations among vision, language, and sound can support neural script knowledge that transfers to tasks without sound.
Method
Reserve jointly represents video frames, audio, and subtitles, masking text or audio spans and matching predictions to independently encoded targets during large-scale pretraining.
Results
Reserve learns strong multimodal representations, improving over prior work on VCR and TVQA and achieving competitive zero-shot performance across four video benchmarks, including outperforming supervised approaches on STAR.
Takeaways & Limitations
Audio-powered multimodal pretraining benefits tasks using fewer modalities, including vision-language reasoning without audio, while enabling zero-shot video reasoning.
Takeaways & Limitations
Reserve learns only from 40-second videos, relies on ASR subtitles, and matches rather than generates text and audio.

Abstract

from arXiv · show

As humans, we navigate a multimodal world, building a holistic understanding from all our senses. We introduce MERLOT Reserve, a model that represents videos jointly over time -- through a new training objective that learns from audio, subtitles, and video frames. Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale: we pretrain on 20 million YouTube videos. Empirical results show that MERLOT Reserve learns strong multimodal representations. When finetuned, it sets state-of-the-art on Visual Commonsense Reasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%, and 1.5% respectively. Ablations show that these tasks benefit from audio pretraining -- even VCR, a QA task centered around images (without sound). Moreover, our objective enables out-of-the-box prediction, revealing strong multimodal commonsense understanding. In a fully zero-shot setting, our model obtains competitive results on four video tasks, even outperforming supervised approaches on the recently proposed Situated Reasoning (STAR) benchmark. We analyze why audio enables better vision-language representations, suggesting significant opportunities for future research. We conclude by discussing ethical and societal implications of multimodal pretraining.

1. Introduction

MERLOT Reserve asks whether time-locked correlations across vision, language, and sound can produce transferable multimodal script knowledge. It introduces a jointly trained model and objective, then reports strong finetuned and zero-shot performance across video and vision-language tasks.

Motivation and contributions: Reserve jointly represents video frames, audio, and subtitles to learn multimodal script knowledge through time-locked cross-modal supervision.The model uses each modality to teach the others rather than learning from only audio-image pairs or subtitled videos.
Motivation and contributions: 20 million YouTube videos provide the pretraining scale for Reserve’s multimodal representation learning.The training collection contains videos with multiple aligned modalities, supporting the paper’s large-scale pretraining experiments.
Motivation and contributions: The contrastive masked span objective hides text or audio spans and trains the model to identify their independently encoded representations.The objective is designed to learn from both text and audio while discouraging shortcut learning from raw perceptual inputs or individual tokens.
Results: Over 5% on VCR and over 7% on TVQA are reported improvements over prior work, while Kinetics-600 reaches 91.1% accuracy.VCR is evaluated without audio; TVQA improves further when audio is provided.
Results: Reserve obtains competitive zero-shot results on four video benchmarks, nearly doubles prior zero-shot MSR-VTT QA accuracy, and surpasses supervised approaches on STAR.The evaluated benchmarks are STAR, EPIC-Kitchens, LSMDC-FiB, and MSR-VTT QA.
Conclusion: The results suggest that time-locked, reentrant learning across modalities is a promising direction for multimodal representations.The authors identify substantial room for future work.

2. Related Work

Prior work studies multimodal fusion, video-language learning, and cross-modal supervision through largely separate approaches. Reserve combines these lines by jointly representing all video modalities while using an objective that enables modalities to supervise one another.

Joint multimodal representations: VisualBERT-style models fuse images and text, often using supervised object detectors and masked language modeling on image-caption pairs.Their masked language objectives can condition token prediction on visual input and unmasked text.
Joint multimodal representations: MERLOT learns vision-text representations from subtitled web videos but lacks audio as both a represented and training modality.Reserve is presented as outperforming MERLOT.
Cross-modal supervision: Joint multimodal training can be shortcut by simpler within-modality patterns, especially when subtitles are noisy.This can lead models to ignore visual input during masked language modeling.
Cross-modal supervision: Independent modality encoders and contrastive matching have been applied to images, video frames, transcripts, and audio, but are also used for single-modality tasks.Late fusion can combine independent encoders, though the paper describes it as less expressive than joint early fusion.
This work: Reserve combines joint video representation across vision, audio, and text with a learning objective that enables cross-modal co-supervision.This addresses the two related-work lines in a single model and training framework.

3. Model: Reserve

Reserve splits videos into aligned temporal segments, independently encodes their modalities, and fuses them with a joint Transformer. Contrastive masked-span and frame-matching objectives train cross-modal representations from a 20-million-video dataset.

3.1. Model architecture: Each temporal segment contains a middle video frame, ASR tokens, and audio, with segments defaulting to 5 seconds.During input construction, the model receives exactly one of text or audio for each segment before masking.
3.1. Model architecture: Independent image, audio, and text encoders produce representations that a bidirectional joint Transformer fuses across modalities and time.The image encoder uses ViT features, audio is divided into subsegments, and text targets come from a separate span encoder.
3.1. Model architecture: The architecture predicts masked text and audio representations using independently encoded targets from the corresponding spans.This design supports zero-shot prediction through target representations computed separately from the joint encoder.
3.2. Contrastive Span Training: Contrastive span training masks 25% of text and audio subsegments and matches each masked prediction only to its independent span encoding.The objective is applied across temporally aligned video segments and operates at semantic spans rather than individual tokens.
3.2. Contrastive Span Training: A separate frame-text contrastive loss matches video frames with contextualized transcript representations, and the final loss sums text, audio, and frame losses.The paper also reports that text remains important because subtitles support shortcut avoidance, fine-tuning, and textual downstream interfaces.
3.2. Contrastive Span Training: The training setup uses audio both as a target and, in another masked-video configuration, as an input while predicting missing text.Using both configurations is intended to avoid shortcut learning from same-modality perceptual correlations.
3.3. Pretraining dataset: YT-Temporal-1B contains 20 million English-subtitled YouTube videos and 1 billion frames for Reserve pretraining.The collection procedure directs scraping toward public, large, and monetized channels as a privacy measure.

4. Experiments

Experiments show that contrastive span pretraining, especially with audio, improves multimodal representations and transfers strongly to finetuned and zero-shot video understanding tasks. Reserve achieves state-of-the-art results while also reducing compute relative to several prior systems.

Experimental scope: Reserve’s experiments cover ablations, finetuned VCR, TVQA, Kinetics-600, and zero-shot video benchmarks.The VCR task combines answer and rationale selection into the Q→AR metric.
Ablations: Contrastive span pretraining outperforms mask LM by over 2% after one vision-and-text pretraining epoch.The authors hypothesize that span-level prediction encourages concept-level representations.
Ablations: Adding audio as a target boosts VCR accuracy by 0.7%, while using audio as input and target increases accuracy by 1% in total.Relaxing strict temporal localization causes a slight performance drop, suggesting localization helps.
VCR: Reserve-L outperforms all prior work by over 5% on the VCR Q→AR metric, including large industry ensembles.The comparison focuses on single models in the table, although the text also notes performance against ensembles.
Efficiency: Reserve-L uses fewer FLOPs than MERLOT while scoring 7% higher, and Reserve-B uses less than one-tenth the FLOPs of base detector-based models.Reserve-B has 200M finetunable parameters and performs similarly to the 378M-parameter UNITER-Large.
TVQA: Reserve-L improves over prior work on TVQA by 7.6%, while subtitle-and-frame input already exceeds prior work by over 3%.Combining subtitle-only and audio-only predictions improves over MERLOT by over 4%.
Kinetics-600: Jointly representing frames and sound improves Kinetics-600 performance by 1.7% and yields a 1.5% advantage over larger models including MTV-Huge.Reserve also outperforms VATT, which represents audio independently from vision.
Zero-shot evaluation: In zero-shot evaluation, Reserve achieves state-of-the-art results on STAR and strong performance across EPIC-Kitchens, LSMDC, and MSR-VTT QA.It outperforms supervised prior work on STAR and significantly outperforms CLIP on LSMDC.

5. Qualitative Analysis: Why does audio help?

Reserve learns to use audio as complementary supervision for temporal, physical, and social understanding beyond what subtitles provide. Across pretraining, it increasingly identifies audio cues and uses them alongside text and video.

Audio supervision: Reserve-B’s audio matching improves across 10 pretraining epochs, showing progressively stronger identification of masked audio spans.The validation examples require selecting the correct missing text or audio span from 48k candidates.
Audio supervision: Audio supplies orthogonal supervision for dynamic state changes, such as popcorn popping, that subtitles may not capture.In the first example, the model assigns 60% probability to the audio span containing slowing popcorn pops by the final epoch.
Audio supervision: Audio also conveys human communication dynamics, including a speaker’s apparent emotion and social interaction.The second example provides only the text “why,” while audio supplies substantially more information about the scene.
Role of text: Text remains a crucial complement: later training increases text matching when recognizing that a tense situation likely prompts a “why” question.The model first matches the audio nearly perfectly, then improves its text-based social inference.
Learning through multimodal reentry: The findings support multimodal reentry, in which time-locked signals from one modality educate representations learned from others.The paper connects this mechanism to hypotheses about how children learn visual and world knowledge across interacting senses.

6. Conclusion, Limitations, Broader Impact

The paper concludes that Reserve learns jointly from sound, language, and vision and performs well in finetuned and zero-shot settings. It also identifies practical and societal limitations, including short video duration, ASR dependence, restricted output modes, privacy concerns, and potential harmful uses.

Conclusion: Reserve learns jointly through sound, language, and vision using a new pretraining objective, with strong finetuned and zero-shot performance.The conclusion presents multimodal learning and downstream performance as the paper’s central contribution.
Limitations: The model is limited to 40-second videos, depends on ASR-generated subtitles, and matches rather than generates text and audio.These constraints define the current scope of the model’s training inputs and outputs.
Broader impact: Potential societal impacts include assistance for low-vision or Deaf users, as well as surveillance, social-bias amplification, privacy risks, dual use, and energy consumption.The authors discuss these issues as broader-impact dimensions of multimodal pretraining.

Abstract

This paper studies multimodal neural script knowledge through joint video representations that incorporate vision, audio, and text. It emphasizes large-scale data construction, privacy-aware release choices, and the distinction between temporal reasoning and person-level memorization.

Contribution: The paper presents a model for multimodal neural script knowledge that incorporates audio alongside text and video frames.It frames this approach as related to holistic human interpretation across multiple modalities.
Privacy: Privacy considerations include user expectations, platform data practices, consent, and the choice to release the model only for research use.The paper contrasts YouTube uploads with private-information exposure on the open web and data-broker ecosystems.
Recognition and memorization: The authors aim to study temporal script knowledge rather than long-tailed visual recognition, while noting that noisy name mentions in ASR subtitles may create a trade-off.Their recognition evaluation uses 120 celebrities with 100 samples each as a conservative upper-bound study.

3. Image to name.

Reserve performs poorly on broad celebrity recognition compared with CLIP, despite recognizing a small set of highly famous individuals. The results suggest a representation emphasis on temporal reasoning and commonsense rather than person-level memorization.

Recognition results: Reserve achieves less than 11% accuracy at celebrity recognition across all tested modality settings.Performance is substantially below the recognition-optimized CLIP model in the image-to-name setting.
Recognition results: CLIP reaches 86% accuracy on image-to-name celebrity classification, far exceeding Reserve.The authors relate this gap to similarity between CLIP’s pretraining data and the evaluation setting.
Recognition results: Reserve reliably recognizes only a few very famous celebrities, while struggling on most others.Examples of relatively well-recognized figures include Oprah Winfrey, Justin Bieber, LeBron James, and Roger Federer.
Interpretation: The findings suggest Reserve’s objectives and data may reduce memorization of particular people relative to models trained on web images with captions.The paper presents this as a possible privacy advantage, while noting that more work is needed.
Interpretation: Reserve’s stronger target is temporal understanding and commonsense reasoning, not person-level recognition.The authors suggest strong temporal reasoning may be learnable without person-level memorization.

A.2. Biases in (pre)training data.

Reserve’s pretraining knowledge is shaped by biases in YouTube content, recommendation dynamics, moderation, and automatically generated captions. These sources can produce and potentially amplify social biases.

Reserve’s knowledge is situated within YouTube and may reflect surface co-occurrences rather than truly causal factors.
YouTube data contains harmful microaggressions, alt-right content, and disproportionately filtered minoritized voices despite platform moderation.
Recommendation incentives favor certain content types, including highly gendered roles that may bias situation understanding.
Because Reserve relies primarily on model-generated ASR text, it may amplify known gender biases and mishandle identity markers such as pronouns.

A.3. Dual use.

The paper discusses dual-use risks from multimodal video representations, including fake-content generation, surveillance, harmful deployment, and substantial pretraining energy use. It limits release to research use while acknowledging unresolved risks.

A finetuned or expanded Reserve could assist actors in generating fake text, audio, or visual content, although the released model cannot directly generate them.
Reserve may support surveillance summaries of activities without identifying people, but its low person-recognition performance may limit recognition-focused applications.
YouTube reporting bias may make Reserve poorly suited to sparse surveillance tasks, though adaptation requirements and risks for large actors remain uncertain.
Linking voices with appearances and recognizing gender can cause harm, especially for transgender individuals, if deployed in production.
3 weeks of TPU v3-512 pretraining produced a reported net carbon footprint of 8.23 tons of CO2 equivalent.
Reserve-B may reduce downstream energy use relative to MERLOT, which is more efficient than earlier detector-based approaches.
The authors release video IDs and checkpoints exclusively for research because privacy, bias, and dual-use risks remain incompletely understood.

B.3. Efficiency metrics of our model

Reserve is evaluated for computational efficiency against UNITER and MERLOT while documenting implementation choices that reduce training and inference costs. The paper reports strong efficiency alongside competitive VCR performance.

Reserve models are far more efficient in FLOPs than UNITER; Reserve-L is roughly on par with MERLOT while exceeding it by 6% in VCR accuracy.
UNITER’s supervised object-detection backbone substantially increases FLOPs through high-resolution dense ResNet processing.
Reserve uses a lighter 288x512 image resolution, while MERLOT uses 384x704 and remains less costly than UNITER.
FLOPs are used as the primary comparison metric because prior work identifies compute as important for model scaling.
Training used AdamW, bfloat16 precision, NaN-to-zero handling, and no gradient clipping; occasional loss spikes remained during Reserve-L pretraining.
Duplicated video copies reuse image and audio encodings and sample disjoint masks, improving efficiency and enlarging contrastive negative pools.
Splitting videos into two groups of eight segments controls quadratic Transformer memory growth, while vectorization combines joint inputs in one call.
Reserve’s contrastive span approach avoids MERLOT’s sequential attention-masking calls, enabling the vectorized implementation.

C. Pretraining Data Formatting: alignment and masking

Reserve formats videos into temporally aligned frame, text, and audio segments, then masks aligned subsegments for cross-modal prediction. Because YouTube timings are imperfect, it refines timestamps and applies conservative masking to reduce cheating.

C. Pretraining Data Formatting: alignment and masking: Pretraining constructs masked segments from frames, ASR tokens, and audio using a 5-second sliding window.
C. Pretraining Data Formatting: alignment and masking: Each audio segment is split into three 1.66-second subsegments so aligned text spans and audio spans can be masked and contrastively predicted.
C. Pretraining Data Formatting: alignment and masking: Long pauses trigger occasional segment merging, producing playback rates of 1x, 2x, or 3x while retaining a middle frame.
C.1. YouTube Caption Timings: Caption timestamps are usually early by roughly 0.1 seconds and provide loose intervals that make pauses difficult to localize precisely.
C.2. Refining timing information: A lightweight MLP regressor trained against Google Cloud ASR refines transcript start and end offsets using lexical, punctuation, timing, and neighboring-word features.
C.2. Refining timing information: The regressor bounds predicted timing offsets with tanh and applies the corrected timings to all transcripts.
C.3. Handling worst-case scenarios in masking, when alignment isn’t perfect: Residual errors are usually local single-word misalignments, while the preprocessing regressor reduces average transcript timing error; caption lead remains typically 0.1 seconds.
C.3. Handling worst-case scenarios in masking, when alignment isn’t perfect: The masking design separates VT→TA and VTA→T prediction and conservatively moves nearby tokens to the predicted span to limit overlap-based cheating and speaker identification.

D.2. Setup and prompting for Zero-shot tasks

Reserve is evaluated zero-shot across four video tasks using prompt conversion, masked text prediction, sampled frames, and optional audio. It achieves competitive or superior results despite using no task supervision.

D.2. Setup and prompting for Zero-shot tasks: Reserve constructs zero-shot label spaces from candidate answers and averages frame-level logits to produce task-specific predictions.CLIP is evaluated similarly, although it cannot jointly encode text or videos.
D.2.1 Zero-shot Action Anticipation on EPIC-Kitchens: EPIC-Kitchens action anticipation predicts the next noun and verb from video context, using class-mean top-5 recall with τa = 1.The task uses 700 unscripted cooking videos and predicts actions one second after the observed clip.
D.2.1 Zero-shot Action Anticipation on EPIC-Kitchens: Reserve-L with audio trails the supervised AVT+ method by 1.6% on verbs and 3.3% on nouns, while outperforming it on Unseen Kitchen and Tail Classes.Adding audio improves Reserve-L by an average of 3.0%.
D.2.2 Zero-shot Situated Reasoning: STAR converts question-answer pairs into masked statements, then scores four candidate answers using sampled frames and optional audio.This reduces domain shift between YouTube pretraining data and visual-question prompts.
D.2.2 Zero-shot Situated Reasoning: STAR performance exceeds all supervised prior work by 3.7%, and adding audio yields a further average improvement of 1.1%.The result is reported for the base model and its audio-equipped variant.
D.2.3 Zero-shot LSMDC: LSMDC fill-in-the-blank replaces the description blank with MASK and scores vocabulary candidates using uniformly sampled frames and optional audio.The task uses 30,354 test examples from 10K movie clips.
D.2.3 Zero-shot LSMDC: 31% accuracy with audio exceeds human text-only performance of 30.2%, while a supervised LSTM reaches 34.4% in the text-only setting.The comparison indicates that this task contains textual bias unavailable to the zero-shot model.
D.2.4 Zero-shot MSRVTTQA: For MSR-VTT-QA, questions are converted into statements and evaluated by comparing the masked representation with the 2000 most common answers.The conversion is intended to minimize drift from the pretraining distribution.

E. Dataset Collection

The dataset pipeline expands from existing YouTube corpora to a filtered, visually grounded collection, using annotations and language-model scoring before download. Additional evaluations probe audio and multimodal recognition.

E. Dataset Collection: The dataset uses YouTube public data to study multimodal script knowledge learned from web videos alone.The collection strategy was motivated by concerns about diversity and local-news concentration in MERLOT’s earlier corpus.
E. Dataset Collection: The pipeline identifies candidate videos, applies inexpensive-to-expensive filters, and avoids downloading unsuitable videos when metadata or captions suffice.This two-phase strategy follows a collection approach similar to MERLOT.
E. Dataset Collection: YouTube recommendations expanded the search to 2 million channels containing 200 million videos.The process began from non-news and non-sports videos in YT-Temporal-180M and retrieved linked channel inventories.
E. Dataset Collection: Mechanical Turk workers annotated 2,000 videos for groundedness, genre, and explicit content, providing training data for metadata-based channel selection.The annotations were used to train a model that predicts video suitability from channel-title examples under a weak-supervision assumption.
E. Dataset Collection: Filtering removed channels with likely non-English, slideshow, racist, or sexist content, while channel budgets favored visually diverse material.Budgeting used predicted variation in objects, actions, and scenes.
E. Dataset Collection: The final process produced 30 million YouTube video IDs judged likely to be high quality.Genre estimates were 1% gaming, 11% news, 20% how-to, 20% chatting, 5% sports, 5% music, 3% movies/drama, 4% documentary, and 31% miscellaneous; gaming was removed.
E. Dataset Collection: Downloaded videos were filtered for dense English speech, requiring at least 50 words in a 30-second interval and at least 80% English-transcript probability.Metadata filtering also removed gaming videos before transcript processing.
E. Dataset Collection: Reserve’s zero-shot audio classification evaluates environmental, urban, and celebrity sounds, with AudioCLIP outperforming it by 16% on ESC50 and 6.4% on US8K.Reserve nevertheless shows multimodal effects: ESC50 and US8K accuracy improves when image and text are supplied together.

F.2. Additional Qualitative Analysis

Qualitative examples track Reserve-B’s masked-span predictions during pretraining. Match performance improves over time, with audio difficulty shaped by accents, lecture context, and span truncation.

F.2. Additional Qualitative Analysis: Figure 10 shows Reserve-B predictions for masked audio and text spans across pretraining, with match performance increasing over time.The examples were selected randomly rather than for unusually large improvements.
F.2. Additional Qualitative Analysis: An Australian accent may make the first-row audio prediction easier, while the lecture-video setting may simplify the second-row prediction.These are proposed explanations for differences among examples, not controlled comparisons.
F.2. Additional Qualitative Analysis: In the third row, both audio and text-span prediction improve, with text slightly favored by the end of pretraining.Audio truncation may make the audio span harder because it is shorter than the corresponding text span.