PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, Zicheng Liu

arXiv:2209.01540v5cs.CV

TL;DR

VidL pre-training lacks a clearly effective masked visual modeling strategy, despite MVM’s success in pure vision. This paper systematically evaluates eight MVM targets in end-to-end VIOLET and develops VIOLETv2, which improves performance across 13 VidL benchmarks. The results identify SIF as the strongest video-text target and show that MVM design matters for downstream transfer.

  • Problem

    Previous VidL studies found only limited benefit from reconstructive visual objectives, leaving unclear which MVM targets can effectively support VidL learning.

    Source: §1 · p. 1

  • Method

    The study evaluates eight MVM target types in end-to-end VIOLET, combining masked visual prediction with video-text matching and masked language modeling.

    Source: §3.2 · p. 3, §3.3 · p. 3

  • Results

    VIOLETv2 achieves strong improvements across 13 VidL benchmarks, including +5.4% video question answering accuracy, +6.6% text-to-video retrieval recall, and +11.4 CIDEr for video captioning.

    Source: §1 · p. 2, §6 · pp. 7–8

  • Takeaways & Limitations

    SIF is the most effective video-text MVM target, and effective MVM training requires target and masking choices tailored to VidL inputs.

    Source: §1 · pp. 1–2, §7 · p. 8

    Prior masked frame modeling uses pre-extracted video features that cannot be refined during pre-training, limiting its effectiveness.

    Source: §1 · p. 1

Abstract

from arXiv · show

Masked visual modeling (MVM) has been recently proven effective for visual pre-training. While similar reconstructive objectives on video inputs (e.g., masked frame modeling) have been explored in video-language (VidL) pre-training, previous studies fail to find a truly effective MVM strategy that can largely benefit the downstream performance. In this work, we systematically examine the potential of MVM in the context of VidL learning. Specifically, we base our study on a fully end-to-end VIdeO-LanguagE Transformer (VIOLET), where the supervision from MVM training can be backpropagated to the video pixel space. In total, eight different reconstructive targets of MVM are explored, from low-level pixel values and oriented gradients to high-level depth maps, optical flow, discrete visual tokens, and latent visual features. We conduct comprehensive experiments and provide insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2. Empirically, we show VIOLETv2 pre-trained with MVM objective achieves notable improvements on 13 VidL benchmarks, ranging from video question answering, video captioning, to text-to-video retrieval.

1. Introduction

The study addresses the limited effectiveness of masked visual modeling for video-language pre-training by systematically testing diverse targets in end-to-end VIOLET. The resulting VIOLETv2 achieves strong improvements across 13 VidL benchmarks, while target effectiveness depends on the input modality.

Motivation

  • Earlier masked frame modeling relies on pre-extracted features that cannot be refined during pre-training, while prior VIOLET discrete-token reconstruction produced only marginal downstream gains.These limitations motivate testing a broader set of end-to-end reconstructive targets.Source: §1 · p. 1
  • Although masked visual objectives have succeeded in pure vision, it remains unclear which MVM variant benefits VidL learning when paired language already supplies high-level semantics.The paper frames this as the central unresolved question for VidL MVM design.Source: §1 · p. 1

Study scope

  • The work systematically evaluates eight MVM targets, spanning low-level pixels and gradients, geometric and motion cues, discrete tokens, and visual or multimodal features.The study uses end-to-end VIOLET so MVM supervision can affect video representations during pre-training.Source: §1 · p. 1, Figure 1 · p. 2

Findings

  • SIF is the most effective MVM target for video-text inputs, whereas the same SIF target causes a large downstream drop when trained with image-text pairs.This indicates that MVM target effects differ between video-text and image-text inputs.Source: §1 · pp. 1–2

Contributions

  • VIOLETv2 improves over models trained on the same 5M corpus by +5.4% accuracy on video question answering, +6.6% recall on text-to-video retrieval, and +11.4 CIDEr on video captioning.It also shows notable advantages over VIOLET despite using much less pre-training data.Source: §1 · p. 2

2. Related Work

Related work spans VidL understanding, multimodal pre-training, end-to-end video representation learning, and masked visual modeling. This study extends VIOLET by comparing diverse MVM targets and masking strategies rather than relying on a single visual-token objective.

Video-Language Understanding

  • VidL research covers text-to-video retrieval [31] [39] [41] [62], video question answering [27] [37] [38], moment retrieval [21] [25] [31] [39], and video captioning [62] [78] [91].These tasks evaluate understanding of the physical world through joint visual and textual perception.Source: §2 · p. 2
  • Prior VidL models often use pre-extracted video features, while later work explores additional raw-video modalities and end-to-end training to improve video representations.This places the paper’s end-to-end design within a broader shift toward learning directly from video inputs.Source: §2 · p. 2

Masked Visual Modeling

  • VIOLET introduced masked visual token modeling for VidL pre-training but reported only marginal downstream improvements.The present study broadens the target space beyond discrete visual tokens.Source: §2 · p. 2
  • Unlike VIOLET, this work evaluates diverse MVM targets and masking combinations, reporting large downstream VidL improvements.The comparison motivates a systematic rather than single-objective treatment of MVM design.Source: §2 · pp. 2–3

3. Method

The method pre-trains an end-to-end VIOLET transformer by masking video patches and reconstructing selected visual targets alongside video-text matching and masked language modeling. It compares eight target-feature families to determine which visual information best supports VidL learning.

End-to-End Video-Language Transformer

  • VIOLET combines a Video Swin Transformer, Language Embedder, and Cross-modal Transformer to produce joint video-language representations from video patches and word tokens.The cross-modal transformer outputs video-patch, [CLS], and word-token representations.Source: §3.1 · p. 3

Problem Setting

  • MVM randomly masks portions of video frames across spatial and temporal dimensions and predicts the corresponding target features using remaining video content and text.This prediction setup is intended to promote cross-modality learning.Source: §3.2 · p. 3
  • The overall pre-training objective combines MVM, video-text matching, and masked language modeling: L = L_MVM + L_VTM + L_MLM.VTM predicts whether a video-text pair matches, while MLM predicts masked word tokens from context.Source: §3.2 · p. 3

Target Features

  • MVM is defined as masked visual feature prediction, with target-feature choice determining which aspect of visual modeling VIOLET learns.The study treats target design as the core open question for VidL MVM.Source: §3.3 · p. 3
  • The study tests Pixel, HOG, Depth, Flow, VQ, SIF, TVF, and MMF targets using linear or two-layer MLP prediction heads and an L1 loss by default.Targets range from raw frame patches and gradients to discrete, spatial, temporal, and multimodal representations.Source: §3.3 · pp. 3–4
  • VIOLETv2 denotes the strongest model obtained from the study’s most effective MVM training strategy.The name is reserved for the final model rather than every evaluated VIOLET variant.Source: §3.3 · p. 4

4. Study: Target Features for MVM

The target-feature study finds that supervised spatial image features are the strongest individual MVM target, while pixels help modestly, HOG and Depth can hurt, and temporal features mainly benefit retrieval. Combining targets does not outperform SIF alone.

One-stage Visual Targets

  • +0.2% TGIF-Frame accuracy and +0.8% DiDeMo AveR are the relatively small gains from reconstructing RGB pixels, whereas HOG yields -0.8% and -2.0% on those metrics.The authors hypothesize that HOG’s missing color information harms VidL understanding.Source: §4 · pp. 4–5

Combining MVM Targets

  • Combining MVM targets performs worse than SIF alone; Pixel+SIF beats Pixel alone but does not improve over SIF by itself.The experiments therefore do not support naive addition of multiple MVM losses as a general improvement.Source: Table 2 · p. 5

Supervised Pseudo-label Targets

  • Depth does not improve over the VTM+MLM baseline, possibly because foreground-background separation can suppress background information relevant to downstream tasks.This is presented as an interpretation of the depth-target behavior.Source: §4 · p. 5

Supervised Visual Feature Targets

  • +5.8% AveR on DiDeMo retrieval and +0.7% TGIF-Frame accuracy make SIF the strongest reported target-feature improvement over the VTM+MLM baseline.TVF instead gives +3.6% AveR on retrieval but -0.1% TGIF-Frame accuracy, indicating task-dependent benefits.Source: §4 · p. 5, Table 3 · p. 5

Discrete and Multimodal Targets

  • VQ and MMF require expensive pre-trained image-text models and produce only slight gains on one task each.VQ helps TGIF-Frame by +0.3%, while the passage does not report a broader benefit for MMF.Source: §4 · p. 5

5. Analyses of MVM

The analyses identify SIF as the strongest video-text MVM target, while showing that masking design, target combinations, and image-text transfer materially affect downstream VidL performance.

Combining MVM Targets

  • Combining MVM targets performs worse than SIF alone, with Pixel+SIF improving over Pixel but not over SIF.The authors hypothesize that pixel-level detail conflicts with high-level semantics in image-classifier features.Source: §5 · pp. 5–6

Masking Ratio

  • 30% masking is the best reported masking ratio for downstream performance under random masking with SIF.The supplied masking-ratio caption specifies WebVid pre-training with VTM+MLM+MVM (SIF) for five epochs.Source: Table 5 · p. 6

Masking Strategy

  • Blockwise masking reduces easy recovery from neighboring patches, and combining blockwise with random masking gives the best DiDeMo retrieval performance.Random masking can permit spurious success because nearby spatial-temporal patches are analogous.Source: §5 · p. 6, Table 4 · p. 6
  • Attention-based masking is ineffective for both downstream tasks and increases training time fourfold relative to random or blockwise masking.The authors attribute the performance drop to masking important visual or textual elements together, leaving less relevant content intact.Source: §5 · pp. 6–7

Applying MVM to Image-Text Data

  • Adding MVM to combined image-text and video-text pre-training yields gains exceeding +0.4% on TGIF-Frame and +2.6% on DiDeMo AveR.The comparison removes MVM from both image and video data while keeping other training settings fixed.Source: §5 · p. 7, Table 7 · p. 7

6. Main Results

VIOLETv2 achieves broad gains across video question answering, captioning, and text-to-video retrieval, including against methods using substantially more pre-training data. It also substantially outperforms the original VIOLET under direct comparison.

Video Question Answering and Captioning

  • VIOLETv2 surpasses ALPRO by 2.4% accuracy on MSRVTT-QA and 8.4% accuracy on MSVD-QA using the same pre-training data.The comparison uses WebVid2.5M + CC3M and covers video question answering.Source: §6 · p. 7

Text-to-Video Retrieval

  • The retrieval comparison includes methods using significantly more pre-training data, which are visually distinguished in Table 9.Table 9 reports R1/5/10 for text-to-video retrieval tasks.Source: Table 9 · p. 8

Video Question Answering and Captioning

  • VIOLETv2 achieves comparable or better performance than models trained on substantially larger video-text datasets despite using less pre-training data.On MSRVTT captioning, it is only two points behind MV-GPT, which uses 53M video-text pairs versus 5M.Source: §6 · p. 8

Text-to-Video Retrieval

  • +10.9% R1 on DiDeMo and +6.1% R1 on LSMDC demonstrate VIOLETv2’s retrieval gains across both benchmarks.With WebVid2.5M + CC3M, VIOLETv2 improves all considered retrieval metrics on DiDeMo and LSMDC.Source: §6 · p. 8, Table 9 · p. 8

Direct Comparison to VIOLET

  • VIOLETv2 gains +3.4% on average across eight video QA datasets and +8.6% absolute R1 across three retrieval benchmarks over VIOLET.These gains hold even though VIOLET was pre-trained with about 37 times more data.Source: §6 · p. 8

7. Conclusion

The paper establishes MVM as a promising direction for VidL pre-training by systematically exploring diverse objectives and masking strategies, yielding VIOLETv2 with strong results across tasks and benchmarks.

  • VIOLETv2 achieves strong performance on three popular VidL tasks across 13 benchmarks after pre-training with MVM on 5M video/image-text data.The tasks include video question answering, video captioning, and text-to-video retrieval.Source: §7 · p. 8
  • The study explores MVM targets spanning low-level pixel information, high-level visual semantics, and extracted latent features in an end-to-end VidL transformer.Source: §7 · p. 8
  • Analyses of target combinations, SIF extractors, masking strategies, and masking ratios provide guidance for effective MVM design in large-scale VidL pre-training.Source: §7 · p. 8
  • The authors identify future directions including scaling MVM to larger datasets and exploring improved targets for video and VidL foundation models.Source: §7 · p. 8

A. Additional Results

Additional experiments show that MVM performance depends strongly on the target, masking strategy, initialization, prediction head, loss, and data modality, with SIF selected as the final target.

Fair Comparison to VIOLET [17]

  • +2.3% on TGIF-Frame and +13.3% AveR on DiDeMo-Retrieval distinguish the best SIF setting from a fair VIOLET reproduction.Both models use the same pre-training setting and corpus, demonstrating the importance of an appropriate MVM configuration.Source: A. Additional Results · p. 9, Table 10 · p. 9

MVM vs. Temporal self-supervised objectives

Initialization and Learning of Video Backbone

  • MVM improves downstream VidL performance across initialization settings, including randomly initialized video transformers and supervised or self-supervised video-backbone variants.Source: A. Additional Results · p. 9, Table 12 · p. 9

MVM Loss and Prediction Head

  • L1 MVM loss is more robust and performs better than L2 on video question answering and text-to-video retrieval, while a 2-layer MLP outperforms a single linear prediction head.The prediction-head comparison includes 31.3 versus 35.4 R1 on DiDeMo-Retrieval.Source: A. Additional Results · p. 9, Table 13 · p. 9, Table 14 · p. 9

Target Extractors and Combination

Additional SIF and Dataset Analyses

  • Swin-B SIF retains a 34.9 R1 retrieval advantage over DeiT, although improved CLIP training recipes reduce the gap and SIF remains strongest on additional datasets.The effects of MVM targets also vary with downstream video length and image-text versus video-text inputs.Source: A. Additional Results · p. 10, Table 17 · p. 10, Table 18 · p. 10

Optical Flow Analysis

B. Additional Pre-training Details

The additional pre-training details specify the VIOLETv2 objectives, MVM configuration, optimization setup, and compute used for training.

Video-Text Matching

Masked Language Modeling

Implementation Details

C. Experimental Setup of Downstream Tasks

The downstream evaluation covers three VidL tasks across 13 datasets using standard splits, task-specific metrics, sparse frame sampling, and AdamW fine-tuning.

Evaluation Tasks

Text-to-Video Retrieval

Video Question Answering and Captioning

Implementation

C.1. Text-To-Video Retrieval

Text-to-video retrieval trains on matched video-text pairs as positives and other within-batch pairings as negatives. The setup covers MSRVTT and DiDeMo, with retrieval-specific finetuning using large batches.

C.1. Text-To-Video Retrieval

  • Matched video-text pairs are positives, while all other pairwise combinations within the batch serve as negatives for retrieval training.A fully connected classifier operates on the [CLS] token representation.Source: §C.1 · p. 12
  • Retrieval finetuning uses 8 80GB A100 GPUs to support larger batch sizes, unlike the other finetuning experiments conducted on 32GB V100 GPUs.Source: §C.1 · p. 12
  • MSRVTT contains 10K YouTube videos and 200K annotations, using 9K training-plus-validation videos and a 1K-A test split for fair comparison.Training uses batch size 20 per GPU for 10 epochs.Source: §C.1 · p. 12
  • DiDeMo contains 10K Flickr videos annotated with 40K sentences.Source: §C.1 · p. 12

C.2. Video Question Answering

Video question answering is evaluated in both multiple-choice and open-ended settings. The tasks use masked language modeling, with answer-choice formulation for multiple-choice QA and free-form responses for open-ended QA.

C.2. Video Question Answering

  • Video QA is formulated as masked language modeling, following LAVENDER because of its reported superior performance.Source: §C.2 · p. 12
  • Multiple-choice QA concatenates the question and answer options with a [MASK] token, while TGIF-Frame is an open-ended task with free-form answers.TGIF-Action and TGIF-Transition use multiple-choice settings.Source: §C.2 · p. 12
  • MSRVTT-MC contains five captions per video with one positive match and can therefore be evaluated as video-to-text retrieval using the MSRVTT-Retr model.Source: §C.2 · p. 13

Related papers

Loading 2209.01540v5…