PaperlayerView exact revision on arXiv

Source-linked AI summary

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, Zicheng Liu

arXiv:2111.12681v2cs.CV

TL;DR

VIOLET addresses the disconnection between fixed video representations and downstream VidL data, including temporal information lost by simple frame aggregation. It uses an end-to-end video transformer with Masked Visual-token Modeling, and reports state-of-the-art or competitive performance across video-language tasks.

  • Problem

    Fixed video representations and simple aggregation of sparsely sampled frames can disconnect video modeling from downstream VidL data and lose temporal information.

    Source: §1 · p. 1

  • Method

    VIOLET combines a Video Swin Transformer for explicit temporal modeling with MVM, which predicts discrete visual tokens for masked spatial-temporal video patches.

    Source: §1 · p. 2

  • Results

    VIOLET achieves state-of-the-art results on 4 text-to-video retrieval tasks and 5 out of 8 video question answering tasks, with ablations supporting temporal modeling and MVM.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Explicit temporal modeling and MVM enhance VIOLET’s video-language representations and support video-language reasoning across different pre-training settings.

    Source: §1 · p. 2, §5 · p. 9

    The authors identify extending VIOLET to full-length videos with densely sampled frames as future work, including for TGIF-Count.

    Source: E. Limitation and Broader Impact · p. 11

Abstract

from arXiv · show

A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling. Specifically, the original video frame patches are "tokenized" into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks.

1. Introduction

VIOLET addresses disconnected video representations and lost temporal information by combining explicit video temporal modeling with Masked Visual-token Modeling. It achieves state-of-the-art results across video question answering and text-to-video retrieval tasks.

Motivation

  • Prior end-to-end VidL methods sparsely sample frames and use 2D CNN features with mean-pooling or concatenation, which can forfeit crucial temporal information.These approaches improve computational feasibility but simplify temporal aggregation.Source: §1 · p. 1

Approach

  • VIOLET uses a Video Swin Transformer to explicitly model video temporal dynamics instead of mean-pooling or concatenating individual frame features.Its spatial-temporal self-attention supports variable sequence lengths and learning from videos or static images.Source: §1 · p. 2

Motivation

  • Masked visual-input pre-training has been limited by imperfect patch labels, excessive feature dimensions, and unsatisfactory performance.Some recent VidL works discard these tasks because improvements are limited.Source: §1 · p. 2

Approach

  • MVM masks video patches across spatial and temporal dimensions and predicts discrete visual tokens produced by a self-reconstructable Discrete VAE.The discrete prediction space avoids excessive feature dimensions and replaces distillation from a supervised visual backbone with self-reconstructed visual tokens.Source: §1 · p. 2

Results

  • VIOLET achieves state-of-the-art results on 4 text-to-video retrieval tasks and 5 out of 8 video question answering tasks.Ablations demonstrate the necessity of temporal video modeling and MVM across different VidL pre-training settings.Source: §1 · p. 2

2. Related Work

Related work established video-language pre-training and masked visual modeling, while VIOLET extends these directions with explicit temporal modeling and discrete visual-token prediction for video inputs.

Video-Language Understanding

  • Earlier VidL systems commonly relied on offline extracted video features, including pre-trained models built on fixed visual representations.VidL tasks include retrieval, question answering, moment retrieval, and captioning.Source: §2 · p. 2
  • VIOLET contrasts with frame-wise image encoding followed by temporal mean-pooling or concatenation by using a video transformer, even with sparse frame sampling.The paper reports temporal modeling as necessary for general VidL understanding.Source: §2 · p. 2

Masked Visual Modeling

  • Visual-text matching and masked language modeling are established vision-language pre-training tasks, whereas masked visual modeling has also been explored for visual inputs.Prior masked visual tasks include recovering masked regions or frames.Source: §2 · p. 2
  • Inspired by BEiT, VIOLET extends recovery of masked visual tokens from images to video inputs by adding a temporal dimension.VIOLET uses visual tokens as prediction targets for video-language modeling.Source: §2 · pp. 2–3

VIOLET Architecture

  • VIOLET combines Video Swin Transformer, Language Embedder, and Cross-modal Transformer components for end-to-end video-language processing.Discrete VAE visual tokens support MVM alongside visual-text matching and masked language modeling.Source: Figure 2 · p. 3

3. VIOLET

VIOLET is an end-to-end video-language transformer that combines explicit spatial-temporal video modeling with cross-modal language fusion and masked visual-token pre-training.

Cross-modal fusion

  • A Language Embedder and Cross-modal Transformer combine video features, word embeddings, positional information, and a [CLS] token for joint VidL learning.The [CLS] representation is used for pre-training and downstream fine-tuning.Source: §3.1 · pp. 3–4

Video Swin Transformer

  • VIOLET processes sparse-sampled video frames with a Video Swin Transformer that models spatial and temporal dimensions using 3D-shifted windows.The transformer preserves the input temporal dimension for Masked Visual-token Modeling.Source: §3.1 · p. 3
  • Learnable spatial and temporal positional embeddings encode patch locations, frame ordering, and shared positions across patches.Each frame’s patches share a temporal embedding, while patches at the same spatial position share a spatial embedding.Source: §3.1 · p. 3

Pre-training tasks

  • VIOLET jointly pre-trains with Masked Language Modeling, Masked Visual-token Modeling, and Visual-Text Matching.These tasks target language reasoning, video scene understanding, and video-text alignment, respectively.Source: §3.2 · p. 4
  • MVM masks video patches and predicts their discrete visual tokens using joint video-language features rather than distilled categories or features.Discrete tokens provide a finite prediction vocabulary and can be learned without human annotations or a supervised visual backbone.Source: §3.2 · p. 4

Masking strategy

  • Blockwise Masking masks spatial-temporal blocks so neighboring patches cannot make masked-token recovery trivially easy.The strategy masks H′ × W′ patches across T′ consecutive frames and is paired with Attended Masking.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4. Experiments

VIOLET is evaluated across text-to-video retrieval and video question answering, with comparisons to prior methods and ablations of temporal modeling and MVM. Results show strong benchmark performance, explicit temporal modeling benefits, and MVM improves downstream VidL tasks.

Text-to-Video Retrieval

  • VIOLET achieves significant gains across all considered text-to-video retrieval datasets and remains competitive against methods using additional modalities.The comparisons include models using sound and speech in addition to video frames.Source: §4.2 · p. 6, Table 1 · p. 6
  • +10% on R@1 on both MSRVTT and DiDeMo is achieved over ClipBERT, while gains over Frozen reach +2.0%, +1.6% and +1.1% on MSRVTT, DiDeMo and LSMDC, respectively.VIOLET uses 5 frames versus ClipBERT’s 16 frames in this comparison.Source: §4.2 · p. 6

Video Question Answering

  • VIOLET surpasses ClipBERT by +9.7% on TGIF-Action, +7.9% on TGIF-Transition, +8.6% on TGIF-Frame, +3.7% on MSRVTT-MC and +6.5% on MSRVTT-QA.The authors attribute these gains to explicit temporal modeling and note that image-text pre-training alone may be insufficient for VidL modeling.Source: §4.2 · p. 6

Temporal Video Modeling

  • VT consistently outperforms Mean and Concat across four datasets, while naive mean pooling loses temporal information and concatenation leaves temporal modeling to the cross-modal transformer.The analysis compares independently computed frame features, concatenated frame features, and Video Swin Transformer representations.Source: §4.3 · p. 7

MVM Analysis

  • MVM produces the best performance on all analyzed tasks, and 21.6% MVM accuracy on 100% of YT-Temporal data corresponds to +2.3% on TGIF-Action and +2.5% R@5 on MSRVTT.MVM-only video pre-training improves all four datasets even without text, and better MVM performance accompanies better downstream performance.Source: §4.3 · p. 8

5. Conclusion

VIOLET combines explicit temporal modeling with Masked Visual-token Modeling for end-to-end video-language learning. Experiments report strong performance and ablation evidence across retrieval and question-answering settings.

5. Conclusion

  • VIOLET uses a Video Swin Transformer to explicitly model temporal dynamics for video-language learning.The architecture is designed to model spatial-temporal video dynamics rather than relying on simpler frame aggregation.Source: §5 · pp. 8–9
  • Masked Visual-token Modeling trains VIOLET to reconstruct masked video patches using self-reconstructable discrete visual tokens.MVM applies a mask-the-predict procedure to support video scene understanding.Source: §5 · p. 9
  • Experiments show SOTA or competitive performance, while ablations demonstrate the necessity of temporal modeling and MVM's effectiveness over MRM/MFM.The findings hold across different video-language pre-training settings and downstream retrieval and question-answering tasks.Source: §5 · p. 9

A. Experimental Setup of Downstream Tasks

The evaluation covers text-to-video retrieval and video question answering across 12 downstream datasets. Experiments use standard dataset splits, sparse five-frame sampling, and common fine-tuning settings.

A. Experimental Setup of Downstream Tasks

  • VIOLET is evaluated on text-to-video retrieval and video question answering across 12 downstream datasets.Retrieval uses MSRVTT, DiDeMo, YouCook2, and LSMDC; question answering includes multiple-choice and open-ended datasets.Source: A. Experimental Setup of Downstream… · p. 9
  • Text-to-video retrieval is measured with Recall at K, while video question answering includes multiple-choice and open-ended settings.The evaluation spans four retrieval datasets and several question-answering benchmarks.Source: A. Experimental Setup of Downstream… · p. 9
  • Unless otherwise stated, experiments sparsely sample T = 5 video frames at 224 resolution with patch size 32.VIOLET is fine-tuned with AdamW using an initial learning rate of 1.2e-5, betas (0.9, 0.98), and weight decay 1e-3.Source: A. Experimental Setup of Downstream… · p. 9

A.1. Text-To-Video Retrieval

For text-to-video retrieval, VIOLET classifies matching and non-matching video-text pairs using the global [CLS] representation. Evaluation follows established dataset-specific protocols across four retrieval benchmarks.

A.1. Text-To-Video Retrieval

  • VIOLET treats corresponding video-text pairs as positives and other pairwise combinations as negatives for retrieval training.A fully connected layer over the global [CLS] representation performs binary classification.Source: §A.1 · p. 9
  • Pre-trained FCVTM supports zero-shot text-to-video retrieval and initializes FCT2V for downstream fine-tuning.The same matching formulation is adapted for task-specific retrieval training.Source: §A.1 · p. 9
  • MSRVTT uses 9K training-plus-validation videos and a 1K-A testing split for fair comparison, with 20 training epochs.The dataset contains 10K YouTube videos and 200K human annotations.Source: §A.1 · p. 9
  • DiDeMo uses paragraph-to-video retrieval by concatenating sentences from each video, while YouCook2 is evaluated on all validation clips.DiDeMo contains 10K videos with 40K sentences; YouCook2 contains 14K clips from 2K cooking videos.Source: §A.1 · p. 9

A.2. Video Question Answering

Video question answering is evaluated in multiple-choice and open-ended formats using classifiers over joint video-language representations. The setup covers spatial-temporal reasoning and dataset-specific answer vocabularies.

A.2. Video Question Answering

  • Multiple-choice QA concatenates each question with an answer option, then predicts option confidence from the joint representation.Cross-entropy trains a classifier over all answer options for each video-question pair.Source: §A.2 · p. 9
  • Open-ended QA is converted into classification over a finite answer vocabulary built from common training-split answers.A fully connected layer over the joint representation predicts the answer class.Source: §A.2 · p. 9
  • TGIF-Action, TGIF-Transition, and TGIF-Frame test spatial-temporal reasoning about repeated actions, state transitions, and specific frames.These tasks use GIF videos from TGIF-QA.Source: §A.2 · p. 9
  • The setup uses 1,540 answer candidates for TGIF-Frame, 1,500 for MSRVTT-QA, and 1,000 for MSVD-QA.MSRVTT-MC and MSRVTT-QA use 20 training epochs, while MSVD-QA uses 40 epochs.Source: §A.2 · p. 10
  • LSMDC-MC selects the only positive caption among five candidates, while LSMDC-FiB recovers a word replaced by [BLANK].LSMDC-FiB is treated as an open-ended video question-answering task.Source: §A.2 · p. 10

B. Impact of Masking Strategy

VIOLET compares four masking strategies and finds that structured masking improves over random masking, with BM+AM performing best. BM reduces spurious reconstruction success, while AM emphasizes important video-text elements.

C. Extending VIOLETto Image Question Answering Task

VIOLET extends to image question answering on VCR and performs comparably to or better than MERLOT under matched pre-training conditions. Higher input resolution and longer pre-training are associated with improved downstream performance.

D. Qualitative Examples of Zero-shot Text-to-Video Retrieval

Qualitative zero-shot retrieval examples show VIOLET matching text with relevant videos across general, long-form, movie, and cooking domains. The examples attribute this capability to pre-training on large-scale visual-text data.

E. Limitation and Broader Impact

The paper frames VIOLET as useful for video-language reasoning while identifying full-length video modeling and audio integration as future directions. It also notes sensitivity to pre-training-data distribution and describes diverse data sources and broad evaluation.

Related papers

Loading 2111.12681v2…