PaperlayerView exact revision on arXiv

Source-linked AI summary

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, Zicheng Liu

arXiv:2111.12681v2cs.CV

TL;DR

VIOLET addresses the disconnection between fixed video representations and downstream VidL data, including temporal information lost by simple frame aggregation. It uses an end-to-end video transformer with Masked Visual-token Modeling, and reports state-of-the-art or competitive performance across video-language tasks.

  • Problem

    Fixed video representations and simple aggregation of sparsely sampled frames can disconnect video modeling from downstream VidL data and lose temporal information.

  • Method

    VIOLET combines a Video Swin Transformer for explicit temporal modeling with MVM, which predicts discrete visual tokens for masked spatial-temporal video patches.

  • Results

    VIOLET achieves state-of-the-art results on 4 text-to-video retrieval tasks and 5 out of 8 video question answering tasks, with ablations supporting temporal modeling and MVM.

  • Takeaways & Limitations

    Explicit temporal modeling and MVM enhance VIOLET’s video-language representations and support video-language reasoning across different pre-training settings.

  • Takeaways & Limitations

    The authors identify extending VIOLET to full-length videos with densely sampled frames as future work, including for TGIF-Count.

Abstract

from arXiv · show

A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling. Specifically, the original video frame patches are "tokenized" into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks.

1. Introduction

VIOLET addresses disconnected video representations and lost temporal information by combining explicit video temporal modeling with Masked Visual-token Modeling. It achieves state-of-the-art results across video question answering and text-to-video retrieval tasks.

  • Motivation: Prior end-to-end VidL methods sparsely sample frames and use 2D CNN features with mean-pooling or concatenation, which can forfeit crucial temporal information.These approaches improve computational feasibility but simplify temporal aggregation.
  • Approach: VIOLET uses a Video Swin Transformer to explicitly model video temporal dynamics instead of mean-pooling or concatenating individual frame features.Its spatial-temporal self-attention supports variable sequence lengths and learning from videos or static images.
  • Motivation: Masked visual-input pre-training has been limited by imperfect patch labels, excessive feature dimensions, and unsatisfactory performance.Some recent VidL works discard these tasks because improvements are limited.
  • Approach: MVM masks video patches across spatial and temporal dimensions and predicts discrete visual tokens produced by a self-reconstructable Discrete VAE.The discrete prediction space avoids excessive feature dimensions and replaces distillation from a supervised visual backbone with self-reconstructed visual tokens.
  • Results: VIOLET achieves state-of-the-art results on 4 text-to-video retrieval tasks and 5 out of 8 video question answering tasks.Ablations demonstrate the necessity of temporal video modeling and MVM across different VidL pre-training settings.

2. Related Work

Related work established video-language pre-training and masked visual modeling, while VIOLET extends these directions with explicit temporal modeling and discrete visual-token prediction for video inputs.

  • Video-Language Understanding: Earlier VidL systems commonly relied on offline extracted video features, including pre-trained models built on fixed visual representations.VidL tasks include retrieval, question answering, moment retrieval, and captioning.
  • Video-Language Understanding: VIOLET contrasts with frame-wise image encoding followed by temporal mean-pooling or concatenation by using a video transformer, even with sparse frame sampling.The paper reports temporal modeling as necessary for general VidL understanding.
  • Masked Visual Modeling: Visual-text matching and masked language modeling are established vision-language pre-training tasks, whereas masked visual modeling has also been explored for visual inputs.Prior masked visual tasks include recovering masked regions or frames.
  • Masked Visual Modeling: Inspired by BEiT, VIOLET extends recovery of masked visual tokens from images to video inputs by adding a temporal dimension.VIOLET uses visual tokens as prediction targets for video-language modeling.
  • VIOLET Architecture: VIOLET combines Video Swin Transformer, Language Embedder, and Cross-modal Transformer components for end-to-end video-language processing.Discrete VAE visual tokens support MVM alongside visual-text matching and masked language modeling.

3. VIOLET

VIOLET is an end-to-end video-language transformer that combines explicit spatial-temporal video modeling with cross-modal language fusion and masked visual-token pre-training.

  • Cross-modal fusion: A Language Embedder and Cross-modal Transformer combine video features, word embeddings, positional information, and a [CLS] token for joint VidL learning.The [CLS] representation is used for pre-training and downstream fine-tuning.
  • Video Swin Transformer: VIOLET processes sparse-sampled video frames with a Video Swin Transformer that models spatial and temporal dimensions using 3D-shifted windows.The transformer preserves the input temporal dimension for Masked Visual-token Modeling.
  • Video Swin Transformer: Learnable spatial and temporal positional embeddings encode patch locations, frame ordering, and shared positions across patches.Each frame’s patches share a temporal embedding, while patches at the same spatial position share a spatial embedding.
  • Pre-training tasks: VIOLET jointly pre-trains with Masked Language Modeling, Masked Visual-token Modeling, and Visual-Text Matching.These tasks target language reasoning, video scene understanding, and video-text alignment, respectively.
  • Pre-training tasks: MVM masks video patches and predicts their discrete visual tokens using joint video-language features rather than distilled categories or features.Discrete tokens provide a finite prediction vocabulary and can be learned without human annotations or a supervised visual backbone.
  • Masking strategy: Blockwise Masking masks spatial-temporal blocks so neighboring patches cannot make masked-token recovery trivially easy.The strategy masks H′ × W′ patches across T′ consecutive frames and is paired with Attended Masking.

4. Experiments

VIOLET is evaluated across text-to-video retrieval and video question answering, with comparisons to prior methods and ablations of temporal modeling and MVM. Results show strong benchmark performance, explicit temporal modeling benefits, and MVM improves downstream VidL tasks.

  • Text-to-Video Retrieval: VIOLET achieves significant gains across all considered text-to-video retrieval datasets and remains competitive against methods using additional modalities.The comparisons include models using sound and speech in addition to video frames.
  • Text-to-Video Retrieval: +10% on R@1 on both MSRVTT and DiDeMo is achieved over ClipBERT, while gains over Frozen reach +2.0%, +1.6% and +1.1% on MSRVTT, DiDeMo and LSMDC, respectively.VIOLET uses 5 frames versus ClipBERT’s 16 frames in this comparison.
  • Video Question Answering: VIOLET surpasses ClipBERT by +9.7% on TGIF-Action, +7.9% on TGIF-Transition, +8.6% on TGIF-Frame, +3.7% on MSRVTT-MC and +6.5% on MSRVTT-QA.The authors attribute these gains to explicit temporal modeling and note that image-text pre-training alone may be insufficient for VidL modeling.
  • Temporal Video Modeling: VT consistently outperforms Mean and Concat across four datasets, while naive mean pooling loses temporal information and concatenation leaves temporal modeling to the cross-modal transformer.The analysis compares independently computed frame features, concatenated frame features, and Video Swin Transformer representations.
  • MVM Analysis: MVM produces the best performance on all analyzed tasks, and 21.6% MVM accuracy on 100% of YT-Temporal data corresponds to +2.3% on TGIF-Action and +2.5% R@5 on MSRVTT.MVM-only video pre-training improves all four datasets even without text, and better MVM performance accompanies better downstream performance.

5. Conclusion

VIOLET combines explicit temporal modeling with Masked Visual-token Modeling for end-to-end video-language learning. Experiments report strong performance and ablation evidence across retrieval and question-answering settings.

  • 5. Conclusion: VIOLET uses a Video Swin Transformer to explicitly model temporal dynamics for video-language learning.The architecture is designed to model spatial-temporal video dynamics rather than relying on simpler frame aggregation.
  • 5. Conclusion: Masked Visual-token Modeling trains VIOLET to reconstruct masked video patches using self-reconstructable discrete visual tokens.MVM applies a mask-the-predict procedure to support video scene understanding.
  • 5. Conclusion: Experiments show SOTA or competitive performance, while ablations demonstrate the necessity of temporal modeling and MVM's effectiveness over MRM/MFM.The findings hold across different video-language pre-training settings and downstream retrieval and question-answering tasks.

A. Experimental Setup of Downstream Tasks

The evaluation covers text-to-video retrieval and video question answering across 12 downstream datasets. Experiments use standard dataset splits, sparse five-frame sampling, and common fine-tuning settings.

  • A. Experimental Setup of Downstream Tasks: VIOLET is evaluated on text-to-video retrieval and video question answering across 12 downstream datasets.Retrieval uses MSRVTT, DiDeMo, YouCook2, and LSMDC; question answering includes multiple-choice and open-ended datasets.
  • A. Experimental Setup of Downstream Tasks: Text-to-video retrieval is measured with Recall at K, while video question answering includes multiple-choice and open-ended settings.The evaluation spans four retrieval datasets and several question-answering benchmarks.
  • A. Experimental Setup of Downstream Tasks: Unless otherwise stated, experiments sparsely sample T = 5 video frames at 224 resolution with patch size 32.VIOLET is fine-tuned with AdamW using an initial learning rate of 1.2e-5, betas (0.9, 0.98), and weight decay 1e-3.

A.1. Text-To-Video Retrieval

For text-to-video retrieval, VIOLET classifies matching and non-matching video-text pairs using the global [CLS] representation. Evaluation follows established dataset-specific protocols across four retrieval benchmarks.

  • A.1. Text-To-Video Retrieval: VIOLET treats corresponding video-text pairs as positives and other pairwise combinations as negatives for retrieval training.A fully connected layer over the global [CLS] representation performs binary classification.
  • A.1. Text-To-Video Retrieval: Pre-trained FCVTM supports zero-shot text-to-video retrieval and initializes FCT2V for downstream fine-tuning.The same matching formulation is adapted for task-specific retrieval training.
  • A.1. Text-To-Video Retrieval: MSRVTT uses 9K training-plus-validation videos and a 1K-A testing split for fair comparison, with 20 training epochs.The dataset contains 10K YouTube videos and 200K human annotations.
  • A.1. Text-To-Video Retrieval: DiDeMo uses paragraph-to-video retrieval by concatenating sentences from each video, while YouCook2 is evaluated on all validation clips.DiDeMo contains 10K videos with 40K sentences; YouCook2 contains 14K clips from 2K cooking videos.

A.2. Video Question Answering

Video question answering is evaluated in multiple-choice and open-ended formats using classifiers over joint video-language representations. The setup covers spatial-temporal reasoning and dataset-specific answer vocabularies.

  • A.2. Video Question Answering: Multiple-choice QA concatenates each question with an answer option, then predicts option confidence from the joint representation.Cross-entropy trains a classifier over all answer options for each video-question pair.
  • A.2. Video Question Answering: Open-ended QA is converted into classification over a finite answer vocabulary built from common training-split answers.A fully connected layer over the joint representation predicts the answer class.
  • A.2. Video Question Answering: TGIF-Action, TGIF-Transition, and TGIF-Frame test spatial-temporal reasoning about repeated actions, state transitions, and specific frames.These tasks use GIF videos from TGIF-QA.
  • A.2. Video Question Answering: The setup uses 1,540 answer candidates for TGIF-Frame, 1,500 for MSRVTT-QA, and 1,000 for MSVD-QA.MSRVTT-MC and MSRVTT-QA use 20 training epochs, while MSVD-QA uses 40 epochs.
  • A.2. Video Question Answering: LSMDC-MC selects the only positive caption among five candidates, while LSMDC-FiB recovers a word replaced by [BLANK].LSMDC-FiB is treated as an open-ended video question-answering task.

B. Impact of Masking Strategy

VIOLET compares four masking strategies and finds that structured masking improves over random masking, with BM+AM performing best. BM reduces spurious reconstruction success, while AM emphasizes important video-text elements.

  • Random masking yields the least performance improvement on both video question answering and retrieval, despite improving over the non-pretrained baseline.
  • BM or AM alone produces more significant performance gains than random masking across video question answering and retrieval tasks.
  • BM+AM achieves the best performance among the four masking strategies, including 26.0% R@1 on DiDeMo-Retrieval for AM.
  • BM reduces spurious MVM success from visually similar neighboring patches, whereas AM weights important video-text elements according to cross-modal attention.

C. Extending VIOLETto Image Question Answering Task

VIOLET extends to image question answering on VCR and performs comparably to or better than MERLOT under matched pre-training conditions. Higher input resolution and longer pre-training are associated with improved downstream performance.

  • VIOLET is evaluated on VCR image question answering using colored highlights around referenced entities and concatenated question-answer choices.
  • 74.9% versus 75.1% shows comparable VIOLET and MERLOT performance at 224x224 input resolution.
  • +1.2% is VIOLET’s absolute gain over MERLOT when VIOLET uses 384x384 input resolution.
  • VIOLET reaches competitive video-language downstream performance with 2K GPU hours and 224x224 inputs, compared with MERLOT’s 30K TPU hours and 384x704 inputs.
  • Higher input resolution and longer pre-training are reported as opportunities for further performance improvement when computational resources permit.

D. Qualitative Examples of Zero-shot Text-to-Video Retrieval

Qualitative zero-shot retrieval examples show VIOLET matching text with relevant videos across general, long-form, movie, and cooking domains. The examples attribute this capability to pre-training on large-scale visual-text data.

  • Large-scale visual-text pre-training enables VIOLET to learn cross-modal alignment for zero-shot text-to-video retrieval.
  • On MSRVTT, VIOLET retrieves relevant clips for the uncommon phrase “grand theft auto 5.”
  • On DiDeMo, VIOLET retrieves relevant videos for paragraph-length queries containing multiple textual cues.
  • LSMDC and YouCook2 examples indicate generalizability to movie and cooking videos.

E. Limitation and Broader Impact

The paper frames VIOLET as useful for video-language reasoning while identifying full-length video modeling and audio integration as future directions. It also notes sensitivity to pre-training-data distribution and describes diverse data sources and broad evaluation.

  • VIOLET targets video question answering and text-to-video retrieval, with potential future applications in video captioning and video grounded dialogue.
  • Extending VIOLET to densely sampled full-length videos and incorporating audio are identified as promising future work.
  • The self-supervised method is sensitive to the distribution of pre-training data.
  • The authors use diverse video and textual sources and evaluate across 12 VidL tasks to mitigate bias in learned cross-modal representations.
Loading 2111.12681v2…