PaperlayerView exact revision on arXiv

Source-linked AI summary

All in One: Exploring Unified Video-Language Pre-training

Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, Mike Zheng Shou

arXiv:2203.07303v1cs.CV

TL;DR

Mainstream video-language pre-training relies on increasingly heavy encoders and fusion networks, motivating a more efficient unified architecture. All-in-one Transformer learns from raw video and text with parameter-free temporal token rolling, and achieves strong results across downstream tasks while retaining efficient retrieval and a bounded temporal-modeling regime.

  • Problem

    Mainstream VLP methods increase parameters by using expensive unimodal encoders or heavier fusion networks, reducing downstream computational efficiency.

  • Method

    All-in-one Transformer jointly processes raw video and text in one unified backbone, using parameter-free temporal token rolling to model video dynamics.

  • Results

    The model demonstrates superiority across four downstream video-text tasks on nine datasets, with efficient retrieval enabled by separate video and text encoding.

  • Takeaways & Limitations

    A unified backbone can support multimodal representation learning and efficient unimodal retrieval without extra unimodal encoders.

  • Takeaways & Limitations

    Excessive rolling ratios lead to unstable training.

Abstract

from arXiv · show

Mainstream Video-Language Pre-training models \cite{actbert,clipbert,violet} consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heavier unimodal encoders or multimodal fusion Transformers, resulting in increased parameters with lower efficiency in downstream tasks. In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture. We argue that the unique temporal information of video data turns out to be a key barrier hindering the design of a modality-agnostic Transformer. To overcome the challenge, we introduce a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner. The careful design enables the representation learning of both video-text multimodal inputs and unimodal inputs using a unified backbone model. Our pre-trained all-in-one Transformer is transferred to various downstream video-text tasks after fine-tuning, including text-video retrieval, video-question answering, multiple choice and visual commonsense reasoning. State-of-the-art performances with the minimal model FLOPs on nine datasets demonstrate the superiority of our method compared to the competitive counterparts. The code and pretrained model have been released in https://github.com/showlab/all-in-one.

1 Introduction

Mainstream video-language pre-training improves performance with increasingly heavy unimodal encoders or fusion networks, but at higher computational cost. All-in-one Transformer unifies raw video and text processing, using parameter-free temporal token rolling to support multimodal and unimodal representations.

  • Motivation: Mainstream VLP methods use expensive unimodal encoders or heavier fusion networks, increasing parameters and reducing downstream computational efficiency.
  • All-in-one Transformer: All-in-one Transformer learns joint video-language representations end-to-end from raw visual and textual signals in one unified backbone.
  • Temporal Modeling: Temporal token rolling cyclically shifts a proportion of visual tokens across sparsely sampled frames to capture temporal representations without extra parameters or increased time complexity.
  • Pre-training and Transfer: The model is pre-trained with video-text matching and masked language modeling, then fine-tuned for downstream video-text tasks.
  • Pre-training and Transfer: All-in-one can encode joint multimodal inputs or separate video and text inputs, enabling a dual-stream mode for efficient retrieval.

2 Related Work

Prior work explores video-language pre-training, temporal modeling, and shared architectures for multimodal data. All-in-one combines these directions through parameter-free temporal token rolling and a unified encoder that also supports efficient unimodal retrieval.

  • Video-Language Pre-training: Video-language pre-training commonly pre-trains on large-scale video-text pairs and transfers representations to retrieval, question answering, and visual storytelling.
  • Temporal Modeling in Video Understanding: Temporal modeling methods include sparse sampling, 3D operations, temporal attention, and shifting channels across frames.
  • Temporal Modeling in Video Understanding: All-in-one uses parameter-free temporal token rolling on sparsely sampled frames instead of heavy 3D-type operations.
  • Unified Architecture Design for Multimodal Data: Shared-encoder architectures process unimodal or multimodal inputs across domains including video, audio, text, images, and 3D data.
  • Unified Architecture Design for Multimodal Data: All-in-one applies cross-modal contrastive learning while supporting dual-stream unimodal retrieval with a shared video-language network.

3 Method

All-in-one uses a unified Transformer to process raw video and text, adding temporal token rolling before self-attention to capture video dynamics efficiently. Video and text are projected into a shared latent space, jointly processed, and trained with matching, masked-language, and contrastive objectives.

  • 3 Method: All-in-one learns joint video-text representations directly from raw inputs using a succinct unified architecture with parameter-free temporal modeling.
  • 3.1 Unified Video-language Transformer: Video frames are sparsely sampled, converted into patch tokens, and projected alongside text tokens into the Transformer's constant latent dimension D.
  • 3.1 Unified Video-language Transformer: Learnable spatio-temporal position embeddings and modality-type embeddings preserve positional and modality information for video and text tokens.
  • 3.1 Unified Video-language Transformer: Each Transformer block applies temporal token rolling before multi-head self-attention and an MLP, with self-attention and MLP weights initialized from ViT or DeiT.
  • 3.2 Temporal Token Rolling: Token rolling reduces self-attention computation to around one third of flattening while modeling cross-modality and inter-frame correspondence.
  • 3.2 Temporal Token Rolling: Deeper layers gradually model longer text-video dependencies, while rolling shifts attention away from centric visual tokens toward rolled tokens.
  • 3.3 Training Objectives: Pre-training uses video-text matching and masked language modeling, with video-text contrastive loss added during retrieval fine-tuning.

4 Experiments

The experiments evaluate All-in-one across four downstream video-language tasks and nine datasets, including retrieval, question answering, multiple choice, and visual commonsense reasoning. Results and ablations examine its transfer performance, temporal token rolling, initialization, fine-tuning objectives, and internal representations.

  • Experimental Setup: All-in-one is evaluated on text-to-video retrieval, video question answering, multiple choice, and visual commonsense reasoning across 9 datasets.The study uses WebVid-2.5M, HowTo100M, and YT-Temporal 180M for pre-training and reports fine-tuning, zero-shot, and ablation analyses.
  • Video-question Answering: With only 1 frame, All-in-one-B outperforms VIOLET by about 5.8% on the TGIF-QA Action subset.More frames benefit FrameQA but not the Action and Transition tasks; All-in-one also performs similarly or better than Just-Ask on MSVD-QA.
  • Retrieval Tasks: All-in-one achieves significant gains over existing methods on MSRVTT retrieval and a 2.1% relative R@1 improvement over OA-Trans.The comparison uses one cross-modality encoder, while OA-Trans uses additional offline object features and focuses only on retrieval.
  • Multiple-choice: 3.2% accuracy improvement over ClipBERT is reported on the MSRVTT multiple-choice test task.Zero-shot accuracy is already close to JSFusion with only three input frames.
  • Visual Commonsense Reasoning: All-in-one outperforms MERLOT on VCR at the same base scale using different data sources.The experiment tests transfer of video-level understanding to single-image visual reasoning and adds color masks for entity identity information.
  • Ablation Studies: Token rolling improves performance by 5.69% over temporal averaging, while excessive rolling ratios lead to unstable training.A separate comparison finds Token Rolling slightly better than Flatten with one-third of Flatten's computation complexity.

5 Conclusions

All-in-one Transformer is a unified end-to-end video-language architecture that uses raw video and text inputs with one fusion network. The authors identify fine-grained word-region alignment and temporal modeling as remaining challenges.

  • All-in-one Transformer presents a unified end-to-end video-language architecture with raw video and text as inputs.
  • Using only one fusion network, All-in-one competes with models that use additional heavy video embedding networks.
  • Fine-grained word-region alignment remains a challenge for unified video-language interaction.
  • Temporal modeling is not fully explored, and the authors encourage future work on single-modality tasks.

A Model Scalability

Model scalability is evaluated across three assessment settings and ten datasets while varying model size from 13M to 320M parameters. Larger models generally help zero-shot and linear-probe performance, but can overfit small fine-tuning datasets.

  • 13M to 320M parameters are evaluated across Fine-tune, Zero-shot, and Linear Probe settings on 10 datasets.
  • Larger models generally produce better results for Zero-shot and Linear Probe tasks.
  • All-in-one-L can perform worse than All-in-one-B on some fine-tuning benchmarks because small datasets cause severe overfitting.MSRVTT-QA contains 10K video-text pairs, while TVQA and TGIF-QA are ten times larger and retain better performance with All-in-one-L.
  • All-in-one-B is presented as a parameter-performance trade-off that is a better choice in most cases.

B Ablation Study

The ablation study compares alternative temporal modeling strategies with the parameter-free token rolling operation. TimeSformer and Decouple Attn improve some pre-training behavior but cost substantially more and perform worse on downstream zero-shot tasks.

  • Evaluation: The study reports top-1 classification accuracy for MLM and VTM as reference measures for the pre-training objectives.
  • Temporal modeling: TimeSformer adds divided spatial and temporal attention, while Decouple Attn separates modality self-attention before cross-modality attention.
  • Temporal modeling: TimeSformer and Decouple Attn are particularly good at Video-text Matching but not at Masked Language Modeling.
  • Efficiency and downstream performance: TimeSformer and Decouple Attn cost about 2-3 times more than All-in-one-B and show worse downstream zero-shot results.

B.2 Do we need to sample more frames?

The study varies sampled video frames from 1 to 16 to examine representation quality and computation trade-offs. More frames generally improve results, but three frames are already close to the best performance and are selected as the default.

  • Pre-training and fine-tuning experiments vary the sampled training frames to assess their relationship with learned representation quality.
  • 1 to 16 frames are evaluated using pre-training VTM and MLM accuracy plus downstream zero-shot multiple-choice accuracy on LSMDC and MSRVTT.
  • More frames generally lead to better results, while All-in-one is already close to its best results with 3 frames.
  • Three frames are used as the default to balance computation cost and performance.
  • Input embeddings: Spatio-temporal position embedding helps VTM, modality type embedding helps MLM, and their combination improves both downstream zero-shot multiple-choice performance and parameter efficiency.

C In-depth Analysis of Token Rolling

The analysis examines where and how temporal token rolling should be applied in All-in-one-B. More rolling layers and block-based selection improve representation, while 25% token selection is adopted by default.

  • Layer placement: Adding temporal token rolling from earlier positions, including the beginning, 3rd, or 6th block, enables comparison of layer placement in All-in-one-B.All-in-one-B contains 12 self-attention blocks, and the reported results examine these three starting positions.
  • Layer placement: More temporal token rolling layers leads to better representation.
  • Token selection: The sampling ablation compares random selection, layer-varying selection, and selecting a contiguous block containing 25% of tokens.
  • Evaluation context: The ablation is reported as a zero-shot multiple-choice study, while the related linear-probe evaluation concerns action-recognition benchmarks.
  • Token selection: Selecting a block of 25% of tokens gives the best result and is used as the default strategy.The authors attribute this preference to preserving position-sensitive information in the multiple perceptron.

D Transferability Evaluation

The transferability evaluation applies the unified representation to linear-probe action recognition and an egocentric multiple-choice benchmark. Results indicate sensitivity to frame count and competitive performance despite a substantial domain gap.

  • Action recognition: The linear probe freezes the unified model and learns linear layers from the final-layer CLS token on K400 and HMDB51.The evaluation compares the base model with Time Average and Frozen.
  • Action recognition: With the same 8-frame input, All-in-one-B clearly outperforms Frozen, especially on the large-scale K400 dataset.The number of frames has a large impact on the task.
  • Egocentric video: Ego-4d is an egocentric dataset with a large domain gap from the paper’s third-view YouTube pre-training videos.
  • Egocentric video: On Ego-4d’s five-choice task, All-in-one outperforms Frozen clearly while using half the parameters, although its zero-shot accuracy is lower than on other multiple-choice benchmarks.

E Complexity Analysis of Retrieval

The retrieval analysis reports search latency components and qualitative masked-word prediction examples. On HowTo100M, total search time is reported as 155.3 ms for text-to-video and 176.41 ms for video-to-text retrieval.

  • Benchmark context: Table 13 is identified as the multiple-choice result on the first-view Ego-4d benchmark.
  • Retrieval latency: 155.3 ms is the reported total HowTo100M search time for text-to-video retrieval.The total is given as 12.05 + 143.25 ms.
  • Retrieval latency: 176.41 ms is the reported total HowTo100M search time for video-to-text retrieval.The total is given as 33.16 + 143.25 ms.
  • Retrieval latency: Table 14 reports running-time analysis for retrieval and inference, with values averaged over 1000 runs.
  • Qualitative analysis: Qualitative examples include masked-word prediction on outdoor scenes and objects, alongside cloze evaluation on outdoor-scene and animal examples.
Loading 2203.07303v1…