PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu

arXiv:2210.11929v1cs.CVcs.CL

TL;DR

Video-language pre-training and detector-based approaches are computationally expensive, while image-language models need temporal modeling for video tasks. LiteVL adapts BLIP with temporal attention and text-conditioned pooling, and reports that it outperforms prior state-of-the-art methods without video-language pre-training.

  • Problem

    Video-language pre-training and off-the-shelf detectors are computationally expensive and inefficient, while image-language models lack the temporal modeling needed for video tasks.

    Source: §1 · p. 1

  • Method

    LiteVL adapts pre-trained BLIP with temporal attention modules using dynamic scaling and non-parametric pooling of video features conditioned on text.

    Source: §1 · p. 1, §3.3 · p. 4

  • Results

    LiteVL outperforms previous state-of-the-art methods trained with much more data; it reaches 50.8% R1 on MSRVTT-9k retrieval and 42.9% accuracy on MSRVTT-QA.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    LiteVL achieves strong video-language results by adapting image-text alignment without heavy video-language pre-training or object detectors.

    Source: §6 · p. 9

    VideoQA is treated as classification, which may limit the model to a fixed number of answer candidates.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive due to the requirement of millions of video-text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text. Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any video-language pre-training.

1 Introduction

Video-language modeling seeks a shared semantic space for video and text, but common pre-training and detector-based approaches are computationally expensive and inefficient. LiteVL adapts a pre-trained image-language model with temporal and text-conditioned feature enhancements, and reports strong retrieval and question-answering results without video-language pre-training.

1 Introduction

  • Video-language modeling learns a shared semantic space for videos and texts to support tasks such as text-video retrieval and video question answering.Prior approaches commonly pre-train on large video-text collections or use object detectors to extract fine-grained visual features.Source: §1 · p. 1
  • Video-text pre-training and off-the-shelf detectors are computationally expensive and inefficient, and inaccurate detections on limited categories may hurt performance.Source: §1 · p. 1
  • LiteVL initializes from BLIP and adds temporal attention layers with learnable task-adjustable scalings, alongside text-conditioned non-parametric pooling of fine-grained spatial-temporal features.The design aims to retain image-text alignment while modeling temporal dependencies.Source: §1 · p. 1
  • LiteVL beats previous state-of-the-art methods by a clear margin without video-language pre-training or detectors, reaching 50.8% R1 on MSRVTT-9k retrieval and 42.9% accuracy on MSRVTT-QA.The paper also reports that visualizations show fine-grained video-text alignment.Source: §1 · pp. 1–2

2 Related Work

Related work describes image Transformers adapted for video through spatial-temporal attention, and video-language models that either pre-train on video-text pairs or fine-tune image-text models on downstream tasks.

2.1 Vision Transformers

  • Video Transformers such as TimeSformer and ViViT initialize from trained image Transformers and add spatial-temporal attention mechanisms to model intertwined dependencies efficiently.Source: §2.1 · p. 2

2.2 Video-Language Modeling

  • Video-language models split between video-text-pair pre-training with spatial-temporal encoders and downstream fine-tuning of image-text models that treat videos as image collections.The pre-training branch uses datasets including WebVid2M and HowTo100M; the other branch includes ClipBERT, BLIP, and X-Pool.Source: §2.2 · p. 2

3 Method

LiteVL adapts BLIP for video-language tasks without video-text pre-training, adding temporal modeling and text-conditioned pooling. Its training objectives support both retrieval and video question answering.

3.1 Model Architecture

  • LiteVL starts from BLIP image-language encoders, adding divided temporal-then-spatial attention to vision and cross-attention to the video-grounded text encoder.The video-grounded text encoder shares parameters with the unimodal text encoder.Source: §3.1 · pp. 2–3

3.2 Dynamic Temporal Scaling

  • Learnable, task-specific scalings adjust temporal-attention strength; values range from 0 to 2, with 0 removing temporal modeling and initialization at 1 matching TimeSformer.The scaled temporal-attention output is applied before the residual connection.Source: §3.2 · p. 3

3.3 Text-dependent Pooling

  • Text-dependent pooling uses text-feature similarities to reweight pooled frame and spatial features, then concatenates them with the original video features.The resulting fine-grained features are conditioned on the textual query.Source: §3.3 · p. 4

3.4 Training Objectives

  • Retrieval combines video-text contrastive and matching losses, while VideoQA classifies answers from multimodal representations using a K-class objective.For VideoQA, a two-layer MLP classification head operates on the multimodal [Encode] token.Source: §3.4 · pp. 4–5

4 Experiments

LiteVL was initialized from BLIP variants and fine-tuned for text-video retrieval and VideoQA, with task-specific frame sampling. Across the experiments, it outperformed prior methods, while ablations and visualizations examined its temporal scaling and text-dependent pooling.

4.1 Text-Video Retrieval

  • LiteVL surpasses prior methods on MSRVTT-7k, MSRVTT-9k, and DiDeMo retrieval, including video-text-pretrained and image-language-based approaches.Against the best BLIP (14M) variant, LiteVLS improves R1 by 2.0 points on MSRVTT-7k, 2.6 on MSRVTT-9k, and 2.4 on DiDeMo.Source: Table 2 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.2 Video Question Answering

  • LiteVL outperforms all compared methods on MSRVTT-QA and MSVD-QA, including methods using large-scale video-text pre-training.LiteVLL, despite starting from BLIP trained on more image-text pairs, performs worse than LiteVLS on both VideoQA tasks; the authors speculate that noisy added data may bias the model.Source: §4.2 · p. 7, Table 5 · p. 7

4.3.1 Dynamic Temporal Scaling

  • Task-learned dynamic temporal scaling performs better than constant scaling at 0 or 1, with the latter two corresponding to ViT and TimeSformer encoders.Using temporal attention improves on ViT, and learned scaling further improves performance.Source: §4.3.1 · p. 7, Table 6 · p. 8
  • Across datasets, average learned scaling is below 0 in the first layer and trends upward with depth, which the authors associate with a shift from frame content to temporal dependencies.Figure 3 visualizes average γ_l,t by frame and layer for retrieval and VideoQA tasks.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2 Text-dependent Pooling

  • Adding spatially or temporally pooled features improves on original features, combining both improves further, and text-dependent reweighting performs best; τ=1.0 gives the best reported results.The authors state that the pooled features add relatively acceptable computation or memory cost to cross-attention because they are smaller than the original features.Source: §4.3.2 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 9
  • When frames differ substantially, text-dependent pooling gives higher weights to caption-relevant frames; when differences are minor, weights are similar and Grad-CAM visualizations examine finer spatial-temporal dependencies.The Grad-CAM maps use averaged cross-attention from the video-grounded text encoder’s eighth layer, with gradients from the video-text matching score.Source: §4.3.2 · pp. 8–9, Figure 4 · p. 8, Figure 5 · p. 8

5 Discussions

The discussions test whether LiteVL’s pooling and temporal scaling extend to CLIP and consider inference at larger retrieval scales.

5.1 Extension to Other Image-language Pre-trained Models

  • CLIP with LiteVL’s method improves on CLIP4ClipmeanP by 1.9% for R1 and 1.7% for R10 on MSRVTT-7k.For CLIP, the method uses text-dependent pooling over video features and a weighted average of reweighted features for retrieval.Source: §5.1 · p. 9

5.2 Scaling to Larger-scale Retrieval Tasks

  • Pairwise VTM scoring over every video-text pair may be slow on huge datasets; a two-stage similarity-filtering approach uses top-k candidates and has negligible reported performance degradation.The authors motivate the original pairwise approach by the relatively small test sets in their three retrieval datasets.Source: §5.2 · p. 9

6 Conclusion

LiteVL adapts an image-language model for video-language tasks without heavy video-language pre-training or object detectors. It adds dynamic temporal attention and text-conditioned non-parametric pooling, and the authors report that it outperforms state-of-the-art methods trained with much more data.

6 Conclusion

  • LiteVL inherits image-text alignment from a pre-trained image-language model, adds dynamically scaled temporal attention, and uses non-parametric text-conditioned pooling for fine-grained video-language alignment.The authors report that LiteVL outperforms state-of-the-art methods trained with substantially more data, without heavy video-language pre-training or object detectors.Source: §6 · p. 9

Limitations

The authors identify limits in their VideoQA formulation and point to broader future evaluation.

Limitations

  • Treating VideoQA as classification may constrain LiteVL to a fixed answer-candidate set; generation on more diverse long-form visual question-answering datasets remains future work.The authors also propose exploring whether adding some video-language pre-training pairs could further improve performance.Source: Limitations · p. 10

Ethics Statement

The paper flags risks from unsuitable information in inherited image-text knowledge and training data, as well as privacy and content concerns in deployment. It recommends filtering harmful training data and conducting risk analysis before real-world use.

Ethics Statement

  • Inherited image-text knowledge and training data may contain unsuitable information, so the trained models may face risks similar to those discussed for large language models.The paper says harmful training data can be filtered to improve model safety.Source: Ethics Statement · p. 10
  • The framework has no generative task, and the authors state that it therefore will not output offensive language.Source: Ethics Statement · p. 10
  • Before real-world deployment, the authors say risk analysis is necessary because data may contain unintended privacy information or video/text that does not conform to social norms.Source: Ethics Statement · p. 10

A Detailed Fine-tuning Setups

The appendix points to tables documenting dataset-specific fine-tuning setups. It also states the general frame resizing, sampling, augmentation, and reproducibility practices used across downstream datasets.

A Detailed Fine-tuning Setups

B Comparison with Previous Work

The comparison describes two common video-text alignment approaches and the video-feature aggregation choices used with them. It also identifies the comparison table's coverage of feature extraction, encoders, and pre-training data.

B Comparison with Previous Work

  • Dual-stream approaches can pool a global video feature and use an MLP with text, while cross-attention approaches use aggregated video features as keys/values and text as the query.Source: B Comparison with Previous Work · p. 12
  • Prior methods aggregate video features either by keeping the original features unchanged or by mean pooling across the spatial or temporal dimension.Source: B Comparison with Previous Work · p. 12
  • The comparison table covers video-feature extraction for alignment with text, along with the video encoder, text encoder, and pre-training data used by different methods.Source: B Comparison with Previous Work · p. 12
  • Table 10 is identified as the experimental setup for fine-tuning on MSRVTT-QA and MSVD-QA.Source: Table 10 · p. 12

C More Qualitative Results

The qualitative analysis finds that text-dependent pooling tends to alter decisions when frames are more dissimilar. A Figure 6 example shows it weighting caption-relevant frames more heavily where vanilla pooling fails.

C More Qualitative Results

  • Text-dependent pooling changes decisions more often when frame dissimilarity is higher, as indicated by greater temporal-weight standard deviation in changed cases.Source: C More Qualitative Results · p. 12
  • In the Figure 6 example, higher weights on the middle two frames match the caption about a girl showing boys her medal, producing a correct decision where vanilla pooling fails.Source: C More Qualitative Results · p. 12, Figure 6 · p. 12
  • Figure 6 plots temporal weights learned by the text-dependent pooling mechanism.Source: Figure 6 · p. 12
  • Table 11 compares prior works and LiteVL by pooling method, video encoder, text encoder, and pre-training dataset.Source: Table 11 · p. 13

Related papers

Loading 2210.11929v1…