PaperlayerView exact revision on arXiv

Source-linked AI summary

Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, Jingjing Liu

arXiv:2102.06183v1cs.CVcs.CL

TL;DR

Video-and-language systems rely on independently trained offline dense features that may mismatch target tasks and are costly to finetune end to end. CLIPBERT instead trains jointly from raw video and text using sparse short-clip sampling, and it outperforms or matches densely featured methods across diverse tasks. The findings support sparse end-to-end learning and image-text pre-training as effective components for video-text understanding.

  • Problem

    Offline video and text features can be disconnected from target tasks and domains, while full-sequence extraction imposes excessive memory and computation demands for end-to-end finetuning.

  • Method

    CLIPBERT performs end-to-end video-and-language learning from raw pixels and text tokens by sparsely sampling one or a few short clips per training step, with image-text pre-training.

  • Results

    CLIPBERT outperforms or matches state-of-the-art methods with densely sampled offline features across diverse video-text tasks and video lengths.

  • Takeaways & Limitations

    Sparse end-to-end learning with a few clips can be more accurate than dense offline video features, and image-text pre-training benefits video-text tasks.

  • Takeaways & Limitations

    Performance may improve with higher-resolution videos, and the authors encourage adding signals such as audio for better performance.

Abstract

from arXiv · show

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second generic domain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available at https://github.com/jayleicn/ClipBERT

1. Introduction

CLIPBERT addresses the task and modality disconnections of offline video-and-language pipelines by enabling end-to-end learning from sparsely sampled clips and raw text. It uses image-text pre-training and achieves strong performance across diverse video-text tasks.

  • Existing video-and-language systems commonly combine offline dense video features and text features before multimodal fusion.
  • Offline extractors may be trained on tasks and domains unlike the target, such as action recognition on human activity videos versus question answering on GIF videos.
  • Independently learned modality features and the cost of full-sequence extraction make efficient end-to-end task-specific finetuning difficult.
  • CLIPBERT sparsely samples one or a few short clips per training step, using raw video pixels and language tokens for end-to-end learning.
  • Image-text pre-training with a 2D visual backbone improves or matches prior performance on video-text tasks while retaining low memory and runtime costs.
  • Across video-text tasks and video lengths from a few seconds to three minutes, CLIPBERT often outperforms densely featured approaches, supporting a “less is more” strategy.

2. Related Work

Related work frames video-and-language understanding around retrieval, captioning, question answering, and moment retrieval, typically using offline features. Action-recognition systems likewise use computationally heavy 2D or 3D backbones and short-clip sampling to handle long videos.

  • Video-and-language research covers text-to-video retrieval, video captioning, video question answering, and moment retrieval.
  • Standard approaches commonly use offline video and text features from action-recognition, image-recognition, and language models.
  • Modern action-recognition models use deep 2D or 3D convolutional backbones that are computationally and memory intensive for long videos.
  • To reduce this burden, action-recognition models are often trained on randomly sampled short video clips rather than full-length videos.

3. CLIPBERT with Sparse Sampling

CLIPBERT enables end-to-end video-and-language learning by sparsely sampling short clips, encoding them with trainable vision and language modules, and aggregating their predictions. This design replaces dense offline features with task-specific learning while reducing the need to process full videos.

  • Framework: CLIPBERT learns joint video-text representations directly from sampled video clips and raw text tokens through end-to-end task-specific finetuning.Its vision and language encoders can be further optimized by the downstream task loss.
  • Sparse Sampling for Training: Instead of processing all N clips, CLIPBERT randomly samples Ntrain clips during training, with Ntrain typically much smaller than N.Each sampled clip is paired with the associated text input to produce an independent prediction.
  • Prediction Aggregation: Independent predictions from sampled clips are aggregated into a video-level consensus used to compute the task-specific loss.The same aggregation principle supports multiple sampled clips during training.
  • Inference: At inference, CLIPBERT uniformly samples Ntest clips with the training clip duration and aggregates their predictions into the final result.The aggregation function G can use mean-pooling, and multiple clips can be treated as a form of multiple instance learning during training.
  • Model Architecture: The architecture uses a 2D ResNet-50 vision encoder, trainable word and position embeddings, and a 12-layer transformer for cross-modal fusion.The 2D backbone is chosen because it typically uses less memory and runs faster than 3D alternatives.
  • Pretraining and Finetuning: Image-text pretraining uses 5.6M caption pairs from COCO Captions and Visual Genome Captions, while downstream finetuning uses task-specific learning rates and epochs.Pretraining also applies pixel random sampling to reduce computation and improve generalization.

4. Experiments

Experiments evaluate CLIPBERT on retrieval and question answering across six diverse datasets, then analyze resolution, sampling, pretraining, end-to-end finetuning, and computational cost. CLIPBERT generally matches or surpasses full-video methods while using sparse clips and frames.

  • Input Resolution: Larger input resolution improves retrieval while maintaining similar QA performance, with the best results around L=448.Further increasing resolution provides no significant performance boost.
  • Sparse Sampling: Sparse sampling saturates after the first few additions, while Ntrain=2 improves retrieval R1 over Ntrain=1 by 2.8% and Ntrain=16 adds only 1.9% over Ntrain=2.LogSumExp is the strongest score aggregation function, and four clips are more memory- and computation-efficient than sixteen.
  • Efficiency: Using more clips or frames substantially increases resource demands: maximum batch size is 190 for Ntrain=2 versus 16 for Ntrain=16 on one NVIDIA V100 GPU.This supports the practical value of sparse training under limited memory and computation.
  • Pretraining and Finetuning: Image-text pretrained weights provide the best performance, while end-to-end finetuning benefits from task supervision compared with freezing parameter subsets.The image-text pretraining result is reported on the MSRVTT retrieval train split and the end-to-end comparison in Table 6.
  • Text-to-Video Retrieval: CLIPBERT 4×1 exceeds HERO by 3.0% on MSRVTT R@1 using four sampled frames, and CLIPBERT 8×2 exceeds it by 5.2%.HERO uses dense full-video SlowFast features, averaging 310 frames per MSRVTT video.
  • Long-Video Retrieval: On longer ActivityNet Captions videos, CLIPBERT surpasses CE by 4.3% on DiDeMo R1 and 3.1% on ActivityNet Captions R1, while remaining competitive with long-range methods.CLIPBERT also obtains 0.8% higher R1 than HSE while sampling only 8 seconds for training from 180-second videos.
  • Video Question Answering: CLIPBERT improves over prior methods on QA, including 6.9%, 6.8%, and 0.6% gains over QueST on TGIF-QA Action, Transition, and FrameQA.CLIPBERT 1×1 uses one randomly sampled training frame, whereas QueST uses ten uniformly sampled frames.

5. Conclusion

CLIPBERT is a generic end-to-end framework that uses sparse sampling for video-and-language learning. Across diverse tasks, it performs as well as or better than methods using densely sampled offline features.

  • CLIPBERT uses only a few sampled short clips from videos at each training step for end-to-end video-and-language learning.The framework adopts sparse sampling instead of processing full-length videos.
  • Experiments across diverse tasks show CLIPBERT outperforms or matches state-of-the-art methods using densely sampled offline features.The reported results support the paper’s “less is more” principle.
  • Ablation studies identify sparse sampling, end-to-end training, and image-text pre-training as key factors associated with CLIPBERT’s success.

A. Additional Experiments

CLIPBERT is also evaluated on image-based question answering using the VQA 2.0 dataset, where it achieves reasonable performance against strong pre-training baselines.

  • CLIPBERT is applied to image-based question answering on the VQA 2.0 dataset.The model is finetuned from image-text pre-trained weights.
  • CLIPBERT shows reasonable performance compared with strong pre-training baselines on VQA 2.0.

B. Downstream Task Adaptation

CLIPBERT is adapted to retrieval and question-answering tasks through classification and multi-clip score aggregation. The downstream evaluation covers three retrieval datasets and three video question-answering datasets.

  • Text-to-video Retrieval: For text-to-video retrieval, CLIPBERT uses a two-layer MLP over the final [CLS] representation for matched-versus-unmatched classification.Training uses LogSumExp loss.
  • Text-to-video Retrieval: LogSumExp aggregates clip-level retrieval logits into a prediction, followed by negative log likelihood training over dataset examples.The ground-truth answer index is denoted by yj.
  • Text-to-video Retrieval: Retrieval experiments use MSRVTT, DiDeMo, and ActivityNet Captions, with average recall at K and median rank as evaluation metrics.Training details are reported for each dataset.
  • Video Question Answering: For video question answering, CLIPBERT classifies from the final [CLS] representation and aggregates predictions from multiple clips with LogSumExp.The output dimension equals the number of answer candidates.
  • Video Question Answering: Video QA evaluation covers TGIF-QA, MSRVTT-QA, and MSRVTT multiple-choice testing across action, transition, FrameQA, open-ended, and multiple-choice settings.The trained retrieval model ranks the five candidate answers for MSRVTT MC Test.
Loading 2102.06183v1…