PaperlayerView exact revision on arXiv

Source-linked AI summary

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Max Bain, Arsha Nagrani, Gül Varol, Andrew Zisserman

arXiv:2104.00650v2cs.CV

TL;DR

Video-text retrieval is limited by visual architectures and noisy, computationally demanding video-text data. The paper proposes an end-to-end dual encoder trained across image and video captions, introduces WebVid-2M, and reports state-of-the-art results across standard retrieval benchmarks. Its curriculum begins with images as frozen video snapshots and gradually adds temporal context.

  • Problem

    Video-text retrieval requires efficient joint embeddings, but large-scale video-text datasets such as HowTo100M are noisy and competitive training requires substantial compute.

  • Method

    The paper introduces an end-to-end dual encoder with space-time transformer attention, joint image-video training, curriculum learning over temporal context, and the WebVid-2M dataset.

  • Results

    The model achieves state-of-the-art performance on MSR-VTT, MSVD, DiDeMo, and LSMDC while using fewer video-text pairs than HowTo100M-based methods.

  • Takeaways & Limitations

    A single flexible transformer model can exploit image and video captioning datasets jointly for downstream video-text retrieval.

Abstract

from arXiv · show

Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute. We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as 'frozen' snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

1 Visual Geometry Group, University of Oxford 2 LIGM, ´Ecole des Ponts, Univ Gustave Eiffel, CNRS

The paper presents joint image–video training with a space-time transformer that treats images as single-frame videos.

  • The listed affiliations are the University of Oxford and LIGM, École des Ponts, Université Gustave Eiffel, CNRS.
  • The visual encoder processes images and videos jointly by treating an image as a single-frame video.
  • One author is now at Google Research.

1. Introduction

The paper unifies image and video retrieval through an end-to-end dual encoder trained on both modalities and introduces WebVid-2M for scalable video-text pretraining.

  • Image and video research commonly use separate architectures, datasets, and benchmarks despite overlapping information.
  • The proposed dual encoder trains on image-caption pairs, video-caption pairs, or both using a transformer visual encoder.
  • Images are treated as videos frozen in time, allowing variable-length training without generating static videos for image examples.
  • WebVid-2M provides 2.5M web-scraped video-text pairs, while the model also uses large-scale image-caption datasets such as Conceptual Captions.
  • The authors report state-of-the-art results on MSR-VTT, MSVD, DiDeMo, and LSMDC while using fewer video-text pairs than HowTo100M-based methods.

2. Related Works

Prior video-text retrieval methods often depend on pretrained expert features or large noisy video datasets, while this work uses a unified self-attention model for images and videos.

  • Many video-retrieval systems combine pre-extracted expert features from multiple tasks and modalities.
  • HowTo100M-based pretraining offers rich supervision but noisy instructional-video text makes competitive performance computationally expensive.
  • CLIP learns joint image-text representations from millions of web-scraped text-image pairs, but is designed for images.
  • Vision transformers improve efficiency by representing images as sequences of patch embeddings rather than individual pixels.
  • The proposed architecture uses divided space-time attention and expandable temporal embeddings for joint image-video training with variable-length inputs.

3. Method

The method encodes spatiotemporal patches with transformer attention, aligns video and text embeddings using bidirectional retrieval losses, and trains efficiently through image-video curriculum learning.

  • Model Architecture: The model accepts image or video inputs with M frames, where images use M = 1, and divides them into non-overlapping spatiotemporal patches.
  • Model Architecture: Spatial and temporal positional embeddings are added to patch tokens so the model can identify their spatial and temporal locations.
  • Model Architecture: Space-time transformer blocks perform temporal self-attention followed by spatial self-attention, with [CLS] tokens producing visual and text embeddings.
  • Training Strategy: Matching video-text pairs are positives and other batch combinations are negatives in the summed video-to-text and text-to-video retrieval loss.
  • Training Strategy: Alternating image and video batches exploits the O(M^2) attention cost, allowing larger image batches when M = 1.
  • Training Strategy: Training begins with fewer frames and expands temporal embeddings through interpolation or zero-padding as the input-frame curriculum grows.
  • Training Strategy: At test time, embeddings from frames sampled across video segments are averaged into the final video embedding.

4. Experiments

Experiments evaluate WebVid-2M, image and video pretraining, curriculum learning, and downstream retrieval across multiple video benchmarks. The model achieves strong results despite using substantially less data than HowTo100M-based systems.

  • 4.1. Pretraining Datasets: WebVid-2M contains 2.5M web-scraped video-text pairs, making it an order of magnitude larger than existing video captioning datasets.Its captions vary in style, while the dataset is described as smaller than HowTo100M in video duration and paired clip-captions.
  • 4.1. Pretraining Datasets: WebVid-2M captions are aligned with videos, generally well formed, and less affected by ASR transcription and grammatical noise than HowTo100M captions.They are also longer and more textually diverse on average.
  • 4.5. Curriculum strategy: Curriculum pretraining reaches 26.6 R@1 at half the GPU-hour cost, while an 8-frame curriculum achieves similar or better performance at almost one-third the cost.The schedule progressively increases temporal context from fewer frames to more frames.
  • 4.6. Comparison to the State of the Art: On MSR-VTT, the model outperforms previous methods, including systems trained on HowTo100M, despite using 13K rather than 135K training hours and 5.5M rather than 136M caption-clip pairs.It also outperforms systems using multiple expert features.
  • 4.6. Comparison to the State of the Art: Zero-shot MSR-VTT performance exceeds MIL-NCE and Support Set, both trained on HowTo100M, while COCO Captions further boosts performance in zero-shot and finetuned settings.The authors interpret the zero-shot result as evidence of broader generalisability and possible domain proximity between WebVid-2M and MSR-VTT.
  • 4.6. Comparison to the State of the Art: The model outperforms prior methods on MSVD and achieves state-of-the-art results on DiDeMo and LSMDC using visual information alone.On DiDeMo, finetuning adds a 14.2% R@1 boost, while the model is equivalent to CLIPBERT’s finetuned zero-shot comparison before finetuning.

5. Extension: Scaling up Further

The paper examines larger and more diverse pretraining sources, finding that image-only pretraining remains competitive while added data improves downstream video retrieval with diminishing returns.

  • WebVid-10M extends WebVid-2M fourfold to 10 million text-video pairs using the same data-collection protocol.The section also evaluates Conceptual-Captions 12M and other additional pretraining sources.
  • Image pretraining alone achieves reasonable downstream video-retrieval performance despite using only a small number of text-image pairs.COCO Captions performs worse than broader pretraining combinations but remains competitive.
  • Increasing pretraining pairs consistently improves downstream performance, but with diminishing returns.The reported comparison favors adding smaller datasets from diverse sources over scaling one source alone.
  • Adding COCO Captions’ 567k pairs to CC3M+WV2M provides a larger boost than adding 7.5 million additional WebVid10M pairs from the same source.

6. Conclusion

The paper concludes that its dual encoder enables end-to-end text-video retrieval training across image and video captioning datasets. It reports state-of-the-art downstream performance while identifying larger and combined pretraining datasets as opportunities for further improvement.

  • The proposed dual encoder is designed for end-to-end text-video retrieval using both large-scale image and video captioning datasets.
  • The model achieves state-of-the-art performance on multiple downstream benchmarks.
  • Performance is not saturated and could improve with full HowTo100M, larger weakly paired image datasets, and multi-dataset combinations.

A. Additional Benchmark Results

Additional results report competitive ActivityNet performance and describe the video encoder’s patch, positional-embedding, and space-time-attention components.

  • A. Additional Benchmark Results: The model achieves comparable results to the state of the art on ActivityNet Captions while using much less training data.Evaluation uses paragraph-to-video retrieval on the val1 set of 4.9K videos.
  • B.1. Video Encoder: The video encoder combines a patch embedding layer, learnable spatial, temporal, and [CLS] embeddings, and 12 space-time attention blocks.
  • B.1. Video Encoder: The patch embedding layer uses a 2D convolution with patch size P = 16 and 768 output channels.
  • B.1. Video Encoder: Spatial and temporal positional embeddings encode patch locations and frame positions, while the [CLS] embedding represents the transformer output.

C.1. Video Backbone

Ablations favor the space-time transformer and the chosen DistilBERT text encoder, while the modified attention block gives modest gains and temporal-embedding expansion remains robust.

  • C.1. Video Backbone: The space-time transformer encoder produces large MSR-VTT improvements over ResNet and 3D backbone variants.
  • C.1. Video Backbone: Frame variants receive an equal number of frames during testing because video embeddings are averaged over multiple strides.
  • C.1. Video Backbone: DistilBERT and BERT perform similarly, while T5 performs significantly worse with more or similar parameter counts.DistilBERT is selected for efficiency because it uses far fewer parameters.
  • C.1. Video Backbone: The modified space-time attention block yields modest zero-shot MSR-VTT retrieval gains over the original divided block.The modification also trains more quickly and more stably than the original block.
  • C.1. Video Backbone: Three temporal positional-embedding expansion methods show robustness across the evaluated approaches.The methods include zero-padding and two interpolation methods.

D. WebVid-2M Dataset Details

WebVid-2M is characterized through video-text examples and distributions of video durations and caption lengths. The appendix also records temporal-expansion experiments and a fourfold extension to WebVid-10M.

  • Dataset statistics: WebVid-2M includes 275,000 videos longer than 30 seconds, providing examples for training long-range video models.
  • Dataset examples: Figure 6 presents qualitative WebVid-2M video-text pairs using video thumbnails.
  • Dataset statistics: Figure 7 reports separate histograms for video duration in seconds and caption length in words.
  • Temporal expansion: Temporal-expansion experiments compare methods for increasing the input from 4 to 8 frames on MSR-VTT text-video retrieval.Models were pretrained on CC3M and WebVid-2M, then finetuned on the MSR-VTT training set.
  • WebVid-10M extension: WebVid was extended fourfold to 10 million text-video pairs using the same data-collection protocol.
Loading 2104.00650v2…