PaperlayerView exact revision on arXiv

Source-linked AI summary

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Max Bain, Arsha Nagrani, Gül Varol, Andrew Zisserman

arXiv:2104.00650v2cs.CV

TL;DR

Image and video research largely develops on separate tracks despite overlapping information. This paper unifies image and video caption training in a transformer-based retrieval model, achieving state-of-the-art performance on multiple downstream benchmarks and outperforming methods pretrained on HowTo100M, which has 20x more video-text pairs.

  • Problem

    Image and video research largely develops on separate tracks despite overlap in the information they convey across tasks.

    Source: §1 · p. 2

  • Method

    A dual-encoder transformer treats images as frozen single-frame videos and uses a curriculum to train on image and video caption datasets.

    Source: §1 · p. 2

  • Results

    The model achieves state-of-the-art performance on multiple downstream benchmarks and outperforms methods pretrained on HowTo100M, which has 20x more video-text pairs.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The work takes a step toward unifying image and video training tracks for video-text retrieval.

    Source: §1 · p. 2

    The model's performance is not saturated, and the authors note it could improve with more pretraining data or dataset combinations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute. We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as 'frozen' snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

1. Introduction

The paper unifies image- and video-text training in an end-to-end transformer dual encoder, treating images as frozen single-frame videos and introducing WebVid-2M. It reports state-of-the-art video-only retrieval results on four benchmarks despite using a dataset 20x smaller than HowTo100M by video-text pair count.

1. Introduction

  • Image and video research largely developed on separate architectures, datasets, and benchmarks, despite overlapping information and the fact that some actions can be recognized from a single frame.The paper motivates joint training as a way to unify these tracks.Source: §1 · p. 2
  • The end-to-end transformer dual encoder trains on image-caption data, video-caption data, or both, treating an image as a single-frame video rather than generating a static video.It uses variable-length sequences and applies modified divided space-time attention directly to pixels, without relying on fixed, externally pretrained expert networks.Source: §1 · p. 2
  • The curriculum begins with images and gradually adds temporal context through temporal embedding interpolation, increasing training efficiency and reducing GPU time.The architecture handles inputs of different lengths, enabling this progression from images to video datasets.Source: §1 · p. 2
  • WebVid-2M contributes 2.5M web-scraped video-text pairs, while the training also takes advantage of large-scale image-caption datasets such as Conceptual Captions.The paper describes the video captions as alt-text pairs.Source: §1 · p. 2
  • The video-only model achieves state-of-the-art performance on MSR-VTT, MSVD, DiDeMo, and LSMDC, outperforming methods using multimodal expert features or HowTo100M pretraining despite HowTo100M having 20x more video-text pairs.The paper states this comparison for its final model and dataset.Source: §1 · p. 2

2. Related Works

Earlier video-text retrieval combined expert features, while large-scale pretraining brought noisy instructional-video supervision and image-text alternatives. This work distinguishes its approach with one self-attention encoder for images and videos, using expandable temporal embeddings for flexible training.

Video-text retrieval

  • MoEE [43], CE [38], MMT [21], and concurrent HiT combine pre-extracted expert features, scoring video-text pairs with a weighted sum of expert-text similarities.Source: §2 · p. 2

Video-text pretraining

  • HowTo100M [44] enabled large-scale pretraining, but noisy instructional-video text requires scale and compute; prior work used MIL-NCE [42], raw audio [1] [55], or image-text training.MoEE [43] zeroes video-only expert streams for images, while ClipBERT [32] restricts feature extractors to 2D CNNs; this paper instead proposes an encoder for both modalities.Source: §2 · p. 2

Image-text pretraining

  • CLIP learns joint image-text representations from millions of scraped pairs with contrastive loss, but is suitable only for images.Source: §2 · p. 2

End-to-end representation learning

  • Action-recognition video models long used spatio-temporal CNNs, while later video self-attention approaches combined attention blocks with CNN layers; ViT [19] instead feeds patch embeddings to a transformer.The cited video CNN families include I3D [8], 3D ResNets, S3D [71], and R(2+1)D [62]; ViT’s patch sequence avoids pixel-level self-attention.Source: §2 · pp. 2–3

Novelty

  • The architecture uses only self-attention, draws on ViT [19] and TimeSformer [6], and adds expandable temporal embeddings for variable-length videos and joint or separate image/video training.The authors report no prior self-attention model trained on both images and videos.Source: §2 · p. 3

3. Method

The method pairs a space-time visual transformer with a text encoder in a shared retrieval embedding, and trains them using joint image/video data and a curriculum over input frame counts.

3.1. Model Architecture

  • Video is split into M×N non-overlapping patches, embedded by a 2D convolution, and tagged with learned spatial and temporal positions shared across matching locations.All patches in one frame share a temporal position, while patches at the same spatial location across frames share a spatial position.Source: §3.1 · p. 3
  • The visual transformer applies temporal then spatial self-attention, pools video and bidirectional-text [CLS] outputs, and projects both into a shared space for dot-product similarity.The model modifies Divided Space-Time attention by placing the residual connection between the block input and spatial-attention output.Source: §3.1 · p. 3
  • Independent dual encoders make retrieval indexable with O(t+v) cost, versus O(tv) for ClipBERT [32], without query-conditioned expert weights.The model needs only a dot product between video and text embeddings at inference.Source: §3.1 · p. 3

3.2. Training Strategy

  • Training minimizes symmetric video-to-text and text-to-video batch losses, treating paired examples as positives and all other in-batch pairings as negatives.The losses use normalized video and text embeddings with temperature σ.Source: §3.2 · pp. 3–4
  • Alternating image-text and video-text batches enables joint pretraining; ViT-initialized spatial and zero-initialized temporal weights start as per-frame ViT, while temporal positions expand with frame count.Expansion uses zero-padding or interpolation, including nearest-neighbour and bilinear options; beginning with fewer frames reduces computation while retaining comparable or better performance.Source: §3.2 · p. 4
  • Training samples one uniformly chosen frame per equal video segment; testing samples segment frames at strides, averages their video embeddings, and uses that mean as the final video embedding.The segment count is the desired number of encoder input frames.Source: §3.2 · p. 4

4. Experiments

Experiments evaluate joint image/video pretraining, training ablations, and retrieval across video and image benchmarks. The model achieves strong reported benchmark comparisons, while the experiments also examine the contributions of WebVid-2M pretraining and temporal curriculum learning.

4.1. Pretraining Datasets

  • 2.5M video-text pairs make WebVid-2M an order of magnitude larger than existing video-captioning datasets, while its aligned, manually generated captions are longer and more diverse than HowTo100M captions.WebVid-2M is 10x smaller than HowTo100M in video duration and over 20x smaller in paired clip-captions; captions average 12 versus 4 words and have MTLD 203 versus 13.5.Source: §4.1 · pp. 4–5

4.2. Downstream Datasets

  • Evaluations use MSR-VTT, MSVD, DiDeMo, and LSMDC video-retrieval benchmarks, plus Flickr30K to test image retrieval.DiDeMo is evaluated with paragraph-to-video queries, with and without ground-truth proposals; the other datasets follow stated train/validation/test protocols.Source: §4.2 · pp. 5–6

4.4. Ablation Study

  • Joint pretraining on CC3M and WebVid-2M performs best on MSR-VTT; CC3M alone does reasonably well and WebVid-2M also outperforms the HowTo-17M subset.The comparison uses four frames for pretraining and finetuning, with one epoch of pretraining; the HowTo-17M subset totals 19K hours.Source: §4.4 · p. 6, Table 2 · p. 6

4.5. Curriculum strategy

  • 26.6 R@1 at half the GPU-hour cost is achieved with pretraining curriculum; a single frame yields 18.8 R@1, while four-frame temporal expansion at pretraining gives 26.0 versus 24.9 R@1 at finetuning.For eight frames, starting with one frame and then expanding through four to eight gives similar or better performance than starting at eight frames, at almost a third of the computational cost; the paper also reports training-speed gains with accuracy gains in Figure 3.Source: §4.5 · pp. 6–7, Table 3 · p. 7, Figure 3 · p. 7

4.6. Comparison to the State of the Art

  • On MSVD, the model outperforms prior methods including Support Set [48]; on DiDeMo, zero-shot matches finetuned CLIPBERT and finetuning adds 14.2% R@1; on LSMDC, it leads except in Median Rank versus MMT.The LSMDC comparison is against MMT, which uses HowTo100M clip-text pairs, expert features, and audio modalities, while this model uses visual information alone.Source: §4.6 · pp. 7–8, Table 5 · p. 8, Table 6 · p. 8
  • On MSR-VTT, the model outperforms previous works, including HowTo100M-pretrained systems and Support Set [48]; zero-shot results also outperform MIL-NCE and Support Set [48].The paper reports these comparisons alongside a further boost from adding COCO Captions for both zero-shot and finetuned settings.Source: §4.6 · p. 7, Table 4 · p. 8
  • 5% improvement in R@1 accompanies competitive Flickr30K retrieval versus works using similar numbers of image-text pairs, with WebVid-2M providing a sizeable boost.The model is single-stage and does not require object detections; some competing methods use high-resolution regions from a Faster-RCNN detector.Source: §4.6 · pp. 8–9, Table 8 · p. 9

5. Extension: Scaling up Further

The section evaluates additional image and video pretraining sources and reports that image-only COCO pretraining remains competitive, while more pretraining pairs improve performance with diminishing returns and source diversity can be more efficient than scaling one source.

5. Extension: Scaling up Further

  • The study adds WebVid-10M, Conceptual-Captions 12M, and COCO Captions as pretraining sources, ranging from 567k image-text pairs to 12 million captioned images.WebVid-10M extends WebVid-2M fourfold; Conceptual-Captions 12M is larger and more diverse than CC3M but has noisier captions.Source: §5 · p. 9
  • COCO Captions-only pretraining performs worse downstream but remains competitive, demonstrating reasonable video-data performance from image pretraining alone.Results are for the 1K-A MSR-VTT test set for text-video retrieval, using four frames per video when finetuning.Source: §5 · p. 9, Table 9 · p. 9
  • Increasing pretraining pairs consistently improves downstream performance with diminishing returns, while adding diverse COCO captions outperforms adding 7.5 million WebVid10M pairs from the same source.The comparison adds 567k COCO caption pairs to CC3M+WV2M versus an extra 7.5 million WebVid10M pairs.Source: §5 · p. 9

6. Conclusion · Appendix

The paper introduces a dual-encoder model for end-to-end text–video retrieval that uses image and video captioning datasets, and reports state-of-the-art downstream benchmark performance while noting that performance is not saturated.

6. Conclusion

  • The dual-encoder model is designed for end-to-end text–video retrieval using both large-scale image and video captioning datasets.The conclusion describes it as a model introduced to take advantage of both dataset types.Source: §6 · p. 9
  • The model achieves state-of-the-art performance on a number of downstream benchmarks, but its performance is not yet saturated.The conclusion notes that further improvements may be possible with additional training data.Source: §6 · p. 9
  • Potential improvements include training on the full HowTo100M dataset, larger weakly paired image datasets such as Google3BN [25], and combinations of multiple datasets.These are proposed future training-data directions, not reported experimental results.Source: §6 · p. 9

A. Additional Benchmark Results … B.2. Text Encoder

The section reports ActivityNet Captions retrieval results and specifies the video encoder’s space-time architecture and DistilBERT-based text encoder.

A. Additional Benchmark Results

  • The model achieves results comparable to the state of the art on ActivityNet Captions with much less training data.Table 10 reports text-to-video retrieval on the ActivityNet val1k set; the passage gives no numerical results.Source: A. Additional Benchmark Results · p. 10, Table 10 · p. 10
  • Training uses 10K videos; evaluation uses 4.9K val1 videos with concatenated segment descriptions for paragraph-to-video retrieval.ActivityNet Captions [29] contains 20K YouTube videos focused on actions, annotated with 100K sentences.Source: A. Additional Benchmark Results · p. 10

B.1. Video Encoder

  • The video encoder combines a patch embedding layer, learnable space, time, and [CLS] embeddings, and 12 space-time attention blocks.Source: §B.1 · p. 10
  • Patch embeddings use a 2D convolution with patch-sized kernel and stride (P = 16) and 768 output channels.Source: §B.1 · p. 10
  • Space and time embeddings have shapes M × d and N × d; N is 196 for 224 × 224 video, and [CLS] has shape 1 × d.Source: §B.1 · p. 10
  • Each space-time attention block contains norm layers, temporal and spatial self-attention layers, and an MLP; Figure 4 shows their order and connections.Source: §B.1 · p. 10, Figure 4 · p. 10

B.2. Text Encoder

  • The text encoder is distilbert-base-uncased [56], which follows BERT [17] with half as many layers and without token-type embeddings or a pooler.The implementation uses the HuggingFace2 transformers library.Source: §B.2 · p. 10

C. Architectural Ablations … C.3. Space-Time Attention

The architectural ablations report that the space-time transformer improves MSR-VTT retrieval over ResNet and 3D variants, text-backbone choice affects downstream performance, and modified space-time attention yields modest gains.

C.1. Video Backbone

  • The space-time transformer encoder improves MSR-VTT retrieval over ResNets and their 3D variants.Table 11 evaluates models pretrained on WebVid-2M and finetuned on the MSR-VTT training set.Source: §C.1 · p. 11, Table 11 · p. 11
  • At testing, all frame-based variants use an equal number of frames because video embeddings are averaged over multiple strides.Source: §C.1 · p. 11
  • The video-backbone ablation fixes the text backbone to distilbert-base-uncased; the text-backbone ablation fixes video to the base space-time transformer at 224 resolution with patch size P = 16.Source: §C.1 · p. 11

C.3. Space-Time Attention

  • The modified space-time attention block yields modest retrieval gains over the compared variant in zero-shot MSR-VTT results after WebVid-2M pretraining.The block is shown in Fig. 5; both variants are compared during pretraining on WebVid-2M.Source: §C.3 · p. 11

C.4. Temporal Expansion

The temporal-expansion experiments compare three ways to expand temporal positional embeddings and report robustness across all three.

C.4. Temporal Expansion

  • They test three temporal positional-embedding expansion methods—zero-padding and two interpolation methods—and report robustness across all three.The passage directs readers to Table 14, which is not included in the supplied evidence.Source: §C.4 · p. 11

D. WebVid-2M Dataset Details · E. WebVid-10M Extension

The sections add descriptive details about WebVid-2M, including long videos and dataset statistics, and report extending the collection to 10 million text-video pairs using the same collection protocol.

D. WebVid-2M Dataset Details

  • 275,000 WebVid-2M videos exceed 30 seconds, providing examples for training long-range video models.Source: D. WebVid-2M Dataset Details · p. 11
  • Figure 6 presents additional WebVid-2M video-text pairs through video thumbnails.Source: Figure 6 · p. 12
  • Figure 7 reports histograms of WebVid-2M video durations and caption lengths.Source: Figure 7 · p. 12
  • Table 14 compares temporal expansion methods that increase the input from 4 to 8 frames.Source: Table 14 · p. 12
  • The temporal-expansion results use the 1K-A MSR-VTT test set, with models pretrained on CC3M and WebVid-2M and fine-tuned on the MSR-VTT training set.Source: Table 14 · p. 12

E. WebVid-10M Extension

Related papers

Loading 2104.00650v2…