PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing Single Frame Bias for Video-and-Language Learning

Jie Lei, Tamara L. Berg, Mohit Bansal

arXiv:2206.03428v1cs.CVcs.AIcs.CL

TL;DR

Video-language models are commonly trained with multiple frames, but it is unclear whether that added temporal input is necessary or worth its computational cost. This paper trains models on single frames, ensembles multiple frames at inference, and finds strong performance that exposes static appearance bias while motivating new temporal retrieval tasks.

  • Problem

    It is unclear whether multiple frames are necessary for video-language learning and whether their performance benefits justify the increased computation and memory costs.

  • Method

    The paper trains SINGULARITY with randomly sampled single frames, uses large-scale image-text and video-text pre-training, and early-fuses uniformly sampled frame representations at inference.

  • Results

    Across text-to-video retrieval and video question answering, single-frame training achieves competitive or mostly better performance than existing multi-frame methods, revealing static appearance bias in current datasets.

  • Takeaways & Limitations

    Two SSv2-based retrieval tasks are proposed to complement existing benchmarks by testing temporal modeling more directly.

  • Takeaways & Limitations

    The system’s predictions can be inaccurate and biased by the training-data distribution, so users should not rely on it completely for real-world decisions.

Abstract

from arXiv · show

Training an effective video-and-language model intuitively requires multiple frames as model inputs. However, it is unclear whether using multiple frames is beneficial to downstream tasks, and if yes, whether the performance gain is worth the drastically-increased computation and memory costs resulting from using more frames. In this work, we explore single-frame models for video-and-language learning. On a diverse set of video-and-language tasks (including text-to-video retrieval and video question answering), we show the surprising result that, with large-scale pre-training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong "static appearance bias" in popular video-and-language datasets. Therefore, to allow for a more comprehensive evaluation of video-and-language models, we propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling. Our code is available at https://github.com/jayleicn/singularity

1 Introduction

The paper investigates whether video-and-language models truly need multiple frames during training. It finds that single-frame training, combined with large-scale pre-training and multi-frame inference, performs strongly while exposing static appearance bias in existing benchmarks.

  • The authors train a video-and-language model on randomly sampled single frames, then ensemble uniformly sampled frames at inference.The model uses paired text during both training and inference.
  • Large-scale pre-training substantially improves single-frame models and largely eliminates their performance gap with similarly trained multi-frame models.The authors attribute this hypothesis to improved robustness against noisy single-frame predictions.
  • SINGULARITY achieves competitive or better performance than existing multi-frame methods across retrieval and video question answering tasks.The comparison covers six datasets, from short MSRVTT videos to 180-second ActivityNet videos.
  • Strong single-frame results reveal that existing benchmarks emphasize static objects and scenes while underweighting temporal dynamics.The paper therefore treats these benchmarks as insufficient tests of full video-language understanding.
  • The authors introduce template-based and label-based SSv2 retrieval tasks to evaluate temporal modeling more directly.The template task removes objects and focuses almost solely on actions, whereas the label task includes objects and motion.
  • The paper’s contributions are single-frame training with inference-time frame ensembling and two temporal-modeling tasks for broader evaluation.

2 Related Work

Prior video-language methods generally train on multiple frames, but related dataset-bias findings motivate testing whether existing benchmarks can be solved from static appearance alone. The paper proposes SSv2-based tasks to assess temporal understanding more comprehensively.

  • Video-and-language methods have typically used multiple video frames during training, a strategy reported effective across many datasets.
  • Prior work found strong static representations in action-recognition datasets, where appearance-based classifiers can substantially outperform chance.
  • The authors find similar static appearance bias in popular video-language datasets, where single-frame models rival methods with explicit temporal modeling.
  • Two SSv2 retrieval tasks are introduced to complement biased benchmarks and test models’ true temporal modeling ability.

3 Methods

SINGULARITY uses image-language components, trains from one randomly sampled frame, and combines multiple frame representations only at inference. Its early-fusion design supplies full frame context to the multimodal encoder.

  • Model Architecture: SINGULARITY comprises a vision encoder, language encoder, and transformer-based multimodal encoder with cross-attention.The cross-attention layer gathers encoded visual information using text as the key.
  • Training: During training, the model randomly samples one frame from each video and predicts from that frame’s representation with the paired text.The video contains T frames, while the sampled frame is encoded by the vision encoder.
  • Inference: At inference, uniformly sampled frames are encoded separately and their representations are concatenated before multimodal processing.This early fusion lets the model make a video-level prediction from the combined frame context.
  • Frame Ensemble: Early fusion consistently outperforms late fusion, whose per-frame predictions can be noisy because each frame provides incomplete information.Late fusion aggregates independently computed frame scores with a manually designed function such as mean-pooling.
  • Pre-Training Objectives: The model uses three pre-training losses: vision-text contrastive learning, masked language modeling, and vision-text matching.
  • Implementation Details: Single-frame training also permits pre-training on image-text data, using image-text and video-text corpora.The implementation uses large-scale image-text and video-text datasets, including COCO, Visual Genome, SBU, CC3M, CC12M, and WebVid.

4 Experiments and Results on Existing Datasets

The experiments evaluate single-frame training across text-to-video retrieval and video question answering, with multi-frame inputs used during testing. Results show competitive or better performance despite substantially smaller training-frame and pre-training-data budgets than several prior methods.

  • Task setup: The evaluation covers text-to-video retrieval on MSRVTT, DiDeMo, and ActivityNet Captions, plus video question answering on MSRVTT-QA, ActivityNet-QA, and MSRVTT-MC.The QA tasks include open-ended and multiple-choice settings.
  • Task setup: Training uses a single frame per video, while testing uses 12 frames for MSRVTT and DiDeMo and 32 frames for ActivityNet Captions.The same downstream input resolution and augmentations are used across tasks.
  • Video question answering: The video question answering evaluation reports accuracy across open-ended MSRVTT-QA and ActivityNet-QA and multiple-choice MSRVTT-MC.MSRVTT-MC selects the best matching caption from five candidates for each video.
  • Text-to-video retrieval: SINGULARITY (5M) outperforms prior work on DiDeMo and ActivityNet Captions despite using 5M pre-training pairs and single-frame training.Some compared methods use 400M image-text pairs, 136M video-text pairs, 64 training frames, or 8 training frames.
  • Video question answering: SINGULARITY (5M) achieves competitive video question answering performance with two orders of magnitude less pre-training data than MERLOT and surpasses the video-QA model JustAsk.Performance improves further when the model is pre-trained with more data.

5 New Tasks that Require Temporal Modeling

The paper introduces two SSv2-based retrieval tasks designed to expose whether video-language models capture fine-grained temporal relationships rather than static appearance. Results show that single-frame SINGULARITY underperforms on the temporally heavy template task, while adding temporal modeling substantially improves performance.

  • Existing video-language benchmarks favor static concepts and poorly indicate fine-grained temporal understanding.
  • Task construction: SSv2-Template Retrieval uses 174 action templates as text queries and tests retrieval over 2,088 validation videos.
  • Task construction: SSv2-Label Retrieval uses annotated action-and-object labels with the same 168,913-video training and 2,088-video testing split.
  • Task construction: Template retrieval emphasizes action understanding without object cues, whereas label retrieval evaluates both static and temporal understanding.
  • Results: SINGULARITY underperforms 4-frame Frozen by 10.9 on SSv2-template R1, while its 4-frame temporal variant surpasses the baseline methods.
  • Results: The SSv2-label task benefits from more pre-training data, but SSv2-template performance remains similar, indicating that the latter primarily requires temporal modeling.

6 Analysis

The analysis shows that inference-time early fusion is more reliable than late fusion, while large-scale pre-training narrows the gap between single- and multi-frame models. Single-frame training also offers substantial efficiency advantages despite noisy frame-level predictions.

  • Frame ensemble strategy: Single-frame predictions fluctuate across frames, and late fusion can be dominated by an inaccurate high-confidence frame.
  • Frame ensemble strategy: Early fusion consistently outperforms late fusion because it considers the whole video when making predictions.
  • Frame ensemble strategy: More inference frames generally improve performance, but late fusion can degrade beyond four frames whereas early fusion improves consistently.
  • Pre-training data size: As pre-training data increases, the performance gap between 1-frame and 4-frame models decreases almost monotonically.
  • Pre-training data size: Single-frame training is noisier because random sampling and incomplete context make predictions less accurate and stable.
  • Training efficiency: The 17M single-frame SINGULARITY model trains 2.8× faster than 4-frame Frozen and 8.5× faster than 64-frame CLIP4Clip while performing better.

7 Conclusion

The paper concludes that single-frame training can perform strongly on video-language tasks when paired with sufficient pre-training and suitable frame ensembling. It also argues that existing benchmarks are biased toward static appearance, motivating temporally focused evaluation tasks.

  • Single-frame training achieves strong performance across text-to-video retrieval and video question answering with sufficient pre-training and proper inference-time frame ensembling.
  • The results reveal that current benchmarks are biased toward static objects and scenes rather than fully testing temporal understanding.
  • The proposed temporal tasks are intended to complement existing benchmarks and provide more comprehensive video-and-language evaluation.
  • Societal impact: The system’s predictions can be inaccurate and biased by the data distribution, so users should not rely on it completely for real-world decisions.

A Appendix

The appendix provides additional model, training, and experimental details, including results beyond the main video-language evaluations.

  • The appendix details the open-ended QA and SINGULARITY-temporal models, pre-training objectives, hyperparameters, and additional experimental setups.
  • Additional results cover existing-dataset temporal models, zero-shot evaluation, image size, and image-text retrieval and question answering.

A.1 Additional Modeling Details

The appendix describes SINGULARITY’s single-frame and temporal variants, their attention-based architecture, pre-training objectives, and reported temporal-modeling results.

  • Model variants: The open-ended QA model adds a multimodal decoder that cross-attends to encoder outputs and generates answer text.The decoder uses “[CLS]” as the start token.
  • Model variants: SINGULARITY-temporal encodes each frame, adds learned temporal position information, concatenates frame representations, and passes them to a temporal encoder.The temporal encoder outputs are supplied to the multimodal encoder’s cross-attention layer for prediction.
  • Model variants: The temporal architecture uses self-attention for temporal inputs and cross-attention to connect temporal representations with multimodal processing.The figure description distinguishes cross-attention inputs from self-attention inputs.
  • Model variants: During inference, temporal position encodings are interpolated when the number of input frames exceeds the training length.This extends the model to longer temporal sequences similarly to spatial position-encoding interpolation.
  • Pre-training objectives: Pre-training uses Vision-Text Contrastive, Masked Language Modeling, and Vision-Text Matching objectives.VTC aligns paired vision-language embeddings, MLM predicts masked text using visual context, and VTM classifies whether a vision-text pair matches.
  • Results: The appendix reports SINGULARITY-temporal results for text-to-video retrieval and video question answering in Tables 4 and 5.The cited passage identifies the result tables but does not provide their numerical values.

A.2 Additional Experiments

Additional experiments examine evaluation protocols, zero-shot and repeated-run performance, image resolution, image-text transfer, and hyperparameter sensitivity.

  • Evaluation setup: Retrieval experiments use average recall, while question-answering experiments use accuracy.Average recall averages R@{1,5,10} across retrieval tasks.
  • Temporal modeling: The 4-frame SINGULARITY-temporal model improves over the 1-frame SINGULARITY model, but the gap is relatively small despite higher resource costs.The comparison covers existing text-to-video retrieval and video question-answering datasets.
  • Zero-shot evaluation: SINGULARITY achieves significantly better zero-shot text-to-video retrieval results than existing methods with a similar amount of pre-training data.The cited passage reports the comparison without providing numerical values.
  • Robustness: Results for text-to-video retrieval are also reported as mean and standard deviation across five random runs.The repeated-run results use the model pre-trained on the 17M corpus.
  • Image resolution: Larger input images generally improve performance, but performance saturates at around 336×336 for the three examined tasks.Higher image sizes can be limited by the low resolution of raw videos, including 320×240 MSRVTT videos.
  • Transfer to image-text tasks: SINGULARITY performs competitively on image-text retrieval and image question answering, although it remains behind state-of-the-art image-text models.The appendix presents comparisons for Flickr30K, COCO, and VQA.
  • Limitations: The experiments did not use an extensive hyperparameter search, so further tuning could improve results.The same hyperparameters were mostly reused across datasets within each task.

A.3 Additional Data Details

The appendix identifies the datasets, licenses, image-text and video-QA comparison tables, and hyperparameter settings used to document the experiments.

  • Dataset documentation: The appendix provides statistics for the pre-training datasets and downstream datasets.These statistics are listed in Tables 13 and 14.
  • Dataset documentation: The appendix provides dataset licenses in Table 15.The passage identifies the table but does not enumerate individual licenses.
  • Experimental tables: Table 8 reports average recall for retrieval tasks and accuracy for the QA task when comparing image sizes.The compared models share a checkpoint pre-trained with 224×224 input images.
  • Experimental tables: Table 10 presents comparisons with existing methods on visual question answering.The table caption identifies the task but does not provide numerical results.
  • Hyperparameters: For SINGULARITY-temporal, the appendix uses four training frames and a smaller second-stage pre-training batch size of 32 per GPU.The four-frame setting is specified in both retrieval and general hyperparameter tables.
  • Dataset documentation: The WebVid pre-training dataset has an average video length of 18 seconds.This statistic is given in Table 13.
Loading 2206.03428v1…