PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning

Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, Jianlong Fu

arXiv:2210.06031v2cs.CV

TL;DR

Long-form video-language pre-training remains underexplored and must capture temporal relationships while handling the cost of more frames. LF-VILA combines temporal contrastive alignment with hierarchical attention and achieves state-of-the-art performance across seven downstream tasks.

  • Problem

    Video-language pre-training mainly focuses on short videos, while long-form videos pose challenges in modeling temporal relationships and processing more frames.

    Source: §1 · pp. 1–2

  • Method

    LF-VILA uses MTC to align clips with temporally related sentences and HTWA to capture long-range dependencies with reduced Transformer computation.

    Source: §1 · p. 2

  • Results

    LF-VILA achieves state-of-the-art performance on four paragraph-to-video retrieval tasks and three long-form video question-answering tasks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Experiments on seven long-form video-language understanding tasks verify the effectiveness of LF-VILA for learning multimodal representations that capture long-range dependencies efficiently.

    Source: §5 · p. 10

    For much longer sequences, the maximum distance between positive pairs needs to be restricted; this work considers M = 4.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on short-form videos (i.e., within 30 seconds) and sentences, leaving long-form video-language pre-training rarely explored. Directly learning representation from long-form videos and language may benefit many long-form video-language understanding tasks. However, it is challenging due to the difficulty of modeling long-range relationships and the heavy computational burden caused by more frames. In this paper, we introduce a Long-Form VIdeo-LAnguage pre-training model (LF-VILA) and train it on a large-scale long-form video and paragraph dataset constructed from an existing public dataset. To effectively capture the rich temporal dynamics and to better align video and language in an efficient end-to-end manner, we introduce two novel designs in our LF-VILA model. We first propose a Multimodal Temporal Contrastive (MTC) loss to learn the temporal relation across different modalities by encouraging fine-grained alignment between long-form videos and paragraphs. Second, we propose a Hierarchical Temporal Window Attention (HTWA) mechanism to effectively capture long-range dependency while reducing computational cost in Transformer. We fine-tune the pre-trained LF-VILA model on seven downstream long-form video-language understanding tasks of paragraph-to-video retrieval and long-form video question-answering, and achieve new state-of-the-art performances. Specifically, our model achieves 16.1% relative improvement on ActivityNet paragraph-to-video retrieval task and 2.4% on How2QA task, respectively. We release our code, dataset, and pre-trained models at https://github.com/microsoft/XPretrain.

1 Introduction

Long-form video-language pre-training must capture temporal relationships across clips and sentences while controlling the cost of processing more frames. LF-VILA introduces MTC and HTWA and reports state-of-the-art results across seven downstream tasks.

1 Introduction

  • Prior work focuses mainly on short-form videos, while long-form videos pose temporal-alignment and computational challenges for video-language pre-training.Vanilla video-paragraph contrastive alignment ignores temporal relationships between clips and sentences; more input frames also increase Transformer cost.Source: §1 · pp. 1–2
  • MTC encourages video clips to be more similar to temporally nearby sentences than to distant sentences, alongside global video-paragraph alignment.The loss is designed to capture temporal relationships between clips and sentences and improve their joint representation.Source: §1 · p. 2
  • HTWA attends to adjacent frames in early layers, then expands temporal windows in later layers to capture long-range dependencies at reduced computational cost.The approach is motivated by sparse sampling that leaves large spatial and motion gaps between frames.Source: §1 · p. 2
  • LF-VILA achieves state-of-the-art performance on four paragraph-to-video retrieval tasks and three long-form video question-answering tasks, surpassing short-video-pretrained models across seven tasks.The paper also reports ablations of MTC and HTWA.Source: §1 · pp. 2–3

2 Related Work

Related work covers video representations, long-form video understanding, and video-language pre-training. The paper positions LF-VILA as end-to-end pre-training with a video Transformer on large-scale long-form video-paragraph data.

2.1 Video Representation

  • TimeSformer factorizes spacetime attention but remains costly with many frames; Video Swin-Transformer uses a fixed temporal window unsuitable for long-duration videos.LF-VILA proposes hierarchical temporal windows to capture long-range dependencies while reducing computation.Source: §2.1 · p. 3

2.2 Long-form Video Understanding

  • Language-based benchmarks include paragraph-to-video retrieval and long-form video question answering, while prior work often uses pre-extracted features.The paper studies end-to-end pre-training and transfer to these long-form video-language tasks.Source: §2.2 · p. 3

2.3 Video-Language Pre-training

  • LF-VILA differs from HERO and MELORT by using a video Transformer for end-to-end pre-training on large-scale long-form video-paragraph data.Prior video-language pre-training mainly focused on short-form videos.Source: §2.3 · p. 3

3 Approach

This section introduces LF-VILA's architecture, then specifies MTC, HTWA, and the pre-training pipeline.

3.1 Model Architecture

  • LF-VILA encodes video and paragraph separately, then concatenates embeddings for cross-modal learning; text attention proceeds sentence-first, while HTWA yields clip and hierarchical video features.Source: §3.1 · p. 4

3.2 Multimodal Temporal Contrastive Learning

  • MTC aligns clip and sentence representations so temporal proximity corresponds to greater similarity, addressing vanilla contrastive loss's neglect of cross-modal temporal relations.Source: §3.2 · p. 5

3.3 Hierarchical Temporal Window Attention

  • HTWA gradually expands temporal windows from local attention to full-video context, targeting long-range relations while avoiding global attention's quadratic cost.With M=4, A and K are randomly sampled; longer sequences require restricting maximum positive-pair distance.Source: §3.3 · pp. 5–6

3.4 Pre-training Pipeline

  • Stage one trains text and video encoders independently with global video–paragraph contrastive alignment plus MTC.Source: §3.4 · p. 6
  • Stage two freezes encoders, concatenates their embeddings for joint learning, and combines masked language modeling with video-text matching.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate LF-VILA on paragraph-to-video retrieval and long-form video question-answering, transfer its video encoder to classification, and ablate frame and temporal-attention choices.

4.1 Pre-training Details

  • LF-VILA-8M contains 8.5 million long-form videos and transcripts, covers ~60% of HD-VILA-100M clips, and averages 100.2 seconds per video and 307.9 words per paragraph.The dataset is built from HD-VILA-100M, which contains 100 million clip-sentence pairs from 3.3 million YouTube videos.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Paragraph-to-Video Retrieval

  • 16.1% higher R@1 than TACo on ActivityNet Captions; LF-VILA also outperforms prior best models on three other paragraph-retrieval datasets.Reported R@1 gains are 12.9% over Frozen on DiDeMo, 29.6% over Frozen on QuerYD, and 12.3% over TeachText on Condensed Movie.Source: Table 2 · p. 7, §4.2 · p. 8, Table 3 · p. 8

4.3 Long-Form Video Question-Answering

  • LF-VILA reaches new state of the art on How2QA and VIOLIN, and outperforms most prior methods on ActivityNet QA except MERLOT.MERLOT uses 180M data pairs and about 30K TPU hours, whereas LF-VILA uses about 5K GPU hours.Source: §4.3 · p. 8, Table 5 · p. 9

4.4 Transfering to Long-form Video Classification

  • LF-VILA surpasses previous methods on COIN and LVU, especially for Scene and Relation classification on LVU.The COIN result is strong despite out-of-domain pretraining videos and lower reported compute than the state-of-the-art method (~2.1K vs. ~7K GPU hours).Source: §4.4 · p. 9, Table 6 · p. 9, Table 7 · p. 10

4.5 Ablation Studies

  • Increasing sampled frames from 8 to 32 substantially improves performance, indicating that long-form video representation needs an efficient backbone that supports more frames.The comparison uses the same backbone.Source: §4.5 · p. 9, Table 8 · p. 10
  • Long-form video-language pre-training with 4 continuous clips yields a significant performance gain, larger on QuerYD than ActivityNet; their videos average 278s and 180s, respectively.Source: §4.5 · p. 9
  • In the long-form pre-training effectiveness analysis, combining the MTC loss further improves performance by 1.7% on ActivityNet and 1.9% on QuerYD, measured in terms of R@1.Source: §4.5 · p. 9, Table 8 · p. 10
  • HTWA offers a better cost-performance tradeoff: a small fixed attention window performs poorly, while a window of 32 adds substantial cost and training time with almost no performance gain.The ablation compares temporal-window choices for the video backbone.Source: §4.5 · p. 10, Table 9 · p. 10

5 Conclusion

LF-VILA pre-trains on long-form video-paragraph data with MTC and HTWA to capture temporal relations and long-range dependencies efficiently. Its effectiveness is verified on seven long-form video-language understanding tasks.

5 Conclusion

  • LF-VILA combines MTC and HTWA to learn multimodal representations that capture long-range dependencies in long-form videos efficiently.MTC captures temporal relations between modalities, while HTWA is applied with an image Transformer.Source: §5 · p. 10
  • The model is pre-trained on a large-scale long-form video-paragraph dataset.Source: §5 · p. 10
  • Experiments on seven long-form video-language understanding tasks verify the effectiveness of LF-VILA.Source: §5 · p. 10

Checklist

The checklist reports that limitations and potential negative societal impacts are discussed, while several reproducibility and asset-disclosure items are marked absent.

Checklist

  • The checklist marks discussion of the work's limitations and potential negative societal impacts as included.Source: Checklist · p. 14
  • The checklist marks provision of code, data, and instructions for reproducing main results and reporting error bars as absent.Source: Checklist · p. 14
  • The checklist reports that existing assets are cited but their licenses are not mentioned, and new assets are not included in the supplement or by URL.Source: Checklist · p. 14

A Model Architecture Details

The architecture uses separate sentence- and paragraph-level text representations, clip- and video-level visual representations, and a cross-modal encoder; HTWA is configured for long-form video processing.

A.1 Text Encoder Specifications

  • The text encoder forms sentence representations from sentence-restricted attention, then combines four sentence embeddings into a global paragraph representation.The sentence representations support MTC, while the global representation supports global alignment.Source: §A.1 · p. 15

A.2 Video Encoder Specifications

  • The video encoder uses HTWA with an eight-frame temporal window in stage 3, then derives clip representations for MTC and a video representation for global alignment.Its patch projection and relative-position embeddings are modified from Swin Transformer-base initialization.Source: §A.2 · p. 15, Table 10 · p. 16

A.3 Cross-modal Encoder Specifications

  • The cross-modal encoder concatenates text and video tokens, using 32 frames with six video tokens per frame.Source: §A.3 · p. 15

A.4 Video Encoder Computational Cost

  • The video encoder cost comparison measures encoding for 8 or 32 frames; the stated constraint is that HD-VILA and Frozen encoders cannot accept as many frames for LF-VILA-8M videos.The paper identifies using more frames as critical for long-form video-language understanding.Source: §A.4 · p. 15, Table 11 · p. 16

B.1 Hyperparameters

  • The authors set the temperature to 0.05, λ1 to 1.0, λ2 to 10.0, and the sizes of A and K to 2 and N to 3.The loss weights and MTC hyperparameters were set empirically.Source: §B.1 · p. 16

B.2 More Details of LF-VILA and other Pre-training Models.

LF-VILA is reported to outperform Frozen with relatively noisy, easy-to-get data and to reduce training cost versus HD-VILA. The authors report better performance with less pre-training data, fewer parameters, and lower input resolution.

B.2 More Details of LF-VILA and other Pre-training Models.

  • LF-VILA achieves much better performance than Frozen using relatively noisy but easy-to-get data rather than a human-annotated dataset.Source: §B.2 · p. 16
  • Compared with HD-VILA, LF-VILA greatly reduces training cost with a smaller model size.Source: §B.2 · p. 16
  • LF-VILA-8M, a subset of HD-VILA-100M comprising about 60% of its clips, achieves better performance with less pre-training data, fewer parameters, and lower input resolution.Source: §B.2 · p. 16

C Downstream Task Details

The appendix details datasets, splits, and fine-tuning setups for paragraph-to-video retrieval, long-form video question-answering, and video classification. It also identifies Table 12 as a comparison of pre-training models.

C.1 Paragraph-to-Video Retrieval

  • C.1 evaluates paragraph-to-video retrieval on ActivityNet Captions, DiDeMo, QuerYD, and Condensed Movie using their stated training and evaluation splits.ActivityNet uses 10K training videos and 4.9K val1 videos; Condensed Movie uses 30K training videos and a 1K-video test set. Table 12 gives more details of LF-VILA and other pre-training models; an asterisk marks a distilled text encoder.Source: §C.1 · pp. 16–17, Table 12 · p. 17

C.2 Long-Form Video Question-Answering

  • C.2 covers ActivityNet QA, How2QA, and VIOLIN, with dataset-specific answer or entailment classifiers and evaluation splits.For How2QA, questions are combined with candidate answers and subtitles; some YouTube videos were unavailable, leaving 1,628 training, 341 validation, and 346 testing videos downloaded.Source: §C.2 · pp. 17–18

C.3 Long-form Video Classification

  • C.3 evaluates COIN and selected LVU content-understanding tasks using official splits and a video encoder with a linear classification layer.The selected LVU tasks are relationship, speaking style, and scene/place classification.Source: §C.3 · p. 18

D LF-VILA-8M Dataset Details

LF-VILA-8M is described as the largest long-form video-language dataset, with examples and dataset statistics presented in figures.

D LF-VILA-8M Dataset Details

  • LF-VILA-8M is described as the largest long-form video-language dataset, with examples and histograms of video duration, text length, and clip-sentence pair counts.The examples appear in Figure 3 and the statistics in Figure 4.Source: D LF-VILA-8M Dataset Details · p. 18

E Limitation and Broader Impact.

The paper notes potential applications of its video-language work and possible risks from user-generated training data.

E Limitation and Broader Impact.

  • The model may support video-text retrieval and video question-answering, while user-generated video-subtitle data may contain biased or offensive content.The authors state these concerns are general to the field and are not amplified by this work.Source: E Limitation and Broader Impact · p. 18

F Responsible AI Considerations

The responsible-AI discussion flags computational cost, fairness and representation concerns, possible residual harmful content, and application-specific risks.

F Responsible AI Considerations

  • The authors identify considerable computation as an environmental concern and say reducing model size and computing effort is important future work.They also note possible misuse through fine-tuning with new data and the need to consider harms specific to production scenarios.Source: F Responsible AI Considerations · pp. 18–19
  • Fairness is not explicitly captured by current accuracy metrics, and standardized fairness measures have not yet been agreed upon.The paper identifies further methods and benchmarks for measuring fairness as an opportunity for future work.Source: F Responsible AI Considerations · p. 18
  • User-generated data may underrepresent demographic groups; category balancing may mitigate disparities, but bias and fairness issues are still likely.The paper also notes possible Western heteronormative bias, stereotypes, and gaps in representation.Source: F Responsible AI Considerations · pp. 18–19
  • Filtering may not eliminate all explicit or violent content from training data, and remaining content could affect results.The paper says approximately 90% of MovieClips videos could be downloaded, while others were broken.Source: F Responsible AI Considerations · pp. 18–19
Loading 2210.06031v2…