PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone

Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, Pengchuan Zhang

arXiv:2307.05463v2cs.CV

TL;DR

Existing egocentric VLP frameworks separate video and language encoders and defer task-specific cross-modal learning to fine-tuning, limiting unified systems. EgoVLPv2 inserts gated cross-modal fusion into the backbones and reuses pretrained attention across tasks. It is lightweight and achieves consistent state-of-the-art performance across a broad range of downstream tasks.

  • Problem

    Existing egocentric VLP separates video and language encoders and learns task-specific cross-modal information only during fine-tuning, limiting unified frameworks.

  • Method

    EgoVLPv2 inserts gated cross-modal fusion directly into video and language backbones and switches between dual and fusion encoders during pretraining and downstream use.

  • Results

    EgoVLPv2 consistently achieves state-of-the-art performance across a wide range of downstream tasks and eight egocentric benchmarks among comparable-sized backbones.

  • Takeaways & Limitations

    Reusing pretrained cross-attention modules unifies dual- and fusion-encoder tasks while reducing task-specific fine-tuning cost.

Abstract

from arXiv · show

Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks. However, existing egocentric VLP frameworks utilize separate video and language encoders and learn task-specific cross-modal information only during fine-tuning, limiting the development of a unified system. In this work, we introduce the second generation of egocentric video-language pre-training (EgoVLPv2), a significant improvement from the previous generation, by incorporating cross-modal fusion directly into the video and language backbones. EgoVLPv2 learns strong video-text representation during pre-training and reuses the cross-modal attention modules to support different downstream tasks in a flexible and efficient manner, reducing fine-tuning costs. Moreover, our proposed fusion in the backbone strategy is more lightweight and compute-efficient than stacking additional fusion-specific layers. Extensive experiments on a wide range of VL tasks demonstrate the effectiveness of EgoVLPv2 by achieving consistent state-of-the-art performance over strong baselines across all downstream. Our project page can be found at https://shramanpramanick.github.io/EgoVLPv2/.

1. Introduction

Egocentric VLP must bridge a domain gap and unify video-language tasks beyond separate encoders with task-specific fusion. EgoVLPv2 inserts cross-modal fusion into the backbones, enabling flexible reuse across tasks with lower computational and fine-tuning costs.

  • Egocentric videos differ substantially from third-person datasets, limiting the performance of third-person-pretrained VLP on egocentric benchmarks.
  • Existing egocentric VLP uses separate video and language encoders, learning task-specific cross-modal information only during fine-tuning.
  • EgoVLPv2 incorporates cross-modal fusion directly into video and language backbones through a gating mechanism.
  • The framework switches between dual and fusion encoders, supporting uni-modal and multi-modal downstream tasks with fewer fusion parameters and reduced resource requirements.
  • EgoVLPv2 reuses pretrained cross-attention modules across tasks, reducing fine-tuning costs and enabling query-focused video summarization with limited annotated data.
  • EgoVLPv2 achieves state-of-the-art performance across eight egocentric benchmarks among comparable-sized backbones.

2. Related Works

VLP frameworks range from separate dual encoders to shared encoders and added fusion layers. EgoVLPv2 introduces fusion inside uni-modal backbones, retaining switching flexibility while using egocentric pretraining data.

  • Dual encoders use separate video and language backbones, commonly train with InfoNCE or MIL-NCE, and have been successful for video-text retrieval.
  • Shared encoders combine video and text representations and can operate across multiple modalities, while fusion-layer approaches add cross-modal modules above dual encoders.
  • EgoVLPv2 proposes a fourth framework category by inserting cross-modal fusion into uni-modal backbones with a gating mechanism.
  • Cross-attention modules switch off for dual-encoder operation and on for fusion-encoder operation, allowing the same framework to support both modes.
  • EgoVLPv2 is trained on EgoClip, a filtered version of Ego4D, which contains 3,670 hours collected from 931 people across 74 locations in nine countries.

3. EgoVLPv2

EgoVLPv2 inserts gated cross-modal attention into the video and language backbones, allowing one pre-trained system to operate as either a dual or fusion encoder. Its pre-training combines EgoNCE, masked language modeling, and video-text matching, while downstream adaptation reuses the same cross-attention modules across tasks.

  • Fusion in the Backbone: Cross-attention is inserted after video space-time attention and within the language backbone, with gating controlling fusion.The video branch combines space-time processing with cross-attention to text, while the text branch analogously incorporates video information.
  • Fusion in the Backbone: Setting α = 0 switches off cross-attention, making EgoVLPv2 behave as a dual encoder; nonzero gating enables fusion.This switching supports tasks requiring unfused or multimodal representations.
  • Downstream Adaptation: EgoVLPv2 adapts to retrieval, video question-answering, video grounding, and query-focused video summarization using dual or fusion encoder modes.For retrieval, cross-attention can be switched off for cosine similarity or on for VTM-based matching scores.
  • Pre-training Objectives: EgoNCE modifies contrastive learning by treating video-text pairs sharing a noun or verb as positives and using augmented batches with modified negatives.The embeddings are L2 normalized and the objective uses a temperature factor τ.
  • Pre-training Objectives: EgoNCE uses dual-encoder features, while masked language modeling and video-text matching use fused multimodal representations.EgoNCE is computed with cross-attention disabled, whereas MLM and VTM are computed with cross-attention enabled.
  • Pre-training Objectives: The total pre-training loss combines EgoNCE, masked language modeling, and video-text matching through weighted loss terms back-propagated end-to-end.The coefficients γ and δ control the contributions of the different objectives.

4. Experiments

EgoVLPv2 is evaluated across diverse egocentric video-language benchmarks and protocols, achieving strong results while using pretrained cross-modal representations and lightweight task adaptation. Its fusion-in-the-backbone design combines broad downstream coverage with reduced computational and fine-tuning costs.

  • Evaluation setup: EgoVLPv2 is evaluated on retrieval, video grounding, question answering, and query-focused summarization across five egocentric datasets.The protocols include zero-shot retrieval, task-specific head-tuning, and end-to-end fine-tuning.
  • Ego4D benchmarks: 91.0% inter-video and 60.9% intra-video accuracy are achieved on EgoMCQ, including a 1% absolute intra-video gain over LAVILA.LAVILA uses 15× more GPT-2-generated narrations.
  • Ego4D benchmarks: 2.11% R@1 for IoU = 0.3 over EgoVLP is gained on EgoNLQ, while EgoMQ improves over EgoVLP by 1.54% R@1 for IoU = 0.3.EgoVLPv2 sets a new state-of-the-art on the uni-modal EgoMQ grounding task.
  • Other downstream tasks: EgoVLPv2 achieves state-of-the-art QFVS F-1 scores across all four videos while learning only a tiny single-layer-transformer head.Existing QFVS baselines are trained end-to-end despite the dataset containing only 135 video-query training samples.
  • Other downstream tasks: EgoVLPv2 reaches state-of-the-art performance in both fine-tuning and zero-shot CharadesEgo regimes and gains 7.8% mAP and 4.4% nDCG over EgoVLP and HierVL on EK-100.CharadesEgo differs substantially from Ego4D because it uses crowd-sourced mobile-camera videos.
  • Ablation study: With six fusion layers, the proposed architecture has 33M fewer parameters and requires 45% lesser computing cost than stacked fusion layers.It performs slightly better than stacked fusion layers, and increasing fusion depth improves both methods.

5. Conclusion

EgoVLPv2 incorporates cross-modal fusion directly into video and language backbones to provide a lightweight, flexible framework for egocentric video-language tasks. Experiments report consistent state-of-the-art performance and visual evidence of learned cross-attention representations.

  • Conclusion: EgoVLPv2 places cross-modal fusion inside the video and language backbones rather than adding separate fusion-specific layers.The strategy is described as lightweight and compute-efficient while supporting flexible downstream task unification.
  • Conclusion: Extensive experiments show consistent state-of-the-art performance across a wide range of downstream tasks.The conclusion reports effectiveness across the evaluated egocentric video-language tasks.
  • Conclusion: Visualizations demonstrate that the learned cross-attention representation attends to semantic regions associated with egocentric actions.Different attention heads attend to different video-frame regions guided by narration.

C. Dataset Details

This section describes the datasets and pre-training procedure used for EgoVLPv2, including Ego4D-derived benchmarks and the model's three training objectives.

  • Pre-training: EgoVLPv2 pre-training combines EgoNCE, masked language modeling, and video-text matching losses.EgoNCE uses dual-encoder features, while MLM reconstructs masked tokens and VTM predicts whether video-text pairs match.
  • Pre-training: The total pre-training loss combines the three objectives with weights γ and δ before end-to-end back-propagation through the encoder.
  • Dataset Details: The downstream datasets include Ego4D benchmarks, QFVS, and EgoTaskQA, covering retrieval, grounding, action localization, summarization, and video question answering.Ego4D benchmarks include EgoMCQ, EgoNLQ, and EgoMQ; QFVS uses query concepts, while EgoTaskQA evaluates descriptions, explanations, anticipation, and counterfactual predictions.
  • Dataset Details: QFVS uses 3–5-hour egocentric videos and defines each consecutive 5-second video clip as a shot.
  • Dataset Details: EgoTaskQA contains long videos segmented into clips averaging 25 seconds and uses machine-generated open-answer or binary-verification questions.

D.1. Pre-training on EgoClip

EgoVLPv2 is pre-trained and then adapted across downstream tasks using dual or fused representations, with specialized handling for very long videos and task-specific settings.

  • Pre-training: Pre-training uses TimeSformer-B and RoBERTa-B backbones with mixed-precision training and gradient checkpointing.
  • Pre-training: The model extracts four frames per video sample during pre-training, resizes them to 224 × 224, and tokenizes narrations to a maximum length of 30.
  • Downstream Settings: EgoNLQ uses pre-extracted video-text features and trains VSLNet because clips can last up to 1200 seconds, making end-to-end fine-tuning difficult.
  • Downstream Settings: QFVS uses a single-layer transformer head, while EgoTaskQA trains linear layers on fused representations and retrieval tasks use task-specific objectives.
  • Downstream Settings: For long-video tasks, EgoVLPv2 extracts unfused clip features, segments videos with KTS, and applies fused layers to compute multimodal representations.

E. Additional Ablations on Pre-training

Ablations show that EgoNCE sampling, learnable gated cross-attention, projector capacity, batch size, and frame count materially affect pre-training performance.

  • EgoNCE: Replacing EgoNCE with InfoNCE reduces EgoMCQ intra-video accuracy by 5.7%, while removing either modified sampling component causes 2.1–1.8% drops.
  • Gated Cross-attention: Learnable gated cross-attention outperforms a fixed gating value, with learned α values ranging from 0.05 to 0.4 across layers.
  • Projector: A 4096-4096-4096 three-layer projector improves EgoMCQ intra-video retrieval by 0.85% over a single 4096-dimensional projector.
  • Batch Size: EgoMCQ performance is best with a pre-training batch size of 256, while larger sizes reduce performance.
  • Number of Frames: Increasing pre-training frames improves EgoMCQ performance up to four frames, after which the improvement diminishes.

F. Ablations on Downstream

Downstream head ablations show that EgoVLPv2's pre-trained cross-modal representations support strong performance with compact task heads, while QFVS favors a single transformer layer.

  • EgoNLQ: On EgoNLQ, a linear conditional span predictor beats EgoVLP by 2.43% R@5 for IoU=0.3.
  • EgoNLQ: Adding the QGH module improves EgoNLQ performance, while the full VSLNet significantly beats EgoVLP and LAVILA across all metrics.
  • EgoNLQ: EgoVLPv2 reaches its best EgoNLQ performance within 100 tuning epochs, compared with 200 epochs used by previous methods.
  • QFVS: For QFVS, a single-layer transformer outperforms linear and multi-layer transformer heads on the small 135-sample training set.Linear layers cannot model cross-shot temporal relations, whereas multi-layer transformers overfit the small dataset.

G. Error Analysis

EgoVLPv2 struggles to attend distinctly to tiny or hindered objects in cluttered environments, although it can track larger or more familiar regions. Attention visualizations illustrate this limitation in the text-to-video cross-attention module.

  • Error Analysis: Tiny and hindered objects in cluttered environments are not distinctly attended by EgoVLPv2.The limitation is visualized through [CLS]-token attention maps from the text encoder in the last text-to-video cross-attention layer.
  • Error Analysis: EgoVLPv2 can recognize larger, familiar objects such as tables and human hands more reliably than barely visible tools.The bicycle handle and T-wrench are difficult to attend to consistently, whereas tables and hands are recognized.
  • Error Analysis: In a wet-lab activity, the model tracks gloved hands and thumb motion but only partially attends to a hindered test tube and wire.The scene combines collaborative activity with fine-grained actions and partially occluded objects.
  • Error Analysis: The 224 × 224 pre-training frame resolution makes tiny objects difficult to distinguish, while higher-resolution frames may help.The resolution constraint is identified as a factor contributing to the observed failures.

H. Qualitative Downstream Performance

Qualitative examples show EgoVLPv2 improving discrimination on challenging multiple-choice questions and producing query-relevant summaries and better-aligned temporal localization predictions. These examples span visually similar clips, long-video summarization, and text-guided localization.

  • EgoMCQ: EgoVLPv2 substantially outperforms EgoVLP on the challenging intra-video EgoMCQ setting with visually similar choices.The VTM head uses hard-negative sampling to distinguish between similar videos.
  • EgoMCQ: The intra-video EgoMCQ setting presents five contiguous clips from one video, making it more challenging than the inter-video setting.The inter-video setting instead uses five clips from different videos.
  • QFVS: On QFVS, EgoVLPv2 generates query-focused summaries containing the scenes relevant to a natural-language query.For example, a query about streets and trees produces a summary of scenes containing those elements.
  • Text-guided Video Localization: EgoVLPv2 produces text-guided localization windows that are significantly more aligned with ground truth than EgoVLP.The task uses an untrimmed video and language query to predict one temporal window containing the answer.
Loading 2307.05463v2…