PaperlayerView exact revision on arXiv

Source-linked AI summary

Egocentric Video-Language Pretraining

Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, Mike Zheng Shou

arXiv:2206.01670v2cs.CVcs.AI

TL;DR

Existing video-language pretraining largely uses third-person video, while egocentric video presents a domain gap. This work introduces EgoClip, EgoNCE, and EgoMCQ for egocentric pretraining and reports strong results on five downstream tasks across three datasets.

  • Problem

    Existing video-language pretraining datasets are often third-person, leaving a domain gap with egocentric video.

    Source: §1 · p. 1

  • Method

    The paper creates EgoClip with 3.8M egocentric clip-text pairs, proposes the EgoNCE contrastive objective, and introduces EgoMCQ for development.

    Source: §1 · p. 2

  • Results

    The paper reports strong egocentric pretraining representations on five downstream tasks across three datasets.

    Source: §7 · p. 10

  • Takeaways & Limitations

    EgoMCQ provides valid feedback and is suitable as a development set, with performance aligned to zero-shot EPIC-KITCHENS-100 results.

    Source: §6.2 · p. 7

    The stated evaluation criterion for EgoClip construction is whether pretraining distinguishes EgoMCQ instances and performs well on public downstream tasks.

    Source: §6.2 · p. 7

Abstract

from arXiv · show

Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.

1 Introduction

Video-Language Pretraining seeks transferable representations for video-text tasks, but existing large-scale web datasets are mainly third-person and leave a domain gap for egocentric video. The introduction presents Ego4D as an opportunity to build egocentric pretraining resources and models.

1 Introduction

  • Existing video-language pretraining datasets are often edited, third-person web videos, and the domain gap limits direct transfer to egocentric downstream tasks.The paper motivates egocentric VLP to better support applications involving first-person video.Source: §1 · pp. 1–2
  • Ego4D’s 3,670 hours of narrated video from 74 worldwide locations cover diverse daily-life scenarios, making egocentric pretraining possible at a scale absent from domain-specific datasets such as EPIC-KITCHENS-100 [14].EPIC-KITCHENS-100 [14] was described as the largest prior egocentric dataset, but focused on kitchens and was far smaller than WebVid-2M [3] and HowTo100M [10].Source: §1 · p. 2
  • 59.4% nDCG on EPIC-KITCHENS-100 video-text retrieval and 32.1% mAP on Charades-Ego action recognition accompany state-of-the-art results and significant boosts on three Ego4D challenges.The Ego4D challenges are natural language query, moment query, and object state change classification.Source: §1 · p. 2
  • EgoClip comprises 3.8M clean first-person clip-text pairs selected from Ego4D and spanning diverse human daily activities.It addresses the lack of a suitable large-scale egocentric video-language pretraining dataset.Source: §1 · p. 2
  • EgoNCE is a video-text contrastive objective proposed to address challenges unique to egocentric pretraining datasets.Source: §1 · p. 2
  • EgoMCQ contains 39K Ego4D questions focused on video-text alignment, enabling validation and faster iteration because it is less discrepant from EgoClip than other downstream benchmarks.Source: §1 · p. 2

2 Related Work

Prior video-language pretraining (VLP) work uses joint or dual encoders, while egocentric datasets were historically small and domain-specific; Ego4D provides large-scale video and benchmarks for this field.

Video-Language Pretraining

  • VLP methods use joint encoders for multimodal fusion or dual encoders to align video and text in a common space, with dual encoders suited to retrieval [24] [25] [3] [26].MIL-NCE matches clips with temporally close multiple captions to address video-text misalignment in HowTo100M [23] [10].Source: §2 · p. 2
  • Frozen [3] uses separate video and text transformers with video-text InfoNCE [29]; this work extends that loss to EgoNCE using egocentric-friendly positive and negative sampling.Source: §2 · p. 3

Egocentric Video Datasets

  • Ego4D comprises 3,670 hours of video from 931 people across 74 locations in 9 countries and introduces benchmarks including natural language and moment queries [15].Earlier egocentric datasets [17] tended to be small-scale and domain-specific because collection is expensive, limiting progress relative to third-person VLP [23] [24] [3].Source: §2 · p. 3

3 EgoClip: An Egocentric Video-Language Pretraining Dataset

EgoClip turns Ego4D’s timestamped narrations into a large, filtered set of egocentric video-text examples. Its variable-length pairing strategy adapts clip duration to narration timing and scenario while centering clips on narrated events.

Data curation

  • EgoClip retains narrations from both annotators and filters videos without narrations, Ego4D challenge validation or test videos, and noisy video-text annotations.The source Ego4D videos have dense timestamp-level narrations describing camera-wearer activities and object interactions; the paper notes their temporal and visual alignment, unlike HowTo100M’s potentially misaligned ASR narrations [10].Source: §3 · p. 3
  • The curation yields 3.85 million narrations covering 2,927 hours of video across 129 scenarios, with clips averaging 1.0 second.EgoClip contains 21.9 clips per minute; clip-length standard deviation is 0.9 seconds and the longest clip is 60 seconds.Source: §3 · p. 3

Creation of clip-text pairs

  • EgoClip pairs each timestamped narration with a contextual variable-length clip because Ego4D provides event timestamps rather than narration intervals.The clip window is centered on the narration timestamp, with duration β_i/α; this avoids directly applying interval-based subtitle pairing used for HowTo100M [10].Source: §3 · pp. 3–4
  • Clip duration adapts to each video’s average inter-narration spacing, while α sets context granularity and averages 4.9 seconds across EgoClip.The approach centers clips to include prior event information and accommodates scenario-specific timing, such as 352.9-second television-watching intervals versus 0.9-second crop-harvesting intervals.Source: §3 · p. 4

4 Video-Language Pretraining Model

The section adapts dual-encoder video-language pretraining to egocentric data with EgoNCE, an objective that changes positive and negative sampling to address challenges in first-person activity clips.

4.1 Architecture: Dual-encoder Pipeline

  • The model uses Frozen’s dual-encoder design, with a TimeSformer video encoder and DistillBERT text encoder, and supports single-modality tasks.The paper notes that the video backbone can be replaced, for example with SlowFast or Video Swin.Source: §4.1 · p. 4

4.2 EgoNCE: An Egocentric-friendly Pretraining Objective

  • EgoClip poses two challenges: the same action occurs in different scenarios, while different actions in the same scenario can look visually similar.Source: §4.2 · p. 4
  • EgoNCE adapts vanilla InfoNCE with action-aware positive sampling and scene-aware hard negatives for egocentric video-text pretraining.The approach is presented as a general extension for existing VLP models.Source: §4.2 · p. 5
  • Action-aware positives share at least one narration noun and verb, with synonyms merged using the Ego4D taxonomy dictionary.This sampling strategy links visual actions to narrations through their verbs and objects.Source: §4.2 · p. 5
  • Scene-aware negatives are adjacent clips sampled close in time from the same video, treating different actions in one scenario as hard negatives.Source: §4.2 · p. 5

5 EgoMCQ: A Benchmark for Egocentric VLP Development

EgoMCQ is an egocentric benchmark designed to enable reliable, fast development of video-language pretraining by evaluating models on a task aligned with pretraining. It uses multiple-choice questions with inter-video and fine-grained intra-video settings.

5 EgoMCQ: A Benchmark for Egocentric VLP Development

  • EgoMCQ provides a benchmark aligned with video-text pretraining to validate choices of datasets such as EgoClip and models such as EgoNCE.The authors argue that existing egocentric benchmarks are mostly domain-specific and single-modality, making them poorly suited to this purpose.Source: §5 · p. 5
  • EgoMCQ uses Ego4D data excluded from EgoClip, removes videos sharing multiple views with EgoClip, samples one annotator narration per video, and follows EgoClip’s clip-pairing strategy.The source mainly covers validation data from Ego4D challenge benchmarks, and view removal is intended to prevent scenes from appearing in pretraining.Source: §5 · p. 6
  • MCQ avoids unreliable retrieval evaluation from duplicate or semantically similar captions, which can make training unreliable [37].Deduplication is challenging for retrieval over a whole validation set, whereas repetitions among a small number of answer choices are highly unlikely.Source: §5 · p. 6
  • Inter-video questions group clips from five different videos, while intra-video questions group five continuous clips to test fine-grained context clues such as hand interaction.Options are deduplicated within each question, and accuracy is the EgoMCQ metric.Source: §5 · p. 6
  • 39K questions cover 198K narrations and 468 hours; inter-video has 24K questions over 290.3 hours, and intra-video has 15K over 178.3 hours.The average duration among the five options is 34.2 seconds.Source: §5 · p. 6

6 Experiments

The experiments evaluate egocentric pretraining on five downstream benchmarks and show consistent gains from EgoNCE over InfoNCE across downstream tasks. Ablations also support EgoClip’s context-aware clip creation and EgoMCQ as a development set.

6.2 Ablation Studies

  • EgoMCQ performance aligns with zero-shot EPIC-KITCHENS-100 results; centering clips helps, variable-length clips matter, and contextual varied lengths outperform centered clips at similar average length.A scale factor greater than 1 is preferred; the authors conclude the strategy and EgoMCQ are effective for clip creation and development.Source: §6.2 · p. 7, Table 2 · p. 7
  • Joint verb-and-noun cues improve EgoMCQ accuracy beyond naive InfoNCE, same-video negatives help, and temporally hard negatives perform best; combining optimal settings yields the best EgoNCE result.Positive samples share at least one verb-noun pair; hard negatives come from within one minute of the paired annotation timestamp.Source: Table 3 · p. 7, §6.2 · pp. 7–8

6.3 Comparisons with State-of-the-arts

  • 1.0 mAP and 5.9% nDCG gains over JPoSE make EgoClip pretraining the strongest reported EPIC-KITCHENS-100 retrieval result, with fewer frames and input modalities.The prose also reports zero-shot gains over CC3M+WebVid-2M and HowTo100M, and a fine-tuned margin over CC3M+WebVid-2M.Source: Table 4 · p. 8, §6.3 · p. 8
  • R@1 for IoU=0.3 improves from 5.45 to 10.84 on Natural Language Query, and the model surpasses VLP pretrained on CC3M+WebVid-2M and HowTo100M.The authors attribute the gains to egocentric data availability and video-text interaction learned from large-scale pretraining.Source: Table 5 · p. 8, §6.3 · pp. 8–9
  • 2.0% improvement over Ego-Exo after fine-tuning gives the model the best reported Charades-Ego action-recognition result, using fewer sampled frames.In zero-shot evaluation, it also outperforms two supervised baselines.Source: Table 6 · p. 9, §6.3 · p. 9
  • 4.66% increase in Avg mAP over SlowFast features gives the model the best reported Moment Query result, with stronger performance than 3rd-person pretraining features.This comparison uses video-only features extracted for the VSGN model.Source: Table 7 · p. 9, §6.3 · p. 9
  • Frozen pretrained on EgoClip with EgoNCE consistently improves over InfoNCE across all downstream tasks, supporting EgoNCE and the decisions informed by EgoMCQ.Source: §6.3 · p. 9
  • 2.4% over CC3M+WebVid-2M counterparts gives the model the best reported Object State Change Classification result, indicating attention to fine-grained state-change clues.Source: Table 8 · p. 9, §6.3 · p. 9

7 Conclusion, Limitations, and Societal Impacts. · Appendix · A Differentiating Egocentric VLP and Ego4D

The work establishes an egocentric video-language pretraining framework with EgoClip, EgoNCE, and EgoMCQ, and reports strong transfer across five tasks. It distinguishes this research program from the Ego4D dataset while noting temporal-dependency and privacy concerns.

7 Conclusion, Limitations, and Societal Impacts.

  • The authors report strong representation from egocentric pretraining across five tasks and three datasets.The conclusion does not specify task-level results or metrics here.Source: §7 · p. 10
  • EgoClip contributes 3.8M curated egocentric clip-text pairs from Ego4D for large-scale pretraining.The paper describes its curation as principled and the pairs as well-chosen.Source: §7 · p. 10
  • EgoNCE adapts pretraining to egocentric video characteristics through meaningful sampling strategies.Source: §7 · p. 10
  • EgoMCQ is a benchmark close to the pretraining set, intended to support efficient exploration and development of EgoClip and EgoNCE.Source: §7 · p. 10
  • The approach does not model long-term temporal dependencies in long Ego4D videos, which the authors leave for future work.Source: §7 · p. 10
  • Egocentric VLP may support applications such as augmented reality and robotics, but Ego4D videos may contain privacy risks and unintended biases.The authors advise cautious use and refer readers to the Ego4D paper for further privacy and societal impacts.Source: §7 · p. 10

Appendix

  • The appendix covers EgoClip and EgoMCQ construction and statistics, model details, and additional experimental details and results.It also includes a section differentiating Egocentric VLP and Ego4D.Source: Appendix · p. 13

A Differentiating Egocentric VLP and Ego4D

  • Ego4D motivates the 1st-person domain, but Egocentric VLP additionally requires a pretraining dataset, development set, model designs, and transferability evaluation.The authors say these elements were less explored than their third-person counterparts and conduct a systematic study of them.Source: A Differentiating Egocentric VLP and… · p. 13

A.1 Pretraining dataset … B.2 Data compression

The sections describe EgoClip curation, EgoMCQ as a development benchmark, and EgoNCE as an egocentric pretraining objective, alongside transferability findings and dataset filtering and compression procedures.

A.1 Pretraining dataset

  • EgoClip addresses Ego4D’s untrimmed videos, lack of direct video-text pairs, and noisy data; its pairing approach outperformed a naive strategy on the development set.The authors describe curating Ego4D into a pretraining dataset as requiring substantial design and experimental validation.Source: §A.1 · p. 13

A.2 Development set

  • EgoMCQ was proposed as a first-person development benchmark aligned with pretraining-data diversity and video-text alignment to support rapid design exploration.The stated targets for exploration are the pretraining dataset and model pretraining objective.Source: §A.2 · p. 13

A.3 Model designs

  • Frozen was selected as a representative scalable dual-encoder baseline, while EgoNCE explores an egocentric objective analogous to MIL-NCE [23] on HowTo100M [10].Source: §A.3 · p. 13

A.4 Transferability evaluation

  • The authors report promising Egocentric VLP transfer results and note that Ego4D had not previously been used for downstream tasks on other datasets.The passage does not provide benchmark-specific values or comparisons.Source: §A.4 · p. 13

B.1 Data filtering

  • EgoClip filtering removes stereo-video excess, videos with aspect ratio above 2, unsure-tag narrations, and narrations shorter than three words.The criteria affect 1.3% of video duration, 0.4% of video duration, 4.0% of texts, and 0.9% of texts, respectively; short narrations are removed because they generally cannot be deduced from video.Source: §B.1 · p. 14

B.2 Data compression

  • One pretraining epoch over 3.8M video-text pairs takes 6 hours on 32 V100 GPUs (192 GPU hours).Because Ego4D videos average 24 minutes and reach 7 hours, the authors resize videos to short size 256, split them into segments up to 10 minutes, and load only the clip’s segment; this enables end-to-end pretraining with raw RGB input.Source: §B.2 · p. 14

B.3 Data analysis … C.2 Multiple-views removing

EgoClip spans diverse sources, scenarios, clips, and narrations, while EgoMCQ construction removes semantically repeated options and keeps multi-view recordings within the same split. The analysis reports geographic and content coverage alongside clip and narration statistics.

B.3 Data analysis

  • EgoClip sources span 13 institutions across nine countries on four continents, inheriting Ego4D’s geographic and participant diversity [15].The distribution is shown in Figure 3.Source: §B.3 · p. 14, Figure 3 · p. 14
  • EgoClip covers 129 daily scenarios, from a largest category of 622K clips (11.1%) to a smallest category with 35 instances.The scenario distribution is long-tailed.Source: Figure 4 · p. 15, §B.3 · p. 16
  • EgoClip averages 13.4 clips per minute and 0.98 seconds per clip; 69.5% of clips are shorter than one second.Clip narration frequency ranges from 0.06 to 175.8 narrations per minute, and clip duration reaches 65.36 seconds; clip length is estimated using the frequency of the source video.Source: §B.3 · p. 16, Figure 6 · p. 16
  • EgoClip narrations average 9.39 words, 1.84 nouns, and 0.87 verbs, covering 116 verbs and 555 nouns; “napkin” appears in 1.0M clips (27.06%).Synonyms are merged in the noun and verb counts.Source: §B.3 · p. 16, Figure 7 · p. 16

C.1 Data deduplication

  • To avoid repeated options while preserving fine-grained verb differences, EgoMCQ deduplicates narrations by their first verb and noun, accounting for synonyms, and resamples until five options have different tags.Sentence-level similarity can treat narrations differing by a single verb, such as “opens” versus “closes,” as similar despite their different meanings.Source: §C.1 · p. 19

C.2 Multiple-views removing

  • Videos from NUS, Minnesota, Georgia Tech, and Indiana collected in the same timeframe are manually grouped into the same split to prevent multi-view recordings of a scene crossing splits.These sources contribute the multi-camera video data.Source: §C.2 · p. 19

C.3 Data analysis · D Technical details of our VLP model

The data analysis characterizes EgoMCQ’s scale, video-setting split, geographic and scenario coverage, narration vocabulary, and example visualizations. The following section introduces additional technical details of the VLP model, focusing on architecture and pretraining objective.

C.3 Data analysis

  • EgoMCQ contains 39K questions covering 198K narrations and 468 hours of video, with five answer options averaging 34.2 seconds.Source: §C.3 · p. 19
  • The inter-video setting has 24K questions across 290.3 hours of video, while the intra-video setting has 15K questions across 178.3 hours.Source: §C.3 · p. 19
  • EgoMCQ covers 13 institutions, and its geographic diversity aligns with EgoClip’s.Source: §C.3 · p. 19, Figure 10 · p. 19
  • EgoMCQ covers 74 scenarios, including 71% of EgoClip’s scenarios plus three not in EgoClip; Cooking has 49K clips (15.3%), while Bus has six instances.The paper describes EgoMCQ as close to EgoClip in geography and scene diversity, making it a development set for EgoClip pretraining.Source: §C.3 · p. 19
  • Across EgoMCQ’s 198K narrations, each contains an average of 3.15 nouns and 0.97 verbs; “hand” occurs in 86K instances (36.2%) and “pick” in 28K (12.0%).Source: §C.3 · p. 19, Figure 11 · p. 20
  • Figure 9 displays examples of both intra-video and inter-video EgoMCQ settings.Source: §C.3 · p. 20

D Technical details of our VLP model

D.1 Architecture: Frozen-in-time [3] … E.5 Results on test set of Natural Language Query

The paper adapts video-text pretraining for egocentric data through tailored encoders and sampling, then evaluates pretrained models on EgoMCQ, EPIC-KITCHENS-100, and Ego4D tasks.

D.1 Architecture: Frozen-in-time [3]

  • The Timesformer video encoder projects its final CLS token, while the DistillBERT text encoder projects its CLS token into the shared representation space.DistillBERT has 40% fewer parameters than BERT while preserving over 95% of its performance.Source: §D.1 · p. 20

D.2 Pretraining objective: EgoNCE

  • EgoNCE extends InfoNCE with noun-and-verb-overlap positives and same-video negatives sampled less than one minute apart, targeting frequent egocentric appearance changes.The temporally adjacent negative strategy improves pretraining beyond the baseline despite halving batch size for memory limits.Source: §D.2 · pp. 20–21

E.1 Implementation details

  • Pretraining uses 4 frames at 224×224 resolution, batch size 512, 10 epochs, and selects downstream checkpoints by average EgoMCQ inter- and intra-video accuracy.The common feature dimension is 256 and the temperature is 0.05; the video encoder starts from ImageNet-21K-trained ViT weights.Source: §E.1 · p. 21

E.2 Downstream settings

  • Downstream tasks use tailored objectives: EPIC-KITCHENS-100 uses multi-instance max-margin retrieval with positives c_ij>0.1 and margin 0.2; Charades-Ego uses InfoNCE; Ego4D object-state classification uses 16-frame inputs and cross-entropy.For Ego4D Natural Language Query, pretrained VLP video and text features replace baseline SlowFast and BERT features, using 1.87-fps extraction. For Moment Query, VLP video features replace baseline VSGN SlowFast features.Source: §E.2 · pp. 21–22

E.3 VLP Evaluation on EgoMCQ

  • EPIC-KITCHENS-100 pretraining with 67.2K first-person pairs performs comparably to HowTo100M pretraining with 136M noisy third-person pairs on EgoMCQ.CC3M+WebVid-2M yields a marked intra-video improvement but only a minor inter-video gain, which the authors attribute to broad appearance coverage but less fine-grained human-object interaction.Source: Table 10 · p. 22, §E.3 · p. 22

E.4 Training Curves of EPIC-KITCHENS-100 video-text retrieval

  • EgoClip-pretrained models nearly converge on EPIC-KITCHENS-100 retrieval in fewer than 20 epochs, and EgoNCE further strengthens the benefit of pretraining.Video-text-pretrained variants rise faster than models without pretraining, except HowTo100M, whose curve is similar to the unpretrained baseline.Source: Figure 12 · p. 22, §E.4 · p. 23

E.5 Results on test set of Natural Language Query

  • EgoClip pretraining with EgoNCE reaches optimum performance on the Natural Language Query test set.The supplied passage reports the result but provides no specific metric values.Source: Table 11 · p. 23, §E.5 · p. 23

E.6 Results on test set of Moment Query · E.7 Visualization

On the Moment Query test set, EgoClip with EgoNCE achieves the best reported results over the baseline; the visualization compares its EPIC-KITCHENS-100 retrieval with third-person pretraining. Overall, the sections report 3.78% R@1 and 4.65% Avg mAP and illustrate retrieval rankings and correlation scores.

E.6 Results on test set of Moment Query

  • 3.78% on R@1 and 4.65% on Avg mAP are the best Moment Query test-set results, achieved by pretraining with EgoClip and EgoNCE over the baseline.Table 12 presents recall and mAP metrics on the Moment Query test set.Source: §E.6 · p. 23, Table 12 · p. 23

E.7 Visualization

  • The EPIC-KITCHENS-100 retrieval visualization compares EgoClip with EgoNCE pretraining against CC3M+WebVid-2M pretraining.Both conditions are fine-tuned with 16 frames.Source: §E.7 · p. 23, Figure 13 · p. 23
  • The retrieval examples display correlation scores between each narration and its retrieved result.Source: §E.7 · p. 23
  • A correlation score of 1 is the best score in the retrieval visualization.Source: §E.7 · p. 23
  • The EPIC-KITCHENS-100 comparison uses 16 frames for fine-tuning in both pretraining conditions.Source: §E.7 · p. 23
  • The visualization shows Top-5 retrieval results for the same text query under first-person and third-person pretraining.Source: Figure 13 · p. 23, §E.7 · p. 23

Related papers

Loading 2206.01670v2…