Source-linked AI summary
Egocentric Video-Language Pretraining
Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, Mike Zheng Shou
TL;DR
Existing VLP is built mainly on third-person video-text data, which leaves a domain gap for egocentric tasks. The paper builds EgoClip, EgoNCE, and EgoMCQ from Ego4D, and reports strong transfer across five egocentric downstream tasks.
Problem
Third-person VLP datasets leave a domain gap for egocentric video, while existing egocentric datasets are too small and domain-specific for egocentric VLP.
Method
The paper creates 3.8M-pair EgoClip, introduces EgoNCE with egocentric-aware sampling, and develops the closely matched EgoMCQ benchmark.
Results
The pretrained representation demonstrates strong performance on five egocentric downstream tasks across three datasets.
Takeaways & Limitations
EgoClip, EgoNCE, and EgoMCQ provide a framework for developing and transferring video-language representations in the egocentric domain.
Takeaways & Limitations
Text-video retrieval evaluation is unreliable because duplicated content can make top-ranked clips correct despite not being labeled ground truth.
Abstract
from arXiv · showhide
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.
1 Introduction
Existing video-language pretraining relies mainly on large-scale third-person datasets, leaving a domain gap for egocentric video. The paper addresses this gap with EgoClip, EgoNCE, and EgoMCQ, enabling evaluation across five egocentric downstream benchmarks.
- Existing video-language pretraining datasets are often edited third-person videos, creating a domain gap with first-person footage.
- Directly transferring existing VLP models to egocentric downstream tasks cannot fully exploit large-scale pretraining.
- Small-scale, domain-specific egocentric datasets previously made egocentric VLP difficult, whereas Ego4D provides massive-scale egocentric video with narrations.
- 3.8M clean first-person clip-text pairs form EgoClip, selected from Ego4D and covering diverse daily activities.
- EgoNCE adapts video-text contrastive learning to egocentric data, while EgoMCQ provides a closely matched 39K-question development benchmark.
- The pretrained representation is evaluated on five egocentric downstream benchmarks across three datasets.
2 Related Work
Prior work established video-language pretraining using large-scale datasets and joint- or dual-encoder architectures, while egocentric datasets remained smaller and domain-specific. Egocentric VLP extends these foundations with sampling designed for first-person video.
- Large-scale video-text datasets enabled VLP methods for video-text retrieval, question answering, and captioning.
- VLP architectures are commonly divided into joint encoders, which fuse modalities, and dual encoders, which encode them separately.
- The paper adopts Frozen’s dual-transformer architecture and extends InfoNCE into EgoNCE through egocentric-aware positive and negative sampling.
- Egocentric videos provide a natural perspective on daily activities but are expensive to collect, so prior datasets tend to be small-scale and domain-specific.
3 EgoClip: An Egocentric Video-Language Pretraining Dataset
EgoClip converts densely narrated Ego4D videos into clean clip-text pairs through filtering and contextual variable-length temporal windows. The strategy uses narration timing and scenario-dependent context to improve pairing quality.
- EgoClip sources 9,645 untrimmed Ego4D videos with timestamp-level narrations describing activities and object interactions.
- Data curation: The curation process filters videos with missing narrations, excludes Ego4D challenge validation and test videos, and retains both narrators’ annotations.
- Creation of clip-text pairs: Because Ego4D narrations have single timestamps rather than intervals, the method creates contextual variable-length clips around each narration.
- Creation of clip-text pairs: Each clip is centered on narration timestamp t_i, with duration β_i/α; β_i reflects local narration spacing and α controls context granularity.The average α across EgoClip videos is 4.9 seconds.
- Creation of clip-text pairs: The strategy uses longer context for slower scenarios such as watching television and shorter context for rapid activities such as harvesting crops.The cited examples report 352.9 seconds for watching television and 0.9 seconds for harvesting crops.
4 Video-Language Pretraining Model
The model uses a dual-encoder VLP architecture and extends InfoNCE with action-aware positives and scene-aware negatives to address egocentric video challenges. EgoNCE is designed for efficient transfer across egocentric downstream tasks, while retrieval evaluation is complicated by duplicated or semantically similar captions.
- Architecture: EgoVLP uses a dual encoder, with separate video and text encoders, to efficiently transfer video-language representations to egocentric tasks.The video encoder uses TimeSformer and the text encoder builds on DistillBERT.
- Baseline objective: InfoNCE treats matched video-text pairs as positives and all other within-batch combinations as negatives, with symmetric video-to-text and text-to-video losses.The video-to-text loss is explicitly formulated, while the text-to-video loss is defined symmetrically.
- Egocentric challenges: EgoClip presents two challenges: the same action can occur in different scenarios, while different actions in one scenario can look visually similar.Examples include unlocking a phone in different settings and typing or moving a mouse near a laptop.
- Evaluation limitation: Text-video retrieval evaluation is unreliable when duplicated or semantically similar captions make multiple top-ranked clips correct.The paper illustrates this issue with retrieval results where top clips are not the annotated ground truth but should still count as correct.
- EgoNCE: EgoNCE addresses these challenges with action-aware positive sampling and scene-aware negative sampling built on vanilla InfoNCE.The objective is presented as a general extension for egocentric video-text pretraining.
- EgoNCE: Action-aware positives group narrations sharing at least one noun and one verb after synonym merging, while scene-aware negatives use adjacent clips from the same video.The augmented batch includes each original sample’s adjacent hard-negative counterpart.
5 EgoMCQ: A Benchmark for Egocentric VLP Development
EgoMCQ is introduced as a development benchmark aligned with EgoClip to validate egocentric VLP dataset and model choices. It avoids unreliable retrieval evaluation through multiple-choice questions with inter-video and more challenging intra-video clip groupings.
- Motivation: EgoMCQ is designed to support validation and rapid exploration of EgoClip and EgoNCE because existing egocentric benchmarks are often domain-specific and single-modality.The benchmark is intended to align closely with the video-text pretraining task.
- Data source: The benchmark uses data excluded from EgoClip, removes videos sharing multiple views with EgoClip, and selects one annotator narration per video.Its clip pairing follows the same strategy as EgoClip for format consistency.
- Task design: Video-text retrieval was rejected because duplicated or semantically similar Ego4D captions make evaluation unreliable and difficult to deduplicate robustly.The paper therefore turns to a multiple-choice task for development evaluation.
- Grouping strategies: The inter-video setting groups five clips from different videos, whereas the intra-video setting groups five continuous clips from one video and is more challenging.The settings distinguish clips across scenarios or among temporally adjacent clips in the same scenario.
- Statistics: EgoMCQ contains 39K questions covering 198K narrations and 468 hours of video.The inter-video portion has 24K questions and 290.3 hours, while the intra-video portion has 15K questions and 178.3 hours.
6 Experiments
Experiments validate EgoClip, EgoNCE, and EgoMCQ through ablations and transfer across five egocentric benchmarks spanning three datasets. The model consistently outperforms relevant baselines, including large-scale third-person pretraining systems.
- Benchmarks and Settings: The evaluation covers five egocentric benchmarks across three datasets, including video-text retrieval, natural language query, action recognition, moment query, and object state change classification.The benchmarks span both video-text and pure-video tasks.
- Ablation Studies: EgoMCQ performance aligns with zero-shot EPIC-KITCHENS-100 results, supporting its use for validating design choices during Egocentric VLP development.The benchmark provides useful feedback despite downstream gains being smaller than EgoMCQ gains.
- Ablation Studies: Variable-length clips outperform fixed-length alternatives, while centering clips at timestamps and using a scale factor greater than 1 improve EgoClip construction.The proposed contextual varied clip length strategy outperforms a comparable-average-length alternative, and larger scale factors focus on densely labeled instantaneous actions.
- Ablation Studies: Jointly matching shared verbs and nouns improves EgoNCE over naive InfoNCE, and temporally hard negatives within one minute produce the best sampling result.The final EgoNCE combines the best positive and negative sampling settings.
- Comparisons with State-of-the-arts: 1.0 mAP and 5.9% nDCG improvements over JPoSE are achieved on EPIC-KITCHENS-100 while using fewer frames and input modalities.EgoClip pretraining also outperforms larger third-person pretraining datasets in zero-shot evaluation and maintains a fine-tuned margin.
- Comparisons with State-of-the-arts: Natural language query R@1 for IoU=0.3 rises from 5.45 to 10.84, while action recognition improves over Ego-Exo by 2.0% with fewer sampled frames.The pretrained model also achieves the best reported performance for moment query and object state change classification, including a 2.4% advantage over CC3M+WebVid-2M for OSCC.
- Comparisons with State-of-the-arts: Frozen pretrained on EgoClip with EgoNCE consistently improves over InfoNCE across all downstream tasks, supporting the effectiveness of the egocentric objective.This conclusion also validates the design decisions selected using EgoMCQ.
7 Conclusion, Limitations, and Societal Impacts.
The work pioneers egocentric video-language pretraining through a curated dataset, an egocentric contrastive objective, and a development benchmark, with strong representation demonstrated across five tasks. It also identifies long-term temporal dependencies in long Ego4D videos as an unresolved limitation.
- Conclusion: The paper creates EgoClip, EgoNCE, and EgoMCQ to establish a foundation for egocentric video-language pretraining.EgoClip contains 3.8M clip-text pairs, EgoNCE uses egocentric sampling strategies, and EgoMCQ supports efficient development.
- Conclusion: Strong representation is demonstrated across five tasks spanning three egocentric datasets.
- Limitations: The pretraining approach does not account for long-term temporal dependencies in long Ego4D videos.
- Conclusion: The study systematically addresses pretraining data, development evaluation, model design, and transferability for the 1st-person domain.
A.1 Pretraining dataset
EgoClip converts noisy, untrimmed Ego4D videos and timestamped narrations into a large-scale 1st-person pretraining dataset, while the accompanying design addresses curation, compression, and efficient loading.
- Motivation: Ego4D cannot be directly used for video-language pretraining because its videos are untrimmed, lack direct video-text pairs, and contain noisy data.The paper therefore treats dataset curation as a central design problem.
- Development benchmark: EgoMCQ is introduced as a development set aligned with EgoClip to support rapid exploration of the dataset and pretraining objective.
- Data cleaning: Noise reduction filters stereo and extreme-aspect-ratio videos, unsure narrations, and narrations shorter than three words.The reported filtering thresholds remove 1.3% stereo-video duration, 0.4% extreme-aspect-ratio duration, 4.0% unsure texts, and 0.9% short texts.
- Efficient processing: Video compression resizes inputs, chunks videos into segments up to 10 minutes, and loads only the segment containing each clip.One epoch over 3.8M video-text pairs costs 6 hours on 32 V100 GPUs.
B.3 Data analysis
EgoClip spans geographically diverse sources and 129 daily-life scenarios, with dense, varied narrations and a verb-noun deduplication strategy for visualized clip-text pairs.
- Geographic diversity: EgoClip covers 13 institutions across 9 countries, inheriting Ego4D’s geographic and participant diversity.The sources span Europe, Asia, America, and Africa.
- Scenario diversity: EgoClip covers 129 human daily scenarios, including household, outdoor, workplace, and leisure activities.The distribution is long-tailed: crafting/knitting/sewing/drawing/painting has 622K instances, while hair and makeup stylist has 35.
- Clip analysis: The 2.9K pretraining videos have varying clip frequencies, averaging 13.4 clips per minute.The reported frequency ranges from 0.06 to 175.8 narrations per minute.
- Narration analysis: EgoClip narrations average 9.39 words and contain 116 verbs and 555 nouns after synonym merging.Each narration averages 1.84 nouns and 0.87 verbs.
- Visualizations: The figures visualize EgoClip clip-text pairs by sampling five frames per clip and using each narration as its caption.
- Deduplication: Deduplication assigns narrations tags from their first verb and noun, including synonyms, to improve fine-grained diversity among candidate captions.This addresses cases where sentence-level embeddings treat minimally different narrations as highly similar.
C.3 Data analysis
EgoMCQ is designed as a development benchmark close to EgoClip in geographic and scenario diversity, with inter-video and intra-video multiple-choice settings covering 198K narrations.
- Benchmark scale: EgoMCQ contains 39K questions covering 198K narrations and 468 hours of video.It includes 24K inter-video questions over 290.3 hours and 15K intra-video questions over 178.3 hours; five-option clips average 34.2 seconds.
- Geographic diversity: EgoMCQ covers 13 institutions and aligns with EgoClip’s geographic diversity.
- Scenario diversity: EgoMCQ covers 74 scenarios, including 71% of EgoClip’s scenarios plus three scenarios absent from EgoClip.Cooking is the largest scenario with 49K clips, while Bus is the smallest with 6 instances.
- Narration analysis: EgoMCQ narrations average 3.15 nouns and 0.97 verbs, with hand and pick as the most frequent noun and verb.Hand appears in 86K instances and pick in 28K instances.
- Benchmark settings: The benchmark includes intra-video and inter-video settings illustrated with text questions and five candidate clips.Correct and incorrect candidate narrations are visually distinguished in the benchmark examples.
D.1 Architecture: Frozen-in-time [3]
The model uses separate video and text encoders and adapts contrastive pretraining to egocentric data through action-aware positives and temporally adjacent negatives. Downstream evaluations replace baseline features or objectives with pretrained representations under task-specific settings.
- Architecture: The video encoder uses Timesformer to tokenize spatiotemporal RGB patches, while DistillBERT encodes each narration into the final text representation.
- Pretraining objective: EgoNCE extends InfoNCE with two sampling strategies for egocentric pretraining.Positive samples are selected using shared nouns and verbs, while negatives come from temporally adjacent instances in the same video.
- Downstream settings: The downstream configurations use task-specific objectives, including Multi-instance Maxmargin loss for EPIC-KITCHENS-100 retrieval and InfoNCE for Charades-Ego action recognition.
- Downstream settings: For Ego4D Natural Language Query, pretrained video and text encoder features replace the baseline SlowFast and BERT inputs because end-to-end training is difficult for long clips.
- Downstream settings: For Ego4D Moment Query and Object State Change Classification, VLP video features replace baseline features, with 16-frame inputs and cross-entropy used for the latter.
- Results: EgoClip and EgoNCE reach optimum Natural Language Query performance, while EgoClip pretraining yields nearly convergent EPIC-KITCHENS-100 retrieval performance after few epochs.
E.6 Results on test set of Moment Query
EgoClip with EgoNCE achieves the strongest reported Moment Query results over the baseline, and qualitative retrieval compares its outputs with 3rd-person pretraining under the same text query.
- Results: 3.78% on R@1 and 4.65% on Avg mAP over the baseline are achieved by EgoClip with EgoNCE on the Moment Query test set.
- Results: Moment Query results are reported using Recall and mAP metrics on the test set.
- Visualization: Figure 13 compares Top-5 retrieval results from 1st-person and 3rd-person pretraining for the same text query, with correlation scores shown for each result.