PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Egocentric Video Captioning

Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie

arXiv:2401.00789v4cs.CV

TL;DR

Egocentric video understanding has often focused on egocentric data, with limited use of large exocentric video collections and synchronized-pair requirements for some joint methods. EgoInstructor retrieves instructional videos as caption references, using automatically paired ego–exo data and EgoExoNCE. The paper reports strong retrieval results across seven benchmarks and captioning benefits from exocentric references.

  • Problem

    Egocentric-only methods overlook large exocentric video collections, while joint ego–exo learning has required time-synchronized videos collected in the same environment.

    Source: §1 · p. 1

  • Method

    EgoInstructor retrieves relevant third-person instructional videos for egocentric captioning, training cross-view retrieval on caption-aligned pseudo-pairs with EgoExoNCE and shared text features.

    Source: §1 · p. 2, §3.1.3 · pp. 4–5

  • Results

    The retrieval module demonstrates strong zero-shot cross-view and cross-modal retrieval ability across seven benchmarks, and results show benefits for egocentric captioning.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper concludes that retrieved exocentric videos can serve as references that benefit egocentric video captioning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Understanding human actions from videos of first-person view poses significant challenges. Most prior approaches explore representation learning on egocentric videos only, while overlooking the potential benefit of exploiting existing large-scale third-person videos. In this paper, (1) we develop EgoInstructor, a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos to enhance the video captioning of egocentric videos. (2) For training the cross-view retrieval module, we devise an automatic pipeline to discover ego-exo video pairs from distinct large-scale egocentric and exocentric datasets. (3) We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions. (4) Through extensive experiments, our cross-view retrieval module demonstrates superior performance across seven benchmarks. Regarding egocentric video captioning, EgoInstructor exhibits significant improvements by leveraging third-person videos as references. Project page is available at: https://jazzcharles.github.io/Egoinstructor/

1. Introduction

Egocentric video methods have often overlooked large exocentric video collections, while joint ego–exo learning has required synchronized pairs. EgoInstructor retrieves instructional videos as caption references, and the paper reports strong results across seven retrieval benchmarks and improved egocentric captioning.

1. Introduction

  • Prior egocentric-only methods are limited by dataset scale, while transfer from exocentric video faces viewpoint, camera-motion, continuity, and domain-shift discrepancies.Joint ego–exo representation learning has also required time-synchronized videos collected in the same environment.Source: §1 · p. 1
  • The paper motivates retrieval-augmented captioning by how people consult third-person demonstrations before unfamiliar tasks and recall their procedures while acting.It presents this as an alternative way to transfer knowledge from exocentric videos to egocentric understanding.Source: §1 · pp. 1–2
  • EgoInstructor retrieves relevant third-person instructional videos as references to assist egocentric captioning.The approach constructs pseudo ego–exo pairs at scale by aligning captions describing similar actions, then trains cross-view retrieval with EgoExoNCE and shared text features.Source: §1 · p. 2
  • The cross-view retrieval module demonstrates superior performance across seven benchmarks, while retrieved exocentric instructional videos benefit egocentric captioning.The benchmarks include EK100 Multi-instance Retrieval, Ego4d Multiple Choice Questions, YouCook2 video-text retrieval, and CharadesEgo cross-view video retrieval.Source: §1 · p. 2

2. Related Work

Related work spans egocentric representation learning, joint ego–exo understanding, and retrieval-augmented vision-language methods. EgoInstructor differs by learning a unified video-language space from unpaired datasets using captions and narrations.

Egocentric video understanding

  • Egocentric representation models have advanced, but the cited work says they lack enough generalization ability on exocentric videos.EgoVLP learns contrastively from paired egocentric videos and narrations, while LaViLa uses pseudolabelled egocentric videos.Source: §2 · p. 2

Ego-exo video understanding

  • Many ego–exo methods rely on simultaneously recorded views, adding data-collection difficulty and cost; alternatives use pseudo-labels, temporal alignment, or semantic categories.EgoInstructor instead learns joint representations on unpaired egocentric and exocentric datasets using captions and narrations, extending to video-language understanding.Source: §2 · p. 2

Retrieval-augmented models

  • Retrieval-augmented vision-language work has largely focused on images and CLIP-based retrieval, which the paper describes as suboptimal for video, especially across egocentric and exocentric views.EgoInstructor trains a cross-view retrieval module for exocentric-video retrieval in egocentric understanding.Source: §2 · p. 2

3. Methodology

EgoInstructor retrieves relevant exocentric instructional videos and uses their videos and captions as references when captioning an egocentric video. Its retrieval module is trained from automatically generated ego-exo pairs and then integrated into a multimodal captioning model.

3. Methodology

  • EgoInstructor retrieves K semantically relevant exocentric instructional videos and associated texts, then conditions caption generation on the egocentric video and retrieved samples.The methodology first trains the cross-view retrieval module using unpaired data, then describes the retrieval-augmented captioning model.Source: §3 · p. 2

3.1. Cross-view Visual Representation Alignment

  • The cross-view retrieval module learns to associate ego- and exo-videos with shared semantics using captioned or narrated videos, without manually annotated ego-exo pairings.Source: §3.1 · p. 2

3.1.1 Architecture Detail

  • Frozen egocentric and exocentric visual encoders provide features that a trained cross-view video encoder maps into an embedding space where semantically similar videos are close.The egocentric representations use VideoMAE and BERT, while exocentric visual features use InternVideo and the text features use the same BERT encoder.Source: §3.1.1 · p. 3

3.1.2 Automatic Ego-Exo Pair Generation

  • HowTo100M ASR transcripts are refined into descriptive captions, then ego-exo pairs are selected by matching scenario and noun-verb overlap across captions.Video-level refinement uses context to infer activity goals and replace pronouns with specific objects; narration boundaries are extended to mitigate temporal shifts.Source: §3.1.2 · pp. 3–4
  • The pipeline adds pseudo long-form training pairs by cropping longer clips and summarizing their accompanying narrations into one sentence with an LLM.These pairs are integrated with the original data, and long-form ego-exo pairing follows the same procedure as short clips.Source: §3.1.2 · p. 4

3.1.3 Retrieval Module Training and Inference

  • EgoExoNCE uses original video-text pairs, language-mined cross-view pairs, and pairs sharing a noun or verb as positives, with remaining minibatch pairs treated as negatives.The total loss combines video-to-text and symmetric text-to-video losses; shared action semantics implicitly pull ego- and exo-video embeddings closer.Source: §3.1.3 · pp. 4–5
  • At inference, the module ranks exocentric candidates using the average of egocentric-video-to-exocentric-video and egocentric-video-to-exocentric-text similarities.Source: §3.1.3 · p. 5

3.2. Retrieval-augmented Captioning

  • The captioning model concatenates retrieved exocentric and egocentric videos for visual encoding, while its text decoder uses retrieved captions as prompts and cross-attends to visual features.A perceiver resampler compresses visual features to a fixed-length bottleneck representation; gated cross-attention conditions text generation on the visual features.Source: §3.2 · p. 5

4. Experiments

Experiments evaluate the cross-view retrieval module across benchmarks and test whether retrieved exocentric instructional videos improve egocentric captioning. Results cover retrieval losses and encoders, captioning on Ego4d and EgoLearner, and qualitative examples.

4.1. Experimental Setups

  • The retrieval module uses Ego4d egocentric videos and HowTo100M instructional videos to build pseudo-pairs, then retrieves exocentric references for captioning at inference.Evaluation spans seven retrieval benchmarks; captioning is tested on Ego4d cooking and EgoLearner clips, while training uses Ego4d cooking captions.Source: §4.1 · pp. 5–6

4.2.1 Results on Cross-view Retrieval

  • Caption refinement consistently improves YouCook2 exocentric retrieval and raises EgoMCQ by 1%–2%, while EgoLearner Exo2Ego gains remain minimal in a similar recording environment.The authors conjecture that refinement aligns ego and exo caption formats, helping egocentric videos match broader, semantically rich captions.Source: §4.2.1 · pp. 6–7
  • EgoExoNCE improves InfoNCE accuracy by an average of 2.0% on EgoLearner and 4.3% on CharadesEgo, with further gains from refined captions; the final alignment outperforms the baseline on all benchmarks.EgoNCE improves egocentric benchmarks but yields only marginal gains on cross-view benchmarks.Source: Table 2 · p. 6, §4.2.1 · p. 7
  • Cross-view visual alignment improves most encoder results, including InternVideo by 15.9% and 7.7% on CharadesEgo.The comparison fixes the exocentric encoder as InternVideo and evaluates multiple egocentric encoders.Source: §4.2.1 · p. 7, Table 3 · p. 7

4.2.2 Results on Retrieval-augmented Captioning

  • On Ego4d cooking captions, one retrieved exocentric reference significantly improves all metrics over zero-shot; MPT reaches 0.468 CIDER at zero-shot, and refined captions yield LLaMA CIDER 1.768 vs. 1.665 with original ASR.Captioning performance gradually improves as the number of retrieved references increases; the model also outperforms reimplemented LaViLA-GPT2.Source: §4.2.2 · p. 7, Table 4 · p. 8
  • Transferred from Ego4d cooking to EgoLearner, EgoInstructor performs poorly at zero-shot, improves substantially with retrieved samples, and performs best with ground-truth paired videos.Random exocentric samples yield performance similar to zero-shot captioning.Source: §4.2.2 · pp. 7–8, Table 5 · p. 8

4.2.3 Qualitative Results

  • Retrieved videos generally match actions or objects in the egocentric clips, and their references help produce more accurate fine-grained object descriptions than captions without references.Examples include naming objects such as “pepper” or “bowl.”Source: §4.2.3 · p. 8, Figure 5 · p. 8

5. Conclusion

EgoInstructor retrieves semantically similar third-person instructional videos as references for egocentric video captioning. Its cross-view retrieval module shows strong retrieval ability across seven benchmarks, while quantitative and qualitative results show benefits for captioning.

  • EgoInstructor retrieves semantically similar third-person instructional videos as references to generate captions for egocentric videos.Source: §5 · p. 8
  • The cross-view retrieval module, trained with EgoExoNCE on curated ego- and exo-video pairs, demonstrates strong zero-shot cross-view and cross-modal retrieval across seven benchmarks.Source: §5 · p. 9
  • Quantitative and qualitative results show benefits from leveraging exocentric videos to enhance egocentric video captioning.Source: §5 · p. 9
Loading 2401.00789v4…