PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

Yanghao Li, Tushar Nagarajan, Bo Xiong, Kristen Grauman

arXiv:2104.07905v1cs.CV

TL;DR

Egocentric datasets are limited in scale and diversity, while third-person pre-training misses distinctive egocentric properties. Ego-Exo distills egocentric cues inferred from third-person video into pre-training, improving results on egocentric benchmarks, including state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

  • Problem

    Egocentric datasets are small and lack diversity, while third-person-only pre-training faces a substantial domain mismatch with egocentric video.

    Source: §1 · p. 1

  • Method

    Ego-Exo uses inferred egocentric signals as knowledge-distillation losses alongside third-person action classification during pre-training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Ego-Exo improves over Kinetics-pretrained models by +3.26 mAP on Charades-Ego and achieves state-of-the-art results on EPIC-Kitchens-100.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pre-trained model is a drop-in replacement for a standard Kinetics-pretrained video model for downstream egocentric understanding tasks.

    Source: §1 · p. 2, §5 · p. 8

    Pre-training assumes access to off-the-shelf egocentric task models to generate pseudo-labels for third-person videos.

    Source: §3.2 · p. 3, §3.3 · p. 4

Abstract

from arXiv · show

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

1 Facebook AI Research 2 UT Austin

Ego-Exo targets the tension between limited, narrow egocentric datasets and third-person pretraining’s viewpoint mismatch by distilling egocentric cues inferred from third-person video. It reports gains on Charades-Ego and state-of-the-art performance on EPIC-Kitchens-100.

  • Wearable-camera video differs through viewpoint, body and head motion, and attention-driven focus on hands, objects, and faces.Source: §1 · p. 1
  • Egocentric datasets are smaller and less diverse, while third-person-only pretraining faces viewpoint mismatch and prior bridging methods require paired videos.EPIC-Kitchens-100 has 90k clips versus 650k in Kinetics-700, and is limited to kitchen scenes.Source: §1 · p. 1
  • The approach assumes third-person video can inform first-person understanding, since it may contain person-centered cues such as hand–object interactions and attentional signals.Source: §1 · pp. 1–2
  • Ego-Exo distills egocentric signals from third-person video into the backbone, using object, hand–object interaction, and egocentricity prediction tasks alongside action classification.Source: Figure 1 · p. 1, §1 · p. 2
  • The learned model is designed as a drop-in encoder and requires neither paired nor activity-labeled egocentric videos during pre-training.Source: §1 · p. 2
  • On Charades-Ego, Ego-Exo improves over Kinetics-pretrained models by +3.26 mAP and achieves state-of-the-art results on EPIC-Kitchens-100.Source: §1 · p. 2

2. Related Work

Prior egocentric work spans many tasks and architectures, but commonly uses encoders pre-trained on third-person video. Ego-Exo differs by adding inferred egocentric cues to third-person pre-training without requiring paired, labeled viewpoints.

  • Egocentric research covers recognition, anticipation, summarization, pose, and gaze across varied architectures, whose encoders are commonly pre-trained on third-person video.Source: §2 · p. 2
  • Ego-Exo adds egocentric losses to exocentric pre-training and then fine-tunes the backbone for a downstream egocentric task.Source: §2 · p. 2, Figure 2 · p. 3
  • Prior viewpoint-bridging approaches use embedding learning, image generation, or domain adaptation, and generally require paired videos recorded together or sharing instance labels.Source: §2 · p. 2
  • Earlier video distillation uses alternate modalities or object-level information, whereas Ego-Exo uses inferred egocentric weak labels alongside third-person labels.Source: §2 · p. 2
  • Prior cue-based models use egocentric labels to detect signals such as gaze, active objects, and hands in contact; Ego-Exo instead uses predicted cue labels as auxiliary losses in third-person models.Source: §2 · p. 2

3. Ego-Exo Approach

Ego-Exo augments third-person video pre-training with automatically generated egocentric cues, aiming to retain large-scale action learning while steering representations toward egocentric properties. Its auxiliary tasks target viewpoint, manipulated objects, and hand-object interaction regions, and the resulting backbone remains compatible with standard video models.

3.2. Ego-Exo pre-training

  • Ego-Exo retains third-person action classification while adding automatically generated egocentric pseudo-labels and auxiliary losses to steer features toward egocentric cues.The pseudo-labels range from scalar scores to categorical labels and spatiotemporal attention maps.Source: §3 · p. 3, §3.2 · p. 4

3.1. Video model pre-training

  • Standard pre-training encodes third-person video clips, predicts action labels, and retains the backbone for fine-tuning on a downstream egocentric task.The standard procedure uses a large-scale third-person dataset before fine-tuning with labeled egocentric video.Source: §3.1 · p. 3

3.3. Auxiliary egocentric tasks

  • Ego-Score uses a binary viewpoint classifier trained on Charades-Ego to generate video-level egocentricity scores, which serve as soft targets for an auxiliary head.The classifier uses viewpoint labels, not paired-view information or egocentric activity labels.Source: §3.3 · p. 4
  • Object-Score trains representations to predict manipulated objects using video-level scores formed by averaging an off-the-shelf ImageNet object recognizer’s logits across frames.The Object-Score is predicted with a knowledge-distillation loss.Source: §3.3 · p. 4
  • Interaction-Map uses detected hands and interacting objects to create spatiotemporal maps, with prediction heads trained to reproduce those maps from clip features.The maps are formed from per-frame bounding-box overlaps; map prediction is used instead of direct bounding-box detection because it better fits standard video backbones and feature learning.Source: §3.3 · pp. 4–5

3.4. Ego-Exo training and evaluation

  • The final pre-training loss combines third-person action classification with weighted losses for Ego-Score, Object-Score, and Interaction-Map, using weights selected by cross-validation.Soft targets preserve information about perceived egocentric cues, including cues that are negative for the actual label.Source: §3.4 · p. 5
  • Auxiliary training changes the learned features, not the network architecture, so the pre-trained model can replace a traditional video encoder directly.The authors describe the encoder as applicable to various egocentric video tasks.Source: §3.4 · p. 5

4. Experiments

Across pre-training comparisons, ablations, and benchmark evaluations, Ego-Exo’s auxiliary egocentric tasks improve transfer from third-person video, with strong results on Charades-Ego and EPIC-Kitchens-100.

4. Experiments

  • Strategy comparisons use a shared Slow-only ResNet-50 backbone; evaluation reports Charades-Ego mAP and EPIC top-1/top-5 accuracy.The compared pre-training strategies include Scratch, Third-only, First-only, Domain-adapt, Joint-embed, and Ego-Exo.Source: §4 · p. 5, §4.1 · p. 6

4.1. Ego-Exo pre-training

  • Ego-Exo achieves the best validation results, improving over Third-only by +1.54% mAP on Charades-Ego and +1.64%/+1.97% on EPIC verbs/nouns.First-only performs poorly despite using the largest available egocentric dataset, indicating that scale alone is insufficient.Source: §4.1 · p. 6, Table 1 · p. 6

4.2. Ablation studies

  • Adding Ego-Score improves both EPIC tasks, while Object-Score and Interaction-Map consistently improve all results; all three together yield the best pre-trained model.The authors describe the three scores as capturing complementary egocentric information despite differences in structure and semantics.Source: §4.2 · pp. 6–7, Table 2 · p. 6
  • Class-wise gains concentrate on active object-manipulation classes, while most under-performing classes are only marginally worse.Figure 4 shows the 35 most- and least-improved classes out of 157.Source: §4.1 · p. 6, Figure 4 · p. 7
  • Adding auxiliary losses during fine-tuning improves both baseline and Ego-Exo models, with larger gains for Ego-Exo, especially on Charades-Ego.The authors suggest pre-training the distillation heads to detect hands and objects may explain the larger gain.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. Comparison with state-of-the-art

  • On Charades-Ego, Ego-Exo and Ego-Exo* exceed the strongest baseline by +2.11% and +3.26% mAP; on EPIC-Kitchens-100, Ego-Exo*-R101 outperforms all existing state-of-the-art methods.Ego-Exo also beats Epic-Fusion on top-1 for all categories despite its additional flow and audio inputs.Source: Table 4 · p. 7, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion

Ego-Exo embeds egocentric signals in third-person video pre-training, aiming to combine the scale and diversity of third-person datasets with representations for egocentric understanding. It serves as a drop-in replacement for standard Kinetics pre-training and achieves state-of-the-art egocentric action-recognition results on two benchmarks.

5. Conclusion

  • Ego-Exo achieves state-of-the-art egocentric action-recognition results on Charades-Ego and EPIC-Kitchens-100.The experiments show it can replace the standard Kinetics-pretrained video model.Source: §5 · p. 8
  • The method embeds key egocentric signals into third-person video pre-training to use the scale and diversity of those datasets for egocentric understanding.Source: §5 · p. 8
  • Future work could explore alternate distillation tasks and instance-specific distillation losses to maximize the impact of third-person data.Source: §5 · p. 8

Supplementary Material

The supplement collects implementation details, additional evaluations and ablations, and qualitative results for the paper's models and experiments.

Supplementary Material

  • Sections S1–S3 provide implementation details for the pretrained egocentric models, Kinetics pre-training, and downstream egocentric fine-tuning.Source: Supplementary Material · p. 12
  • Section S4 reports additional results on EPIC-Kitchens, Charades-Ego, and Something-Something v2.Source: Supplementary Material · p. 12
  • Section S5 presents ablations of the Interaction-map model, pretrained-model and feature use, dataset scale, implicit pairing, and backbone choice.Source: Supplementary Material · p. 12
  • Section S6 adds Ego-Score distributions, qualitative examples, and class-wise breakdowns of auxiliary-task improvements; a supplementary video animates the examples.Source: Supplementary Material · p. 12

S1. Details: Pre-trained egocentric models M τ

The auxiliary egocentric models estimate egocentricity, object recognition, and hand-object interactions. The resulting auxiliary losses do not require network modification, allowing the pretrained model to serve as a drop-in replacement for downstream tasks.

S1. Details: Pre-trained egocentric models M τ

S2. Details: Pre-training on Kinetics

Kinetics pre-training follows the same training recipe for Slow-only and SlowFast backbones across methods, with models trained from scratch for 200 epochs using synchronized SGD.

S2. Details: Pre-training on Kinetics

S3. Details: Fine-tuning on Ego-datasets

The section reports fine-tuning settings for Charades-Ego, EPIC-Kitchens, and EPIC-Kitchens-100.

S3. Details: Fine-tuning on Ego-datasets

S4. Additional results

Additional results report ensemble and single-model performance on EPIC-Kitchens benchmarks, a Charades-Ego multi-task setting, and Something-Something V2.

S4. Additional results

  • The EPIC-Kitchens Ego-Exo ensemble consistently outperforms the Epic-Fusion ensemble across categories.Source: Table S1 · p. 13, S4. Additional results · p. 13
  • The Ego-Exo ensemble outperforms the EPIC-Kitchens-100 leaderboard’s best model across categories, with +5% on noun and +3.5% on action Overall Top-1.The single model also ranks first and performs better than the best leaderboard model in most categories.Source: Table S2 · p. 13, S4. Additional results · p. 13
  • With joint first- and third-person classification fine-tuning on Charades-Ego, Ego-Exo reaches 28.32 mAP versus SlowFast’s 25.06 mAP.Both use a ResNet-50 backbone.Source: S4. Additional results · p. 13
  • On Something-Something V2, Ego-Exo improves over Third-only from 59.49% to 60.41% accuracy.The paper notes potential impact beyond egocentric video for general interaction scenarios.Source: S4. Additional results · p. 13

S5. Additional Ablation studies

The additional ablations test distillation choices, egocentric supervision scale, pairing information, and backbone generality.

S5. Additional Ablation studies

  • Combining Hand-Map and Object-Map in Interaction-Map achieves the best results overall, while either map alone improves over Third-only.The results are averaged over three runs.Source: S5. Additional Ablation studies · p. 13, Table S3 · p. 14
  • Distillation outperforms directly fine-tuning auxiliary-task models as pre-trained models, which capture specific egocentric properties but are insufficient alone.The auxiliary models considered are the ego-classifier and object-recognition model; the hand-object detector backbone is incompatible with the video backbone.Source: S5. Additional Ablation studies · p. 14, Table S4 · p. 14
  • Appending embeddings from auxiliary-task models is less effective than Ego-Exo’s distillation scheme.The tested embeddings come from the auxiliary models’ global-pooling layers and are concatenated with ego-model features during fine-tuning.Source: S5. Additional Ablation studies · p. 14
  • Ego-Exo outperforms Third-only with both I3D and TSM backbones on Charades-Ego and EPIC-Kitchens.This supports the reported versatility across the tested backbone structures.Source: Table S6 · p. 15, S5. Additional Ablation studies · p. 15
  • With 10% of EPIC-Kitchens-100 training data, Ego-Exo improves over Third-only by +2.68% on verb and 1.99% on noun tasks.Ego-Exo outperforms Third-only at all tested dataset scales, although both perform worse with less egocentric fine-tuning data.Source: S5. Additional Ablation studies · pp. 14–15, Figure S1 · p. 14
  • Methods achieve similar results with or without implicit pairing information when training the Ego-Score classifier.The ablation uses one view from each Charades-Ego pair and does not use explicit pairing information.Source: S5. Additional Ablation studies · p. 15, Table S5 · p. 15

S6. Additional qualitative results

The qualitative results examine Ego-Score distributions, class-wise effects of auxiliary tasks, and examples ranked by egocentric scores.

S6. Additional qualitative results

Loading 2104.07905v1…