PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World

Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, Yu Qiao

arXiv:2403.16182v3cs.CV

TL;DR

EgoExoLearn addresses the challenge of studying how people map demonstrated actions across asynchronous egocentric and exocentric views. It pairs demonstrations with egocentric task recordings and multimodal annotations, and provides four benchmarks for investigating cross-view activity bridging.

  • Problem

    Studying how humans bridge asynchronous procedural actions across egocentric and exocentric views matters because following demonstrations can help when physical trials are costly.

    Source: §1 · p. 1

  • Method

    EgoExoLearn pairs demonstration videos with egocentric recordings of people replicating the tasks and provides gaze, multimodal annotations, and benchmarks.

    Source: §1 · pp. 1–2

  • Results

    EgoExoLearn provides four novel benchmarks for investigating how asynchronous cross-view activities can be bridged.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The dataset supports research into how gaze and hand-associated annotations can be used to bridge asynchronous procedural actions across views.

    Source: §5 · p. 8

Abstract

from arXiv · show

Being able to map the activities of others into one's own point of view is one fundamental human skill even from a very early age. Taking a step toward understanding this human ability, we introduce EgoExoLearn, a large-scale dataset that emulates the human demonstration following process, in which individuals record egocentric videos as they execute tasks guided by demonstration videos. Focusing on the potential applications in daily assistance and professional support, EgoExoLearn contains egocentric and demonstration video data spanning 120 hours captured in daily life scenarios and specialized laboratories. Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints. To this end, we present benchmarks such as cross-view association, cross-view action planning, and cross-view referenced skill assessment, along with detailed analysis. We expect EgoExoLearn can serve as an important resource for bridging the actions across views, thus paving the way for creating AI agents capable of seamlessly learning by observing humans in the real world. Code and data can be found at: https://github.com/OpenGVLab/EgoExoLearn

1. Introduction

Humans can follow demonstrations across viewpoints, but existing approaches and datasets do not capture realistic, asynchronous procedural activities. EgoExoLearn introduces paired demonstration and egocentric recordings with gaze, multimodal annotations, and benchmarks for studying this ability.

1. Introduction

  • Existing datasets do not support studying how people bridge procedural activities recorded in realistic ego- and exocentric views at different times.The introduction notes that existing real-world datasets are limited in quality or scale, while prior ego-exo datasets record views in the same environment and synchronously.Source: §1 · p. 1
  • EgoExoLearn pairs demonstrations with egocentric videos of participants performing the same task in a different environment.The dataset is designed to emulate asynchronous demonstration following.Source: §1 · p. 1, Figure 1 · p. 2
  • The dataset contains 747 video sequences spanning 120 hours, including gaze signals in the egocentric videos.The authors target daily assistance and professional support, with activities ranging from daily food-making to specialized laboratory experiments.Source: §1 · p. 2
  • EgoExoLearn defines benchmarks for cross-view association, action understanding, referenced skill assessment, and referenced video captioning.The benchmarks are annotated and supported by baseline implementations, and the paper also explores gaze in these tasks.Source: §1 · p. 2

2. Related Work

Prior ego-exo resources often lack the scale, quality, or asynchronous demonstration-following setup needed to study high-level procedural actions. EgoExoLearn complements them with a larger dataset, calibrated gaze, and fine-grained annotations.

Ego-exo datasets

  • Existing ego-exo datasets often use the same environment or synchronized views, whereas EgoExoLearn studies demonstration following across environments and at different times.The paper positions its setting as complementary to datasets focused on single actions, fixed-camera views, or 3D pose and activity understanding.Source: §2 · p. 2
  • EgoExoLearn is described as 100x larger than AE2 and adds gaze and fine-grained multimodal annotations for task-centric procedural understanding.AE2 combines existing datasets with a newly collected ego tennis dataset, but its largest subset has 322 clips and it focuses on clip-level actions.Source: §2 · p. 2

Egocentric and ego-exo video understanding

  • The demonstration-following setting complements Ego4D and can benefit from its knowledge and representations.The paper evaluates joint view-invariant learning, domain adaptation, and knowledge distillation in its benchmarks.Source: §2 · p. 3

Egocentric gaze

  • EgoExoLearn provides calibrated gaze positions for all egocentric videos, enabling study of gaze in cross-view understanding.The paper notes that large-scale egocentric gaze datasets were previously absent.Source: §2 · p. 3

3. Dataset

EgoExoLearn records people following demonstrations across daily-life and laboratory tasks, then provides layered language, hand-associated, gaze, and skill annotations. Its collection and annotation design supports comparison of procedural actions across views.

3.1. Data Collection

  • The dataset covers five daily tasks and three specialized laboratory tasks, recorded in four kitchens and three labs.Participants learn from selected exocentric demonstrations before replicating the task with gaze-calibrated glasses; they may revisit the demonstration during recording.Source: §3.1 · p. 3
  • 432 egocentric videos span 96.5 hours, while 315 demonstration videos span 23.5 hours.Demonstrations are shorter on average because they are edited to remove repeated steps, while egocentric recordings capture the full procedure.Source: §3.1 · p. 3

3.2. Annotation

  • Coarse annotations mark timed middle-level task steps with action labels and descriptions of what is done, how, and why.The dataset defines 39 coarse action categories and averages 41.2 coarse annotations per video, each averaging 21.5 seconds.Source: §3.2 · p. 4
  • Fine-level annotations describe atomic actions, including the hand and object involved and why they are used.The annotations are translated and parsed into hand-associated verb and noun labels, with 95 verb and 254 noun categories.Source: §3.2 · p. 4
  • Pairwise skill judgments yield 34,239 valid video pairs after filtering for agreement among annotators.Four annotators label each of 40,191 pairs, and pairs with fewer than three consistent opinions are removed.Source: §3.2 · p. 4

3.3. Statistics & Comparisons

  • EgoExoLearn adds temporal-bounded captions, cross-view associations, and multi-label video segments to a demonstration-following dataset.The authors also highlight hand-specific verb and noun labels and calibrated gaze as resources for analyzing how people bridge views.Source: §3.3 · p. 4

4. Dataset Properties & Benchmarks

EgoExoLearn pairs egocentric recordings of people following exocentric demonstrations in different environments with fine-grained annotations and gaze. Its benchmarks assess how models bridge asynchronous actions across views through association, action understanding, and referenced skill assessment.

4.1. Dataset Properties

  • EgoExoLearn’s demonstration-following setting has participants perform the same task as an exocentric demonstration, but in a different environment.The paper presents this as an emulation of human observational learning and a resource for embodied agents learning across perspectives.Source: §4.1 · p. 4
  • The dataset includes fine-grained multimodal annotations, including hand-specific verb and noun labels, temporally bounded captions, and calibrated gaze signals.These annotations support analysis of how people bridge ego- and exocentric activities.Source: §4.1 · pp. 4–5

4.2. Benchmarks

  • The paper introduces four benchmarks: cross-view association, action understanding, referenced skill assessment, and referenced captioning; action understanding includes anticipation, planning, and segmentation.It also explores gaze and reports that the dataset is split to eliminate annotation leakage across benchmarks.Source: §4.2 · p. 5

4.2.1 Cross-view association

  • Association tests whether a model can match the same semantics across ego and exo views, in both ego-to-exo and exo-to-ego directions.Ground-truth pairs are built through semantic-aware matching and verification; pairs are not provided for training.Source: §4.2.1 · p. 5
  • In association, gaze-cropped ego-only training substantially improves over center crops, hybrid training is strongest over single-view models, and single-view models struggle across views.The benchmark retrieves the matching video from 20 candidates and evaluates averaged Top-1 accuracy.Source: §4.2.1 · pp. 5–6, Table 3 · p. 6

4.2.2 Cross-view action anticipation & planning

  • Co-training with both views achieves the best performance in all action anticipation and planning setups, while gaze markedly diminishes the cross-view performance disparity.UDA and KD improve over direct zero-shot inference; the reported comparisons are on the test set.Source: §4.2.2 · p. 7, Table 4 · p. 7

4.2.3 Cross-view referenced skill assessment

  • Referenced skill assessment asks which of two egocentric clips shows higher skill, using an exocentric demonstration of the same action as a reference.Pairs are assessed by four annotators, and pairs with fewer than three agreements are removed before forming video triplets.Source: §4.2.3 · p. 8
  • In referenced skill assessment, relation-network and triplet-loss methods benefit from the exocentric reference, and adding gaze yields the best performance, though the reference gain is marginal.The authors conclude that current models may still struggle to bridge asynchronous activities across views.Source: §4.2.3 · p. 8, Table 5 · p. 8

5. Conclusion

EgoExoLearn provides a realistic platform for studying asynchronous ego–exo activity bridging through four benchmarks, while benchmark results reveal substantial room for improvement. Its multimodal annotations also open directions for studying gaze and hand-associated information.

5. Conclusion

  • EgoExoLearn combines egocentric recordings of replicated demonstrations across different environments and times with multimodal annotations in a platform for four novel benchmarks.The benchmarks investigate bridging asynchronous ego–exo activities.Source: §5 · p. 8
  • Benchmark results show weaknesses in current models' ability to bridge asynchronous ego–exo activities, leaving significant room for future work.Source: §5 · p. 8
  • Gaze and hand-associated annotations offer directions for further research on bridging activities across views.Source: §5 · p. 8

S6. Additional Benchmark Details

The association benchmark evaluates whether ego- and exo-view videos of the same action can be matched, in either query direction. Its baseline learns from unpaired training videos, while validation and test pairs are constructed and manually verified.

S6.1.1 Detailed task definition

S6.1.2 Implementation details

  • The co-trained dual encoder aligns videos and narrations with contrastive loss; inference selects the candidate with highest cosine similarity, with an added gaze branch for fusion.The original-video feature cross-attends to the gaze-cropped feature at transformer blocks 5, 8, and 11.Source: S6.1.2 Implementation details · p. 10, Figure S5 · p. 10

S6.1.3 Annotation details

  • Validation/test pairs are built through scenario, noun-and-verb, and sentence matching, followed by negative sampling and two rounds of manual verification.The sets contain 868 validation and 2,200 test pairs; training pairs are not provided.Source: S6.1.3 Annotation details · p. 10

S6.2. Cross-view action anticipation & planning

The benchmark tests action anticipation and planning across ego and exo views, from direct transfer to approaches that use information from both views. Results show cross-view transfer is difficult, while gaze inputs and cross-view training methods improve performance.

S6.2.1 Detailed task definition

  • Anticipation forecasts the verb and noun of the next fine-level action one second ahead, while planning forecasts the next eight coarse-level actions.Anticipation uses class-mean Top-5 recall; planning uses ED@K and evaluates five predicted sequences.Source: S6.2.1 Detailed task definition · pp. 10–11

S6.2.2 Implementation details

  • Models use a two-second video context with five sampled frames and task-specific prediction heads for anticipation and planning.Anticipation predicts verb or noun categories; planning predicts category-by-step classes for K = 8.Source: S6.2.2 Implementation details · p. 11
  • The benchmark compares zero-shot transfer, UDA without target-view labels, teacher-to-student knowledge distillation, and co-training on both views.These settings represent increasing degrees of cross-view information use.Source: S6.2.2 Implementation details · pp. 11–12, Figure S7 · p. 12

S6.2.3 Annotation details

  • Anticipation annotations retain 19 verb and 31 noun categories after alignment and long-tail filtering; planning uses 27 coarse-level classes.Fine-level planning labels were not used because environmental constraints and unskilled performance produced substantial action variation.Source: S6.2.3 Annotation details · p. 12

S6.2.4 Additional results

  • Gaze-cropped inputs improve zero-shot cross-view results, and UDA, KD, and co-training provide further gains, though improvement varies by setting.Table S6 reports validation performance; its gray cells indicate cross-view results.Source: Table S6 · p. 12, S6.2.4 Additional results · p. 13

S6.3. Cross-view referenced skill assessment

The referenced skill benchmark ranks two egocentric performances using an exocentric expert demonstration. Pairwise annotations complement self-assessment, and supplementary results show that demonstration-reference gains vary with the feature backbone.

S6.3.1 Detailed task definition

S6.3.2 Implementation details

  • Triplet loss treats the exo clip as an anchor, pulling the higher-skill ego clip closer than the lower-skill clip; a relation network compares concatenated ego–exo features.Both approaches build on pairwise ranking, with I3D and VideoMAE used as feature encoders.Source: S6.3.2 Implementation details · p. 13

S6.3.3 Annotation details

  • Pairwise rankings use four criteria, and 40% differ from rankings based on subjects' self-evaluations; the dataset retains challenging pairs using four raters.It includes gaze and exo demonstrations and six action categories.Source: S6.3.3 Annotation details · pp. 13–14

S6.3.4 Additional results

  • VideoMAE features improve overall ranking performance, but exo-reference gains are less significant than with I3D features.The authors suggest that intrinsic properties of the extracted features may account for the variation.Source: S6.3.4 Additional results · p. 14, Table S8 · p. 14

S6.4. Cross-view action segmentation

Cross-view action segmentation classifies video frames and delineates procedural steps, with benchmarks for using information from another viewpoint. Baselines benefit from gaze data and cross-view assistance, with Co-Training tending to outperform Knowledge Distillation and Unsupervised Domain Adaptation.

S6.4.1 Detailed task definition

S6.4.2 Implementation details

  • The implementation uses I3D temporal features and SSTDA code, downsampling features and labels by five; training combines classification and smoothing losses and uses Adam at 1e-3 for 150 epochs.The cross-view settings follow those used for action anticipation and planning, adapted for dense prediction.Source: S6.4.2 Implementation details · p. 14

S6.4.3 Annotation details

  • For cross-view action segmentation, annotations derived from coarse-level annotations form non-overlapping temporal segments by placing boundaries at centers of overlapping portions and adding “no action” background segments to uncovered regions.The egocentric train/validation/test split contains 173/57/85 videos; the exocentric split contains 210/24/32.Source: S6.4.3 Annotation details · p. 14

S6.4.4 Experimental results

  • Gaze improves performance in ego-only and cross-view settings, while cross-view assistance improves segmentation; Co-Training tends to outperform Knowledge Distillation, which outperforms Unsupervised Domain Adaptation.Without assistance, models perform well only when the test data are from the same view.Source: Table S9 · p. 15, S6.4.4 Experimental results · p. 15

S6.5. Cross-view referenced video captioning

Cross-view referenced captioning tests whether exocentric videos can help caption egocentric videos when egocentric data are limited. Both co-training and referenced-training baselines outperform single-view models.

S6.5.1 Detailed task definition

  • At inference, the model captions test egocentric videos using exocentric videos as references, with 0-shot, 1-shot, and 2-shot settings.Training uses limited egocentric videos alongside a larger set of exocentric videos paired with fine-grained descriptions.Source: S6.5.1 Detailed task definition · p. 15

S6.5.2 Annotation

  • The annotation uses 1,000 egocentric training videos and 6,270 referenced exocentric videos; validation and test sets contain 8,181/2,143 and 18,243/4,930 videos, respectively.Each pair gives egocentric videos followed by referenced exocentric videos.Source: S6.5.2 Annotation · p. 15

S6.5.4 Results

  • Co-training and referenced-training models outperform single-view models, indicating that exocentric videos can improve egocentric captioning when egocentric data are limited.Table S10 reports cross-view referenced captioning performance; referenced-training uses one- or two-shot exocentric references.Source: Table S10 · p. 16, S6.5.4 Results · p. 16

S6.5.3 Implementation details

  • The baseline prepends reference videos to the input and supplies their captions as text-decoder prompts, selecting references with the cross-view association network.It uses a Flamingo-style captioning model.Source: S6.5.3 Implementation details · p. 16

S6.6. Zero-shot action recognition

The zero-shot action recognition benchmark evaluates verb and noun classification using CLIP-style prompts and features. Its reported results show weaker performance on egocentric than exocentric videos, especially for Top-1 accuracy.

S6.6. Zero-shot action recognition

S6.6.1 Annotation

  • The evaluation is closed-set and does not test cross-view settings; egocentric and exocentric categories need not be consistent across their validation and test sets.The respective ego and exo verb/noun validation and test sizes are 14.4k/32.6k/20.2k/44.8k and 4.2k/10.4k/5.7k/13.1k.Source: S6.6.1 Annotation · p. 16

S6.6.2 Implementation details

  • The method samples each clip’s center frame and extracts visual and text features with OpenAI CLIP, using 16 prompts for each verb and noun subset.The prompts are used for zero-shot classification.Source: S6.6.2 Implementation details · p. 16

S6.6.3 Experimental results

  • Zero-shot performance on egocentric videos is worse than on exocentric videos on validation and test sets, particularly for Top-1 accuracy.The authors identify this as a limitation in cross-view action understanding for the current method.Source: S6.6.3 Experimental results · p. 16

S6.7. Fine-tuned action recognition

Fine-tuned action recognition is framed as closed-set multilabel classification and evaluated with class-wise macro mAP. MViT-S outperforms SlowFast-R50 in reported performance and generalization.

S6.7.1 Detailed task definition

S6.7.4 Experimental settings and results.

S6.7.2 Annotation

  • Ego and exo annotations maintain consistent categories across splits; the ego set has 81 verb and 211 noun categories, while the exo set has 69 verb and 183 noun categories.Train/validation/test sizes are 36k/8k/18k for ego and 6.2k/2.1k/4.8k for exo.Source: S6.7.2 Annotation · p. 17

S6.7.3 Implementation details

  • The benchmark uses Kinetics-pretrained SlowFast-R50 and MViT-Small backbones, uniformly samples action-clip frames, and trains multilabel classification with standard cross-entropy.Training hyperparameters are listed in Table S14.Source: S6.7.3 Implementation details · p. 17, Table S14 · p. 17

S7. Additional Dataset Details

The supplementary details describe the dataset’s language annotations, video and gaze processing, ethical review, and multi-step daily and laboratory tasks. They also explain how the annotations support captioning and hand-action analysis.

S7.1. Language annotation

  • Annotations have coarse and fine levels with manually marked temporal boundaries; their average lengths are 21.5 seconds and 4.6 seconds, respectively.Annotators describe daily tasks from their own knowledge and receive additional experiment and terminology training for laboratory tasks.Source: S7.1. Language annotation · pp. 17–18
  • A web interface lets annotators mark video segments, assign categories, and describe them in their mother language; non-English annotations are translated and manually checked.Unsatisfactory translations are translated again with Google Translation API.Source: Figure S10 · p. 18, S7.1. Language annotation · p. 18
  • A rule-based iterative process extracts hand-associated verbs and nouns, masks left/right mentions to disambiguate “left” as a verb, and manually reviews errors; the figure shows the top 30 categories.Source: S7.1. Language annotation · p. 18

S7.2. Post-processing.

  • Videos are standardized to 25 fps, while 120 Hz gaze samples are time-aligned to the egocentric camera, assigned to the closest frame, and averaged per frame.Source: S7.2. Post-processing · p. 18
  • The dataset introduces a demonstration-following setting, and its longer language annotations than existing egocentric datasets support use in captioning benchmarks.Figure S14 pairs video frames with fine-level annotations.Source: S7.2. Post-processing · p. 19, Figure S14 · p. 20

S7.3. IRB approval

  • Data collection received IRB approval, and participants received consent forms and information sheets explaining capture, study purpose, and future data use.Source: S7.3. IRB approval · p. 19

S7.4. Tasks

  • The dataset covers five daily and three laboratory task types, collected in four kitchens and three laboratories, with long procedures intended to reflect complex real-life activities.The task table gives example procedures, and laboratory annotations describe reagents by visual appearance rather than scientific name.Source: S7.4. Tasks · pp. 19, 21, Table S15 · p. 23
Loading 2403.16182v3…