Source-linked AI summary
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, Karteek Alahari
TL;DR
Egocentric video understanding lacks a large paired resource connecting first- and third-person views. The paper constructs and annotates Charades-Ego from scripted paired recordings, producing a diverse dataset with temporal annotations and baselines for cross-view activity recognition. It positions the dataset for classification, localization, captioning, and cross-modal research.
Problem
The work targets the need to connect first- and third-person video understanding so third-person video resources can support egocentric research.
Method
The authors collect paired scripted recordings, temporally annotate both views jointly, and provide baselines for first- and third-person training and testing.
Results
Charades-Ego contains 68,536 activity instances in 68.8 hours of paired first- and third-person video, with temporal annotations and egocentric baselines.
Takeaways & Limitations
The dataset supports egocentric classification, localization, captioning, and research connecting first- and third-person activity recognition.
Abstract
from arXiv · showhide
In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Charades-Ego has temporal annotations and textual descriptions, making it suitable for egocentric video classification, localization, captioning, and new tasks utilizing the cross-modal nature of the data.
1. Introduction
Charades-Ego links first- and third-person activity understanding through paired recordings of scripted activities. It extends Charades with a large, diverse egocentric dataset sharing classes, scripts, and methodology.
- The dataset addresses egocentric action understanding while connecting it to abundant third-person video resources.
- Crowd workers recorded themselves performing prescribed activity scripts from both first- and third-person viewpoints.
- 68,536 activity instances span 68.8 hours of first- and third-person video in Charades-Ego.
- Charades-Ego shares activity classes, scripts, and methodology with Charades, which adds 82.3 hours of third-person video and 66,500 activity instances.
2. The Charades-Ego Dataset
Charades-Ego pairs third-person and first-person recordings of the same acted scripts, using shared and newly collected activities to build a diverse dataset across many homes.
- Data collection: Users recorded two videos of each script: one third-person performance and one same-script performance with a forehead-mounted camera.The paired recordings compromise between synchronized laboratory capture and broader data collection.
- Scripts and activities: 78.7% of Charades-Ego scripts are shared with the Charades training set, while 1000 additional scripts were collected for diversity.Most scripts originate from Charades, supplemented by newly collected scripts.
- Data collection: 60% of submitted videos contained activities featuring both hands, supported by incentivizing users to create their own camera mounts.Workers received $1.5 per video pair and a $0.5 bonus for using a home-made head mount.
- Dataset scale: 68,536 activity instances span 68.8 total hours across 157 activities, equally divided between 34.4 hours of third-person and first-person video.The videos were collected in various rooms across 112 homes worldwide.
- Dataset split: The dataset uses an 80/20 training-test split with 6167 training videos and 1693 test videos, ensuring subjects do not cross sets.The smallest category contains 52 training examples and 24 test examples.
3. Video annotation
Charades-Ego obtains temporal annotations by presenting paired videos simultaneously and using the third-person view to improve annotation of the first-person view.
- Activity labeling: Activities were pooled by transferring labels from same-script Charades videos and applying video-level annotation to third-person videos.This creates candidate activities likely to appear in the paired videos.
- Temporal annotation: Annotators viewed third- and first-person videos from the same subject simultaneously and annotated both videos temporally.The third-person video provided better observability for annotating the first-person video.
- Annotation quality: 77.0% versus 72.8% is the annotator consensus for third-person versus egocentric videos, a 4.2% lower consensus for egocentric annotation.The comparison is reported for fully observable third-person videos and egocentric videos.
4. Baselines on Charades-Ego v1.0
The paper establishes Charades-Ego baselines for first- and third-person recognition using models trained on third-person, first-person, or combined labeled data, including zero-shot egocentric evaluation.
- Baseline setup: Baselines train and test on first- and third-person videos to encourage research across both viewpoints.The experiments are reported in Table 1 using video-level mean average precision.
- Baseline setup: All models start from the same ResNet-152 models pretrained on the original Charades third-person dataset.Third-Person Training uses the original model, while First-Person or First/Third Training uses labeled Charades-Ego data.
- Annotation interface: The annotation interface displays paired videos simultaneously, with sliders that seek to corresponding timestamps for temporal labeling.Workers independently annotate related first- and third-person videos rather than treating them as identical.
- Baseline setup: First-person labeled data are used for testing on the egocentric test set in the First-Person Training and First/Third Training settings.This continuation specifies the evaluation condition for models trained with Charades-Ego first-person labels.
- Egocentric benchmark: Charades-Ego v1.0 is evaluated with multiple baselines, including zero-shot egocentric recognition without first-person labeled training data.The ResNet-152 Transfer baseline predicts activities in third-person videos using the Charades model before transferring labels to egocentric recognition.
5. Discussion
The paper positions Charades-Ego as a step toward bringing third-person and first-person activity recognition together. It frames this integration as a shared research direction.
- The authors aim to bring third-person and first-person activity recognition together.
- Charades-Ego is presented as a step toward linking the two activity-recognition fields.
- The discussion encourages research spanning both first-person and third-person video understanding.