Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

• The dataset addresses egocentric action understanding while connecting it to abundant third-person video resources.Source: §1 · p. 1
• Crowd workers recorded themselves performing prescribed activity scripts from both first- and third-person viewpoints.Source: §1 · p. 1
• 68,536 activity instances span 68.8 hours of first- and third-person video in Charades-Ego.Source: §1 · p. 1
• Charades-Ego shares activity classes, scripts, and methodology with Charades, which adds 82.3 hours of third-person video and 66,500 activity instances.Source: §1 · p. 1

Data collection

• Users recorded two videos of each script: one third-person performance and one same-script performance with a forehead-mounted camera.The paired recordings compromise between synchronized laboratory capture and broader data collection.Source: §2 · p. 1

Scripts and activities

• 78.7% of Charades-Ego scripts are shared with the Charades training set, while 1000 additional scripts were collected for diversity.Most scripts originate from Charades, supplemented by newly collected scripts.Source: §2 · p. 1

Data collection

• 60% of submitted videos contained activities featuring both hands, supported by incentivizing users to create their own camera mounts.Workers received $1.5 per video pair and a $0.5 bonus for using a home-made head mount.Source: §2 · p. 2

Dataset scale

• 68,536 activity instances span 68.8 total hours across 157 activities, equally divided between 34.4 hours of third-person and first-person video.The videos were collected in various rooms across 112 homes worldwide.Source: §2 · p. 2

Dataset split

• The dataset uses an 80/20 training-test split with 6167 training videos and 1693 test videos, ensuring subjects do not cross sets.The smallest category contains 52 training examples and 24 test examples.Source: §2 · p. 2

Activity labeling

• Activities were pooled by transferring labels from same-script Charades videos and applying video-level annotation to third-person videos.This creates candidate activities likely to appear in the paired videos.Source: §3 · p. 2

Temporal annotation

• Annotators viewed third- and first-person videos from the same subject simultaneously and annotated both videos temporally.The third-person video provided better observability for annotating the first-person video.Source: §3 · p. 2

Annotation quality

• 77.0% versus 72.8% is the annotator consensus for third-person versus egocentric videos, a 4.2% lower consensus for egocentric annotation.The comparison is reported for fully observable third-person videos and egocentric videos.Source: §3 · p. 2

Baseline setup

• Baselines train and test on first- and third-person videos to encourage research across both viewpoints.The experiments are reported in Table 1 using video-level mean average precision.Source: §4 · p. 2, Table 1 · p. 2
• All models start from the same ResNet-152 models pretrained on the original Charades third-person dataset.Third-Person Training uses the original model, while First-Person or First/Third Training uses labeled Charades-Ego data.Source: §4 · p. 2

Annotation interface

• The annotation interface displays paired videos simultaneously, with sliders that seek to corresponding timestamps for temporal labeling.Workers independently annotate related first- and third-person videos rather than treating them as identical.Source: Figure 2 · p. 2

Baseline setup

• First-person labeled data are used for testing on the egocentric test set in the First-Person Training and First/Third Training settings.This continuation specifies the evaluation condition for models trained with Charades-Ego first-person labels.Source: §4 · p. 2

Egocentric benchmark

• Charades-Ego v1.0 is evaluated with multiple baselines, including zero-shot egocentric recognition without first-person labeled training data.The ResNet-152 Transfer baseline predicts activities in third-person videos using the Charades model before transferring labels to egocentric recognition.Source: §4 · p. 2

• The authors aim to bring third-person and first-person activity recognition together.Source: §5 · p. 2
• Charades-Ego is presented as a step toward linking the two activity-recognition fields.Source: §5 · p. 2
• The discussion encourages research spanning both first-person and third-person video understanding.Source: §5 · p. 2