Source-linked AI summary
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray
TL;DR
Egocentric video understanding lacks sufficiently large, realistic annotated datasets. EPIC-KITCHENS introduces a diverse, densely annotated benchmark with object, action, and anticipation challenges, whose baselines remain far from high-precision performance.
Problem
Egocentric video understanding lacks large, annotated datasets capturing longer, nonscripted activities and natural multi-tasking.
Method
EPIC-KITCHENS records 55 hours from 32 participants in native kitchens and densely annotates narrations, action segments, and interacting-object bounding boxes.
Results
Baselines across object detection, action recognition, and anticipation show the dataset’s potential, while existing methods remain far from high-precision performance.
Takeaways & Limitations
EPIC-KITCHENS provides a benchmark for advancing fine-grained egocentric video understanding across object, action, and anticipation tasks.
Takeaways & Limitations
Participant narrations use individual vocabularies and free language, making the annotation process challenging.
Abstract
from arXiv · showhide
First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict nonscripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labeled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. Dataset and Project page: http://epic-kitchens.github.io
1 Introduction
The introduction motivates first-person vision as a rich view of daily activities while noting that existing video datasets often miss natural action progression and multi-tasking. It presents EPIC-KITCHENS as a large-scale, densely annotated egocentric dataset collected in participants’ native kitchens and evaluated through object, action, and anticipation challenges.
- Motivation: Existing video datasets often depict less natural behavior and lack the progression and multi-tasking found in real-life actions.
- Motivation: First-person vision captures people’s goals, motivation, multi-tasking, and varied ways of performing mundane daily activities.The passage also identifies applications in human-to-robot imitation learning and HCI.
- Dataset: EPIC-KITCHENS was collected from 32 participants of 10 nationalities in their native kitchens during daily kitchen activities.Participants recorded sequences regardless of duration, including video and sound.
- Dataset: 55hrs of recording were densely annotated with start/end times for actions and interactions and bounding boxes around interacted objects.
- Evaluation: The benchmark defines object, action, and anticipation challenges and reports baselines for seen and unseen kitchens.Held-out test ground truth and leaderboards are provided through the dataset project page.
2 Related Datasets
EPIC-KITCHENS differs from prior egocentric datasets by capturing unscripted, natural kitchen activities rather than instructed action sequences. It also substantially exceeds ADL in frames, annotated action segments, and object bounding boxes.
- Scripted versus natural activities: Most datasets capturing activities in native environments are recorded in third-person, while some use scripted storylines, recipes, or given actions.These collection procedures can produce less natural-looking actions.
- Scripted versus natural activities: Prior egocentric datasets use scripted activities in which participants are told what actions to perform.Following instructions encourages sequential steps, unlike real-life scenarios involving multitasking, searching, changing one’s mind, and surprises.
- Dataset scale: EPIC-KITCHENS contains 11.5M frames vs 1M in ADL, 90x more annotated action segments, and 4x more object bounding boxes.These comparisons make it the largest first-person dataset to date.
3 The EPIC-KITCHENS Dataset
EPIC-KITCHENS combines voluntary, unscripted kitchen recordings with participant narrations and crowdsourced temporal, object, and semantic annotations. The resulting benchmark contains diverse recordings, 39,564 localized action segments, 454,255 bounding boxes, and manually or semi-automatically grouped verb and noun classes with reported annotation errors.
- Data collection: 32 individuals recorded all kitchen visits for three consecutive days across four cities in North America and Europe.The participants recorded voluntarily without financial reward, starting immediately before entering and stopping before leaving the kitchen.
- Data collection: Participants recorded 1.7 hours on average, up to 4.6 hours, across 13.6 sequences per participant on average.Sequence counts and durations varied with each participant’s kitchen engagement.
- Narration and annotation: 39,596 participant narrations provide an initial action labeling every 4.9 seconds, with phrases averaging 2.8 words.The narrations offer rough temporal alignment but may be incomplete, belated, and expressed in participant-specific vocabulary.
- Narration and annotation: 39,564 action segments cover 99.9% of narrated segments after crowdsourced temporal adjustment, with mean length 3.7 seconds and standard deviation 5.6 seconds.Annotators imposed a minimum action length of 0.5 seconds and prevented an action from starting before the preceding action’s start time.
- Object annotation: 454,255 object bounding boxes were collected, with 77% of requested annotations yielding at least one box and 1.64 boxes per frame on average.Potential annotation frames extended from 2 seconds before to 2 seconds after each action segment.
- Semantic classes: The taxonomy contains 125 verb classes and 331 noun classes used across all three challenges.Verbs were manually clustered and nouns semi-automatically clustered after automated WordNet, Word2Vec, and Lesk combinations produced meaningless clusters.
4 Benchmarks and Baseline Results
The paper defines three benchmark challenges—object detection, action recognition, and action anticipation—and evaluates existing methods under seen- and unseen-kitchen protocols. Baselines reveal substantial difficulty, especially for object detection and future-action prediction.
- Benchmark Design: Three challenges are introduced with baseline results and online leaderboards, using held-out annotations and test splits designed to assess generalization to novel kitchens.The seen-kitchen split contains kitchens represented in both training and testing, whereas the unseen-kitchen split separates participants and kitchens.
- Object Detection: Object detection evaluates annotated active objects across many-shot and few-shot classes, with mean average precision measured at IoU thresholds of 0.05, 0.5, and 0.75.The evaluation includes 202 many-shot classes and 88 few-shot classes, reporting performance for the 15 most frequent classes.
- Object Detection: Below 40% performance at the standard IoU > 0.5 indicates that EPIC-KITCHENS objects are generally harder to detect than those in most existing datasets.The paper also notes relatively low performance even at a very small IoU threshold.
- Action Recognition: Action recognition classifies provided action segments into verb, noun, or combined action classes, reporting aggregate top-1 and top-5 accuracy plus per-class precision and recall.The combined action class pairs the verb with the first noun in the narration when multiple nouns are present.
- Action Anticipation: Action anticipation predicts the next verb and noun jointly from a video segment preceding the action start, using TSN baselines compared with 2SCNN.Qualitative results show a tendency to over-predict ‘put’; the paper identifies long-term goal understanding and multi-scale history as possible remedies.
5 Conclusion and Future Work
EPIC-KITCHENS is presented as the largest and most varied egocentric vision dataset, captured in participants’ native environments and annotated through participant narrations. Baseline results across object detection, action recognition, and anticipation demonstrate the dataset’s potential.
- Conclusion and Future Work: EPIC-KITCHENS contains 55 hours of head-mounted GoPro video recorded in participants’ native environments.The dataset is described as the largest and most varied in egocentric vision to date.
- Conclusion and Future Work: Participant live commentary anchors annotations of narrations, action segments, and objects in a unified pipeline.The pipeline begins with participants commenting on their recorded videos.
- Conclusion and Future Work: Baseline results cover object detection, action recognition, and anticipation challenges, showing EPIC-KITCHENS’ potential.The passage reports baseline results across all three challenge types.