Source-linked AI summary
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray
TL;DR
Video understanding lacks large, naturalistic egocentric datasets that capture daily activity, multitasking, and object interaction. EPIC-KITCHENS addresses this gap through participant-recorded kitchen videos and dense multimodal annotations, then evaluates object, action, and anticipation challenges across seen and unseen kitchens. The baselines show temporal modelling is important, while existing methods remain far from high-precision performance on these tasks.
Problem
Video understanding has lacked large annotated datasets, while first-person datasets have often been smaller and captured in single environments.
Method
The paper constructs EPIC-KITCHENS from nonscripted kitchen recordings by 32 participants and annotates actions and objects using participant narrations and crowd-sourced ground truths.
Results
Baselines across object detection, action recognition, and anticipation show temporal modelling is important, while existing methods remain far from high-precision performance.
Takeaways & Limitations
The three challenges provide a base for higher-level understanding of the wearer’s goals and motivate novel approaches to anticipation.
Takeaways & Limitations
Free participant vocabulary makes evaluation challenging, and limited annotation context prevented reliable automatic clustering of verb and noun classes.
Abstract
from arXiv · showhide
Since its introduction in 2018, EPIC-KITCHENS has attracted attention as the largest egocentric video benchmark, offering a unique viewpoint on people's interaction with objects, their attention, and even intention. In this paper, we detail how this large-scale dataset was captured by 32 participants in their native kitchen environments, and densely annotated with actions and object interactions. Our videos depict nonscripted daily activities, as recording is started every time a participant entered their kitchen. Recording took place in 4 countries by participants belonging to 10 different nationalities, resulting in highly diverse kitchen habits and cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.2K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos after recording, thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and. anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. We introduce new baselines that highlight the multimodal nature of the dataset and the importance of explicit temporal modelling to discriminate fine-grained actions e.g. 'closing a tap' from 'opening' it up.
1 INTRODUCTION
EPIC-KITCHENS addresses limited large-scale, naturalistic egocentric video data by recording daily kitchen activities in native environments and annotating actions and interacting objects. It supports object, action, and anticipation challenges with held-out evaluation across seen and unseen kitchens.
- Motivation: Video-understanding datasets have often been short, single-action, or scripted, limiting natural activity progression and multitasking.Existing egocentric datasets were also substantially smaller and often captured in a single environment.
- Dataset contribution: EPIC-KITCHENS records nonscripted daily kitchen activities by 32 participants in their native kitchens, including parallel-goal interactions.The recordings include activities such as washing dishes while cooking, which existing datasets had not captured.
- Dataset contribution: The dataset contains 55 hours of video with dense action-segment timing and object bounding-box annotations.It defines object, action, and anticipation challenges and reports baselines for seen and unseen kitchens.
- Annotation pipeline: Participants’ narrations, active-object annotations, and frames from 32 environments form the dataset’s annotation pipeline.Figure 1 presents the environments, participant narrations used for action segments, and active object bounding boxes.
2 RELATED DATASETS
EPIC-KITCHENS is positioned against egocentric and third-person activity datasets, especially those using scripted or sequential activities. Compared with ADL, it substantially expands the scale of native-environment egocentric recordings and annotations.
- Dataset landscape: Related datasets include short clips, native-environment recordings, cooking activities, and object-interaction videos across egocentric and third-person perspectives.The comparison excludes datasets focused on inter-person interactions or instructional videos because they address different research questions.
- Naturalistic activity: Many related datasets use scripted activities, whereas EPIC-KITCHENS targets natural scenarios involving multitasking and searching for objects.Scripted instructions typically impose a sequential order on actions.
- Scale comparison: EPIC-KITCHENS has 11.5M frames versus 1M in ADL, with 90x more annotated action segments and 4x more object bounding boxes.The paper identifies EPIC-KITCHENS as the largest first-person dataset at the time of comparison.
3 THE EPIC-KITCHENS DATASET
EPIC-KITCHENS combines natural kitchen recordings with participant narrations and dense temporal and object annotations. Its pipeline addresses noisy timing, overlapping actions, object-label variability, and diverse unscripted routines through crowdsourced refinement and manual class clustering.
- 3.1 Data Collection: 32 participants recorded kitchen visits in native environments, while post-recording speech provided coarse action annotations without disturbing daily activities.Narrations used five languages and were collected after filming so participants could perform activities without labeling concerns.
- 3.2 Action Segment Annotations: 39,596 action narrations were collected, but narrator timing was imperfect: 57.4% fell within the action segment and 30.6% fell within a different action.Narrations were often delayed because participants first recognized the action, although some anticipated their own actions.
- 3.2 Action Segment Annotations: 39,564 action segments were retained, with mean length 3.7 seconds, and 24% overlapped another action segment.The overlap rate highlights concurrent interactions in natural recordings.
- 3.3 Object Annotations: Object annotations covered action-associated nouns within a window extending 2 seconds before and after each action segment.Each object annotation task targeted one object for up to 25 seconds of video.
- 3.3 Object Annotations: 454,158 bounding boxes and 125,375 true-negative object labels were collected for the Object Detection challenge.True negatives represent active objects absent because of occlusion or camera viewpoint.
- 3.4 Verb and Noun Classes: Manual verb and noun clustering produced 125 verb classes and 331 noun classes after automatic methods failed because the short annotations lacked sufficient context.The dataset also reports visualisations across 19 supercategories, with object sizes varying substantially between furniture, appliances, and food ingredients.
4 BENCHMARKS AND BASELINE RESULTS
EPIC-KITCHENS benchmarks object detection, action recognition, and action anticipation across seen and unseen kitchens. Baselines show challenges from fine-grained temporal distinctions, multimodal cues, object ambiguity, and limited anticipation of future actions.
- Evaluation protocols: 27% of the data is held out for testing, with S1 sharing kitchens across training and testing and S2 holding out complete kitchens.S2 uses all sequences from four participants for testing and represents 7% of the dataset by frame count.
- Evaluation protocols: 220 actions are zero-shot in S2, mostly combining verbs and nouns that were individually observed during training.S2 also contains one unseen verb and 13 unseen nouns.
- Object Detection Benchmark: Object detection performance is below 30% at IoU>0.5 and falls below 10% for few-shot classes, while seen and unseen splits remain comparable.The comparable split performance indicates generalization across environments despite low overall detection accuracy.
- Action Recognition Benchmark: Flow performs best for verb classification, RGB for noun classification, and fused modalities outperform or match the strongest individual modality.Top-1 action accuracy is 26.1% on seen kitchens and 15.6% on unseen kitchens.
- Action Recognition Benchmark: Temporal modelling improves top-1 verb accuracy by 6.4–8.0% in S1 and 5.6–6.0% in S2, with less effect on noun classification.The benchmark specifically targets fine-grained action distinctions requiring temporal progression.
- Action Anticipation Benchmark: Longer observation times challenge 2SCNN and TSN in anticipation, while failures also arise from ambiguous futures and insufficient temporal context.Qualitative examples include confusing “put oil” with “roll ball” and “take bag” with “check knife.”
5 CONCLUSION AND FUTURE WORK
EPIC-KITCHENS combines large-scale native-environment egocentric video with participant narrations, dense annotations, and multiple challenges. Its baseline results underscore temporal modelling, the difficulty of anticipation, and opportunities beyond the reported tasks.
- 55 hours of native-environment video are annotated with narrations, action segments, and object annotations.The annotation pipeline begins with live commentary by the participants themselves.
- Baseline results across object detection, action recognition, and anticipation show the dataset’s potential for fine-grained video understanding.
- Temporal modelling is important for this dataset, while anticipation requires novel approaches.
- The three challenges provide a foundation for higher-level understanding of the wearer’s goals.
- EPIC-KITCHENS also supports less explored tasks including video object reasoning, action retrieval, novel-word learning, domain adaptation, and environmental affordances.
B. IMPLEMENTATION DETAILS
This section reports implementation details for the baselines evaluated across the paper’s three challenges.
- The section reports implementation details of the compared baselines for the three challenges.
B.1. Object Recognition Benchmark
The object recognition benchmark uses a Faster R-CNN detector with a ResNet-101 architecture pretrained on MS-COCO.
- Faster R-CNN uses a ResNet-101 base architecture pretrained on MS-COCO for object recognition.
B.2. Action Recognition Benchmark
The action recognition benchmark compares established temporal architectures using separate verb and noun prediction outputs, with RGB, optical-flow, and audio modalities.
- TSN, TRN, and TSM are implemented in PyTorch and pretrained on ImageNet for action recognition.
- Each action baseline uses separate fully connected outputs to predict verbs and nouns.
- RGB models use one frame per segment, while audio models extract 1.28 seconds from untrimmed video, potentially extending beyond action boundaries.
- Models are trained with SGD and momentum 0.9 for 80 epochs, without stratification or weighted sampling.
B.3. Action Anticipation Benchmark
The benchmark evaluates action anticipation methods using video observations sampled one second before the anticipated action, with distinct input and fusion schemes across models.
- 2SCNN, TSN, and TRN process observations of length τa sampled one second before the anticipated action begins.
- RGB and Flow predictions are merged by equal-weight late fusion for the video-based methods.
- DMR anticipates actions from single RGB images, whereas ED uses early fusion of concatenated RGB and Flow representations.