The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines

1 INTRODUCTION

• Video understanding has been hindered by scarce annotated data; most cited datasets use clips only seconds long, while Hollywood in Homes uses scripted performances that lack natural action progression.The paper contrasts scripted recordings with the progression and multitasking of real-life activity.Source: §1 · p. 1
• First-person video captures daily goals, motivation, multitasking, and varied everyday actions, but existing egocentric datasets are smaller and often limited to one environment.Daily interactions from wearable cameras are also scarcely available online.Source: §1 · p. 1
• EPIC-KITCHENS records daily kitchen activities from 32 participants of 10 nationalities in their native kitchens, capturing natural multitasking and parallel-goal interactions.Participants recorded sequences regardless of duration, with video and sound.Source: §1 · p. 1
• The dataset comprises 55 hours of recordings, annotated with action start and end times and object bounding boxes, and supports object, action, and anticipation challenges.Baselines are reported for seen and unseen kitchens; released data and held-out test ground truth support an online leaderboard.Source: §1 · p. 1

• The paper compares EPIC-KITCHENS with six egocentric datasets,,,,, and five third-person object-interaction datasets,,,,, excluding inter-person interaction and instructional-video datasets.Source: §2 · p. 2
• Unlike scripted egocentric activities, EPIC-KITCHENS captures natural scenarios involving multitasking, searching, deciding what to do, changing one’s mind, and unexpected surprises.The passage notes that instructions often lead participants to perform steps sequentially, whereas these kitchen scenarios are more natural.Source: §2 · p. 2
• EPIC-KITCHENS has 11.5M frames vs 1M in ADL, 90x more annotated action segments, and 4x more object bounding boxes, making it the largest first-person dataset to date.Source: §2 · p. 3

3.1 Data Collection

• Thirty-two individuals in four cities recorded kitchen visits and narrated their own actions after completing recordings, yielding 39,596 action narrations, corresponding to one narration every 4.9s in the video.Post-recording narration let participants perform daily activities undisturbed; evaluation included only actions that had been narrated.Source: §3.1 · pp. 3–4

3.2 Action Segment Annotations

• 39,564 action segments (µ = 3.7s, σ = 5.6s) were finalized using union of best agreements; 57.4% of narration timestamps fell within their segment, while 30.6% fell within another.The union method combined annotations when they strongly agreed, increasing average action length from 3.5 sec to 3.7 sec; 18.5% of narrations preceded their action and 24.1% followed it.Source: §3.2 · pp. 5–6

3.3 Active Object Bounding Box Annotations

• 454,158 bounding boxes and 125,375 true negative object labels were collected; sequence-level annotator selection raised consecutive-frame average IoU from 0.41 to 0.46.Three workers annotated each object HIT, and the annotator with maximum agreement across the sequence was selected, with ties broken by tighter boxes.Source: §3.3 · p. 6

3.4 Verb and Noun Classes

• Free-text multilingual narrations were manually clustered into minimally overlapping classes, yielding 125 verb classes and 331 noun classes.Automatic clustering methods proved inadequate for the short, context-limited annotations, so the authors iteratively adjusted clusters manually.Source: §3.4 · pp. 6–7

3.5 Annotation Visualisations

• Visualizations show 19 object super-categories and natural variation in one participant’s breakfast routine across four days, with few unlabelled background frames.The super-categories include nine food and drink groups; object box sizes vary with furniture and appliances being larger than food ingredients.Source: §3.5 · pp. 7–8

3.6 Annotation Quality Assurance

• Reported annotation error rates were 5.7% for action segment boundaries, 6.3% for object bounding boxes, 3.3% for verb classes, and 6.0% for noun classes.The authors state that these rates are comparable to other datasets.Source: §3.6 · p. 8

4 BENCHMARKS AND BASELINE RESULTS

• The test protocols reserve 27% of annotations and distinguish S1, where kitchens occur in training and testing, from S2, which holds out complete kitchens for four participants.S2 is only 7% of the dataset by frame count; it includes one unseen verb, 13 unseen nouns, and 220 zero-shot actions, mostly new combinations of observed verbs and nouns.Source: §4 · p. 8, Table 5 · p. 9

4.1 Object Detection Benchmark

• Object detection evaluates annotated active objects using Faster R-CNN and mAP at IoU thresholds 0.05, 0.5, and 0.75, separating 202 many-shot from 78 few-shot classes.The challenge assesses images only where the object is annotated and reports per-class results for classes with at least 10 training boxes.Source: §4.1 · p. 9
• Object detection remains difficult: performance at IoU > 0.5 is below 30%, few-shot performance falls short of 10%, and seen and unseen split performance is comparable.Performance across five bounding-box size groups is also comparable, with the best performance on the smallest objects in both splits.Source: §4.1 · pp. 9–10, Table 6 · p. 9, Table 7 · p. 10

4.2 Action Recognition Benchmark

• TRN and TSM explicitly model temporal relationships, unlike TSN’s average pooling across snippets; 2SCNN samples one snippet, while TSN, TRN, and TSM use multiple snippets.TRN computes relations over ordered snippet sets, whereas TSM shifts filter responses across time to enable later convolutional layers to learn temporal correlations.Source: §4.2 · pp. 10–11
• Fused modalities outperform or match the best individual modality, yet top-1 action accuracy is only 26.1% in S1 and 15.6% in S2.Flow performs best for verbs, RGB for nouns, and flow transfers best to unseen environments for noun classification; audio exceeds RGB for unseen top-1 verb accuracy.Source: §4.2 · pp. 11–12, Table 8 · p. 11
• Verb top-1 accuracy improves 6.4–8.0% in S1 and 5.6–6.0% in S2 with temporal modeling; TSM leads on 11 of 12 seen metrics, while TRN leads on 7 of 12 unseen metrics.The differing unseen results suggest TSM may generalize less well to unseen actions.Source: Table 9 · p. 11, §4.2 · pp. 12–13

4.3 Action Anticipation Benchmark

• No action anticipation method leads across evaluation measures or both splits; ED has an advantage on action measures in S1 that disappears in S2, while DMR performs poorly on both.The comparison covers five methods at a fixed anticipation time of 1 second; for 2SCNN, TSN, and TRN, reported results use 0.5 seconds of observed video.Source: §4.3 · p. 13, Table 10 · p. 13
• Anticipation accuracy varies with observation time: 2SCNN and TSN struggle with longer contexts, while TRN is less sensitive in S1 and benefits more from longer observations in S2.The models’ differing sensitivity suggests limited ability among these baselines to reason about past and future context.Source: §4.3 · p. 13, Figure 15 · p. 14

5 CONCLUSION AND FUTURE WORK

• The 55-hour EPIC-KITCHENS dataset was captured in participants’ native environments and annotated through a pipeline beginning with their live commentary of recorded videos.The annotations include narrations, action segments, and object annotations.Source: §5 · p. 14
• Baseline results show temporal modelling matters for this dataset, anticipation needs novel approaches, and existing methods remain far from high-precision performance.The object detection, action recognition, and anticipation challenges form a base for higher-level understanding of the wearer’s goals.Source: §5 · p. 14
• Future work considered includes dense captioning, skill understanding, and multi-day routine modelling.The dataset’s unscripted and untrimmed recordings allow explorations beyond the challenges reported in the paper.Source: §5 · pp. 14–15
• Beyond action recognition and anticipation, EPIC-KITCHENS has supported object reasoning and detection [67],, action retrieval [69], novel-word learning, domain adaptation [70], and affordance learning [71].The authors attribute this breadth to collecting natural interactions rather than selecting a task before data collection.Source: §5 · p. 15

B. IMPLEMENTATION DETAILS

• The section covers implementation details of the compared baselines for all three challenges in Section 4.Source: B. IMPLEMENTATION DETAILS · p. 17

B.1. Object Recognition Benchmark

• The benchmark uses TensorFlow Object Detection API Faster R-CNN, with an MS-COCO-pretrained ResNet-101 backbone,.Source: §B.1 · p. 17
• Training uses an initial learning rate of 0.0003, decayed by 10 after 30,000 and 40,000 iterations, with an overall minibatch size of 32 across eight NVIDIA P100 GPUs.The per-GPU minibatch size is 4; training uses distributed training and parameter synchronisation on one NVIDIA DGX-1 compute node.Source: §B.1 · p. 17
• Images are resized to a 600-pixel shortest side, with stride 16 features, four anchor scales, three aspect ratios, NMS IoU threshold 0.7, and 300 RPN proposals.The resizing maintains aspect ratio; anchor scales are 0.25, 0.5, 1.0, and 2.0, and aspect ratios are 1:1, 1:2, and 2:1. Resizing follows.Source: §B.1 · p. 17

B.2. Action Recognition Benchmark

• TSN, TRN and TSM use two output FC layers for verb and noun prediction; 2SCNN is implemented as one-segment TSN.The models use official PyTorch implementations and ImageNet-pretrained weights.Source: §B.2 · p. 17
• RGB networks sample single frames, flow networks use stacks of five optical-flow pairs, and audio networks process 1.28-second waveform segments.Audio is converted to mono and resampled to 24 kHz, then represented as a 256 × 256 log-spectrogram; segments may extend beyond action boundaries. Flow is extracted with TV-L1 [12].Source: §B.2 · p. 17
• Models train for 80 epochs with SGD and momentum 0.9, then use 10-crop testing; modality scores are averaged after softmax for fusion.Training uses batches of 64 (128 for audio), no stratification or weighted sampling, and ImageNet-pretrained initialization; crop scores are averaged pre-softmax for clip-level predictions.Source: §B.2 · p. 17

B.3. Action Anticipation Benchmark

• Recognition models anticipate from a 1-second video observation, while DMR uses a single image and ED uses 16 frames sampled at 4 fps over 4 seconds.The video observation is sampled 1 second before the action begins.Source: §B.3 · p. 17
• TSN and TRN use the authors’ official PyTorch implementations, 2SCNN uses TSN with one segment, and DMR is re-implemented from its paper.ED follows without the reinforcement module because this task does not require early action-versus-background discrimination.Source: §B.3 · p. 17
• 2SCNN and TSN train for 160 epochs, TRN uses similar settings with batch size 32, and DMR trains in two stages while ED uses pre-training and fine-tuning.DMR’s first stage trains a multimodal regressor for 65 epochs; its second trains a 1024-unit hidden-layer MLP for 500 epochs, following. ED follows and uses TSN features trained for action recognition.Source: §B.3 · p. 17
• 2SCNN, TSN, and TRN use late fusion of RGB and Flow predictions with equal weights, DMR uses RGB only, and ED early-fuses concatenated RGB and Flow representations.ED’s input features come from the RGB and Flow branches of a three-segment TSN model.Source: §B.3 · pp. 17–18