Source-linked AI summary
HD-EPIC: A Highly-Detailed Egocentric Video Dataset
Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhifan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Davide Moltisanti, Michael Wray, Hazel Doughty, Dima Damen
TL;DR
HD-EPIC addresses the lack of realistic egocentric video datasets that combine dense, interconnected annotations with long-video and 3D grounding. It constructs such a dataset from unscripted home kitchen recordings and evaluates models across VQA and recognition tasks. The VQA benchmark remains difficult, with current models far below the human baseline, while recognition results expose limited robustness to new scenes or devices.
Problem
Existing egocentric datasets are often sparsely annotated, while richly annotated datasets are commonly synthetic or collected in controlled settings.
Method
The paper collects unscripted kitchen videos and manually links recipe, action, nutrition, fixture, object-motion, gaze, audio, and 3D digital-twin annotations into validation benchmarks.
Results
The VQA benchmark is challenging: average model performance is 37.6% versus a 90.3% human baseline, while audio models show large performance drops on HD-EPIC relative to EPIC-Sounds.
Takeaways & Limitations
HD-EPIC supports comprehensive validation across video-language, action, sound, and long-term video-object segmentation capabilities.
Takeaways & Limitations
Gaze-priming analysis excludes objects taken or placed off screen and objects already being manipulated before pickup.
Abstract
from arXiv · showhide
We present a validation dataset of newly-collected kitchen-based egocentric videos, manually annotated with highly detailed and interconnected ground-truth labels covering: recipe steps, fine-grained actions, ingredients with nutritional values, moving objects, and audio annotations. Importantly, all annotations are grounded in 3D through digital twinning of the scene, fixtures, object locations, and primed with gaze. Footage is collected from unscripted recordings in diverse home environments, making HDEPIC the first dataset collected in-the-wild but with detailed annotations matching those in controlled lab environments. We show the potential of our highly-detailed annotations through a challenging VQA benchmark of 26K questions assessing the capability to recognise recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze direction. The powerful long-context Gemini Pro only achieves 38.5% on this benchmark, showcasing its difficulty and highlighting shortcomings in current VLMs. We additionally assess action recognition, sound recognition, and long-term video-object segmentation on HD-EPIC. HD-EPIC is 41 hours of video in 9 kitchens with digital twins of 413 kitchen fixtures, capturing 69 recipes, 59K fine-grained actions, 51K audio events, 20K object movements and 37K object masks lifted to 3D. On average, we have 263 annotations per minute of our unscripted videos.
1. Introduction
HD-EPIC addresses the gap between sparsely annotated large-scale egocentric datasets and richly annotated but synthetic or controlled datasets. It introduces unscripted home recordings with interconnected recipe, action, nutrition, 3D, object-motion, gaze, and audio annotations, evaluated through challenging benchmarks.
- Egocentric videos challenge comprehensive understanding through camera motion, subtle actions, occlusions, and objects leaving view.
- HD-EPIC bridges realism and annotation density with 41 hours of multi-day unscripted recordings and highly detailed labels.
- Recipe steps link to preparatory actions, while ingredients are weighed, nutritionally labelled, and tracked as they are added.
- Actions include dense what, how, and why descriptions; kitchen fixtures, object movements, 3D boxes, and gaze are interconnected.
- The 26K-question VQA benchmark covers recipe nutrition, ingredients, fine-grained actions, 3D perception, object motion, gaze, and related multi-hop questions.
- The paper additionally benchmarks action recognition, sound recognition, and long-term video-object segmentation on HD-EPIC.
2. Related Work
Prior egocentric benchmarks provide useful scale or selected capabilities, but often rely on sparse annotations, static environments, short narrations, or limited 3D information. HD-EPIC instead combines comprehensive annotations with unscripted home footage and a 3D digital twin.
- Prior benchmarks cover selected embodied abilities but include static-environment views, short narrations, or limited annotation types.
- HD-EPIC combines detailed narrations, object and hand segmentations, and comprehensive 3D scene and object digital twins from unscripted home footage.
3. Data Collection
HD-EPIC records participants’ unscripted kitchen activities across multiple days using multimodal glasses and supporting nutritional equipment. The resulting data span diverse recipes, ingredients, narrations, and recipe modifications.
- The collection includes diverse objects, activities, and recipes from in-the-wild recordings.
- Participants recorded daily kitchen activities for at least 3 consecutive days using Project Aria glasses with cameras, microphones, eye tracking, and SLAM.
- Participants freely prepared 69 recipes, averaging 6.6 steps and 8.1 ingredients per recipe.
- Recipes included participant-reported modifications to ingredients and preparation steps.
- Participants weighed and manually logged ingredients, yielding nutrition information for 558 ingredients across dishes with varied calorie content.
- Participants narrated what they were doing, along with how and why, extending the collected action descriptions.
4. Annotation Pipeline
HD-EPIC builds a dense annotation pipeline linking recipe structure, actions, ingredients, nutrition, audio, gaze, objects, and kitchen fixtures in 3D. The resulting annotations support fine-grained temporal, spatial, and interaction-aware analysis of unscripted kitchen videos.
- Recipe steps and ingredients: Prep-step pairs capture essential actions needed before each recipe step, with 93.1% of steps having paired prep annotations.Average prep and step durations are 54.5s and 78.2s, respectively.
- Recipe steps and ingredients: Ingredients are weighed, nutritionally labelled, and temporally linked to their addition so dish nutrition can be tracked as recipes progress.The pipeline annotates 283 weighing sequences and 501 adding sequences, excluding spices.
- Fine-grained actions: Narrations receive precise action boundaries, parsed verb, noun, hand, how, and why labels, while audio annotations identify timed events across 44 classes.The dataset contains 59,454 action segments and 50,968 audio annotations.
- Digital twins and interactions: Each kitchen is reconstructed as a Blender digital twin with labelled fixtures linked to narrated interactions, object placements, and gaze.Kitchens average 45.9 labelled fixtures, including counters, cupboards, drawers, and appliances.
- Digital twins and interactions: Objects are tracked through movements, masks, lifted 3D locations, fixture assignments, and gaze-based priming of future interactions.Among objects feasible for priming, 94.8% are primed before pickup and 88.5% before placement.
5. Benchmarks and Results
The paper evaluates HD-EPIC through a broad VQA benchmark plus action, sound, and long-term object-segmentation tasks. Models struggle across capabilities and input lengths, with especially large difficulty on long videos and newly encountered scenes.
- VQA benchmark: HD-EPIC’s VQA benchmark covers seven annotation types, including recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze.Questions include long-video multi-hop reasoning such as counting an object’s movements.
- VQA benchmark: 26,650 five-way multiple-choice questions across 30 prototypes use difficult within-dataset negatives to create realistic evaluation challenges.Input lengths range from single frames to videos exceeding seven hours.
- VQA results: 37.6% average VQA performance versus a 90.3% human baseline shows the benchmark remains difficult for current models.Gemini performs best overall, while models struggle particularly with inputs at least one minute long.
- Sound recognition: Sound-recognition performance drops by 28.4, 25.9, and 26.4 points for SSAST, ASF, and TIM on HD-EPIC relative to EPIC-Sounds.The reported comparison indicates limited robustness to new scenes or devices.
- Long-term video object segmentation: SAM2 surpasses Cutie for hand segmentation but performs worse on objects, while both models struggle with long-term object segmentation.The benchmark contains 1,000 sequences with one to five objects and two hand masks.
6. Onwards...
HD-EPIC is released with videos, audio, gaze, camera poses, digital twins, and detailed annotations spanning object movements, masks, 3D locations, tracks, and object-action-fixture relations.
- Release and future use: The release includes 41 hours of multimodal recordings and annotations intended to support broader research on holistic egocentric perception.The authors state that HD-EPIC is intended to direct future research in this area.
Supplementary Material
The supplementary video showcases HD-EPIC’s annotation hierarchy, linked 3D digital twin, detailed sequence annotations, and examples from multiple kitchens.
- Video showcase: The showcase walks through annotation hierarchy, one fully annotated sequence, and examples of narrations, segmentations, audio, object-fixture assignments, and nutrition.The examples include running and total nutrition for ingredients and complete recipes.
B. Data Collection
The appendix details participant recruitment, equipment, and the post-processing and annotation sections covered in this part of the paper.
- The appendix directs readers to recruitment and equipment, collected-data instructions, narrations, and post-processing details.
- Participants received recording equipment packaged in one backpack for home data collection.
B.2. Instructions and Collected Data
HD-EPIC combines unscripted multi-day kitchen recordings with recipe, nutrition, narration, 3D, gaze, and audio-related processing, producing dense annotations across diverse activities.
- B.2. Instructions and Collected Data: Participants recorded unscripted daily kitchen activities for at least three consecutive days while wearing the glasses.
- B.2. Instructions and Collected Data: 41.3 hours across 156 videos and 9 participants were collected, averaging 15.9 minutes per video.
- B.2. Instructions and Collected Data: 69 recipes covered diverse cuisines, averaging 6.6 steps and 8.1 ingredients per recipe.
- B.2. Instructions and Collected Data: Participants interleaved multiple dishes, capturing concurrent preparation patterns absent from edited online videos and short controlled recordings.
- B.2. Instructions and Collected Data: Nutrition tracking combined weighed ingredients with manually logged MyFitnessPal values, while spices lacked captured nutrition values.
- B.3. Narrations: Participant narrations described actions with what, why, and how information, yielding denser and more detailed annotations than previous egocentric datasets.
- B.4. Post-Processing: Multi-Video SLAM and Gaze: Multi-video SLAM reconstructed a single kitchen point cloud from unscripted recordings across multiple days without additional scene scanning.
- C.4. Density of Our Annotations: 263 annotations per minute were provided on average, excluding hand masks generated at 30 fps.
D. Benchmark and Results
The benchmark and results section introduces supplementary experiment details, including VQA ablations, model descriptions, and additional recognition and segmentation evaluations.
- The supplementary material describes VQA question construction and sampling before presenting further benchmark details.
- Additional experiments cover VQA input ablations and prediction bias, model configurations, long-term VOS, and recognition benchmarks.
D.1. VQA Benchmark Details
The VQA benchmark samples diverse questions from annotations spanning recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze.
- D.1.1 Recipes: Recipe questions test recognizing recipes, localizing steps, and identifying activities associated with recipe-step completion.
- D.1.1 Recipes: Recipe recognition uses participant videos concatenated chronologically and manually edited similar-recipe negatives.
- D.1.2 Ingredients: Ingredient questions assess ingredient identity, exact weight, adding order, and temporal localization of adding segments.
- D.1.3 Nutrition: Nutrition questions compare ingredient nutrition from selected frames and measure changes in dish nutrition across adding segments.
- D.1.4 Fine-Grained Action: Fine-grained action questions evaluate recognition of 1–3 actions and descriptions of how actions were performed.
- D.1.5 3D Perception: 3D perception questions test reasoning about fixture locations from viewing direction using projected fixture meshes and visibility constraints.
- D.1.6 Object Motion: Object-motion questions assess movement itineraries, movement counts, and periods when tracked objects remain stationary.
- D.1.7 Gaze: Gaze estimation asks what the camera wearer is looking at in a trimmed video segment.
D.2. Additional VQA Experiments
The additional VQA experiments examine input ablations, model prediction bias, and evaluation procedures for long videos and related benchmarks.
- Answers alone produce random performance for Llama and Gemini, except on recipe and ingredient prototypes, while adding the question improves performance slightly.
- Ground-truth narrations outperform ground-truth action clusters across all question categories, showing the value of detailed short-term descriptions.
- 63.6% is achieved for fine-grained action questions with ground-truth narrations, while nutrition, 3D perception, object motion, and gaze remain difficult.
- 0.2% of questions were blocked or refused by the Gemini API and counted as failures.
- Action recognition uses a chance baseline from randomly shuffled ground-truth labels, while TIM additionally incorporates audio during inference.
D.4. Long-Term VOS Benchmark
The long-term video object segmentation benchmark evaluates models on lengthy annotated sequences and reports a frame-selection limitation that may constrain performance.
- 1000 sequences form the HD-EPIC long-term VOS benchmark, averaging 561.0 seconds per sequence.
- SAM2 and Cutie receive each object’s first appearance frame as working memory, while Static copies that mask across evaluation frames as a naive baseline.
- The benchmark evaluates segmentation with Jaccard index J, contour accuracy F, and their average.
- Because of the dataset’s large frame count, only memory and evaluation frames are passed to models, and additional frame sampling could improve results.
D.5. Recognition Benchmarks
The recognition benchmarks evaluate action and sound recognition with strong video models using preprocessing procedures from the original works.
- Action and sound recognition are evaluated as fundamental downstream tasks for video models.
- Five action recognition methods are assessed using publicly available weights fine-tuned on EPIC-KITCHENS-100.