Source-linked AI summary
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations
Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, Dima Damen
TL;DR
VISOR addresses the lack of pixel-level video annotations that remain consistent as hands and objects transform during egocentric activities. It introduces a scalable annotation pipeline and releases a large dataset with three benchmarks spanning video segmentation, hand-object interaction, and long-term source tracing. The benchmark analyses show diverse object and hand interactions, while source tracing remains difficult because temporally localising evidence frames is challenging.
Problem
Existing computer vision models tend to treat objects discretely, while egocentric activities require pixel-level annotations that capture hands, active objects, and transformations over short and long timescales.
Method
VISOR builds on EPIC-KITCHENS videos with an AI-assisted annotation pipeline and defines benchmarks for semi-supervised VOS, hand-object segmentation, and source tracing.
Results
The release contains 271.6K manual masks, 9.9M interpolated dense masks, and 67.2K hand-object contact states across 36 hours, while WDTCF source tracing achieves 24.8% accuracy for the model baseline.
Takeaways & Limitations
VISOR enables research on long-term understanding of action-object interplay, object transformations, and hand-object relations in egocentric video.
Takeaways & Limitations
Hand-object contact and segmentation remain difficult because of hand overlap and contact ambiguity, diverse held objects, and hand-object or hand-hand occlusion.
Abstract
from arXiv · showhide
We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning. For data, code and leaderboards: http://epic-kitchens.github.io/VISOR
1 Introduction
VISOR extends egocentric video understanding with pixel-level annotations that capture hands, active objects, and object transformations across time. It releases these annotations with three benchmarks spanning short-term segmentation, hand-object relations, and long-term source tracing.
- VISOR captures object transformations such as flour becoming dough through sparse pixel-level annotations of hands and active objects.The annotations represent both short actions such as adding salt and longer actions such as kneading dough.
- 271.6K masks across 50.7K images and 36 hours were interpolated into 9.9M dense masks for benchmark analysis.
- VISOR defines Semi-Supervised VOS, Hand Object Segmentation, and Where Did This Come From benchmarks at different temporal scales.The tasks cover tracking segments across actions, hand-object contact and segmentation, and tracing objects to their source containers.
2 Related Efforts
VISOR builds on prior datasets with pixel- and action-level annotations while emphasizing longer videos, diverse objects, and transformations beyond translation or deformation. Its design targets temporal consistency and hand-object understanding in egocentric activity.
- VISOR videos average 12 minutes, but are divided into approximately 12-second subsequences to make current-model evaluation feasible.
- VISOR provides longer sequences and the largest number of manually annotated masks among compared datasets with pixel- and action-level annotations.Its masks are semantically consistent across videos through class knowledge.
- VISOR addresses object transformations beyond translation and deformation, requiring long-term relations across sequences and semantic classes.
3 VISOR annotation pipeline
VISOR combines structured subsequence selection, entity discovery, AI-assisted manual segmentation, and propagation-based interpolation to scale annotations across diverse egocentric activities. The pipeline also collects metadata for object classes, hand relations, and benchmark analysis.
- 3.2 Tooling, annotation rules and annotator training: Annotators use AI-enhanced tooling alongside manual verification and extensive training to obtain consistent pixel-level masks.The project worked with 13 annotators over 22 months using the TORAS annotation suite.
- 3.1 Entities, frames and sub-sequences: Entity selection combines narration parsing, manual rules, and human annotations to identify active objects that narrations may omit.The selected entities are then manually segmented.
- 3.1 Entities, frames and sub-sequences: Untrimmed videos are divided into subsequences of three consecutive actions, with six sampled frames per subsequence.
- 3.4 Dense annotations: VISOR analysis covers a long-tailed distribution of entity classes, verb-entity and intra-entity co-occurrences, and diverse hand-object contacts.The dataset also records glove relations and spatial metadata for active objects and hands.
- 3.4 Dense annotations: Sparse manual annotations are propagated forward and backward between annotated frames, with logits averaged by distance from the manual frames.Interpolations longer than 5 seconds and objects absent from one endpoint are excluded, except hands are always interpolated.
4 Dataset Statistics and Analysis
VISOR captures diverse hand-object interactions and object categories, including frequent co-occurrences alongside a substantial long tail. Most objects are exhaustively annotated, while ambiguity and motion blur account for inconclusive cases.
- VISOR segments frequent hands, appliances, cutlery, and cookware while retaining rare entities such as watch and candle.
- Substantial co-occurrence entropy demonstrates dataset diversity beyond frequent action-object and object-object pairings.
- 54% of images with conclusive annotations for both hands include both hands contacting the world.
- 91.1% of objects are exhaustively annotated, while 4.1% are inexhaustive and 4.8% inconclusive.
- Motion blur and semantic ambiguity, including sauce and pan/pot distinctions, contribute to inconclusive samples.
5 Challenges and Baselines
VISOR defines three benchmarks spanning short-term segmentation, hand-object relations, and long-term source tracing. Baselines reveal strong hand segmentation but difficulty with contact recognition, temporal evidence localization, and challenging transformations.
- Challenges: Three challenges target short-term object transformations, in-frame hand-object relations, and ultra-long video understanding.
- VOS: VOS tracks multiple masks from the first frame through a short subsequence, while allowing occlusion and later reappearance.
- VOS: VOS failures commonly involve reference-frame occlusion, tiny objects, motion blur, and drastic transformations such as cutting or mixing.
- HOS: HOS segments hands and contacted or active objects, but contact identification remains challenging despite accurate hand segmentation.
- WDTCF: WDTCF traces a query mask to one of 15 source types using spatiotemporally grounded evidence frames.
- WDTCF: 24.8% accuracy for the model contrasts with 37% for the most frequent-source prior, while the prior cannot localize or segment evidence.
6 Conclusion and Next Steps
VISOR provides rich pixel-level annotations across space and time and introduces benchmarks for studying long-term interactions between actions and object transformations.
- VISOR releases rich pixel-level annotations across space and time with benchmarks for long-term action-object transformation understanding.
Checklist
The checklist documents dataset availability, reproducibility details, ethical sourcing, benchmark documentation, and known scope and resource limitations.
- Documentation: The paper describes its dataset, annotation pipeline, benchmarks, metrics, baselines, qualitative results, and limitations across the paper and appendix.
- Reproducibility: The dataset, code, instructions, splits, training details, and leaderboard are reported as publicly available or documented.Test annotations remain hidden, including those for the third challenge.
- Resources and limitations: The experiments lack error bars because repeated runs were considered too computationally costly.
- Assets and ethics: The work builds on EPIC-KITCHENS, follows its asset licenses, and reports consent and ethics approval for the underlying data.
- Scope and impact: The dataset is not representative of kitchen activities globally or within the recorded countries, so models are intended for exploratory research and investigation.
- Entity preparation: Additional-object lists address active objects omitted from narrations, including cutlery, utensils, furniture, and common kitchen tools.
B.2 Frame Extraction
VISOR samples frames to represent transformations within actions while balancing annotation efficiency, temporal coverage, and benchmark comparability.
- Frame selection: The majority of videos use variable-rate sampling with 6 frames per subsequence spanning 3 nonoverlapping actions.Only 7 videos use fixed-rate sampling.
- Frame selection: Variable-rate sampling concentrates frames within actions, reducing between-action frames from 10.2% to 3.7% relative to fixed-rate sampling.
- Frame selection: Fixed-rate videos are included to match standard benchmark practice and enable comparisons between sampling regimes.Three fixed-rate videos are annotated in Train/Val, while four are reserved for Test.
- Frame selection: Frames are sampled near 25% and 75% of each action, with motion-blur filtering and random shifts used to select representative images.
- Frame statistics: Most variable-rate actions contain 1–2 sampled frames, while 2720 actions have no annotated frames because they overlap other actions.
- Subsequences: Each subsequence contains six frames from three consecutive nonoverlapping actions with a consistent set of entities.
C Appendix - Annotation rules and annotator training (Main §2.2) / Annotator Training and Rules
VISOR trains annotators through structured onboarding and rules designed to produce temporally and cross-dataset consistent masks, supported by AI-assisted tooling.
- Training process: Annotators are recruited, trained, and instructed after frames and active entities have been identified.
- Training process: The onboarding tutorial covers project roles, the AI tool, step-by-step annotation, annotation rules, and example situations.
- Annotation rules: Annotation rules target temporal consistency within videos and consistency across the dataset while minimizing errors from automatic tools.
- Annotation rules: Annotators review six-frame subsequences, segment visible requested objects, and mark listed objects that are invisible in individual frames.
- Annotation rules: Rules specify how to handle containers, contained objects, occlusions, small pieces, transparent packaging, hands, and active objects.
- Annotation tooling: TORAS combines polygon-based manual editing with AI box, trace, correction, refinement, path, paint/erase, and boolean operations.
D.4 Annotation Efficiency
AI-assisted annotation reduces labeling time while preserving measured agreement, and a correction workflow checks masks for frame-level and temporal errors.
- Annotation efficiency: 37.5 seconds/object with AI tools versus 50.5 seconds/object manually reduced annotation time by approximately one quarter.The comparison used 50 images containing 253 objects.
- Annotation efficiency: 90.11 IoU with AI tools versus 89.92 IoU manually showed no accuracy degradation on the evaluated subset.The IoUs were within one standard deviation of each other.
- Annotation efficiency: Annotators achieved up to 2x speed improvement in images annotated per hour as the project progressed.
- Correction: The correction stage reviews annotations across images and sequences to confirm or correct pixel-level accuracy and temporal consistency.
- Correction: The correction interface displays image, mask, overlay, progress, and color-coded entity lists while collecting frame-specific comments.
- Correction: Train and Val videos typically receive one correction round, whereas every Test video receives two rounds.
E.3 Statistics of Collected Comments
This section details annotation quality controls, dense-mask interpolation, and semi-supervised VOS evaluation and results. VISOR uses manual verification and filtering to support consistent annotations and reports that small objects remain hardest to segment.
- Collected Comments: 24.5% of 50.7K images were returned for correction, averaging 1.8 comments per corrected image.80% of comments requested adding or renaming an active object, while 20% refined segmentation boundaries.
- Annotation Quality Control: Hive quality control combines qualifiers, gold-standard checks, and consensus labeling requiring six of up to nine workers to agree.Workers needed at least 80% on qualifiers and 90% accuracy on known samples to continue annotating.
- Annotation Quality Control: 295 frames with more than four potentially in-contact objects were manually annotated because Hive displayed only four colored options.Potentially in-contact objects were defined as segmented objects sharing a border with the hand.
- Dense Annotation Interpolation: Interpolated objects with J &F below 85% were filtered from the automatic dense annotations.STM was pretrained on MS-COCO, fine-tuned on VISOR, and used bidirectional inference for interpolation.
- Semi-Supervised VOS: The VOS benchmark tracks masks present in the first frame across remaining frames, excluding objects that first appear later.VISOR splits long videos into shorter subsequences and evaluates with Jaccard Index and Boundary F-Measure.
- VOS Evaluation: Small objects are harder to segment than medium and large objects in VISOR evaluation.The benchmark reports size- and class-based performance for STM models trained on different datasets.
H.4 Additional Results
Additional HOS results analyze performance by object size and class while documenting hand-object annotation conventions and evaluation setup. VISOR fine-tuning yields its largest class margins for hands and spatula, whereas knife and fork remain difficult.
- Size-based Performance: 16% better performance was achieved by the VISOR-fine-tuned model than by the COCO-pretrained model fine-tuned on YTVOS and DAVIS for small objects.The gap was 7.4% for medium objects and 10.6% for large objects.
- Class-based Performance: The VISOR fine-tuning advantage is largest for left/right hands and spatula, but smaller for COCO classes such as pie and food.The comparison covers the best and worst EPIC-KITCHENS-100 classes for three models.
- Class-based Performance: Knife and fork have poor scores because their appearance changes across subsequences and they are often occluded during use.These difficulties arise from viewpoint or orientation changes and functional occlusion as tools are used.
- Annotation Conventions: When worn, gloves are included in the hand mask and their contacted object is treated as being in contact with the hand.Gloves not worn on hands remain ordinary object masks.
- HOS Setup: Hand-Contact-Relations evaluates hand and contacted-object instances with hand side, contact state, and offset annotations.Hand-And-Active-Objects instead segments hands and all other objects.
- HOS Setup: HOS baselines use PointRend with an R50-FPN backbone, while Hand-Contact-Relations adds predictors for hand side, contact state, and offset.Evaluation uses COCO Mask AP and retains only images with conclusive validation and test annotations.
I.4 Additional Results
This section presents qualitative HOS validation results. The examples show accurate hand segments and many correctly segmented objects, while the task remains challenging.
- Qualitative Results: Qualitative HOS validation results show often accurate hand segments and many correctly segmented objects.The examples are presented on the validation set.
J.1 Data Preparation and Annotation
The Where Did This Come From? benchmark selects query and evidence frames to identify a query object’s source, while evaluating source classification and segmentation. Its annotations and baselines expose challenges from long temporal gaps, occlusion, and competing source candidates.
- Data Preparation and Annotation: Query candidates are selected from linked object occurrences, while evidence candidates are the first co-occurring frames with one of 15 containers.Three query and three evidence candidates are used per object.
- Data Preparation and Annotation: The annotation interface records the video, query object, query and evidence candidates, and source candidates.For example, yogurt is shown emerging from a fridge, with one evidence frame annotated for the taster challenge.
- Data Preparation and Annotation: 19.4% of query-evidence gaps exceed 10 minutes, although many gaps fall within 2 minutes.The benchmark includes 222 test examples, and the gap distribution varies substantially.
- Data Preparation and Annotation: Each method predicts a source class, an evidence frame, and source and query segments in that evidence frame.Evaluation uses sparse frames; Query IoU and Source IoU are zero when the evidence frame is not localized correctly.
- Baseline Details: PointRend obtains 90.5% overall query-mask prediction accuracy, but tiny objects and liquids are difficult to detect.The examples include a correctly detected bottle and an occluded yogurt object that is missed.
- Baseline Details: Evidence-frame segmentation is straightforward with one detected source candidate but difficult when multiple candidates, such as a bottle and cupboard, are present.Additional qualitative results are shown alongside ground-truth and PointRend predictions.
Q. What (other) tasks could the dataset be used for?
VISOR supports long-term pixel-grounded video understanding, but its uses are bounded by annotation selection, dataset composition, class coverage, licensing, and inherited source-data constraints.
- Potential Uses: VISOR is intended to support many long-term pixel-grounded video understanding tasks.The dataset is publicly available for non-commercial research purposes.
- Composition and Preprocessing: Annotated frames are selected for ease of annotation and may contain little motion blur, so motion-blur augmentation may be needed for generalisation.This is a stated preprocessing consideration for future users.
- Composition and Preprocessing: The data contain fewer unique individuals than internet data, which may make generalisation harder.This limitation follows from the collection process.
- Composition and Preprocessing: Verb and noun classes do not cover all actions and objects present in the recorded kitchens.The class vocabulary therefore bounds coverage of possible kitchen activities and entities.
- Composition and Preprocessing: The dataset is naturally long-tailed, so models will be biased toward better recognition of head classes.Class imbalance is identified as a property affecting downstream modeling.
- Usage Restrictions: VISOR is restricted to non-commercial research use under a Creative Commons BY-NC 4.0 license.A commercial license can be negotiated through the University of Bristol.