PaperlayerView exact revision on arXiv

Source-linked AI summary

Rescaling Egocentric Vision

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray

arXiv:2006.13256v4cs.CVcs.LG

TL;DR

The paper addresses the limited number of tasks supported by action-understanding datasets by extending EPIC-KITCHENS with a scalable annotation pipeline and new footage. EPIC-KITCHENS-100 provides denser, more complete annotations and six challenges with baselines and evaluation metrics, while exposing performance changes across footage collected two years apart.

  • Problem

    Action and activity understanding datasets support relatively few tasks, while models trained on earlier footage may face changed source and target distributions over time.

  • Method

    The paper constructs EPIC-KITCHENS-100 using a scalable pause-and-talk annotation pipeline and defines six challenges on the resulting dataset.

  • Results

    The dataset provides denser and more complete action annotations, six challenge benchmarks, and evidence that models trained only on earlier videos do not withstand the test of time.

  • Takeaways & Limitations

    EPIC-KITCHENS-100 supports shared evaluation of fine-grained action understanding across recognition, detection, anticipation, retrieval, and domain adaptation settings.

  • Takeaways & Limitations

    Bounding-box correctness for hands and objects has been evaluated, but R-CNN masks have not been quantitatively evaluated and are error-prone.

Abstract

from arXiv · show

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the "test of time" - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics

1 Introduction and Related Datasets

EPIC-KITCHENS-100 extends egocentric action datasets with unscripted kitchen footage, richer annotations, and multiple challenges on shared video data.

  • Annotation and benchmarks: The paper frames EPIC-KITCHENS-100 as a multi-task benchmark for fine-grained video understanding rather than a dataset for one isolated task.The supplied passages describe the dataset as supporting shared-input learning across several annotations and challenges.
  • Dataset motivation: Action and activity understanding datasets have proposed fewer tasks than datasets for driving and scene understanding.Earlier video benchmarks nevertheless demonstrated multi-challenge evaluation, including localisation, dense captioning, object detection, and active speaker detection.
  • Dataset contribution: EPIC-KITCHENS-100 extends EPIC-KITCHENS with 100 hours of diverse, unscripted and unedited kitchen interactions.Its domain-specific footage captures naturally unbalanced hand-object interactions and novel action compositions across environments.
  • Dataset contribution: The dataset adds two-level noun and verb annotations to support domain knowledge in fine-grained kitchen activities.Examples group terms such as carrot/courgette under vegetable and put/throw/drop under leave.
  • Annotation and benchmarks: A refined annotation pipeline produces denser and more complete action annotations in untrimmed videos.The pipeline supports multiple tasks on the same dataset and provides baselines and evaluation metrics for six challenges.

2 Data Collection and Scalable Pipeline

EPIC-KITCHENS-100 uses a scalable collection and annotation pipeline centered on pause-and-talk narration, visual transcription support, temporal segmentation, and parsing.

  • Data collection: 37 participants and 45 kitchen environments contribute additional footage collected over 2–4 days of typical kitchen activities.The collection includes returning participants and newly recruited subjects, including participants who moved homes.
  • Narration: The pause-and-talk narrator lets participants pause video while speaking actions in their native language, reducing mental overload and improving annotation density.Participants can take breaks and correct recordings while maintaining a scalable one-pass viewing process.
  • Transcription: The pipeline transcribes narrations, translates unique transcriptions into English when needed, and uses visual information to reduce transcription errors.The revised transcriber addresses failures caused by workers lacking visual context.
  • Parsing: The parser extracts verbs and nouns, groups them into minimally overlapping classes, and reparses prior annotations with the updated pipeline.The update improves handling of compound nouns and missing verbs or nouns.
  • Temporal segmentation: Five workers annotate temporal bounds for each action instance, including repeated consecutive instances, to avoid merging actions.The pipeline prompts annotators from narration timestamps and increases the worker count from four to five.
  • Quality improvements: Pause-and-talk produces denser videos, fewer gaps, more labelled frames, shorter actions, and greater overlap than non-stop narration.Its timestamps are also closer to the relevant actions, with more contained timestamps and smaller distances to remaining timestamps.
  • Quality improvements: 56 annotated actions replace 20 in equal-length example sequences using pause-and-talk narration.Short and overlapping actions are captured more reliably, and denser narration makes temporal boundaries more accurate.

3 Statistics, Scalability and the Test of Time

EPIC-KITCHENS-100 expands the dataset to nearly twice the footage and 2.3 times the action segments, while supporting tests of temporal generalisation and data scalability. Its annotations and splits also enable analyses of long-tail classes, unseen participants, and spatial hand-object information.

  • Statistics: 89,977 fine-grained action segments span 700 long videos and 100 hours of footage.The extension rescales the previous dataset to 1.8x the hours and 2.3x the action segments.
  • Statistics: The dataset has a clear long-tail distribution across 97 verb and 300 noun classes, with some classes appearing only in newly collected videos.Examples include the new verb classes ‘lock’ and ‘bend’, and noun classes ‘orange’ and ‘hoover’.
  • Statistics: Automatic spatial annotations provide 66M masks, 31M hand boxes, and 38M object boxes for future spatial or spatio-temporal attention models.The annotations combine Mask R-CNN detections with hand-object interaction detections, including multiple object interactions.
  • Splits: The unseen-participant subsets contain 1,065 validation and 4,110 test action segments from participants absent from training.These subsets evaluate model generalisability across the benchmarks.
  • Scalability and the Test of Time: Models trained only on the original videos do not withstand the test of time, while adding new data improves performance but saturates after 50% of the new data.The effect is particularly limited for unseen participants, motivating better models and more diverse data.
  • Scalability and the Test of Time: The evaluation includes overall and unseen-participant action-recognition results for both temporal generalisation and scalability.The test examines older training footage on videos collected two years later, while scalability progressively adds new annotated data.

– Hardware and capturing as in [53,55]. Extended

The extension introduces several sources of domain variation beyond ordinary dataset changes, including camera, location, participants, temporal conditions, and long-term environmental changes. These variations form a compound domain gap that the dataset explores without using property labels.

  • Hardware and capturing as in [53,55]. Extended: The extension uses a newer camera model with onboard video stabilisation, creating a capture-method difference from the original footage.This is one of the documented sources of domain variation.
  • Hardware and capturing as in [53,55]. Extended: Eight participants moved home, changing surroundings while preserving the appearance of many objects and tools.Additional unseen participants also recorded in environments with different object and surroundings appearances.
  • Hardware and capturing as in [53,55]. Extended: Short-term temporal offsets alter scene lighting and background-object positions, while long-term offsets include wear, new objects, and changed object positions.The newer footage was filmed two years after EPIC-KITCHENS-55, and participant behaviour can also change over time.
  • Hardware and capturing as in [53,55]. Extended: The dataset explores this compound domain gap without using property labels, including changes in behaviour and object appearance that are difficult to label.Some properties have labels, but other changes vary between samples without associated labels.

4 Challenges and Baselines

EPIC-KITCHENS-100 defines six challenges on one densely annotated dataset, spanning recognition, detection, anticipation, retrieval, and domain adaptation, with task definitions, baselines, and metrics.

  • Challenge suite: Six challenges combine two modified tasks with four new ones, including weakly supervised recognition, detection, domain adaptation, and retrieval.The modified tasks are action recognition and anticipation.
  • Action recognition: Action recognition predicts verb, noun, and action classes for video segments, evaluated with Top-1/5 accuracy on validation and test sets.Overlapping segments are considered independently.
  • Weak supervision: Weak supervision uses single narration timestamps rather than temporal boundaries, and performs 11% worse than strong supervision on top-1 action accuracy.The benchmark is challenging because short, overlapping actions are difficult to localize from roughly aligned timestamps.
  • Action detection: Action detection predicts temporal boundaries and verb, noun, and action classes, with mAP computed across IoU thresholds from 0.1 to 0.5.The BMN-plus-SlowFast baseline finds detection particularly challenging, especially at higher IoU thresholds.
  • Domain adaptation: Unsupervised domain adaptation uses 2018 videos as labelled source data and newly collected videos as unlabelled target data.Multimodal TA3N partially reduces the domain gap, improving verb accuracy by 2.5% and noun accuracy by 2.4%.
  • Action retrieval: Action retrieval evaluates semantic caption relevance with nDCG, while JPoSE outperforms an MLP baseline but both struggle with longer captions.Relevance is defined from the mean IoU of verb and noun classes, rather than binary matching alone.

5 Conclusion and Future Work

EPIC-KITCHENS-100 is presented as a large-scale egocentric dataset built with a scalable, higher-quality annotation pipeline and organized around six challenges. The authors release the dataset, annotations, models, evaluation scripts, and leaderboards to support further research and collection efforts.

  • The paper presents EPIC-KITCHENS-100 through a scalable annotation pipeline described as higher quality than previous approaches.
  • Six challenges provide leaderboard baselines for action understanding and highlight retrieval, domain adaptation, and dense overlapping-action detection.The challenges use free-form captions, footage collected two years later, and dense annotations of overlapping actions.
  • The dataset, annotations, models, evaluation scripts, challenge leaderboards, and updates are made available through the project resources.
  • The released pause-and-talk narrator and visually supported transcription interfaces are intended to support other large-scale data-collection efforts.

Appendices

The appendices document the annotation demonstration and collection procedures, emphasizing pause-and-talk narration, visually supported transcription, and timestamp-based temporal annotation. These changes were designed to improve annotation density, correctness, and agreement.

  • Annotation pipeline demonstration: The demonstration follows one sequence through narration, transcription, translation, parsing, class mapping, and automatic annotations.
  • Annotation pipeline demonstration: The video also presents predictions for the six challenges on a training sequence to illustrate their potential and links.The displayed performance is described as near-perfect rather than representative of current baseline performance.
  • Collection and annotation procedures: Head-mounted GoPro Hero 7 cameras recorded the collection at 50fps with video stabilisation.The 50fps choice avoids overhead-light flickering associated with the frame rate and national-grid frequency mismatch in the previous collection.
  • Collection and annotation procedures: The pause-and-talk narrator produces single-action audio clips, replacing previously combined 30-second speech chunks.
  • Collection and annotation procedures: Images centered at {−0.25s, 0s, +0.25s} around each narration timestamp provide transcription context.
  • Collection and annotation procedures: Temporal annotators refine start and end times from accurate single timestamps instead of assuming action length equals narration length.
  • Collection and annotation procedures: Five annotators per segment, compared with four previously, resulted in higher agreement between annotators.

C Challenges’ Implementation Details

This section provides implementation and training details for all baselines, together with challenge-specific information such as evaluation-metric definitions, to enable replication.

  • Implementation and training details are provided for all baselines to enable replication of the reported results.The section also includes additional challenge-specific details, including definitions of evaluation metrics.

C.1 Action Recognition

The action-recognition baselines use publicly available video models with RGB and optical-flow inputs, separate verb and noun predictions, and combined action scores. Training configurations vary across architectures but use standard supervised optimization and pretrained backbones.

  • Model inputs and outputs: TSN, TRN, and TSM use ResNet-50 backbones with ImageNet or Kinetics initialisations and separate RGB and optical-flow inputs.Each model uses 8 RGB frames or 8 stacks of 5 optical-flow fields.
  • Model inputs and outputs: The models predict verbs and nouns with a two-way output, and action classes come from the most likely verb-noun combinations.
  • Training configurations: TSN and TRN train on 8 GPUs with batch-size 128, while TSM trains on 4 GPUs with batch-size 64 for 80 epochs.The learning rate starts at 0.01 and is decayed at epochs 20 and 40 by a factor of 10.
  • Training configurations: TBN training uses batch size 64, six segments, and learning-rate drops at epochs 40 and 60.
  • Training configurations: SlowFast uses an 8x8 ResNet-50 Kinetics-pretrained backbone and randomly sampled 1-second clips during training.Its training lasts 30 epochs, with learning-rate decay at epochs 20 and 25.

C.2 Weakly-Supervised Action Recognition

The weakly supervised action-recognition baselines use TSN-based RGB and flow models, with fixed-length or iteratively updated frame sampling and late fusion. Temporal proposals are generated with BMN and classified by SlowFast.

  • Baselines: TSN baselines use Inception with batch normalization, Kinetics-400 pretraining, and late-fused RGB and flow predictions.Testing uses 25 RGB frames or optical-flow stacks.
  • Baselines: The fixed-length baseline divides 5-second segments into equal parts and randomly samples frames, while the alternative updates sampling distributions iteratively.Both baselines sample five frames for training.
  • Proposal generation: BMN generates temporal action proposals from TSN-based features using a rescaled observation window and maximum action length of 400.The observation window and maximum action length are both set to 400.
  • Proposal generation: Soft Non-Maximum Suppression reduces overlapping proposals, retaining the top 1,000 instances per video before classification.Proposals shorter than 10 frames are discarded during update generation.
  • Proposal classification: Each proposal is classified with SlowFast, using a training-set model for validation and a training-plus-validation model for testing.

C.4 Action Anticipation

Action anticipation uses RGB, optical-flow, and object features with an RU-LSTM architecture and modality attention. An oracle based on the observed action outperforms chance but remains below the RU-LSTM baseline.

  • Features and model: The anticipation pipeline extracts RGB and flow features with TSN and object-based features from a detector trained on manually annotated object boxes.
  • Features and model: RU-LSTM trains single-modality branches with sequence completion, fine-tunes them for anticipation, and combines them through modality attention.The full architecture is trained for 29 epochs.
  • Results: 20.84%, 25.00% and 8.92% oracle accuracy are obtained for Top-5 verb, noun and action labels, respectively, on validation data.The oracle uses the ground-truth observed action and therefore is not reproducible during inference.
  • Results: The oracle exceeds the chance baselines of 6.39%, 2.00% and 0.20% for Top-5 verb, noun and action labels, respectively, but remains below the RU-LSTM baseline.

C.5 Unsupervised Domain Adaptation (UDA) for Action Recognition

The UDA setup tunes models using labelled source and unlabelled target validation splits, then retrains on the corresponding training splits for target-test evaluation. It also defines retrieval relevance from verb and noun overlap and ranks captions with nDCG.

  • Validation protocol: Source Val and Target Val use four participants, while Source Train and Target Train use the remaining twelve participants.The validation participants include two returning and two changing kitchens.
  • Validation protocol: Hyper-parameters are selected using labelled Source Val and unlabelled Target Val performance before final evaluation on Target Test.The leaderboard model is trained on labelled Source Train and unlabelled Target Train.
  • Domain scope: Only 0.2% of target-domain verb classes and 2.3% of noun classes are absent from the source domain, while 9.46% of target actions are zero-shot verb-noun combinations.Zero-shot actions can still be predicted when their component verbs and nouns are known.
  • UDA implementation: TA3N is extended for multimodal input by concatenating modality features, and its domain discriminators are initialized before gradient reversal and back-propagation.
  • Retrieval evaluation: Video-caption relevance averages the Intersection-over-Union of verb and noun classes, using the corresponding verb and noun class sets.
  • Retrieval evaluation: nDCG compares ranked-caption DCG with an ideal ranking formed by sorting captions by video-caption relevance, with stable ID ordering for ties.
  • Retrieval baselines: The MLP baseline projects visual and textual features into a shared embedding space and samples triplets using semantic relevance based on identical verb and noun classes.The final embedding size is 512, and the text-to-visual triplet-loss weight is 2.0.
Loading 2006.13256v4…