Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

Fadime Sener; Dibyadip Chatterjee; Daniel Shelepov; Kun He; Dipika Singhania; Robert Wang; Angela Yao

Source-linked AI summary

Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, Angela Yao

arXiv:2203.14712v2Mar 28, 2022cs.CV

TL;DR

Procedural activity understanding lacks large-scale, non-scripted datasets with diverse sequencing, multi-view recordings, and rich annotations. Assembly101 addresses this gap with synchronized static and egocentric toy-assembly videos, multi-granular actions, 3D hand poses, and mistake labels. It benchmarks recognition, anticipation, temporal segmentation, and mistake detection across varied generalization settings.

Problem
Existing datasets often use produced instructional videos or are small and limited in task or sequence diversity, leaving goal-oriented procedural activities underrepresented.
Method
Assembly101 records non-scripted assembly and disassembly of 101 take-apart toy vehicles from synchronized static and egocentric views with multi-granular action, mistake, and 3D hand-pose annotations.
Results
Assembly101 benchmarks action recognition, anticipation, temporal segmentation, and mistake detection, while supporting analyses of new-toy generalization, cross-view transfer, long-tailed distributions, skill level, and pose versus appearance.
Takeaways & Limitations
The dataset provides a broad challenge for studying goal-oriented sequence learning, multi-view action understanding, and 3D hand-object interaction recognition.

Abstract

from arXiv · show

Assembly101 is a new procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 "take-apart" toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natural variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with simultaneous static (8) and egocentric (4) recordings. Sequences are annotated with more than 100K coarse and 1M fine-grained action segments, and 18M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal segmentation. Additionally, we propose a novel task of detecting mistakes. The unique recording format and rich set of annotations allow us to investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose vs. appearance. We envision that Assembly101 will serve as a new challenge to investigate various activity understanding problems.

1. Introduction

Assembly101 addresses the lack of large-scale, goal-oriented procedural datasets with diverse sequencing by combining synchronized static and egocentric recordings with rich action and mistake annotations.

Dataset gap: Existing instructional datasets are produced online videos with multiple shots, while non-kitchen goal-oriented datasets are often small or limited in task and sequence diversity.These limitations motivate a dataset recorded in a controlled, embodied setting.
Dataset contribution: Assembly101 contains 362 unique sequences of people assembling and disassembling 101 take-apart toy vehicles across 8 static and 4 egocentric viewpoints.The dataset totals 4321 sequences and 513 hours of footage.
Annotations: The dataset combines fine-grained and coarse actions, 3D hand poses, skill levels, and mistake labels for sequence understanding.Mistakes and corrections are annotated on coarse segments, while actions are represented at multiple granularities.
Dataset contribution: The activities are non-scripted and goal-oriented, with natural variation in action ordering, participant skill, mistakes, and corrections.Unlike strictly ordered recipes or scripts, participants work through assembly and disassembly procedures without fixed instructions.
Multi-view setting: Synchronized outside-in and egocentric views support full 3D hand-object understanding and transfer between viewpoints.Static views provide information unavailable in egocentric recordings alone.

2. A Comparison of Action Datasets

Assembly101 extends action-dataset coverage beyond curated instructional videos and constrained recorded benchmarks by supporting multi-step, multi-view, and diverse action-understanding research.

Existing datasets: Online instructional datasets offer scale and variety but often contain produced footage with mixed viewpoints, fast-forwarding, and unrelated narration unsuitable for AR settings.Their primary emphasis is often multimodal vision-plus-language learning.
Existing datasets: Recorded multi-step datasets are commonly small or provide little variation in action ordering, while existing assembly datasets have limited scale.These constraints reduce their suitability for studying sequencing and deep learning.
Assembly101: Assembly101 combines multi-step activities with synchronized third-person and egocentric views, enabling cross-view domain analysis.The setting provides simultaneous privileged outside-in information and multi-view egocentric data.
Assembly101: Its multi-granular labels support both short-horizon fine-grained and long-horizon coarse action anticipation.Fine-grained actions last a few seconds, whereas coarse actions support longer temporal reasoning.
Assembly101: Assembly101 introduces procedural mistake detection, addressing a task not previously studied in procedural activities according to the authors.Coarse assembly segments carry mistake labels alongside action annotations.

3. Recording and Annotation

Assembly101 uses a synchronized desk-based camera rig and structured multi-granular annotations to capture diverse toy-assembly interactions, including 3D hand poses.

Recording setup: The recording rig uses 8 RGB cameras at 1920 × 1080 and 4 monochrome headset cameras at 640 × 480.Five RGB cameras are overhead and three are mounted on the side; the monochrome cameras occupy the headset corners.
Recording setup: All cameras are synchronized with SMPTE timecode and geometrically calibrated to sub-pixel accuracy.Participants are positioned so their hands and assembled toys remain visible across views.
Participants and toys: The study recruited 53 adults to assemble and disassemble 101 unique toys spanning 15 vehicle categories.Toy categories include construction and emergency-response vehicles with variations in colour, size, and style.
Annotations: Fine-grained labels mark hand-object interactions across pre-contact, interaction, and post-contact stages, while sequential actions are merged into coarse actions.The annotation interface uses multiple static views and labelled toy diagrams to support temporal and object-identification accuracy.
3D hand poses: 3D hand poses are estimated by fusing four egocentric views, regressing hand joint angles and global transformations, and extracting 21 keypoints per hand.The resulting keypoints are represented in world coordinates.

4. Dataset Statistics

Assembly101 provides large-scale, multi-view procedural activity data with fine- and coarse-grained labels, substantial 3D hand-pose coverage, and natural temporal variation including mistakes and corrections.

Dataset scale and recording: 4321 videos from 362 sequences cover 101 toy vehicles across 12 viewpoints and 513 hours of footage.Each video averages 7.1 ± 3.4 minutes.
Dataset scale and recording: More than 1M fine-grained and 100K coarse action segments make Assembly101 the largest procedural activity dataset described here.The dataset is also reported as the largest in number of segments and richest in multi-view recordings.
Label structure: 1380 fine-grained labels combine 90 objects, five tools plus the hand, and 24 interaction verbs, with a natural long tail.Thirty percent of the data accounts for 1238, or 89%, of fine-grained actions.
Label structure: 202 coarse actions represent assembly or disassembly of vehicle parts, average 24 actions per sequence, and average 16.5 ± 15.7 seconds per action.Each coarse action comprises an average of 10 fine-grained actions.
3D hand poses: Compared with FPHA and H2O, Assembly101 contains 82× more action segments and 200× more frames with 3D hand-pose annotations.These annotations support hand-object action understanding and pose-based analysis.

5. Benchmark Experiments

Assembly101 benchmarks recognition, anticipation, temporal segmentation, and mistake detection across views, skills, class frequencies, toy novelty, and pose versus appearance. Results expose substantial gaps across viewpoints, long-tailed classes, unseen toys, procedural complexity, and mistake detection.

5.1. Recognition, anticipation & segmentation: Four tasks evaluate action recognition, one-second anticipation, temporal segmentation, and three-way mistake detection.Recognition uses fine-grained pre-trimmed clips; segmentation predicts frame-wise labels, while mistake detection distinguishes correct, mistake, and correction segments.
5.2. Camera viewpoints: 16.2% lower Overall recognition, 4.9% lower Overall anticipation recall, and 6.5% lower segmentation MoF occur for egocentric than fixed viewpoints.Models are trained jointly on fixed and egocentric instances but evaluated separately by view.
5.3. Head vs. tail classes: 37% separates head and tail action recognition accuracy, while 89% of action classes are tail classes.The reported imbalance motivates few-shot action recognition methods.
5.4. Seen vs. unseen, assembly vs. disassembly: Seen toys outperform unseen toys across recognition, anticipation, and segmentation, with the largest recognition and anticipation gap occurring for objects.Verbs are shared across toys, whereas objects are not, with 13% unseen objects.
5.5. 3D pose-based action recognition: 28.7% is the highest reported pose-based action accuracy, still 5.1% below fused egocentric video recognition.Pose recognition has 6.7% higher verb accuracy but 10.2% lower object accuracy than video-based recognition; context improves pose performance.
5.5. 3D pose-based action recognition: 23.2% separates EPIC-KITCHENS pre-training from native Assembly101 pre-training, while pose features outperform features from the other datasets.The comparison indicates a considerable domain gap between Assembly101 and existing recognition benchmarks.
5.4. Seen vs. unseen, assembly vs. disassembly: Assembly segmentation is consistently lower than disassembly for the least skilled groups, likely because assembly has more ordering variations and mistake segments.Skill has little impact on disassembly, while assembly complexity produces lower scores for groups a1 and a2.
5.7. Mistake detection: 46.6% mistake recall and 29.6% correction recall are achieved with TSM features after the segment ends, dropping another 11.6% and 3.2% for early prediction.Even the GT coarse oracle reaches only 62.7% mistake recall and 84.9% correction recall.

6. Conclusion

The paper presents Assembly101 as a large procedural activity dataset combining synchronized views, multi-granular action segments, mistake labels, and 3D hand poses. It positions the dataset as a basis for broad assembly-task challenges and applications including recognizing actions, predicting steps, and detecting non-compliant behavior.

6. Conclusion: Assembly101 combines synchronized egocentric and static views, multi-granular action segments, mistake labels, and 3D hand poses.The benchmark covers recognition, anticipation, temporal segmentation, mistake detection, new-toy generalization, cross-view transfer, long-tailed distributions, skill level, and pose versus appearance.
6. Conclusion: Existing methods show promising results but remain far from high-precision performance, including in oracle experiments.The conclusion identifies room for future exploration across the dataset’s challenges.
6. Conclusion: Assembly101 is proposed for assistants that recognize current actions, predict subsequent steps, check non-compliant steps, and offer alerts or help.The paper also invites additional applications and tasks after release.

Supplementary Assembly101: A Large-Scale Multi-View Video Dataset

The supplementary material documents Assembly101’s recording setup, annotation interface, label distributions, dataset statistics, baseline implementations, and additional experimental comparisons. Its overview centers on the dataset’s multi-view capture and annotation resources.

Recording setup: Figure 1 presents the desk-based rig with sample frames from eight RGB and four monochrome cameras.The figure provides the visual overview of the multi-camera recording arrangement.
Annotation interface: Figure 2 shows annotation input views, predefined verb/tool/object lists, free-form entry, temporal boundaries, and the list of annotated actions.The interface combines video inspection, label selection, and temporal action marking.
Supplementary overview: The supplementary covers recording settings, annotations, experiments, label distributions, train/validation/test statistics, related-dataset comparisons, baseline architecture, and further results.These topics are organized across four supplementary sections.

1. Recording and Annotation

Assembly101 was recorded with a dedicated multi-camera rig and annotated using a custom interface designed for temporally precise, visible action labeling. The dataset combines participant sessions, multi-view footage, structured and free-form labels, and annotator training.

Recording: Each sequence uses eight RGB cameras at 1920 × 1080 and four monochrome cameras at 640 × 480 resolution.The dedicated desk-based rig captures the dataset’s synchronized views.
Participants and data release: 53 adults recorded approximately one-hour sessions across 18 consecutive days, with informed consent for digital capture.The footage and collected annotations were made available to the research community.
Annotation: Annotators marked fine-grained segment start and end frames using predefined verbs, tools, and objects, with optional free-form labels.Three static views were displayed to reduce self-occlusion and improve action visibility.
Annotator training: Annotators received four days of training before labeling was completed by 21 annotators over 213 hours.Slow annotators or those making many mistakes were not selected to continue.
Label space: The label vocabulary comprises 90 objects and 24 verbs, forming 1380 fine-grained action labels.The object and verb definitions are summarized in Figure 3.

2. Dataset Statistics & Splits

Assembly101 defines fine- and coarse-grained action labels over multi-view assembly sequences, with splits designed to expose long-tailed classes, unseen toys, and participant-skill variation. Its scale includes 4321 videos, over 1M fine-grained segments, and extensive 3D hand-pose data.

Fine-grained actions: 1380 fine-grained labels combine 90 objects and 24 verbs, including six attempt verbs for incomplete actions.Fine-grained actions can span reaching, grasping, and lifting stages; coarse actions combine sequential fine-grained interactions.
Coarse actions: 202 coarse actions use 11 verbs and 61 objects, with each video averaging 24 coarse actions and each coarse action averaging 10 fine-grained actions.Coarse actions correspond to assembling or disassembling vehicle parts; assembly sequences average 14 coarse actions and disassembly sequences average 10.
Training, validation, and test splits: The dataset uses a 60/15/25 recording split for training, validation, and test, with held-out test annotations reserved for challenge leaderboards.Validation and test sets examine generalization under varying conditions, including unseen toys and long-tailed labels.
Long-tailed and unseen classes: 30% of the training data contains 1238 (89%) tail fine-grained action classes and 171 (84%) tail coarse action classes.The dataset also ensures unseen toys across splits: only 25 of 101 toys are shared across all three splits.
Participant skill: Participants receive skill labels from 1 to 5 based on assembly speed and mistakes, with 47% of sequences assigned the highest level.The sequence distribution from levels 1 through 5 is 9%, 6%, 13%, 25%, and 47%.

3. Implementation Details

The benchmark covers recognition, anticipation, temporal segmentation, and mistake recognition using established video, pose, and temporal models. Evaluation uses task-specific metrics and temporal context, including long-range features for mistake detection.

Tasks: Four challenges are evaluated: action recognition, anticipation, temporal segmentation, and mistake recognition.
Action recognition: TSM with a ResNet-50 backbone serves as the action-recognition baseline, with either one action head or separate object and verb heads.Both variants are trained with SGD for 50 epochs.
Anticipation: Anticipation predicts upcoming fine-level actions 1 second before onset and reports class-mean Top-5 recall using TempAgg.TempAgg aggregates past snippets over multiple temporal scales and predicts objects, verbs, and actions separately.
Temporal segmentation: Temporal segmentation compares MS-TCN++ and C2F-TCN using mean frame-wise accuracy.MS-TCN++ uses dilated temporal convolutions, whereas C2F-TCN uses a shrink-then-expand encoder-decoder.
Mistake detection: Mistake detection uses a modified TempAgg model designed to capture relationships spanning several minutes around each action segment.The model uses a 60-second surrounding context and weighted cross-entropy for mistake and correction classes.
Evaluation: Recognition is evaluated by Top-5 accuracy, while fused multi-view recognition and anticipation use Top-1 accuracy and Top-5 recall, respectively.Fusion averages scores from multiple views.

4. Results

Multi-view score fusion substantially improves action recognition, while helping anticipation less. Performance varies across skill levels, toy categories, and label types, with tail verbs generally harder to recall.

Multi-view performance: Multi-view score fusion significantly increases recognition performance, while producing a smaller improvement for anticipation.The comparison uses overall results and fused scores across multiple views.
Skill-level analysis: Recognition and anticipation show no significant difference across participant skill levels.The authors suggest skill may be more relevant to coarse actions because these tasks use fine-level labels.
Toy-category analysis: The transporter category achieves the highest recognition and segmentation score, potentially reflecting its 22 participant recordings despite containing only four toys.The authors associate the high performance with the large number of recordings.
Fine-grained class evaluation: The verbs “clap,” “pick up,” and “put down” have the highest fine-grained recall, while tail verbs involving “attempt to” have the lowest.Enumerated wheels are among the top object classes in fine-grained evaluation.
Coarse class evaluation: For coarse actions, “demonstrate,” “attach,” and “detach” have the highest verb recall, while “position,” “remove,” and “attempt to screw” have the lowest.“Chassis” and “interior” are the highest-recall objects and the most common objects across toys.