Source-linked AI summary
The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose
Yizhak Ben-Shabat, Xin Yu, Fatemeh Sadat Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, Stephen Gould
TL;DR
Existing datasets for human activity understanding often lack task-specific, multi-view and richly annotated data for furniture assembly. IKEA ASM addresses this gap with a multi-view, multi-modal dataset and benchmarks across vision tasks, while its real-world visual similarities expose segmentation limitations.
Problem
Existing assembly datasets cover few actions and mainly use pose and color information, while broader datasets often center on single-view RGB data.
Method
The paper constructs IKEA ASM with multi-view, multi-modal, fine-grained annotations and evaluates baseline methods across action recognition, segmentation, tracking and pose estimation.
Results
The dataset provides a challenging benchmark across multiple vision tasks and supports analysis using multi-view and multi-modal data.
Takeaways & Limitations
IKEA ASM supports development of holistic methods that integrate semantic and geometric information for furniture assembly understanding.
Takeaways & Limitations
Instance segmentation remains challenged by similar furniture parts and confusion with visually similar environmental surfaces.
Abstract
from arXiv · showhide
The availability of a large labeled dataset is a key requirement for applying deep learning methods to solve various computer vision tasks. In the context of understanding human activities, existing public datasets, while large in size, are often limited to a single RGB camera and provide only per-frame or per-clip action annotations. To enable richer analysis and understanding of human activities, we introduce IKEA ASM -- a three million frame, multi-view, furniture assembly video dataset that includes depth, atomic actions, object segmentation, and human pose. Additionally, we benchmark prominent methods for video action recognition, object segmentation and human pose estimation tasks on this challenging dataset. The dataset enables the development of holistic methods, which integrate multi-modal and multi-view data to better perform on these tasks.
1. Introduction
IKEA ASM addresses the limited datasets available for furniture assembly understanding by providing diverse, fine-grained, multi-view and multi-modal annotations. It supports analysis of actions, objects, pose, geometry and semantics under realistic assembly conditions.
- Existing assembly datasets cover few actions and mainly provide human pose and color information, limiting training for longer-timescale activity understanding.
- IKEA ASM varies assemblers, furniture, colors and body visibility to represent real-life assembly environments.
- The dataset combines color, depth, surface normals, human pose and object instance segmentation across three calibrated camera views.
- Fine-grained labels distinguish visually similar furniture parts and actions such as aligning, spinning in and tightening a leg.
- Symmetric parts, similar objects and visually similar actions require contextual tracking relative to other parts and sub-assemblies.
- The paper benchmarks action recognition, pose estimation, object instance segmentation and tracking using the multi-view, multi-modal dataset.
2. Background and related work
Prior instructional and action datasets often lack task specificity, multi-view coverage or rich modalities. IKEA ASM is positioned as a real-world, domain-specific dataset with fine-grained actions and labels supporting multiple vision tasks.
- Related datasets: Large action datasets such as Kinetics, ActivityNet and Charades are not domain specific or task-oriented and mainly center on single-view RGB data.
- Instructional video datasets: Instructional datasets commonly provide fine-grained action labels with selected modalities, while COIN favors larger scale without additional modalities.
- Multi-view, multi-modal datasets: Drive & Act and NTU RGB+D provide multi-view, multi-modal data, but they target other activity domains and differ in dataset scale or task orientation.
- Related assembly resources: The IKEA Furniture Assembly Environment is a simulated robotic manipulation testbed, whereas IKEA ASM captures real-world human assembly data for domain adaptation.
- IKEA ASM: IKEA ASM includes multi-view and multi-modal data with fine-grained actions, human pose, object instance segmentation and tracking labels.
- Related methods: The paper surveys benchmark methods for action recognition, instance segmentation, multiple object tracking and human pose estimation.
3. The IKEA assembly dataset
The IKEA ASM dataset combines multi-view video with temporal action labels, spatial annotations, and diverse assembly environments. It supports benchmarking action recognition and related perception tasks under challenging conditions.
- Dataset composition: The dataset provides three RGB views, depth, atomic actions, human poses, object segments, and camera calibration.It also includes processing code for depth conversion, surface-normal estimation, visualization, and evaluation.
- Dataset composition: Data collection involved 48 subjects assembling furniture across five environments with varied layouts, lighting, and background activity.Participants assembled furniture in different body configurations, including on the floor or on tables.
- Dataset statistics: The dataset contains 16,764 annotated actions, with each action averaging 150 frames, or approximately six seconds.The action distribution is imbalanced because some assembly operations occur repeatedly within a furniture assembly.
- Dataset split: An environment-based split assigns 254 scans to training and 117 scans to testing, with test environments absent from training.The paper also provides scripts for alternative subject-, environment-, and joint subject-environment splits.
- Spatial annotation: Object instance segmentation uses enclosing polygons on keyframes covering diverse poses, while the remaining frames receive pseudo-ground-truth.Human pose annotations include 12 body joints and five facial key points, plus confidence values for occluded or uncommon poses.
4. Experiments and benchmarks
The paper benchmarks action recognition, instance segmentation, tracking, and human pose estimation on IKEA ASM, including multi-view and multi-modal action-recognition experiments. Results reveal gains from combining views and pose, improved segmentation from pseudo-ground-truth and PointRend, environment-dependent tracking errors, and substantial pose-estimation difficulty.
- Action recognition: P3D outperforms I3D, C3D, and frame-wise ResNet for action recognition under the reported evaluation protocol.The evaluation uses frame-wise accuracy, macro-recall, and mean average precision, with weighted sampling for class imbalance.
- Action recognition: Combining multiple camera views improves action-recognition performance over the best single view, while adding pose provides a further increase.Combining views, depth, and pose instead produces a small disadvantage because the depth-based method performs worse.
- Instance segmentation: Pseudo-ground-truth annotations yield a slight instance-segmentation improvement across backbone architectures.The pseudo-ground-truth combines manual and automatically generated annotations, and any backbone architecture can benefit from it.
- Instance segmentation: PointRend improves instance segmentation by refining boundaries and estimating tighter bounding boxes when training with 1% manually annotated data.The comparison concerns Mask R-CNN with a ResNet-50-FPN backbone.
- Instance segmentation: Mask R-CNN errors arise from visually similar furniture parts and confusing relevant parts with similar environmental surfaces, producing misclassification and false positives.Figure 5 illustrates drawer-panel confusion and working-surface segmentation as a shelf.
- Multiple furniture part tracking: SORT tracking is affected by view-specific occlusions and distractors: occlusions increase false negatives and identity loss, whereas similar irrelevant parts increase false positives.The side view has more occlusions, while the front view preserves identities better but produces more false positives in a cluttered environment.
- Human pose: The fine-tuned Mask R-CNN is the best 2D pose baseline, with 11.5-pixel MPJPE, 64.3% PCK @ 10 pixels, and 87.8 AUC, but lower-body and demographic disparities remain.Upper-body joints are detected more accurately than lower-body joints, and female subjects are detected less accurately than male subjects.
- Human pose: VIBE is the best 3D pose baseline, with 153mm median Procrustes-aligned PJPE and 50% PA-PCK @ 150mm, while baselines perform worse than on standard pose datasets.The results demonstrate the difficulty of pose estimation on IKEA ASM.
5. Conclusion
The paper introduces a comprehensively labeled furniture-assembly dataset for task-oriented activity understanding and benchmarks baseline methods across several perception tasks. Its multi-view and multi-modal design supports research on human-object interactions and longer-timescale structured activities.
- The IKEA ASM dataset provides comprehensive labels for fine-grained actions and common furniture parts in task-oriented human activities.
- It serves as a test bed for action recognition, human pose estimation, object instance segmentation, and tracking.
- Multi-view and multi-modal data supports algorithms that use these modalities for the evaluated tasks.
- Recognizing actions, poses, and object positions can facilitate human-object-interaction understanding and support perceptual understanding of long-timescale structured activities.
6. Appendix
The appendix surveys methods for action recognition, pose estimation, segmentation, and tracking, and documents additional action-localization evaluation details and visualizations. It also lists atomic action classes and presents baseline-comparison tables and figures.
- Action Recognition: Action-recognition methods commonly extend image-based models to handle the temporal dimension, while pose-based approaches use representations such as skeletons.
- Instance Segmentation: Instance-segmentation approaches include proposal-based two-stage systems and region-based methods whose coarse grids can limit masks for large instances.
- Multiple Object Tracking: Multi-object tracking can be online or offline and can use geometry, appearance, or combinations of information depending on the scenario.
- Human Pose Estimation: Human pose-estimation methods are categorized as bottom-up or top-down, with bottom-up methods reaching real-time speeds and top-down methods having better performance.
- Additional Results: The appendix provides the atomic action list, per-class accuracy visualizations, and baseline-comparison tables for action localization.
- Action Localization: Action localization identifies action instances and their start and end frames in untrimmed videos, evaluated using mean average precision over action classes.