Source-linked AI summary
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, Otmar Hilliges
TL;DR
ARCTIC addresses the shortage of ground-truth 3D data for physically consistent manipulation of articulated objects. It provides synchronized multi-view recordings with accurate meshes, defines reconstruction and interaction-field tasks, and supplies ArcticNet and InterField baselines evaluated on the dataset.
Problem
Existing datasets provide limited evidence for physically consistent hand-object motion because they mostly contain grasping of rigid objects and few dexterous articulated interactions.
Method
The paper constructs ARCTIC with synchronized RGB and motion-capture data, accurate hand and object meshes, and baselines for motion reconstruction and interaction-field estimation.
Results
ARCTIC provides 2.1M RGB images and evaluates ArcticNet for consistent motion reconstruction and InterField for dense relative hand-object distance estimation.
Takeaways & Limitations
ARCTIC establishes a dataset and benchmark framework for studying spatio-temporally consistent bimanual manipulation of articulated objects.
Takeaways & Limitations
The ArcticNet baselines assume that object models are known, leaving articulated 3D shape estimation for unknown objects to future work.
Abstract
from arXiv · showhide
Humans intuitively understand that inanimate objects do not move by themselves, but that state changes are typically caused by human manipulation (e.g., the opening of a book). This is not yet the case for machines. In part this is because there exist no datasets with ground-truth 3D annotations for the study of physically consistent and synchronised motion of hands and articulated objects. To this end, we introduce ARCTIC -- a dataset of two hands that dexterously manipulate objects, containing 2.1M video frames paired with accurate 3D hand and object meshes and detailed, dynamic contact information. It contains bi-manual articulation of objects such as scissors or laptops, where hand poses and object states evolve jointly in time. We propose two novel articulated hand-object interaction tasks: (1) Consistent motion reconstruction: Given a monocular video, the goal is to reconstruct two hands and articulated objects in 3D, so that their motions are spatio-temporally consistent. (2) Interaction field estimation: Dense relative hand-object distances must be estimated from images. We introduce two baselines ArcticNet and InterField, respectively and evaluate them qualitatively and quantitatively on ARCTIC. Our code and data are available at https://arctic.is.tue.mpg.de.
1. Introduction
ARCTIC addresses the lack of datasets for physically consistent dexterous manipulation of articulated objects by pairing synchronized multi-view video with accurate 3D hand and object meshes. It introduces two tasks and corresponding baselines for reconstructing motion and estimating dense hand-object spatial relations.
- Existing hand-object datasets mostly cover grasping rigid objects, with few examples of dexterous manipulation involving articulated objects.
- ARCTIC contains multi-view RGB frames paired with accurate 3D hand and object meshes from subjects manipulating articulated objects.Capture uses synchronized color cameras and high-resolution motion capture to fit pre-scanned human and object meshes.
- The consistent motion reconstruction task requires reconstructing two hands and an articulated object with spatio-temporally consistent contact, articulation, and motion from monocular video.The task emphasizes precise hand-object alignment across frames despite depth-related challenges.
- ArcticNet provides single-frame and recurrent baselines for reconstructing two hands and an articulated object from video.Both variants estimate MANO hand parameters and parameters of the articulated object model.
- Interaction field estimation predicts shortest distances between each hand vertex and the object mesh, including configurations where hands are nearby but not contacting.InterField is benchmarked in single-frame and recurrent versions on ARCTIC.
- The paper contributes ARCTIC, two articulated hand-object interaction tasks, and baselines for both tasks.
2. Related Work
Prior human-object datasets and contact-estimation methods cover several interaction settings, but articulated-object manipulation remains insufficiently addressed. ARCTIC is positioned against these gaps through its focus on dexterous interactions and richer contact dynamics.
- Existing human-object datasets include body interaction with rigid objects, but differ from ARCTIC in hand capture, 3D accuracy, or articulated manipulation coverage.
- Related motion methods model hand-object interaction or motion priors, but none of the cited methods handle articulated objects.
- Contact-estimation methods span mesh-based, image-space, and binary contact approaches, while ARCTIC targets richer hand-object interaction fields.
- ARCTIC includes fast-changing hand-object contact during dexterous manipulation.
3. ARCTIC Dataset
ARCTIC is a multi-view motion dataset of bimanual dexterous manipulation with articulated objects, designed to provide diverse poses, contacts, and accurate 3D annotations. Its capture setup combines synchronized RGB views with motion capture to address fast motion and occlusion.
- Dataset overview: ARCTIC contains 339 sequences involving 10 subjects and 11 articulated objects, totaling 2.1M RGB images from 8 static and 1 egocentric view.Sequences include both object use and grasp interactions, paired with 3D hand and object meshes.
- Dataset features: ARCTIC uniquely combines two hands, the full human body, articulated objects, calibrated multi-view cameras, and bimanual dexterous manipulation.Compared with existing datasets, it supports monocular, multi-view, and egocentric reconstruction settings.
- Hand pose and contact variations: ARCTIC exhibits significantly greater hand-pose diversity than compared datasets, attributed to unconstrained dynamic manipulation.
- Hand pose and contact variations: Its contact heatmaps show broader hand-region spread and higher palm contact likelihood than other datasets.Object contact regions vary with object size and interaction mode, such as one-handed support of smaller objects.
- Capture setup: The capture system synchronizes motion capture with 8 static allocentric and 1 moving egocentric RGB view at 30 FPS.Small 1.5mm-radius dorsal hand markers balance annotation accuracy with participant freedom during interaction.
- Object representation: Each articulated object is represented by a 6D base pose and a 1D articulation relative to a canonical pose.
4. Evaluation Protocol
ARCTIC evaluation separates subject-based allocentric and egocentric protocols and measures reconstruction using pose, articulation, object, relative-root, contact, motion, and smoothness metrics. ArcticNet-SF maps image features to two hand and articulated-object meshes, while interaction-field evaluation measures per-vertex distance accuracy and temporal smoothness.
- Data split: Eight subjects train, one male validates, and one female tests, with allocentric and egocentric protocols for third- and first-person evaluation.The split is subject-disjoint and balances gender in evaluation.
- Contact and motion metrics: Stable-contact tuples are evaluated over windows where hand-object vertices remain within α throughout the interval and are absent from contact immediately outside it.This identifies longest stable-contact windows before computing motion deviation.
- Additional metrics: The protocol also reports acceleration error for reconstruction smoothness, relative-root position error for hand-hand and hand-object translation, and average distance error for interaction fields.Interaction-field error averages absolute per-vertex distance differences in millimeters, with acceleration error assessing temporal smoothness.
- Pose and object metrics: MPJPE measures root-relative 21-joint hand error, AAE measures articulation error, and Success Rate measures object vertices within 5% of object diameter.Success Rate is object-size independent and uses root-subtracted vertices.
- Baseline architecture: ArcticNet-SF encodes an image into features, decodes left and right MANO parameters and translations, and decodes articulated-object pose into parametric 3D meshes.The object pose includes articulation, rotation, and translation.
5. Baselines and Experiments
ARCTIC benchmarks consistent motion reconstruction and interaction field estimation, introducing ArcticNet and InterField baselines with single-frame and temporal variants. Temporal modeling improves consistency, smoothness, and distance-related predictions.
- Consistent motion reconstruction: ARCTIC evaluates 3D reconstruction of two hands and an articulated object from video, requiring temporally consistent contact and motion.
- Consistent motion reconstruction: ArcticNet benchmarks single-frame and recurrent LSTM models for estimating MANO hand parameters and articulated-object motion.
- Consistent motion reconstruction: Temporal ArcticNet-LSTM produces more consistent contact and motion, with smoother acceleration, than the single-frame baseline.
- Interaction field estimation: Interaction field estimation predicts each hand vertex’s closest distance to the object and each object vertex’s closest distance to either hand.
- Interaction field estimation: InterField concatenates image features with canonical subsampled vertices, uses PointNet and regression heads, and upsamples predictions to the full mesh.
- Interaction field estimation: The temporal InterField-LSTM aggregates image features over a window to jointly reason about hand-object motion, yielding distance and acceleration predictions.
6. Conclusions
ARCTIC provides synchronized video and high-quality 3D ground truth for two-handed manipulation of articulated objects. It supports consistent motion reconstruction and interaction field estimation through ArcticNet and InterField baselines.
- ARCTIC is a dataset of two hands dexterously manipulating articulated objects with synchronized video and high-quality 3D ground-truth meshes.
- The dataset contains 2.1M RGB images from 8 static and 1 egocentric view, covering 10 subjects and 11 articulated objects.
- The paper presents consistent motion reconstruction and interaction field estimation, with ArcticNet and InterField baselines evaluated on ARCTIC.
1. Dataset Details
ARCTIC contains large-scale multi-view recordings of 10 subjects manipulating 11 articulated objects. The dataset includes 2.1M images organized into long sequences and distinct use and grasp interaction modes.
- ARCTIC includes 2.1M images across 10 subjects and 11 articulated objects, with more than 170k images for every object.
- The dataset’s average view-agnostic sequence contains 698 frames, corresponding to 23.3 seconds.
- Subjects perform “use” sequences that allow articulation and “grasp” sequences that prohibit articulation, with more data collected for “use”.
2. Data Capture Details
ARCTIC combines synchronized RGB and motion-capture recording with scanned subject and object meshes to recover articulated-object motion, hand-object contact, and pose annotations. Its capture and processing pipeline estimates object axes and articulation angles before computing mesh-level contact labels.
- Capture setup: The capture system uses 54 high-resolution MoCap cameras to reduce hand self-, hand-hand, and hand-object occlusion.
- Capture setup: Small markers track the hands, full body, articulated object, egocentric camera, and props, while larger markers are used for the body.
- Mesh preparation: Subject-specific SMPL-X meshes are obtained from 3D scans and registered into aligned canonical templates.
- Object articulation: Each articulated object has two rigid parts rotating about a shared axis, which is estimated from marker trajectories by fitting circles and a 3D line.
- Object articulation: The refined axis and framewise articulation angles are estimated relative to an object-specific rest pose, producing motion consistent with marker trajectories.
- Contact computation: Ground-truth binary contact labels are computed at the vertex level across the two hands and both rigid object parts, including proximity and interpenetration cases.
3. Model Details and Results
The paper details ArcticNet and InterField architectures for reconstructing articulated hand-object motion and estimating interaction fields. Temporal modeling improves reconstruction quality and contact, while interaction-field predictions correlate with ground truth.
- ArcticNet: ArcticNet-SF uses an encoder-decoder with a 2048-dimensional pooled image feature and iterative refinement for hand and object parameters.It predicts MANO hand parameters and articulated object parameters from an input image.
- ArcticNet: ArcticNet-LSTM passes image features from a moving window through an LSTM before estimating 3D meshes for each frame.The model otherwise uses the same structure as ArcticNet-SF.
- Training objectives: Training supervises hand and object 3D and 2D predictions, pose or articulation parameters, camera parameters, and temporal or contact-related losses.The contact deviation loss is applied separately to left-hand/object and right-hand/object interactions.
- Results: The ArcticNet-LSTM model has lower overall errors and better contact than ArcticNet-SF in qualitative comparisons.The authors hypothesize that recurrent processing helps jointly reason about hand and object motions.
- InterField: InterField predicts four directed interaction fields between the left hand, right hand, and object using image features and subsampled vertices.Training uses an L1 loss over the four hand-object and object-hand fields, with distances thresholded at 10cm.
- Results: InterField predictions correlate well with ground truth in qualitative visualization, supporting the feasibility of interaction-field estimation.Brighter mesh colors represent closer predicted or ground-truth distances.
4. Metrics and Experiments
The experiments define metrics for reconstruction accuracy, smoothness, articulation, relative positioning, and contact, then analyze model behavior across objects and settings. Results highlight object-dependent difficulty and gains from articulated interaction data.
- Metrics: Acceleration error measures motion smoothness in m/s2 using centered differences with the 30-FPS stencil width w=1/30s.The prediction dimension is 3 for reconstruction and 1 for interaction fields.
- Metrics: MRRPE measures relative root translation error between two entities by comparing predicted and ground-truth root differences.Entities may be the left hand, right hand, or object.
- Per-object analysis: The microwave is hardest for hand reconstruction and contact because fingers are often heavily occluded when opening its door.Scissors has the lowest hand reconstruction error, partly because it is small.
- Per-object analysis: The box has the smallest articulation-angle error, whereas the espresso machine has the highest because its small handle is heavily occluded.The box’s large size makes its articulation angle easier to observe, according to the authors’ hypothesis.
- Per-object analysis: Scissors is hardest for object-pose reconstruction because it is small and its dark texture can resemble the background in some views.The analysis identifies this difficulty using the success rate metric.
- Transfer analysis: Pretraining on ARCTIC improves HO3D hand MPJPE by 9.2% and object vertex-to-vertex error by 7.1% relative to training only on HO3D.The comparison follows the HO3D protocol and uses scale-translation-aligned hand error and root-aligned object error.
5. Visualizing ARCTIC
Figure 10 visualizes ARCTIC’s 3D hand and object annotations by overlaying randomly sampled meshes on dataset images.
- 3D annotation visualization: Figure 10 overlays randomly sampled 3D hand and object meshes on images from ARCTIC.The project page also provides rendered manipulation sequences.
6. Discussions and Limitations
ARCTIC’s current scope includes known object models, toy objects, 1-DoF articulation, and marker-based capture limitations. The authors identify expanding object complexity and addressing representation limits as future directions.
- Known object models are assumed by the ArcticNet baselines, leaving articulated shape estimation for unknown objects as future work.
- Some ARCTIC objects are toys that are not to scale and lack the visual complexity of real objects.
- The human representation does not capture skin deformation during hand-object contact because it uses SMPL-X/MANO geometry.
- Optical hand-marker capture provides accurate poses but may introduce label noise, despite minimally intrusive markers that are barely visible after resizing.
- ARCTIC restricts objects to 1 DoF, and future work should expand object number and complexity to study depth ambiguity and occlusion further.