PaperlayerView exact revision on arXiv

Source-linked AI summary

TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding

Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, Li Yi

arXiv:2401.08399v2cs.CV

TL;DR

Existing hand-object datasets largely focus on single-hand actions or single-object bimanual manipulation, leaving bimanual tool-object cooperation and generalization under-supported. TACO combines a real-world bimanual dataset and automatic acquisition pipeline with benchmarks for three generalization tasks; those benchmarks reveal challenges, including a substantial accuracy drop on the most difficult compositional-recognition split.

  • Problem

    Existing hand-object datasets primarily focus on unimanual actions or bimanual manipulation of a single object, and the closest dataset covers fewer than 20 tool-action-object triplets.

    Source: §1 · p. 2

  • Method

    TACO provides real-world bimanual tool-action-object data with automatic hand-object mesh and segmentation recovery, and benchmarks three tasks for generalizable interaction understanding.

    Source: §1 · p. 2

  • Results

    TACO’s benchmarks reveal generalization challenges, including a significant accuracy decrease on the most difficult S4 compositional-recognition set compared with the non-generalizable S1 set.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    TACO provides a dataset and benchmark for studying generalizable bimanual hand-object interaction across analysis and synthesis tasks.

    Source: §1 · p. 2, §6 · p. 8

    TACO does not currently cover articulated objects.

    Source: §6 · p. 8

Abstract

from arXiv · show

Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities. However, existing technical approaches for analyzing and synthesizing hand-object manipulation are mostly limited to handling a single hand and object due to the lack of data support. To address this, we construct TACO, an extensive bimanual hand-object-interaction dataset spanning a large variety of tool-action-object compositions for daily human activities. TACO contains 2.5K motion sequences paired with third-person and egocentric views, precise hand-object 3D meshes, and action labels. To rapidly expand the data scale, we present a fully automatic data acquisition pipeline combining multi-view sensing with an optical motion capture system. With the vast research fields provided by TACO, we benchmark three generalizable hand-object-interaction tasks: compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis. Extensive experiments reveal new insights, challenges, and opportunities for advancing the studies of generalizable hand-object motion analysis and synthesis. Our data and code are available at https://taco2024.github.io.

1. Introduction

Existing hand-object datasets largely focus on unimanual actions or bimanual manipulation of a single object, leaving tool-action-object coordination and generalization under-supported. TACO contributes a large-scale bimanual dataset, an automatic acquisition pipeline, and benchmarks for generalizable hand-object understanding.

1. Introduction

  • Bimanual behaviors coordinate two hands, a tool, and a target object in potentially asymmetric roles, posing challenges for current vision systems.The paper connects this understanding to VR/AR, human-robot interaction, and dexterous manipulation.Source: §1 · p. 2
  • Existing datasets focus mainly on unimanual actions or single-object bimanual manipulation, while the closest dataset covers only 12 objects and fewer than 20 tool-action-object triplets.The paper identifies this coverage gap as a barrier to benchmarking generalization to novel geometries and unseen triplets.Source: §1 · p. 2
  • TACO covers 131 tool-action-object triplets across 20 object categories, 196 object instances, and 15 daily actions.The dataset is organized by triplets with varying overlap to support study of different extents of generalization.Source: §1 · p. 2
  • The automatic acquisition pipeline recovers hand-object meshes and segmentation alongside markerless vision data, and TACO contains 2.5K motion sequences and 5.2M video frames.The frames include third-person and egocentric views.Source: §1 · p. 2
  • TACO benchmarks compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis for unseen geometries and novel tool-action-object combinations.The benchmarks examine both hand-object motion analysis and synthesis.Source: §1 · p. 2

2. Related Work

Prior hand-object datasets span 2D, 3D, and 4D capture, but many emphasize single-hand grasping, and the available multi-object dataset lacks object diversity. Related work also addresses generalization in motion analysis and synthesis, while existing synthesis approaches do not focus on multi-hand, multi-object cooperation.

2.1. Hand-object Interaction Datasets

  • Most existing datasets emphasize single-hand grasping; KIT captures multi-object interactions but lacks object diversity for generalizable studies.Other datasets cover tool use, soft-body manipulation, and multi-person handovers.Source: §2.1 · pp. 2–3

2.2. Generalizable Hand-object Motion Analysis

  • Generalizable motion-analysis methods use object boxes, separate action and object features, or transfer category-level interaction priors to unseen instances.These approaches target recognition or detection of actions and action-object pairs, or estimation of hand and object poses.Source: §2.2 · p. 3

2.3. Generalizable Hand-object Motion Synthesis

  • Motion synthesis research models hand-object relations through contact and spatial features or CVAEs, and conditions generative models on object geometry, but existing approaches do not focus on multi-hand, multi-object cooperation.CAMS maps objects into a canonicalized object space to reduce the geometry gap.Source: §2.3 · p. 3

3. Constructing TACO

TACO combines multi-view RGB capture with optical motion capture and scanned object meshes, then automatically estimates object poses, hand meshes, masks, and marker-removed images. Its dataset spans daily tool-use behaviors, multiple object categories and instances, and 131 tool-action-object triplets.

3.1. Data Capturing

  • TACO captures hand motion with 12 synchronized industrial cameras, tracks object markers with an optical motion-capture suite, and records egocentric RGBD video with a helmet-mounted camera.The system operates at 30 Hz and uses scanned object meshes with up to 100K triangular faces.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Data Annotating

  • The automatic annotation pipeline combines color images, marker positions, and object models to perform object-pose optimization, hand-keypoint localization, hand-pose optimization, segmentation, and marker removal.Figure 3 depicts the pipeline inputs and stages.Source: §3.2 · p. 4, Figure 3 · p. 5
  • Object poses are recovered from tracked markers associated with object surfaces, while multi-view hand keypoints are triangulated after RANSAC filters imprecise 2D detections.The hand-keypoint process detects the two hands separately to address occlusion and separation challenges.Source: §3.2 · p. 4
  • MANO hand meshes are fitted to precomputed 3D keypoints and object meshes using losses for keypoint alignment, pose priors, temporal smoothness, contact, and interpenetration.The attraction and penetration terms balance encouraging contact against preventing hand-object penetration.Source: §3.2 · p. 4
  • Hand-object masks are rendered from meshes and refined with SAM; tracked markers are covered by rendered masks and inpainted with a pre-trained LAMA network.Figure 4 shows a captured patch, its marker mask, and the inpainted patch.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3. Dataset Statistics

  • TACO contains 2.5K sequences across 20 object categories, 196 fine-grained object models, 14 participants, 15 actions, and 131 triplets, with 12 allocentric views and one egocentric RGBD view.Tools and target objects can each be used for up to seven kinds of actions.Source: §3.3 · pp. 4–5

4. Data Quality Evaluation

The data-quality evaluations show that combining contact attraction with penetration avoidance balances the competing goals of hand–object contact and non-penetration. Cross-dataset hand-pose results support TACO as a complement to DexYCB, while marker removal makes image regions harder to segment as markers.

Qualitative contact optimization evaluation

  • Combining attraction and penetration losses encourages hand–object contact while mitigating penetration.Attraction alone may worsen penetration; penetration loss alone does not resolve missing contact.Source: §4 · p. 5, Figure 6 · p. 5

Quantitative hand pose evaluation

  • Joint training on TACO and DexYCB improves both methods on both datasets, despite performance dropping when training and test datasets differ.The evaluation uses PA-MPJPE and MPJPE; Table 2 reports results in millimeters, with lower values better.Source: §4 · p. 6, Table 2 · p. 6

Marker removal quality evaluation

  • Marker segmentation falls from 63.8% mIoU on raw images to 11.1% mIoU after marker removal.The result indicates that the inpainted regions appear more natural and ambiguous to the segmentation network.Source: §4 · p. 6, Figure 7 · p. 6

5. Experiments

Across the benchmarks, generalization is challenging: novel interaction triplets hurt action recognition, and novel geometries or triplets challenge forecasting and grasp synthesis. The experiments use four test splits to distinguish these kinds of generalization.

5. Experiments

  • The experiments benchmark compositional action recognition, hand-object motion forecasting, and cooperative grasp synthesis on TACO.The section introduces the data split and points to supplementary materials for metric details.Source: §5 · p. 6

5.1. Data Split

  • The four test sets separate no generalization, novel tool geometry, novel interaction triplets, and compound generalization to a novel tool category.The compound split includes unseen tool geometries and triplets.Source: §5.1 · p. 6

5.2. Compositional Action Recognition

  • The action-recognition task identifies action labels in first-person RGB video, optionally using hand and object bounding boxes.Evaluation uses Top-1 and Top-5 accuracy.Source: §5.2 · pp. 6–7
  • Novel interaction triplets cause a significant action-recognition accuracy drop, while novel tool geometries within known categories perform similarly to the non-generalization set.CACNF outperforms AIM on the challenging S3 and S4 sets.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Generalizable Hand-object Motion Forecasting

  • Forecasting errors increase for the right hand and tool under geometry or interaction-triplet generalization, while results for other entities remain close to the non-generalization set.Generative models yield larger errors than predictive models for both hands and the tool.Source: §5.3 · p. 7, Table 4 · p. 8

5.4. Cooperative Grasp Synthesis

  • Novel object geometries and categories raise collision ratios and FID scores across grasp-synthesis approaches, with most also declining in contact precision and penetration prevention.HALO-VAE− performs worse than HALO-VAE on contact and collision ratios when interaction-environment inputs are omitted.Source: §5.4 · p. 8, Table 5 · p. 8

6. Limitations and Conclusion

TACO provides a large real-world bimanual manipulation dataset and benchmarks three generalizable hand-object-interaction tasks. Its scope is bounded by gaps in articulated-object and scene diversity, and imperfect recovery of original object appearances after marker removal.

6. Limitations and Conclusion

  • TACO contains 2.5K motion sequences and 5.2M frames from 12 third-person views and one egocentric view, supporting three generalizable hand-object-interaction benchmarks.The benchmarks cover compositional action recognition, hand-object motion forecasting, and cooperative grasp synthesis.Source: §6 · p. 8
  • The dataset does not cover articulated objects, lacks scene diversity, and cannot perfectly recover original object appearances after generative marker removal.The authors identify these as three major limitations of TACO.Source: §6 · p. 8

7. Interaction Field Estimation

The section benchmarks estimating six hand-, tool-, and target-object interaction fields from RGB images. Its baseline results show that generalization varies across tool/action test sets, with unseen geometries and actions posing challenges.

7. Interaction Field Estimation

  • The task estimates six directed interaction fields among the left hand, right hand, tool, and target object from an RGB frame.Each field gives the distance from every vertex in one mesh to its nearest vertex in the other mesh.Source: §7 · p. 13
  • The evaluation uses Mean Distance Error for field precision and Acceleration Error for estimate smoothness.The two baselines are based on InterField-SF, using separated or concatenated inputs to predict fields.Source: §7 · p. 13
  • For RT and TR interaction fields, performance is significantly better on S1 than S3, while S3 outperforms S2 and S4, indicating challenges with unseen geometries and actions.The test-set differences primarily concern tools and actions, with weaker correlation to target objects; S3 combines seen geometries with unseen actions.Source: §7 · p. 13

8. Data Capturing Details

The data-capturing details describe camera calibration and synchronization across the recording setup. Intrinsics are estimated with a checkerboard, while extrinsics use scene markers and synchronized timestamps align sensor streams.

8.1. Camera Calibration

  • Intrinsic calibration estimates the camera matrix and distortion from a checkerboard with known grid scales; extrinsic calibration uses 12 scene markers and minimizes their reprojection error.Mocap provides marker positions with errors below 1 mm, while pixel coordinates are manually annotated; Figure 9 depicts the extrinsic calibration setup.Source: §8.1 · p. 13, Figure 9 · p. 13

8.2. Time Synchronization

  • Timestamps are matched by nearest neighbor to synchronize the 12 industrial cameras with mocap and the Realsense L515, with a maximum matched-frame difference of 17 ms.The industrial cameras share signals from the same generator; UTC timestamps are used across modalities.Source: §8.2 · p. 13

9. Data Annotating Details

The annotation details optimize object pose from surface markers, fuse multi-view hand keypoints, and fit MANO hand meshes with geometric and temporal constraints. Object fitting uses contact and penetration losses, while hand fitting adds contact and interpenetration terms after keypoint alignment.

9.1. Details on Object Pose Optimization

  • Object pose optimization fits a relative 6D transform from the object model to a rigid body of attached markers, combining contact and penetration losses.The procedure uses α=1 cm and K=14 for optimization, though only four markers track the object during capture; Adam uses learning rate 1e-4.Source: §9.1 · pp. 13–14

9.2. Details on 3D Hand Keypoint Localization

  • Hand keypoints are estimated per view, triangulated across camera pairs, and filtered by RANSAC, which accepts 2D points within 30 pixels of the projected 3D estimate.The process first obtains per-hand 2D keypoints and then selects the triangulated 3D point with the most nearby 2D keypoints.Source: §9.2 · p. 14

9.3. Details on Hand Pose Optimization

  • The initial MANO fit aligns hand joints to 2D and 3D keypoints while constraining joint angles and temporal changes to promote natural, smooth poses.Per-participant shape parameters are precomputed and held fixed during subsequent pose optimization.Source: §9.3 · pp. 14–15
  • A subsequent hand-pose refinement adds attraction and penetration losses to encourage hand-object contact and prevent interpenetration.The attraction term is computed for the right hand with the tool and the left hand with the object; the penetration term uses the same pairings.Source: §9.3 · pp. 14–15

10. Detailed Statistics on TACO

TACO combines varied object geometries and tool-action-object interactions with motion statistics and a hand-pose comparison against HO3D.

Object diversities

  • The dataset includes 20 object categories—17 used as tools and 9 as targets—and 12 brush instances illustrating geometry diversity.The categories are drawn from everyday interaction scenarios, and each object is sized for stable single-hand manipulation.Source: §10 · p. 15, Figure 10 · p. 16, Figure 11 · p. 16

Interaction diversities

Motion speed

  • Tools average 16.6 cm/s and 71.3 °/s, and the right hand moves faster than the left across action types.The authors attribute this pattern to right-handed participants commonly controlling the tool.Source: §10 · p. 16

Hand pose distribution

  • TACO hand-pose distributions mostly differ from HO3D, which the authors attribute to different human behaviors.Figure 14 visualizes the distributions with t-SNE.Source: §10 · p. 16, Figure 14 · p. 16

11. Details on Marker Removal Evaluation

This section details marker-removal data processing and training, evaluation metrics, interactive-grasp baseline modifications, qualitative forecasting results, and dataset visualizations.

11. Details on Marker Removal Evaluation

  • For marker removal, image patches are centered in 512x512 inputs, then a U-Net estimates inpainted-region heatmaps using mean-square error and Adam at 5e-4.The patches are blurred with a Gaussian kernel of σ=1.0 before input.Source: §11 · pp. 16–17

12. Details on Evaluation Metrics

  • Cooperative grasp evaluation measures contact, interpenetration, environmental collision, and realism via FID on interaction-feature distributions.The extractor encodes hand-object vertices into 64-dimensional features and is supervised by action-label decoding.Source: §12 · p. 17, Figure 15 · p. 18

13. Baseline Designs for Interactive Grasp Synthesis

  • The grasp baselines incorporate the left hand and target object as interaction-environment conditions for a CVAE, encoded from their point clouds.Figure 16 compares modified HALO-VAE with HALO-VAE−.Source: §13 · p. 18, Figure 16 · p. 18

14. Qualitative Results on Hand-object Motion Forecasting

  • CAHMP performs best among four forecasting baselines but commonly misses fast right-hand and tool movements and struggles to infer interaction intentions.Figure 17 shows predicted and ground-truth motions at five frames from 0.12 to 0.60 seconds.Source: §14 · p. 18, Figure 17 · p. 18

15. TACO Visualization

  • The visualizations show 12 third-person RGB views, egocentric RGB and depth, hand-object meshes, segmentation, and marker-removed image patches.The section points to a supplementary video for additional visualizations.Source: §15 · p. 18, Figure 18 · p. 18, Figure 19 · p. 19

Related papers

Loading 2401.08399v2…