PaperlayerView exact revision on arXiv

Source-linked AI summary

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox

arXiv:2104.04631v1cs.CV

TL;DR

DexYCB targets the lack of realistic datasets for joint hand-object perception by capturing markerless human grasping with multi-view RGB-D cameras. It constructs and annotates the dataset, benchmarks three vision tasks, and evaluates safe robot grasps for handover; cross-dataset results show stronger generalization from DexYCB-trained models than from HO-3D-trained models.

  • Problem

    Existing datasets often separate hand and object pose estimation or omit hand-object interactions, while device-based capture can bias natural motion and appearance.

  • Method

    The paper introduces DexYCB using multi-camera markerless recordings, crowdsourced pose annotations, cross-dataset evaluation, three-task benchmarking, and safe-handover grasp generation.

  • Results

    DexYCB-trained models generalize better across datasets than HO-3D-trained models, with ResNet50 root-relative MPJPE increasing 1.4× versus 3.7× in the reported directions.

  • Takeaways & Limitations

    DexYCB provides a basis for joint evaluation of hand-object perception and for studying safe robot grasps in human-to-robot handover.

  • Takeaways & Limitations

    Specialized gloves, magnetic sensors, and marker-based mocap can achieve high accuracy, but attached devices may bias natural motion and alter hand appearance.

Abstract

from arXiv · show

We introduce DexYCB, a new dataset for capturing hand grasping of objects. We first compare DexYCB with a related one through cross-dataset evaluation. We then present a thorough benchmark of state-of-the-art approaches on three relevant tasks: 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation. Finally, we evaluate a new robotics-relevant task: generating safe robot grasps in human-to-robot object handover. Dataset and code are available at https://dex-ycb.github.io.

1. Introduction

DexYCB addresses the lack of datasets capturing interacting hands and objects by introducing a markerless, multi-view dataset and benchmarking tasks spanning perception and safe handover.

  • Motivation: Existing 3D hand and object pose estimation datasets typically address the two problems separately and often omit hand-object interactions.Deep-learning approaches require large labeled datasets, but many existing datasets contain static objects or bare hands rather than interactions.
  • Motivation: Interaction scenarios are harder because object motion and mutual occlusions compound the difficulty of estimating hand and object pose together.Models trained on separate-task datasets therefore do not generalize well to interaction scenarios.
  • Motivation: Device-based capture methods can achieve high accuracy but may bias natural hand motion and alter hand appearance, potentially harming generalization.Prior systems used gloves, magnetic sensors, or marker-based motion capture to obtain accurate hand motion.
  • Motivation: Synthetic data translates effectively to real images for object pose, but realistic hand-object interaction synthesis remains challenging.Realistic grasp poses for generic objects and natural-looking human motions remain difficult to synthesize.
  • Dataset: DexYCB records real hand-object interactions with a larger multi-camera workspace and crowdsourced human pose annotations, producing 582K RGB-D frames over 1,000 sequences.The dataset captures grasping objects from a table.
  • Contributions: DexYCB supports cross-dataset evaluation, joint benchmarking of 2D detection, 6D object pose, and 3D hand pose, plus safe robot grasp generation for handover.The paper presents these as three contributions and evaluates the robotics-relevant handover task.

2. Constructing DexYCB

DexYCB is constructed from synchronized multi-view RGB-D recordings of grasping trials, with human-annotated keypoints and pose recovery through multi-view geometric optimization.

  • Data Collection: The capture system uses 8 synchronized, extrinsically calibrated RGB-D cameras recording a tabletop workspace at 30 fps with minimal blind spots.The setup is designed to provide a sizable capture space for free interaction with multiple objects.
  • Data Collection: Trials record 20 YCB-Video objects manipulated by 10 subjects, with repeated right- and left-hand pickups among varied surrounding-object arrangements.Each trial lasts 3 seconds, and each target object is repeated 5 times with randomized companions and placement.
  • Annotation: Human annotators label and track required hand and object keypoints in every RGB view, marking hand keypoints invisible when occluded.Hands use 21 predefined joints, while objects use two distinctive user-defined landmark points.
  • Pose Representation: Hand pose is represented with MANO, whose deformable 778-vertex mesh is parameterized by pose θ ∈R51 and shape β ∈R10.The differentiable model also outputs the 3D positions of 21 hand joints.
  • Pose Representation: Object pose uses a standard 6D representation as a 3D rotation matrix and translation vector T ∈R3×4.The objects have texture-mapped 3D mesh models from YCB-Video.
  • Pose Optimization: The overall hand-and-object pose P is solved per frame in world coordinates by minimizing an energy using multi-view depth and keypoint annotations.The formulation leverages known camera extrinsics and jointly represents hand and object poses.
  • Pose Optimization: The objective combines depth, keypoint-reprojection, and MANO regularization terms to fit observed data while discouraging irregular hand articulation.Depth fitting uses a GPU implementation because signed-distance computations and gradients are expensive for large point clouds and meshes.
  • Pose Optimization: Object keypoint correspondences are established by back-projecting annotated landmarks onto visible model surfaces and fixing the mappings thereafter.The first-frame pose is initialized manually using PoseCNN predictions, and later frames use the previous solved pose.

3. Related Datasets

Existing hand and object datasets often isolate tasks, omit visual modalities or object poses, or rely on intrusive capture systems. DexYCB addresses these gaps with markerless, multi-view RGB-D videos of dynamic hand-object interactions and human annotations.

  • Prior 6D object pose datasets largely depict static objects, while category-level datasets also use static scenes.
  • Some hand pose datasets use depth only or capture bare hands without object interactions, whereas others focus on hand-hand interactions.
  • Synthetic hand datasets still struggle to generate realistic grasp poses and natural motion, often providing static images rather than videos.
  • DexYCB is markerless, captures dynamic grasp motions, and provides multi-view RGB-D sequences with both hand-object interaction data and object pose.
  • Compared with HO-3D, DexYCB includes 20 versus 10 objects, 8 versus 1 to 5 views, 582K versus 78K frames, and 1,000 versus 27 sequences.
  • DexYCB captures complete grasping processes and tabletop object poses, while HO-3D usually shows an object already held and labels only that object.

4. Cross-Dataset Evaluation

The cross-dataset evaluation tests a hand-pose model trained separately on HO-3D, DexYCB, or their combination, using held-out data from both datasets. Models generalize across datasets asymmetrically: DexYCB-trained models transfer better to HO-3D than the reverse.

  • The evaluation trains Spurr et al.'s hand-pose model on HO-3D, DexYCB, or their combined training sets and tests it across both datasets.
  • When trained on DexYCB and tested on HO-3D, root-relative MPJPE rises 1.4× to 1.9×, compared with a 3.4× to 3.7× rise in the reverse direction.
  • For ResNet50, root-relative MPJPE changes from 18.05 to 31.76 mm when transferring DexYCB to HO-3D, and from 12.97 to 48.30 mm in the reverse direction.
  • Combining training sets reduces HO-3D error from 18.05 to 15.79 mm for ResNet50 but increases DexYCB error from 12.97 to 13.36 mm.
  • The authors conclude that DexYCB complements HO-3D better than HO-3D complements DexYCB.

5. Benchmarking Representative Approaches

DexYCB benchmarks representative methods across 2D detection, 6D object pose, and 3D hand pose under default and generalization setups. Results show interaction, unseen subjects, views, and grasping patterns create task-specific challenges.

  • Benchmark scope: DexYCB benchmarks 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation.The benchmark uses multiple representative approaches and four train/validation/test setups: default, unseen subjects, unseen views, and unseen grasping.
  • 2D Object and Keypoint Detection: Mask R-CNN and SOLOv2 perform similarly in object-detection mAP, with Mask R-CNN leading bbox mAP 75.76 versus 75.13 and SOLOv2 leading segmentation mAP 71.56 versus 69.58 on S0.Mask R-CNN has a slight bounding-box advantage, whereas SOLOv2 directly predicts segmentations.
  • 2D Object and Keypoint Detection: Hand detection is harder than object detection, with Mask R-CNN achieving 71.85 hand AP versus 75.76 object bbox mAP on S0.Performance also drops for unseen subjects, such as Mask R-CNN bbox mAP decreasing from 75.76 on S0 to 72.69 on S1.
  • 6D Object Pose Estimation: 6D pose benchmarking follows the BOP protocol, averaging recall from VSD, MSSD, and MSPD into an aggregate recall score.Representative methods include PoseCNN, DeepIM, DOPE, PoseRBPF, and CosyPose, with RGB and selected RGB-D variants.
  • 6D Object Pose Estimation: PoseCNN object-pose AR drops significantly on grasped objects compared with static objects, motivating evaluation focused on grasped objects.Across setups, AR falls on unseen subjects and unseen grasping, while unseen views slightly increase AR from 41.65 to 45.18 on all objects.
  • 3D Hand Pose Estimation: HRNet32 absolute hand-pose MPJPE rises from 52.26 mm on S0 to 70.10 mm on S1 and 80.63 mm on S2, while depth-based A2J improves absolute position but worsens Procrustes MPJPE.On S0, A2J records 12.07 mm Procrustes MPJPE versus 6.83 mm for HRNet32.

6. Safe Human-to-Robot Object Handover

The handover task generates diverse robot grasps from RGB-D images of people holding objects while avoiding contact with the person’s hand. Evaluation balances successful-reference coverage against grasp precision, using estimated object pose and hand segmentation.

  • Task: The goal is to generate diverse SE(3) robot grasps that take over an object without pinching the person’s hand.The setup assumes a parallel-jaw Franka Panda gripper, and diversity matters because not every grasp is kinematically feasible.
  • Evaluation: The reference set contains 100 farthest-point-sampled grasps per YCB object, transformed using ground-truth object pose and filtered for collisions with object and hand meshes.This produces a dense coverage of the pose space and a set of successful grasps.
  • Evaluation: Coverage is the percentage of successful reference grasps matched by at least one non-colliding predicted grasp, using translation and quaternion-orientation thresholds.The matching thresholds are σt = 0.05 m and σq = 15°.
  • Evaluation: Precision is the percentage of predicted grasps that match at least one successful reference grasp, complementing coverage against exhaustive sampling.The precision-coverage trade-off is evaluated across collision thresholds ϵ ∈[0, 0.07 m] on S1.
  • Results: Better object-pose estimation produces better grasp generation, while most failures arise from inaccurate object pose and some hand collisions from missed partially occluded hands.Qualitative results distinguish successful-covering, collided, and unsuccessful grasps by green, red, and gray colors.

7. Conclusions

DexYCB introduces a dataset for hand grasping of objects, benchmarks three vision tasks, and evaluates safe robot grasp generation for human-to-robot handover. The authors present these results as evidence of the dataset’s merits and anticipated role in future progress.

  • Conclusions: DexYCB captures hand grasping of objects and supports benchmarking current approaches on three relevant tasks plus a robotics-relevant task.The evaluated robotics task is safe robot grasp generation for human-to-robot object handover.

Appendices

The appendices detail differentiable depth-term optimization, mesh-distance computation, qualitative pose results, and fingertip reprojection-error reporting. The depth term supports gradient-based optimization for both rigid objects and deformable hands.

  • A.1. Optimizing Edepth: The depth term Edepth is optimized with an efficient point-parallel GPU implementation, and its gradient is derived for gradient-based pose optimization.The backward pass can also be parallelized on a GPU.
  • A.1. Optimizing Edepth: For a rigid object pose P ∈SE(3), Edepth uses a point cloud and the model mesh M(P), with signed distance computed from each point to the mesh.A BVH-based axis-aligned bounding-box tree accelerates nearest-face distance queries.
  • A.1. Optimizing Edepth: The closest mesh point q is represented with barycentric coordinates q = u · a + v · b + w · c, where a, b, and c are triangle vertices.The barycentric representation enables differentiating the depth term through mesh vertices.
  • A.1. Optimizing Edepth: The resulting gradient applies the multivariable chain rule to mesh vertices and is valid for rigid YCB objects and the deformable MANO hand model.This allows the joint optimization formulation to use both object and hand models.

B.2. Analysis on 3D Annotation Accuracy

DexYCB provides diverse real hand-object interactions and evaluates annotation quality through reprojection error and pose-distribution analysis. Its grasp diversity exceeds prior datasets while background diversity remains limited and controlled.

  • 3D Annotation Accuracy: Mean reprojection errors for all five finger tips are below 5 pixels.Large errors often arise from fast hand motion or ambiguous annotations.
  • Diversity of Hand Pose: DexYCB expands grasp diversity beyond prior datasets, as shown by the distribution of the first two MANO PCA coefficients.The dataset increases both objects and grasps per object, but its main advantage is the diversity of captured grasps.
  • Dataset Diversity: DexYCB focuses on grasp diversity rather than background diversity and uses a controlled environment with limited scene variation.HO-3D also has only two background scenes, so its natural backgrounds provide limited additional diversity.
  • Cross-Dataset Evaluation: DexYCB generalizes better than HO-3D because it offers greater grasp diversity despite similar scene-background diversity.The paper identifies this as explaining DexYCB’s edge in cross-dataset evaluation.

C. Benchmarking Representative Approaches

The benchmark evaluates representative methods across predefined dataset splits and examines qualitative behavior in 2D detection, 6D object pose, 3D hand pose, and grasp generation. Results expose failures from occlusion and pose-estimation errors, while refinement methods and articulated hand-pose estimation show useful strengths.

  • Evaluation Setups: The four evaluation setups are S0 (default), S1 (unseen subjects), S2 (unseen views), and S3 (unseen grasping).Statistics include subjects, objects, views, sequences, images, object annotations, and hand annotations.
  • 2D Detection: 2D detection produces false object detections and inaccurate hand keypoints when occlusions arise from objects or hand interaction.The qualitative failures are shown for Mask R-CNN on S0.
  • 6D Object Pose Estimation: Refinement methods such as DeepIM and CosyPose improve coarse PoseCNN estimates under severe hand occlusion.The qualitative 6D pose results are shown on S1 using rendered object models over darkened input images.
  • 3D Hand Pose Estimation: 6.83mm mean per joint position error after Procrustes alignment is reported for 3D hand pose estimation.The method generates sensible articulated poses under object occlusions, while translation, rotation, and scale contribute more error than articulation.
  • Grasp Generation: Two objects lack feasible pre-generated grasps for the Franka Panda gripper, so grasp visualizations cover 18 of DexYCB’s 20 objects.The evaluation uses 100 sampled grasps per object.
  • Grasp Generation: Grasp-generation failures arise when pose errors cause collisions, missed grasps, or potential pinches around occluded hands.Larger objects can require a nearly fully open gripper, whereas smaller objects may be missed by pose errors.

D.3. Full Quantitative Results

Across S0, S2, and S3, more accurate object pose estimation consistently improves grasp generation. Similar 6D pose recall can nevertheless yield different downstream handover performance, showing that AR alone is insufficient for robotics grasp quality.

  • Full Quantitative Results: More accurate object pose estimation leads to better grasp-generation performance on S0, S2, and S3.The paper reports this same trend as on S1 using precision-coverage curves.
  • Full Quantitative Results: 57.54 versus 57.43 AR shows that DeepIM (RGB-D) and CosyPose have nearly identical 6D object-pose performance, yet CosyPose performs significantly better on grasp generation.DeepIM wins by larger margins on fewer objects, while CosyPose performs better across more objects.
  • Full Quantitative Results: Higher AR on 6D pose metrics does not necessarily translate to higher performance on downstream robotics tasks such as object handover.The comparison indicates that object-level average recall can mask per-object differences relevant to grasp generation.

E. Results on In-the-Wild Images

DexYCB is collected in a controlled setting with a constant background, limiting expected transfer to in-the-wild images. Qualitative COCO examples show false positives, false negatives, and inaccurate keypoints, while the other figures provide qualitative task visualizations.

  • In-the-Wild Generalization: DexYCB-trained RGB models are not expected to generalize well to in-the-wild images because the dataset uses a constant background.Evaluation on COCO images showed an expected performance drop.
  • In-the-Wild Generalization: Figure 10 shows 2D hand and keypoint detection samples on COCO.The accompanying evaluation reports false positives, false negatives, and inaccurate keypoint detections.
  • Qualitative Benchmarks: Figure 11 presents Mask R-CNN qualitative results for 2D object and keypoint detection, including failure examples in the last two rows.The figure complements the in-the-wild evaluation with detection examples.
  • Qualitative Benchmarks: Figure 12 visualizes 6D object pose estimates by rendering object models on darkened input images.Figure 13 shows predicted 3D hand poses from front and side views, while Figures 14 and 15 visualize pre-generated and predicted grasps.
Loading 2104.04631v1…