PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAB: A Dataset of Whole-Body Human Grasping of Objects

Omid Taheri, Nima Ghorbani, Michael J. Black, Dimitrios Tzionas

arXiv:2008.11200v1cs.CV

TL;DR

Existing grasping data are limited for modeling complex, occluded interactions involving the whole body, detailed contact, and varied 3D objects. GRAB captures these interactions as full 3D sequences and demonstrates their use by training GrabNet to generate hand grasps for unseen objects. The dataset provides whole-body motion, object motion, contact, and articulated body detail for research and machine learning.

  • Problem

    Understanding and synthesizing human grasping requires data covering detailed contact, hand pose and shape, complex objects, and full-body motion, but capturing such interactions is difficult because of occlusion and instrumentation trade-offs.

  • Method

    GRAB adapts MoSh++ and SMPL-X to fit MoCap markers into moving 3D meshes of the body, articulated face and hands, and objects, then infers body-object contact.

  • Results

    GRAB contains whole-body grasps from 10 subjects interacting with 51 objects, and supports GrabNet, which generates natural-looking 3D hand grasps for unseen object shapes.

  • Takeaways & Limitations

    GRAB enables richer modeling of human-object interaction than hand-only data because it includes whole-body motion, while also supporting analysis and grasp generation.

  • Takeaways & Limitations

    GRAB prioritizes accurate MoCap over synchronized image data, reflecting a trade-off between accurate motion without RGB images and less accurate motion with RGB images.

Abstract

from arXiv · show

Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at https://grab.is.tue.mpg.de.

1 Introduction

GRAB addresses the lack of rich data for whole-body human-object interaction by capturing detailed 3D body, hand, face, object, and contact information over time. It also demonstrates the dataset’s value for generating plausible hand grasps on unseen 3D objects.

  • Motivation: Images obscure contact regions, while instrumentation can impair natural interaction and fails to capture full-body motion.These limitations have left complex human-object interaction datasets unavailable.
  • Motivation: Whole-body grasping involves gross body motion, dexterous fingers, and contact between objects and multiple body parts.Examples include passing, eating, using binoculars or a hammer, and drinking from a bowl.
  • Dataset: GRAB adapts MoSh++ and SMPL-X to recover full-body, articulated hands and face, object motion, and detailed 3D meshes from MoCap markers.A 3D scan is captured for each subject before fitting the body model.
  • Dataset: The dataset captures 10 people interacting with 51 everyday objects, including dynamic in-hand manipulation and re-grasping.Body-object contact is inferred from the reconstructed meshes.
  • Application: GrabNet uses GRAB to generate natural-looking 3D hand grasps for unseen object shapes with a coarse-to-refined conditional neural network.The model predicts wrist pose and full finger articulation using a BPS object representation.
  • Contributions: The work contributes a whole-body grasp dataset, mesh-based contact analysis, and a conditional network for generating grasps on unseen 3D objects.The dataset, models, and code are released for research purposes.

2 Related Work

Prior research studies hand grasps, whole-body interaction, motion capture, contact, and 3D interaction models, but each addresses only parts of the problem. GRAB builds on these directions while introducing real whole-body grasps with detailed human and object modeling.

  • Hand Grasps: Most grasping studies focus on single-hand prehensile interaction and analyze hand pose, contact areas, finger involvement, or grasp purpose.These approaches use methods including ink, data gloves, and tactile gloves.
  • Whole-Body Grasps: Whole-body grasping has received limited attention, with prior datasets using unrealistic humanoids, simple objects, or synthetic objects.Existing work includes MoCap-based body pose taxonomies and teleoperated simulated-robot demonstrations.
  • Capturing Interactions with MoCap: MoCap research captures body skeletons or hand interactions with scenes and objects, but these representations do not provide GRAB’s full articulated body detail.Prior examples include interactions with large geometric objects and phones.
  • Capturing Contact: Contact capture is difficult because humans and objects occlude one another, while tactile or pressure instrumentation can bias natural grasps.Prior methods place force transducers or use data and tactile gloves.
  • 3D Interaction Models: Hand-crafted interaction models approximate contact, with accuracy improving when humans are represented by detailed 3D meshes rather than skeletons or primitives.Mesh-based approaches may use generic, personalized, or statistical human models.
  • 3D Interaction Models: Synthetic grasp datasets manually articulate hand models or optimize generated grasps with robotics metrics, limiting their correspondence to natural human interaction.Other work captures body skeletons or tracked objects for synthesis and reconstruction.
  • GRAB: GRAB goes beyond prior partial solutions by introducing a real whole-body grasp dataset with detailed body, hand, and object interaction data.The dataset is designed to support modeling and understanding human-object interaction.

3 Dataset

GRAB addresses the challenge of capturing whole-body human-object interactions by combining motion capture, detailed body and object models, and contact annotation. The resulting dataset represents dynamic interactions across varied objects and intents, revealing contact patterns beyond the hands.

  • 3.1 Motion Capture (MoCap): GRAB combines multi-camera MoCap with 3D human and object reconstruction to capture interactions across body, fingers, face, and objects.The setup prioritizes accurate motion and temporal contact detail, using 54 infrared cameras at 120 fps and minimally intrusive markers.
  • 3.2 Reconstruction: The dataset uses personalized SMPL-X models with articulated body, face, and hands, fitted to cleaned marker observations while object pose is solved from detected markers.SMPL-X represents body shape, pose, facial expression, and translation; rigid object pose is estimated from three or more markers.
  • 3.3 Contact Annotation: Contact is inferred from human-object mesh proximity using a tolerance threshold, with intersection-ring heuristics addressing penetration and undershooting errors.The procedure accommodates measurement and fitting errors, limited mesh resolution, soft-tissue deformation, and thin-object intersections.
  • 3.5 Analysis: 92% of “use” contact frames involve the right hand, 39% the left hand, 31% both hands, and 8% the head, while fingers contact more often than palms.The per-finger contact likelihoods from thumb to pinky are 100%, 96%, 92%, 79%, and 39%; palm contact is 24%.
  • 3.5 Analysis: Interaction intent changes contact locations: “use” follows object functionality, whereas “pass” favors one side while leaving the other available to the receiver.For example, subjects avoid a knife blade or pan hot area but contact a flashlight button during “use” sequences.

4 GrabNet: Learning to Grab an Object

GrabNet generates plausible MANO hand grasps for unseen 3D objects using coarse prediction followed by neural refinement. Its mesh-based predictions also provide anthropomorphically valid hand-object contact estimates, evaluated quantitatively and through human ratings.

  • Architecture: GrabNet trains separate CoarseNet and RefineNet modules to generate an initial grasp and improve its contacts with the object.CoarseNet predicts a plausible grasp, while RefineNet uses hand-object distances and learned contact likelihoods for iterative refinement.
  • Inference: Given an unseen 3D object, GrabNet first estimates a grasp with CoarseNet and then produces the final grasp through RefineNet.The two networks are trained separately, and the final output contains full MANO hand pose and finger articulation.
  • Evaluation: 12.1 mm, 14.1 mm and 18.4 mm are CoarseNet’s reconstruction vertex-to-vertex errors on the training, validation and test sets, respectively.The supplied passage reports these values for CoarseNet’s three dataset splits.
  • Evaluation: A user study evaluates 20 generated and 20 ground-truth grasps per object across 6 test objects using a 5-level Likert scale.Participants rated agreement with whether humans could grasp each object as shown.
  • Contact evaluation: GrabNet contact is a mesh-derived by-product of MANO inference and is anthropomorphically valid by construction, unlike the compared bottom-up contact approach.The comparison notes that the alternative can produce zero-contact predictions or red regions that are not anthropomorphically plausible.

5 Discussion

GRAB extends beyond prior motion-capture and grasping datasets by retaining whole-body motion, supporting richer interaction modeling and several downstream applications. Its main scope limitation is the absence of synchronized image data.

  • Discussion: GRAB includes whole-body motion, enabling richer modeling than the demonstrated GrabNet application alone.The discussion distinguishes the dataset’s broader whole-body information from GrabNet’s hand-grasp prediction.
  • Limitations: Accurate MoCap was prioritized over synchronized image capture, so GRAB does not contain synced image data.The paper notes that synthetic rendering and learned priors can nevertheless support image-based inference.
  • Future work: The dataset can support interaction modeling, robotic grasping from imitation, marker-to-mesh mapping, synthetic-image rendering, object inference, and temporal analysis.These are listed as future or potential uses of GRAB.

GRAB: A Dataset of Whole-Body Human Grasping of Objects *Supplemental Material*

The supplemental material identifies the paper’s authors and their institutional affiliation at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.

  • The paper is authored by Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas.
  • The authors are affiliated with the Max Planck Institute for Intelligent Systems in Tübingen, Germany.
  • The supplemental material lists the contact emails otaheri, nghorbani, black, and dtzionas at tuebingen.mpg.de.

S.1 Supplementary Video

The supplementary video presents an overview of the method, GRAB sequences, GrabNet predictions for unseen objects, and failure cases.

  • Supplementary Video: The video includes a narrated overview of the method and a wide variety of GRAB sequences showing 3D moving meshes.
  • Supplementary Video: The video shows GrabNet predictions for unseen objects from several viewpoints, alongside GrabNet failure cases.

S.2 GRAB Dataset Content

GRAB’s research release provides the meshes, markers, subject templates, pose parameters, reconstruction code, contact annotations, and Vicon MoCap files needed to reproduce the dataset.

  • The release includes modified object meshes and marker locations for each object.
  • It provides body-shape templates plus pose parameters for subjects and objects.
  • Code reproduces the interacting meshes shown in the authors’ video.
  • Per-vertex contact annotations are provided for body and object meshes at each frame.
  • Vicon MoCap files contain labeled marker positions, including markers on the floor and table.

S.3 Why MoCap Instead of 3D Scan Sequences?

The authors choose optical MoCap and model fitting over full interaction scans because it is more practical for accurate, scalable human-object capture and object tracking under occlusion.

  • 3D scanning does not scale to interaction sequences because it generates huge data volumes and requires substantial processing.
  • Object tracking remains challenging with scans under occlusion because scan-to-model correspondence is ill-posed.
  • A minimum of 3 MoCap marker observations is sufficient for reliable object pose estimation.
  • Many small object markers support object-pose estimation despite occlusion.
  • MoSh++ produces body meshes comparable to 3D scanning when given ground-truth body shape, while requiring a more practical capture process.
  • The dataset therefore uses a high-end optical MoCap system and fits full 3D human and object meshes to marker data.

S.4 Why not Capture RGB Images?

GRAB prioritizes accurate, naturalistic whole-body interaction geometry over RGB capture, avoiding instrumentation-related limitations and reducing penetration artifacts relative to prior RGB-based datasets.

  • RGB interaction datasets capture only the hand rather than the whole body, and some use visible hand instrumentation.
  • Prior RGB-based methods exhibit severe hand-object inter-penetration, measured at 11.0 ± 8.9 mm and 4.36 ± 0.94 mm in two cited datasets.
  • GRAB’s “use” grasps have 3.25 ± 0.68 mm average surface penetration and realistic body-object contact while including whole-body and dexterous interactions.
  • The precision-focused setup sacrifices RGB images because the MoCap suit and 3D-printed objects have uniform, artificial textures.
  • GRAB’s contact heatmaps show frequent thumb and fingertip involvement, unlike the cited HO-3D comparison.

S.5 Penetration Plots

The penetration analysis evaluates realistic “use” sequences, where the reported average inter-penetration is attributed largely to missing soft-tissue deformation.

  • 3.25 ± 0.68 mm average penetration is reported for GRAB’s “use” sequences.
  • “Use” sequences are selected because they present the most realistic occlusions and capture challenges.
  • The measured penetration effectively corresponds to missing soft-tissue deformation.

S.6 Protocol Details

GRAB varies object placement and captures four interaction intents to increase motion and grasp diversity. Motion-capture markers are recorded on the body, face, fingers, object, floor, and table.

  • Interaction intents: GRAB captures “use,” “pass,” “lift,” and “off-hand pass” interaction intents.The protocol includes everyday use, object passing, lifting, and bimanual transfer followed by use.
  • Scene randomization: Object position and pose are randomized on a resting table, whose height varies from 75 cm to 120 cm.These randomizations increase motion variance across sequences.
  • Penetration measurement: For “use” grasps, mean vertex penetration is 3.25 ± 0.68 mm.The supplementary plot also stores maximum, mean, and median penetration per frame.
  • Grasp variation: Repeated placement and re-grasping encourage exploration of contact configurations and relative hand-object orientations.Subjects repeat trials after leaving the object on the table in random poses.
  • Instrumentation: MoCap markers are placed on the body, face, fingers, object, floor, and table.Subjects gave informed consent to share their motion data for research.

S.7 Computing Contact

The contact pipeline cleans noisy mesh collisions, supports binary and fine-grained labels, and underlies GRAB’s modeling and GrabNet application. Supplementary analyses report contact patterns, plausible generated grasps, and remaining penetration failures.

  • Contact cleaning: Noise, fitting errors, and unmodeled contact-dependent skin deformation can make fingers penetrate both outer and inner object surfaces.This is illustrated for thin-walled cup meshes.
  • Contact cleaning: The pipeline clusters colliding triangles into connected rings, removes rings associated with inner-surface penetration, and labels enclosed vertices as contact.For thin objects, the outer-surface contact is retained while inner-surface artifacts are discarded.
  • Contact representations: Contact labels range from binary contact status to hand-part identities and fine-grained points on the 3D hand surface.Object contact vertices can be assigned the ID of their closest SMPL-X/MANO bone.
  • Mesh reconstruction: GRAB uses adapted MoSh++ with SMPL-X to estimate body, face, and hand motion from markers, producing moving meshes for contact inference.The adaptation uses ground-truth body shape and extends estimation to facial pose and expression parameters.
  • GrabNet: GrabNet predicts grasping poses for 3D objects with a coarse estimate followed by refinement, using GRAB’s right-hand data for this application.The two networks are trained separately.
  • GrabNet results: Most generated grasps look natural across 10 grasps for each of 6 unseen objects, but thin parts produce penetrating-finger failures.The frying pan is especially challenging because of its large size, thin walls, and handle; penetration and contact are soft training constraints.
  • Contact statistics: For “use” sequences, the right hand contacts the object in 90.62% of frames, with the thumb fingertip contacting in 99.88% of those frames.These percentages are computed over all frames and over the subset of right-hand contact frames, respectively.
  • Contact statistics: Clustering reveals that several grasp clusters, typically 3–6 of 20, contain fingers contacting markers.The analysis uses k-means with k=20 across grasps for five objects.
Loading 2008.11200v1…