GRAB: A Dataset of Whole-Body Human Grasping of Objects

Motivation

• The dataset addresses a gap in complex human-object interaction data by representing 3D body shape, full-body motion, and detailed contact, which images and instrumentation do not capture together.Everyday object use can involve multiple body parts, not only the hand; examples include bringing an object to the mouth and drinking from a bowl.Source: §1 · pp. 1–2, Figure 1 · p. 2

Capture method

• GRAB adapts MoSh++ to recover body, hand, and facial motion from MoCap markers and tracks manipulated objects, enabling detailed moving meshes and inferred body-object contact.The method fits SMPL-X to a scan of each subject before recovering pose; small hemispherical object markers are reported not to impact grasping behavior.Source: §1 · pp. 2–3

Dataset

• GRAB captures whole-body grasps across 10 people and 51 objects, including full-body motion, object motion, in-hand manipulation, re-grasps, and body-object contact.Interactions span lifting, handing over, passing an object between hands, and using it.Source: §1 · pp. 2–3

Applications

• The paper analyzes observed body-object contact patterns and demonstrates GRAB’s machine-learning value by training a conditional network to generate plausible hand grasps for unseen 3D objects.The network uses coarse prediction followed by refinement, represents object shapes with basis point sets, and predicts wrist pose and full finger articulation.Source: §1 · p. 3

Hand Grasps

• Hand-grasp studies define taxonomies using object shape and purpose, contact areas, hand pose, or finger and thumb configuration, making accurate hand poses and hand-object contact central.Approaches include ink, data gloves, and tactile gloves.Source: §2 · p. 3

Whole-Body Grasps

• Prior whole-body grasp work is limited and uses unrealistic humanoids with simple objects or synthetic objects, whereas this work uses SMPL-X to capture whole-body, face, and dexterous in-hand interactions.Borras et al. present a multi-contact body-pose taxonomy, while Hsiao et al. study imitation learning from teleoperated simulated-robot demonstrations.Source: §2 · pp. 3–4

Capturing Interactions with MoCap

• MoCap studies capture bodies or hands interacting with scenes and objects, but none captures full 3D bodies, hands, and faces together with 3D object manipulation and contact.Mandery et al. capture interaction with large and handheld objects but omit articulated faces and fingers; other body datasets omit fingers, body shape, or object manipulation.Source: §2 · p. 4

Capturing Contact

• Human-object contact is difficult to capture under occlusion: sensors can bias natural grasps or be hard to synchronize, and recent realistic hand-object or body-scene datasets avoid direct contact measurement.Thermal imaging captures only static grasps held long enough for heat transfer.Source: §2 · p. 4

3D Interaction Models

• Hand-crafted contact-and-physics methods model body-scene, body-object, and hand-object interactions, with rough contact from skeletons or primitives and greater accuracy from 3D meshes.Meshes may be generic, personalized, or based on statistical models.Source: §2 · pp. 4–5
• Existing hand-interaction data include synthetic manually posed grasps and automatically generated MANO grasps, while magnetic-sensor captures of real hands have noisy poses and severe inter-penetrations.GraspIt optimizes hand-crafted metrics that may not reflect the human-grasp distribution.Source: §2 · p. 5

Novelty

• Prior work addresses only parts of the problem; drawing especially on, this work introduces a dataset of real whole-body grasps.The paper positions its dataset as going beyond those prior efforts.Source: §2 · p. 5

3.1 Motion Capture (MoCap)

• The capture setup uses 54 infrared cameras recording 16 MP at 120 fps to reduce occlusion and capture contact detail; four trained technicians clean marker occlusions, swaps, and ghost markers.Small markers allow high-resolution capture while minimizing their influence on finger and face motion.Source: §3.1 · p. 6

3.2 From MoCap Markers to 3D Surfaces

• Personalized SMPL-X models are fitted to cleaned human markers, while object-specific marker correspondences and detected object markers recover rigid object poses frame by frame.SMPL-X models body, articulated face, and fingers; human shape is informed by a 3D scan, and object pose is solved from three or more detected markers.Source: §3.2 · pp. 7–8

3.3 Contact Annotation

• Contact is estimated from mesh proximity within a tolerance, with intersection-ring heuristics addressing penetrating contacts and proximity checks addressing missed contacts.The empirically selected contact threshold is 4.5 mm.Source: §3.3 · pp. 8–9

3.5 Analysis

• GRAB contains 1,334 sequences and over 1.6M MoCap frames; its contact analysis selects 952,514 frames using object vertical movement and a minimum of 50 contacting body vertices.The analysis assumes objects start static on a table; the stated thresholds have little influence on the analysis.Source: §3.5 · p. 10
• In “use” contact frames, 92% involve the right hand, 39% the left, 31% both hands, and 8% the head; right-hand finger contact likelihood ranges from 100% at the thumb to 39% at the pinky.For right-hand contact, the palm likelihood is 24%; lips, nose, temporal head area, and ears are also contact hotspots.Source: §3.5 · pp. 10–11
• “Use” contact follows object function, avoiding knife blades and hot pan areas, while “pass” contact favors one side and leaves the other free for the receiver.For example, people contact a flashlight’s on/off button during “use” sequences.Source: §3.5 · p. 11, Figure 6 · p. 11
• Markers often lie in contact-hot areas, suggesting subjects do not avoid grasping those locations; contact patterns also vary with object and hand size.Hands and face both contribute to everyday-object use, and fingers contact objects more often than the palm.Source: §3.5 · pp. 10–11, Figure 5 · p. 10

GrabNet

• GrabNet predicts full MANO hand grasps for unseen objects via separately trained CoarseNet and neural RefineNet, unlike ’s robotic-gripper prediction and optimization-based refinement.Source: §4 · pp. 11–12, Figure 7 · p. 13

CoarseNet

• CoarseNet encodes object shape as BPS distances and samples a 16-dimensional grasp latent space to decode fully articulated MANO hands, trained with mesh, penetration, contact, and cVAE losses.Source: §4 · pp. 11–12

RefineNet

• RefineNet uses GRAB-learned vertex contact likelihoods and hand-object distances to refine CoarseNet poses in three iterations, trained on perturbed synthetic CoarseNet grasps.Source: §4 · p. 12

Evaluation - CoarseNet/RefineNet

• CoarseNet’s train/validation/test reconstruction vertex-to-vertex errors were 12.1/14.1/18.4 mm; RefineNet’s were 3.7/4.1/4.4 mm.The components were trained separately and worked reasonably well before being combined.Source: §4 · pp. 12–13

Evaluation - GrabNet

• An AMT study compared 20 generated with 20 ground-truth grasps on each of six test objects using five-level human-graspability ratings and implausible-grasp catch trials.Source: §4 · p. 13

Evaluation - Contact

• ContactDB produces only 10 predictions per object, sometimes none, whereas GrabNet’s MANO-derived contacts are anthropomorphically valid and its learned grasp space supports infinite samples.Source: §4 · pp. 13–14, Figure 9 · p. 14

Discussion

• GRAB includes whole-body motion, enabling richer modeling than GrabNet, and its data and variability suffice to train a network that predicts human grasps of objects.The discussion presents GrabNet as an example of the dataset’s potential, not the limit of what whole-body data can support.Source: §5 · p. 14
• GRAB lacks synchronized image data because it prioritizes accurate MoCap, but can support image-based inference through synthetic interaction rendering or priors for ill-posed 2D inference.The cited methods are potential uses enabled by GRAB, not experiments reported in this discussion.Source: §5 · p. 14
• Future directions include interaction modeling, robotic grasping by imitation, marker-to-mesh mapping, synthetic rendering, object shape/pose inference, and temporal-pattern analysis.Source: §5 · p. 14

• The video shows a narrated method overview, 3D moving-mesh GRAB sequences, GrabNet predictions for unseen objects from several viewpoints, and GrabNet failure cases.Source: §S.1 · p. 20

• The website provides modified object meshes from, marker locations on each mesh, and body-shape templates for the subjects.Source: §S.2 · p. 20
• It includes pose parameters for subjects and objects, plus code to reproduce the interacting meshes shown in the video.Source: §S.2 · p. 20
• The release also contains per-vertex body and object contact annotations for each frame and labeled Vicon MoCap marker positions, including floor and table markers.Source: §S.2 · p. 20

S.3 Why MoCap Instead of 3D Scan Sequences?

• MoCap scales better to interaction sequences than 3D scanning, which creates substantial data-processing demands and makes object tracking under occlusion difficult.With many small markers on objects, at least three marker observations suffice for reliable object-pose estimation.Source: §S.3 · p. 20
• Using a ground-truth body shape, MoSh++ produces meshes on par with 3D scans, enabling a practical, scalable approach that fits human and object meshes to optical MoCap markers.The authors use a high-end optical MoCap system and fit full 3D meshes to markers for both the human and the object.Source: §S.3 · p. 21

• Capturing accurate interactions and natural RGB images is challenging: prior RGB datasets show only hands, and visible hand instrumentation makes imagery less natural.The paper frames the choice as accurate grasping requiring instrumentation versus natural images reducing ground-truth accuracy.Source: §S.4 · p. 21
• [4] reports 11.0 ± 8.9 mm average skeleton penetration, while has 4.36 ± 0.94 mm surface penetration and rarely contacts thumb and fingertips, unlike GRAB.The contact heatmaps show thumb and fingertip contact more frequently in GRAB than in HO-3D.Source: §S.4 · p. 21, Figure S.1 · p. 22
• GRAB’s “use” grasps average 3.25 ± 0.68 mm surface penetration, significantly below, with realistic body-object contact in whole-body dexterous manipulation.The captured scenarios include dexterous in-hand manipulation and the whole body rather than only the hand.Source: §S.4 · p. 21
• GRAB’s precision-focused setup forgoes RGB because its MoCap suit and 3D-printed objects have uniform, artificial textures; its accurate meshes could serve as priors for improving.The authors present this as a sensible trade-off for learning 3D interactions and improving future hand or whole-body methods.Source: §S.4 · p. 21

S.5 Penetration Plots

• “Use” grasps have 3.25±0.68 mm average penetration, effectively corresponding to missing soft-tissue deformation.The authors note that no human-body model with articulated fingers and face captures this soft-tissue deformation with contact.Source: §S.5 · p. 22
• For “use” grasps, 67% have ≤3.5 mm penetration, 86% ≤4.0 mm, and 96% ≤4.5 mm.Source: §S.5 · p. 22
• For “use” grasps, 99.9% have ≤5.8 mm penetration.Source: §S.5 · p. 22

S.6 Protocol Details

• Object positions and poses were randomized on tables whose heights varied from 75 cm to 120 cm to increase motion variance.Source: §S.6 · p. 22
• The four intents were use, pass, lift, and off-hand pass, with multiple use sequences for objects with multiple everyday uses and inspection for objects without a clear use.Pass directions were randomized; lift sequences involved stable natural grasps, repeated re-grasping, and random object poses to explore contact configurations and hand-object orientations.Source: §S.6 · pp. 22–23
• 3.25 ± 0.68 mm was the mean vertex penetration for “use” grasps.The figure plots maximum, mean, and median penetration per frame.Source: Figure S.2 · p. 23
• Off-hand pass sequences involved grasping with the off hand, passing the object to the dominant hand, and then using it.Source: §S.6 · pp. 22–23
• Markers were recorded on the body, face, fingers, object, floor, and table, and all subjects consented to research sharing of their motion data.Source: §S.6 · p. 23

S.7 Computing Contact

• Cup meshes can have closely spaced inner and outer surfaces, and fitting noise causes fingers to penetrate both, although the intended contact is on the outer surface.The passages attribute this to noise, fitting errors, and models that do not represent contact-dependent skin deformations.Source: §S.7 · p. 23
• The method clusters colliding triangles into connected rings, maps each ring to penetrating hand areas, ignores inner-surface areas, and labels vertices enclosed by the remaining rings as contact vertices.For the cup example, hand areas contacting the inner surface are a subset of those contacting the outer surface.Source: §S.7 · pp. 23–24, Figure S.3 · p. 24
• Beyond binary contact labels, annotations can identify the contacting hand part or a point on the hand surface; hand-part labels use the ID of the closest SMPL-X/MANO bone to each contacting object vertex.Figure S.3 illustrates binary labels and hand-part labels; the text also describes point-on-surface labels as a finer option.Source: §S.7 · p. 24, Figure S.3 · p. 24

S.8 Adapting MoSh++

• MoSh++ is adapted to fit scanned body shape, SMPL-X body, hand, and face parameters, while object pose is estimated independently per frame.Because synchronized scans with clear body, face, and finger regions are impractical, the authors tune weights on synthetic sequences assembled from body, hand, and face datasets with simulated marker noise and dropouts; the adapted fitting improves over vanilla MoSh++ on the synthetic test set.Source: §S.8 · pp. 24–26, Table S.1 · p. 26

S.9.1 Data Preparation

• GrabNet training uses stable right-hand grasps, object-distance BPSo representations, 41 training objects, four validation objects, and six held-out test objects.The stable-grasp filters require right-hand contact, no left-hand contact, object lift, and thumb plus another finger contact; the resulting splits contain roughly 320k, 31k, and 65k data points.Source: §S.9.1 · pp. 26–27

S.9 GrabNet; S.9.2 Results: Success and Failure Cases

• Most generated grasps for six unseen objects look natural and plausible, but failures mainly involve finger penetration on thin parts, especially the frying pan.The authors attribute this partly to sparse BPSo shape representation and soft penetration/contact losses, and suggest hard-constraint refinement as a possible improvement.Source: §S.9.2 · p. 27

S.9.3 GrabNet Implementation Details

• CoarseNet conditions on object BPSo and a latent code to predict MANO hand parameters, while RefineNet iteratively refines grasps using hand-to-object vertex distances.Validation selected a 16-dimensional latent space; CoarseNet and RefineNet are trained separately for 16 and 23 epochs, respectively, with validation-based learning-rate reduction and early stopping.Source: §S.9.3 · pp. 27–28

S.9.4 Filtering out Unreliable Turkers

• The user study filters unreliable Turkers by removing those who rate noisy perturbed grasps as realistic at least 20% of the time, excluding 54 of 170 participants.Source: §S.9.4 · p. 28

S.9.5 Heatmaps for Various Intents and Fine-Grained Numbers

• For “use” sequences, the right hand contacts the object in 90.62% of frames, and the thumb fingertip contacts it in 99.88% of those contact frames.The supplementary heatmaps report contact percentages across intents for the hands and head, both over all frames and within frames where each part is in contact.Source: §S.9.5 · p. 28, Figure S.9 · p. 34

S.10 Bias from MoCap Markers

• K-means grasp clusters show marker contact in several clusters, providing additional evidence that the 1.5 mm hemispherical markers cause no or minimal grasp bias.Subjects did not report discomfort or bias, although the authors say more evidence is needed; markers are preferentially placed in areas less likely to be contacted, with some elsewhere to address occlusion.Source: §S.10 · pp. 28–29, Figure S.10 · p. 35

S.11 Influence of Contact Heuristic Thresholds; S.12 Influence of Object Size; S.13 Vicon Software

• Contact heuristic threshold changes have minimal effect on integrated heatmaps, while smaller objects are grasped mainly one-handed and larger objects draw more help from the other hand.Smaller subjects show more bimanual grasps and greater finger and metacarpophalangeal contact; Vicon Shōgun-Live handles calibration, synchronization, and capture, while Shōgun-Post supports semi-automatic marker cleaning by trained annotators.Source: §S.11 · p. 29, §S.12 · p. 29, §S.13 · p. 29