Source-linked AI summary
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping
Omid Taheri, Vasileios Choutas, Michael J. Black, Dimitrios Tzionas
TL;DR
GOAL addresses the challenge of generating coordinated full-body, head, and hand motions for grasping unknown objects. It uses GNet to generate a goal grasp and MNet to generate the motion from a starting pose, combining SMPL-X parameters with 3D offsets. On GRAB, GOAL generalizes to unseen objects, outperforms baselines, and produces realism comparable to ground-truth motions.
Problem
Existing methods separately model body motion or static hand grasps, leaving coordinated full-body object grasping with head, hands, and physical contact insufficiently addressed.
Method
GOAL uses GNet to generate a whole-body grasp and MNet to autoregressively generate motion from a starting pose to that goal, using SMPL-X parameters and 3D vertex offsets.
Results
GOAL generalizes to withheld GRAB objects, outperforms baselines, and achieves perceptual realism comparable to GRAB ground-truth motions.
Takeaways & Limitations
GOAL takes a step toward automatic synthesis of natural and physically plausible whole-body grasping motions for realistic avatars.
Takeaways & Limitations
GOAL is constrained to starting close to the object and does not generate motions when the body is far away; longer pre-interaction walking remains future work.
Abstract
from arXiv · showhide
Generating digital humans that move realistically has many applications and is widely studied, but existing methods focus on the major limbs of the body, ignoring the hands and head. Hands have been separately studied, but the focus has been on generating realistic static grasps of objects. To synthesize virtual characters that interact with the world, we need to generate full-body motions and realistic hand grasps simultaneously. Both sub-problems are challenging on their own and, together, the state-space of poses is significantly larger, the scales of hand and body motions differ, and the whole-body posture and the hand grasp must agree, satisfy physical constraints, and be plausible. Additionally, the head is involved because the avatar must look at the object to interact with it. For the first time, we address the problem of generating full-body, hand and head motions of an avatar grasping an unknown object. As input, our method, called GOAL, takes a 3D object, its position, and a starting 3D body pose and shape. GOAL outputs a sequence of whole-body poses using two novel networks. First, GNet generates a goal whole-body grasp with a realistic body, head, arm, and hand pose, as well as hand-object contact. Second, MNet generates the motion between the starting and goal pose. This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact. To achieve this, the networks exploit a representation that combines SMPL-X body parameters and 3D vertex offsets. We train and evaluate GOAL, both qualitatively and quantitatively, on the GRAB dataset. Results show that GOAL generalizes well to unseen objects, outperforming baselines. GOAL takes a step towards synthesizing realistic full-body object grasping.
1. Introduction
GOAL addresses coordinated whole-body grasping by jointly generating body, head, hand, and object interaction motions. It uses separate networks for a realistic goal grasp and the motion from a starting pose, and generalizes to unseen objects in GRAB.
- Existing motion-generation methods commonly omit the head and hands or ignore object context, while grasp-generation methods often ignore the body.
- Whole-body grasping requires coordinated walking, head orientation, torso leaning, arm extension, and dexterous hand-object contact.
- GOAL takes a 3D object, its position and orientation, and a starting body pose and shape to generate a sequence ending in an object grasp.
- GNet generates a varied goal whole-body grasp, while MNet autoregressively generates the motion between the starting and goal poses.
- GOAL combines SMPL-X parameters with 3D vertex offsets and uses optimization to improve pose realism and physical plausibility.
- Evaluation on withheld GRAB objects shows natural full-body grasping motions, baseline improvements, and perceptual realism comparable to ground-truth motions.
2. Related Work
Prior work studies body motion, scene interaction, hand motion, and grasp pose generation largely as separate problems. GOAL instead targets full-body SMPL-X motion from approaching an object to grasping it, using a two-stage goal-and-motion pipeline.
- Motion generation for bodies “in isolation”: Body-motion methods often operate without scene context and may struggle with discontinuities between observed and predicted poses.
- Motion for full-body interactions: Scene-interaction methods use contact annotations, optimization, reinforcement learning, or geometric representations for body or hand interactions.
- Motion generation for bodies in 3D scenes: Data-driven approaches generate context-aware or start-to-goal motion, sometimes dividing long sequences into short pairs and stitching them together.
- Motion generation for hands: Hand-motion research addresses physically plausible instrument, grasping, and interaction trajectories, but focuses on hands rather than coordinated full-body motion.
- Pose generation for bodies and hand-object grasps: Static pose-generation methods fit bodies or hands to scene and object contacts, including unseen-object hand grasps refined with optimization.
- Summary: GOAL learns full-body SMPL-X motion from walking toward an object through grasping, given a 3D object and a starting human pose.
3. Method
GOAL combines SMPL-X parameters, interaction-aware vertex offsets, and two networks to generate whole-body grasping motion. GNet synthesizes and optimizes a grasp goal, while autoregressive MNet generates the motion from the starting pose to that goal with improved hand guidance and foot-ground contact.
- Method overview: GOAL takes an object, its pose, and a nearby starting body pose and shape, then generates SMPL-X motion through a grasping goal.The starting pose is roughly 0.5–1.5 m from the object and oriented approximately ±10° toward it.
- Human model: SMPL-X jointly models the body, head, face, and hands; GOAL samples 400 interaction-relevant body vertices for its representations.The model outputs a mesh with 10,475 vertices and uses interaction-guided sampling based on GRAB heatmaps.
- Interaction representation: Interaction-aware attention exponentially emphasizes body vertices near the object and downweights distant vertices, using a learnable distance-scaling parameter set to w = 5.This representation focuses on regions such as hands for grasping and feet for walking rather than treating all body vertices equally.
- GNet: GNet is a conditional variational auto-encoder that predicts a whole-body grasp, head direction, and hand-to-object offsets from object conditions and a sampled 16-dimensional grasp code.Its inputs include SMPL-X parameters, sampled vertex coordinates, head orientation, object translation, and a BPS object-shape representation.
- GNet optimization: GNet predictions are refined by optimizing SMPL-X pose and translation with hand-offset, head-orientation, pose-coupling, and foot-ground constraints.The final energy combines five terms, and the optimization post-processing is evaluated perceptually.
- MNet: MNet autoregressively generates arbitrary-length motion toward GNet’s goal using recent frames, vertex locations and velocities, hand offsets, and goal-hand BPS features.It predicts relative changes for future frames; foot losses and velocities address skating, while hand motion is guided toward the goal grasp.
4. Experiments
Experiments evaluate GOAL’s static grasps and generated motions qualitatively, perceptually, and through an architectural ablation. Optimization improves GNet grasps, while MNet produces realistic motions and more accurate reconstructions with more output frames.
- Qualitative results: GOAL generates representative motions across different object shapes, locations, and body shapes.
- Perceptual evaluation: The perceptual study finds optimization especially improves the realism of hand grasps.
- Perceptual evaluation: Optimized grasps receive better grasping-pose and head-orientation ratings than ground truth, while overall quality remains close to ground truth.The reported explanation is that GNet consistently orients the head toward the object, unlike some GRAB examples.
- Motion evaluation: MNet generates realistic grasping motions approaching ground-truth realism across body motion, foot-ground contact, final hand-object grasp, and head orientation.
- Ablation study: Increasing MNet’s output frames from 1 to 10 improves reconstruction losses and helps motions converge to a final grasp.Lower-frame networks can produce motions whose hands gradually deviate from the object.
5. Conclusion and Future Work
The paper concludes that GOAL synthesizes natural, physically plausible grasping motions for previously unseen objects using staged goal-grasp and motion generation. Future work targets longer pre-interaction walking, scene-aware interaction, and responsible research-only release in response to deepfake concerns.
- Conclusion: GOAL combines GNet goal-grasp generation, MNet motion inpainting, Euclidean interaction features, and optimization to synthesize natural, physically plausible grasping motions.
- Future work: GOAL is constrained to starting near the object and cannot generate motions when the body is far away.Future work should extend it to longer walking motions before object interaction.
- Future work: The current work focuses on human-object interaction rather than scene-aware grasping motions.The authors propose combining GOAL with human-scene interaction models in future work.
- Social impact: The authors identify potential full-body deepfake misuse and state that models will be released only for research purposes.
*Supplemental Material*
The supplemental material documents the GNet architecture and optimization post-processing, and provides a video because motion is important for evaluating output realism.
- Supplemental overview: The supplemental material includes an architectural overview of GNet and its optimization post-processing step.
- Supplemental overview: A supplemental video is important for evaluating the realism of movement-based results.
6. Data Preparation
Data preparation constructs separate static-grasp and motion datasets from GRAB. GNet uses stable right-hand grasp frames, while MNet uses sliding 21-frame subsequences aligned to the current body coordinate system.
- GNet data: GNet data preparation selects GRAB frames with stable right-hand grasps and creates training, testing, and validation sets of 160K, 26K, and 12.5K frames.
- MNet data: MNet data preparation gathers each sequence from its start until the right hand first establishes a stable grasp.
- MNet data: MNet forms 21-frame subsequences with 10 past frames, one current frame, and 10 future frames using a stride of one.
- MNet data: Past and future frames are expressed relative to the current frame’s body coordinate system while gravity remains upward.
- MNet data: The resulting MNet dataset contains roughly 40K, 7K, and 3K training, testing, and validation motion subsequences.
7. GNet Architecture
The referenced material directs readers to R.7 for an architectural overview of GNet and its optimization-based post-processing.
- R.7 provides an architectural overview of GNet.
- The referenced overview also covers GNet’s optimization-based post-processing.
- The architecture and post-processing details are presented outside this passage, in R.7.
8. MNet Architecture
The referenced material directs readers to R.8 for an architectural overview of MNet and its optimization-based post-processing.
- R.8 provides an architectural overview of MNet.
- The referenced overview also covers MNet’s optimization-based post-processing.
- The architecture and post-processing details are presented outside this passage, in R.8.
9. Video
The paper provides a narrated video explaining its motivation and method, showing many qualitative motion results; R.8 presents MNet’s architecture and post-processing overview.
- The narrated video explains the paper’s motivation.
- The narrated video explains the paper’s method.
- The video shows many results, including qualitative motion results.
- R.8 provides an architectural overview of MNet and its optimization-based post-processing.