PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGA: Stochastic Whole-Body Grasping with Contact

Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, Siyu Tang

arXiv:2112.10103v3cs.CV

TL;DR

Existing grasp synthesis typically focuses on the hand, while whole-body grasping must also model body dynamics, dexterous fingers, and contacts. SAGA jointly generates static grasp poses and human–object contacts, then infills contact-aware motions between starting and grasping poses. It generates diverse motions toward unseen objects, with synthesized motions rated 3.15 versus 4.04 for ground-truth sequences in the reported user study.

  • Problem

    Existing grasp methods typically synthesize hand–object interaction alone, while whole-body grasping must combine body dynamics, dexterous fingers, and complex contacts.

    Source: §1 · pp. 1–2

  • Method

    SAGA jointly generates static whole-body grasp poses and human–object contacts, then uses contact-aware generative infilling to create motions between start and grasping poses.

    Source: §1 · p. 2, §3.2 · pp. 6–7, §3.3 · p. 9

  • Results

    SAGA generates diverse motions toward unseen objects; synthesized sequences received a perceptual score of 3.15 versus 4.04 for ground-truth sequences.

    Source: §1 · p. 2, §4.3 · p. 14

  • Takeaways & Limitations

    SAGA provides a framework for generating realistic whole-body approaches and grasps of random objects, with realistic ground and object interactions.

    Source: §5 · p. 14

    Some generated grasps of frying pans fail to grasp the handle despite contact, producing perceptually unrealistic results and low perceptual scores.

    Source: D Limitations · p. 29

Abstract

from arXiv · show

The synthesis of human grasping has numerous applications including AR/VR, video games and robotics. While methods have been proposed to generate realistic hand-object interaction for object grasping and manipulation, these typically only consider interacting hand alone. Our goal is to synthesize whole-body grasping motions. Starting from an arbitrary initial pose, we aim to generate diverse and natural whole-body human motions to approach and grasp a target object in 3D space. This task is challenging as it requires modeling both whole-body dynamics and dexterous finger movements. To this end, we propose SAGA (StochAstic whole-body Grasping with contAct), a framework which consists of two key components: (a) Static whole-body grasping pose generation. Specifically, we propose a multi-task generative model, to jointly learn static whole-body grasping poses and human-object contacts. (b) Grasping motion infilling. Given an initial pose and the generated whole-body grasping pose as the start and end of the motion respectively, we design a novel contact-aware generative motion infilling module to generate a diverse set of grasp-oriented motions. We demonstrate the effectiveness of our method, which is a novel generative framework to synthesize realistic and expressive whole-body motions that approach and grasp randomly placed unseen objects. Code and models are available at https://jiahaoplus.github.io/SAGA/saga.html.

1 Introduction

Prior work largely separates hand–object grasp synthesis from whole-body motion, leaving realistic whole-body grasping with diverse body–scene and hand–object contacts unsolved. SAGA combines static grasp-pose generation with contact-aware motion infilling to synthesize varied approaches and grasps.

1 Introduction

  • Existing hand–object methods typically omit whole-body movement, while whole-body motion synthesis often ignores scene objects.This leaves whole-body grasping synthesis comparatively unaddressed.Source: §1 · pp. 1–2
  • Whole-body grasping must combine full-body dynamics with dexterous fingers, realistic body–scene and hand–object contacts, and many plausible approaches to objects.The motion should include natural hand conformity and avoid foot skating.Source: §1 · p. 2
  • SAGA generates diverse static whole-body grasping end poses and diverse motions between a given start pose and end pose.The framework targets stochastic approaches and grasps of 3D objects.Source: §1 · p. 2
  • SAGA uses surface markers to represent body shape and pose, and learns contact representations to guide interaction synthesis.The authors report that markers are easier for neural networks to learn than parametric-body latent parameters; contact is used in both components.Source: §1 · pp. 2–3
  • A multi-task conditional VAE jointly learns whole-body marker positions and fine-grained marker–object contacts for static grasp poses.At inference, it generates marker locations and contact labels; contact-aware optimization recovers a parametric mesh while enforcing hand–object contact.Source: §1 · p. 3
  • Motion infilling generates diverse trajectories and local pose articulations between the initial and grasping poses, while predicting foot–ground contacts.Contact-aware motion optimization uses predicted human–object contacts to produce grasp-oriented whole-body motions.Source: §1 · p. 3

2 Related Work

Prior work spans hand-grasp synthesis, general human-motion synthesis, and motion generation in scene context, but the cited approaches leave whole-body grasp learning and stochastic grasp-oriented infilling unaddressed. SAGA distinguishes itself by jointly generating whole-body poses and mutual contacts, then infilling diverse trajectories and articulations.

2 Related Work

  • Hand-grasp methods generate hand poses and contacts, while ManipNet generates dexterous finger motion from wrist and object trajectories; the cited works do not study 3D whole-body grasp learning and synthesis.SAGA instead jointly generates consistent whole-body pose and mutual human–object contacts.Source: §2 · p. 4
  • General deterministic and stochastic motion-prediction methods contribute to motion understanding but do not study interaction with the 3D environment.The cited stochastic approaches include VAE-based models.Source: §2 · p. 4
  • Scene-context motion methods include skeleton-based path and pose prediction, motion infilling, and contact-aware models, with cited limitations including contact modeling, foot-ground naturalness, or deterministic output.One cited infiller requires ten start frames and one end frame; another contact-aware infiller predicts better foot-ground contact but is deterministic.Source: §2 · pp. 4–5
  • Unlike cited scene-motion infillers, SAGA captures diversity in both global trajectories and local body articulations from the first and last frames.The concurrent GOAL pipeline also uses two stages, but its deterministic autoregressive infilling does not fully explore grasping-motion uncertainty.Source: §2 · p. 5

3 Method

SAGA represents people with sparse body markers and uses a two-stage pipeline: it generates object-conditioned grasping end poses, then infills motions between the initial and target poses. Contact-aware optimization converts the predicted markers and contacts into body meshes and smooth grasping motions.

3.1 Overview

  • The pipeline first generates object-conditioned grasping end poses and then infills motions from the initial pose to that target, with optimization recovering body meshes and grasping motions.The method represents people using sparse surface markers and recovers SMPL-X body meshes from them.Source: §3.1 · pp. 5–6, §3 · p. 5, Figure 2 · p. 6

3.2 Whole-Body Grasping Pose Generation

  • WholeGrasp-VAE jointly learns diverse whole-body marker poses and mutual human–object contacts, using reconstruction, KL-divergence, and consistency losses.Its reconstruction loss covers marker positions and body-marker and object contact probabilities.Source: §3.2 · pp. 6–7
  • GraspPose-Opt converts predicted markers into a body mesh by optimizing SMPL-X parameters while leveraging predicted contacts for high-fidelity interaction.The optimization includes marker fitting, mutual object contact, collision, and ground-contact terms.Source: §3.2 · pp. 7–8

3.3 Generative Motion Infilling

  • MotionFill-VAE infills motion through two conditional VAEs: TrajFill predicts an intermediate root trajectory, and LocalMotionFill generates local motion conditioned on that trajectory and the endpoint poses.Its four-channel input combines local motion and foot–ground contact labels with root velocity; training reconstructs trajectory, local motion, and foot-contact labels.Source: §3.3 · p. 9

3.4 Contact-aware Grasping Motion Optimization

  • GraspMotion-Opt recovers smooth motions from generated markers and predicted foot–ground and human–object contacts, applying object-contact loss only in the final five frames.It also encourages the palm to face the object when close to its surface and uses foot-contact labels to reduce foot skating.Source: §3.4 · pp. 9–10

4 Experiments

The experiments evaluate SAGA’s grasp-pose generator, stochastic motion infiller, and full grasping pipeline. Results report improved grasp interactions and motion quality, with generated grasping motions rated favorably in a user study.

4 Experiments

  • The evaluation uses GRAB to train and evaluate WholeGrasp-VAE, GRAB to fine-tune MotionFill-VAE, AMASS to pretrain LocalMotionFill-CVAE, and unseen HO3D objects to test generalization.The study evaluates each stage and the full pipeline.Source: §4 · p. 10

4.1 Stochastic Whole-body Grasp Pose Synthesis

  • WholeGrasp without optimization surpassed extended GrabNet in diversity, contact ratio, and intersection, while contact-aware optimization reduced collisions and maintained high contact.The comparison uses WholeGrasp without GraspPose-Opt because the extended GrabNet baseline has no corresponding mesh-refinement step.Source: §4.1 · p. 11, Table 1 · p. 11
  • Five samples for an unseen object illustrate diverse whole-body grasp poses alongside hand details and predicted object-contact maps.The samples use the same object positions; red indicates high contact probability.Source: §4.1 · p. 11, Figure 5 · p. 12
  • Multi-task contact learning outperformed single-task pose learning with a predefined contact pattern, and collision and contact losses significantly improved recovered interactions.Fitting only to ground-truth markers recovered an accurate body mesh with interaction metrics comparable to the ground-truth mesh.Source: §4.1 · p. 12, Table 2 · p. 12

4.2 Stochastic Motion Infilling

  • The motion infiller yielded lower marker error, less foot skating, and lower PSKL-J scores than Route+PoseNet and local-motion baselines.The reported non-collision scores were 0.9771 for GRAB motions and 0.9743 for generated motions.Source: §4.2 · pp. 13–14, Table 3 · p. 13

4.3 Whole-body Grasp Motion Synthesis

  • Generated grasping motions received a realism score of 3.15, versus 4.04 for ground-truth sequences, on a five-point user-study scale.The pipeline was tested on 14 unseen GRAB and HO3D objects, with two-second motions.Source: §4.3 · p. 14

5 Conclusion and Discussion

SAGA combines static whole-body grasp-pose generation with motion infilling to produce diverse motions approaching and grasping objects. The paper limits its scope to grasping, not subsequent hand–object manipulation.

5 Conclusion and Discussion

  • SAGA pairs WholeGrasp-VAE for static grasp poses with MotionFill-VAE for motions from an initial pose to the generated end pose.The authors describe the resulting sequences as diverse, with realistic interactions with the ground and random objects.Source: §5 · p. 14
  • The pipeline synthesizes approaches and grasps but does not address subsequent hand–object manipulation.Future work is proposed to incorporate object affordance, physics, and interaction goals.Source: §5 · p. 14

SAGA: Stochastic Whole-Body Grasping with Contact ** Appendix **

  • The appendix covers implementation, datasets, ablations, additional results, and limitations; grasp-pose training and optimization use 143 body-surface markers, including 44 palm markers.Motion-infilling training instead uses a sparse set of palm markers with 10 fingertip markers.Source: SAGA: Stochastic Whole-Body Grasping… · p. 20, §A.1 · p. 20

A.2 Architecture Details

The architecture combines marker-and-contact prediction for static grasp poses with trajectory and local-motion infilling. It uses different marker inputs for the two motion and pose models.

A.2 Architecture Details

  • WholeGrasp-VAE training uses 30 finger markers and 44 palm markers, while MotionFill-VAE uses 10 fingertip markers for its sparse hand input.The figure distinguishes these finger and palm markers from the 69 body-surface markers.Source: Figure S1 · p. 21
  • WholeGrasp-VAE encodes body-marker positions and contacts with object contacts and geometry, then decodes marker positions and body- and object-contact probabilities.The model fuses the body and object features into a joint 16-dimensional latent space; PointNet++ encodes object geometry.Source: WholeGrasp-VAE · pp. 20–21
  • TrajFill learns deviations from a linear-interpolation-and-extrapolation root trajectory conditioned on the supplied start and end root states.Its output is added to the initial rough trajectory.Source: MotionFill-VAE · p. 21
  • LocalMotionFill infills a masked 4-channel local-motion image containing foot-ground contacts, normalized local markers, and root local velocities.The condition image masks the input motion image.Source: MotionFill-VAE · pp. 21–22

A.3 Dataset Processing

The models are trained and evaluated using processed GRAB and AMASS data. GRAB supplies grasping poses and approach-to-grasp sequences, while AMASS supplies motion-infilling clips.

A.3 Dataset Processing

  • GRAB pose training uses right-hand grasps from 41 objects, reserving 4 validation objects and 6 test objects from the 51-object dataset.Male and female models are trained separately, and object point clouds and body markers are centered at the object geometry center.Source: §A.3 · p. 22
  • GRAB motion training clips 2-second approach-and-grasp segments, with final frames at stable grasp poses and first-frame padding for shorter sequences.The selected segments are drawn from sequences that begin in a T-pose and proceed toward object interaction.Source: §A.3 · p. 22
  • AMASS is down-sampled to 30 fps and divided into 2-second grasping-infilling clips, with 4- and 6-second clips used for general motion infilling.Training excludes EKUT, KIT, SFU, SSM synced, TCD handMocap, and TotalCapture; testing uses TCD handMocap, TotalCapture, and SFU.Source: §A.3 · p. 22
  • WholeGrasp-Opt takes sampled markers and marker-object contacts as input, then fits and refines a body mesh through staged optimization.Its three optimization stages use 300, 400, and 500 steps, respectively.Source: §A.3 · p. 23

A.4 Implementation Details

The implementation specifies training losses and staged mesh optimization for the pose and motion models. The user study evaluates synthesized grasping motions on unseen objects against available GRAB ground truth.

A.4 Implementation Details

  • WholeGrasp-VAE uses weighted binary cross-entropy for contact reconstruction, with in-contact weights 3 for objects and 5 for markers, and trains for 40 epochs.The contact-map weights λM and λO are 1, and KL weight annealing is used during training.Source: §A.4 · p. 23
  • GraspPose-Opt uses multi-stage, global-to-local optimization because jointly optimizing high-dimensional SMPL-X parameters can hinder convergence.The three stages use learning rates 0.016, 0.012, and 0.008.Source: §A.4 · pp. 23–24
  • GraspMotion-Opt first fits SMPL-X meshes to predicted marker sequences, then refines them with overall losses for basic fit, ground contact, smoothness, and skating.The two stages take 100 and 300 steps; the smoothness loss acts on motion latents and hand vertices, while the skating loss uses foot-ground contact labels.Source: §A.4 · p. 24

A.5 Amazon Mechanical Turk (AMT) User Study

  • The AMT study scores 50 generated sequences per each of 14 unseen objects, with three users rating each sequence on a 0-to-5 scale.Ground-truth motion is evaluated only for GRAB objects because HO3D has no ground-truth whole-body grasping motions; the interface presents first- and third-person views.Source: §A.5 · p. 25, Table S1 · p. 25, Figure S2 · p. 26

B Baselines Implementation Details

The baseline experiments compare whole-body pose generation and motion infilling against adapted prior approaches and single-task variants. The implementation details specify how those baselines represent pose, contact, and motion.

B Baselines Implementation Details

B.1 Baselines to WholeGrasp-VAE

  • GrabNet-SMPLX extends GrabNet to whole-body grasping by learning compact SMPL-X parameters for global configuration, shape, and full-body pose.It uses PointNet++ for object-shape encoding, matching WholeGrasp-VAE.Source: §B.1 · p. 26
  • WholeGrasp-single predicts only body-marker positions and replaces predicted contact maps with a fixed hand-contact prior and heuristic contact loss.The heuristic encourages vertices with prior contact probability above 0.7 and distance below 2 cm to contact the object.Source: §B.1 · p. 27

B.2 Baselines to MotionFill-VAE

  • MotionFill-VAE is compared with CNN-AE, LEMO, RouteNet, and PoseNet using a shared body representation across experiments.For RouteNet and PoseNet, the scene-encoding branch is removed while the route- and pose-encoding branches are retained.Source: §B.2 · p. 27

C.1 Ablation Study on GraspPose-Opt optimization losses

The ablation examines how GraspPose-Opt loss choices affect recovered hand–object interaction. Marker fitting with foot-ground contact alone produces perceptually unrealistic interaction; collision loss mitigates interpenetration but does not yield a steady grasp.

C.1 Ablation Study on GraspPose-Opt optimization losses

  • Object collision loss mitigates hand–object interpenetration, but the optimized hand still does not grasp steadily.The comparison is against using only marker-fitting and foot-ground-contact losses.Source: §C.1 · p. 28
  • Marker fitting and foot-ground-contact losses alone recover hand poses that hardly interact with the object perceptually realistically.The hand pose is sensitive to small marker-position noise.Source: §C.1 · p. 28
  • The visualization compares optimized hand poses under different loss designs, with red object-surface areas indicating higher contact probability.The caption identifies the visualization as an illustration of the loss-design ablation in Table 2.Source: Figure S4 · p. 28

C.2 Additional Visualizations and Results on MotionFill-VAE

MotionFill-VAE generates diverse intermediate motions, including different trajectories and local motions. The supplementary visualizations also show motion sequences at multiple time lengths, while the GRAB experiments are limited to two-second infilling.

C.2 Additional Visualizations and Results on MotionFill-VAE

  • MotionFill-VAE generates diverse trajectories and diverse local motions in the qualitative results.The visualization uses the model finetuned on GRAB; Fig. S5 shows distinct trajectories and multiple sequences on the same ground-truth trajectory.Source: §C.2 · p. 28, Figure S5 · p. 29
  • The GRAB dataset’s short sequence length limits MotionFill-VAE experiments to two-second infilling.The stated condition is specific to the GRAB experiments.Source: §C.2 · p. 28
  • Visualization results show infilled sequences at 2, 4, and 6 seconds using MotionFill-VAE models trained on matching-length AMASS clips.The authors report that the model can generalize to different time lengths.Source: §C.2 · p. 29

D Limitations

The method produces realistic grasping poses and motions for most unseen test objects, but some objects remain difficult to grasp realistically. Frying pans are a challenging case because generated humans often contact the pan without grasping its handle.

D Limitations

  • For most unseen objects in the test set, the method generates realistic grasping poses and motions, but some synthesized humans fail to grasp realistically.The authors identify these as failure cases.Source: D Limitations · p. 29
  • Frying pans are especially challenging: generated humans typically contact the pan but fail to grasp its handle, producing perceptually unrealistic results.The authors associate this failure with a low perceptual score in Table S1.Source: D Limitations · p. 29, Figure S7 · p. 30
  • The supplementary time-length visualization covers 2-, 4-, and 6-second sequences trained on AMASS clips of the corresponding lengths.The authors state that the motion infilling model can be generalized to different time lengths.Source: Figure S6 · p. 30

Related papers

Loading 2112.10103v3…