PaperlayerView exact revision on arXiv

Source-linked AI summary

VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation

Zhuo Chen, Yihua Cheng, Aleš Leonardis, Hyung Jin Chang

arXiv:2610.10197v1cs.CV

TL;DR

Existing contact representations are surface-bound and often rely on hand-crafted rules that leave penetration depth uncontrolled. VolCo encodes contact in hierarchical volumes, and VolCoDiff models local configurations and their global composition. The authors report state-of-the-art grasp-generation quality, while noting limitations in controllability and scale adaptivity.

  • Problem

    Existing contact representations preserve limited hand information, remain surface-bound, and rely on penetration losses or hand-crafted rules that can yield severe penetrations.

    Source: §1 · pp. 1–2, §2 · p. 3

  • Method

    VolCo encodes contact likelihood and hand correspondences in local volumes, while VolCoDiff uses SDF-conditioned VolumeVAE and prior-guided latent diffusion to model local configurations and global hand geometry.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    On two public datasets, VolCoDiff achieves the highest stability, lowest penetration depth, and highest plausible contact area, with state-of-the-art grasp-generation quality.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The experiments support hierarchical contact volumes as a way to preserve fine-grained hand-object contact while achieving high geometric plausibility and grasp stability.

    Source: §5 · p. 10

    The method has limited controllability and scale adaptivity, prefers tighter grasps, and uses fixed volume number and scale that limit object-scale coverage.

    Source: §5 · p. 10

Abstract

from arXiv · show

Accurate contact modeling is fundamental to understanding hand-object interaction, yet existing contact representations are typically restricted to object surfaces and rely on hand-crafted rules to recover contact details, leading to severe penetrations and implausible results. To better exploit the rich detail in motion-capture data, we introduce Volumetric Contact (VolCo), a representation that expands surface points to a set of 3D volumetric grids. VolCo encodes 3D contact that allows precise hand part recovery, and is organized in an inherent hierarchy: local contact details within each volume and global hand geometry across all volumes. Our framework, VolCoDiff, employs two modules to capture local and global features following this hierarchy. For local contact details, we use a 3D variational autoencoder to model the possible hand configurations conditioned on the local object signed distance field (SDF). For global hand geometry, we design a prior-guided diffusion model that learns the distribution of compressed latent features aggregated from the volumetric grids. We evaluate our method on two benchmark datasets and demonstrate state-of-the-art performance in penetration and stability, indicating the capability to generate tight grasps with much less severe penetrations. Our code is available at https://github.com/chzh9311/volco.

1 Introduction

VolCo represents contact in volumes around object surfaces to retain fine-grained hand details and avoid reliance on unstable nearest-neighbor penetration rules. VolCoDiff models local hand configurations with a VAE and global hand geometry with diffusion, and the paper reports leading reconstruction and grasp-generation performance.

1 Introduction

  • Existing contact methods rely on shape-dependent test-time rules and nearest-neighbor penetration losses, which can get stuck in local minima and cannot control penetration depth.Slight penetration is expected at contact with rigid MANO models, but these methods can still produce severe penetrations and implausible grasps.Source: §1 · p. 1
  • The approach aims to preserve more training-data contact detail by sampling contact points around, rather than only on, object surfaces.The paper motivates hierarchically organizing these points to balance granularity against the computational cost of 3D grids.Source: §1 · p. 2
  • VolCo expands object-surface points into local volumetric grids that encode contact likelihood and hand-surface correspondences, preserving fine-grained local configurations.Its hierarchy separates local contact details from global hand geometry.Source: §1 · p. 2
  • VolumeVAE models feasible local hand configurations conditioned on object SDF values, while prior-guided diffusion learns plausible combinations of those configurations.An auxiliary hand branch and force-aware stability loss guide generation toward valid hand poses and stable grasps.Source: §1 · p. 2
  • VolCo achieves the highest precision in hand-pose reconstruction from contact and state-of-the-art grasp-generation quality on two public datasets.The grasp-generation benchmarks report the highest stability, lowest penetration depth, and highest plausible contact area.Source: §1 · pp. 2–3

2 Related Work

Prior contact representations enrich surface maps with additional hand and force information, but remain surface-bound and leave penetration-depth recovery to hand-crafted rules. VolCo is presented as a response to these information and optimization limitations.

2 Related Work

  • Contact representations evolved from surface likelihoods to maps with part labels, CSE, orientations, and forces, yet remain confined to object surfaces and lack explicit penetration-depth modeling.Recovering penetration depth is consequently left to hand-crafted rules.Source: §2 · p. 3
  • Implicit and volumetric hand-object representations capture relations in a shared feature space, but can be limited to known templates or low resolution.VolCo is rooted in known templates and aims to balance fine detail with computational cost.Source: §2 · p. 3
  • Grasp-generation methods use global-pose models or recover contact before fitting a hand, but contact-based fitting can require nearest-neighbor penetration losses and risk local minima.The paper attributes these issues to contact representations that preserve incomplete information and are non-deterministic.Source: §2 · p. 3
  • Prior methods typically use test-time adaptation with non-differentiable penetration losses, which the authors attribute to the limited information preserved by contact maps.VolCo is designed to address that information-preservation limitation.Source: §2 · p. 3

3 Methodology

VolCo encodes hand-object contact in local volumes that can be converted to hand geometry. VolCoDiff models feasible local configurations with VolumeVAE, uses prior-guided latent diffusion to compose them, and fits the generated contact to a hand pose.

3.1 Volumetric Contact Definition

  • Each local volume samples contact likelihood and CSE at grid points around an object-surface center, encoding distance-dependent contact and the nearest hand-surface correspondence.The representation discretizes the object surface into N centers and expands each into a local grid.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • VolCo recovers hand vertices by finding mesh triangles with matching CSE and aggregating corresponding volume points with weighted averages.The hand mesh and VolCo are bidirectionally convertible, allowing recovered local configurations to be concatenated into hand geometry.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Modeling Contact Prior using VolumeVAE

  • VolumeVAE models the conditional distribution of contact volumes given local object SDF values, compressing feasible local hand configurations into latent codes.Its training uses contact, CSE, and CSE-weight reconstruction losses because direct nearest-triangle lookup is non-differentiable.Source: §3.2 · p. 5

3.3 Prior-Guided Latent Diffusion

  • VolCoDiff concatenates volume-wise latent codes and an auxiliary hand latent, then uses object-conditioned diffusion to generate their plausible global combinations.The auxiliary hand branch supplies hand-pose supervision, while a consistency loss links the predicted hand back to reconstructed contact latents.Source: §3.3 · p. 5
  • A force-aware stability loss uses contact likelihood and object SDF penetration to infer contact forces and guide grasp generation.The formulation uses a spring-damper model with a 1 mm margin and slack variables allowing penetration error within 1 mm.Source: §3.3 · p. 6

3.4 Finetuning the Local Details.

  • Final fitting optimizes the hand pose to match predicted contact while penalizing large changes from the auxiliary hand-branch initialization.Contact volumes use weighted point-to-hand reconstruction, while non-contact volumes receive a repulsive loss.Source: §3.4 · p. 6

4 Experiment

Experiments evaluate contact-based hand reconstruction and grasp generation on GRAB and HO3D, where VolCo performs best on reconstruction and improves grasp stability and penetration measures. Ablations and runtime analysis examine design choices and efficiency.

4 Experiment

  • The experiments cover reconstruction from ground-truth contact and grasp generation on GRAB and HO3D, with HO3D objects unseen during GRAB training.GRAB supplies training data and one test set; HO3D tests adaptability to out-of-domain objects.Source: §4 · p. 6

4.1 Reconstruction from VolCo

  • At the same point count, changing from contact maps to volumes reduces reconstruction EPE by 40.1%, and VolCo performs best across reconstruction metrics.The authors attribute the improvement to sampling along the surface-normal direction and to point-wise correspondence.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Grasp Generation Analysis

  • On GRAB, VolCoDiff beats prior SOTA by 14.8% in SD, 32.5% in PD, and 1.6% in CA; on HO3D, by 12.3% (SD), 28.5% (PD), and 18.2% (CA).The reported GRAB results have the lowest SD and PD and largest CA; on HO3D, they indicate adaptability to unseen objects.Source: Table 2 · p. 8, Table 3 · p. 8, §4.2 · p. 8
  • Figure 5 shows prior methods producing loose or deeply penetrating grasps, while VolCo favors tighter grasps with mostly non-severe penetrations resembling motion-capture data.The comparison marks hovering fingers in purple and penetrations in red.Source: §4.2 · pp. 8–9, Figure 5 · p. 9

4.3 Ablation Study

  • VolCo improves stability over point-based contact; prior guidance lowers PD, initialization further improves stability and CA, and stability loss reduces displacement only with prior guidance.The auxiliary branch both regularizes generated VolCo for fitting with fewer penetrations and provides a better test-time initialization.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 Runtime Analysis

  • Among diffusion methods, VolCoDiff has the second-fastest inference time and second-fastest optimization time, while batched inference time barely changes from 1 to 10 samples.Inference time changes from 5.28s to 5.47s as sample count rises from 1 to 10; the point-contact baseline rises from 2.90s to 19.57s.Source: §4.4 · p. 10, Table 5 · p. 10

5 Conclusion

The conclusion presents VolCoDiff as a hierarchical approach to preserving fine-grained hand-object contact, while the stability-loss appendix explains how slack accommodates uncertainty in penetration depth.

5 Conclusion

  • VolCoDiff pairs VolumeVAE for local contact details with diffusion for global hand-configuration composition, reporting state-of-the-art geometric plausibility and stability in grasp generation.The conclusion also reports highest precision in reconstruction from contact.Source: §5 · p. 10
  • Fixed volume number and scale limit object-scale coverage, and the method currently favors tighter grasps while offering limited controllability.The authors propose controlling VolumeVAE latents and adapting volume number and scale as future work.Source: §5 · p. 10
  • The method may improve AR/VR experiences and dexterous robotic deployment, but could also foster harmful robot misuse.Source: §5 · p. 10

A Stability Loss With Slack Variables

  • Slack variables expand the stability-loss bounds so penetration depths within ±∆d are treated as correct, accommodating approximate spring-damper forces and motion-capture depth errors.The original formulation assumes precise forces; the slack-adjusted bounds address that mismatch.Source: A Stability Loss With Slack Variables · p. 14

B Model Architectures

This section gives architecture details for VolumeVAE, the VolCoDiff UNet, and the hand VAE to support reproduction.

B.1 VolumeVAE

  • VolumeVAE processes VolCo and object SDF volumes through two downsampling stages, residual MLP blocks, and a projection layer, injecting multiscale object features by channel concatenation.Its 3D convolutional downsampling uses kernel size 2 to minimize information loss from the small input dimensionality.Source: §B.1 · p. 15

B.2 VolCoDiff

  • The VolCoDiff UNet encodes flattened object SDF volumes with PointNet; cross-attention passes object and contact information to the hand branch, not vice versa.Source: §B.2 · p. 15, Figure 7 · p. 16

B.3 Hand VAE

  • The hand VAE maps flattened hand vertices to a sampled 16-dimensional latent and decodes it to 61-dimensional MANO parameters; it is trained on GRAB with reconstruction and KL-divergence losses.Training uses AdamW at learning rate 10^-4 for 100 epochs.Source: §B.3 · pp. 15–16

C Experimental Details

The reproduction experiments use a Linux machine with an Intel Xeon w7-3445 CPU, 64GB RAM, and two 4090 GPUs.

C Experimental Details

C.1 Data Pre-processing

The preprocessing uses GRAB hand-object frames and constructs separate local-volume datasets for VolumeVAE, with downsampling to reduce redundancy.

C.1 Data Pre-processing

  • GRAB preprocessing yields 323,622 closely interacting training frames, with validation and test sets of 7,757 and 1,012 frames, respectively.Training data is augmented using randomly selected axis-aligned rotations, which require no interpolation of local volumes.Source: §C.1 · p. 16
  • VolumeVAE training uses one local volume per sample; sampling 1,024 object-surface points per object initially provides 331.4M local volumes.The authors downsample because the dataset exceeds the compact VAE's capacity and contains redundant information.Source: §C.1 · pp. 16–17
  • To reduce redundancy, the pipeline downsamples frames and object points, then retains only volumes containing at least one hand vertex.Frame sampling is every 5 frames for training and every 10 for validation and testing; object points are reduced by 40% and 20%, respectively.Source: §C.1 · p. 17
  • The resulting local-volume datasets contain 7,320,315 training, 165,651 validation, and 273,315 test volumes.Source: §C.1 · p. 17

C.2 Contact-Map Based Method

The appendix specifies a contact-map diffusion baseline and explains how different correspondence representations initialize the global hand pose before reconstruction refinement.

C.2 Contact-Map Based Method

  • The contact-map baseline follows [23] but omits the hand branch, uses PointNet++ on point positions and normals, and diffuses contact and CSE values rather than latent vectors.Its diffusion loss combines contact and CSE losses, weighting the latter by 0.05 to focus more on contact.Source: §C.2 · p. 17

C.3 Initialization for Reconstruction From Contact

  • All Tab. 1 methods refine an average-pose initialization; they differ only in how they obtain the global 6D pose.Source: §C.3 · p. 17
  • Without correspondence information, the initial 6D pose is set to zero.Source: §C.3 · p. 17
  • With hand-part correspondence, initialization registers hand anchor centers to contact-likelihood-weighted centers computed from the contact map.The initial rotation and translation are obtained by point-cloud registration.Source: §C.3 · p. 17
  • With CSE correspondence, each VolCo point is matched to a weighted-average hand-surface point, and those pairs determine the initial pose by point registration.Source: §C.3 · p. 17

C.4 Contact Area Metric Definition

Contact area measures object surface area near opposing hand surfaces. The appendix also describes how the initial hand guess and VolCo guide pose refinement.

C.4 Contact Area Metric Definition

  • A vertex counts as contact when its closest hand-mesh distance is below δ and the corresponding object and hand surface normals oppose one another.Source: §C.4 · p. 17
  • Contact area sums the areas of object faces containing at least one contact vertex; larger CA indicates more extensive contact, generally desirable for stable grasps.Source: §C.4 · p. 17

D.1 Initial Guess and Refinement

  • Figure 8 shows initial guesses, predicted VolCo, and final fitted grasps for several objects from two viewpoints.Source: Figure 8 · p. 18
  • The initial hand guess is usually close to predicted VolCo, and reconstruction and repulsive losses let fitting converge to a consistent pose.The reconstruction loss is quadratic in hand-vertex locations, while the nearby initial pose supports convergence to the global minimum.Source: §D.1 · p. 18

D.2 Generated Grasps

Additional generated samples illustrate diversity and plausible grasps on GRAB and out-of-domain HO3D; each sample is shown from four viewpoints.

D.2 Generated Grasps

  • The additional randomly selected samples are presented as representative examples of diverse, highly plausible generations.Source: §D.2 · p. 18
  • Figure 9 shows further qualitative grasp samples from the GRAB dataset.Source: Figure 9 · pp. 19–20

NeurIPS Paper Checklist

The checklist reports the paper's contribution claims, reproducibility provisions, and selected experimental and societal-impact disclosures.

NeurIPS Paper Checklist

  • The paper claims a local-volume contact representation and a grasp-generation framework that produces tighter grasps with less severe penetrations.Source: NeurIPS Paper Checklist · p. 20
  • The authors state that training and testing hyperparameters are provided, the architecture is detailed in the appendix, and computational resources are described there.Hyperparameter choices mostly follow prior work.Source: NeurIPS Paper Checklist · pp. 24–26
  • The authors say code will be released upon acceptance.Source: NeurIPS Paper Checklist · p. 25
  • The main result tables report 1-sigma error bars indicating standard deviation.Source: NeurIPS Paper Checklist · p. 25
  • Experiments use public research datasets without self-collected human data; the paper reports no user study and says existing datasets are properly cited and used under their terms.Source: NeurIPS Paper Checklist · pp. 26–28
  • The authors identify a possible risk that dexterous-robot deployment could foster harmful contexts, while describing that risk as outside this paper's scope.Source: NeurIPS Paper Checklist · p. 27
Loading 2610.10197v1…