PaperlayerView exact revision on arXiv

Source-linked AI summary

CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, Cewu Lu

arXiv:2012.00924v4cs.CV

TL;DR

The paper tackles the limited integration of hand-object contact modeling with joint pose estimation. It proposes CPF, a spring-based contact representation, together with A-MANO and the MIHO learning-fitting framework. Experiments report state-of-the-art performance on several reconstruction metrics and more physically plausible poses under interpenetration or disjointedness.

  • Problem

    Joint hand-object pose estimation has advanced, but simultaneous pose estimation and contact modeling remains insufficiently explored, despite contact cues being important for natural grasp recovery.

  • Method

    CPF models each contacting hand-object vertex pair as a spring-mass system, while MIHO combines this representation with learning and fitting; A-MANO constrains hand joint rotations anatomically.

  • Results

    The method achieves state-of-the-art performance on several benchmarks and produces more physically plausible hand-object poses when ground truth contains severe interpenetration or disjointedness.

  • Takeaways & Limitations

    CPF explicitly represents contact semantics and uses elastic-energy minimization to avoid interpenetration and control disjointedness while supporting precise hand-object pose recovery.

Abstract

from arXiv · show

Modeling the hand-object (HO) interaction not only requires estimation of the HO pose, but also pays attention to the contact due to their interaction. Significant progress has been made in estimating hand and object separately with deep learning methods, simultaneous HO pose estimation and contact modeling has not yet been fully explored. In this paper, we present an explicit contact representation namely Contact Potential Field (CPF), and a learning-fitting hybrid framework namely MIHO to Modeling the Interaction of Hand and Object. In CPF, we treat each contacting HO vertex pair as a spring-mass system. Hence the whole system forms a potential field with minimal elastic energy at the grasp position. Extensive experiments on the two commonly used benchmarks have demonstrated that our method can achieve state-of-the-art in several reconstruction metrics, and allow us to produce more physically plausible HO pose even when the ground-truth exhibits severe interpenetration or disjointedness. Our code is available at https://github.com/lixiny/CPF.

1. Introduction

The paper addresses joint hand-object pose and contact modeling, arguing that contact semantics should guide reconstruction rather than be treated only as a consequence of accurate pose. It introduces CPF, A-MANO, and MIHO to produce physically plausible grasps.

  • Single-image hand-object reconstruction must estimate both pose and a natural grasp configuration for applications including VR/AR, teleoperation, and grasping.
  • Existing joint-estimation methods commonly treat contact as a consequence of correct pose, although perfect pose recovery is impractical and contact provides cues for accurate, natural grasps.
  • Prior contact approaches use distance-based attraction and repulsion or physics simulators, but integrating contact modeling into pose estimation while preserving contact semantics remains open.
  • CPF represents contacting hand-object vertex affinities with attractive and repulsive forces, reaching equilibrium when paired vertices form the grasp contact.
  • Each contact vertex pair becomes a spring-mass system whose minimum elastic energy defines the grasp, assigning per-vertex semantics while controlling interpenetration and disjointedness.
  • A-MANO constrains joint rotations within a twist-splay-bend frame, while MIHO combines CPF with a learning-fitting framework; the method reports state-of-the-art performance on several benchmarks.

2. Related Work

Related work spans hand and object reconstruction, joint hand-object pose estimation, and contact heuristics. Existing contact methods include physical simulation and multi-point formulations, motivating comparison with distance-based baselines.

  • 3D Hand Reconstruction: 3D hand reconstruction commonly uses MANO, but high-dimensional joint rotation regression can produce anatomically abnormal poses.
  • Hand-object Pose Estimation: Hand-object pose estimation evolved from separate hand or object estimation toward learning-based joint estimation frameworks.
  • Contact Heuristic: Contact heuristics have used shape-specific priors, predefined grasps, physical simulators, and multi-point contact formulations across interaction tasks.
  • Contact Heuristic: The paper treats prior methods [1] and [25] as strong contact-heuristic baselines for comparison.

3. Anatomically Constrained A-MANO

A-MANO retains MANO’s parametric skinning formulation while constraining joint rotations anatomically and representing hand contact through regional anchors.

  • A-MANO uses MANO’s pose parameters θ ∈ R^15×3 and shape parameters β ∈ R^10 for an articulated hand mesh.
  • Its main changes are restricting joint rotation axes and angles within a twist-splay-bend frame and adding anchors to subdivided hand regions.
  • The twist-splay-bend frame constrains unwanted rotation axes because most hand joints primarily rotate about a bend axis.
  • Anchors interpolate representative points in hand subregions, reducing the number of hand-object vertex pairs used for spring attachments.
  • The hand palm is divided into 17 subregions, with up to four anchors interpolated per region while backside vertices are ignored.

4. Contact Potential Field

CPF models hand-object contact as interacting attractive and repulsive springs, forming a potential field whose grasp configuration minimizes elastic energy. Hand anchors and distance-based heuristics make this field learnable and computationally manageable.

  • Spring-Mass Contact Model: A contact is modeled as a spring-mass system whose elastic potential energy is zero at rest and increases when stretched or compressed.The spring endpoints represent hand and object vertices, while elasticity represents their contact affinity.
  • Attractive and Repulsive Springs: Attractive springs pull affiliated hand and object vertices together, whereas repulsive springs push nearby hand vertices away along object normals.Attraction is tied to a specific vertex affinity; repulsion acts only locally and is affinity-agnostic.
  • Energy Formulation: CPF estimates hand-object pose by minimizing the combined elastic energy of attractive and repulsive springs.The field is constructed from all springs, and grasping is formulated as elastic-energy minimization.
  • Efficient Contact Representation: Hand vertices are simplified to representative subregion anchors, reducing the number of hand-object vertex pairs used by the field.For attractive springs, each hand vertex is replaced by its closest anchor; repulsion is zero beyond trpl = 20 mm.

5. Hybrid Framework – MIHO

MIHO combines coarse pose estimation, learned CPF recovery, and energy-based fitting to refine hand-object meshes. PiCR predicts contact structure and spring elasticities, while GeO optimizes pose under contact and anatomical constraints.

  • MIHO Pipeline: MIHO processes an RGB image through HoNet, PiCR, and GeO to obtain refined hand-object meshes.HoNet predicts coarse meshes, PiCR recovers the CPF and elastic energy, and GeO minimizes that energy for refinement.
  • HoNet: HoNet predicts coarse hand-object poses using object and wrist 6D poses, MANO pose components, and hand shape parameters.The baseline output contains 37 coefficients and places the meshes into the camera frame.
  • PiCR: PiCR predicts Vertex Contact, Contact Region, and Anchor Elasticity to construct the CPF from coarse hand-object meshes.These outputs identify contacting object vertices, their likely hand subregions, and attractive-spring elasticities.
  • PiCR Architecture: PiCR uses image features, projected object vertices, and a PointNet encoder before sequential multilayer-perceptron heads produce its three contact outputs.The feature pipeline aligns projected object vertices with image features through bilinear sampling and appends root-relative depth.
  • GeO: GeO refines object pose, hand joint rotations, and wrist pose by optimizing CPF elastic energy together with anatomical and offset costs.The anatomical term penalizes unwanted twist, splay, and excessive bend, while the offset term discourages deviation from the initial estimate.

6. Experiments and Results

Experiments on FHB and HO3D evaluate reconstruction and grasp quality jointly, using quantitative comparisons, qualitative examples, and ablations of CPF and anatomical constraints. MIHO improves physical interaction quality, while dataset scope and ground-truth quality affect interpretation of reconstruction scores.

  • Datasets: FHB and HO3D are the evaluated datasets, with HO3D comparisons restricted to suitable samples and customized splits or augmentation where required.The method requires known object models and stable grasping configurations, excluding many HO3Dv2 test samples.
  • Metrics: The evaluation combines hand and object reconstruction errors with penetration depth, solid intersection volume, and disjointedness distance.The five metrics jointly assess pose estimation and grasp quality rather than relying on a single measure.
  • Comparison with State-of-the-Arts: On FHB, lower vertex error is not treated as definitive because the ground-truth and prior results exhibit substantial interpenetration and disjointedness.The paper therefore emphasizes physical plausibility alongside reconstruction metrics.
  • Comparison with State-of-the-Arts: 3.71 mm lower penetration depth, 9.34 cm3 lower solid intersection volume, and 14.99 mm lower disjointedness distance are reported for MIHO versus [24] on FHB.These physical-quality gains come with 2.03 mm and 0.51 mm hand and object MPVPE costs, respectively.
  • Ablation Study: CPF surpasses simple contact baselines on most metrics and reduces average time per iteration by 46% relative to ObMan Contact.Directly optimizing disjointedness gives those baselines better resistance on that metric, while CPF learns contact semantics for more valid grasps.
  • Ablation Study: Removing repulsive springs increases penetration depth and solid intersection volume, while anatomical constraints prevent abnormality during optimization.Attractive springs can still push an embedded hand outward, but repulsive springs improve physical quality further.

7. Conclusion

The paper presents CPF and MIHO for hand-object interaction, recovering precise poses while addressing interpenetration, disjointedness, and abnormal hand pose. It also identifies future work on object-agnostic CPF.

  • CPF represents hand-object contact, while MIHO combines learning and fitting to model their interaction.
  • The method recovers precise hand-object pose while avoiding interpenetration, controlling disjointedness, and preventing abnormal hand configurations.
  • The supplemental material covers A-MANO, spring elasticity, HO3D analysis, additional experiments, and qualitative results.

A.1. Derivation of Twist-splay-bend Frame.

The twist-splay-bend frame decomposes each articulation’s rotation into anatomically motivated twist, splay, and bend axes. The section also describes how hand regions, control points, and differentiable surface anchors are constructed.

  • Twist-splay-bend frame: A-MANO constrains each articulation’s axis-angle rotation using twist, splay, and bend components in a Euclidean frame.The frame is designed to reduce abnormal hand poses during optimization.
  • Twist-splay-bend frame: The frame is derived by computing a twist axis, then bend and splay axes through cross products with MANO’s y axis.
  • Hand subregions and control points: MANO vertices are assigned to 17 anatomically defined hand subregions using numbered links and control points.Most links receive one midpoint control point, while selected links receive two control points at upper and lower thirds.
  • Anchor construction: Anchors are required to lie on the hand surface, distribute uniformly within their regions, and remain differentiable from hand vertices.
  • Anchor construction: Each anchor is computed as a barycentric-style linear combination of three face vertices, allowing anchor losses to backpropagate to the hand mesh.The weights and face are precomputed and reused during optimization.
  • Anchor construction: Control points generate anchors by ray-surface intersection for non-tip regions, while tip regions use three anchors per control point.The Carpal region uses only three of its five control points for anchor derivation.

B.1. Elastic Energy Analysis

The elastic-energy analysis models hand-object vertex affinities through attraction and repulsion around the object surface. Spring annotations use distance-based rules, with valid contacts restricted to a 20 mm range.

  • Elastic energy: The elastic energy is defined between a hand-surface vertex and an object-surface vertex, whose object normal supplies the repulsion direction.
  • Elastic energy: The signed projection of the hand-to-object offset distinguishes hand vertices outside the object from those inside it.The projection is positive outside and negative inside.
  • Elastic energy: Exponential repulsive energy increases as penetration grows, providing an optimizer heuristic with stronger gradients for deeper intersections.
  • Elastic energy: For an intersecting hand vertex, attraction and repulsion point toward the object surface; outside the object, opposing forces create a nearby balance point.
  • Attractive spring annotation: Anchor-vertex pairs beyond 20 mm are labeled invalid with zero attractive stiffness, while closer pairs receive inverse-proportional stiffness.The scale factor for the distance rule is 20 mm.
  • Attractive spring annotation: The attractive-stiffness function is not unique, provided it equals 1 at zero distance, decreases inversely through 20 mm, and remains bounded between 0 and 1.The cosine choice is motivated by smoothness.

C.1. Analysis and Selection

The HO3Dv2 testing set is filtered to retain samples suitable for MIHO’s known-object and stable-grasp requirements. The section also documents the selected evaluation set, augmentation, and module implementation details.

  • Selection criteria: HO3Dv2 samples with unknown object requirements or unstable interactions are excluded because they can make GeO-based evaluation unsuitable.Pitcher grasps are removed, and sliding interactions are excluded because they may mislead optimization.
  • Selection criteria: 6076 samples are selected from the HO3Dv2 test set for MIHO evaluation.
  • Dataset augmentation: HO3Dv1 training data are augmented by randomizing hand-object poses and fitting additional stable grasps before rendering synthetic images.The augmentation operates in the object canonical coordinate system.
  • Implementation: The supplemental section includes an objective combining LA, Erpl., and Lanat.
  • Implementation: The implementation uses HoNet with a ResNet-18 backbone, PiCR with stacked hourglass and PointNet components, and GeO as a nonlinear fitting module.

D.2. Ablation Study

The ablations examine repulsion strength, hand-pose parameterization, and anatomical constraints. They show trade-offs between reducing interpenetration, preserving reconstruction quality, and avoiding visually abnormal poses.

  • The Impact of the krpl: Larger krpl reduces solid interpenetration volume but can separate attraction, worsening hand and object reconstruction metrics.The default krpl is 1 × 10^-3; tested values range from 0.2 to 8.0 × 10^-3.
  • A-MANO with PCA Pose: Full MANO with 15 relative joint rotations achieves lower hand MPVPE than PCA MANO.The paper attributes this to PCA MANO's tendency toward the mean flat pose, whereas full MANO provides greater pose flexibility.
  • A-MANO with PCA Pose: Fitting 15 joint rotations in so(3) introduces 45 degrees of freedom and is less stable against pose abnormality.The authors therefore combine full joint rotations with anatomical constraints.
  • Unwanted Twist Correction: A-MANO constrains the thumb's twist component and produces a more visually pleasing result when ground-truth contains an unwanted twist.This example comes from the HO3Dv1 testing set.

E. More Qualitative Results

MIHO is qualitatively evaluated on FHB and HO3D examples, with the final row of the figure showing failure cases. Ground truth is unavailable for HO3Dv2−, limiting direct qualitative verification there.

  • More Qualitative Results: Figure 16 presents qualitative MIHO results on the FHB and HO3D datasets.The figure includes examples from FHB, HO3Dv1, and HO3Dv2−.
  • More Qualitative Results: Ground truth is unavailable for HO3Dv2−, so those qualitative examples lack direct ground-truth comparison.
  • More Qualitative Results: The final row of Figure 16 shows failure cases.
Loading 2012.00924v4…