Source-linked AI summary
StableGrasp: Reconstructing Physically Stable Human Hand Grasps from Single Images
Han Jiang, Etienne Vouga, Qixing Huang, Georgios Pavlakos
TL;DR
Image-based grasp reconstruction must account for more than the visually constrained hand pose: stability also depends on inferring the torques that maintain contact. StableGrasp jointly optimizes hand pose and a PD control target through kinetic-energy minimization under visual and geometric constraints. Experiments report improved stability over baselines while preserving agreement with the input image.
Problem
A simulation-ready hand and object do not suffice to assess stability, because contact forces depend on unobserved joint torques that must also be inferred.
Source: §3 · p. 4Method
Results
Takeaways & Limitations
StableGrasp treats stable grasp reconstruction as requiring a control target, not only a visually plausible hand pose.
Source: §1 · p. 2, §3 · p. 4Object geometry remains fixed during local pose optimization, making results dependent on the initial reconstruction; object reconstruction errors accounted for 27 of 48 failures among 362 examples.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Reconstructing a physically stable human grasp from a single RGB image is challenging because physically modeling grasps is itself difficult, and the problem requires estimating not only a visually constrained hand pose but also a control target that stabilizes the grasp. Existing methods either model only visual hand geometry without considering physics, or rely on less plausible physical modeling, which limits the physical validity of the resulting grasps. In this paper, we present StableGrasp, a differentiable simulation-based optimization framework that explicitly separates the visual hand pose from the control target that determines the grasping forces. Our method jointly optimizes hand geometry and control by minimizing the kinetic energy of the grasp in a differentiable simulator, while regularizing the hand geometry to preserve visual consistency and geometric plausibility. The reconstructed grasps are substantially more stable under rigorous physical simulation, while remaining visually consistent with the input images and geometrically plausible. Experiments show that our approach produces far more stable grasps than alternative hand-control strategies, benefiting visual-only grasp reconstruction pipelines by turning their outputs into physically stable grasps.
1 INTRODUCTION
Image-based grasp reconstruction is useful across applications, but visual plausibility does not ensure a grasp can be executed and maintained. StableGrasp treats stability as a control problem, jointly reconstructing the grasp and a control target; it reports greater physical stability than baseline hand-control methods while preserving visual consistency and geometric plausibility.
1 INTRODUCTION
- Image-based grasp reconstruction supports applications including digital human modeling, AR/VR, and robot learning, but visual plausibility alone does not establish physical executability.A grasp can look convincing in 3D while remaining impossible to execute or physically unstable.Source: §1 · pp. 1–2
- Force-closure analysis considers whether static contact forces could hold an object against gravity, but does not establish whether a hand can pose and actuate itself to maintain those forces.The resulting grasp may still be impossible if the hand cannot produce and maintain the required contact forces.Source: §1 · p. 2
- StableGrasp frames stable grasp reconstruction as a control problem that must infer joint torques, rather than optimizing visual or geometric fit alone.The method evaluates stability through simulation of coupled hand-object dynamics.Source: §1 · p. 2
- StableGrasp substantially improves physical stability over heuristic and learning-based hand-control baselines while producing visually consistent, geometrically plausible grasps.The evaluation uses DexYCB, HO3D, and CORe50.Source: §1 · p. 2
- The framework uses kinetic-energy-driven optimization to preserve visual and geometric fidelity while recovering a grasp configuration and a control target whose induced torques stabilize it in simulation.The paper formulates stability through forward simulation of the full hand-object system.Source: §1 · p. 2
2 RELATED WORK
Prior hand-object reconstruction methods focus on visual plausibility, while physics-aware approaches vary in whether they model an actuated hand and two-way hand-object dynamics. StableGrasp models that coupling with a fixed control target for the static reconstruction setting.
2 RELATED WORK
- Hand-object reconstruction methods use interaction constraints, contact-aware refinement, implicit representations, unified estimation, and object-geometry reconstruction, but generally target visual plausibility.Many represent the hand with MANO, a parametric pose-and-shape model mapped to a skinned mesh.Source: §2 · pp. 2–3
- Static-hand approaches assess object stability from contact forces, but that stability does not show whether an actuated hand can generate or maintain those forces over time.Prior work also uses force-closure criteria, simulation objectives, learned simulation surrogates, explicit fingertip forces, or contact priors.Source: §2 · p. 3
- StableGrasp reconstructs both a contact-consistent hand pose and a control target whose induced joint torques realize the contacts and maintain stability under forward simulation.This models full two-way coupling between the actuated hand and object.Source: §2 · p. 3
- Some prior methods model strict two-way coupling, but D-Grasp and Grasp-and-Lift focus on dynamic settings where control can vary over time.StableGrasp instead represents the unobserved control with one fixed PD target in a static setting.Source: §2 · p. 3
3 METHODOLOGY
StableGrasp reconstructs an image-consistent hand and object while inferring a control target that keeps the coupled system stationary in simulation. Its two-phase refinement uses simulated kinetic energy for stability and visual and geometric regularization to preserve plausible hand geometry.
3 METHODOLOGY
- The goal is to recover the hand, object, and control target so the grasp remains visually consistent and holds both hand and object in place under simulated frictional contact.Stability is framed as balancing the system so neither hand nor object moves over time.Source: §3 · pp. 3–4
- The hand is represented visually with MANO and a simulation-ready articulated rigid-body proxy, while the object is treated as a rigid body with known uniform density.The method uses MuJoCo to simulate the coupled system from an initial configuration with zero initial configurational velocity.Source: §3 · p. 4
- A separate target hand pose drives a PD controller, avoiding reliance on fixed torques that may let small force or torque residuals accumulate into motion.The target is a pose the hand attempts to reach through joint actuation, and it may intersect the object.Source: §3 · p. 4
3.1 PIPELINE
- Visual initialization estimates the hand and object from the image; refinement freezes the object mesh and hand shape while jointly optimizing the visual pose and control target.HORT supplies the initial hand estimates and object point cloud, which is meshed and convexly decomposed for simulation.Source: §3.1 · p. 5
3.2 DIFFERENTIABLE PHYSICS-BASED OPTIMIZATION
- The stability objective sums simulated hand–object kinetic energy over the first To steps, a short horizon chosen because instability emerges early and enables efficient backpropagation.The generalized mass matrix includes hand and object mass and inertia tensors.Source: §3.2 · p. 5
- Visual and geometric regularization combines image-joint reprojection, interpenetration, adhesion for exterior non-palm points, and a joint-limit term.Palm points are excluded from the adhesive term because the rigid proxy palm does not contribute to stability like a deformable human palm.Source: §3.2 · pp. 5–6
- Optimization first minimizes visual regularization to reduce severe penetrations and establish plausible contact, then activates kinetic energy and optimizes the full objective.The loss bundles are normalized at the start of the second phase so their weights, rather than raw loss magnitudes, control relative influence.Source: §3.2 · p. 6
4 EXPERIMENTS
The experiments evaluate StableGrasp on selected DexYCB, HO3D, and CORe50 images, with stated assumptions and simulation settings. They assess physical motion and visual consistency using separate metrics.
4.1 EXPERIMENTAL SETUP
- The benchmark contains 242 DexYCB images, 100 HO3D frames, and 20 manually selected CORe50 images, with the first two drawn from HORT test splits.DexYCB samples are final frames with clear hand–object visibility, while HO3D samples near sequence ends can depict ongoing manipulation.Source: §4.1 · p. 6
- For CORe50, object density is set to 1000 kg/m3, and gravity is approximated as camera-down at 9.8 m/s2; DexYCB and HO3D use YCB density statistics.The CORe50 density is used because that dataset provides neither 3D annotations nor object masses.Source: §4.1 · p. 6
- Refinement differentiates through 20 simulation steps of 2 ms and early-stops after 200 iterations without sufficient loss improvement; evaluation runs 5 seconds at 0.2 ms steps.Hand proxies use a box palm and capsule finger links, and evaluation uses unified PD gains.Source: §4.1 · p. 6
4.2 METRICS
- Physical stability is measured by hand-joint displacement, object center-of-mass translation and rotation, success within a 1 cm object-position threshold, and mean drop time among failures.Success Rate counts grasps whose object center stays within 1 cm of its initial position throughout simulation.Source: §4.2 · p. 6
- Visual stability compares initial and final image projections using object-mask IoU and hand PCK, the fraction of 21 joints displaced by less than τ in normalized image coordinates.Source: §4.2 · p. 6
(c) HO3D
StableGrasp produces more stable simulated grasps than both baselines across three datasets while preserving visual consistency. Warm-up improves outcomes, and most pose changes occur before physics-based refinement.
4.3.1 QUALITATIVE RESULTS
- Qualitative comparisons show object drops or visible motion with PPO and Pinching, while StableGrasp preserves grasps and flexibly adjusts its control target.Even when baselines retain objects, their larger shifts remain noticeable; Pinching prescribes inward displacement.Source: Table 1 · p. 7, §4.3.1 · p. 8
(c) HO3D
- For HO3D, hand reconstruction quality is compared against annotations, which may not reflect grasp dynamics or interpenetration.Stable contact with the rigid proxy and reconstructed object can require pose adjustments relative to these visual references.Source: (c) HO3D · p. 7
4.3 BASELINE COMPARISONS
- The comparisons initialize each control baseline from the visual warm-up pose; PPO trains instance-specific policies, while heuristic pinching displaces fingers inward.PPO can update finger PD targets online during evaluation; heuristic pinching uses a PD controller for a target pose.Source: §4.3 · p. 7
4.3.2 QUANTITATIVE RESULTS
- StableGrasp achieves higher success rates, lower hand and object motion, and better image-space stability than both baselines across all three datasets.Table 2 evaluates 357 common examples using a 1 cm displacement threshold; Success Rate uses all examples, while motion, IoU, and PCK compare successful grasps.Source: §4.3.2 · p. 8, Table 2 · p. 8
- Most hand-pose changes occur during visual warm-up, while physics-based refinement changes the pose only marginally.The authors attribute differences from annotations to the simulation-ready hand proxy, reconstructed object, and dataset reference; visually faithful results may therefore differ from annotated poses.Source: §4.3.2 · p. 8, Table 3 · p. 8
4.4 ABLATION STUDIES
- Removing visual warm-up reduces success from 90.70% to 76.74% and increases hand motion on both datasets.The ablation compares 600 warm-up plus 2,000 physics-based iterations with 2,600 physics-based iterations without warm-up.Source: §4.4 · pp. 8–9, Table 4 · p. 9
- Each warm-up regularizer addresses a distinct failure: without Lobs the grasp drifts, without Llim a joint twists, without Lint the hand penetrates, and without Ladh contact is insufficient.Figure 4 compares the complete warm-up with individual regularizer ablations.Source: Figure 4 · p. 9, §4.4 · p. 9
5 CONCLUSION
StableGrasp reconstructs stable grasps from single RGB images by jointly optimizing the visual hand pose and a PD control target. It improves stability over baselines while preserving agreement with the input image, but its local pose optimization depends on the initial object reconstruction.
5 CONCLUSION
- StableGrasp jointly optimizes the visual hand pose and a PD control target by minimizing coupled hand–object kinetic energy while enforcing visual and geometric constraints.The method reconstructs stable human grasps from single RGB images.Source: §5 · p. 9
- Experiments show improved grasp stability over baselines while preserving agreement with the input image.Source: §5 · p. 9
- Local pose optimization keeps object geometry fixed, so it depends on the initial reconstruction; alpha-shape meshing can distort thin objects or fill concave and hollow features.Object reconstruction errors account for 27 of 48 failures across 362 tested examples; the rigid hand proxy also omits local tissue deformation.Source: §5 · p. 9