Source-linked AI summary
GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Image
Semin Kim, Haechan Shin, Jongyoo Kim
TL;DR
Monocular full-body HOI methods lack a combination of category-agnostic object reconstruction and explicit finger-level grasp optimization, leaving hand-object contact unreliable. GraspHOI separately reconstructs the body, hands, and object, aligns them in metric camera space, and jointly refines contact and articulation. Across four benchmarks and six baselines, it improves relative placement, hand pose, and contact plausibility while preserving image alignment.
Problem
Existing monocular full-body HOI methods do not combine category-agnostic object reconstruction with explicit finger-level grasp optimization, although fingers may float, penetrate, or remain generically open.
Method
GraspHOI separately reconstructs the human, hands, and object, aligns them in a shared metric camera frame, seats the object using occlusion-aware palmar correspondences, and refines arm and finger articulation through contact-aware optimization.
Results
Across four benchmarks and six baselines, GraspHOI improves relative human-object placement, hand pose, and contact plausibility while preserving image alignment.
Takeaways & Limitations
Its hand-accurate, category-agnostic outputs can serve as editable assets or keyframes toward HOI video generation.
Takeaways & Limitations
The method targets static, grasp-centric interactions with at least one contacting hand and does not model temporal motion, deformable-object physics, or sitting and leaning.
Abstract
from arXiv · showhide
Existing monocular full-body 3D human-object interaction (HOI) methods do not combine explicit finger-level grasp optimization with category-agnostic object reconstruction. Despite plausible body-object configurations, their fingers may float from or penetrate objects instead of forming a grasp. We present GraspHOI, the first framework that reconstructs a full-body 3D HOI from a single image while explicitly optimizing finger articulation against the reconstructed object. GraspHOI recovers object geometry directly, without predefined meshes or a fixed category vocabulary. It reconstructs the body, hands, and object separately, aligning them in metric camera space via depth-based registration and image-space alignment. Occlusion-aware palmar correspondences seat the object against the grasping hand, and contact-aware optimization refines arm and finger articulation to form surface contact without excessive penetration. Across four benchmarks and six baselines, GraspHOI improves relative human-object placement, hand accuracy, and contact plausibility. Full pipeline code will be released.
1. Introduction
GraspHOI addresses the missing combination of category-agnostic object reconstruction and explicit finger-level grasp optimization in single-image full-body HOI. It separately reconstructs the body, hands, and object, aligns them metrically, and refines their contact and relative configuration.
- Motivation: In-the-wild HOI assets require category-agnostic object coverage and articulated hand contact, because plausible body-object placement alone can leave fingers floating, penetrating, or generically open.Such failures reduce the usefulness of reconstructions for animation, game cutscenes, virtual reality, and downstream grasp motion.
- Research gap: Existing methods do not jointly provide category-agnostic object reconstruction and finger-level grasp optimization from one in-the-wild image.Template methods use predefined meshes, learned priors remain category-bounded, retrieval methods depend on databases, and prior contact representations do not optimize individual fingers against the surface.
- Contribution: GraspHOI reconstructs a full-body 3D HOI from one image while explicitly optimizing finger articulation against the reconstructed object without predefined meshes, category-specific models, or contact retrieval.The framework accepts higher runtime in exchange for iterative refinement of small but visible hand-object errors.
- Method: Occlusion-aware palmar correspondences seat the object against the hand before joint optimization refines arm and finger articulation, object scale, and cross-component placement.The refinement acts on reconstructed surfaces to correct contact and penetration errors after initialization.
- Evaluation: Evaluation spans four benchmarks and six recent baselines, covering object reconstruction, joint HOI recovery, hand accuracy, and grasp plausibility.The compared methods include SAM 3D Objects, HOI-TG, HDM, TeHOR, EasyHOI, and PICO.
- Method: The pipeline reconstructs the human, hands, and object independently, then aligns them in a shared metric camera frame using depth registration and image-space alignment.This decoupling keeps object-shape errors out of body and hand initialization while exposing meshes, cameras, masks, and keypoints to the optimizer.
4. Experiments
GraspHOI is evaluated across four benchmarks and multiple complementary metrics for object reconstruction, HOI recovery, hand accuracy, and grasp plausibility. It generally improves relative placement, hand accuracy, image alignment, and contact quality, while ablations show benefits from transplanted hand estimates and penetration loss.
- Evaluation Protocol: The evaluation spans ARCTIC, ProciGen-GRAB, BEHAVE, and PICO-db, comparing GraspHOI with six recent baselines across object, HOI, hand, and grasp metrics.
- Object Shape and Image Alignment: GraspHOI achieves higher sF-sc@2 cm and 2D-IoU than SAM 3D Objects, although SAM 3D Objects has lower mean sCD.
- HOI Baseline Comparison: GraspHOI has lower ARCTIC human error, higher raw-object and H+O F-scores on both datasets, and the lowest hand PA-MPJPE on ARCTIC and PGG.
- HOI Baseline Comparison: At similar average PD, GraspHOI obtains higher CPP than HOI-TG, indicating more frequent near-surface contact without excessive penetration.
- Qualitative Results: Qualitative examples show GraspHOI better preserves observed body pose, object geometry, scale, placement, and articulated grasp than PICO, HDM, TeHOR, and EasyHOI.
- Ablation Study: The WiLoR hand transplant improves wrist-relative accuracy and Procrustes-aligned articulation over SAM 3D Body’s native hands on 3,591 matched ARCTIC frames.
- Ablation Study: 5.84 cm3 to 0.93 cm3: penetration loss reduces hand-object intersection volume during joint refinement, occasionally sacrificing agreement with the input finger pose.
5. Conclusion
GraspHOI reconstructs full-body 3D HOI from a single image while separately aligning the body, hands, and object before refining arm and finger contact. Across object, joint-HOI, and grasp evaluations, it improves relative placement, hand pose, and contact plausibility while preserving image alignment.
- 5. Conclusion: GraspHOI reconstructs the object, body, and hands separately, aligning their scale and placement in a shared metric frame.This lets final refinement focus on arm and finger articulation, surface contact, and physical plausibility.
- 5. Conclusion: The final optimization refines arm and finger articulation, surface contact, and physical plausibility instead of re-estimating the entire scene.
- 5. Conclusion: Across object, joint-HOI, and grasp evaluations, GraspHOI improves relative placement, hand pose, and contact plausibility while preserving image alignment.
- 5. Conclusion: Its hand-accurate, category-agnostic outputs can serve as assets or keyframes toward HOI video generation.
Supplementary Material
The supplementary material documents implementation and evaluation details omitted from the main paper, including model conversion, conditioning, registration, contact construction, optimization, datasets, baselines, metrics, and failures.
- Supplementary Material: The supplement specifies model conversions, amodal conditioning, metric placement, contact construction, optimization schedules, dataset preparation, baseline preprocessing, and evaluation metrics.
- Supplementary Material: It also presents representative failures of the current pipeline.
S1. Additional Design Details
The method assumes at least one grasping hand and uses automatic interaction detection, geometric contact selection, and independently initialized body and hand reconstructions.
- S1. Additional Design Details: The method assumes that the target interaction contains at least one grasping hand, but it does not force every detected hand to contact the object.
- S1. Additional Design Details: HOI-DETR predicts hand-object links, which are associated with left and right SMPL-H hands and the target object mask.
- S1. Additional Design Details: Amodal overlap rejects geometrically inconsistent links before retained contacts are paired using palmar ray intersections and visible-object occlusion order.
- S1. Additional Design Details: The pipeline requires a visible object mask and an object-name prompt, while human boxes, hand boxes, and contact decisions are inferred automatically.
- S1. Additional Design Details: The same hyperparameters are used across all datasets.
- S1. Additional Design Details: SAM 3D Body supplies full-body camera-space initialization, while WiLoR supplies finger pose for the SMPL-H hand representation.
S2. Details on Our Pipeline
The pipeline completes the visible object, reconstructs and metrically registers its geometry, initializes pose, and iteratively refines placement, contact, arm, and finger configurations.
- S2. Details on Our Pipeline: The pipeline receives an RGB image, a visible-object mask, and an object name used only to prompt amodal inpainting.
- S2. Details on Our Pipeline: Relative depth from PPD is aligned to metric depth from MoGe-2 and then to the reconstructed human’s global gauge.The resulting map supports amodal masking, object scaling, FoundationPose registration, and contact-depth alignment.
- S2. Details on Our Pipeline: The occluder region is restricted to the expanded visible-object box intersected with pixels whose depths do not exceed the estimated far-object depth.
- S2. Details on Our Pipeline: The far visible-object depth is estimated from the 90th percentile of valid depths, excluding deeper background pixels from inpainting.
- S2. Details on Our Pipeline: A narrow boundary rim may be repainted, while visible pixels away from the rim remain fixed to prevent growth into distant background or invalid-depth regions.
- S2. Details on Our Pipeline: Stable Diffusion inpainting produces an amodal object cutout whose generated alpha is unioned with the visible-object support.
- S2. Details on Our Pipeline: Hunyuan3D 2.1 normally generates the textured mesh, with deterministic retries and Hunyuan3D 2.0 fallback for degenerate outputs.
- S2. Details on Our Pipeline: Metric scale uses filtered depth points and coordinate-wise median centers for the observed cloud and generated mesh.
S3. Dataset and Baseline Selection
The section constructs and characterizes the PGG and ARCTIC evaluation data, defines protocol-compatible baseline comparisons, and details metric and runtime evaluation choices.
- Dataset construction: PGG combines GRAB interactions with ProciGen rendering to preserve metric body-hand-object configurations while providing clothed appearance, calibrated monocular views, and articulated-hand ground truth.PGG is subsampled to approximately 4K frame-views, while ARCTIC is subsampled to approximately 5K frame-views.
- Dataset construction: The intentionally small image footprint of people, hands, and objects in PGG evaluates reconstruction under small visual evidence.
- Baseline protocols: Comparisons are restricted to settings where each baseline’s representation and established protocol support meaningful evaluation across object, HOI, hand, and grasp measures.The reported comparisons span ARCTIC, PGG, BEHAVE, and PICO-db, with dataset-specific omissions where required annotations are unavailable.
- Baseline protocols: TeHOR returns finite outputs for 85.5% of evaluation inputs, with failures concentrated in person detection and very small object voxelization.Finite outputs are averaged, while failed frames remain missing.
- Baseline protocols: EasyHOI’s small hand and object regions provide weak evidence for later camera and object optimization on full-body images.On PGG, its estimated field of view reaches a median of 150°, consistent with low projected-hand coverage.
- Evaluation protocol: GraspHOI typically requires 120–150 s per image on an RTX 4090, with two-hand cases taking longer.PICO is reported as an observed runtime range because its time varies with object and mesh complexity.
- Evaluation protocol: Human and object predictions are pelvis-centered without removing relative placement, while normalized object-shape evaluation independently scales objects by enclosing-sphere diameter.The common F-score threshold is 5 cm for scene evaluation; normalized object evaluation uses 0.05 of object diameter.
S4. Hand and Contact Metrics
The section defines hand-object metrics that separately measure intersection, penetration, projected hand explanation, and near-surface contact plausibility.
- Hand-object metrics: Intersection volume measures shared object and wrist-capped hand volume on a deterministic 5 mm voxel grid, reported in cm3.Evaluation includes only dataset-annotated contacting hands.
- Hand-object metrics: Penetration depth is the per-frame mean signed-distance depth of penetrating vertices in the 778-vertex hand region, reported in millimeters.Frames without penetration contribute zero.
- Hand-object metrics: Projected-hand coverage measures the fraction of the reference hand region explained by the projected contacting-hand mesh.It is interpreted alongside physical grasp metrics because it is recall-like and does not subtract object occlusion.
- Hand-object metrics: The same semantic wrist-and-joint rule selects a 778-vertex hand region for SMPL-H and SMPL-X, without surface-topology conversion during scoring.
- Contact–Penetration Plausibility: CPP measures whether hand vertices lie in a plausible near-surface band, complementing IV and PD when low penetration could instead indicate a detached hand.Its signed-distance band permits slight compression from −2 mm through 10 mm outside the object, and CPP is zero for an empty denominator.
S5. Extended Qualitative Results
Supplementary qualitative figures extend interaction comparisons and document a joint-optimization failure caused by freezing body regions outside optimized arm-hand chains.
- Extended qualitative results: Additional qualitative interactions include a comparison with TeHOR and a second comparison that omits it when valid reconstructions were unavailable.The omitted cases involve unreliable person detection or heavy occlusion.
- Extended qualitative results: A large object contacting the body can penetrate it when refinement optimizes arm-hand chains while freezing the rest of the body.
S6. Additional Object Recon Comparison
On complex GSO objects, Hunyuan3D 2.1 remains more faithful to input geometry and texture than SAM 3D Objects, whose reconstructions can depart from structure and fragment textures.
- Object reconstruction comparison: Hunyuan3D 2.1 better preserves complex input geometry and attempts to reproduce fine texture details.The comparison uses rendered views of two complex Google Scanned Objects selected to make instance fidelity visible.
- Object reconstruction comparison: SAM 3D Objects can depart from observed structure, produce fragmented textures, and sometimes generate non-watertight outputs.
S7. Failure Cases and Limitations
The pipeline’s decoupled stages can fail when upstream reconstructions or pose initialization are inaccurate, and fixed-body refinement limits correction for large body-contacting objects. The method is scoped to static, grasp-centric interactions and depends on several input and reconstruction components.
- Failure Cases: Decoupled reconstruction can leave individually plausible components geometrically incompatible with 3D contact.The paper attributes representative failures to upstream initialization and subsequent optimization stages.
- Object Reconstruction Failures: Hunyuan3D 2.1 may generate elongated, flattened, distorted objects or artificial floors, causing most full-pipeline reconstruction failures.The authors recommend inspecting meshes, retrying seeds, or screening candidates before regeneration.
- Object Pose Initialization Failures: FoundationPose failures arise from generated meshes that mismatch the input object and sensitivity to incorrect object-scale initialization.Either mismatch can produce an incorrect object pose.
- Joint Refinement: Joint refinement cannot correct the fixed body around large body-contacting objects, which may therefore penetrate the body despite plausible hand alignment.The optimized arm-hand chains receive greater emphasis while the rest of the body remains frozen.
- Scope Boundaries: GraspHOI targets static, grasp-centric interactions with at least one contacting hand and excludes temporal motion, deformable physics, and large-scale body-object support.It also requires a visible-object mask and noun prompt, while errors across initialization and reconstruction stages can propagate to the final scene.
S8. Licensing Details
The supplementary materials identify licenses for internet images and released assets used in the figures, alongside dataset sources and additional qualitative or amodal-conditioning examples.
- Image Licenses: Internet images used in retained figures are listed with their individual Creative Commons licenses and attribution details.The listed images include basketball, mariachi, pink guns, pineapple, and musical-performance examples.
- Dataset Sources: Dataset examples come from ProciGen-GRAB, PICO-db, and GSO.The GSO assets used in Figure S2 are released under CC BY 4.0.
- Supplementary Figures: Figures S6 and S7 provide extended qualitative comparisons with recent baselines, while Figure S8 shows amodal conditioning on PICO-db.Figure S7 omits TeHOR because it did not return valid reconstructions for those samples; Figure S8 retains visible object support while generating inferred occluder geometry.