PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions

Zikai Wang, Zhilu Zhang, Yiqing Wang, Hui Li, Wangmeng Zuo

arXiv:2603.25791v1cs.CV

TL;DR

ArtHOI targets the gap between rigid-object HOI methods and articulated-object reconstruction methods requiring scans or multiview videos. It integrates and optimizes foundation-model priors with ASR and MLLM-guided alignment, and experiments report robust reconstruction across datasets, including in-the-wild videos.

  • Problem

    Existing HOI methods largely target rigid objects, while articulated-object reconstruction typically requires pre-scanned geometry or multiview videos, leaving monocular articulated HOI reconstruction underexplored.

  • Method

    ArtHOI integrates multiple foundation-model priors through optimization, using ASR for metric scale and pose and MLLM-inferred contact information to constrain hand-object alignment.

  • Results

    ArtHOI reconstructs physically plausible 4D hand-object interactions across two datasets, outperforms prior approaches relying on pre-scanned articulated objects, and generalizes to in-the-wild Internet videos.

  • Takeaways & Limitations

    The model-free framework supports robust articulated hand-object reconstruction across diverse objects, interactions, and controlled or in-the-wild videos.

  • Takeaways & Limitations

    FoundationPose performance degrades in this setting because generated meshes and inaccurate depth are inconsistent, causing poor or unstable predictions.

Abstract

from arXiv · show

Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.

1. Introduction

ArtHOI addresses monocular 4D reconstruction of hand-articulated-object interactions by refining foundation-model priors through optimization. It introduces scale-and-pose refinement, contact-constrained hand-object alignment, and evaluation on challenging benchmarks.

  • Research gap: Existing HOI methods largely assume rigid objects, while articulated-object reconstruction often requires pre-scanned geometry or multi-view videos.These requirements limit applicability to unconstrained, in-the-wild scenarios.
  • Research gap: The task is ill-posed because monocular videos provide limited visual cues and frequent occlusions.Foundation models offer geometric, motion, semantic, and hand-mesh priors that can help address these ambiguities.
  • Challenges: Naively combining foundation-model predictions can produce physically implausible reconstructions because predictions may be inaccurate or lack metric and physical grounding.Typical failures include incorrect world-space scale and pose, hand-object interpenetration, and disjointed contact.
  • ArtHOI framework: ArtHOI integrates and refines multiple foundation-model priors to reconstruct 4D hand-articulated-object interactions from monocular videos.Its optimization explicitly addresses inconsistencies and mismatches among priors.
  • ArtHOI framework: Adaptive Sampling Refinement estimates an articulated object's metric scale and 6-DoF pose, grounding its normalized mesh in world space for motion reconstruction.The method then uses MLLM-inferred frame-wise contact states and contacting fingers as constraints for jointly refining object scale and hand pose.
  • Evaluation: ArtHOI evaluates preprocessing, canonical mesh reconstruction, part-wise motion reconstruction, and hand-object alignment across existing and newly introduced datasets.The evaluation includes ArtHOI-RGBD, ArtHOI-Wild, and RSRD, with experiments reporting robustness across diverse objects and interactions.

2. Related Works

Related work has addressed monocular HOI and articulated-object reconstruction using increasingly general priors, but existing approaches remain constrained by occlusion, depth ambiguity, input requirements, or limited data domains.

  • Hand-object interaction reconstruction: Monocular HOI reconstruction is difficult because severe occlusions and depth ambiguities obscure hand-object structure.Earlier methods addressed these challenges with known object templates or pretraining on small-scale 3D object datasets.
  • Articulated-object reconstruction: Articulated-object reconstruction methods commonly require 3D point clouds or multiview observations, restricting datasets and weakening generalization to in-the-wild data.Recent work has begun reconstructing articulated objects from monocular RGB video captured in the wild.

3. Method

ArtHOI reconstructs monocular 4D hand-articulated-object interactions by combining foundation-model priors with optimization that resolves scale, motion, and hand-object alignment inconsistencies.

  • Pipeline: ArtHOI preprocesses videos with foundation models, reconstructs a normalized canonical object mesh, estimates part-wise motion, and aligns reconstructed hands and objects.Preprocessing extracts masks, depth, and camera parameters; inpainting restores hand-occluded object regions before image-to-3D reconstruction.
  • Metric Pose and Scale Optimization of Object: FoundationPose predictions can become unstable when generated mesh geometry and estimated depth are inconsistent, motivating ASR’s joint refinement of scale and pose.The method is designed for normalized generated meshes and noisy depth rather than accurate metric-scaled ground-truth geometry.
  • Metric Pose and Scale Optimization of Object: ASR converts the normalized object mesh into a metric world-space reconstruction by optimizing scale and 6-DoF pose against depth, masks, and pose hypotheses.It begins with a coarse depth-based scale estimate, samples candidate scales adaptively, and evaluates rendered silhouettes against object masks.
  • MLLM-guided Articulated HOI Alignment: Hand reconstruction uses WiLoR to estimate MANO-based 4D hands, while contact and motion regularization terms reduce spatial mismatch and temporally smooth hand parameters.The contact loss minimizes fingertip distance to the closest point on the object mesh; SLERP fills missing or unreliable hand predictions.
  • MLLM-guided Articulated HOI Alignment: MLLM-guided alignment infers frame-wise contact states and contacting fingers, then uses these constraints to optimize object scale followed by hand pose and global transformations.The structured prompts incorporate camera perspective, neighboring RGB frames, and colorized depth maps for contact reasoning.

4. Experiments

Experiments evaluate ArtHOI on newly collected RGBD and in-the-wild videos, RSRD, and ARCTIC, measuring articulated-object reconstruction, HOI alignment, and contact reasoning. ArtHOI achieves consistently strong reconstruction and alignment, including settings where scan-dependent methods cannot operate.

  • Datasets: ArtHOI-RGBD contains five RealSense sequences, while ArtHOI-Wild contains eight internet and smartphone videos; evaluation also uses nine RSRD videos and an ARCTIC subset.RGBD videos are captured at 1280 × 720 and 30 FPS with metric depth.
  • Evaluation Settings: Evaluation measures articulated-object reconstruction with CD, MSSD, and F-scores, and hand-object alignment with the Collision-Contact (Co2) score.Lower CD/MSSD and Co2 are better, while higher F-scores are better.
  • Quantitative Results: ArtHOI achieves the lowest reconstruction errors across all five ArtHOI-RGBD sequences and results comparable to RSRD on the RSRD dataset without pre-scanning.It also handles ArtHOI-Wild and ARCTIC videos, whereas RSRD fails without surrounding scans.
  • Quantitative Results: MLLM-guided alignment produces the lowest Co2 scores and visually plausible, well-aligned 4D reconstructions compared with unaligned, RSRD, and heuristic-contact strategies.Removing MLLM-derived contact cues consistently degrades reconstruction accuracy.
  • Qualitative Results: Qualitative results show coherent articulated geometry, motion, and hand alignment under severe hand-part and part-part occlusions, including in-the-wild videos.RSRD struggles with heavy occlusions and requires a pre-scanned canonical object.
  • Ablation Study: ASR achieves the highest silhouette IoU and optimization success rates across all videos, outperforming direct FoundationPose application and Any6D in the ablation.FoundationPose often fails under mesh-depth inconsistency, while Any6D struggles with metric-scale recovery.

5. Conclusion

ArtHOI reconstructs 4D hand-object interactions with articulated objects from monocular videos by combining foundation-model priors with optimization strategies. Experiments show model-free reconstruction outperforms prior approaches using pre-scanned articulated objects and generalizes to in-the-wild videos.

  • ArtHOI reconstructs 4D hand-object interactions with articulated objects from monocular videos.
  • The approach unifies multiple foundation-model priors through optimization strategies that handle cross-prior inconsistencies and estimation noise.
  • Experiments on two datasets show that the model-free method outperforms approaches relying on pre-scanned articulated objects and generalizes to Internet videos.

Supplementary Material

The supplementary material describes a staged MLLM prompting pipeline for perspective detection, hand mapping, and contact reasoning. It uses RGB, depth, and neighboring frames to infer hand-specific contact information.

  • Prompting pipeline: Hand mapping assigns visible hands to left or right labels using spatial and thumb cues for first-person views and camera-body relationships for third-person views.
  • Prompting pipeline: The contact prompt analyzes neighboring RGB frames and colorized depth maps to infer binary contact states and contacting fingers for each hand and frame.

A.1. Coarse Metric Scale Estimation of Object

Coarse metric scale estimation aligns a normalized canonical object with metric depth in world space. The method cleans the depth-derived point cloud and estimates scale from image-plane extents.

  • Metric depth maps are back-projected into 3D using camera intrinsics and the object mask after mask erosion and statistical outlier removal.
  • The method compares bounding-box extents of the normalized canonical object and the back-projected depth point cloud.
  • Coarse scale uses the maximum x- and y-axis extent ratio, excluding z because visible-surface depth is noisier and less reliable.

A.2. Object Part Segmentation

The supplementary material details object-part segmentation and MLLM-based contact reasoning within ArtHOI. The pipeline uses staged perspective and hand mapping, conservative contact inference, structured outputs, and concurrent processing during reconstruction.

  • Object Part Segmentation: PartField features and agglomerative clustering produce vertex-group labels for articulated-object part segmentation.
  • Evaluation: MLLM contact reasoning remains robust on in-the-wild videos, whereas a rule-based mask-intersection heuristic degrades notably outside the controlled RSRD dataset.
  • MLLM Contact Reasoning: Neighboring RGB frames and colorized depth maps are concatenated so the MLLM can infer spatio-temporally consistent hand identities, contact flags, and contacting fingers in structured JSON.
  • MLLM Contact Reasoning: The MLLM pipeline uses three stages: perspective detection, perspective-specific hand mapping, and frame-wise contact reasoning.
  • MLLM Contact Reasoning: The contact prompt marks uncertain cases as no-contact to suppress false positives, which are more common than false negatives in real-world videos.
  • Computational Performance: Part-wise motion reconstruction takes roughly 30 minutes, while canonical scale and pose optimization takes less than 2 minutes and final hand-object alignment up to 5 minutes.

C. Additional Results

The contact-reasoning procedure analyzes horizontally merged RGB-D frames to determine hand visibility and solid contact with the articulated object. It separately evaluates the left and right hands while accounting for occlusion and depth cues.

  • Contact Reasoning: A Stage 3 contact-reasoning prompt is appended after Stage 2.
  • Contact Reasoning: Contact reasoning processes K horizontally merged frames, with RGB frames in the top row and depth frames below, where blue denotes near and red denotes far.Black bars separate frames in the merged image.
  • Contact Reasoning: The procedure analyzes the left and right hands separately after they are identified in an earlier step.
  • Contact Reasoning: Visibility checks skip a hand only when it is not visible, while still requiring partially occluded hands around the articulated object to be identified.
  • Contact Reasoning: Object-contact estimation uses RGB and depth to distinguish solid hand-object contact from mere proximity or incidental contact.Hands clearly distant from the object, or merely contacting it without solid-state interaction, receive contact:false.

C. Depth Map Verification (Critical Phase):

The contact-verification procedure combines RGB and depth evidence to distinguish true physical contact from proximity, then identifies engaged fingers and enforces temporally logical decisions.

  • C. Depth Map Verification (Critical Phase):: Depth-map verification marks contact true only when hand and object depth values merge without discontinuity at the interaction point.A sharp depth edge or color contrast indicates false contact.
  • C. Depth Map Verification (Critical Phase):: When contact is true, the procedure identifies specific engaged fingers and excludes fingers that are occluded or ambiguous.
  • C. Depth Map Verification (Critical Phase):: The final decision reviews frame-wise findings and requires contact-status transitions to remain logically supported across neighboring frames.Mere contact is explicitly judged as false contact.
  • C. Depth Map Verification (Critical Phase):: The output lists appeared hands and per-frame contacts in a JSON structure, with valid fingers only and empty finger lists when there is no contact.
  • C. Depth Map Verification (Critical Phase):: ArtHOI’s in-the-wild evaluation shows that MLLM contact reasoning is more reliable than mask-intersection heuristics under casually captured conditions.The heuristic performs slightly worse on controlled lab datasets but drops substantially on in-the-wild videos.
Loading 2603.25791v1…