PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

Agniv Chatterjee, Georgios Pavlakos

arXiv:2608.27407v1cs.CV

TL;DR

Single-image 3D human-object interaction reconstruction is difficult because of depth ambiguity, occlusion, and object-shape variability. MILO interprets a Large Reconstruction Model’s joint human-object mesh, and it surpasses existing methods across multiple benchmarks while using less privileged information.

  • Problem

    Single-image HOI reconstruction must recover human pose and object geometry despite object variability, occlusions, and depth ambiguities.

  • Method

    MILO applies an LRM to obtain a holistic human-object mesh, segments its components, fits SMPL-H to the human part, and optionally aligns an object template.

  • Results

    MILO surpasses existing HOI reconstruction methods across multiple benchmarks without relying on ground-truth contact information.

  • Takeaways & Limitations

    The LRM output serves as a geometric scaffold that captures human-object arrangement and proximity cues while remaining agnostic to object category and shape.

  • Takeaways & Limitations

    MILO’s pose estimates can become image-inconsistent when severe self-occlusion or object occlusion causes the LRM to hallucinate missing body parts.

Abstract

from arXiv · show

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.

1 Introduction

MILO reframes single-image 3D human-object interaction reconstruction as interpreting a joint mesh generated by a Large Reconstruction Model. It addresses object variability, occlusion, and depth ambiguity while avoiding required contact information.

  • Single-image HOI reconstruction is difficult because objects vary in shape and affordance, interactions have multiple modes, and images contain occlusions and depth ambiguities.
  • Existing approaches rely on human parametric models, limited controlled datasets, object-mesh retrieval, and sometimes privileged contact or depth cues.
  • MILO uses an LRM to generate an image-specific holistic human-object mesh that preserves relative arrangement and proximity cues.
  • The pipeline segments the LRM mesh, fits SMPL-H to the human component, and optionally aligns an object template using semantic correspondences.
  • MILO is evaluated quantitatively on InterCap, HODome, and IMHD, with qualitative results on in-the-wild images.
  • The method achieves state-of-the-art performance across multiple benchmarks without relying on ground-truth contact information.

2 Related Works

Prior work spans parametric and non-parametric human reconstruction, diverse object representations, joint HOI modeling, and contact- or template-based pipelines. MILO instead uses an LRM-generated joint mesh and does not require explicit contact information, while object templates remain optional.

  • Human Reconstruction: Human reconstruction methods use parametric body models or directly predict meshes and point clouds from monocular inputs.
  • Object Reconstruction: Monocular object reconstruction spans volumetric, point-based, mesh, implicit, radiance-field, Gaussian-splatting, and diffusion-based representations.
  • Large Reconstruction Models: Recent LRMs provide efficient, view-consistent reconstruction of diverse objects, and MILO uses their output as an image-specific joint human-object mesh.
  • Human-Object Reconstruction: Prior HOI methods jointly reason about pose, contact, affordance, spatial relations, or dense human-object offsets.
  • Object Templates: Object-retrieval approaches depend on the coverage and quality of shape repositories, while MILO can reconstruct interaction objects without explicit CAD templates.
  • Limitations of Prior Work: Many previous methods assume object templates or ground-truth contact annotations, limiting scalability and generalization to novel objects and in-the-wild scenes.
  • Contact Estimation: MILO does not require contact annotations and can extract competitive contact estimates from its HOI reconstructions.

3 Technical Approach

MILO interprets an LRM-generated holistic human-object mesh, fitting a parametric human model and optionally aligning an object template. Its pipeline uses multi-view keypoints, staged optimization, visibility-aware consistency, and mesh segmentation.

  • LRM Reconstruction: MILO uses an LRM to reconstruct a holistic human-object mesh from a single RGB image before fitting separate components.The pipeline then interprets this mesh rather than relying only on image reprojection.
  • 3D Keypoint Estimation: The method renders 60 virtual views and estimates body and hand keypoints, triangulating consistent observations into confidence-weighted 3D keypoints.ViTPose supplies body detections and HaMeR supplies hand detections; low-confidence observations are discarded before robust triangulation.
  • Human Optimization: SMPL-H fitting proceeds in two stages: root fitting optimizes orientation and translation, then pose fitting additionally optimizes shape, body pose, and hand pose.Initialization comes from HMR2.0 and HaMeR, while the first stage also optimizes scale to accommodate uncertain LRM metric scale.
  • Object Extraction and Alignment: MILO segments the LRM mesh into human and object components using multi-view masks, then optionally aligns an available object template through semantic correspondences.The resulting object component is composed with the fitted parametric human mesh for evaluation and visualization.

4 Experiments

MILO is evaluated against prior methods across multiple datasets, reconstruction settings, contact estimation, LRM geometries, and scaffold designs. It consistently performs strongly without ground-truth contact information, while ablations identify the joint reconstruction scaffold as central to accuracy.

  • Comparison with the state of the art: MILO achieves the best overall InterCap reconstruction accuracy without using contact information.The evaluation reports PA-CD for human, object, and combined human-object meshes, with lower values being better.
  • Comparison with the state of the art: MILO is compared on InterCap, HODome, and IMHD, including settings with and without object templates.InterCap evaluation includes regression- and optimization-based baselines under an out-of-domain training protocol.
  • Comparison with the state of the art: MILO outperforms PICO across all reported metrics on both HODome and IMHD, in template and template-free modes.Qualitative results also show better 3D alignment, more accurate contact, and fewer penetrations than prior approaches.
  • Contact evaluation: MILO’s inferred contact achieves higher F1, Precision, and Recall and lower geometric error than the DECO RICH checkpoint on InterCap.MILO infers contact from the proximity of the fitted SMPL-H mesh and aligned object template rather than directly predicting contact.
  • Effect of mesh geometry: SAM3D provides comparable performance to Hunyuan3D-2.0, while the oracle result indicates remaining error is largely bounded by upstream reconstruction quality.InstantMesh performs worse, although it represents an older method.
  • Comparison with individual reconstruction: MILO outperforms object-only and independently reconstructed baselines, indicating that a joint human-object scaffold is more effective for HOI reconstruction.The controlled independent-reconstruction comparison attributes the substantial improvement to the joint scaffold rather than the downstream fitting procedure.

5 Conclusion

MILO reconstructs single-image human-object interactions by interpreting a holistic LRM mesh as a geometric scaffold. It surpasses existing methods across benchmarks with less privileged information, while its performance remains bounded by the underlying LRM quality.

  • 5 Conclusion: MILO obtains a holistic human-object mesh from an LRM, segments human and object parts, and fits SMPL-H and an optional object template.The scaffold preserves arrangement and proximity cues while remaining agnostic to object category and shape.
  • 5 Conclusion: MILO surpasses existing HOI reconstruction methods across multiple benchmarks despite using significantly less privileged information.Qualitative in-the-wild results cover diverse objects, poses, and interaction contexts.
  • 5 Conclusion: MILO’s performance is bounded by the quality of the underlying LRM.The conclusion identifies this as a limitation and notes the rapid progress of LRM methods.

Supplementary Material for: Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

The supplementary material adds implementation details, extended qualitative results, and representative failure cases for MILO. It expands the pipeline description and provides additional dataset-specific comparisons.

  • Supplementary Material: The supplement expands multiview rendering, keypoint triangulation, object-template alignment, and pointcloud segmentation details.These components support keypoint detection, pointcloud segmentation, and correspondence estimation.
  • Supplementary Material: The supplement includes discussion of representative failure cases.These cases are presented as part of the extended qualitative analysis.

S.1 MILO implementation details

MILO uses multi-view rendering and confidence-aware geometric processing to segment the LRM mesh and fit human-object components. The implementation emphasizes robust keypoint triangulation and boundary-sensitive segmentation.

  • Multi-view rendering: MILO renders the LRM mesh from 60 viewpoints spanning azimuth intervals of 30 degrees and elevations of 0, ±30, and ±60 degrees.These views support keypoint detection, correspondence estimation, and mesh segmentation.
  • 3D keypoint estimation: Hand detections are reduced to at most one candidate per side per view by comparing candidate centroids, with image-plane extent as a fallback when the body wrist is unavailable.The centroid over 21 hand keypoints is preferred because wrist localization is often unstable.
  • Confidence filtering and triangulation: Keypoints are retained only from views exceeding confidence threshold 0.6, invalidated when fewer than 3 observations remain, and triangulated using consensus under a reprojection threshold of 5.0.The selected 3D hypothesis is refined nonlinearly over its inlier views.
  • Confidence aggregation: Aggregated body and hand keypoints form a confidence-weighted 67-point set used to fit SMPL-H to the LRM mesh.The set contains 25 body, 21 left-hand, and 21 right-hand keypoints.
  • Point-cloud segmentation: Segmentation combines viewpoint quality with multi-scale boundary closeness, weighting vertices near human-object boundaries more heavily before thresholding and geometric cleanup.The resulting non-human vertices are filtered by outlier removal, neighborhood pruning, and largest-cluster selection.
  • Point-cloud segmentation: Boundary weighting normalizes vertex distance by d_max, applies an adaptive exponent γ ∈ [0.3, 0.8], and optionally adds a boost when the object is much smaller than the human.Scores are aggregated across visible viewpoints to obtain final human and object labels.

S.2 Further Qualitative Results

MILO preserves global object geometry and positions objects consistently with the body across qualitative datasets. Template alignment further improves regularity for rigid objects when templates are available.

  • Qualitative trends: MILO better preserves global object geometry while positioning objects relative to the body, especially for large carried or supported objects.Examples include backpacks, suitcases, chairs, tables, and box-like objects, where PICO often produces implausible orientation or placement.
  • Dataset comparisons: On PICO-db, the non-template variant is visually more accurate than PICO across varied in-the-wild appearances, viewpoints, and object categories.On HODome and IMHD, the template-based variant further improves geometric regularity and alignment for rigid canonical objects.

PICO Ours PICO Ours

The qualitative comparison indicates that MILO improves interaction plausibility relative to PICO, including for slender objects where detailed surface recovery is less decisive. These reconstructions support downstream reasoning about contact, support, and affordances.

  • Variant comparison: For slender objects such as bats, the gap between MILO’s template and non-template variants is smaller because pose estimation is the main challenge.The comparison concerns the relative benefit of detailed surface recovery across variants.
  • Interaction plausibility: Even when recovered object geometry lacks detail, MILO often produces more physically plausible interactions than PICO.The visual results connect this plausibility to downstream reasoning about contact, support, and affordances.

S.2.1 Results Discussion

The InterCap quantitative comparison excludes HDM because its training data construction uses both BEHAVE and InterCap, giving it a competitive advantage over methods not trained on InterCap.

  • Evaluation scope: HDM is excluded from the InterCap quantitative evaluation because its ProciGen-based training construction draws on both BEHAVE and InterCap.This creates a competitive advantage over methods not trained on InterCap.

PICO Ours

MILO produces more physically plausible and ground-truth-proximate reconstructions than PICO on HODome, with and without object templates. On IMHD, the supplementary material provides qualitative evaluation of both methods.

  • MILO outputs are more physically plausible and closer to ground truth than PICO on HODome, with or without templates.
  • PICO’s retrieval strategy is often unsuccessful because several HODome and IMHD object categories lack sufficiently similar counterparts in PICO-db.
  • MILO uses LRM meshes as geometric scaffolds, avoiding reprojection objectives while prioritizing 3D interaction consistency.
  • The supplementary material presents qualitative comparisons between PICO and MILO on IMHD images.

S.2.2 Failure Cases

MILO’s failures stem mainly from imperfect LRM geometry, noisy segmentation, and ambiguous template correspondences. Template alignment helps thin or small objects but can harm larger objects, while runtime and segmentation ablations expose practical trade-offs.

  • Failure Cases: LRM reconstruction and point-cloud segmentation failures cause incomplete or warped large-object geometry, truncated objects, background leakage, and extra-object reconstructions.
  • Failure Cases: Partial visibility can restrict template correspondences to visible geometry, while symmetric objects and missing segmented regions further impair alignment.
  • Failure Cases: Template alignment reduces PA-CDo for bottles from 34.23 to 9.97 cm and cups from 42.91 to 18.64 cm, but increases trolley error from 17.46 to 21.79 cm.
  • Failure Cases: Symmetric objects can produce orientation flips, with mean rotation error reaching 90◦-120◦ versus 21.71◦ for chairs.
  • Failure Cases: MILO can fail for small objects under severe occlusion, although larger-object occlusion is often handled through LRM priors and SMPL-H completion.
  • Ablations: On InterCap, Ours achieves 6.85 cm, 20.74 cm, and 9.36 cm versus P3-SAM’s 7.30 cm, 55.86 cm, and 14.68 cm across the reported metrics.
  • Ablations: Removing visibility coverage, mask quality, or boundary-closeness weighting degrades point-cloud segmentation performance.
  • Runtime Analysis: Core optimization requires 32.3s, while PICO’s three stages require 475.84s; optional template alignment adds semantic-correspondence cost dominated by 375.75s.
Loading 2608.27407v1…