PaperlayerView exact revision on arXiv

Source-linked AI summary

CHORE: Contact, Human and Object REconstruction from a single RGB image

Xianghui Xie, Bharat Lal Bhatnagar, Gerard Pons-Moll

arXiv:2204.02445v3cs.CV

TL;DR

Single-image 3D reconstruction has largely treated people apart from surrounding objects, despite the difficulty of inferring their interactions under occlusion and depth ambiguity. CHORE jointly reconstructs the person, object, and contacts using neural fields and model fitting, with reported gains over PHOSA across benchmark and user-study evaluations.

  • Problem

    Most prior single-image 3D human reconstruction reasons about people in isolation, while joint human-object inference is difficult because of occlusion, depth ambiguity, and sparse contact evidence.

    Source: §1 · p. 1

  • Method

    CHORE predicts human and object distance fields, a body-part correspondence field, and an object-pose field, then fits a parametric body and object template while reasoning about contacts.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 6

  • Results

    Over 50% improvement in Chamfer distance over PHOSA on BEHAVE; users preferred CHORE over PHOSA on NTU-RGBD (84%) and COCO (72%).

    Source: §6 · p. 14

  • Takeaways & Limitations

    Depth-aware scaling addresses depth-scale ambiguity while preserving pixel alignment for learning on real data.

    Source: §6 · p. 14

    The method assumes a known object template; its stated scope is one person interacting closely with one dynamic movable object.

    Source: §3 · p. 4, §5 · p. 14

Abstract

from arXiv · show

Most prior works in perceiving 3D humans from images reason human in isolation without their surroundings. However, humans are constantly interacting with the surrounding objects, thus calling for models that can reason about not only the human but also the object and their interaction. The problem is extremely challenging due to heavy occlusions between humans and objects, diverse interaction types and depth ambiguity. In this paper, we introduce CHORE, a novel method that learns to jointly reconstruct the human and the object from a single RGB image. CHORE takes inspiration from recent advances in implicit surface learning and classical model-based fitting. We compute a neural reconstruction of human and object represented implicitly with two unsigned distance fields, a correspondence field to a parametric body and an object pose field. This allows us to robustly fit a parametric body model and a 3D object template, while reasoning about interactions. Furthermore, prior pixel-aligned implicit learning methods use synthetic data and make assumptions that are not met in the real data. We propose a elegant depth-aware scaling that allows more efficient shape learning on real data. Experiments show that our joint reconstruction learned with the proposed strategy significantly outperforms the SOTA. Our code and models are available at https://virtualhumans.mpi-inf.mpg.de/chore

1 Introduction

CHORE addresses the difficulty of reconstructing interacting humans and objects from one image, where occlusion, depth ambiguity, and sparse contact cues complicate inference. It jointly learns their interactions and uses neural fields to fit both models, with a reported improvement of over 50% over prior art.

1 Introduction

  • Occlusion, uncertain relative size and spatial arrangement, and contact cues limited to a small image region make joint human-object reconstruction challenging.Prior work reconstructs humans and objects separately and uses hand-crafted rules, which are difficult to scale across varied contacts and can leave errors uncorrected.Source: §1 · pp. 1–2
  • CHORE learns human-object interactions from data and jointly fits a parametric body and object using surface-distance, body-correspondence, and object-pose fields.The approach predicts the neural fields before fitting the SMPL model and object template to the image.Source: §1 · p. 2
  • A perspective-camera training strategy supports pixel-aligned implicit learning on real data, unlike prior approaches described as using weak perspective cameras and synthetic data.The paper presents the strategy as enabling more effective learning with perspective cameras.Source: §1 · p. 2
  • A remarkable performance improvement of over 50% compared to prior art is reported.Source: §1 · p. 2

2 Related Work

Prior work has addressed human or object reconstruction separately, and interaction methods have limitations in scope or input setting. CHORE instead jointly reconstructs humans, objects, and contacts from one RGB image, learning contact priors rather than relying on hand-crafted heuristics.

2 Related Work

  • Human reconstruction methods do not reconstruct objects, while rigid-object methods do not address the added challenge of articulated humans and their interactions.The paper situates CHORE as jointly reconstructing humans, objects, and interactions.Source: §2 · p. 3
  • Hand-object interaction methods do not predict the full body, while cited image-based contact methods do not reconstruct both a 3D human and object from one RGB image.Other full-body interaction approaches address scene constraints, multiview capture, or human-scene reconstruction.Source: §2 · p. 3
  • PHOSA uses separate human and object fitting with predefined contact pairs, while Weng et al. [62] predict human and scene layout separately and use predefined contact weights.The paper describes these heuristic approaches as prone to errors or not scalable and accurate.Source: §2 · p. 3
  • CHORE predicts distance, SMPL-part, and object-pose fields from an image and human and object masks, then uses them to fit the body and object while reasoning about contacts.The paper reports that its learning-based approach outperforms heuristic-based PHOSA and [62].Source: Figure 2 · p. 4, §2 · p. 4

3 Method

CHORE predicts neural fields for the human, object, their relative pose, and contacts, then fits a parametric body and object template to those fields. For real-image training, depth-aware scaling preserves perspective pixel alignment while reducing depth-scale ambiguity.

3.1 CHORE neural fields from single images

  • CHORE jointly predicts human and object distance fields, SMPL part correspondences, and an object pose field, then fits a controllable body model and object template to them.The learned fields provide a 3D data term for fitting rather than directly regressing body parameters from the image.Source: §3 · p. 4, §3.1 · p. 5

3.2 3D human and object fitting with CHORE fields

  • Human fitting combines distance-to-surface and part-label losses, while object fitting uses its template’s surface distance, silhouette agreement, and center regularization.Part labels help avoid mismatches such as aligning an arm to a torso or a hand to a head.Source: §3.2 · pp. 6–7
  • A joint contact term uses predicted corresponding human and object contact-point sets, encouraging the object to remain close to the SMPL parts predicted to touch it.The term is zero when no contacts are detected.Source: §3.2 · p. 7

3.3 Learning pixel-aligned neural fields on real data

  • Depth-dependent mesh scaling preserves perspective image projections while placing mesh centers at a fixed depth, reducing ambiguity for pixel-aligned learning on real images.The paper contrasts this strategy with weak-perspective misalignment and poor results from direct perspective training under depth-scale ambiguity.Source: §3.3 · pp. 8–9, Figure 3 · p. 8

4 Experiments

Experiments compare CHORE with prior methods on BEHAVE and test generalization to COCO and NTU-RGBD. Ablations examine human reconstruction, depth-aware training, contact modeling, and fitting choices.

4 Experiments

  • Evaluation uses 10.7k BEHAVE training frames and 4.5k unseen-location test frames; COCO and NTU-RGBD comparisons rely on user surveys because they lack 3D ground truth.Source: §4 · pp. 9–10

4.1 Comparison on BEHAVE dataset

  • CHORE outperforms PHOSA and [62] on BEHAVE Chamfer-distance comparisons, using combined-mesh alignment for BEHAVE and SMPL-only alignment for BEHAVE−−.Source: §4.1 · p. 10, Table 1 · p. 11

4.2 Generalisation beyond BEHAVE

4.3 Comparison with a human reconstruction method

  • Joint human-object reasoning performs on par with human-only FrankMocap for human reconstruction, and slightly improves it by accounting for the interacting object.Source: Table 2 · p. 12, §4.3 · p. 12

4.5 Contacts modeling

  • Contact prediction helps place the object at the contact location, improving physical plausibility; without contact information, alignment is inaccurate.Source: Figure 7 · p. 13, §4.5 · p. 13

4.4 Pixel-aligned learning on real data

  • Depth-aware training significantly improves implicit-function and mesh reconstruction on real data over weak-perspective and direct-perspective baselines.Source: Table 3 · p. 13, §4.4 · p. 13

4.6 Other ablation studies

  • Removing rotation or translation initialization worsens fitting and can trap the object in local minima; joint fitting also outperforms raw neural point-cloud reconstruction by using human and object priors.Source: §4.6 · pp. 13–14

5 Limitations and Future Work

CHORE assumes known object templates for objects in the images. The authors propose template retrieval from a 3D CAD database or direct image-based shape reconstruction as future directions.

  • CHORE assumes known object templates; future work could select templates from a 3D CAD database or reconstruct object shape directly from images.Source: §5 · p. 14

6 Conclusion

CHORE jointly reconstructs humans, objects, and contacts from single images, using neural implicit predictions and model-based fitting learned from real data. Experiments report improved Chamfer distance over PHOSA on BEHAVE and user preference for CHORE over PHOSA on NTU-RGBD and COCO.

6 Conclusion

  • CHORE learns joint reconstruction of humans, objects, and contacts from single images, combining neural implicit predictions with robust model-based fitting.The method learns from real data rather than synthetic scans.Source: §6 · p. 14
  • Depth-aware scaling addresses depth-scale ambiguity while preserving pixel alignment for learning from real data.Source: §6 · p. 14
  • On BEHAVE, joint reasoning improved Chamfer distance by over 50% compared to PHOSA; users preferred CHORE over PHOSA by 84% on NTU-RGBD and 72% on COCO.The COCO dataset has no ground truth.Source: §6 · p. 14

Related papers

Loading 2204.02445v3…