PaperlayerView exact revision on arXiv

Source-linked AI summary

BEHAVE: Dataset and Method for Tracking Human Object Interactions

Bharat Lal Bhatnagar, Xianghui Xie, Ilya A. Petrov, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

arXiv:2204.06950v1cs.CV

TL;DR

BEHAVE addresses the lack of diverse, full-body 3D human-object interaction data and flexible capture methods for natural environments. It introduces a portable multi-view RGBD dataset and a tracker that predicts human-object contacts through correspondences to a statistical body model. The resulting benchmark and method jointly represent humans, objects, and their surface contacts in 3D.

  • Problem

    Existing interaction data and capture methods do not provide diverse, accurate, and flexible 3D human-object interaction capture in natural environments.

  • Method

    BEHAVE uses portable multi-view RGBD capture and neural correspondence and unsigned-distance predictions within joint registration to track humans, objects, and contacts.

  • Results

    BEHAVE provides around 15.2k frames spanning 20 objects, 8 subjects, and 5 locations, with annotated humans, objects, and contacts.

  • Takeaways & Limitations

    BEHAVE provides a benchmark for modeling human-object interactions and supports challenges including tracking, single-image reconstruction, and pose estimation.

Abstract

from arXiv · show

Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record around 15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave

1. Introduction

BEHAVE addresses the lack of diverse 3D interaction data and flexible capture methods by combining a portable RGBD setup, a dataset, and a tracker for humans, objects, and contacts.

  • Motivation: Existing capture systems are expensive, spatially confined, vulnerable to occlusion, and restrictive in the interactions, scenes, and objects they support.Wearable sensors avoid volume restrictions but cannot accurately capture close-range interaction.
  • Approach: BEHAVE uses portable, cheap, easy-to-use RGBD cameras to capture diverse 3D human interactions in natural environments.The setup must address noisy, incomplete depth, frequent mutual occlusion, and small contact regions.
  • Dataset: Around 15.2k frames span 20 objects, 8 subjects, and 5 locations, with ground-truth SMPL models, object meshes, and contacts.The dataset is intended for training and evaluating the proposed tracker.
  • Contributions: BEHAVE jointly tracks humans, objects, and contacts in 3D from multi-view RGBD images.The authors describe this as the first approach with this capability in natural environments.
  • Contributions: The dataset provides multi-view RGBD sequences with corresponding human models, object annotations, and contact annotations.The authors identify this as the largest such dataset and plan to release code and data for further research.

2. Related Work

Related work spans human and object reconstruction, humans in static scenes, and dynamic interactions, but existing datasets and methods remain limited in full-body, 3D, contact-aware capture.

  • Human reconstruction and performance capture: Human reconstruction increasingly models details such as gestures, expressions, self-contacts, interactions, and clothing, often using parametric body models.These developments motivate the paper’s use of implicit diffused representations as a tracker backbone.
  • Object reconstruction: Object reconstruction from RGB or RGBD data usually treats objects in isolation, despite the challenge of reconstructing them under human-induced occlusions.The paper frames dynamic reconstruction with human involvement as a more challenging setting.
  • Humans in static scenes: Methods for humans in static scenes address placement, motion prediction, pose reconstruction, or interaction priors, but focus on static objects.The paper targets dynamic human-object interactions instead.
  • Dynamic human object interactions: Hand-object methods use 3D, 2.5D, or 2D data, whereas full-body methods may lack full 3D reasoning, target specific objects, or omit contact prediction.Recent RGB and RGBD methods reconstruct and track interactions but do not consider contact prediction.
  • Dataset comparison: BEHAVE is presented as the first dataset capturing dynamic human-object interactions in diverse environments with full 3D human, object, and contact annotations.The comparison criteria include RGB input, 3D human and contact availability, quality, and scalability.

3. BEHAVE Dataset

BEHAVE records natural full-body interactions with portable multi-view RGBD cameras and provides fitted human, object, segmentation, and contact annotations for several benchmark tasks.

  • Recording multi-view RGBD data: Four calibrated Kinects record 8 subjects interacting with 20 diverse objects at 5 indoor locations.Recorded activities include lifting, carrying, sitting, pushing, pulling, and free interactions.
  • Human segmentation and SMPL fitting: Human annotations use DetectronV2 segmentation, lifted multi-view depth point clouds, FrankMocap initialization, and instance-specific SMPL fitting.Manual mask correction is included before fitting the SMPL model to the segmented human point cloud.
  • Object segmentation and fitting: Object annotations fit pre-scanned meshes using manually marked multi-view keypoints, achieving a 2.42cm Chamfer error.Fitted meshes are projected back into the images to obtain object segmentation masks.
  • Contact annotation: Contacts are automatically detected when a registered human-surface point lies within 2cm of the object surface.Each object point stores a binary contact label and correspondence to the human contact location.
  • Challenges and uses: BEHAVE supports tracking, single-image joint reconstruction, and pose and shape estimation in heavily occluded natural environments.The dataset is also intended for additional community applications.

4. Method: Tracking human, object and contacts

BEHAVE jointly registers a human SMPL model and a template object mesh to multi-view RGBD point clouds while predicting correspondences that recover human-object contacts. Its fitting pipeline uses learned correspondences, implicit surface distances, object orientation initialization, and explicit contact optimization to improve robustness under incomplete observations and occlusion.

  • Joint registration: BEHAVE formulates per-frame registration of SMPL and a pre-scanned object mesh to multi-view RGBD data, with contacts represented as mesh correspondences.The formulation fits both models to lifted human and object point clouds while enforcing contact consistency.
  • Pipeline overview: Figure 3 depicts the pipeline from segmented multi-view RGBD frames to 3D point clouds, learned correspondences, model registration, and joint human-object tracking.The network predicts both human-to-body-model and object-to-body-model correspondences, with the latter supporting contact modelling.
  • Human fitting: Learned dense correspondences associate input human points with SMPL locations, helping the fitting process avoid degenerate local minima such as left-right body-part flips.The network predicts correspondences from voxel-aligned point-cloud features, and the correspondence term constrains transformed SMPL points to match the input.
  • Object fitting: Implicit unsigned-distance predictions let object mesh vertices be fitted even when severe occlusions make corresponding object point-cloud regions missing.The object objective combines point-to-mesh fitting with vertex-based surface constraints for incomplete observations.
  • Object fitting: Predicted global object orientation initializes rotation optimization, preventing object fitting from getting stuck in local minima and correcting semantically misaligned parts.Orientation is estimated from filtered near-surface query points and used to initialize the object rotation before refinement.
  • Contact refinement: Explicit contact optimization jointly adjusts SMPL and object parameters so predicted contact vertices align, reducing floating-object and hovering-hand artefacts.The contact term selects near-surface object vertices and matches them to corresponding SMPL locations.

5. Experiments

BEHAVE outperforms comparison methods for jointly tracking humans, objects, and contacts, while ablations show that contact prediction improves physical plausibility and object orientation prediction improves localization.

  • Comparison with PHOSA: BEHAVE clearly outperforms PHOSA for jointly tracking humans, objects, and contacts.PHOSA relies on fixed contact regions and hand-crafted contact heuristics, whereas BEHAVE predicts full-body contacts from data.
  • Direct fitting baseline: Directly fitting SMPL and object meshes to point clouds easily gets stuck in local minima.Noisy point clouds and heavy human-object occlusion cause missing parts that make direct fitting unreliable.
  • Ablations: Object orientation prediction noticeably improves object localization error, while contact loss has little quantitative effect but a noticeable qualitative effect.Without contact information, humans and objects may fail to lock into the correct location, producing floating objects.
  • Comparison with LoopReg: BEHAVE outperforms LoopReg because distances to the SMPL surface allow fitting with nonsurface points under missing and noisy observations.LoopReg assumes a complete, noise-free human point cloud and cannot predict correspondences for occluded parts.

6. Conclusions

BEHAVE jointly tracks humans, objects, and explicit contacts in natural environments using neural predictions integrated into joint registration. It also provides a large annotated RGBD dataset and benchmark intended to support research on human-object interaction.

  • 6. Conclusions: BEHAVE jointly tracks humans, objects, and explicit contacts in natural environments.The method predicts correspondences to a 3D human body model and uses unsigned distance fields over human and object surfaces.
  • 6. Conclusions: The BEHAVE dataset is the largest RGBD sequence dataset with annotated humans, objects, and contacts to date.It is presented as the first benchmark for research on modelling human-object interactions.
  • 6. Conclusions: The released dataset and code provide real-world challenges for reconstructing and tracking human-object interactions from RGB and RGBD inputs.The challenges include single-image reconstruction, multiple- and single-view tracking, and pose estimation.
Loading 2204.06950v1…