PaperlayerView exact revision on arXiv

Source-linked AI summary

Detailed 2D-3D Joint Representation for Human-Object Interaction

Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, Cewu Lu

arXiv:2004.08154v2cs.CVcs.LG

TL;DR

HOI detection is challenged by viewpoint-dependent 2D ambiguity, sparse 3D joints, and limited modeling of interacted objects. DJ-RN combines detailed single-view 3D human reconstruction with an estimated 3D object representation and joint 2D-3D consistency learning. It achieves state-of-the-art results on HICO-DET and Ambiguous-HOI, while object-depth estimation depends on empirical priors and regularization.

  • Problem

    HOI learning lacks sufficiently detailed, view-robust human information and comprehensive 3D modeling of the interacted object.

  • Method

    DJ-RN combines detailed single-view 3D body, face, and hand reconstruction with a hollow-sphere object representation and cross-modal 2D-3D consistency learning.

  • Results

    DJ-RN achieves state-of-the-art results on HICO-DET and Ambiguous-HOI.

  • Takeaways & Limitations

    The method provides a joint representation paradigm and a benchmark for evaluating HOI disambiguation under difficult 2D ambiguity.

  • Takeaways & Limitations

    Object-depth estimation is sensitive to sphere radius and is regularized using empirical object-size and relative-depth priors.

Abstract

from arXiv · show

Human-Object Interaction (HOI) detection lies at the core of action understanding. Besides 2D information such as human/object appearance and locations, 3D pose is also usually utilized in HOI learning since its view-independence. However, rough 3D body joints just carry sparse body information and are not sufficient to understand complex interactions. Thus, we need detailed 3D body shape to go further. Meanwhile, the interacted object in 3D is also not fully studied in HOI learning. In light of these, we propose a detailed 2D-3D joint representation learning method. First, we utilize the single-view human body capture method to obtain detailed 3D body, face and hand shapes. Next, we estimate the 3D object location and size with reference to the 2D human-object spatial configuration and object category priors. Finally, a joint learning framework and cross-modal consistency tasks are proposed to learn the joint HOI representation. To better evaluate the 2D ambiguity processing capacity of models, we propose a new benchmark named Ambiguous-HOI consisting of hard ambiguous images. Extensive experiments in large-scale HOI benchmark and Ambiguous-HOI show impressive effectiveness of our method. Code and data are available at https://github.com/DirtyHarryLYL/DJ-RN.

1. Introduction

HOI detection needs view-robust representations because 2D appearance and pose can be ambiguous, while rough 3D joints miss fine-grained body cues and object configuration. DJ-RN addresses these gaps through detailed 3D human and object representations, joint 2D-3D learning, and the Ambiguous-HOI benchmark.

  • Motivation: 2D HOI cues can be ambiguous across viewpoints, while rough 3D joints lack the detailed hand, face, and body-shape information needed for complex interactions.The paper motivates detailed 3D body modeling with examples such as cutting with a knife and eating or talking.
  • Method: DJ-RN represents HOI in 3D using detailed human shape plus estimated object location and size derived from 2D configuration and object-category priors.The object is represented with a hollow sphere in a normalized 3D volume.
  • Method: The method jointly learns 2D and 3D features with spatial, semantic, and body-part consistency tasks.The consistency design aligns 2D spatial features with 3D features and encourages corresponding semantic and attention patterns.
  • Evaluation: Ambiguous-HOI contains difficult ambiguous examples selected from HICO-DET, V-COCO, OpenImage, and HCVRD to evaluate 2D disambiguation.The benchmark is designed specifically to assess processing of 2D ambiguities.
  • Results: DJ-RN achieves state-of-the-art results on HICO-DET and Ambiguous-HOI.The paper reports extensive experiments on both the established benchmark and the proposed ambiguous-image benchmark.

2. Related Work

Prior HOI methods largely use appearance, spatial relations, and sometimes 2D pose, while related action-recognition work increasingly uses 3D human pose. The paper identifies 2D ambiguity and limited 3D body detail as underexplored issues for HOI learning.

  • Human-Object Interaction Detection: Earlier HOI methods mainly rely on visual appearance and relative human-object locations, with some incorporating estimated 2D pose.Representative approaches include multi-stream, graph, action-density, and part-state models.
  • Human-Object Interaction Detection: The paper distinguishes its approach from prior work by targeting 2D ambiguity rather than relying only on 2D cues.The related-work discussion states that 2D ambiguity in HOI had not been well studied.

3. Representing HOI in 3D

The method constructs a 3D HOI representation from single-view detailed human reconstruction and an estimated 3D object sphere. It uses projection geometry, category and depth priors, and normalized point-based volume construction to encode the interaction.

  • 3. Representing HOI in 3D: Given 2D human and object detections plus human pose, the method recovers a detailed 3D human body and estimates the object’s 3D location and size.These outputs form the input to the 3D HOI spatial configuration volume.
  • 3.1. Single-view 3D Body Capture: SMPLify-X fits the SMPL-X model to the image and 2D pose to estimate body pose, body shape, face shape, hand shape, and facial expression, from which a 3D mesh is generated.The mesh is blended and deformed to match the observed posture and shape.
  • 3.2. 3D Spatial Configuration Volume: Instead of reconstructing object geometry, the method uses a hollow sphere whose center and depth are estimated from perspective projection, object size priors, and human-object distance.The sphere avoids difficult single-view 6D object-pose estimation across varied categories.
  • 3.2. 3D Spatial Configuration Volume: Projection equations constrain the sphere center using tangent planes, while category-dependent radius and depth priors regularize the estimated object position.The depth is sensitive to the radius, so empirical regularization factors constrain the estimate relative to the recovered human depth range.
  • 3.2. 3D Spatial Configuration Volume: The normalized 3D volume aligns configurations at the human pelvis, standardizes gravity and shoulder orientation, and samples points from the body and sphere.The described construction downsamples the body to 916 points and samples 312 spherical-surface points.

4. 2D-3D Joint Learning

DJ-RN combines 2D and detailed 3D human-object representations, aligning their spatial, body-part, and semantic information for HOI learning.

  • Framework overview: DJ-RN uses separate 2D-RN and 3D-RN modules to extract modality-specific HOI features before learning a joint representation.The framework aligns 2D and 3D spatial features and applies body-part attention and semantic consistency objectives.
  • 2D representation: The 2D-RN extracts appearance features from human and object regions and spatial features from pose maps and human-object spatial maps.Its spatial input combines two 64 × 64 box maps with 17 pose-joint heatmaps.
  • 3D representation: The 3D-RN encodes a normalized human-object volume with PointNet and detailed SMPL-X body, face, hand, expression, and pose information.The volume stream also pairs 3D part and object regions with semantic category embeddings.
  • 2D-3D spatial alignment: Triplet-based spatial alignment uses same-HOI 3D features as positives and non-overlapping-HOI features as negatives to gather varied 2D configurations.The alignment uses Euclidean distance with margin α = 0.5.
  • Joint inference: The network combines modality-specific classification with an early-fused joint classifier, producing separate 2D, 3D, and joint HOI predictions.The joint classifier concatenates features from the final fully connected layers of 2D-RN and 3D-RN.

5. Experiment

Experiments evaluate DJ-RN on HICO-DET and the ambiguity-focused Ambiguous-HOI benchmark, including comparisons, visualizations, and ablations. Joint 2D-3D learning improves performance, while detailed body information and consistency components contribute materially.

  • 5.1. Ambiguous-HOI: Ambiguous-HOI contains hard examples selected using 2D pose and spatial-configuration ambiguity criteria, yielding 8,996 images and 25,188 annotated human-object pairs.Selection combines clustering relative to pose templates with an MLP ambiguity probe.
  • 5.4. Results and Comparisons: 21.34 mAP is achieved by unified DJ-RN on HICO-DET Default Full, exceeding joint learning at 20.61 mAP and the 2D and 3D streams.The 2D stream reaches 13.53 mAP and the 3D stream 12.41 mAP on the same setting.
  • 5.4. Results and Comparisons: Visualizations show aligned 2D and 3D attentions that capture relevant body parts across HOIs, alongside predicted 3D spatial configuration volumes.The 3D representation network uses pre-extracted features and is reported to run at about 50 FPS, whereas GPU-accelerated SMPLify-X runs at about 5 FPS.
  • 5.4. Results and Comparisons: DJ-RN outperforms the previous method by 0.65, 2.15, and 2.23 mAP on Ambiguous-HOI evaluation settings.All methods obtain relatively low scores because Ambiguous-HOI is substantially more difficult.
  • 5.5. Ablation Study: Removing detailed face and hand shape degrades DJ-RN, especially without hand shape, while removing the volume or body block reduces performance by 1.00 and 1.33 mAP.Removing the alignment, triplet, or semantic losses further decreases performance by 0.64, 0.51, and 0.54 mAP.

6. Conclusion

The paper concludes that DJ-RN learns a joint 2D-3D HOI representation from detailed 3D human and estimated object geometry. It reports state-of-the-art results on HICO-DET and Ambiguous-HOI.

  • 6. Conclusion: DJ-RN combines detailed 3D body and estimated object geometry with separate 2D and 3D representation networks and cross-modal consistency tasks.The method is presented as a 2D-3D joint HOI representation learning paradigm.
  • 6. Conclusion: DJ-RN achieves state-of-the-art results on both HICO-DET and the Ambiguous-HOI benchmark.

Appendices

The appendix visualizes human body-part attentions and estimated 3D spatial configuration volumes.

  • Appendices: Figure 10 presents visualized human body-part attentions together with estimated 3D spatial configuration volumes.

A. Visualized Results

The visualized results illustrate DJ-RN across varied human-object interactions and object scales. They also show examples of estimated object sizes and locations in 3D volumes.

  • A. Visualized Results: DJ-RN handles interactions ranging from holding a small bottle to sitting on a large chair, covering both local and whole-body actions.
  • A. Visualized Results: The visualizations show appropriate object-size and location estimates, including a chair that covers the baby sitting inside it.

B. Prior Object Size and Depth Regularization Factor

The method estimates 3D spatial configuration using object-category priors, including prior size ratios and depth regularization factors collected from volunteers.

  • B. Prior Object Size and Depth Regularization Factor: Object-category priors provide size ratios relative to human shoulder width and depth regularization factors for estimating 3D spatial configuration.These priors cover the 80 COCO object categories used in HICO-DET.
  • B. Prior Object Size and Depth Regularization Factor: Approximately 50 volunteers contributed to the collection of prior object information.Volunteer backgrounds are documented separately in Table 5.

D. Spatial Alignment Illustration

The paper illustrates spatial alignment between 2D and 3D spatial features in latent space, alongside a performance comparison on Ambiguous-HOI.

  • D. Spatial Alignment Illustration: A visualization illustrates how 2D and 3D spatial features are aligned in latent space.The accompanying figure is identified as an illustration of spatial alignment.
  • D. Spatial Alignment Illustration: Figure 11 compares the proposed method with TIN on Ambiguous-HOI.

E. Characteristics of Ambiguous-HOI

Ambiguous-HOI covers diverse human-object interactions with hard ambiguous images and shows that DJ-RN outperforms TIN across various HOIs.

  • E. Characteristics of Ambiguous-HOI: DJ-RN outperforms TIN on various HOIs in Ambiguous-HOI, demonstrating the effectiveness of detailed 2D-3D joint HOI representation.
  • E. Characteristics of Ambiguous-HOI: Ambiguous-HOI contains 87 HOI types spanning 40 verb categories and 48 object categories.Its statistics include selected objects, verbs, and annotated human-object pair counts for each interaction.
Loading 2004.08154v2…