PaperlayerView exact revision on arXiv

Source-linked AI summary

CG-HOI: Contact-Guided 3D Human-Object Interaction Generation

Christian Diller, Angela Dai

arXiv:2311.16097v2cs.CV

TL;DR

CG-HOI addresses the gap in generating dynamic 3D human-object interactions from text by jointly modeling human motion, object motion, and contact. Its contact-guided diffusion process produces realistic, physically plausible sequences, supports human-motion generation from object trajectories without retraining, and applies to static 3D scene scans.

  • Problem

    Existing motion-generation approaches often model humans in isolation or static environments, while realistic interactions require interdependent human and object motion and full-body contact.

  • Method

    CG-HOI jointly models human, object, and contact representations in a cross-attended diffusion process and uses predicted contact to guide synthesis.

  • Results

    CG-HOI generates realistic, diverse, physically plausible interactions, improves performance over baselines, and supports object-trajectory-conditioned human motion without retraining.

  • Takeaways & Limitations

    The learned human-object interdependencies enable text-conditioned HOI generation, conditional human-motion synthesis from object trajectories, and interaction placement in static real-world 3D scans.

  • Takeaways & Limitations

    The method handles realistic interactions with a single object, requires expensive 3D HOI captures and manual text annotations, and does not model multiple objects together.

Abstract

from arXiv · show

We propose CG-HOI, the first method to address the task of generating dynamic 3D human-object interactions (HOIs) from text. We model the motion of both human and object in an interdependent fashion, as semantically rich human motion rarely happens in isolation without any interactions. Our key insight is that explicitly modeling contact between the human body surface and object geometry can be used as strong proxy guidance, both during training and inference. Using this guidance to bridge human and object motion enables generating more realistic and physically plausible interaction sequences, where the human body and corresponding object move in a coherent manner. Our method first learns to model human motion, object motion, and contact in a joint diffusion process, inter-correlated through cross-attention. We then leverage this learned contact for guidance during inference to synthesize realistic and coherent HOIs. Extensive evaluation shows that our joint contact-based human-object interaction approach generates realistic and physically plausible sequences, and we show two applications highlighting the capabilities of our method. Conditioned on a given object trajectory, we can generate the corresponding human motion without re-training, demonstrating strong human-object interdependency learning. Our approach is also flexible, and can be applied to static real-world 3D scene scans.

1. Introduction

CG-HOI addresses text-to-3D human-object interaction generation by jointly modeling human motion, object motion, and contact to produce realistic, physically plausible sequences.

  • 1. Introduction: Realistic human-object interaction requires interdependent motion because objects constrain human actions and human actions can affect object motion.Prior approaches often generate humans in isolation or assume a static environment throughout the sequence.
  • 1. Introduction: CG-HOI jointly predicts 3D human body motion and object motion from text, explicitly modeling contact as a bridge between them.Contact is represented by distances from the human body surface to the closest point on the interacted object's surface.
  • 1. Introduction: Contact guidance encourages semantic coherence and physical plausibility, including discouraging unsupported objects from floating.The method uses predicted contact during inference to refine synthesized interactions.
  • 1. Introduction: Evaluations on BEHAVE and CHAIRS report realistic and physically plausible interactions, alongside applications for object-trajectory conditioning and static 3D scene scans.The object-trajectory application generates human motion without retraining.
  • 1. Introduction: The method jointly models human, object, and contact modalities through cross-attention in a diffusion process, with contact-based guidance during synthesis.Object motion hypotheses for multiple human body parts are aggregated according to predicted contact distances.

2. Related Work

Prior work largely models human motion in isolation, static scenes, or limited hand-object settings, whereas CG-HOI targets text-conditioned full-body human-object motion with contact.

  • 3D Human Motion Generation: Existing 3D human motion methods use recurrent, graph, attention, or diffusion models, with deterministic or stochastic generation and optional text conditioning.These methods primarily focus on human motion rather than jointly modeling interacting object motion.
  • 3D Human Motion in Scenes: Scene-based methods model human placement, context, or walking and sitting in static environments, limiting interactions when the environment moves.The static-scene assumption particularly focuses generated behavior on walking or sitting movements.
  • 3D Human Motion in Scenes: Other methods generate human motion around a single static object, condition on object motion, or synthesize interactions from an observed sequence.CG-HOI instead formulates a flexible text-conditioned approach modeling both human and object motion.
  • Contact Prediction for Human-Object Interactions: Object-motion methods commonly predict objects in isolation, despite interactions typically depending on human motion.This limits the interdependency represented in generated interactions.
  • Contact Prediction for Human-Object Interactions: Contact prediction has mainly addressed fine-grained hand-object interaction, while whole-body interaction generation requires full human and object surface geometry.CG-HOI models full-body contact jointly with human and object motion from object geometry and text.

3. Method Overview

CG-HOI takes text and object geometry as conditions and generates framewise human, object, and contact representations through a jointly guided diffusion process.

  • 3. Method Overview: The model jointly generates human, object, and contact sequences in a denoising diffusion process, reasoning about all three modalities during training and inference.This joint formulation supports semantically meaningful human-object interaction sequences.
  • 3. Method Overview: Given text T and static object geometry G, CG-HOI generates F frames containing object transformations, human body-surface representations, and human-object contact.The sequences are denoted H, O, and C for human, object, and contact representations.
  • 3. Method Overview: Cross-attention modules exchange information among human motion, object motion, and contact, while contact-weighted hypotheses emphasize body parts closer to the object.Multiple object-motion hypotheses are aggregated according to predicted contacts.
  • 3. Method Overview: During inference, predicted contact guides each diffusion step by encouraging recomputed human-object contact to match the predicted contact.This refinement produces more realistic interactions overall.

4. Human-Object Interaction Diffusion

CG-HOI jointly diffuses human motion, object transformations, and contact distances conditioned on object geometry and text. Cross-attention and contact-weighted object hypotheses model how nearby body regions influence object motion.

  • 4. Human-Object Interaction Diffusion: Contact is represented as distances from uniformly distributed human-surface markers to the closest points on object geometry.The representation uses M = 128 motion markers on the body surface.
  • 4. Human-Object Interaction Diffusion: CG-HOI jointly generates human poses, object transformations, and contact distances through a diffusion process conditioned on object geometry and text.The noised sequence concatenates human, object, and contact representations, while geometry and text provide conditioning.
  • 4. Human-Object-Contact Cross-Attention: Cross-attention exchanges information among human, object, and contact modalities using modality-specific latent features.For human features, object and contact features form the key and value inputs, and analogous operations update the other modalities.
  • 4.3. Contact-Based Object Transform Weighting: Object motion hypotheses are generated for each human contact point and weighted inversely by predicted contact distance.This emphasizes body regions closest to the object when aggregating object transformations.
  • 4.4. Loss Formulation: Training minimizes separate human, object, and contact reconstruction terms on the jointly predicted sequence.The common objective uses λh = 1.0, λo = 0.9, and λc = 0.9.

5. Interaction Generation

During synthesis, CG-HOI uses contact-based diffusion guidance to correct inconsistencies between predicted human and object motion. The same trained model can generate human motion conditioned on an object trajectory without retraining.

  • 5. Interaction Generation: Novel interactions are generated by reversing diffusion from Gaussian noise while combining contact-weighted object transformations with custom guidance.The process iteratively estimates a cleaned sample from the noised sequence.
  • 5.1. Contact-Based Diffusion Guidance: The guidance cost compares predicted and actual contact distances from the denoised human, object, and contact predictions.Its input is zt = [ht, ot, ct], and the gradient ∇ztG(zt) guides the diffusion mean.
  • 5.1. Contact-Based Diffusion Guidance: Contact-based diffusion guidance corrects physical contact inconsistencies during inference without explicit post-processing.The guidance uses a contact-distance cost and applies its gradient densely across diffusion steps.
  • 5.2. Conditioning on Object Trajectory: Given an object sequence and text description, the trained model generates the corresponding human sequence without retraining.The method injects the given object motion O′ into the diffusion process at every inference step.

6. Results

CG-HOI generates realistic and physically plausible human-object interactions across BEHAVE and CHAIRS, outperforming baselines and benefiting from contact-based design choices. Additional applications condition human motion on object trajectories and populate static 3D scene scans, while limitations remain for multi-object interactions and training data requirements.

  • 6.3. Comparison to Baselines: CG-HOI outperforms MDM and InterDiff in all three evaluation settings, indicating strong learned correlation between human and object motion.The settings include human-only, motion-conditioned, and text-conditioned comparisons.
  • 6.4. Ablation Studies: Joint contact prediction with cross-attention improves performance by sharing information among human, contact, and object modalities.The ablation results associate this design with more realistic human-object contact.
  • 6.3. Comparison to Baselines: Participants significantly favor CG-HOI over baselines for overall realism and text coherence.The perceptual study compares 10 baseline-versus-method pairs with 32 participants.
  • 6.4. Ablation Studies: Joint contact prediction, contact-based weighting, and inference-time guidance together improve fidelity, physical plausibility, and interaction realism.The ablations compare joint versus separate contact prediction, no contact weighting, and contact-based guidance.
  • 6.5. Applications: Given an object trajectory, CG-HOI generates corresponding human motion without retraining.This application uses the learned human-object correspondence and contact distance predictions.
  • 6.5. Applications: CG-HOI generates interactions for segmented objects in static 3D scene scans, extending the method beyond isolated object settings.The application uses semantic object segmentation from a ScanNet++ scene.
  • 6.6. Limitations: The method focuses on single-object interactions, requires expensive 3D HOI captures, and depends on manual text annotations.The authors identify multi-object modeling, weak supervision, and more specific prompts as possible future directions.

7. Conclusion

The paper concludes that contact modeling within a diffusion process enables realistic, diverse, and physically plausible dynamic human-object interactions. Learned human-object correlations also support conditioning human motion on object sequences and improve performance over baselines.

  • 7. Conclusion: CG-HOI jointly models human motion, object motion, and contact through cross-attention in a diffusion process.Contact-based object transformation weighting and diffusion guidance further refine the generated interactions.
  • 7. Conclusion: Extensive evaluation confirms the fidelity and diversity of generated sequences and shows improved performance compared to baselines.The conclusion attributes these results to the contact-based modeling approach.
  • 7. Conclusion: The learned human-object correlation enables conditional human-motion generation from given object sequences.This capability does not require additional training.

A. Additional Qualitative Results

Additional qualitative results show that CG-HOI generates diverse human-object interactions for fixed text prompts and visualizes object geometry, text conditions, and generated sequences. The examples include varied ways of moving or sitting on a stool.

  • A. Additional Qualitative Results: CG-HOI produces additional 3D human-object interaction sequences from object geometry and text conditions.The supplementary visualization places the object geometry and text condition alongside the generated sequence.
  • A. Additional Qualitative Results: For fixed prompts such as “Move a stool” and “Sit on a stool,” CG-HOI visually demonstrates diverse interactions.The training data associates moving with hands or feet, while sitting may also involve the butt.

B. Additional Quantitative Results

Additional evaluations assess physical plausibility, contact prediction, interaction novelty, and robustness to body representation. The method matches ground-truth penetration and floating levels and generates novel rather than memorized interactions.

  • Physical Plausibility: Penetration and floating in generated sequences are similar to the ground-truth training data.The evaluation considers both penetration and non-contact ratios across frames and vertices.
  • Contact Evaluation: Contact predictions are evaluated with precision, recall at ≤5cm, and mean contact-distance ℓ1 error.The evaluation focuses on body parts within 1m of the object; better contact prediction corresponds with better HOI generation.
  • Novelty Analysis: Generated interactions are novel: distances to the closest training sequences are mostly larger than intra-training-set distances.The novelty analysis generates approximately 500 sequences per dataset and compares their nearest training samples.
  • Representation Analysis: Using HumanML3D skeletons and fitting SMPL afterward degrades performance because contact guidance is less effective.The authors also observe slight pose jitter and foot skating in the ground-truth data, especially BEHAVE sequences captured with Kinect sensors.
  • Qualitative Evaluation: Qualitative results show diverse, realistic interactions with reduced floating and penetration artifacts through contact modeling.The additional evaluation is conditioned on object geometry and short action descriptions.

C. Baseline Evaluation Setup

Because no prior method directly generates 3D human-object interactions from text and object geometry, the evaluation compares CG-HOI with the two closest approaches across multiple settings.

  • Comparison Rationale: No previous approach directly models 3D human-object interactions from text and object geometry, so comparisons use the two closest methods.The authors compare methods in multiple settings to provide a fair evaluation.
  • InterDiff: InterDiff is evaluated in its native setting, generating likely interaction continuations from observed interaction sequences and geometry without text.The comparison uses InterDiff’s full diffusion and post-processing refinement approach.
  • InterDiff: InterDiff is compared in both evaluation scenarios and on both datasets.
  • MDM: MDM is compared for human-only motion generation using SMPL parameters against CG-HOI and InterDiff outputs.CG-HOI’s comparison condition uses its model trained for full human-object interaction generation.
  • Body Representation: HumanML3D skeletons followed by SMPL fitting yield degraded performance relative to CG-HOI’s SMPL-body representation.The comparison attributes this degradation to less effective contact guidance.

D. Fidelity and Diversity Metrics

The evaluation adapts established human-motion fidelity and diversity metrics to both human-only and full human-object representations. The implementation uses SMPL-based data and point-cloud object geometry encoded with PointNet.

  • Metric Definition: Full HOI evaluation uses 216 features: 79 SMPL parameters, 128 contact distances, and 9 object transformations.Human-only evaluation uses 79 features, with metric adaptations for the different input dimensions.
  • Metric Definition: The fidelity and diversity measures are R-Precision, FID, Diversity, and MultiModality, adapted from human-motion-generation practice.The same evaluation networks are used, with input dimensions adjusted for each representation.
  • Network Architecture: The network architecture contains encoder, bottleneck, and decoder components.
  • Datasets: CHAIRS provides approximately 1,300 SMPL-X interaction sequences involving 46 subjects and 81 chair or sofa types.Sequences contain six actions and manually annotated textual descriptions.
  • Datasets: BEHAVE contains approximately 130 longer SMPL-H sequences from eight participants interacting with 20 object types.The dataset includes per-frame object transformations and canonical object geometries.
  • Object Representation: Object geometry is represented by 256 uniformly sampled surface points processed with a PointNet encoder.Each object category is sampled once during preprocessing and reused during training and inference.
Loading 2311.16097v2…