PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Human-Object Interaction Synthesis

Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, C. Karen Liu

arXiv:2312.03913v2cs.CV

TL;DR

Synthesizing realistic human-object interactions in 3D scenes requires language-guided, synchronized human and object motion, including precise contact and scene-aware trajectories. CHOIS uses conditional diffusion with initial states and sparse object waypoints, adding geometry loss and sampling guidance; it reports realistic interactions and integration into a long-horizon scene pipeline.

  • Problem

    Generating realistic, synchronized human-object motion from language and initial states in cluttered 3D scenes remains challenging, particularly when maintaining contact and accommodating diverse objects.

    Source: §1 · pp. 1–2

  • Method

    CHOIS uses a conditional diffusion model conditioned on language, initial human and object states, and sparse object waypoints, with object geometry loss and sampling-time contact guidance.

    Source: §1 · p. 3

  • Results

    CHOIS synthesizes realistic interactions adhering to language and sparse waypoint conditions, with reported performance on FullBodyManipulation and generalization to novel 3D-FUTURE objects.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The interaction module integrates with path planning to synthesize long-horizon, environment-aware interactions from 3D scenes and language input.

    Source: §1 · p. 3

Abstract

from arXiv · show

Synthesizing semantic-aware, long-horizon, human-object interaction is critical to simulate realistic human behaviors. In this work, we address the challenging problem of generating synchronized object motion and human motion guided by language descriptions in 3D scenes. We propose Controllable Human-Object Interaction Synthesis (CHOIS), an approach that generates object motion and human motion simultaneously using a conditional diffusion model given a language description, initial object and human states, and sparse object waypoints. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Naively applying a diffusion model fails to predict object motion aligned with the input waypoints; it also cannot ensure the realism of interactions that require precise hand-object and human-floor contact. To overcome these problems, we introduce an object geometry loss as additional supervision to improve the matching between generated object motion and input object waypoints; we also design guidance terms to enforce contact constraints during the sampling process of the trained diffusion model. We demonstrate that our learned interaction module can synthesize realistic human-object interactions, adhering to provided textual descriptions and sparse waypoint conditions. Additionally, our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.

1 Introduction

CHOIS targets realistic, language-guided human-object interactions in cluttered 3D scenes, where both motions must be synchronized and contacts maintained. It uses sparse object waypoints with a conditional diffusion model, adding geometry loss and sampling guidance to improve waypoint adherence and interaction realism.

1 Introduction

  • Language-guided human behavior synthesis in 3D environments is motivated by applications requiring coordinated interaction among people, objects, and surroundings.Such behaviors also express purpose, which language can articulate.Source: §1 · pp. 1–2
  • Existing approaches leave a gap: some address static scenes or smaller objects, while others require past interaction states or complete object-motion sequences.The target setting synthesizes both human and object motion from initial states while accommodating cluttered environments.Source: §1 · p. 2
  • CHOIS conditions a conditional diffusion model on language, initial human and object states, and sparse object waypoints to generate synchronized motions.The waypoints are derived from 3D scene geometry and ground the generated trajectories in environmental constraints.Source: §1 · pp. 2–3
  • Object geometry loss improves object-motion matching to input waypoints, while sampling-time guidance enforces contact constraints and enhances interaction realism.The guidance addresses hand-object contact, foot floating, and object-floor penetration.Source: §1 · p. 3
  • The approach synthesizes realistic interactions on FullBodyManipulation and generalizes to novel 3D-FUTURE objects.The module is also integrated into a pipeline for long-horizon interactions from 3D scenes and language input.Source: §1 · p. 3

2 Related Work

Related work spans language-conditioned motion generation, scene-aware synthesis, and dynamic human-object interaction modeling. CHOIS differs by generating both human and object motion from language and sparse object waypoints, supporting integration with path planning for long-term interactions.

2 Related Work

  • Language-conditioned motion work extends generative human-motion modeling, while CHOIS generates both object and human motion from text.Prior work includes VAE-based and diffusion-based text-conditioned motion synthesis.Source: §2 · p. 3
  • Scene-aware methods include paired-data interaction synthesis, path-planning-guided motion generation, and reinforcement-learning policies, often focused on static scenes or objects.CHOIS instead targets dynamic objects and conditions its interaction module on sparse object waypoints for pipeline integration.Source: §2 · p. 4
  • Dynamic interaction methods include hand-motion synthesis, small-object manipulation, reinforcement-learning approaches, and models conditioned on past states or object-motion sequences.The cited approaches have limitations in object diversity or cannot synthesize interactions in 3D scenes solely from initial states.Source: §2 · p. 4
  • Concurrent CG-HOI and HOI-Diff synthesize human and object motion from text, whereas CHOIS combines text with sparse waypoints to connect with path planning.The work also uses FullBodyManipulation, described as offering greater data scale and motion quality than BEHAVE.Source: §2 · p. 4

3 Method

The method uses conditional diffusion to generate synchronized human and object motion from language, initial states, object geometry, and sparse waypoints, then applies sampling-time guidance to improve interaction constraints.

3.1 Data Representation

  • Human poses, object transformations, BPS geometry features, initial states, and sparse object waypoints form the motion and conditioning representations.The geometry features are concatenated with masked motion conditions; language embeddings also condition the denoising network.Source: §3.1 · pp. 5–6

3.2 Interaction Synthesis Model

  • A conditional diffusion model jointly predicts human and object motion and hand-contact labels, using a transformer denoiser conditioned on geometry, masked states, noise level, and language.The predicted left- and right-hand contact labels guide sampling toward more realistic hand-object contact.Source: §3.2 · p. 6
  • An object geometry loss compares transformed rest-pose vertices with ground-truth object vertices to improve object-motion prediction.The loss uses 100 sampled points from 1,024 BPS nearest neighbors to capture a rough object outline.Source: §3.2 · p. 7

3.3 Guidance

  • Sampling-time reconstruction guidance uses analytical constraint gradients; test-time guidance avoids adding costly, difficult-to-balance contact and floor losses during training.The method perturbs the predicted clean representation using gradients with respect to the noisy input.Source: §3.3 · pp. 7–8
  • Hand-object guidance uses predicted contact labels to mask distances between each hand and its nearest object-mesh points.Contact is identified when a predicted label exceeds 0.95.Source: §3.3 · p. 8

4 Experiments

The experiments compare CHOIS with adapted baselines, assess perceptual quality and guidance ablations, and demonstrate long-term interaction synthesis in 3D scenes.

4.1 Datasets

  • The FullBodyManipulation dataset provides 10 hours of paired motion across 15 objects; 3D-FUTURE tests generalization to 17 furniture objects using motion from matching categories.Articulated-object sequences are excluded, and the FullBodyManipulation split uses 15 training subjects and two test subjects.Source: §4.1 · p. 9

4.2 Evaluation Metrics

  • Evaluation covers waypoint matching, human-motion quality, interaction quality, and deviation from ground-truth motion.Metrics include waypoint errors, foot sliding and height, FID and R-precision, hand-contact and penetration measures, and position and rotation errors.Source: §4.2 · pp. 9–10

4.3 Results

  • CHOIS outperforms the baselines on 3D-FUTURE, while on FullBodyManipulation object geometry loss improves condition matching and inference guidance improves contact and floor realism.Guidance improves contact accuracy and reduces hand-object penetration and foot floating.Source: Table 2 · p. 11, §4.3 · pp. 11–12, Table 1 · p. 11
  • InterDiff struggles with waypoint and text matching, MDM lacks realistic contact constraints, and GT-OMOMO still exhibits foot floating despite comparable interaction metrics.Lin-OMOMO has zero deviation at sparse input locations because it does not alter the supplied object motion; Pred-OMOMO remains inferior to CHOIS in condition matching and human-motion quality.Source: §4.3 · pp. 11–12

4.4 Ablation Study

  • Ablations show that hand-object guidance matters for contact percentage and feet-floor guidance matters for foot height; object-floor guidance targets uncommon penetration artifacts.The study does not ablate object-floor guidance because object-floor penetration is uncommon.Source: Table 3 · p. 12, §4.4 · p. 13

4.3 Results

  • The perceptual evaluation uses 800 sequence pairs, with each pair reviewed by 10 AMT workers, to assess text consistency and overall motion quality.The results are presented in Figure 4.Source: §4.3 · pp. 12–13

4.5 Application

  • In Replica scenes, Habitat collision-free paths become model waypoints at fixed 30-frame intervals, and Figure 6 varies text or waypoints to demonstrate control.The module synthesizes long-term human-object motion from language, scene information, and initial human and object states.Source: §4.5 · pp. 13–14, Figure 5 · p. 13, Figure 6 · p. 14

5 Conclusion

CHOIS generates synchronized human and object motions conditioned on language and sparse object waypoints. Object geometry loss improves object-motion generation, while sampling guidance enhances realism and supports integration into long-term interaction synthesis.

5 Conclusion

  • The learned interaction module integrates into a pipeline that synthesizes long-term interactions from language and 3D scenes.Source: §5 · p. 14
  • CHOIS generates synchronized human and object motions that resonate with the provided language descriptions.Source: §5 · p. 14
  • Object geometry loss improves object-motion generation, and sampling-time guidance enhances the realism of generated interactions.Source: §5 · p. 14

Related papers

Loading 2312.03913v2…