PaperlayerView exact revision on arXiv

Source-linked AI summary

InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion

Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, Liang-Yan Gui

arXiv:2308.16905v1cs.CVcs.AIcs.GR

TL;DR

Existing HOI synthesis has often omitted comprehensive whole-body motion, dynamic objects, or physical validity, motivating a more demanding forecasting task. InterDiff addresses it with diffusion generation plus physics-informed correction based on simple contact-relative motion, and experiments report realistic, diverse, long-term predictions across datasets. Its demonstrated scope leaves more complex environments, articulated or deformable objects, multiple dynamic objects, and interactions with other humans for future work.

  • Problem

    Existing HOI synthesis often lacks comprehensive whole-body interactions with dynamic objects and physically valid contact behavior.

  • Method

    InterDiff combines a DDPM-based interaction diffusion generator with a physics-informed interaction corrector that predicts object motion relative to human contact regions.

  • Results

    InterDiff produces realistic, diverse, and long-term HOI predictions across multiple datasets, while reducing contact floating and penetration artifacts.

  • Takeaways & Limitations

    Simple contact-relative motion provides a practical physics prior for generating plausible interactions with dynamic objects.

  • Takeaways & Limitations

    The demonstrated framework does not yet cover more than one dynamic object, articulated or deformable objects, more complex environments, or interactions with other humans.

Abstract

from arXiv · show

This paper addresses a novel task of anticipating 3D human-object interactions (HOIs). Most existing research on HOI synthesis lacks comprehensive whole-body interactions with dynamic objects, e.g., often limited to manipulating small or static objects. Our task is significantly more challenging, as it requires modeling dynamic objects with various shapes, capturing whole-body motion, and ensuring physically valid interactions. To this end, we propose InterDiff, a framework comprising two key steps: (i) interaction diffusion, where we leverage a diffusion model to encode the distribution of future human-object interactions; (ii) interaction correction, where we introduce a physics-informed predictor to correct denoised HOIs in a diffusion step. Our key insight is to inject prior knowledge that the interactions under reference with respect to contact points follow a simple pattern and are easily predictable. Experiments on multiple human-object interaction datasets demonstrate the effectiveness of our method for this task, capable of producing realistic, vivid, and remarkably long-term 3D HOI predictions.

1. Introduction

The paper formulates 3D HOI prediction as whole-body human and dynamic-object forecasting under physical-validity constraints. InterDiff combines diffusion generation with physics-informed interaction correction, using simpler contact-relative object motion to improve plausible long-term synthesis.

  • Motivation: Existing HOI synthesis often focuses on ego-centric hand-object interactions, small objects, simplified skeletons, or static object dynamics.These limitations motivate forecasting more comprehensive whole-body interactions with dynamic objects.
  • Task Challenges: The proposed task jointly models mesh-based whole-body movements and object dynamics across varied object shapes.Predictions should fit the object surface during contact while avoiding body-object penetration.
  • Task Challenges: Directly extending motion diffusion can produce floating contact and penetration artifacts, with errors accumulating during long-term autoregressive inference.Existing approaches commonly add physical constraints through post-hoc optimization or simulation.
  • Key Insight: InterDiff injects intuitive physics into a diffusion model by exploiting short-term object motion relative to contact points, which is simple and nearly deterministic.For example, a juggling ball can move up and down relative to the juggler’s hand despite complex global motion.
  • Method: InterDiff combines interaction diffusion, which models future HOI distributions, with interaction correction, which predicts contact-relative object motion and promotes contact fitting.The components can be trained separately and conform during inference without fine-tuning.
  • Contributions: The framework is presented as the first mesh-based 3D HOI prediction task and diffusion framework using past motion and shape information to generate future interactions.The proposed corrector incorporates physics priors to infill denoising generation and targets generalization to unseen objects and long-term inference.

2. Related Work

Related work spans diffusion-based motion generation, human-object interaction modeling, and statistical 3D body animation. InterDiff differs by jointly modeling human and object dynamics for HOI prediction.

  • Denoising Diffusion Models: Denoising diffusion models add noise to data samples and generate denoised samples through an iterative reverse process.Prior work extends this framework to human motion generation and motion infilling.
  • Denoising Diffusion Models: Motion diffusion models provide a foundation that the paper extends from human motion prediction to 3D human-object interaction prediction.The related comparison includes PhysDiff, which injects a physics-simulation motion imitation policy into diffusion.
  • Human-Object Interaction: Human-object interaction research has mainly addressed detection, reconstruction, static-scene interaction, or hand-object synthesis rather than comprehensive dynamic HOIs.This work targets the less-developed setting of jointly modeling interacting humans and objects.
  • Human Motion and Object Dynamics: Generative models including VAEs, GANs, normalizing flows, and diffusion models have supported skeleton-based motion prediction and 3D body animation.The paper uses SMPL parameters to drive the 3D human mesh on BEHAVE.
  • Human Motion and Object Dynamics: Unlike prior motion-generation work centered on human bodies, the proposed method jointly models human and object dynamics in their interaction.This joint modeling is the paper’s distinguishing connection between human motion generation and HOI prediction.

3. Methodology

InterDiff combines conditional interaction diffusion with physics-informed correction to generate future 3D human-object interactions. Its correction pipeline detects contact or penetration problems, predicts object motion in contact-centered reference systems, and blends the correction into denoising.

  • Problem formulation: InterDiff formulates HOI prediction as generating future mesh-based human and object states conditioned on object shape and historical motion, while requiring physical validity.Physical validity requires natural surface conformity during contact and avoidance of body-object penetration.
  • Interaction diffusion: The diffusion model samples future HOIs from noise through iterative denoising, using a transformer conditioned on past HOI states and a PointNet object-shape embedding.The model directly estimates the clean HOI at each reverse-diffusion step.
  • Correction schedule: A correction scheduler activates interaction correction when penetration exceeds a threshold or contact is absent, applying corrections during selected late denoising iterations.For mesh HOIs, contact and penetration states are computed from human-object geometry and signed distance fields.
  • Correction schedule: The scheduler selects a ground or human-marker reference system from contact information, using markers to index candidate human surface reference points efficiently.The ground reference is retained without contact; otherwise, the selected reference is the contacting human-surface marker.
  • Interaction prediction: The interaction predictor transforms past object motion into ground- and marker-relative spatial-temporal graphs, uses an STGNN to forecast future motion, then transforms the selected prediction back to ground coordinates.The resulting corrected HOI is blended with the denoised HOI before the reverse process continues.

4. Experiments

Experiments evaluate InterDiff on three 3D human-object interaction datasets using task-specific metrics, baselines, quantitative comparisons, qualitative visualizations, and ablations. The results show improved physical plausibility, generalization to unseen objects and datasets, diversity, and long-term forecasting, with interaction correction reducing artifacts and improving motion accuracy.

  • Experimental Setup: Evaluation covers BEHAVE, GRAB, and the Human-Object Interaction dataset, including cross-dataset testing from BEHAVE to GRAB.BEHAVE contains ordinary objects, GRAB focuses on whole-body grasping of small objects, and the Human-Object Interaction dataset includes unseen instances.
  • Experimental Setup: The study evaluates joint-position, object-translation, object-rotation, and object-keypoint errors alongside comparisons with VAE-, RNN-, and prior skeletal baselines.The reported metrics include MPJPE-H, Trans. Err., Rot. Err., and MPJPE-O.
  • Quantitative Results: InterDiff outperforms two baselines across all metrics on both BEHAVE and the Human-Object Interaction dataset.The Human-Object Interaction evaluation specifically tests challenging unseen instances in the training data.
  • Quantitative Results: Interaction correction produces more plausible interactions, reduces penetration artifacts, and improves object-motion precision; improved object motion also improves predicted human motion.Under Best-of-Many evaluation, the full method improves over pure diffusion with more samples and longer horizons.
  • Qualitative Results: InterDiff generalizes to unseen objects and a different dataset without fine-tuning, while generating diverse and legitimate HOIs.Qualitative results cover unseen objects, skeletal representations, small-object grasping, and diverse predictions.
  • User Study: 67.8% of human judges preferred InterDiff against baselines in a double-blind study of 100-frame future interactions.The study used 30 pairwise comparisons with 23 human judges.
  • Ablation Study: The full model is superior in long-horizon ablations, whereas removing correction, reference transformation, contact and penetration losses, or scheduling causes floating or penetration artifacts.The ablations show that reference-system transformation, contact and penetration losses, and scheduled correction each contribute to stable physical interactions.

5. Discussions

The discussion frames 3D human-object interaction prediction as a novel task requiring physical validity and presents InterDiff as a diffusion framework with physics-informed correction. It reports vivid, diverse predictions with reduced artifacts while identifying remaining contact inconsistency and broader environment generalization as boundaries.

  • Discussions: The paper introduces 3D human-object interaction prediction as a task addressing intricate real-world interaction challenges.The task concerns mesh-based whole-body movements and object dynamics with physically valid contact.
  • Discussions: InterDiff injects physics priors into diffusion to reduce contact floating and penetration with minimal additional computational cost.The framework uses interaction diffusion and an interaction correction step rather than post-optimization or physics simulation.
  • Discussions: The demonstrated scope excludes more complex environments involving multiple dynamic objects, articulated or deformable objects, or other humans.These settings are identified as future directions for generalization.
  • Discussions: Contact inconsistency remains in some generated results, although interaction correction largely alleviates the artifact.The corrected outputs can also be directly applied to post-optimization to improve quality.

A. Visualization Video

The visualization demos show that InterDiff can forecast diverse, extremely long-term, physically valid HOIs while maintaining smooth and vivid motion. They also illustrate the effect of physics-informed correction and the contact-based coordinate system.

  • Visualization Video: Without physics-informed correction, pure diffusion produces implausible interactions, whereas InterDiff maintains physical validity in extremely long-term forecasts.The demos report this qualitative pattern as consistent with the main-paper experiments.
  • Visualization Video: InterDiff produces diverse, smooth, and vivid HOIs, including cases where motion-capture ground truth contains jitter patterns.The demos provide additional visualizations of 3D HOI forecasting and emphasize the contact-based coordinate system.

B.1. Interaction Diffusion

Interaction Diffusion uses a reverse diffusion model to generate future human and object motion while training against ground-truth denoised motion and temporal velocity consistency.

  • B.1. Interaction Diffusion: The diffusion model predicts denoised human and object motion from noisy interactions, conditioning on timestep and context.The reverse process outputs x_0(t) through a function of x_t, t, and c.
  • B.1. Interaction Diffusion: Separate reconstruction objectives supervise the generated human motion and object motion against their respective ground-truth data.The objectives compare h_0(t) with h and o_0(t) with o.
  • B.1. Interaction Diffusion: Velocity regularizations promote smooth temporal transitions by penalizing differences between consecutive predicted human and object frames.The regularizers compare shifted future and preceding motion sequences for both humans and objects.

B.2. Interaction Correction

Interaction Correction represents object motion across global and contact-centered reference systems, processes it with graph convolutions in the frequency domain, and applies motion and physical-validity objectives.

  • B.2. Interaction Correction: The interaction predictor represents object motion as a spatial-temporal graph across one ground reference system and marker-based human reference systems.The representation includes past and future frames and uses body markers as contact-related reference systems.
  • B.2. Interaction Correction: The predictor transforms the graph into the frequency domain with DCT bases, applies spatial-temporal graph convolutions, and transforms the result back.Future object motion is then indexed using an informed reference system and converted back to the ground reference.
  • B.2. Interaction Correction: Its learning objectives supervise recovered object motion and temporal velocity consistency over the reconstructed sequence.The recovered motion includes the past sequence together with predicted future frames.
  • B.2. Interaction Correction: Contact and penetration losses encourage surface proximity while penalizing mutual penetration between human and object meshes.These losses are applied for mesh representations, but not for skeletal representations.

C. Additional Details of Experimental Setup

The additional setup specifies loss-term weights and evaluates long-term forecasting by autoregressively generating 100 future interaction frames with 10 DCT bases.

  • C. Additional Details of Experimental Setup: Interaction diffusion uses loss weights (λh, λo, λvh, λvo) = (1, 0.1, 0.2, 0.02).
  • C. Additional Details of Experimental Setup: Interaction prediction uses loss weights (λo, λvo, λc, λp) = (1, 0.1, 1, 0.1).
  • C. Additional Details of Experimental Setup: 100 future interaction frames are generated autoregressively to evaluate long-term forecasting on BEHAVE.The ablation balances rotation and translation performance using 10 DCT bases.

D. Additional Ablation Studies

The DCT-basis ablation finds a trade-off between translation and rotation errors, motivating the choice of 10 bases.

  • D. Additional Ablation Studies: As the number of DCT bases increases, translation error increases while rotation error decreases.The authors attribute this trade-off to rotation requiring more parameters, whereas translation relative to the reference system is easy to model.
  • D. Additional Ablation Studies: 10 DCT bases are selected to balance translation and rotation errors.
Loading 2308.16905v1…