PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Up Dynamic Human-Scene Interaction Modeling

Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Siyuan Huang

arXiv:2403.08629v2cs.CV

TL;DR

HSI modeling lacks datasets that jointly provide scalable coverage, high-quality motion, and realistic dynamic contacts. The paper introduces TRUMANS and a scene- and action-conditioned autoregressive diffusion model, achieving motion quality close to MoCap data with strong zero-shot generalization. Its main scope boundary is that it does not reliably generate interaction behaviors absent from training.

  • Problem

    HSI research lacks scalable, high-quality datasets that capture diverse realistic interactions, dynamic objects, and 3D contacts.

  • Method

    The paper constructs TRUMANS and trains an autoregressive conditional diffusion model using scene context and action labels to generate arbitrary-length HSI sequences.

  • Results

    The generated motions closely mirror original MoCap quality and outperform existing methods in quality and zero-shot generalizability on unseen 3D scenes.

  • Takeaways & Limitations

    TRUMANS supports HSI motion synthesis as well as human pose and contact estimation through broad, realistic interaction data.

  • Takeaways & Limitations

    The method cannot reliably generate human-object interaction behaviors beyond those represented in its training set.

Abstract

from arXiv · show

Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

1. Introduction

HSI modeling is limited by a shortage of high-quality, scalable data that captures realistic human motion, dynamic objects, and 3D contact. The paper addresses this gap with TRUMANS and a scene- and action-conditioned diffusion model, reporting strong generalization and motion quality.

  • Motivation: Existing HSI datasets trade off motion quality, scalability, scene diversity, or realistic dynamic contacts.MoCap systems provide high-quality motion but often lack diverse HSI coverage, while RGBD and synthetic datasets have tracking or realism limitations.
  • Contributions: TRUMANS provides over 15 hours of motion-captured interactions across 100 indoor scenes, including whole-body motion and part-level object dynamics.The dataset emphasizes realistic contact and extends physical captures with accurate virtual scene models and human- and object-level augmentations.
  • Contributions: The proposed autoregressive conditional diffusion model generates arbitrary-length motion using scene and action conditions.Its local scene perceiver supports 3D-aware collision avoidance in cluttered environments.
  • Results: Across evaluations, the method exceeds existing synthesis approaches in motion quality and zero-shot generalizability on unseen 3D scenes.Generated motions closely approximate original motion-captured data, while TRUMANS also supports pose and contact estimation benchmarks.
  • Contributions: The work contributes a scalable MoCap HSI dataset and a controllable diffusion-based generator that can produce MoCap-quality motion in novel environments.The contributions combine dataset diversity, arbitrary-length generation, scene and action conditioning, and extensive empirical evaluation.

2. Related Work

Prior HSI datasets struggle to combine accurate motion capture, scalable scene diversity, and realistic dynamic interactions. TRUMANS addresses this balance through optical capture in replicated environments, while its conditional autoregressive diffusion model targets controllable arbitrary-length generation.

  • HSI Datasets: Image- and RGBD-based HSI datasets improve accessibility but often produce noisy 3D poses.PROX uses scene scans to constrain SMPL-X estimation, yet image-based capture remains limited in motion quality.
  • HSI Datasets: IMU and optical MoCap improve capture quality but remain constrained by scalability, static scenes, or single-object interactions.These limitations reduce coverage of complex behaviors such as navigating cluttered spaces or performing concurrent actions.
  • HSI Datasets: Synthetic datasets offer cost efficiency and adaptability but do not fully capture realistic 3D interactions, especially dynamic object poses.Their simulated environments remain limited in representing the full range of human-scene interactions.
  • HSI Datasets: TRUMANS combines optical motion capture with replicated synthetic environments to improve interaction fidelity, scalability, and scenario diversity.The setup captures humans and objects in complex HSIs while also enabling photorealistic renderings.
  • HSI Generation: Existing HSI generation research uses diffusion, cVAE, autoregressive, and semantic-conditioning techniques for temporal motion synthesis.These directions include arbitrary-length generation and action-label or language-based control.
  • HSI Generation: The paper combines diffusion with autoregressive conditioning to generate arbitrary-length HSI motion with improved controllability.The model conditions generation on scene and action information rather than treating motion synthesis as an unconditioned temporal process.

3. TRUMANS Dataset

TRUMANS is built by capturing diverse human-object interactions in replicated 3D environments and enriching them with structured annotations, renderings, and motion augmentation. Its augmentation pipeline preserves contact relationships while adapting motion to altered scene geometry.

  • Dataset Overview: TRUMANS provides accurate 3D ground truths and photorealistic multi-view, ego-view, and other rendered data for HSI tasks.The dataset section covers statistics, capture, postprocessing, and augmentation procedures.
  • Dataset Statistics: TRUMANS contains 15 hours of VICON data across 100 indoor scenes, 1.6 million frames, 20 object types, and seven participants.Objects include static, dynamic, and articulated categories, with at least five distinct instances per type.
  • Scene-aware Capture: The capture process replicates 3D-FRONT and BlenderKit scenes in physical MoCap environments and uses real-world placeholders to preserve object affordances.Movable objects are marked for VICON tracking so humans and objects can be captured together.
  • Postprocessing: Postprocessing converts poses to SMPL-X, computes per-vertex contact annotations, and renders photorealistic RGBD, segmentation, and egocentric data.More than 200 digital human models diversify the renderings without obstructing movement.
  • Motion Augmentation: Motion augmentation adapts human trajectories to altered scene objects while preserving physical plausibility and interaction fidelity.The pipeline is designed for complex scenes involving concurrent or successive interactions.
  • Motion Augmentation: Figure 2 illustrates motion adaptation when chair and bed heights change by 15cm while interactions remain consistent.The example demonstrates augmentation across altered object dimensions rather than merely visual variation.
  • Motion Augmentation: The augmentation pipeline transfers contact points, smooths joint-offset trajectories, and recomputes motion with an enhanced CCD-based IK solver.Clipping, regularization, and distance-dependent rotational limits help maintain fluid, natural motion after scene changes.

4. Method

The method uses an autoregressive conditional diffusion model to generate arbitrary-length human-scene interaction motions from scene, goal, and action conditions. It combines episode-wise generation, local scene perception, and action-progress encoding to support controllable motion and object interaction.

  • Method overview: The model generates physically plausible, frame-wise controllable human-scene interactions of arbitrary length in real time using autoregressive motion diffusion.It conditions synthesis on 3D scenes and action labels.
  • Problem formulation: The synthesis objective takes a 3D scene, goal location, and action labels as conditions while also estimating dynamic-object poses when interactions involve objects.Human motion is represented through SMPL-X meshes, and objects through rotations and translations.
  • Autoregressive generation: Long sequences are generated episode by episode, with each new segment extending from the final k frames of the preceding episode.Discrete subgoals specify episode-level navigation targets, including desired pelvis xy-coordinates at episode completion.
  • Episode synthesis: Within each episode, a Transformer-based DDPM predicts noise over human joint locations while its first token encodes diffusion-step, scene, and action information.The sampled joints are converted into SMPL-X parameters through an MLP and further refined for interaction realism.
  • Scene and action conditioning: A local occupancy grid is encoded with a Vision Transformer and supplied as the diffusion condition, while progress-augmented frame-wise action labels encode action evolution across episodes.Grid discretization improves efficiency, although direct mesh-based occupancy computation is reported to make training approximately 300 times slower.

5. Experiments

The experiments evaluate action-conditioned HSI generation in static and dynamic settings, using unseen scenes and complementary motion-quality metrics. TRUMANS-trained models outperform relevant baselines, while human studies and real-time tests support motion realism and practical generation.

  • Evaluation setup: Evaluation covers static locomotion and scene interaction, dynamic object interaction, ablations, qualitative comparisons, and additional image-based tasks.Unseen scenes come from PROX, Replica, ScanNet, and ScanNet++, with five motion variations generated per trajectory.
  • Evaluation metrics: Contact and Penetration metrics assess static motion, while FID and Diversity assess dynamic human-object interaction generation.A MoCap-differentiation study additionally measures the Success Rate of Discrimination.
  • Static setting: TRUMANS-trained models surpass baselines across most static-setting metrics, and removing augmentation increases penetration.Compared with PROX-trained models, the results show significant improvements.
  • Dynamic setting: In dynamic interaction, TRUMANS captures more detailed scene-adherent motions, whereas GRAB-trained methods show high penetration; removing Aind causes method failure.Table 3 evaluates object-level interaction against GRAB-trained methods.
  • Human study: Only about one quarter of participants distinguish synthesized motions from real MoCap, near the 1/5 random-guessing rate, while the model exceeds the second-best method by over 30% in support rate.The study uses MoCap discrimination to assess realism and comparative preference to assess naturalness.
  • Real-time control: The method samples a 1.6-second episode at 10 FPS in 0.7 seconds on an A800 GPU, enabling uninterrupted long-term generation with incremental sampling for new controls.The incremental strategy begins with 2 frames and increases sampled chunks up to 16 frames.
  • Additional image-based tasks: Integrating TRUMANS with 3DPW significantly improves human mesh estimation and enhances contact estimation, particularly by reducing geodesic error.These results support combining synthetic TRUMANS data with real-world data for image-based tasks.

6. Conclusion

The paper introduces TRUMANS and a diffusion-based autoregressive method to address scalability, data quality, and advanced HSI motion synthesis. The method generates real-time sequences of arbitrary length whose quality closely mirrors original MoCap data, but it does not generalize beyond interaction behaviors represented in training.

  • TRUMANS is a large-scale MoCap dataset paired with a novel motion synthesis method for scalability, data quality, and advanced HSI modeling challenges.
  • The diffusion-based autoregressive method generates real-time HSI sequences of arbitrary length using TRUMANS.
  • Generated motions closely mirror the quality of original motion-captured data.
  • The method cannot generate human-object interaction behaviors beyond those present in training, such as realistic motions for climbing off a table.

A.1. Experiment Settings

The experiment settings separate static scene interaction from dynamic object interaction and require multiple motion variations for each specified task.

  • Static setting: Static evaluation tests locomotion and scene-level interaction between predefined start and end points in indoor scenes.Five start-end pairs are used per test scene, often involving furniture such as chairs.
  • Dynamic setting: Dynamic evaluation asks methods to approach and interact with an object from paired human and object starting locations under a designated action type.Each method generates five unique motion variations.

A.2. Implementation Details

Implementation uses a scene- and action-conditioned diffusion model with Transformer-based processing, SMPL-X reconstruction, and a real-time-oriented evaluation pipeline.

  • Model: The motion generator uses a DDPM with a linear variance scheduler, conditioned on scene and action embeddings.Its Transformer encoder has 6 layers, 16 attention heads, 512-dimensional embeddings, and 0.1 dropout.
  • SMPL-X reconstruction: A four-layer MLP converts joint locations into coarse SMPL-X parameters, which are refined by minimizing joint-location L2 error.The MLP uses three consecutive frames and predicts parameters for the middle frame.
  • Training: Training uses Adam at learning rate 1e-4 and batch size 512 on four NVIDIA A800 GPUs for 500k steps over 48 hours.
  • Baseline adaptation: Baseline methods are adapted with milestone transition and subgoal poses to support fair autoregressive long-term motion comparisons.
  • Human evaluation: Human studies compare synthesized and MoCap motions in shared TRUMANS-test scenes, while pairwise demonstrations measure preference against recent methods.
  • Additional results: Additional qualitative results visualize 3D contact estimation.

A.5. Image-based Tasks

TRUMANS data improves image-based human mesh and contact estimation when incorporated into training, including alongside existing real data.

  • 3D Human Mesh Estimation: Adding TRUMANS data to 3DPW training markedly enhances I2L and SGRE performance on the 3DPW test set.The methods are evaluated with real-only and mixed training at varying ratios.
  • 3D Contact Estimation: Training baseline contact-estimation methods with TRUMANS improves the precision of contact prediction.The comparison includes models trained with and without TRUMANS.

B.1. Additional Details of Dataset Comparison

The dataset comparison standardizes object categories and combines related data, while documenting TRUMANS splits, scene alignment, capture aids, and object coverage.

  • Dataset comparison: Object categories from SAMP, BEHAVE, and iReplica are consolidated into common types for fairer dataset comparison.SAMP’s seven objects become three types, BEHAVE’s 20 items become 14, and iReplica’s nine objects become five classes.
  • Dataset comparison: iReplica’s combined comparison data includes 0.25 hours from EgoHOI and 0.5 hours from H-contact.EgoHOI provides ego-view and multiview RGBD videos, while H-contact provides per-frame hand-object contact.
  • Dataset split: TRUMANS uses scenes 1–70 for training, 71–80 for validation, and 91–100 for testing, yielding an approximately 7:1:2 frame split.The listed scene ranges define the respective dataset partitions.
  • Object coverage: The dataset includes 20 indoor object types, spanning articulated and rigid chairs, furniture, handheld items, and articulated appliances.Examples include tables, sofas, beds, books, phones, laptops, drawers, cabinets, microwaves, and doors.
  • Environment alignment: Virtual scenes and objects are manually aligned with real-world counterparts selected for interactability and compatibility with the motion-capture area.Digital characters are also retargeted to actors using matching bone lengths and real-time inverse kinematics.
  • Capture process: Transparent object placeholders and real-time inspection support accurate capture in clustered scenes while filtering tracking and inverse-kinematics errors.The placeholders enable tracking through occluding geometry, such as a transparent acrylic table.

B.5. Motion Augmentation Implementation Details

The motion augmentation pipeline preserves human–object contact by smoothly blending object-induced joint offsets, including when multiple contacts change over time.

  • Single-contact augmentation: Object-shape changes induce joint offsets that are applied within a proximity window using a linearly decreasing norm for smoother pose transitions.The offset is computed from the changed object contact point and blended into the original joint trajectory.
  • Smoothness analysis: With W set to 30 and practical offset magnitudes, the updated inverse-kinematics target trajectories exhibit sufficient smoothness.The condition depends on reasonable object variation scales and stable CCD inverse-kinematics results.
  • Multi-contact augmentation: When contacts shift between bones, the method blends multiple offset vectors using a norm-weighted mean over the smoothing window.This compensates for deviations caused when inverse kinematics assigns a new target to another contacting bone.
  • Smoothness analysis: The resulting target trajectories are bounded by the original frame-to-frame motion plus an offset term proportional to 4(|v_1| + |v_2|)/W.The bound is used to establish smoothness for the blended trajectory.

B.6. Annotations

TRUMANS provides fitted SMPL-X human motion, contact annotations, detailed object dynamics, action labels, and adaptive camera trajectories for rendered interaction data.

  • Human motion: VICON pose sequences are converted into SMPL-X through vertex pairing, manual correction, framewise optimization, and temporal initialization.The optimization refines body shape and pose parameters from the preceding frame until convergence.
  • Contact annotations: Per-vertex contact is determined from mesh penetration or proximity and orientation relative to scene and object meshes.The orientation criterion helps prevent false contact assignments, such as misidentifying a hand holding a bottle.
  • Contact annotations: More than 95% of frames have both mean and median penetration distances below 2cm.These statistics characterize penetration quality in the contact annotations.
  • Object annotations: Objects include watertight meshes, framewise 6D poses, and part-level annotations with URDF files for articulated objects.The representation covers both rigid and articulated object dynamics.
  • Action annotations: Each sequence receives framewise multi-hot action labels, allowing multiple concurrent actions to be represented in one frame.Each action is assigned a distinct label.
  • Video rendering: The rendering pipeline uses adaptive camera tracking with constrained rotations, visibility checks, and dynamic programming to maximize interaction coverage.Candidate camera poses are evaluated at keyframes, and the selected rotations are interpolated between them.
Loading 2403.08629v2…