Source-linked AI summary
HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, Siyuan Huang
TL;DR
Existing HSI datasets limit language-aware, scene-grounded motion generation through restricted scale, quality, and semantics. HUMANISE constructs a large synthetic dataset with language descriptions and introduces a scene-and-language conditioned cVAE model; experiments report diverse, semantically consistent motions, while object grounding and physical interaction remain failure cases.
Problem
Existing HSI datasets have limited scale and quality and lack scene, interaction, and action semantics, making instruction-aware scene-conditioned motion generation challenging.
Method
HUMANISE aligns captured motions with 3D scenes and language descriptions, while a cVAE model encodes scene and language conditions to generate conditioned human motion.
Results
The model generates visually appealing, diverse, and semantically consistent motions, with auxiliary losses improving spatial and action grounding in the action-agnostic setting.
Takeaways & Limitations
HUMANISE enables language-conditioned human motion generation in 3D scenes requiring joint understanding of scenes, motion, and language.
Takeaways & Limitations
The model sometimes misgrounds interacting objects and produces physically implausible human-object relations or collisions, partly because it lacks explicit object-level HOI modeling.
Abstract
from arXiv · showhide
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.
1 Introduction
HUMANISE addresses limited, semantically poor HSI datasets by introducing a large synthetic dataset and a language-conditioned motion-generation task with a scene-and-language model.
- Existing HSI datasets have limited scale and quality, including 12 PROX scenes with jittering motions and 49 GTA-IM scenes lacking 3D shapes and movement diversity.
- Existing datasets also lack complete semantic segmentation and human action descriptions, hindering instruction-aware motion generation.
- HUMANISE aligns captured human motions with scanned indoor scenes, applies collision and contact constraints, and automatically adds template-based descriptions of actions and interacting objects.
- The new task conditions motion generation jointly on 3D scenes and language, requiring correct actions, target-object grounding, realism, and physical plausibility.
- The proposed cVAE model separately encodes scene and language conditions, fuses them with self-attention, and uses auxiliary grounding and action-specific losses.
2 Related work
Prior work studies motion synthesis and human-scene interaction using varied conditioning signals and datasets, but existing HSI data remain noisy, limited, and difficult to ground semantically.
- Human motion synthesis has been conditioned on past motion, 3D scenes, objects, action labels, and language descriptions.
- Physical and virtual HSI datasets include PiGraphs and PROX, but suffer from inaccurate reconstructions, noisy poses, low frame rates, or visibly poor pose fitting.
- Grounding language to 3D objects has applications in reference, question answering, and navigation, yet referential object localization remains challenging.
3 The HUMANISE dataset
HUMANISE is synthesized by aligning labeled human motions with indoor scenes under physical constraints, enriching them with compositional language and diverse affordances.
- Dataset construction: HUMANISE combines AMASS motions, ScanNet scenes, BABEL action labels, and Sr3D referential descriptions through an automatic synthesis pipeline.
- Motion alignment: The alignment process samples target objects and interacting positions, then selects global translation and rotation using collision and contact constraints.
- Motion alignment: Contact constraints enforce foot-ground support and action-specific proximity between the human and interacting object.
- Diverse affordance: HUMANISE includes diverse affordances such as standing from a coffee table, lying on an office table, and sitting on a toilet.
- Language description generation: Descriptions follow the template < action >< target-class > [< spatial-relation >< anchor-class(es) >], with optional relations identifying the target object uniquely.
- Dataset statistics: 19.6k motion sequences span 643 3D scenes, covering sit, stand up, lie down, and walk, with rich semantic information about actions and interacting objects.
4 Language-conditioned human motion generation in 3D scenes
The model generates human motions conditioned jointly on a 3D scene and language description. It fuses multimodal context, encodes motion into a latent representation, and decodes motion while training with reconstruction, regularization, and auxiliary grounding and action objectives.
- Problem definition and notations: The task models human-motion generation conditioned on both a 3D scene and a language description.The target is a motion sequence that is semantically consistent with the language and physically plausible in scene interaction.
- Overall architecture: The proposed architecture adopts a cVAE framework with motion encoder and decoder components for conditional human-motion generation.The cVAE models p(Θ1:T | S, L1:D), where T is the motion-sequence length.
- Condition module: Scene and language features are fused through self-attention and mapped to a joint conditional latent embedding.Point-level scene features and word-level language features attend across modalities before producing the conditional embedding zc.
- Motion encoder and decoder: A bidirectional GRU encodes the input motion, while a transformer decoder uses sampled latent and conditional embeddings to generate motion parameters.The decoder converts its outputs into body meshes with the differentiable SMPL-X model.
- Training loss: Training combines reconstruction, KL-divergence, object-grounding, and action-specific generation losses.Reconstruction compares predicted body parameters and SMPL-X vertex positions with ground truth, while auxiliary tasks regress the target-object center and classify the action.
- Implementation details: The model is trained on HUMANISE with Adam for 150 epochs using a point transformer for scene processing and BERT for language descriptions.The scene encoder downsamples 32768 points to 128 points with 512-D features, while BERT descriptions have maximum length 32.
5 Experiments
Experiments evaluate HUMANISE motion generation through action-specific and action-agnostic settings, reconstruction and generation metrics, qualitative studies, ablations, duration variation, and downstream synthesis. Results indicate semantically consistent generation, benefits from auxiliary grounding tasks and architecture choices, and substantial downstream gains, while failure cases remain in object grounding and physical interaction.
- Action-specific results: Our model generates visually appealing and semantically consistent motions, with attention focused on point-cloud features around target interacting objects.The action-specific results include walk, sit, and stand up; lie down achieves smaller goal distance, possibly because its targets are often large flat furniture.
- Action-agnostic results: Auxiliary object-location and action-recognition tasks improve spatial and action grounding in the more difficult action-agnostic setting.Removing Lo increases goal distance, while La improves the action score; the differences in goal distance, quality score, and action score occur without loss of reconstruction quality.
- Architecture ablations: Self-attention fusion and the proposed scene encoder improve generation metrics and grounding compared with their ablated alternatives.The authors attribute the fusion benefit to attention across scene and language modalities and the encoder benefit to finer-grained scene understanding.
- Duration analysis: Sampling with different durations produces motions for both walk-to-object and sit-on-object descriptions, demonstrating generation across varied sequence lengths.The tested durations are 30, 60, 90, and 120 frames, with rendered bodies shown every 15 frames.
- Downstream task: HUMANISE pretraining followed by PROX fine-tuning outperforms PROX-only training on all downstream reconstruction metrics by a large margin.RouteNet and PoseNet are trained on HUMANISE for ten epochs, fine-tuned on PROX for ten epochs, and compared against a PROX-only 20-epoch model.
6 Conclusion
HUMANISE provides a large-scale, semantic-rich HSI dataset with diverse, physically plausible interactions and language descriptions, enabling language-conditioned motion generation in 3D scenes. The authors identify short motion duration and unresolved language grounding as limitations.
- HUMANISE contains diverse and physically plausible human-scene interactions equipped with language descriptions.
- The dataset enables language-conditioned human motion generation in 3D scenes.
- The proposed generative model produces diverse and semantically consistent motions from 3D scenes and language descriptions.
- Limitations: Human motions are relatively short because action-specific motions usually last within 5 seconds.
- Limitations: Language grounding in 3D scenes remains far from solved, and a robust grounding module could support more meaningful motions.
A Statistics of HUMANISE
HUMANISE provides varied motion sequences, language descriptions, and interactive objects across a substantial synthetic dataset. Compared with PROX, its motions receive higher quality assessments across collision, smoothness, HSI, and overall quality.
- Dataset statistics: HUMANISE contains motions ranging from 30 to 120 frames, with language descriptions averaging 7.63 words.
- Dataset statistics: The dataset includes 9 interactive object categories, with chair, table, and couch occurring most frequently.
- Dataset statistics: 19.6k motion sequences were synthesized from 1305 AMASS motions across 643 3D scenes.
- Dataset statistics: 8264 walk, 5578 sit, 3463 stand-up, and 2343 lie-down sequences were synthesized from 717, 365, 190, and 33 selected motions, respectively.
- Comparison with PROX: HUMANISE motions receive higher quality ratings than PROX for collision, smoothness, human-scene interaction, and overall quality.The comparison uses 100 motion segments from each dataset, scored by three workers on 1-to-5 scales.
C Generalization of motion alignment pipeline
The motion alignment pipeline generalizes beyond the original scanned scenes and core action types. It is demonstrated with Replica scenes and additional actions including jump up, turn to, open, and place something.
- The pipeline generalizes to other 3D scene datasets and action types, enabling large-scale synthetic HSI data generation.
- Generalize to other 3D scene datasets: The pipeline generates aligned human motions in hand-crafted Replica 3D scenes.
- Generalize to other actions: Additional aligned motions cover jump up, turn to, open, and place something.
D Additional qualitative results
Additional qualitative results cover reconstruction and generation. The model can sometimes fail to ground the correct interaction target or generate the correct human-object interaction relation.
- Reconstruction: Additional qualitative reconstruction results are provided in Fig. A4.
- Generation: Additional qualitative generation results are provided in Fig. A5.
- Failure cases: The model sometimes fails to ground the correct target object for interaction and to generate the correct human-object interaction relation.
F.1 Ablations of the lie-down action
Ablations show that lie-down grounding remains relatively easy as grounding-loss weight and training data vary, although reduced data harms reconstruction quality. The broader pipeline comparison indicates that providing ground-truth target positions substantially improves motion-generation metrics.
- F.1 Ablations of the lie-down action: Reducing the grounding-loss weight increases lie-down goal distance, but it remains smaller than for other actions.The original weight is αo = 0.1; experiments also test αo = 0.01, 0.001, and 0.
- F.1 Ablations of the lie-down action: Training with fewer lie-down examples preserves approximately the same goal distance but significantly reduces reconstruction metrics.The reduced-data settings use 10% and 50% of the original lie-down data.
- F.1 Ablations of the lie-down action: The authors attribute lie-down’s easier grounding to frequent interaction with beds or tables, whose flat surfaces cover larger areas than other furniture.
- F.2 Multi-stage pipeline and end-to-end pipeline: Using ground-truth action reaches approximately the same performance as the proposed model, while ground-truth action plus target position significantly improves motion-generation metrics.This comparison evaluates multi-stage and end-to-end pipeline variants.