Source-linked AI summary
InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, Lan Xu
TL;DR
InterGen addresses the limited treatment of multi-human interaction in motion generation by enabling text-guided customization of high-quality two-person motions. It combines a multimodal interaction dataset with a diffusion model tailored to symmetry and spatial relations, and experiments show compelling, diverse, and plausible interaction motions.
Problem
Human motion generation needs realistic, customizable motions that reflect interactions between people and match textual themes, while prior work faces challenges including unnatural interactions and limited generalization to novel texts.
Method
InterGen combines the InterHuman multimodal dataset with a diffusion model using world-frame motion representation, cooperative weight-sharing denoisers, mutual attention, and spatial-relation regularization.
Results
InterGen outperforms other baselines on FID, R precision, and MM Dist, while generating diverse, plausible, and naturally aligned two-person interaction motions.
Takeaways & Limitations
InterGen supports text-controlled two-person interaction generation and downstream interaction applications across diverse scenarios.
Takeaways & Limitations
InterGen models only two-person interactions and generates sequences from text prompts without incorporating user feedback, limiting complex-scenario applicability and creativity with vague prompts.
Abstract
from arXiv · showhide
We have recently seen tremendous progress in diffusion advances for generating realistic human motions. Yet, they largely disregard the multi-human interactions. In this paper, we present InterGen, an effective diffusion-based approach that incorporates human-to-human interactions into the motion diffusion process, which enables layman users to customize high-quality two-person interaction motions, with only text guidance. We first contribute a multimodal dataset, named InterHuman. It consists of about 107M frames for diverse two-person interactions, with accurate skeletal motions and 23,337 natural language descriptions. For the algorithm side, we carefully tailor the motion diffusion model to our two-person interaction setting. To handle the symmetry of human identities during interactions, we propose two cooperative transformer-based denoisers that explicitly share weights, with a mutual attention mechanism to further connect the two denoising processes. Then, we propose a novel representation for motion input in our interaction diffusion model, which explicitly formulates the global relations between the two performers in the world frame. We further introduce two novel regularization terms to encode spatial relations, equipped with a corresponding damping scheme during the training of our interaction diffusion model. Extensive experiments validate the effectiveness and generalizability of InterGen. Notably, it can generate more diverse and compelling two-person motions than previous methods and enables various downstream applications for human interactions.
1 Introduction
InterGen addresses the limited treatment of human-to-human interaction in motion diffusion by combining a multimodal interaction dataset with an interaction-aware diffusion model. Its designs target text-guided, diverse, realistic two-person motion generation while preserving symmetry and spatial relations.
- Motivation: Existing motion diffusion models mainly target single-person settings and overlook rich human-to-human interactions.Existing datasets also struggle to provide both accurate captured interaction motions and natural prompt labels for diverse interactions.
- Dataset: InterHuman provides diverse two-person interaction data with accurate skeletal motions and natural-language descriptions.The dataset includes about 107 million video frames from 7,779 motion sequences, lasting 6.56 hours, with 23,337 unique descriptions.
- Approach: InterGen enables layman users to customize high-quality two-person interaction motions using only text guidance.The approach is designed for interaction motions spanning daily activities and professional interactions.
- Approach: Cooperative denoisers share weights and use mutual attention to model the symmetry between two interacting performers.The two denoisers correspond to the two performers while explicitly connecting their denoising processes.
- Approach: A non-canonical motion representation explicitly encodes global relations between performers in a common world frame.This addresses the loss of precise spatial relations in canonical single-person representations and avoids drift associated with naive relative translation and rotation inputs.
- Approach: Joint distance-map and relative-orientation losses, together with damping during training, encode spatial relations and support diverse interaction generation.The distance-map loss captures spatial interference and implicit physical constraints, while the relative-orientation loss captures interaction-facing information.
2 Related Work
Prior work advances text- and action-conditioned human motion generation, but interactive multi-person generation remains constrained by single-person datasets, limited annotations, and persistent motion-quality challenges.
- Text-driven motion generation: Text-driven diffusion models provide convenient control for generating human motions from natural language.Recent work builds on diffusion methods’ ability to model complex motion distributions and accept accessible user prompts.
- Open challenges: Prior interaction-generation approaches still face drifts, foot-sliding, unnatural interactions, and limited generalization to novel texts.Limited data quality and quantity impede further exploration of two-person interaction generation.
- Dataset comparison: Table 1 compares datasets using motion clips, vocabulary size, and total textual descriptions.These measures distinguish the scale and annotation coverage of existing human-motion datasets.
- Interactive motion datasets: Existing text-annotated motion datasets primarily contain single-person motions, limiting their application to interactive generation involving multiple individuals.This mismatch makes generalization to two-person interactions difficult.
- Interactive motion datasets: Multi-person datasets provide interaction data but remain limited in size and textual or multimodal annotations.ComMDM’s text-annotated 3DPW subset contains only 27 two-person motion sequences.
3 InterHuman Dataset
InterHuman is a large-scale, text-annotated dataset of diverse two-person interactions, captured with 76 cameras and organized across daily and professional activities.
- Dataset overview: InterHuman is a comprehensive 3D dataset containing diverse motions of two interacting people, each paired with natural-language annotations.It is presented as the most extensive 3D human-to-human interaction dataset available.
- Interaction coverage: The dataset covers daily interactions such as passing objects and greeting, alongside professional activities including Taekwondo, Latin dance, and boxing.Its collection separates daily motion from professional motion categories to cover real-world interaction distributions.
- Data collection: 76 calibrated RGB cameras captured the human interaction motions.The motion-capture system is illustrated in Fig. 2.
- Annotation: Annotators segmented captured videos into clips of at most 10 seconds and wrote descriptions for every clip from separate perspectives.The annotation process aimed to preserve each interaction’s semantic meaning while capturing varied descriptions.
- Dataset scale: 7779 motions are labeled with 23,337 unique descriptions using 5656 distinct words across 6.56 hours of captured data.The paper characterizes InterHuman as the largest and most diverse known scripted dataset of human-to-human interactions.
4 InterGen Approach
InterGen adapts diffusion-based motion generation to two-person interactions through a world-frame motion representation, cooperative denoisers, and interaction-aware regularization. Its designs preserve spatial relations, exploit performer symmetry, and connect the two denoising branches.
- Overview: InterGen combines a world-frame interaction representation, cooperative denoising networks, and additional regularization losses for text-conditioned two-person motion generation.These are the approach’s three primary technical designs.
- 4.1 Human Interaction Representation: Canonical representations lose global spatial information, while cumulative integration of noisy local velocities can cause unbounded trajectory drift.This motivates retaining global motion information directly in the representation.
- 4.1 Human Interaction Representation: The non-canonical representation keeps both performers’ joint positions in one world frame, exposing global translation and rotation without cumulative integration.Local rotations remain represented in the root frame, alongside world-frame joint positions and foot-contact features.
- 4.2 Human Interaction Diffusion: The model assumes swapping performer identities preserves interaction semantics, motivating symmetric denoising branches for the two people.This commutative property treats {xa, xb} and {xb, xa} as equivalent interactions.
- 4.2 Human Interaction Diffusion: InterGen models interaction diffusion with two cooperative denoisers that share weights and use counterpart hidden states through mutual attention.Each branch receives its own and the other performer’s noisy motion, together with timestep and conditioning information.
- 4.3 Additional Regularization Losses: The masked joint distance-map loss matches cross-person joint distances to ground truth, with masking activated when horizontal separation is sufficiently small.The loss uses global joint positions and a 2D XZ-plane distance threshold.
5 Experiments
InterGen is evaluated against prior methods, through ablations of its representation, cooperative networks, interactive losses, and loss schedule, and through downstream interaction applications. The experiments report stronger quantitative and qualitative interaction generation, while identifying scope limitations for more complex or unconstrained use.
- Comparisons: InterGen outperforms other baselines on FID, R precision, and MM Dist over the whole test set.These results indicate more compelling interaction generation and more accurate text-motion matching.
- Comparisons: Qualitative comparisons show more natural interaction states, diverse motions, and accurate global relative orientations and translations between performers.The corresponding results are presented in Fig. 8.
- Evaluations: Removing the non-canonical representation causes significant R precision and FID degradation, while cumulative trajectory errors separate the two performers spatially.The common-frame representation preserves spatial relations and avoids harmful cumulative integration.
- Evaluations: Replacing mutual attention with concatenation reduces interaction plausibility and motion alignment, while removing weight sharing worsens R precision and FID.Without weight sharing, the two performers also show dissimilar activation and motion capacity during interaction.
- Evaluations: Removing the distance-map or relative-orientation loss produces unrealistic interactions, including body penetration or mismatched orientations.The results support the effectiveness and necessity of both interactive losses.
- Evaluations: The t ≤ 0.7T regularization schedule with cosine noise outperforms applying regularization at all timesteps or omitting it.This schedule is selected for the final model.
- Applications: Downstream applications include person-to-person generation, trajectory control combined with text, and diverse interaction inbetweening.The inbetweening workflow freezes initial and ending motions while incorporating text prompts into generated motions.
- Limitations: The method is limited to two-person interactions, text-only conditioning without user feedback, and motion sequences with a fixed maximum length.These constraints limit crowd scenarios, preference-specific control, and extremely long-sequence generation.
6 Conclusion
The paper presents InterGen for text-controlled two-person motion generation across diverse interactions, supported by a multimodal dataset and interaction-specific diffusion designs. Its conclusion emphasizes cooperative denoisers, global relational representation, and damped regularization for modeling interacting performers.
- 6 Conclusion: InterGen generates two-person motions under diverse interactions using only text-prompt controls.The paper presents it as a diffusion-based approach for convenient interaction motion generation.
- 6 Conclusion: The paper contributes a multimodal dataset with rich motion results and natural-language descriptions covering diverse interaction scenarios.The dataset is introduced as support for text-guided interaction motion generation.
- 6 Conclusion: Cooperative weight-sharing denoisers with mutual attention model the symmetry of human identities during interactions.The design connects the two performers’ denoising processes.
- 6 Conclusion: A non-canonical motion representation models global relations between performers, while damped regularization encodes spatial relations.Together, these designs target relational structure in the interaction setting.