Source-linked AI summary
Human Motion Diffusion as a Generative Prior
Yonatan Shafir, Guy Tevet, Roy Kapon, Amit H. Bermano
TL;DR
Human-motion diffusion models are limited by scarce annotated data, single-person training, and coarse control. The paper composes pretrained diffusion priors sequentially, in parallel, and across models to address long motions, two-person interactions, and fine-grained control. It reports long fluent sequences, few-shot coordination, and flexible joint- and trajectory-level editing, while noting limitations in long-range consistency and interaction generalization.
Problem
Scarce annotated motion data and predominantly short, single-person datasets limit long-sequence, multi-person, and detailed-control generation.
Method
The paper composes pretrained diffusion priors sequentially, in parallel, and across fine-tuned models for long motions, two-person coordination, and composite control.
Results
The methods produce 10-minute fluent motions from a prior trained on clips up to 10 seconds, enable two-person generation from as few as a dozen examples, and support flexible joint and trajectory control.
Takeaways & Limitations
Diffusion priors can support advanced motion generation and control with little to no new data by handling integration between composed components.
Takeaways & Limitations
Long-sequence generation remains limited by the initial model’s quality and may show inconsistencies between distant intervals; ComMDM generalizes only to interactions seen during training.
Abstract
from arXiv · showhide
Recent work has demonstrated the significant potential of denoising diffusion models for generating human motion, including text-to-motion capabilities. However, these methods are restricted by the paucity of annotated motion data, a focus on single-person motions, and a lack of detailed control. In this paper, we introduce three forms of composition based on diffusion priors: sequential, parallel, and model composition. Using sequential composition, we tackle the challenge of long sequence generation. We introduce DoubleTake, an inference-time method with which we generate long animations consisting of sequences of prompted intervals and their transitions, using a prior trained only for short clips. Using parallel composition, we show promising steps toward two-person generation. Beginning with two fixed priors as well as a few two-person training examples, we learn a slim communication block, ComMDM, to coordinate interaction between the two resulting motions. Lastly, using model composition, we first train individual priors to complete motions that realize a prescribed motion for a given joint. We then introduce DiffusionBlending, an interpolation mechanism to effectively blend several such models to enable flexible and efficient fine-grained joint and trajectory-level control and editing. We evaluate the composition methods using an off-the-shelf motion diffusion model, and further compare the results to dedicated models trained for these specific tasks.
1 INTRODUCTION
Human-motion diffusion models offer strong short-sequence generation but remain constrained by scarce annotated data, limited multi-person coverage, and insufficient detailed control. This paper uses pretrained diffusion models as compositional priors for long-sequence generation, two-person coordination, and fine-grained control.
- Motivation: Motion datasets are expensive, small, homogeneous, and composed largely of short single-person sequences, limiting long-sequence and multi-person generation.The paper attributes these constraints to elaborate motion-capture settings or artist-created data.
- Core idea: Pretrained diffusion-based motion models can serve as priors for out-of-domain generation and efficient control in few-shot or zero-shot settings.The approach uses a pretrained Motion Diffusion Model as the prior.
- Three compositions: The paper defines sequential, parallel, and model composition for concatenating motions, coordinating two people, and blending models with different control capabilities.These three forms organize the paper’s proposed composition framework.
- Parallel composition: ComMDM coordinates two frozen motion priors through a slim communication block, enabling textually driven two-person generation with as few as a dozen training examples.The communication signal passes through intermediate activation maps during diffusion.
- Model composition: DiffusionBlending composes fine-tuned control models to support flexible joint- and trajectory-level motion control and editing.The method extends classifier-free guidance to combine models with different keypoint-control capabilities.
- Evaluation: The composition methods quantitatively and qualitatively outperform dedicated previous methods on their respective tasks.The comparison uses an off-the-shelf motion diffusion model and dedicated prior art.
2 RELATED WORK
Prior work addresses motion generation, editing, long sequences, and multi-person synthesis, but remains limited by data availability, annotation, generalization, or task-specific design. This paper positions diffusion-prior composition as a way to extend these capabilities with limited additional data.
- Long-sequence generation: Motion graphs synthesize long motions by traversing discrete poses, but depend on existing data and do not generalize well to elaborate textual conditions.RNN-based alternatives also tend toward constant poses, while prefix-completion methods require generated-frame training.
- Multi-person generation: Multi-person motion datasets are scarce, and existing datasets described here lack textual or other annotations for text-guided generation.The cited datasets contain only limited numbers of two-person or multi-person sequences.
- Diffusion composition: Soft blending assigns frame-wise mask values between 0 and 1 so a second denoising pass partially refines generated motion around transitions.A short linear transition of b frames connects hard and soft masking over the interval and handshake regions.
- Multi-person generation: This work textually annotates 3DPW and learns text-guided two-person motion generation for the first time.Earlier approaches learned unsupervised prefix completion under the DCT transform.
- Human motion priors: VPoser is a human-pose auto-encoder trained on AMASS and used as a prior for denoising, SMPL fitting, and motion-generation pose coding.The cited examples illustrate prior-based motion applications before this paper’s diffusion composition methods.
- Diffusion methods: MDM adapts diffusion-image inpainting for motion editing, while this work extends diffusion principles to non-trivial few-shot and zero-shot motion tasks.Other related work includes latent diffusion models that generate motion latent codes for a larger motion generator to translate.
3 METHOD
The method composes pretrained diffusion priors for long, interactive, and finely controlled human-motion generation. It combines DoubleTake for sequential composition, ComMDM for two-person coordination, and DiffusionBlending for composite control.
- Sequential composition: DoubleTake generates arbitrarily long motions from a fixed short-sequence prior by composing separately prompted intervals and refining their transitions.The method operates at inference time and supports different text prompts and sequence lengths for individual intervals.
- Sequential composition: DoubleTake’s first take generates intervals in parallel while forcing neighboring handshakes to agree during every denoising iteration.A second take partially noises transition sandwiches and denoises them with soft masks conditioned on neighboring intervals.
- Parallel composition: ComMDM coordinates two frozen MDM instances through a slim communication block that exchanges intermediate activations during diffusion.It can additionally predict each person’s initial pose, addressing datasets that place both actors at the same origin and orientation.
- Model composition: The method trains control-specific diffusion priors by preserving ground-truth control features during the forward diffusion pass.The controlled features can include a root trajectory, a single joint, or combinations of these signals.
- Model composition: DiffusionBlending combines separately fine-tuned control models so multiple capabilities, such as trajectory and left-hand control, can be applied together.The approach is introduced because training a separate model for every possible composite control task is sub-optimal.
4 EXPERIMENTS
The experiments evaluate DoubleTake for long-sequence generation, ComMDM for two-person motion, and model composition for controlled motion synthesis. Results compare against dedicated or original models using quantitative metrics, qualitative examples, and a user study.
- Long Sequences Generation: DoubleTake outperforms TEACH in FID across all evaluated methods, with larger advantages on transition evaluations.The comparison includes both fair and minimal transition margins.
- Two-Person Generation: ComMDM is trained with a slim communication block between fixed priors for few-shot two-person motion generation.The two-person data setting leaves 10 training examples and 4 validation examples after processing 3DPW.
- Two-Person Generation: ComMDM generates lively, semantically correct prefix completions, although MRT achieves lower mean L2 error.A 30-user study preferred ComMDM over MRT and MDM across interaction, prefix completion, and overall motion quality.
- Two-Person Generation: Text-to-motion ComMDM generates diverse two-person interactions from unseen prompts, but generalization remains limited to interaction types seen during training.The model is trained using five textual annotations for 14 training and validation motions.
- Model Composition: Fine-tuning MDM is crucial for the evaluated joint and trajectory control tasks and produces high-quality results.The study compares fine-tuned models and DiffusionBlending with original MDM inpainting.
5 CONCLUSION
The paper presents diffusion motion priors as a basis for sequential, parallel, and model composition with little to no new data. It also identifies limitations in long-range consistency, interaction generalization, and human contact modeling.
- Conclusion: Diffusion-based motion priors support three composition methods for advanced motion generation and control.The methods address sequential composition, parallel coordination, and blending models with different control capabilities.
- Conclusion: The approach enables new motion-generation tasks with little to no additional data by addressing integration between composed parts.The paper frames the diffusion model as a proxy for the human motion manifold.
- Limitations: Long-sequence generation remains limited by the initial model’s quality and may develop inconsistencies between distant intervals.Long sequences also increase the need for motions that interact with rich environments.
- Limitations: ComMDM synchronizes two priors effectively for training-seen interactions but lacks generalization and does not yet ensure valid contacts between people.The conclusion identifies valid interpersonal contacts as a future challenge.
- Future Work: The proposed techniques are not specifically designed for motion, motivating adaptation to other generative domains.The paper also proposes investigating additional ways to combine knowledge from pretrained generative models.
A USER STUDY
The paper conducted a user study of two-person prefix completion, with a sample screenshot provided as an example of the study interface.
- User Study: The user study evaluates two-person prefix completion, and Figure 11 shows a sample screenshot from the study form.The study details appear in Section 4.2, with results presented in Figure 8.