Source-linked AI summary
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, Ziwei Liu
TL;DR
Text-driven motion generation performs well on common motions but remains limited on more diverse or uncommon conditions. ReMoDiffuse adds retrieval-guided denoising through hybrid semantic-kinematic retrieval, selective transformer integration, and condition mixing. Experiments report improved generalizability, diversity, and balance between text-motion consistency and motion quality, especially for uncommon motions.
Problem
Existing text-driven motion-generation methods are not versatile enough for uncommon condition signals and require better use of diverse motion knowledge.
Method
ReMoDiffuse is a retrieval-augmented diffusion pipeline using Hybrid Retrieval, a Semantics-Modulated Transformer, and Condition Mixture to refine denoising with retrieved samples.
Results
ReMoDiffuse outperforms existing motion-generation pipelines and achieves state-of-the-art performance on both condition-consistency and fidelity, particularly for uncommon motions.
Takeaways & Limitations
Retrieval can provide auxiliary multimodal knowledge that improves generalizability and helps generate diverse, high-quality motion sequences.
Abstract
from arXiv · showhide
3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process. ReMoDiffuse enhances the generalizability and diversity of text-driven motion generation with three key designs: 1) Hybrid Retrieval finds appropriate references from the database in terms of both semantic and kinematic similarities. 2) Semantic-Modulated Transformer selectively absorbs retrieval knowledge, adapting to the difference between retrieved samples and the target motion sequence. 3) Condition Mixture better utilizes the retrieval database during inference, overcoming the scale sensitivity in classifier-free guidance. Extensive experiments demonstrate that ReMoDiffuse outperforms state-of-the-art methods by balancing both text-motion consistency and motion quality, especially for more diverse motion generation.
1. Introduction
Text-driven motion generation has advanced through generative models, but existing approaches remain insufficiently versatile for uncommon conditions. ReMoDiffuse addresses this gap with retrieval-augmented diffusion and reports stronger generalizability across common and uncommon prompts.
- Text-driven motion generation supports applications including games, films, and virtual reality, while traditional modeling is time-consuming and equipment-intensive.
- Diffusion models generate realistic and diverse motions, but prior pipelines are not versatile enough for uncommon condition signals.
- Retrieval augmentation for motion requires combining semantic and kinematic similarity, selectively using retrieved samples, and accounting for semantic differences between prompts and references.
- ReMoDiffuse combines Hybrid Retrieval, a Semantics-Modulated Transformer, and Condition Mixture to guide denoising with retrieved knowledge.
- ReMoDiffuse achieves higher generalizability on both common and uncommon prompts in qualitative and quantitative experiments.
2. Related Work
Related work spans diffusion models, retrieval-augmented generation, and text-driven motion generation. ReMoDiffuse extends diffusion-based motion generation with retrieval capability to improve generalizability, particularly for uncommon condition signals.
- Diffusion Models: Diffusion models established strong progress in text-to-image generation, including classifier-free guidance and CLIP-based text-image alignment.
- Retrieval-Augmented Generation: Retrieval-based image-generation methods improve generalizability by using nearest-neighbor, user-provided, or external-database examples.
- Text-Driven Motion Generation: Earlier text-driven motion methods learned joint text-motion embeddings or used variational mechanisms to improve motion reconstruction and diversity.
- Text-Driven Motion Generation: ReMoDiffuse builds on MotionDiffuse with multimodal retrieval, feeding retrieved motion sequences into transformer layers during denoising.
- Text-Driven Motion Generation: MotionDiffuse and MDM adapted diffusion models to motion generation, offering realistic, controllable, and diverse actions through different transformer-based designs.
3. Our Approach
ReMoDiffuse builds a retrieval-augmented motion diffusion pipeline with retrieval and refinement stages. It retrieves samples using semantic, kinematic, and length-related information, then fuses retrieved knowledge into denoising while addressing computational cost and classifier-free guidance.
- Diffusion Model for Motion Generation: The diffusion model represents motions as pose sequences and learns to denoise noisy states into clean motion sequences.The model predicts the clean state x0 from a noisy state xt, timestamp t, retrieved samples, and text.
- Overall Architecture: ReMoDiffuse combines a retrieval stage for selecting informative motion samples with a refinement stage that uses them during diffusion denoising.The pipeline is presented as a retrieval-augmented extension of motion diffusion.
- Retrieval-Augmented Motion Generation: Hybrid Retrieval scores database samples using text-feature cosine similarity and a penalty based on relative motion-length differences.The hyper-parameter λ balances semantic similarity against motion-length similarity, and closer lengths increase the retrieval score.
- Retrieval-Augmented Motion Generation: The retrieval design evaluates semantic and kinematic similarities while down-sampling motion sequences to 1/4 original FPS to reduce fusion cost.The motion encoder uses Semantics-Modulated Attention and feed-forward layers to retain low-level motion information at lower computational cost.
- Condition Mixture: Condition Mixture combines four model estimations conditioned on retrieved samples and text, using learned coefficients to balance them during inference.The four estimations correspond to joint, retrieval-only, text-only, and unconditional conditions.
4. Experiments
Experiments evaluate ReMoDiffuse on KIT-ML and HumanML3D using standard motion-generation metrics, retrieval ablations, qualitative comparisons, and new measures for diverse generation. Results report improved balance between condition consistency and fidelity, with stronger performance on uncommon motions.
- Datasets and Metrics: Experiments use KIT and HumanML3D, containing 3,911 and 14,616 motions respectively, with FID, R Precision, Diversity, Multimodality, and Multi-Modal Distance.HumanML3D also provides 44,970 descriptions, while KIT provides 6,363 natural-language annotations.
- Main Results: ReMoDiffuse achieves a better balance between condition-consistency and fidelity than other diffusion-model pipelines.The authors report state-of-the-art performance on both metrics.
- Retrieval Techniques: λ = 0 makes retrieval quality unacceptable, while λ = 0.1 is optimal on both KIT-ML and HumanML3D.λ balances semantic and kinematic similarity, with larger values emphasizing kinematic similarity.
- Motion Refinement: Using text features markedly improves FID in motion refinement, and the proposed retrieval techniques outperform the baseline by a remarkable margin.The ablation treats semantic and kinematic similarity as factors in both retrieval and refinement stages.
- Metrics on Diverse Generation: Rareness measures the maximum text-prompt similarity to training prompts, with lower similarity producing higher rareness.The test distribution is divided into 100 bins, and higher rareness commonly reflects unseen expressions, detailed descriptions, or action combinations.
- Results and Analysis: With retrieval, ReMoDiffuse outperforms both its retrieval-free baseline and state-of-the-art methods on generalization ability.Qualitatively, it is the only compared approach reported to convey both accurate action and path information.
5. Conclusion
ReMoDiffuse is a retrieval-augmented motion diffusion model that uses multimodality retrieval, semantics-modulated attention, and learnable condition mixing to refine denoising. Experiments report superior text-driven motion-generation performance, particularly for uncommon motions, while noting potential misuse for fake media.
- Conclusion: ReMoDiffuse combines multimodality retrieval, semantics-modulated attention, and learnable condition mixture to use an auxiliary database during denoising.The framework is designed to refine motion without expensive computation.
- Conclusion: Quantitative and qualitative experiments report superior text-driven motion-generation performance, particularly for uncommon motions.The conclusion summarizes the method’s reported effectiveness across both evaluation types.
- Conclusion: The technique can be used to create fake media when combined with 3D avatar generation, including malicious depictions of events that never happened.This is identified as a social impact concern.