Motion Mamba: Efficient and Long Sequence Motion Generation

Zeyu Zhang; Akide Liu; Ian Reid; Richard Hartley; Bohan Zhuang; Hao Tang

Source-linked AI summary

Motion Mamba: Efficient and Long Sequence Motion Generation

Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, Hao Tang

arXiv:2403.07487v4Mar 12, 2024cs.CV

TL;DR

Long-sequence human-motion generation remains difficult because diffusion models face temporal modeling and inference-efficiency limitations. Motion Mamba adapts selective state space models through hierarchical temporal and bidirectional spatial blocks in a latent-diffusion U-Net. It reports up to 50% FID improvement and up to four-times faster inference on HumanML3D and KIT-ML benchmarks.

Problem
Diffusion-based motion generators struggle with long-range sequence modeling and efficient inference, while motion-specific SSM designs remain limited.
Method
Motion Mamba uses HTM for hierarchical temporal scanning and BSM for bidirectional spatial processing within a latent-diffusion U-Net.
Results
Motion Mamba achieves state-of-the-art motion-generation quality and efficiency, including up to 50% FID improvement and four-times faster inference.
Takeaways & Limitations
The framework demonstrates high-quality long-sequence motion modeling with fast inference for conditional human-motion generation.

Abstract

from arXiv · show

Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging. Recent advancements in state space models (SSMs), notably Mamba, have showcased considerable promise in long sequence modeling with an efficient hardware-aware design, which appears to be a promising direction to build motion generation model upon it. Nevertheless, adapting SSMs to motion generation faces hurdles since the lack of a specialized design architecture to model motion sequence. To address these challenges, we propose Motion Mamba, a simple and efficient approach that presents the pioneering motion generation model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba (HTM) block to process temporal data by ensemble varying numbers of isolated SSM modules across a symmetric U-Net architecture aimed at preserving motion consistency between frames. We also design a Bidirectional Spatial Mamba (BSM) block to bidirectionally process latent poses, to enhance accurate motion generation within a temporal frame. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets compared to the previous best diffusion-based method, which demonstrates strong capabilities of high-quality long sequence motion modeling and real-time human motion generation. See project website https://steve-zeyu-zhang.github.io/MotionMamba/

1 Introduction

Motion Mamba addresses the difficulty of generating long, efficient human-motion sequences by adapting selective state space models with motion-specific temporal and spatial designs. It reports improved generation quality and faster inference than prior diffusion-based methods.

Motivation: Diffusion-based motion models face long-range modeling and inference-efficiency limitations, especially when using convolutional or transformer architectures.Transformers increase computational requirements and are not intrinsically designed for temporal sequential modeling.
Background: SSMs provide a promising foundation because they capture long-range dependencies while supporting parallel training for sequential data.The passage situates LSSL, S4, DSS, and S4D as SSM-based approaches designed for long-range sequence modeling.
Motivation: The paper identifies a lack of motion-specific SSM designs for sensitive temporal details and latent-space aggregation.Motion Mamba is presented as a diffusion-based architecture tailored to long-term sequence generation with near-linear-time complexity.
Results: Up to 50% FID improvement and 4× faster inference establish Motion Mamba’s accuracy-efficiency advantage over prior diffusion-based motion generation.On HumanML3D, FID decreases from 0.473 to 0.281, while inference takes 0.058 seconds per sequence versus 0.217 seconds for MLD.
Contribution: Motion Mamba introduces HTM and BSM modules for specialized temporal and spatial modeling within motion generation.HTM targets temporal dependencies and frame consistency, while BSM bidirectionally processes latent pose information.

2 Related Works

Prior human-motion generation work spans autoencoders, GANs, diffusion models, and state space models. The related literature highlights diffusion quality and diversity, alongside SSM advantages for efficient long-sequence modeling.

Human Motion Generation: Text-to-motion generation commonly learns a shared latent space between language and human motion for applications including 3D modeling and robot manipulation.This framing treats text-to-motion as the predominant recent formulation of human-motion generation.
Generative Models: Autoencoder methods compress motion into latent spaces, while GAN-based approaches use discriminators to improve realism and motion completion.Examples include RNN-based language-pose representations and densely validated multi-scale discriminators.
Diffusion Models: Diffusion models have surpassed GANs and VAEs in image generation and have been adopted for text-driven motion synthesis.MotionDiffuse introduced diffusion-based text-to-motion generation with probabilistic mapping, realistic synthesis, and multi-level manipulation.
State Space Models: State space models, including S4 and Mamba, are promising sequence architectures because they model long-range dependencies efficiently.Mamba adds selective, time-varying parameters and hardware-aware algorithms for efficient training and inference.

3 The Proposed Method

Motion Mamba is a latent-diffusion denoiser built around a symmetric U-Net with Mamba-based temporal and spatial processing. HTM uses hierarchical scans across depth, while BSM processes latent poses bidirectionally.

Latent Diffusion: The latent-diffusion pipeline encodes motion into a low-dimensional latent space, denoises it conditionally, and decodes it back into motion.The VAE encodes z = E(x1:L), decodes with D(z), and trains the latent denoiser using MSE between true and predicted noise.
Motion Mamba Architecture: Motion Mamba uses a denoising U-Net with encoder-decoder blocks, an attention mixer, and Mamba-based modules for long-range motion modeling.The denoiser contains encoder blocks E1..N, decoder blocks D1..N, and a transformer-based mixer M.
Hierarchical Temporal Mamba: HTM applies multiple isolated SSM scans with a hierarchical scan schedule, then aggregates their outputs into transformed temporal representations.The scan sequence ranges from S2N−1 to S1, distributing scanning complexity across encoder and decoder depth.
Design Rationale: The hierarchical scan strategy increases scan counts for higher generation quality while retaining lower complexity than attention-based methods.The design explicitly balances model parameters and computational efficiency.
Hierarchical Temporal Mamba: The symmetric encoder-decoder distributes hierarchical scans from high complexity at the outermost layers to low complexity at the innermost layers.This arrangement is intended to distribute processing capability across the architecture and support detailed temporal-sequence analysis.
Bidirectional Spatial Mamba: BSM rearranges latent representations, applies normalized linear projection, runs forward and backward SSMs, and gates and sums their outputs.Its output is rearranged back into temporal latent format after bidirectional processing.

4 Experiments

Experiments evaluate Motion Mamba on HumanML3D and KIT-ML using standard text-to-motion metrics, long-sequence data, and architectural ablations. Results report strong benchmark performance, long-sequence capability, and improved inference efficiency.

Comparative Studies: 40.5% lower FID than MLD was reported on HumanML3D, alongside up to 10% higher R Precision and a best multi-modal score of 3.060.The authors also report improvements across FID, R Precision, multi-modal distance, and diversity, with KIT-ML gains in FID and multi-modal distance.
Long Sequence Motion Generation: HumanML3D-LS contains sequences longer than 190 frames and is used to evaluate long-sequence motion generation.The variant is extracted from the original test set to expose performance on the dataset’s long-sequence tail.
Ablation Studies: Block-based bidirectional scanning achieves the best performance among the tested BSM designs, indicating that bidirectional latent-space information exchange benefits motion generation.The ablations also examine HTM arrangements, latent dimensions, and layer counts.
Ablation Studies: A latent dimension of 2 with 11 layers is reported as the superior configuration for Motion Mamba.The authors attribute the dimensionality preference to HTM’s multiple scans correlated with sequence length.
Inference Time: Motion Mamba reduces inference time relative to prior methods through an efficient Mamba block and lightweight architecture.The reported comparison uses MLD as a strong baseline and describes approximately four times less computational effort.

5 Discussion and Conclusion

Motion Mamba is presented as an efficient framework for extended-sequence motion generation using hierarchical temporal and bidirectional spatial modeling. The reported overall result is 0.281 FID at 0.058 seconds per sentence, outperforming previous methods.

5 Discussion and Conclusion: 0.058s average inference time per sentence and 0.281 FID are reported for Motion Mamba overall on a single V100 GPU.The figure compares average inference time per sentence against FID.
5 Discussion and Conclusion: Motion Mamba integrates HTM blocks for hierarchical temporal alignment and BSM blocks for bidirectional information exchange within latent spaces.The conclusion describes these designs as improving temporal consistency and skeleton-level feature capture.

1 Implementation Details

Implementation uses a latent-space Motion Mamba denoiser with fixed architectural, optimization, training, and diffusion-sampling settings. A user study compares text-motion correspondence and motion quality against prior methods and ground truth.

1 Implementation Details: Training uses AdamW with learning rate 10^-4, global batch size 512 across 4 GPUs, and 2,000 epochs.The diffusion sampling process uses 1,000 and 50 steps for the stated training and inference settings.
1 Implementation Details: The user study evaluates text-motion correspondence and overall motion quality while comparing Motion Mamba with MDM, T2M, MLD, and ground truth.Sequences are presented anonymously without revealing their generating models.

2 User Study

The user study evaluates generated motions for text-motion correspondence and overall quality using anonymous comparisons against multiple methods and ground truth. Motion Mamba outperformed the benchmark methods on both dimensions, with large reported margins.

User Study: The evaluation compared Motion Mamba with MDM, T2M, and MLD alongside a ground-truth motion baseline.The study used 20 randomly prompted motion-sequence sets from the HumanML3D test set and involved 50 participants.
User Study: The study assessed both text-to-motion correspondence and the overall quality of generated motions.Participants evaluated anonymously presented motion sequences through a user-study interface.
User Study: Motion Mamba exceeded MDM, T2M, and MLD by 79%, 74%, and 62% in text-motion correspondence, respectively.The generated sequences also closely adhered to the intended text descriptions when compared with ground-truth motions.

3 Visualization

The visualization study compares Motion Mamba with established motion-generation methods using prompted HumanML3D sequences and additional randomly selected examples. It presents motion sequences to illustrate the models’ generated behaviors and capabilities.

3 Visualization: Three prompted motion sequences were visualized to compare Motion Mamba with MotionDiffuse, MDM, and MLD.The comparison was conducted using prompts and prompt variations derived from HumanML3D.
3 Visualization: The visual comparison reported superior performance for Motion Mamba relative to MotionDiffuse, MDM, and MLD.The methods were compared through visualized motion sequences generated from three distinct prompts.
3 Visualization: Additional examples used randomly selected HumanML3D prompts to show Motion Mamba’s generated motions.These examples provide further visualizations beyond the three primary comparison sequences.
3 Visualization: The user-study interface displayed two randomly selected videos from outputs of three methods and ground truth for comparison.Participants answered evaluative questions about the presented motions.