Source-linked AI summary
MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Space
Lixing Xiao, Shunlin Lu, Huaijin Pi, Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, Jingbo Wang
TL;DR
Streaming motion generation must respond to sequential text while maintaining coherent motion from variable-length history, but existing diffusion and discrete-token autoregressive methods impose fixed lengths, delayed decoding, or error accumulation. MotionStreamer addresses this with continuous causal latents, diffusion-based autoregression, causal online decoding, and Two-Forward training; experiments report state-of-the-art performance across text-to-motion and long-term synthesis, with multi-round and compositional applications. Its unidirectional streaming design limits motion inbetweening and localized editing.
Problem
Streaming motion generation requires online response to sequential text and coherent synthesis from historical motion, while existing methods face fixed-length, delayed-decoding, or discrete-token limitations.
Method
MotionStreamer combines a diffusion head and autoregressive model with a continuous causal motion latent space, causal decoding, and Two-Forward training.
Results
MotionStreamer achieves state-of-the-art performance on text-to-motion and long-term motion synthesis across HumanML3D and BABEL, while supporting multi-round generation, long-term generation, and dynamic motion composition.
Takeaways & Limitations
The framework provides streaming motion generation with progressive text inputs, online response, and flexible multi-round, long-term, and dynamic composition capabilities.
Takeaways & Limitations
Because streaming generation is unidirectional, MotionStreamer limits motion inbetweening and localized editing of intermediate tokens.
Abstract
from arXiv · showhide
This paper addresses the challenge of text-conditioned streaming motion generation, which requires us to predict the next-step human pose based on variable-length historical motions and incoming texts. Existing methods struggle to achieve streaming motion generation, e.g., diffusion models are constrained by pre-defined motion lengths, while GPT-based methods suffer from delayed response and error accumulation problem due to discretized non-causal tokenization. To solve these problems, we propose MotionStreamer, a novel framework that incorporates a continuous causal latent space into a probabilistic autoregressive model. The continuous latents mitigate information loss caused by discretization and effectively reduce error accumulation during long-term autoregressive generation. In addition, by establishing temporal causal dependencies between current and historical motion latents, our model fully utilizes the available information to achieve accurate online motion decoding. Experiments show that our method outperforms existing approaches while offering more applications, including multi-round generation, long-term generation, and dynamic motion composition. Project Page: https://zju3dv.github.io/MotionStreamer/
1. Introduction
MotionStreamer targets streaming motion generation with sequential text inputs, variable-length motion history, online response, and contextual consistency. It combines continuous causal latents, diffusion-based autoregression, and specialized training strategies to support accurate and flexible generation.
- Motivation: Streaming motion generation must incrementally process arriving text while continuously synthesizing contextually consistent human motions.The target setting requires online response and integration of historical motion with incoming textual conditions.
- Limitations of Existing Methods: Fixed-length diffusion models cannot dynamically adapt to evolving text, while non-causal autoregressive tokenization delays partial decoding.Existing paradigms are constrained either by static conditioning and fixed-length generation or by decoding requirements that prevent immediate responses.
- Proposed Framework: MotionStreamer combines a diffusion head with an autoregressive model and a causal motion compressor to predict continuous motion latents for online decoding.The autoregressive model conditions on text features and historical motion latents, while the causal compressor supports streaming decoding.
- Training: Two-Forward and Mixed training address error accumulation, exposure bias, and compositional generalization for multi-round and long-term generation.Two-Forward mixes generated and ground-truth latents during training, while Mixed training supports unseen motion combinations.
- Results and Applications: Experiments on HumanML3D and BABEL report state-of-the-art performance on text-to-motion and long-term synthesis, alongside multi-round, long-term, and dynamic composition applications.The framework supports progressive text inputs, multiple texts, and regeneration of subsequent motions while preserving an initially generated prefix.
2. Related Work
Prior work spans text-conditioned motion generation, real-time autoregressive control, and motion compression. MotionStreamer is positioned as a strictly causal alternative that handles variable-length history and incremental generation.
- Text-conditioned Motion Generation: Text-conditioned motion generation synthesizes 3D human motion from natural-language descriptions using VAE, diffusion, and other architectures.These approaches target naturalness and expressiveness for digital humans and motion animation.
- Real-time Motion Control: Real-time autoregressive methods have been applied to interactive character control and text-driven motion control.Examples include Ready-to-React, CLoSD, and DART.
- Causality and Streaming: Existing real-time methods rely on fixed-length context windows, whereas MotionStreamer handles variable-length historical information and incrementally generates motions.This distinction concerns strict causality and streaming generation.
- Motion Compression: VQ-based motion compression discretizes motion into tokens before GPT-style generation, while residual quantization reduces but does not eliminate quantization error.This compression paradigm motivates continuous latent representations.
3. Method
The method builds a causal continuous latent pipeline for streaming motion generation from sequential text prompts and historical motion. A diffusion-based autoregressive Transformer predicts latents that are decoded online, with explicit stopping and training mechanisms.
- 3.1. Problem Formulation: The task maps sequential text prompts to an online sequence of motion frames rather than generating from one fixed text prompt.Each prompt arrives incrementally, and the output frames are generated online.
- 3.1. Motion Representation: Each pose uses a 272-dimensional SMPL-based representation that directly supports character animation without post-processing.For K = 22 joints, the representation totals 272 dimensions.
- 3.2. Causal Temporal AutoEncoder: Causal TAE uses 1D temporal causal convolutions in its encoder and decoder to produce continuous latents while preserving temporal causality.The encoder outputs Gaussian parameters and reparameterized latents Z with temporal downsampling.
- 3.2. Causal Temporal AutoEncoder: Causal TAE training combines reconstruction, KL-divergence, and root-joint losses, with dc = 16 selected as the best latent dimension in experiments.The motion dimension is D = 272, and the temporal downsampling rate is l = 4.
- 3.2. Causal Temporal AutoEncoder: Causal decoding enables online motion reconstruction without future frames and avoids the discretization bottleneck of VQ-based methods.Continuous token representations address information loss associated with discrete tokenization.
- 3.3. MotionStreamer: MotionStreamer concatenates text embeddings, historical latents, and current latents, then uses a causal diffusion-based autoregressive Transformer to predict motion latents.Transformer condition features guide a diffusion head, and predicted latents are passed to the causal decoder as they are generated.
- 3.3. MotionStreamer: Two-Forward training replaces some ground-truth latents with first-pass predictions before a gradient-bearing second pass to reduce exposure bias and error accumulation.A cosine scheduler controls the proportion of replaced latents.
- 3.3. MotionStreamer: An encoded impossible-pose latent provides a continuous stopping reference, allowing generation length to be determined automatically for each text prompt.Generation stops when the current latent is sufficiently close to the reference end latent.
4. Experiment
Experiments evaluate MotionStreamer on text-to-motion, long-term generation, latency, qualitative behavior, applications, and ablations. Results show strong generation quality, online response, dynamic composition, and benefits from continuous causal compression and the proposed training design.
- Experimental Setup: MotionStreamer is evaluated on HumanML3D and BABEL using text-to-motion and long-term motion generation benchmarks.HumanML3D provides text-to-motion evaluation, while BABEL supports long-term generation with sequential textual inputs.
- Quantitative Results: The method achieves better performance across multiple metrics on the HumanML3D test set and outperforms existing approaches for long-term generation on BABEL.The comparisons include adapted streaming versions of T2M-GPT and a VQ-discretized variant of MotionStreamer.
- Quantitative Results: Causal TAE achieves the lowest First-frame Latency because predicted latents can be decoded immediately, unlike non-causal or fixed-length alternatives.Non-causal VAE latency increases with generated-frame count because decoding waits for the entire sequence.
- Qualitative Results: Continuous latents preserve finer motion details and improve qualitative alignment, while VQ-based baselines exhibit incorrect actions, detail loss, or sliding artifacts.For one-leg jumping, several baselines generate two-leg or alternating-leg motion, whereas MotionStreamer preserves the described action more accurately.
- Ablation Study: Causal TAE outperforms VQ-VAE and non-causal VAE in reconstruction and generation, while latent dimension and hidden size require a balance between compression and model capacity.Larger latent dimensions improve reconstruction but can worsen generation because insufficient compression makes meaningful motion generation harder.
- Applications: MotionStreamer supports multi-round generation, long-term generation, and dynamic motion composition while preserving previously generated motion prefixes.Causal TAE avoids full-sequence re-decoding when subsequent motion latents are generated.
5. Conclusion
MotionStreamer combines diffusion-based autoregression with causal motion latents to support online streaming generation, mitigate cumulative errors, and enable multiple motion-generation applications.
- MotionStreamer integrates a diffusion-based autoregressive model to directly predict causal motion latents.
- Causal TAE supports online response to progressive textual inputs.
- Two-Forward training mitigates cumulative errors during streaming generation.
- The framework supports multi-round, long-term, and dynamic motion composition applications.
A. Implementation Details
The implementation specifies training configurations for the causal autoencoder, autoregressive Transformer, diffusion head, and Two-Forward replacement schedule.
- The Causal TAE downsamples temporally by 4 and trains on motion sequences cropped to N = 64 frames.
- Training uses AdamW with batch sizes of 128 for the Causal TAE and 256 for the autoregressive model.
- The AR Transformer uses 12 layers, 12 attention heads, and a 768-dimensional hidden state, while its diffusion head has 9 layers.
- The replacement schedule increases from γ_t = 0 at t = 0 to γ_t = 1 at t = T, transitioning from ground-truth to generated motion tokens.
B. Causal TAE Architecture
The Causal TAE encodes motion into a lower-dimensional latent space and progressively decodes it to reconstruct motion.
- Input motion sequences are encoded into a latent space using a 1D causal ResNet.
- The latent representation is projected to Gaussian distribution parameters before a linear adapter reduces its dimensionality.
- Sampling occurs in the lower-dimensional latent space, while the decoder mirrors the encoding process for progressive reconstruction.
C. AR Model Architecture
The AR model architecture is selected through ablations over Transformer and diffusion-head capacity, resulting in a 12-layer configuration with 12 heads and a 768-dimensional hidden state.
- The ablation varies Transformer layers, attention heads, hidden dimension, and diffusion-head layers.
- The final architecture uses 12 Transformer layers, 12 attention heads, a 768-dimensional hidden state, and a 9-layer diffusion head.
D. Classifier-free guidance
Classifier-free guidance conditions the diffusion head during inference, with the guidance scale selected through ablation. The section also describes the 263-dimensional motion representation and post-processing challenges involving inverse kinematics and SMPLify.
- D. Classifier-free guidance: During training, 10% of text is replaced with blank text to create unconditioned samples for classifier-free guidance.
- D. Classifier-free guidance: During inference, classifier-free guidance adjusts denoising noise by combining unconditioned and conditioned predictions with scale s.The formulation is ε_g = ε_u + s(ε_c − ε_u).
- D. Classifier-free guidance: The ablation study evaluates CFG scale s, and the authors choose s = 4.0 for all experiments.
- Post-processing: The motion representation contains root motion, local joint positions, velocities, rotations, and contact labels, totaling 263 dimensions for SMPL characters with K = 22 joints.The representation includes 2 + 1 + 1 + 3 × 21 + 3 × 22 + 6 × 21 + 4 dimensions.
- Post-processing: Direct inverse kinematics loses twist rotation and causes rotation errors, while SMPLify-based recovery is time-consuming and can produce jittering results.SMPLify takes around 60 seconds for a 10-second motion clip.
F. Limitations and Future work
The streaming paradigm restricts bidirectional operations on intermediate motion tokens, limiting inbetweening and localized editing. Future work proposes predicting future latents to support these operations while retaining streaming generation.
- Limitations: Unidirectional streaming modeling limits motion inbetweening and localized editing of intermediate tokens.
- Limitations: Fine-grained insertion or refinement is restricted because intermediate adjustments must preserve global motion coherence.
- Future work: Hybrid bidirectional refinement is proposed as future work to add editing flexibility without compromising streaming generation.
- Future work: Predicting a set of future latents at each step could enable motion in-between and localized editing while preserving streaming operation.
Sample
The figures and table describe the Causal TAE and Transformer-block architectures used in the framework. The Causal TAE samples motion latents in a continuous causal latent space.
- Causal TAE: The Causal TAE samples motion latents in a continuous causal latent space.
- Autoregressive model: The Transformer-block architecture applies QK Norm to enhance training stability in the autoregressive model.
- Causal TAE: A table provides the detailed architecture of the proposed Causal TAE.