Source-linked AI summary
LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens
Zekun Li, Sizhe An, Chengcheng Tang, Chuan Guo, Ivan Shugurov, Linguang Zhang, Amy Zhao, Srinath Sridhar, Lingling Tao, Abhay Mittal
TL;DR
Motion-language models face scarce paired data, language forgetting from LLM fine-tuning, and artifacts or duration constraints from existing motion representations. LLaMo addresses these issues with modality-specific MoT adaptation, continuous causal motion latents, and flow matching, achieving unified motion generation and understanding while preserving base language capabilities.
Problem
Motion-language modeling is underexplored, while scarce motion-text data, language forgetting, and discrete-motion artifacts constrain unified systems.
Method
LLaMo freezes text-related modules in a modality-specific MoT architecture and combines causal continuous motion latents with flow matching for autoregressive motion synthesis.
Results
LLaMo achieves unified motion generation and understanding while preserving the base LLM’s language capabilities.
Takeaways & Limitations
LLaMo provides a foundation for unified motion-language modeling within a continuous autoregressive paradigm.
Takeaways & Limitations
The MoT architecture substantially increases training cost, and the continuous autoregressive formulation requires careful tuning of training dynamics.
Abstract
from arXiv · showhide
Recent progress in large models has led to significant advances in unified multimodal generation and understanding. However, the development of models that unify motion-language generation and understanding remains largely underexplored. Existing approaches often fine-tune large language models (LLMs) on paired motion-text data, which can result in catastrophic forgetting of linguistic capabilities due to the limited scale of available text-motion pairs. Furthermore, prior methods typically convert motion into discrete representations via quantization to integrate with language models, introducing substantial jitter artifacts from discrete tokenization. To address these challenges, we propose LLaMo, a unified framework that extends pretrained LLMs through a modality-specific Mixture-of-Transformers (MoT) architecture. This design inherently preserves the language understanding of the base model while enabling scalable multimodal adaptation. We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone through a lightweight flow-matching head, allowing for streaming motion generation in real-time (>30 FPS). Leveraging the comprehensive language understanding of pretrained LLMs and large-scale motion-text pretraining, our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings, especially zero-shot motion generation, marking a significant step towards a general unified motion-language large model.
1. Introduction
LLaMo addresses scarce motion-text data, language forgetting, and limitations of discrete or fixed-length motion representations by extending pretrained LLMs for unified motion understanding and generation. Its MoT architecture, continuous causal motion latents, flow matching, and large-scale pretraining target high-fidelity, real-time motion-language modeling.
- Paired motion-text data is scarcer and more expensive than comparable image or video data, complicating large-scale motion-language modeling.
- Fine-tuning LLM text parameters on limited motion-text data causes catastrophic forgetting and weakens language-based reasoning for downstream motion tasks.
- Discrete motion tokenization introduces jitter artifacts, while continuous fixed-length approaches restrict synthesis to predetermined durations.
- The framework aims to preserve pretrained text-only performance while supporting unified motion generation and understanding across settings.
- LLaMo uses modality-specific MoT parameters, freezes text-related modules, and enables cross-modal communication through shared self-attention.
- LLaMo combines causal continuous motion latents with flow matching to support high-fidelity arbitrary-length generation and real-time streaming.
- Pretraining uses over 3 million motion sequences totaling 3,076 hours, followed by text-to-motion and motion-to-text evaluation.
2. Related Works
Related work extends unified multimodal modeling through discrete autoregressive or hybrid autoregressive-diffusion designs, while motion-language systems commonly fine-tune LLMs and use discrete motion codebooks. LLaMo instead combines continuous streaming motion generation with modality-specific parameterization intended to preserve native language performance.
- Unified multimodal models commonly use either autoregressive discrete tokens or hybrid autoregressive-diffusion generation for non-text modalities.
- These designs have limited support for continuous token generation and flexible-length context generation.
- LLaMo uses a flow-matching head to sample continuous motion latents from the autoregressive backbone for streaming generation.
- Unified human-motion methods typically fine-tune pretrained LLMs and rely on discrete vector-quantized motion codebooks.
- Prior methods fine-tune original LLM text parameters, which is associated with severe degradation in language-modeling performance.
- LLaMo is presented as integrating human motion into foundational LLMs without hurting native language performance while supporting real-time streaming generation.
3. Method
LLaMo extends a decoder-only pretrained LLM with continuous motion representations, modality-specific transformer parameters, and autoregressive motion decoding. Its design combines causal VAE tokenization, flow matching, shared self-attention, and an exit head for flexible-length motion generation while preserving language-modality processing.
- Motion Representation: The 272-dimensional motion representation contains root velocities, root angular velocity, local joint positions, local joint velocities, and local rotations.This representation is intended to mitigate inverse-kinematics errors while preserving redundant motion information.
- Continuous Motion Tokenization: A causal CNN-based VAE converts motion sequences into continuous latent tokens while preserving temporal causality and reconstructing motion frames.The encoder models temporal Gaussian latent distributions, with a temporal downsampling rate controlling latent sequence length.
- Unified Motion-Language Model Architecture: Modality-specific Mixture-of-Transformers separates text and motion parameters while shared self-attention enables cross-modal interaction and frozen text modules preserve the base language pathway.The modality-disentangled design is presented as model-agnostic and supports extending pretrained LLMs without degrading language performance.
- Unified Motion-Language Model Architecture: Motion and text embeddings are interleaved in a language-model sequence using [BOM] and [EOM] boundaries, with teacher-forcing noise added to motion latents during instruction tuning.The motion adapter aligns VAE latents with the language embedding space, while the backbone retains next-token prediction for motion understanding.
- Continuous Motion Decoding: A lightweight flow-matching head predicts continuous next-motion latents from autoregressive hidden states instead of using discrete motion-token decoding.The flow objective interpolates random noise with clean VAE latents; timestep resampling is used to stabilize training under a shifting condition distribution.
- Motion Generation Exit Head: A binary exit head predicts when continuous motion generation should end, enabling flexible-length sequences without relying on an [EOM] token.The ending signal is trained with a binary cross-entropy loss.
4. Experiments
The experiments evaluate LLaMo's motion codec, unified text-to-motion generation, motion-to-text captioning, zero-shot generation, and modality-specific parameter design. Results show comparable generation and captioning performance, plausible zero-shot motion, and severe language degradation when the full LLM is fine-tuned without MoT.
- Motion Codec: The continuous causal motion VAE is compared with quantization-based FSQ-VAE for motion tokenization.FSQ uses a 64k codebook and 512-dimensional embeddings, whereas LLaMo uses compact continuous latent vectors with z = 32.
- Text-to-Motion: LLaMo achieves semantically coherent HumanML3D text-to-motion generation comparable to MotionMillion and specialist models.The evaluation notes that FID is unreliable on HumanML3D because the dataset is limited and distributionally different from large-scale motion corpora.
- Motion-to-Text: LLaMo delivers competitive HumanML3D motion-to-text captioning, with superior CIDEr and competitive BERTScore without fine-tuning text parameters.Lower BLEU@1 alongside good BLEU@4 and ROUGE is attributed to diverse or natural wording.
- Zero-Shot Generation: Zero-shot evaluation shows plausible, semantically aligned motions for unseen complex compositional prompts from MotionMillion-Eval.The reported examples also include motion generation from non-English inputs despite no non-English text during unified training.
- Modality-Specific Parameters: Removing MoT and fine-tuning the full LLM causes MMLU and IFEval scores to collapse to near-random levels.Without text-only corpus training, both scores fall to ≤25 and ≤30, respectively, indicating severe catastrophic forgetting.
5. Limitations And Discussions
LLaMo’s modality-specific design increases training cost despite unchanged per-token inference activation cost, and its continuous autoregressive training requires careful tuning. Future work targets broader instruction-tuning coverage for motion editing and question answering.
- Doubling the model’s parameter count leaves per-token inference activation cost identical to the base LLM but substantially increases training cost.
- Continuous motion-token autoregressive training achieves better results than discrete motion codecs but requires careful tuning.
- Future work will add instruction-tuning tasks such as motion editing and motion question answering to broaden downstream applications.
6. Conclusion
LLaMo presents a continuous autoregressive motion-language model that unifies motion generation and understanding while preserving the base LLM’s language capabilities. Its large-scale pretraining and language priors establish a foundation for unified motion-language modeling.
- LLaMo introduces a large-scale continuous autoregressive framework for unified motion generation and understanding while preserving base-LLM language capabilities.
- The model combines a modality-specific Mixture-of-Transformers, frozen text-branch parameters, a causal VAE-based codec, and flow matching for continuous motion prediction.
- Results position LLaMo as a foundation for next-generation motion-language models spanning unified motion generation and understanding.
7. Implementation Details
LLaMo is implemented in multiple model sizes with frozen language components, specialized motion and flow-matching modules, and efficiency measures for training and real-time inference. Its 4× temporal downsampling supports a 30 FPS target through lower token latency.
- LLaMo is implemented at 1B, 3B, and 8B scales, including an 8B model trained for implementation and analysis.
- The causal VAE uses warmup and cosine-decay learning rates, AdamW optimization, batch size 256, and eight A100 GPUs.
- The Mixture-of-Transformers freezes language-related parameters except special token embeddings and initializes the motion transformer from the text branch.
- The flow-matching head uses a 12-layer, 1536-dimensional MLP and Euler ODE integration with 50 inference steps.
- The motion-generation exit head predicts a stop signal with a five-layer Swish MLP and binary classification loss.
- With infrastructural optimizations, an 8B model reaches real-time streaming motion generation because 7.5 FPS token generation corresponds to 30 FPS motion generation after 4× temporal downsampling.
- Training uses DeepSpeed-Zero2 and BF16 precision, while the 4× temporal downsampling sets the real-time target token latency at 133.33 ms.
- Ablation evaluation uses approximately 30K test samples and reports effects of predicted-variance VAE and multistage training choices.
8. More Ablation Studies
Ablations show that scaling beyond 3B yields limited additional gains, robust VAE noise is important for motion synthesis, and multistage training improves fidelity and text–motion consistency.
- Scaling LLaMo from 3B to 8B yields negligible improvements in FID and R-precision compared with scaling from 1B to 3B.
- Replacing the robust VAE variance sampling with a predicted-variance VAE significantly degrades motion synthesis but does not affect motion understanding.
- Multistage training produces steady improvements in motion fidelity and text–motion consistency, stabilizing learning dynamics in large models.
- The staged procedure progressively specializes optimization, mitigating early instability and improving modality alignment and zero-shot performance.
9. Data Curation Details
The data curation pipeline combines Gemini-based semantic annotation with kinematic filtering to remove static or under-expressive motion sequences.
- Gemini is used to annotate human-motion videos, with the complete annotation prompt provided in the supplementary materials.
- Sequences are filtered when all end-effector velocities remain below 5 cm/s, targeting static or near-static and under-expressive motions.The threshold is intended to capture natural micro-movement during quiet standing.
10. Zero-shot Text-to-Motion Generation
LLaMo is evaluated for zero-shot text-to-motion generation through human studies, cross-language prompts, and semantic-distribution analysis. The results indicate smoother, more plausible, better-aligned motion and broader semantic coverage than the compared baseline and dataset.
- User Study: 14 participants preferred LLaMo over MotionMillion across Physical Plausibility, Motion Smoothness, and Text Alignment.Participants compared motions generated from the same prompts without knowing the model identity.
- User Study: LLaMo produces smoother, more physically plausible, and better text-aligned motions than MotionMillion despite comparable text-token parameter budgets.The comparison attributes these qualities to high-fidelity continuous motion representations and retained native language capabilities.
- Unseen Languages: LLaMo generates motion from prompts in languages beyond English despite training on English language-motion data.The paper presents this as a qualitative emergent behavior with examples in the supplementary materials.
- Dataset Comparison: The t-SNE analysis compares text-embedding distributions from MotionMillion and an internal dataset after Qwen3-Embedding-0.6B sentence encoding.K-means clustering is performed independently for each dataset, with density-based subsampling for visualization.
- Dataset Comparison: MotionMillion shows pronounced inter-cluster gaps and smaller intra-cluster spread, while the internal dataset covers the embedding space more uniformly.The authors interpret this pattern as limited semantic diversity and greater semantic redundancy in MotionMillion.
11. Motion Auto-encoder Comparison
The comparison contrasts MotionMillion’s discrete FSQ tokenizer with LLaMo’s continuous sigma-causal TAE for motion reconstruction. Continuous latent representations preserve more motion fidelity than quantized representations.
- FSQ quantization introduces information loss that produces jittery motion and removes fine-grained reconstruction details.
- LLaMo’s continuous sigma-causal TAE preserves higher fidelity to the original motion than the discrete FSQ tokenizer.The passage frames this as a qualitative reconstruction advantage over quantization.
- The comparison links discrete tokenization artifacts to quantization and contrasts them with continuous latent motion representation.