Source-linked AI summary
FrankenMotion: Part-level Human Motion Generation and Composition
Chuqiao Li, Xianghui Xie, Yong Cao, Andreas Geiger, Gerard Pons-Moll
TL;DR
Existing motion-generation methods lack fine-grained temporal and body-part control because datasets and models do not capture temporally coherent part-level features. FrankenMotion addresses this with LLM-annotated atomic motion data and hierarchical conditioning, outperforming adapted baselines while composing novel movements.
Problem
Existing datasets and methods lack temporally aligned, body-part-specific annotations, limiting fine-grained control over motion dynamics and individual parts.
Method
FrankenMotion trains a transformer-based diffusion model on the FrankenStein dataset to compose motion using sequence-, action-, and temporally structured part-level text.
Results
FrankenMotion consistently outperforms adapted state-of-the-art methods for part-based motion generation in semantic correctness and realism, while composing movements unseen during training.
Takeaways & Limitations
The framework provides fine-grained spatial–temporal control and a foundation for compositional motion generation.
Takeaways & Limitations
The annotation framework allows atomic action and part labels to remain unknown because annotating every element is expensive and unnecessary.
Abstract
from arXiv · showhide
Human motion generation from text prompts has made remarkable progress in recent years. However, existing methods primarily rely on either sequence-level or action-level descriptions due to the absence of fine-grained, part-level motion annotations. This limits their controllability over individual body parts. In this work, we construct a high-quality motion dataset with atomic, temporally-aware part-level text annotations, leveraging the reasoning capabilities of large language models (LLMs). Unlike prior datasets that either provide synchronized part captions with fixed time segments or rely solely on global sequence labels, our dataset captures asynchronous and semantically distinct part movements at fine temporal resolution. Based on this dataset, we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion, where each body part is guided by its own temporally-structured textual prompt. This is, to our knowledge, the first work to provide atomic, temporally-aware part-level motion annotations and have a model that allows motion generation with both spatial (body part) and temporal (atomic action) control. Experiments demonstrate that FrankenMotion outperforms all previous baseline models adapted and retrained for our setting, and our model can compose motions unseen during training. Our code and dataset will be publicly available upon publication.
1. Introduction
FrankenMotion addresses the lack of temporally aligned, part-level motion annotations that limits fine-grained control. It introduces the FrankenStein dataset and a compositional model for generating complex motions from body-part, action, and sequence prompts.
- Existing motion-generation methods lack fine-grained temporal and body-part control because datasets rarely provide aligned annotations.Prior part-level approaches still fail to capture temporally coherent part-level features needed for controllable generation and editing.
- FrankenMotion treats body parts as atomic elements, mapping their prompts into temporal motion segments while using action descriptions to compose meaningful complex motions.The framework is designed around compositional relationships among simpler motion elements.
- The FrankenStein dataset provides automatically generated structured part-level annotations for motion data.It is presented as the largest part-level human motion dataset at the time of the paper.
- FrankenMotion uses hierarchical conditioning on part-, action-, and sequence-level text for fine-grained control over body parts and timing.The model supports control at sequence, action, and part levels.
- 93.08% of FrankenAgent annotations were considered correct by human experts.On the annotated FrankenStein data, FrankenMotion also outperformed state-of-the-art methods in semantic correctness and realism.
2. Related Work
Related work has expanded controllable motion generation across modalities and granularity levels. However, existing spatial-control methods remain limited by synchronized stage annotations or incomplete part-motion composition.
- Controllable motion generation uses text, inter-person signals, key poses or trajectories, and audio to guide motion synthesis.The field includes methods for environmental context, interaction, structural control, and cross-modal transitions.
- FineMoGen enables per-body-part control but uses synchronized stage-based intervals across parts, limiting temporal flexibility.This constraint distinguishes it from methods targeting asynchronous part movements.
- STMC performs spatial composition by assembling body-part motions from pretrained diffusion components.
3. FrankenStein Dataset Construction
FrankenStein extends existing motion–language datasets with hierarchical, temporally aligned annotations for sequences, atomic actions, and individual body parts. An LLM-based pipeline generates these annotations, producing a large-scale dataset with fine-grained spatial–temporal grounding.
- Annotation Structure: The annotation scheme spans three granularities: full sequences, coarse atomic actions, and fine-grained body-part movements over time.This hierarchical design represents individual parts such as the head, arms, legs, spine, and trajectory.
- Dataset Motivation: Existing datasets mainly provide coarse sequence or action labels, lacking hierarchical temporal decomposition and body-part-specific annotations.These limitations restrict fine-grained understanding of which body parts perform each motion and when.
- Annotation Structure: Each annotation element pairs a text label with the start and end times of its described motion segment.The structured collection covers sequence-level annotations, non-overlapping atomic actions, and body-part-specific atomic annotations.
- LLM-based Annotation Framework: An LLM-based FrankenAgent constructs body-part annotations from existing sequence and atomic-action annotations, using prompts for temporal alignment and explicit body-part coverage.The pipeline instructs the agent to output unknown when uncertain and uses Deepseek-R1 as its primary model.
- FrankenMotion: FrankenMotion is trained on the paired structured annotations and supports sequence-, action-, and part-level conditioning for composing complex motions.Its overview describes a transformer-based diffusion model trained to learn essential motion elements and compose them into complex motions.
- Dataset Statistics: FrankenStein provides hierarchical and atomic labels across approximately 16k sequences and 265k atomic motion segments.The dataset spans 39 hours, contains around 15.7k, 31.5k, and 46.1k annotations at its three levels, and has an average segment duration of 4.8 seconds.
4. FrankenMotion: Part-Based Spatiotemporal Composition
FrankenMotion uses a transformer-based diffusion model to generate motion from sequence-, action-, and temporally structured body-part prompts. Its joint embedding and stochastic masking strategy support fine-grained composition and robustness to sparse conditioning.
- Multi-granularity control: The model supports sequence-level text, windowed action labels, and per-frame body-part prompts as three control granularities.Body-part prompts are the most fine-grained condition and enable part-based motion composition and editing.
- Multi-granularity control: Users can provide only sequence descriptions, sparse part prompts, or edits to existing control signals during inference.
- Diffusion generation: The diffusion model predicts a clean motion sequence from noisy motion, the diffusion timestep, and hierarchical sequence, action, and part-level text conditions.The output uses the pose representation defined for the generated motion sequence.
- Spatio-temporal embedding: Sequence, action, part-level text, and motion features are fused into a joint embedding, with action embeddings expanded across their temporal windows.A global sequence feature and diffusion-timestep embedding are appended or added to the fused representation.
- Robust training: Random Beta-distributed masking of labelled body-part text conditions improves robustness to incomplete conditioning under sparse annotations.Unknown text labels are zeroed, and each training step samples a masking probability for body-part labels.
- Robust training: Training uses the standard DDPM objective with a cosine noise schedule, 100 diffusion steps, AdamW, a 2×10^-4 learning rate, and batch size 32.The model trains for approximately 47.5 hours on one NVIDIA H100 GPU.
5. Experiments
Experiments evaluate FrankenMotion’s dataset quality, hierarchical conditioning, fine-grained generation, and flexible inference control. Across the reported settings, it outperforms adapted baselines in semantic correctness and realism while supporting multiple conditioning granularities.
- Dataset Quality: 93.08% annotation accuracy and AC1 = 0.91 indicate that FrankenAgent produces highly reliable part, action, and sequence labels.The human evaluation sampled 50 motion sequences and used three experts, with binary correctness scores averaged across labels.
- Fine-grained Motion Generation: FrankenMotion supports hierarchical conditioning from sequence, atomic-action, and body-part text, unlike prior methods adapted for this task.Baselines include adapted UniMotion, STMC, and DART models trained or conditioned on the FrankenStein annotations.
- Fine-grained Motion Generation: Across part, action, and sequence settings, FrankenMotion achieves the best semantic correctness and realism, outperforming all prior baselines.Table 2 reports 95% confidence intervals after 20 repeated evaluations.
- Fine-grained Motion Generation: Qualitatively, FrankenMotion composes complex body-part motions coherently and follows detailed prompts and high-level semantics more precisely than the baselines.The comparison contrasts its behavior with STMC’s composition difficulties, DART’s repetitive motions, and UniMotion’s weaker adherence to intricate details.
- Ablation Study: Part-only conditioning already performs strongly, while adding atomic-action and sequence text further improves part-level correctness and realism.The added higher-level texts provide semantics for the desired motion and make generated motions more meaningful.
- Flexible Input Control Applications: Its modular design and sparse dataset structure allow users to control generation with a dominant body part, an action phrase, or a sequence description.The available conditioning granularity can vary with the text provided or the user’s preference.
6. Conclusion and Limitation
The paper introduces FrankenStein and FrankenMotion for multi-level spatiotemporal motion conditioning and compositional generation. The main stated limitation is that FrankenMotion cannot generate minute-long sequences in one pass.
- Conclusion: FrankenStein provides fine-grained, temporally aligned atomic part-level annotations derived through LLM reasoning.FrankenMotion learns to compose motion from these atomic elements.
- Conclusion: FrankenMotion enables controllable generation at sequence, atomic-action, and atomic-part levels with fine-grained spatial–temporal control.The conclusion describes this as multi-level spatiotemporal motion conditioning.
- Conclusion: Experiments show that FrankenMotion outperforms adapted baselines and establishes a foundation for compositional motion generation.The conclusion frames compositional generation as the supported scientific outcome.
- Limitation: FrankenMotion cannot yet generate minute-long motion sequences within a single pass.Extending long-term temporal structure is identified as future work.