PaperlayerView exact revision on arXiv

Source-linked AI summary

FLAME: Free-form Language-based Motion Synthesis & Editing

Jihoon Kim, Jiseob Kim, Sungjoon Choi

arXiv:2209.00349v2cs.CVcs.GR

TL;DR

FLAME addresses limited flexible conditional generation in text-to-motion systems with a diffusion-based model for free-form synthesis and motion editing. Its transformer-based design handles spatio-temporal, variable-length motion and supports synthesis, editing, forecasting, and in-betweening without fine-tuning.

  • Problem

    Prior free-form text-to-motion methods lack flexible conditional generation, while motion data also requires handling temporal structure and variable lengths.

  • Method

    FLAME uses diffusion with a transformer-based architecture, language cross-attention, and motion tokens to synthesize and edit motion from free-form text.

  • Results

    FLAME achieves state-of-the-art generation performance on HumanML3D, BABEL, and KIT and extends editing to motion prediction and in-betweening without fine-tuning.

  • Takeaways & Limitations

    A unified model can support text-to-motion generation, frame-wise and joint-wise editing, motion prediction, and in-betweening.

  • Takeaways & Limitations

    Sampling 1,000 diffusion steps takes more than 30 seconds per sample, hindering practical application; the authors identify real-time sampling as future work.

Abstract

from arXiv · show

Text-based motion generation models are drawing a surge of interest for their potential for automating the motion-making process in the game, animation, or robot industries. In this paper, we propose a diffusion-based motion synthesis and editing model named FLAME. Inspired by the recent successes in diffusion models, we integrate diffusion-based generative models into the motion domain. FLAME can generate high-fidelity motions well aligned with the given text. Also, it can edit the parts of the motion, both frame-wise and joint-wise, without any fine-tuning. FLAME involves a new transformer-based architecture we devise to better handle motion data, which is found to be crucial to manage variable-length motions and well attend to free-form text. In experiments, we show that FLAME achieves state-of-the-art generation performances on three text-motion datasets: HumanML3D, BABEL, and KIT. We also demonstrate that editing capability of FLAME can be extended to other tasks such as motion prediction or motion in-betweening, which have been previously covered by dedicated models.

Introduction

The paper addresses limited expressiveness and flexible conditional-generation capabilities in prior text-to-motion systems by introducing FLAME, a diffusion-based model for free-form motion synthesis and editing.

  • Behavioral labels such as walking, jumping, or dancing limit motion diversity and controllability because they lack descriptive power.
  • FLAME generates motion aligned with free-form prompts and edits reference motions from textual descriptions.
  • Diffusion enables text-to-motion synthesis and conditional editing, including forecasting and in-betweening, without fine-tuning or modifying the trained model.
  • FLAME introduces a transformer decoder architecture for spatio-temporal motion and variable-length sequences, using language and motion tokens with cross-attention.
  • The model is presented as a unified system for free-form motion synthesis and editing that also supports more diverse motions and classical prediction tasks.

Related Work

Prior diffusion research established high-quality conditional generation mainly for images, while motion-generation research developed separate approaches for labels, text, prediction, and in-betweening.

  • Diffusion models improved multimodal image generation quality compared with GANs and VAEs, motivating their use for text-to-motion generation.
  • Diffusion models progressed from unconditional image generation to class-conditional and text-conditional generation.
  • Classifier-free guidance enables conditional generation without a separate classifier, and FLAME employs it alongside learned reverse-diffusion variances.

3D Human Motion Generation

Earlier motion models addressed prediction, in-betweening, and text-to-motion synthesis separately, whereas FLAME combines high-quality text-to-motion generation with flexible text-based editing without fine-tuning.

  • Motion prediction forecasts subsequent frames, while in-betweening generates natural sequences satisfying starting and target poses.
  • Existing prediction and in-betweening models lack a single-model solution for multiple tasks and cannot synthesize motion from textual descriptions.
  • FLAME performs text-based motion editing, including motion prediction and in-betweening, without fine-tuning or modifying the trained generative model.

Proposed Method: FLAME

FLAME applies conditional diffusion to motion synthesis and editing, using a transformer decoder designed for spatio-temporal, variable-length motion. The model supports text conditioning, denoising-based generation, and frame- or joint-wise editing.

  • Diffusion-based Model: The diffusion process gradually adds noise to motion, reducing complex distribution modeling to a sequence of denoising problems.
  • Diffusion-based Model: FLAME reverses a diffusion process that transforms Gaussian noise into motion conditioned on language descriptions.Inference starts from M_T ∼ N(0, I) and repeatedly samples pθ(M_t−1|M_t, c) from t = T to 1.
  • Diffusion-based Model: FLAME trains a model to predict noise and reverse-process variance, using a hybrid loss because the simplified loss alone cannot learn variance.The model predicts the noise component and uses the hybrid loss proposed by Nichol and Dhariwal to train variance.
  • Model Architecture for Motion Data: Because motion combines temporal and spatial patterns with variable sequence lengths, FLAME replaces image-oriented U-Nets with a transformer-based architecture.Variable length is handled by masking in the transformer decoder.
  • Model Architecture for Motion Data: The transformer projects motion frames into tokens, conditions them through RoBERTa-derived cross-attention, and uses time-step and motion-length tokens.Its output represents both the predicted noise and variance for the reverse distribution.
  • Inference: Classifier-free guidance strengthens text conditioning during denoising, trading a small amount of sample diversity for substantially higher precision.The guided prediction amplifies the conditioning effect with a scalar s > 1.
  • Inference for Motion Editing: Motion editing uses a diffuse-then-conditionally-denoise strategy to fill masked frame-wise or joint-wise regions while bridging unedited and edited distributions.

Experiments

FLAME is evaluated across three datasets and multiple motion-generation settings, including text alignment, diversity, editing, and conditional tasks. The experiments also examine architectural components and the sampling-speed trade-off.

  • Datasets: Experiments use HumanML3D, BABEL, and KIT, with SMPL-based representations for HumanML3D and BABEL and a 64-dimensional representation for KIT.HumanML3D contains 44,970 descriptions for 14,616 motions; BABEL uses 63,353 frame-level annotations; KIT contains 3,911 sequences paired with 6,353 descriptions.
  • Evaluation Metrics: Evaluation measures positional and variance errors, motion-text alignment, feature distance, mutual-information divergence, and retrieval accuracy.R-Precision ranks motion and text features by Euclidean distance, while mCLIP uses cosine similarity between separately trained motion and text embeddings.
  • Quantitative Results on Text-to-motion: FLAME outperforms compared models on all reported metrics except variance metrics, while additional tests assess joint variance, multimodality, and mCLIP alignment.The paper averages reported metrics over three trials and samples 10 motions per test annotation for the diversity analysis.
  • Ablation Study: The ablation study evaluates self-attention, a motion-length token, cross-attention, and language-model freezing as components of the architecture.The motion-length token explicitly supplies the number of frames to generate, and cross-attention uses the transformer decoder architecture.
  • Sampling Speed: Sampling with 1,000 diffusion steps takes more than 30 seconds per sample, but reduced steps preserve sample quality unless reduced extremely.The speed measurements are recorded on a single NVIDIA Tesla V100 SXM2 32GB machine.
  • Application on Other Motion Tasks: FLAME’s editing method extends to motion prediction and in-betweening through flexible conditional generation.These applications use conditioning frames for the generated motion sequence.

Conclusion

The paper presents FLAME as a unified diffusion-based model for text-to-motion generation and text-based motion editing. It emphasizes sample quality and flexible conditional generation, while identifying real-time sampling and cross-domain feature use as future directions.

  • Conclusion: FLAME unifies text-to-motion generation and text-based motion editing in a diffusion-based model.The paper distinguishes the model from previous work by its sample quality and flexibility in conditional generation.
  • Conclusion: The authors expect FLAME to streamline motion generation and lower the barrier to 3D motion synthesis.These are stated expectations rather than measured outcomes in the supplied conclusion passage.
  • Conclusion: Future work targets improved sampling for real-time application and reuse of features learned in image-vision domains.

Appendix A: Data Preprocessing

The appendix describes preprocessing for HumanML3D, BABEL, and KIT, including motion representations, annotation selection, dataset splitting, and evaluation compatibility.

  • HumanML3D: HumanML3D combines AMASS and HumanAct12 annotations, uses SMPL motion data, follows the original train-validation-test split, and processes motions at 20Hz.The representation primarily uses 24 body joints because most dataset motions contain those joint values.
  • BABEL: BABEL annotations include sequence-level and frame-level labels, but experiments use only frame-level annotations because they align more precisely with motion.
  • Motion Length: Figure 1 reports the motion-length distribution used in the preprocessing context.
  • KIT: KIT uses the same motion pipeline and preprocessing as TEMOS for fair comparison.

Appendix B: Derivation

The appendix derives the diffusion training objective for continuous motion, combining simple noise prediction with a variational lower-bound term and adapting the final reverse step to continuous values.

  • Variational objective: The reverse-process objective decomposes into LT, intermediate Lt−1 terms, and L0, with LT constant when forward variances are fixed.The derivation represents each intermediate term as a KL divergence between the forward posterior and learned reverse transition.
  • Forward process: The forward process uses Gaussian noise ϵ ∼ N(0, I), with αt = 1 − βt and cumulative product ¯αt.These quantities parameterize the noisy motion sequence at timestep t.
  • Continuous motion: For continuous motion, the same Gaussian-derived reverse-process form is used for L0 rather than an independent discrete decoder.This differs from DDPM’s image setting, where pixel values are discrete.
  • Training objective: The combined objective is Lhybrid = Lsimple + λLvlb, where Lvlb sums L0 through LT.Improved DDPM learns the variance instead of treating the forward-process variances as fixed constants.

Appendix C: Evaluation Details

The evaluation appendix defines motion-diversity and position-error measurements, describes feature extraction and training stabilization, and lists the associated implementation settings.

  • Error and diversity metrics: APE and AVE are computed as described in TEMOS, while the same joint-diversity equation also measures diversity across joints.Motion notation distinguishes temporal averages, joint indices, frame indices, and the total sample count.
  • Diversity metrics: Multimodality measures motion diversity generated for each text annotation by comparing feature vectors from two sample subsets.The experiments generate Sl samples per annotation and set Sl = 10.
  • Implementation: Exponential moving average is used to stabilize FLAME training, with separate hyperparameter tables for HumanML3D, BABEL, and KIT.The HumanML3D and BABEL settings are listed together, while KIT uses a separate table.
  • Feature extraction: A separate CLIP-like contrastive model extracts high-level features from motion and text for evaluation.Its training hyperparameters are listed separately from FLAME’s dataset-specific settings.

Appendix E: Additional Experiments

Additional experiments compare DDPM and DDIM sampling and examine how classifier-free guidance scale affects text-conditional generation.

  • Sampling strategies: FLAME supports both DDPM and DDIM sampling strategies for generation.Sampling-step comparisons are reported for the two strategies.
  • Guidance scale: Classifier-free guidance is varied across sampling runs to evaluate the effect of guidance strength on text-conditional generation.The results are presented in a quantitative table organized by guidance scale.

Additional Qualitative Results

Additional qualitative material illustrates FLAME’s text-to-motion synthesis and free-form editing capabilities, alongside quantitative experiment references.

  • Quantitative references: The additional experiments reference quantitative comparisons across sampling steps and guidance scales.These include DDPM/DDIM sampling results and guidance-scale results.
  • Text-to-motion synthesis: Figure 2 provides additional examples of text-to-motion synthesis.The examples are qualitative demonstrations of generated motion from language.
  • Motion editing: Figure 3 demonstrates free-form language-based editing of upper-body motion, including the root pelvis joint.The figure highlights joint-selective editing in a reference motion.
Loading 2209.00349v2…