PaperlayerView exact revision on arXiv

Source-linked AI summary

FLAME: Free-form Language-based Motion Synthesis & Editing

Jihoon Kim, Jiseob Kim, Sungjoon Choi

arXiv:2209.00349v2cs.CVcs.GR

TL;DR

FLAME addresses limited flexible conditional generation in text-to-motion systems with a diffusion-based model for free-form synthesis and motion editing. Its transformer-based design handles spatio-temporal, variable-length motion and supports synthesis, editing, forecasting, and in-betweening without fine-tuning.

Abstract

from arXiv · show

Text-based motion generation models are drawing a surge of interest for their potential for automating the motion-making process in the game, animation, or robot industries. In this paper, we propose a diffusion-based motion synthesis and editing model named FLAME. Inspired by the recent successes in diffusion models, we integrate diffusion-based generative models into the motion domain. FLAME can generate high-fidelity motions well aligned with the given text. Also, it can edit the parts of the motion, both frame-wise and joint-wise, without any fine-tuning. FLAME involves a new transformer-based architecture we devise to better handle motion data, which is found to be crucial to manage variable-length motions and well attend to free-form text. In experiments, we show that FLAME achieves state-of-the-art generation performances on three text-motion datasets: HumanML3D, BABEL, and KIT. We also demonstrate that editing capability of FLAME can be extended to other tasks such as motion prediction or motion in-betweening, which have been previously covered by dedicated models.

Introduction

The paper addresses limited expressiveness and flexible conditional-generation capabilities in prior text-to-motion systems by introducing FLAME, a diffusion-based model for free-form motion synthesis and editing.

  • Behavioral labels such as walking, jumping, or dancing limit motion diversity and controllability because they lack descriptive power.Source: Introduction · p. 1
  • FLAME generates motion aligned with free-form prompts and edits reference motions from textual descriptions.Source: Introduction · p. 1
  • Diffusion enables text-to-motion synthesis and conditional editing, including forecasting and in-betweening, without fine-tuning or modifying the trained model.Source: Introduction · p. 2
  • FLAME introduces a transformer decoder architecture for spatio-temporal motion and variable-length sequences, using language and motion tokens with cross-attention.Source: Introduction · p. 2
  • The model is presented as a unified system for free-form motion synthesis and editing that also supports more diverse motions and classical prediction tasks.Source: Introduction · p. 2

Related Work

Prior diffusion research established high-quality conditional generation mainly for images, while motion-generation research developed separate approaches for labels, text, prediction, and in-betweening.

3D Human Motion Generation

Earlier motion models addressed prediction, in-betweening, and text-to-motion synthesis separately, whereas FLAME combines high-quality text-to-motion generation with flexible text-based editing without fine-tuning.

  • Motion prediction forecasts subsequent frames, while in-betweening generates natural sequences satisfying starting and target poses.Source: 3D Human Motion Generation · p. 2
  • Existing prediction and in-betweening models lack a single-model solution for multiple tasks and cannot synthesize motion from textual descriptions.Source: 3D Human Motion Generation · p. 2
  • FLAME performs text-based motion editing, including motion prediction and in-betweening, without fine-tuning or modifying the trained generative model.Source: 3D Human Motion Generation · p. 2

Proposed Method: FLAME

FLAME applies conditional diffusion to motion synthesis and editing, using a transformer decoder designed for spatio-temporal, variable-length motion. The model supports text conditioning, denoising-based generation, and frame- or joint-wise editing.

Diffusion-based Model

  • The diffusion process gradually adds noise to motion, reducing complex distribution modeling to a sequence of denoising problems.Source: Diffusion-based Model · p. 3
  • FLAME reverses a diffusion process that transforms Gaussian noise into motion conditioned on language descriptions.Inference starts from M_T ∼ N(0, I) and repeatedly samples pθ(M_t−1|M_t, c) from t = T to 1.Source: Diffusion-based Model · p. 3
  • FLAME trains a model to predict noise and reverse-process variance, using a hybrid loss because the simplified loss alone cannot learn variance.The model predicts the noise component and uses the hybrid loss proposed by Nichol and Dhariwal to train variance.Source: Diffusion-based Model · pp. 3–4

Model Architecture for Motion Data

Inference

  • Classifier-free guidance strengthens text conditioning during denoising, trading a small amount of sample diversity for substantially higher precision.The guided prediction amplifies the conditioning effect with a scalar s > 1.Source: Inference for Motion Synthesis · p. 4

Inference for Motion Editing

  • Motion editing uses a diffuse-then-conditionally-denoise strategy to fill masked frame-wise or joint-wise regions while bridging unedited and edited distributions.Source: Inference for Motion Editing · p. 4

Experiments

FLAME is evaluated across three datasets and multiple motion-generation settings, including text alignment, diversity, editing, and conditional tasks. The experiments also examine architectural components and the sampling-speed trade-off.

Datasets

  • Experiments use HumanML3D, BABEL, and KIT, with SMPL-based representations for HumanML3D and BABEL and a 64-dimensional representation for KIT.HumanML3D contains 44,970 descriptions for 14,616 motions; BABEL uses 63,353 frame-level annotations; KIT contains 3,911 sequences paired with 6,353 descriptions.Source: Datasets · p. 5

Evaluation Metrics

  • Evaluation measures positional and variance errors, motion-text alignment, feature distance, mutual-information divergence, and retrieval accuracy.R-Precision ranks motion and text features by Euclidean distance, while mCLIP uses cosine similarity between separately trained motion and text embeddings.Source: APE and AVE · p. 6, R-Precision · p. 7, Motion CLIP Score (mCLIP) · p. 7

Quantitative Results on Text-to-motion

  • FLAME outperforms compared models on all reported metrics except variance metrics, while additional tests assess joint variance, multimodality, and mCLIP alignment.The paper averages reported metrics over three trials and samples 10 motions per test annotation for the diversity analysis.Source: Quantitative Results on Text-to-motion · p. 7

Ablation Study

  • The ablation study evaluates self-attention, a motion-length token, cross-attention, and language-model freezing as components of the architecture.The motion-length token explicitly supplies the number of frames to generate, and cross-attention uses the transformer decoder architecture.Source: Table 4 · p. 6, Ablation Study · p. 7

Sampling Speed

  • Sampling with 1,000 diffusion steps takes more than 30 seconds per sample, but reduced steps preserve sample quality unless reduced extremely.The speed measurements are recorded on a single NVIDIA Tesla V100 SXM2 32GB machine.Source: Table 5 · p. 6, Ablation Study · p. 7

Application on Other Motion Tasks

Conclusion

The paper presents FLAME as a unified diffusion-based model for text-to-motion generation and text-based motion editing. It emphasizes sample quality and flexible conditional generation, while identifying real-time sampling and cross-domain feature use as future directions.

Conclusion

  • FLAME unifies text-to-motion generation and text-based motion editing in a diffusion-based model.The paper distinguishes the model from previous work by its sample quality and flexibility in conditional generation.Source: Conclusion · p. 7
  • The authors expect FLAME to streamline motion generation and lower the barrier to 3D motion synthesis.These are stated expectations rather than measured outcomes in the supplied conclusion passage.Source: Conclusion · p. 7
  • Future work targets improved sampling for real-time application and reuse of features learned in image-vision domains.Source: Conclusion · p. 7

Appendix A: Data Preprocessing

The appendix describes preprocessing for HumanML3D, BABEL, and KIT, including motion representations, annotation selection, dataset splitting, and evaluation compatibility.

HumanML3D

  • HumanML3D combines AMASS and HumanAct12 annotations, uses SMPL motion data, follows the original train-validation-test split, and processes motions at 20Hz.The representation primarily uses 24 body joints because most dataset motions contain those joint values.Source: HumanML3D · p. 10

BABEL

  • BABEL annotations include sequence-level and frame-level labels, but experiments use only frame-level annotations because they align more precisely with motion.Source: BABEL · p. 10

Motion Length

  • Figure 1 reports the motion-length distribution used in the preprocessing context.Source: Figure 1 · p. 10

KIT

  • KIT uses the same motion pipeline and preprocessing as TEMOS for fair comparison.Source: KIT · p. 10

Appendix B: Derivation

The appendix derives the diffusion training objective for continuous motion, combining simple noise prediction with a variational lower-bound term and adapting the final reverse step to continuous values.

Variational objective

  • The reverse-process objective decomposes into LT, intermediate Lt−1 terms, and L0, with LT constant when forward variances are fixed.The derivation represents each intermediate term as a KL divergence between the forward posterior and learned reverse transition.Source: Appendix B · p. 11

Forward process

  • The forward process uses Gaussian noise ϵ ∼ N(0, I), with αt = 1 − βt and cumulative product ¯αt.These quantities parameterize the noisy motion sequence at timestep t.Source: Appendix B · p. 11

Continuous motion

  • For continuous motion, the same Gaussian-derived reverse-process form is used for L0 rather than an independent discrete decoder.This differs from DDPM’s image setting, where pixel values are discrete.Source: Appendix B · p. 11

Training objective

  • The combined objective is Lhybrid = Lsimple + λLvlb, where Lvlb sums L0 through LT.Improved DDPM learns the variance instead of treating the forward-process variances as fixed constants.Source: Appendix B · p. 11

Appendix C: Evaluation Details

The evaluation appendix defines motion-diversity and position-error measurements, describes feature extraction and training stabilization, and lists the associated implementation settings.

Error and diversity metrics

  • APE and AVE are computed as described in TEMOS, while the same joint-diversity equation also measures diversity across joints.Motion notation distinguishes temporal averages, joint indices, frame indices, and the total sample count.Source: Appendix C · p. 11

Diversity metrics

  • Multimodality measures motion diversity generated for each text annotation by comparing feature vectors from two sample subsets.The experiments generate Sl samples per annotation and set Sl = 10.Source: Appendix C · p. 12

Implementation

  • Exponential moving average is used to stabilize FLAME training, with separate hyperparameter tables for HumanML3D, BABEL, and KIT.The HumanML3D and BABEL settings are listed together, while KIT uses a separate table.Source: Hyperparameters of FLAME · p. 12, Table 1 · p. 12

Feature extraction

  • A separate CLIP-like contrastive model extracts high-level features from motion and text for evaluation.Its training hyperparameters are listed separately from FLAME’s dataset-specific settings.Source: Hyperparameters of Motion CLIP · p. 12

Appendix E: Additional Experiments

Additional experiments compare DDPM and DDIM sampling and examine how classifier-free guidance scale affects text-conditional generation.

Sampling strategies

Guidance scale

  • Classifier-free guidance is varied across sampling runs to evaluate the effect of guidance strength on text-conditional generation.The results are presented in a quantitative table organized by guidance scale.Source: Benchmark on Different Guidance Scale · p. 12

Additional Qualitative Results

Additional qualitative material illustrates FLAME’s text-to-motion synthesis and free-form editing capabilities, alongside quantitative experiment references.

Quantitative references

  • The additional experiments reference quantitative comparisons across sampling steps and guidance scales.These include DDPM/DDIM sampling results and guidance-scale results.Source: Table 4 · p. 13, Table 5 · p. 13

Text-to-motion synthesis

Motion editing

Related papers

Loading 2209.00349v2…