PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Motion Generation: A Survey

Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, Yizhou Wang

arXiv:2307.10894v3cs.CV

TL;DR

Human motion generation seeks natural, realistic motion under conditional signals, but its complex relationship with text, audio, and scenes remains challenging. This survey synthesizes methods, datasets, metrics, and open problems across these settings. It highlights incomplete evaluation, data constraints, and unresolved semantic and multimodal challenges.

  • Problem

    Human motion generation must model intricate motion and its implicit relationship with conditional signals such as text, audio, and scenes.

  • Method

    The survey reviews fundamentals, conditional generation methods, datasets, evaluation metrics, and future directions for text-, audio-, and scene-conditioned motion.

  • Results

    The survey identifies persistent challenges in evaluation, physical plausibility, audio-motion consistency, data quality, and semantic alignment.

  • Takeaways & Limitations

    The field has broad application potential, but progress depends on addressing limitations in data, evaluation, multimodal conditioning, and semantic relationships.

  • Takeaways & Limitations

    The survey is limited by the difficulty and expense of collecting high-quality motion data, along with variation in representations and conditional signals.

Abstract

from arXiv · show

Human motion generation aims to generate natural human pose sequences and shows immense potential for real-world applications. Substantial progress has been made recently in motion data collection technologies and generation methods, laying the foundation for increasing interest in human motion generation. Most research within this field focuses on generating human motions based on conditional signals, such as text, audio, and scene contexts. While significant advancements have been made in recent years, the task continues to pose challenges due to the intricate nature of human motion and its implicit relationship with conditional signals. In this survey, we present a comprehensive literature review of human motion generation, which, to the best of our knowledge, is the first of its kind in this field. We begin by introducing the background of human motion and generative models, followed by an examination of representative methods for three mainstream sub-tasks: text-conditioned, audio-conditioned, and scene-conditioned human motion generation. Additionally, we provide an overview of common datasets and evaluation metrics. Lastly, we discuss open problems and outline potential future research directions. We hope that this survey could provide the community with a comprehensive glimpse of this rapidly evolving field and inspire novel ideas that address the outstanding challenges.

1 INTRODUCTION

Human motion generation seeks natural, realistic, and diverse motion for applications across multiple fields, but its articulated, nonlinear, and perceptually sensitive nature makes generation challenging. This survey organizes recent progress around generative methods, conditional signals, datasets, and evaluation.

  • Human motion generation targets natural, realistic, and diverse motion for film, games, AR/VR, human-robot interaction, and digital humans.
  • Deep learning has driven rapid adoption of autoregressive models, VAEs, normalizing flows, GANs, and diffusion models for generation.
  • Recent video-based motion capture and large-scale datasets have increased interest in data-driven human motion generation.
  • Human motion remains difficult to generate because it is nonlinear, articulated, physically constrained, and judged by sensitivity to unnatural kinematics.
  • The survey reviews motion fundamentals, conditional generation from text, audio, and scenes, datasets, evaluation metrics, and future directions.

2 SCOPE

The survey defines its scope around human motion generation conditioned on signals, especially text, audio, and scenes, while covering multiple motion representations.

  • The survey primarily covers human motion generation conditioned on text, audio, and scene signals.
  • Covered motion targets include 2D/3D body-keypoint sequences, joint rotations, and parametric human body models.

3 PRELIMINARIES

This section introduces human motion representations, data collection techniques, and generative models used to generate new motion sequences. It covers keypoint and rotation-based representations, four collection approaches, and major generative-model families.

  • 3.1.1 Motion Data Representation: Human motion data is represented as temporal pose sequences using keypoints or joint rotations, with conversions possible through forward and inverse kinematics.Keypoint representations include 2D/3D body points, while rotation-based models describe joint rotations.
  • 3.1.1 Motion Data Representation: SMPL represents body pose and shape through joint rotations and ten shape parameters, producing a 3D human mesh.The model uses K = 24 joints and shape parameters β ∈R10, with Figure 3 showing the rotation-based representation.
  • 3.1.2 Motion Data Collection: Human motion data is collected through marker-based capture, markerless capture, pseudo-labeling, and manual annotation.Marker-based systems use optical markers or IMUs; optical markers are more accurate, whereas IMUs are more portable and support outdoor use.
  • 3.2 Motion Generation Methods: Human motion generation methods include regression models and generative models that learn motion distributions or conditional joint distributions.The survey reviews GANs, VAEs, normalizing flows, diffusion models, and motion graphs as representative generative approaches.
  • 3.2 Motion Generation Methods: GANs, VAEs, normalizing flows, diffusion models, and motion graphs offer different trade-offs in stability, likelihood estimation, sample quality, and computational cost.GANs can suffer instability and mode collapse; VAEs may produce less sharp samples; flows provide exact likelihoods; diffusion models produce high-quality samples but require long reverse chains; motion graphs generate sequences consistent with dataset characteristics.

4 TEXT-CONDITIONED MOTION GENERATION

Text-conditioned motion generation uses action labels or natural-language descriptions to produce human motion, with text-to-motion offering broader expressive variety but requiring detailed language-motion alignment.

  • Action to Motion: Action-to-motion methods commonly use GANs, graph convolutions, VAEs, or transformer-based conditional VAEs to generate labeled action sequences.Examples include dual-discriminator GANs, disentangled latent variations, per-frame VAEs, and non-autoregressive sequence generation.
  • Action to Motion: Existing action-to-motion methods primarily generate single-action sequences, while multi-action motion remains challenging and may require post-processing.MultiAct addresses longer multi-action sequences by recurrently using past motion and canonicalizing each step to ground geometry.
  • Text to Motion: Text-to-motion generates motion from natural-language descriptions, offering more varied outputs than action-to-motion’s predefined labels.The central challenge is converting linguistic intricacies into corresponding movements.
  • Text to Motion: Text-to-motion research progresses from joint text-motion embeddings and GANs toward zero-shot alignment with pretrained vision-language representations.MotionCLIP aligns motion latents with CLIP text and image spaces, while AvatarCLIP uses reference-based synthesis.
  • Text to Motion: Recent text-to-motion methods use VQ-VAE tokenization, autoregressive transformers, and diffusion models to generate variable-length or diverse motion.Examples include TM2T, T2M-GPT, FLAME, MDM, MLD, and MoFusion.

5 AUDIO-CONDITIONED MOTION GENERATION

Audio-conditioned motion generation maps music or speech to motion while balancing semantic and rhythmic alignment, with music-to-dance and speech-to-gesture as its main tasks.

  • Overview: Audio-conditioned generation must align motion with audio at both high-level semantics and low-level rhythms, despite audio lacking explicit motion descriptions.The survey covers music-to-dance and speech-to-gesture, using waveforms, spectrograms, or MFCCs and sometimes style or transcript conditions.
  • Music to Dance: Music-to-dance methods span supervised sequence models, GANs, normalizing flows, motion graphs, VQ-VAE transformers, and diffusion models.Representative systems include FACT, MNET, Transflower, Bailando, and EDGE.
  • Music to Dance: Long-song dance generation remains difficult because long sequences can accumulate errors and produce freezing motions.Curriculum learning and VQ-VAE-based approaches are introduced to address this long-horizon problem.
  • Speech to Gesture: Speech-to-gesture systems generate primarily upper-body gestures from speech audio or transcripts, using adversarial, retrieval-refinement, and diffusion-based approaches.DiffGesture uses cross-modal transformers and classifier-free guidance to balance diversity and gesture quality.
  • Speech to Gesture: Speaker-specific variability motivates multimodal systems that incorporate text, audio, identity, affect, style, facial expressions, and body-part structure.These methods aim to support multiple speakers and richer expressive gesture generation.

6 SCENE-CONDITIONED MOTION GENERATION

Scene-conditioned motion generation seeks motions consistent with environmental layout and physical constraints, using varied scene representations and predominantly multi-stage pipelines.

  • Overview: Scene-to-motion generation produces human movement that is reasonable for the scene context and supports interaction with surrounding environments.The survey focuses on data-driven approaches organized by scene representation and generation pipeline.
  • Scene Representation: Methods represent scenes with RGB images, point clouds, meshes, 3D objects, goal positions, and sometimes language or action labels.Many systems combine multiple scene representations with past motion or a starting pose.
  • Generation Pipeline: A common pipeline predicts a goal or interaction anchor, plans a path or trajectory, and infills motion along that path.This decomposition is used across image-, object-, and geometry-conditioned methods.
  • Generation Pipeline: Alternative pipelines use given sub-goals to generate static poses, synthesize connecting motions, and stitch clips into long-term movement.CIRCLE represents another paradigm that initializes motion before subsequent generation steps.

7 DATASETS

Human motion generation datasets pair motion with text, audio, or scenes and vary substantially in capture technique, scale, modality, and environmental scope.

  • Dataset Organization: The survey categorizes datasets by conditional signal into text-motion, audio-motion, and scene-motion collections, while also listing unconditioned motion datasets.Table 2 summarizes their key properties.
  • Text-Motion Datasets: Text-motion datasets include paired language annotations and motion recordings, while action datasets provide labeled categories across modalities such as RGB, depth, and skeleton sequences.KIT-Motion Language pairs motion with language, and UESTC contains 40 action categories across capture modalities.
  • Audio-Motion Datasets: Audio-motion datasets are collected in controlled settings through capture or annotation, or in the wild from online videos processed with pose estimators.They cover dance and conversational gesture, including multimodal datasets with transcripts, styles, emotions, and facial expressions.
  • Scene-Motion Datasets: Scene-motion datasets capture humans together with objects and environments, whereas scene-only datasets such as Matterport3D, Replica, and ScanNet are often used for testing.WBHM includes interacting objects, and CIRCLE contains reaching motions across nine scenes.

8 EVALUATION METRICS

Evaluating synthesized human motion is non-trivial because conditional inputs can admit multiple valid motions and human judgments are subjective. The survey therefore organizes commonly used metrics and discusses their strengths and limitations.

  • Motion evaluation is difficult because the task has one-to-many mappings, subjective human judgments, and high-level conditional cues.
  • The survey summarizes evaluation metrics from different perspectives and discusses their strengths and limitations.

8.1 Fidelity

Fidelity evaluation covers generated-motion quality through ground-truth comparison, distributional similarity, naturalness, and physical plausibility. These measures remain incomplete because one reference may omit valid alternatives, learned quality scores can diverge from human judgments, and physical-plausibility metrics lack standardization.

  • Ground-truth distance metrics directly compare synthesized and reference motions, but one reference cannot cover all feasible conditional outputs.
  • Motion quality metrics compare generated and real motion manifolds in motion space or feature space.
  • Distributional measures include AVE, Hellinger distance, FID, IS, chi-square distance, MMD, MMS, CCA, and realistic score.
  • Naturalness metrics depend on dataset distributions and pretrained feature extractors, and EDGE reports that FID can be inconsistent with human evaluations.
  • Physical plausibility metrics assess foot sliding, foot-ground contact, and related artifacts, but no standardized metric currently exists.

8.2 Diversity

Diversity evaluation examines both variation within individual motion sequences and diversity across different generated sequences. Existing metrics assess intra-motion variation, inter-motion coverage, and multi-modality using distances and prediction-oriented errors.

  • 8.2 Diversity: Intra-motion diversity metrics evaluate variation within a sequence and help detect freezing in long-sequence generation.Methods include average pairwise distances across non-overlapping clips, temporal position and velocity variations, and freezing rate.
  • 8.2 Diversity: Inter-motion diversity metrics evaluate the generated motion manifold through coverage and multi-modality.Coverage commonly samples multiple conditional signals and measures the diversity of their generated motions, including normalized APD relative to ground truth.
  • 8.2 Diversity: Diversity evaluation also uses ADE, FDE, MMADE, and MMFDE, while some methods assess diversity around interaction anchors or planned paths.These metrics reflect the multi-modal nature of motion prediction and can target specific interaction structures.

8.3 Condition Consistency

Condition-consistency evaluation measures whether generated motions match text, audio, and scene conditions. The survey also highlights objectivity, cultural and emotional nuance, and generality as limitations of existing metrics.

  • 8.3 Condition Consistency: Condition-consistency metrics evaluate agreement between generated motions and their corresponding conditional signals, with measures varying by task.The survey organizes these metrics according to text-, audio-, and scene-conditioned generation.
  • Text Consistency: Text-motion consistency is commonly assessed with recognition accuracy from a pretrained action-recognition model.The metric tests whether generated motions are identified as their corresponding action classes.
  • Evaluation Limitations: Pretrained-model-based metrics depend strongly on model performance and training-data quality and distribution, limiting objective evaluation.These dependencies constrain how objectively such metrics can assess generated motion.
  • Audio Consistency: Audio-motion consistency metrics measure beat alignment and semantic correspondence, including beat coverage, hit rate, beat alignment scores, and semantic-relevant gesture measures.Some approaches refine beat matching or use semantic weighting to better reflect perceived correspondence.
  • Audio Consistency: Existing audio-motion metrics often overlook stylistic and emotional compatibility, such as whether dance style or gesture affect matches the music or speech.The survey notes that beat alignment alone may not capture these subtler cultural and emotional relationships.
  • Scene Consistency: Scene-motion consistency is evaluated through non-collision, human-scene contact, goal-reaching, execution time, success rate, and body-to-goal distance.These metrics address safety, physical plausibility, contact, and interaction with target objects, but are often tailored to specific research contexts and lack broad generality.

8.4 User Study

User studies are essential for evaluating generated human motion because they reveal quality dimensions that objective metrics may miss. They are particularly important for detecting subtle artifacts and culturally nuanced qualities.

  • User studies can uncover motion-quality aspects that objective metrics alone may not capture.
  • Human evaluators are sensitive to subtle biological-motion artifacts such as jittering and foot skating.
  • Subjective evaluation can assess nuanced cultural qualities, including aesthetics and emotional impact.

9 CONCLUSION AND FUTURE WORK

The survey synthesizes human motion generation research by organizing methods around conditional signals and reviewing datasets, metrics, and fundamentals. It concludes that data, semantic understanding, and broader research directions remain important challenges.

  • The survey classifies existing studies by conditional signals and reviews datasets, evaluation metrics, and foundational generation methods.
  • High-quality human motion data are difficult and expensive to collect, creating a trade-off between data quantity and quality.
  • Future work should improve semantic alignment between motion and signals such as text, music, speech, and environmental affordances.
Loading 2307.10894v3…