PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Diverse Dance Motions with Transformer

Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, Hao Li

arXiv:2008.08171v1cs.CVcs.GR

TL;DR

The paper tackles the costly production and limited diversity of choreographed dance animation by learning from online dance videos. It introduces a music-conditioned two-stream transformer and reports more diverse, realistic motion than existing methods, while identifying limitations in pose detail and audio representation.

  • Problem

    Dance animation is complex and labor-intensive, while existing motion data and regression-based models provide limited diversity for realistic dance synthesis.

  • Method

    The authors recover 50 hours of synchronized music and dance poses from online videos and train a conditional autoregressive two-stream transformer with discrete pose representations.

  • Results

    The system outperformed existing methods on reported quality, diversity, and musicality evaluations and generated diverse dances with new music.

  • Takeaways & Limitations

    Online videos can train dance-motion models for high-quality, diverse animations suitable for virtual concerts and professional animation reference.

  • Takeaways & Limitations

    Video-derived poses usually omit finger animations, and the audio representation does not cover musical layers such as bass and lyrics.

Abstract

from arXiv · show

With the ongoing pandemic, virtual concerts and live events using digitized performances of musicians are getting traction on massive multiplayer online worlds. However, well choreographed dance movements are extremely complex to animate and would involve an expensive and tedious production process. In addition to the use of complex motion capture systems, it typically requires a collaborative effort between animators, dancers, and choreographers. We introduce a complete system for dance motion synthesis, which can generate complex and highly diverse dance sequences given an input music sequence. As motion capture data is limited for the range of dance motions and styles, we introduce a massive dance motion data set that is created from YouTube videos. We also present a novel two-stream motion transformer generative model, which can generate motion sequences with high flexibility. We also introduce new evaluation metrics for the quality of synthesized dance motions, and demonstrate that our system can outperform state-of-the-art methods. Our system provides high-quality animations suitable for large crowds for virtual concerts and can also be used as reference for professional animation pipelines. Most importantly, we show that vast online videos can be effective in training dance motion models.

1. Introduction

The paper addresses expensive, labor-intensive dance animation by combining diverse online video data with a generative transformer system for music-conditioned motion synthesis. It targets the limited diversity and determinism of prior approaches while introducing evaluation metrics for synthesized dance quality.

  • Prior mocap-based datasets contain limited dance diversity, producing monotonous sequences and restricting models’ generalization to realistic dance motions.The cited example contains only two kinds of dances and less than one hour of data.
  • Regression-based motion models are deterministic, so they struggle to generate highly diverse and complex dance movements.
  • 50 hours of synchronized music and dance pose sequences are recovered from thousands of online videos using 2D detection, tracking, and 3D pose estimation.
  • The authors report that their real-time method generates more diverse and realistic dance motions than acLSTM and ChorRNN, while Transformers train more efficiently than LSTM architectures.
  • The system combines a large-scale online dance-motion dataset, a two-stream motion transformer, and evaluation metrics for diverse music-conditioned dance synthesis.The contributions include scalable data collection, discrete pose representations, long-term dependency modeling, and metrics for synthesized-motion quality.

2. Related Work

Related work frames dance synthesis as motion representation learning, long-term prediction, or audio-to-motion translation. These approaches provide motion control and sequence modeling, but audio-to-dance translation can impose a one-to-one music-to-motion mapping that generalizes poorly beyond training songs.

  • The paper positions dance motion synthesis as an interdisciplinary problem spanning computer graphics and computer vision.
  • 2.1. Motion Synthesis: Motion synthesis research uses autoencoders and related representation-learning methods to recover valid motion manifolds and support motion control.
  • 2.1. Motion Synthesis: Recurrent models address long-term human-motion prediction through residual strategies, sampling, and autoconditioning for extended or potentially unlimited sequences.
  • Audio-to-dance methods commonly formulate synthesis as translation, enforcing a one-to-one mapping from music to dance that does not generalize well beyond training songs.

3. Methods

TSMT is an autoregressive dance-synthesis model that separately encodes motion history and music before fusing them to predict future poses. It uses discrete pose representations and Transformer blocks to model motion distributions and long-term dependencies.

  • 3.3.2. TSMT: TSMT processes pose sequences and music separately, then fuses both streams to predict the next motion.The pose transformer captures dance history, while the audio transformer extracts music context before late fusion.
  • 3.1. Problem Formulation: The model formulates synthesis autoregressively, conditioning each predicted motion on music and previously generated poses.The joint conditional probability is defined over 3D motion frames and audio features.
  • 3.3. TSMT: Pose coordinates are discretized into 300 intervals and embedded as categorical representations before temporal processing.Each pose dimension becomes a 300-dimensional one-hot vector, then a shared learnable embedding reduces memory cost.
  • 3.3.2. TSMT: Audio inputs combine 26-dimensional MFCC features with derivatives and embedded beat signals, followed by convolution and positional embedding.Beat and non-beat positions receive different embedding vectors.
  • 3.3.1. Transformer: Each Transformer uses multi-head self-attention and position-wise feed-forward layers to capture sequential dependencies.Inputs are projected into keys, queries, and values, split across heads, concatenated, and processed with convolutional layers and normalization.
  • 3.3.2. TSMT: Training computes time steps in parallel with causal attention masks, whereas inference samples from the likelihood at each new time step.The mask ensures each step attends only to its past.

3.4. Implementation Details

The implementation trains the model with Adam using mini-batches of 32 and an initial learning rate of 10^-4, decayed after 200 epochs. Global motion is inferred with the Global Path Predictor from Zhou et al.

  • 3.4. Implementation Details: Training uses Adam with mini-batches of 32 and an initial learning rate of 10^-4, decayed by 0.3 after 200 epochs.
  • 3.4. Implementation Details: The system infers global motion using the Global Path Predictor from Zhou et al.

4. Dataset

The YouTube-Dance3D dataset is built from thousands of online dance videos through a pipeline combining trimming, pose detection, tracking, manual cleaning, 3D estimation, jitter removal, and synchronized audio extraction.

  • Dataset collection: 3809 videos from five YouTube street-dance channels were filtered and trimmed into dance segments for dataset construction.The pipeline removed irrelevant content and annotated performance start and end times.
  • Pose processing: The pipeline detects and tracks poses, manually corrects audience tracks and identity swaps, and estimates 3D poses with 17 joints.Manual review addresses tracking noise caused especially by occlusion and group-formation changes.
  • Pose and audio preprocessing: 3D joint trajectories are smoothed with an HP filter, interpolated to 24 fps, and paired with MFCC and beat-interval audio features.The filter removes motion jitters, while standardized audio features are extracted at the same frame rate.
  • Dataset analysis: The dataset uses an inexpensive pipeline that can scale through online videos and offers greater scale, quality, and variety than prior dance datasets.Its urban-dance sources emphasize choreography and variation, while the processing pipeline supports future expansion.

5. Experiments

The experiments evaluate synthesized dance through automatic metrics and qualitative assessment, covering physical plausibility, beat consistency, diversity, and generated-motion quality.

  • Evaluation overview: The evaluation includes metrics for physical plausibility, beat consistency, and generation diversity.
  • Evaluation overview: The experiments report quantitative results for the proposed dance-generation system.
  • Evaluation overview: The experiments also include qualitative results for synthesized dance motions.

5.1. Metrics

The paper evaluates dance synthesis using physical plausibility, beat consistency, diversity, and learned perceptual features, supplemented by newly proposed metrics and visual examples.

  • Metrics: The proposed evaluation combines a humanoid physics simulator for plausibility with a beat-consistency metric for musical alignment.Plausibility is divided into authenticity and coherence, while beat matching allows a flexibility of 2 frames.
  • Physical Plausibility: Authenticity measures joint-limit validity, whereas coherence measures whether joint angular velocities remain within realistic ranges.
  • Diversity: Diversity is measured with FID, inter-sequence diversity, intra-sequence diversity, and same-music diversity.These metrics compare feature distributions or distances across generated sequences, within sequences, and across generations conditioned on identical music.
  • Perceptual features: A two-block Transformer dance-style classifier reaches 61.0% top-1 and 71.5% top-2 classification accuracy for perceptual feature extraction.

5.2. Comparisons

The comparison evaluates the proposed model against acLSTM and ChorRNN in a non-audio setting before comparing audio-enabled model variants.

  • Non-audio comparisons: The non-audio comparison uses acLSTM and ChorRNN as the main baseline techniques.
  • acLSTM: acLSTM uses a three-layer LSTM with alternating ground-truth and model samples to reduce motion freeze and support long sequences.
  • ChorRNN: ChorRNN is a mixture-density model using a three-layer LSTM that predicts a pose distribution at each time step.

5.3. Results

The proposed model generates diverse and realistic dance motions, with Transformer-based synthesis remaining efficient and real-time. Audio conditioning affects diversity, while qualitative results align with the quantitative evaluation.

  • The model generates more diverse and realistic motions than existing techniques while maintaining pose validity comparable to acLSTM.In the non-audio setting, acLSTM is coherent but insufficiently diverse, whereas ChorRNN is diverse but produces invalid poses.
  • Beat-only audio conditioning achieves higher diversity because it imposes fewer constraints on the generated motion.
  • 5.3.2. Qualitative Results: The proposed model produces valid and diverse poses, unlike acLSTM’s static poses and ChorRNN’s difficult-to-perform poses.
  • 5.3.3. Computation Time: The Transformer model trains more efficiently than LSTM-based baselines while generating motion in real time at 24 fps.

5.4. Human Evaluation

Human evaluation uses crowd-sourced pairwise judgments to assess whether automatic metrics align with perceived dance quality. The proposed model is preferred over several baselines, although synthetic motion remains below ground-truth quality.

  • 5.4. Human Evaluation: The human evaluation samples metric-level pairs and asks workers to choose the better motion, with validation pairs filtering noisy annotations.
  • 5.4.2. Overall Quality: Human evaluators prefer the proposed model over acLSTM, ChorRNN, and discrete-representation LSTM baselines.The study compares 200 generated pairs and includes ground-truth comparisons.
  • 5.4.2. Overall Quality: Synthetic motions still trail ground truth because timestep-wise sampling can introduce low-probability poses and sequence noise.The authors suggest adding constraints during pose sampling to reduce this noise.

6. Discussion and Conclusion

The paper presents an audio-driven dance synthesis system trained on online videos, together with a large-scale dataset and evaluation metrics. It reports effective handling of diverse motion but identifies missing finger and facial motion and incomplete musical representations as future-work boundaries.

  • The system handles highly diverse dance movements using online dance videos as training data.
  • The paper introduces new evaluation metrics covering motion quality, diversity, and musicality, and reports that its conditional generative model outperforms existing methods.
  • 6.1. Limitation and Future Work: Video-derived pose data usually omits finger animations, and future work targets finger joints, facial expressions, and motion-capture-based denoising.
  • 6.1. Limitation and Future Work: The audio representation uses MFCC features and beats but does not yet model other musical layers such as bass and lyrics.
Loading 2008.08171v1…