Source-linked AI summary
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Ruilong Li, Shan Yang, David A. Ross, Angjoo Kanazawa
TL;DR
Music-conditioned 3D dance generation requires realistic, continuous motion aligned with music, while existing dance data and sequence models have limitations. The paper introduces AIST++ and FACT, combining a large multimodal dataset with full-attention cross-modal future-motion prediction. The reported evaluations show that these design choices support high-quality, long, music-correlated dance generation.
Problem
Music-conditioned 3D dance generation must produce continuous, kinematically complex motion aligned with music, but dance motion-capture datasets are limited in sequence, dancer, and music diversity.
Method
The paper constructs AIST++ and trains FACT to fuse music with 2-second seed motion using full-attention cross-modal prediction of N future motions.
Results
Extensive evaluations report that FACT's design choices are critical for generating high-quality, multimodal, long motion sequences that correlate with input music.
Takeaways & Limitations
AIST++ provides a large multimodal benchmark, while FACT offers a supported approach for long, realistic 3D dance generation conditioned on music.
Abstract
from arXiv · showhide
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses -- the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict $N$ future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively.
1. Introduction
The paper addresses music-conditioned 3D dance generation, introducing the AIST++ dataset and FACT to produce realistic, music-correlated long motion sequences.
- Dance generation requires continuous, kinematically complex motion that captures the nonlinear relationship between movement and accompanying music.
- The paper introduces AIST++, a large-scale multimodal dataset, and FACT, a transformer for generating realistic 3D dance from music and seed motion.Given music and a 2-second seed, FACT generates long future motion sequences.
- FACT encodes audio and seed motion separately, fuses them with a cross-modal transformer, predicts N future motions, and generates continuously autoregressively.Its design uses full attention and future-N supervision.
- Existing motion-capture datasets are limited in dance sequences, dancer diversity, and music diversity because collection requires heavily instrumented environments.
- The contributions include FACT for long, realistic, music-correlated motion and evaluations using motion-quality and music-motion-correlation metrics.
- AIST++ contains 5.2 hours of 3D dance motion, while the paper reports extensive evaluations validating design choices for high-quality multimodal long-sequence generation.
2. Related Work
Prior work spans statistical, graph-based, and neural 3D motion synthesis, audio-conditioned 2D pose generation, and cross-modal sequence modeling, but 3D dance remains constrained by realism and data limitations.
- 3D human motion synthesis has used statistical models, motion graphs, CNNs, GANs, RBMs, RNNs, and transformers.
- Autoregressive RNNs and vanilla transformers can generate unbounded motion in theory but may freeze through regression to the mean or drift toward unnatural motions.
- Audio-to-motion research often generates 2D pose from audio, but 2D representations limit expressiveness and downstream applications.
- Compared with a related transformer approach, FACT uses full-attention cross-modal modeling to preserve music-motion correlation and generate realistic long 3D motion with global translation.
- 3D dance motion-capture datasets remain limited because they require heavy instrumentation and expert dancers, leading prior work toward small-scale or private datasets.
- The paper connects 3D dance generation to broader cross-modal sequence-to-sequence tasks including text-to-speech, speech-to-gesture, and image or video captioning.
3. AIST++ Dataset
AIST++ reconstructs 3D human motion from multiview AIST videos and pairs it with music, providing broad annotations and coverage across subjects, genres, and synchronized images.
- AIST++ is reconstructed from multiview AIST videos by recovering camera calibration parameters and 3D human motion in SMPL parameters.
- The dataset provides camera parameters, 2D and 3D COCO-format joints, and SMPL pose parameters with global scaling and translation.
- The dataset pairs 3D dance motion with music and includes synchronized multiview images, unlike prior 3D dance datasets described in the comparison.
- AIST++ contains 1408 sequences, 30 subjects, and 10 dance genres, with motions distributed across basic and advanced choreographies.
- Its genres span Old School and New School styles, with varied tempos and an 85% basic versus 15% advanced choreography distribution.
- Train and validation subsets avoid sharing choreography or music, addressing overlap risks created by instructional recordings of repeated choreography.
4. Music Conditioned 3D Dance Generation
The task generates future 3D dance motion from a short seed motion and longer music sequence. FACT combines modality-specific encoding, deep cross-modal fusion, full attention, and multi-future supervision in an autoregressive generation framework.
- Given a 2-second seed motion and longer music sequence, the model generates future motion from time T + 1 through T′.
- A transformer attention layer maps input sequences to context vectors using queries, keys, values, and an optional mask, whose design determines causal or full attention.
- FACT encodes seed motion and audio separately, concatenates their embeddings, and uses a cross-modal transformer to generate N future motion sequences.
- FACT uses full attention and predicts N unseen future motions beyond the context window rather than only the immediate next motion.
- A deep 12-layer cross-modal transformer is used to combine audio and motion embeddings, and the authors report that it is essential for attention to input music.
5. Experiments
Experiments validate AIST++ reconstruction quality and evaluate FACT on realism, diversity, music-motion correlation, ablations, and user preference. FACT outperforms recent methods across the reported criteria, while ablations identify future-N supervision and early cross-modal fusion as important design choices.
- Quantitative Evaluation: FACT outperforms three recent state-of-the-art methods on motion quality, generation diversity, and motion-music correlation.The evaluation uses FID-based motion quality, feature-space diversity, and BeatAlign correlation measures.
- Quantitative Evaluation: FACT generates varied dance motions from different music while keeping the same 2-second hip-hop seed motion, including style adaptation to Ballet Jazz.The visualization shows Break, Ballet Jazz, Krump, and Middle Hip-hop conditioning genres.
- Quantitative Evaluation: BeatAlign measures average distance from each kinematic beat to its nearest music beat, and FACT is better correlated with input music than the baselines.Kinematic beats are local minima of kinetic velocity; the authors note substantial remaining improvement relative to real data.
- Ablation Study: Early cross-modal fusion improves alignment with conditioning music and makes the cross-modal transformer attend more to music than late fusion.The comparison includes no fusion, late fusion with one cross-modal layer, and early fusion with twelve cross-modal layers.
- User Study: 81%, 71%, and 77% of generated dances were preferred over Li et al., Dancenet, and DanceRevolution, respectively, in user comparisons.The random real-dance counterpart was preferred over FACT 75% of the time, although quantitative metrics favored FACT for music alignment.
6. Conclusion and Discussion
The paper presents FACT for music-conditioned 3D dance generation and AIST++, a multi-view, multi-genre dataset. It identifies physical interaction modeling and stochastic generation as important directions for future work.
- FACT learns audio-motion correspondence and generates non-freezing, high-quality 3D dance sequences conditioned on music.
- AIST++ is a large multi-view, multi-genre, cross-modal 3D dance dataset supporting conditional motion generation and human understanding research.
- The kinematic approach does not model physical interactions between the dancer and floor, so global translation can produce foot sliding and floating artifacts.
- The deterministic model does not generate multiple realistic dances for the same music, leaving multimodal generation as an open direction.
A. AIST++ Dataset Details
AIST++ reconstructs 3D dance motion from multi-view AIST videos and pairs it with music across diverse genres. Reconstruction quality is validated through reprojection consistency, while the dataset statistics establish its scale and coverage.
- 3D reconstruction: The reconstruction optimizes body pose, global scale, and transformation against triangulated joint locations while fixing body shape to the average shape.
- Dataset statistics: AIST++ contains 5.2 hours, 1.1M frames, and 1408 sequences of 3D dance motion accompanied by music.The dataset covers 10 dance genres and 60 music pieces.
- Dataset coverage: The dataset visualizes 10 types of 3D human dance motion and is built from a single-person dance subset of the AIST database.
- Validation: Over 86% of video sequences have less than 10 pixels of average MPJPE-2D error at 1920x1080 resolution.
- Validation: 98.4% averaged PCKh@0.5 across all joints indicates high consistency between re-projected and detected 2D keypoints.
B.1. Comparison User Study
The comparison user study presents generated and baseline dance motions side by side and asks participants which dancer follows the music better. Participants span a range of dance experience, and video-side order is randomized.
- Study design: Participants compare the proposed and baseline motions side by side using stickman visualizations and judge which person is dancing more to the music.
- Study design: The left-right order is randomly shuffled for each video to reduce positional effects in the comparison.
- Participant demography: Participants range from professional dancers to people who rarely dance, with a majority reporting at least one year of dance experience.