Source-linked AI summary
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, Ziwei Liu
TL;DR
Music-conditioned 3D dance generation must satisfy choreography-specific spatial constraints and maintain temporal coherence across music genres. Bailando addresses these challenges with an unsupervised quantized choreographic memory and an actor-critic GPT, achieving state-of-the-art qualitative and quantitative performance on AIST++ while discovering human-interpretable dancing-style poses.
Problem
Music-conditioned 3D dance generation requires spatially qualified dancing poses and temporal coherence with diverse music beats, but existing approaches face these challenges.
Method
Bailando encodes dancing-style poses into a compositional quantized choreographic memory and uses an actor-critic GPT with cross-conditional attention and beat-align rewards to generate future pose codes.
Results
Bailando achieves state-of-the-art performance qualitatively and quantitatively on the AIST++ benchmark, outperforming existing methods across automatic metrics and visualization judgments.
Takeaways & Limitations
The learned choreographic memory discovers human-interpretable dancing-style poses in an unsupervised manner while supporting fluent music-coherent dance generation.
Takeaways & Limitations
Existing fixed-length, fixed-speed dance units cannot be reused for different kinds of music beats.
Abstract
from arXiv · showhide
Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated dance sequence also needs to maintain temporal coherency with different music genres. To tackle these challenges, we propose a novel music-to-dance framework, Bailando, with two powerful components: 1) a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequence to a quantized codebook, 2) an actor-critic Generative Pre-trained Transformer (GPT) that composes these units to a fluent dance coherent to the music. With the learned choreographic memory, dance generation is realized on the quantized units that meet high choreography standards, such that the generated dancing sequences are confined within the spatial constraints. To achieve synchronized alignment between diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a newly-designed beat-align reward function. Extensive experiments on the standard benchmark demonstrate that our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively. Notably, the learned choreographic memory is shown to discover human-interpretable dancing-style poses in an unsupervised manner.
1. Introduction
Bailando addresses the spatial and temporal challenges of music-conditioned 3D dance generation with a choreographic memory and an actor-critic motion GPT. The framework quantizes dancing-style pose units and composes them into music-synchronized sequences.
- 1. Introduction: Music-conditioned 3D dance generation must satisfy strict dancing-style spatial constraints while remaining temporally coherent with varied music rhythms.Dancing poses must be visually expressive and choreography-compliant, while movements remain fluent and aligned with beats across genres.
- 1. Introduction: Existing methods that directly map music to continuous 3D joint sequences can be unstable and produce nonstandard poses such as freezing or meaningless swaying.These approaches lack explicit constraints restricting synthesized motion to the dancing subspace.
- 1. Introduction: Bailando learns an unsupervised choreographic memory by encoding dancing-style 3D poses into a finite codebook of reusable units.Separate VQ-VAEs for upper and lower body halves create compositional pose-code pairs and enlarge the representable pose range.
- 1. Introduction: An actor-critic motion GPT predicts future upper-lower pose code pairs from music features and starting pose codes.Cross-conditional causal attention preserves coherence between the generated upper and lower body sequences.
- 1. Introduction: The predicted pose codes are embedded through choreographic memory and decoded by CNN-based decoders into a 3D dance sequence.This inference pipeline autoregressively composes quantized pose units into motion.
- 1. Introduction: Bailando’s actor-critic GPT uses reinforcement learning to align diverse motion tempos with music beats, and experiments report improvements over existing methods.The contributions include beat alignment and favorable results on automatic metrics and visualization judgments.
2. Related Work
Related work includes graph-based motion synthesis and two-stage generative approaches that first encode data and then generate the resulting representations. Bailando follows the latter strategy for meaningful dancing units.
- 2. Related Work: Graph-based music-to-dance methods splice motion clips while imposing constraints such as music similarity, beat-wise connectivity, and rhythm signatures.These methods synthesize motion by treating existing clips as nodes and linking them according to rules.
- 2. Related Work: Two-stage generation methods first encode data and then learn a probabilistic model to generate the encoded representations.Prior applications include generating songs from audio and lyrics or generating images from quantized patches.
- 2. Related Work: Bailando applies the two-stage paradigm by encoding and quantizing meaningful dancing units before generating them with a motion model.Its pose VQ-VAE is designed to summarize dancing units into choreographic memory and reconstruct pose sequences.
3. Our Approach
Bailando represents spatially standard dance movements with compositional quantized pose codes, then uses a motion GPT and actor-critic rewards to generate coherent, music-synchronized 3D dance.
- Choreographic Memory: Bailando encodes spatially standard dance movements into a finite choreographic-memory codebook of reusable dancing positions.The codebook is learned unsupervised with pose VQ-VAEs, whose entries decode into interpretable dance positions.
- Choreographic Memory: Separate upper- and lower-body VQ-VAEs provide compositional pose-code pairs that enlarge the range of covered dance positions.The two codebooks can combine upper- and lower-body codes while preserving separate body representations.
- Pose VQ-VAE Training: Velocity-and-acceleration reconstruction terms play essential roles in preventing jitters in generated dance.The reconstruction loss regresses joint positions together with first- and second-order temporal derivatives.
- Motion GPT: The motion GPT predicts future upper- and lower-body pose codes from music features and source pose codes, then decoders reconstruct the 3D dance sequence.The GPT embeds body codes and music features, predicts code probabilities, and the VQ-VAE decoders reconstruct motion from selected codes.
- Motion GPT: Cross-conditional causal attention exchanges information among music and both body components while preventing future information from flowing to the past.Its mask uses a repeated block structure whose elements are lower-triangular temporal matrices.
- Actor-Critic Learning: Actor-critic learning addresses nondifferentiable sequence-level constraints by using beat-align and compositional consistency rewards during GPT training.The beat-align reward penalizes missing dance beats during music-beat intervals, while the consistency reward penalizes opposite upper- and lower-body directions.
4. Experiments
Experiments on AIST++ evaluate Bailando’s dance quality, diversity, music-motion alignment, user preference, and component effectiveness. Bailando outperforms existing methods across reported evaluations, while ablations support the roles of quantization, body decomposition, velocity modeling, and velocity-acceleration losses.
- Experimental Setup: Evaluation measures dance quality with kinetic and geometric FID, motion diversity with feature distance, and music-motion alignment with Beat Align Score.The metrics target generated-dance quality, diversity, and temporal correspondence to music beats.
- Comparison to Existing Methods: 20% and 56% improvements over FACT are reported for FIDk and FIDg, respectively, with Bailando achieving a better FIDg than ground truth.The reported FID improvements are 7.19 for FIDk and 12.49 for FIDg; FIDg is 9.62 versus 10.60 for ground truth.
- Comparison to Existing Methods: Bailando achieves high choreographic diversity, improves music-motion correlation, and wins at least 84.5% of user comparisons against competing methods.In comparisons with ground truth, 40% of generated dances are preferred on average; participants cite rhythm stability and diversity.
- Ablation Studies: Removing upper-lower body separation prevents effective reconstruction of test-set dancing poses, supporting compositional representations in the choreographic memory.The ablation examines pose VQ-VAE components on reconstructed AIST++ ground-truth results.
- Ablation Studies: Removing global velocity prediction sharply worsens motion-quality scores, while removing velocity-and-acceleration loss increases FIDk and produces strong motion jitters.The reported drops without global velocity are 42.72 (151%) and 5.89 (47%); FIDk worsens by 2.68 without velocity-and-acceleration loss.
- Interpretability of Choreographic Memory: Choreographic memory codes decode into unique dance positions and arbitrary code combinations can produce fluent movements through smooth transitions.Single codes decode to static poses, while different code sequences decode transitions between poses.
5. Discussion and Conclusion
Bailando addresses spatial and temporal challenges in 3D dance generation through choreographic memory and actor-critic GPT. On AIST++ and user studies, it achieves state-of-the-art qualitative and quantitative performance.
- Bailando combines choreographic memory for encoding and quantizing dancing-style poses with actor-critic GPT for music-coherent temporal generation.The framework targets spatial constraints and temporal coherency with music.
- Bailando translates and aligns varied motion tempos with music beats while preserving dancing-style pose constraints.
- Bailando achieves state-of-the-art performance qualitatively and quantitatively on the AIST++ benchmark and in user studies.