Source-linked AI summary
EDGE: Editable Dance Generation From Music
Jonathan Tseng, Rodrigo Castellon, C. Karen Liu
TL;DR
Creating realistic, music-faithful dances is difficult, while prior methods struggle with user constraints and imperfect evaluation metrics. EDGE uses a transformer-based diffusion model with Jukebox features and dance-specific editing, and reports state-of-the-art performance with strong user-study results, while its physical-plausibility metric has defined scope limitations.
Problem
Creating new dances is difficult and time-consuming, while prior systems have limited success satisfying user constraints and existing quality metrics can be flawed.
Method
EDGE combines a transformer-based diffusion model with Jukebox music representations, editable conditioning, Contact Consistency Loss, and Physical Foot Contact Score.
Results
EDGE achieves state-of-the-art results on AIST++, generalizes to in-the-wild music, and improves dance-quality evaluations in a large user study.
Takeaways & Limitations
EDGE enables realistic long-form dance generation with temporal and joint-wise constraints that users can specify through editing.
Takeaways & Limitations
PFC assumes feet are the only joints with static contact and therefore requires modification for motions such as gymnastics with integral hand contacts.
Abstract
from arXiv · showhide
Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods. Qualitative samples from our model can be found at our website.
1. Introduction
EDGE addresses the difficulty of creating music-faithful dances by combining diffusion-based generation, strong music representations, and dance-specific editing. It also introduces new evaluation and physical-plausibility methods to address shortcomings in prior work.
- Dance creation is difficult because movements are expressive and freeform while remaining precisely structured by music, making hand animation and motion capture costly or impractical.
- Previous methods had limited success satisfying user constraints, while existing quantitative metrics for generated-dance quality were shown to be flawed.
- EDGE combines a transformer-based diffusion model with Jukebox music features to generate realistic, physically plausible dances from input music.
- The diffusion-based approach supports powerful editing capabilities, including joint-wise conditioning, in-betweening, and arbitrarily long sequences.
- EDGE analyzes prior metrics against human evaluations and introduces Physical Foot Contact Score plus Contact Consistency Loss for physical plausibility and foot-sliding reduction.
- Jukebox replaces previous hand-crafted audio-feature strategies with representations from a pretrained generative music model.
2. Related Work
Dance-generation research has progressed from retrieval-based methods to neural synthesis and conditional generative models, but physical realism, complexity, data scarcity, and music-conditioned scaling remain challenges.
- Motion matching retrieves and interpolates database sequences, producing physically plausible outputs but mainly serving simple domains such as locomotion.
- Deep neural networks generate diverse motions but may miss physical laws or require difficult-to-train reinforcement learning solutions.
- Dance generation must produce choreography stylistically faithful to music, a challenge addressed by retrieval methods and later synthesis from large datasets.
- Later dance systems use adversarial learning, recurrent networks, and transformers, but many are complex and rely on multiple conditioning layers and subnetworks.
- Diffusion models learn data distributions by reversing scheduled noising and support conditional generation, with strong results in image, video, and text-conditioned motion generation.
- Compared with text-to-motion, music-conditioned dance generation faces harder computational scaling and substantially lower data availability.
3. Method
EDGE uses a transformer-based diffusion framework to generate music-conditioned dance sequences, with Jukebox features providing cross-attention context. Its auxiliary losses, masked denoising, and temporally constrained sampling support physical realism, editing, and arbitrary-length choreography.
- Pose Representation: EDGE represents each dance frame with 151 pose features: 147 SMPL rotation and translation values plus four binary foot-contact labels.Sequences contain N frames and are generated under arbitrary music conditioning c.
- Music Conditioning: Jukebox music representations condition a transformer decoder through cross-attention, replacing hand-crafted audio features with learned music features.The Jukebox model is frozen, and its features are extracted as conditioning input for the diffusion model.
- Diffusion Framework: The diffusion model reverses a scheduled noising process by estimating the clean dance sequence at each timestep and iteratively denoising until t = 0.Sampling starts from noise zT ∼ N(0, I), predicts a denoised sequence, and noises it back for the next timestep.
- Physical Realism: Auxiliary losses supplement reconstruction by matching joint positions, velocities, and foot velocities to improve physical realism.Contact Consistency Loss additionally encourages foot-contact prediction and consistency with the model’s own predictions, which significantly improves generated-motion realism.
- Editing: Masked denoising enables arbitrary temporal and joint-wise constraints, including motion in-betweening without special training procedures.Known regions are replaced with forward-diffused constraint samples at every denoising timestep; unconstrained regions are generated to connect them.
- Long-form Sampling: EDGE generates arbitrarily long choreographies by enforcing temporal consistency across multiple shorter sequences before concatenating them.This approach avoids directly scaling the model’s sequence length and associated Jukebox memory demands; the example joins 5-second clips through overlapping 2.5-second constraints.
4. Experiments
EDGE is evaluated against prior dance-generation methods using human judgments and quantitative measures of physical plausibility, beat alignment, diversity, and in-the-wild performance. The experiments also examine editing capabilities and reveal that several existing FID-based metrics do not track human-perceived quality.
- Experimental setup: EDGE is compared with FACT and Bailando on human evaluations and automated measures of dance quality, physical plausibility, beat alignment, and diversity.The evaluation uses AIST++ dance-and-music data and includes 147 human raters assessing 11,610 clip pairs.
- Human evaluation: Human raters overwhelmingly preferred EDGE over previous methods and even over real dances in overall quality evaluations.The study asked raters which dance looked and felt better overall.
- Physical plausibility metric: The Physical Foot Contact score uses physically inspired acceleration conditions without explicit physical modeling, and Contact Consistency Loss improves both this metric and human-rated plausibility.The loss ablation reports EDGE with CCL winning 61.8% of matchups against the version without it.
- Physical plausibility: EDGE achieves greater physical plausibility than previous methods and approaches ground-truth motion-capture plausibility, while fixed bone lengths avoid variations observed in Bailando.Bailando’s generated sequences showed average bone-length changes of up to ±20%.
- Diversity: At guidance weight w = 1, EDGE matches ground-truth diversity closely; at w = 2, fidelity and Elo increase while diversity decreases.The observed fidelity–diversity trade-off is consistent with prior guidance-weight studies.
- In-the-wild music: Jukebox features bring human-rated quality on in-the-wild music almost to in-distribution levels, and EDGE continues to outperform baselines in human evaluations.The comparison used an ablation removing Jukebox features and included Bailando without its optional additional fine-tuning.
- Editing: EDGE supports dance completion, keyframe mid-choreography, joint-conditioned generation, in-betweening, and long-form generation under temporal and joint-wise constraints.Constraints can specify selected frames, joints, upper- or lower-body angles, and root trajectories.
- FID results: FIDg is unreliable on AIST++: FACT scores 12.75, Bailando 24.82, EDGE 23.08, and ground truth 41.4 despite human evaluations favoring EDGE over prior methods.The authors conclude that generated-output FIDg values do not reliably measure quality on this dataset.
5. Discussion
Dance-generation evaluation is difficult because dance is subjective and culturally specific, while current automated metrics can misrepresent human judgments. EDGE’s PFC metric targets foot sliding but has explicit contact and motion assumptions.
- Automated dance-quality metrics are challenging because dance is complex, subjective, and culturally specific.
- Overall Quality Evaluation: Current FID metrics on AIST++ are unreliable indicators of dance quality, potentially because the test set is small and features are superficial.
- Overall Quality Evaluation: More representative motion features could make distribution-based FID evaluations reliable without abandoning the underlying comparison.
- Beat Alignment Scoring: Beat alignment metrics can penalize valid dances that use double-time, half-time, skipped, or between-beat motions.
- Beat Alignment Scoring: EDGE’s beat alignment scores exceed both competing baselines and ground-truth samples, suggesting the metric may lose meaning at ground-truth-level quality.
- Physical Plausibility: PFC assumes only feet experience static contact and that center-of-mass acceleration comes from static contact, limiting use for gymnastics and extended sliding.
6. Conclusion and Future Work
EDGE generates realistic, long-form music-conditioned dances and supports temporal and joint-wise editing. It achieves state-of-the-art results on AIST++ and generalizes to in-the-wild music, but chaining short clips cannot represent very-long-term choreographic dependencies.
- EDGE generates realistic, long-form dance sequences conditioned on music and achieves state-of-the-art results on AIST++ while generalizing to in-the-wild inputs.
- Users can specify temporal and joint-wise constraints through the model’s editing capabilities.
- Chaining locally consistent shorter clips enables arbitrarily long sequences but cannot generate choreographies with very-long-term dependencies.
- Future work could extend editable dance generation to multi-person and scene-aware choreographies.
A. User Study
The user study used screened U.S. participants and standardized pairwise comparisons to evaluate dance quality. Responses were collected over 24 days and converted into Elo and win-rate metrics with binomial confidence intervals.
- Quality-control mechanisms and U.S.-participant restrictions removed 20 participants, leaving 147 raters.
- The study evaluated 10 training checkpoints alongside models to broaden skill-level coverage for Elo calculations.
- Physical-plausibility surveys asked raters which dance was more physically plausible, using the same remaining survey procedure.
- Responses collected over 24 days were summarized with win tables, Elo scores averaged over 1,000 randomized trials, and win rates.
- Pairwise 95% confidence intervals treated each matchup outcome as an independent Bernoulli trial using the binomial distribution.
- FACT was retrained for 300k steps because its published checkpoints were not fully trained, while Bailando used its official implementation and checkpoints.
C. Metrics and Evaluation
The evaluation uses short clips for automatic metrics and longer clips for qualitative studies, with official feature-extraction implementations and standardized method labels. Figure 6 shows the survey interface and stick-figure skeleton choice used for comparison and efficiency.
- Automatic PFC, beat-alignment, Distk, and Distg evaluation used 5-second clips sampled every 2.5 seconds from test music.
- Qualitative Prolific evaluation used 10-second clips sampled every 5 seconds from test music.
- The survey interface uses simple stick figures to support fair comparison with Bailando and improve computational efficiency.
- FID and diversity features were extracted directly with the official FACT repository, which uses fairmotion code.
- Table 4 reports pairwise user-study wins and abbreviates Jukebox features, CCL, guidance weight, out-of-domain inputs, and training checkpoints.
D. PFC
The section explains PFC’s assumptions and evaluates whether it tracks physical plausibility. PFC scores tend to improve during training, supporting its validity as a plausibility metric.
- Metric assumptions: Because AIST++ lacks explicit masses, PFC approximates center-of-mass acceleration using root-joint acceleration.
- Metric assumptions: PFC normalizes center-of-mass acceleration but not foot velocities, so foot-contact plausibility depends on more than root acceleration alone.Under static-contact assumptions, differing root accelerations with identical foot velocity can remain equally plausible, whereas differing foot velocities do not.
- Evaluation: PFC scores tend to improve throughout training, providing evidence that the metric measures physical plausibility.
G. Guidance Weight at Inference Time
Dropping guidance during early denoising steps increases diversity for the model sampled at guidance weight w = 1.
- 40% early-step guidance dropout increases diversity when sampling the model at w = 1.Guidance is set to w = 0 from denoising step 1000 through step 800.
H. Memory-efficient Jukebox Implementation
The section improves Jukebox feature extraction efficiency and describes related long-form and constrained-generation capabilities. These changes make music representations more accessible while supporting editable dance generation.
- Jukebox implementation: Previous Jukebox extraction implementations were memory-inefficient and slow on short clips.They could not load the full model on a GPU with 16GB VRAM and processed short clips as if they had full sequence length.
- Jukebox implementation: 2x memory efficiency and 4x faster extraction for 5-second clips improve the Jukebox implementation.The new implementation takes approximately 5 seconds for a 5-second clip on a Tesla T4 GPU.
- Jukebox implementation: The implementation uses meta-device initialization for memory savings and shorter-clip inference for speed.
- Jukebox implementation: These improvements make extracting Jukebox representations more accessible to researchers and practitioners.
- Editable generation: EDGE supports joint-wise conditioning by generating upper-body motion from lower-body constraints and lower-body motion from upper-body constraints.
- Editable generation: EDGE supports temporal in-betweening from start and end poses and continuation from a seed motion under arbitrary music conditioning.