Source-linked AI summary
DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer
Buyu Li, Yongchi Zhao, Zhelun Shi, Lu Sheng
TL;DR
Music-conditioned 3D dance generation must produce long, complex, coherent motion aligned with musical beats. DanceFormer addresses this with beat-level key-pose generation followed by parametric motion-curve prediction, supported by the animator-labeled PhantomDance dataset. Experiments report state-of-the-art performance that significantly surpasses prior works qualitatively and quantitatively.
Problem
Prior sequence-generation approaches struggle to render long-term 3D dance sequences with high kinematic complexity and coherent, music-aligned movements.
Method
DanceFormer cascades DanTrans networks for beat-synchronized key-pose generation and in-between parametric motion-curve regression, using kinematic enhancements and PhantomDance annotations.
Results
DanceFormer achieves state-of-the-art results and significantly surpasses other works qualitatively and quantitatively on PhantomDance and AIST++.
Takeaways & Limitations
The key-pose and curve representation, together with PhantomDance, is compatible with industrial animation software and supports downstream applications.
Abstract
from arXiv · showhide
Generating 3D dances from music is an emerged research task that benefits a lot of applications in vision and graphics. Previous works treat this task as sequence generation, however, it is challenging to render a music-aligned long-term sequence with high kinematic complexity and coherent movements. In this paper, we reformulate it by a two-stage process, ie, a key pose generation and then an in-between parametric motion curve prediction, where the key poses are easier to be synchronized with the music beats and the parametric curves can be efficiently regressed to render fluent rhythm-aligned movements. We named the proposed method as DanceFormer, which includes two cascading kinematics-enhanced transformer-guided networks (called DanTrans) that tackle each stage, respectively. Furthermore, we propose a large-scale music conditioned 3D dance dataset, called PhantomDance, that is accurately labeled by experienced animators rather than reconstruction or motion capture. This dataset also encodes dances as key poses and parametric motion curves apart from pose sequences, thus benefiting the training of our DanceFormer. Extensive experiments demonstrate that the proposed method, even trained by existing datasets, can generate fluent, performative, and music-matched 3D dances that surpass previous works quantitatively and qualitatively. Moreover, the proposed DanceFormer, together with the PhantomDance dataset (https://github.com/libuyu/PhantomDanceDataset), are seamlessly compatible with industrial animation software, thus facilitating the adaptation for various downstream applications.
1 Introduction
DanceFormer reframes music-conditioned 3D dance generation as key-pose generation followed by parametric motion-curve regression. Its DanTrans networks and PhantomDance dataset target synchronized, coherent, expressive dances compatible with animation pipelines.
- Prior work models music-conditioned 3D dance as sequence generation, but long-term sequences with high kinematic complexity and coherent movements remain challenging.
- The proposed coarse-to-fine representation generates beat-synchronized key poses and regresses in-between parametric curves for fluent, rhythm-consistent motion.The representation also permits user-controlled temporal resolutions.
- The motion decoder incorporates kinematic propagation and structured multi-head attention to enhance spatial correlations using human-body topology.These modules increase physical significance and aim to alleviate unreasonable motion.
- PhantomDance is animator-produced rather than reconstructed or motion-captured, and stores key poses with motion-curve parameters for fluent, expressive, music-matched sequences.The dataset is designed to be compatible with industrial animation pipelines.
- DanceFormer achieves state-of-the-art results and significantly surpasses other works qualitatively and quantitatively on PhantomDance and AIST++.
- DanceFormer uses two cascaded DanTrans networks: the first generates key poses, while the second predicts motion-curve parameters.DanTrans uses adaptive temporal attention and explicitly enhances kinematic correlations.
2 Related Works
Existing 3D dance datasets and synthesis methods leave gaps in dance-specific coverage, music alignment, and pose quality. PhantomDance addresses these gaps with animator-created, higher-quality data and motion representations suited to animation.
- Deep learning approaches to 3D dance synthesis include convolutional, recurrent, and variational auto-encoder models, alongside earlier retrieval-based methods.
- General motion datasets focus on everyday actions and are poorly matched to dance because of distribution differences and absent music-dance alignment.
- PhantomDance uses experienced animators and professional dancer guidance, with stored key poses and parametric curves that support industrial animation software.
3 The PhantomDance Dataset
PhantomDance contains animator-created 3D dances represented with key poses and parametric motion curves. Its design emphasizes broad genre coverage, music alignment, dense temporal sampling, and high-quality labels for animation and learning.
- The dataset contains 260 dance videos spanning more than 13 genres and over 100 subjects, totaling 9.6 hours.Experienced animators created the 3D sequences under professional dancer instruction.
- PhantomDance uses a standard SMPL character with 24 joints and parameterizes root position plus joint rotations using a subject-invariant skeleton.
- Parametric motion curves provide analytical, continuous representations that can be densely sampled at varied temporal resolutions for real-time rendering.
- The dataset covers 13 genres with BPM values from 75 to 178 and provides pose labels, music beats, and motion-curve parameters.These annotations directly facilitate animation or model learning compatible with industrial software.
- Professional animators created PhantomDance motion from scratch with dancer guidance and strict quality checking, requiring about 15 months of labeling.The paper contrasts this process with noise in motion-capture and video-reconstruction data.
4 DanceFormer
DanceFormer generates music-conditioned 3D dances in two stages: beat-aligned key poses followed by parametric motion curves between adjacent poses. Its DanTrans networks incorporate kinematic structure through KPM and structured multi-head attention, while the motion curves use TCB splines.
- Two-stage framework: DanceFormer cascades DanTrans networks to generate beat-aligned key poses first and then predict motion-curve parameters between adjacent poses.Stage 1 uses music features around beats; stage 2 combines generated key poses with features between beats.
- Network Structure of DanTrans: Each DanTrans combines a transformer-based wave encoder with a motion decoder containing KPM and Structured Multi-head Attention to model human-body spatial correlations.KPM propagates and fuses features along kinematic chains, while structured attention organizes joint-node features as attention heads.
- Stage 1: Key Pose Generation: Stage 1 represents each key pose by 168 joint position-and-quaternion values and targets poses synchronized with extracted music beats.The beat times are obtained with a dynamic programming beat-tracking algorithm.
- Stage 2: Parametric Motion Curve Prediction: Stage 2 models each joint transformation-parameter trajectory with multi-knot Kochanek-Bartels splines, also called TCB splines.The curves encode translation and quaternion rotation as functions of time.
- Stage 2: Parametric Motion Curve Prediction: The stage-2 decoder autoregressively predicts seven control parameters per four-knot TCB spline between adjacent key poses.The seven parameters comprise tension, continuity, bias, and the positions of two intermediate knots.
- Training: Adversarial training uses separate discriminators for the two DanTrans networks to assess alignment between generated and ground-truth output distributions.The discriminators process music features together with key-pose sequences or motion-curve parameters.
5 Experiments
Experiments evaluate DanceFormer on PhantomDance and AIST++ using motion quality, distribution, diversity, beat consistency, ablations, attention variants, and user studies. Results show advantages from two-stage curve prediction and kinematics-aware modules, with stronger quantitative and qualitative performance than compared methods.
- Experimental Setup: DanceFormer is evaluated on PhantomDance and AIST++, with PhantomDance using 900 training pairs and 100 test pairs spanning 13 genres and 80–180 BPM.The study follows the official AIST++ train/test split.
- Evaluation Metrics: NPSS measures long-term motion synthesis, while PFD and VFD measure joint position and velocity distribution distances.NPSS is computed in joint motion space containing quaternion rotations and joint positions.
- Ablation Study: The pose-to-curve two-stage framework outperforms direct 60-FPS sequence prediction in the Curve to Frame ablation.The baseline uses a standard transformer with 0.5-second sliding music windows and a 1/60-second step.
- Ablation Study: KPM surpasses linear modules and a graph neural network, with gains evaluated by NPSS, PFD, and VFD.The comparison supports incorporating spatial correlations and physics constraints into the motion representation.
- Comparison with Other Methods: DanceFormer surpasses AI Choreographer by 30% in NPSS, 28% in PFD, 27% in VFD, 46% in PVar, and 93% in BC.It also outperforms Li et al. (2020) and Music2Dance by a considerably large margin; BC has an upper bound of 1.
- Comparison with Other Methods: Qualitative results show diverse, high-kinematic-complexity movements, while a 100-participant user study preferred DanceFormer on performance quality and music matching.The study compared 400 pairs across 100 validation-set musics, including baselines and ground truth.
6 Conclusion
The paper reformulates music-conditioned 3D dance generation around key poses and in-between motion curves, introducing DanceFormer and the PhantomDance dataset.
- DanceFormer models dance as key poses plus in-between motion curves rather than pose sequences.The curve representation is intended to produce more fluent and graceful generated motion.
- The transformer-based DanTrans architecture includes a Kinematic Propagation Module for modeling kinematic correlations.
- PhantomDance is presented as the first music-conditioned 3D dance dataset using curves to represent body motion and as the largest dataset with the best visual quality to date.