PaperlayerView exact revision on arXiv

Source-linked AI summary

MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generation

Kaixing Yang, Jiashu Zhu, Xulong Tang, Ziqiao Peng, Xiangyue Zhang, Puwei Wang, Jiahong Wu, Xiangxiang Chu, Hongyan Liu, Jun He

arXiv:2512.18181v3cs.CV

TL;DR

Music-driven dance video generation must jointly produce realistic motion and high-quality, coherent appearance, but existing approaches struggle with both. MACE-Dance addresses this with cascaded Motion and Appearance Experts, achieving state-of-the-art performance in 3D dance generation, pose-driven image animation, and the overall task.

  • Problem

    Music-driven dance video generation must jointly achieve kinematically plausible, artistically expressive motion and high-fidelity, spatiotemporally consistent appearance.

  • Method

    MACE-Dance cascades a Motion Expert for music-to-3D motion with an Appearance Expert for motion- and reference-conditioned video synthesis, supported by a motion–appearance dataset and evaluation protocol.

  • Results

    MACE-Dance achieves state-of-the-art performance in overall dance video generation, music-driven 3D dance generation, and pose-driven image animation.

  • Takeaways & Limitations

    MACE-Dance provides a benchmarked framework for generating dance videos with plausible motion and spatiotemporally coherent visual appearance.

  • Takeaways & Limitations

    The 14B-parameter Appearance Expert has substantial computational demands that preclude real-time applications.

Abstract

from arXiv · show

With the rise of online dance-video platforms and rapid advances in AI-generated content (AIGC), music-driven dance generation has emerged as a compelling research direction. Despite substantial progress in related domains such as music-driven 3D dance generation, pose-driven image animation, and audio-driven talking-head synthesis, existing methods cannot be directly adapted to this task. Moreover, the limited studies in this area still struggle to jointly achieve high-quality visual appearance and realistic human motion. Accordingly, we present MACE-Dance, a music-driven dance video generation framework with cascaded Mixture-of-Experts (MoE). The Motion Expert performs music-to-3D motion generation while enforcing kinematic plausibility and artistic expressiveness, whereas the Appearance Expert carries out motion- and reference-conditioned video synthesis, preserving visual identity with spatiotemporal coherence. Specifically, the Motion Expert adopts a diffusion model with a BiMamba-Transformer hybrid architecture and a Guidance-Free Training (GFT) strategy, achieving state-of-the-art (SOTA) performance in 3D dance generation. The Appearance Expert employs a decoupled kinematic-aesthetic fine-tuning strategy, achieving state-of-the-art (SOTA) performance in pose-driven image animation. To better benchmark this task, we curate a large-scale and diverse dataset and design a motion-appearance evaluation protocol. Based on this protocol, MACE-Dance also achieves state-of-the-art performance. Code is available at https://github.com/AMAP-ML/MACE-Dance.

1 Introduction

MACE-Dance addresses music-driven dance video generation with cascaded Motion and Appearance Experts that jointly target plausible, expressive motion and coherent visual identity. The paper also introduces MA-Data and a motion–appearance evaluation protocol to benchmark the task.

  • Existing dance-generation methods primarily focus on 3D motion, while rendered 2D videos often lack realistic human–scene interactions and detailed appearance.
  • MACE-Dance uses cascaded experts: the Motion Expert generates kinematically plausible, artistically expressive 3D motion, and the Appearance Expert synthesizes motion- and reference-conditioned videos with coherent visual identity.The framework uses 3D SMPL parameters as an intermediate representation because they provide view-invariant and physically consistent supervision.
  • MA-Data contains 70k dance clips lasting 5–10 seconds each, totaling 116 hours across more than 20 dance genres.The dataset combines complementary motion-centric 3D-rendered data and other sources for benchmarking music-driven dance video generation.
  • The paper reports that MACE-Dance achieves SOTA performance in music-driven dance video generation, 3D dance generation, and pose-driven image animation.The Motion Expert uses a diffusion model with a BiMamba-Transformer hybrid architecture and the Appearance Expert uses decoupled Kinematic-Aesthetic fine-tuning.

2 Related Work

Prior work mainly addresses music-to-dance generation through 3D motion models or human-centric image animation, but existing dance-video methods struggle with occlusions, complex locomotion, and dance’s inherently 3D nature. MACE-Dance responds with a cascaded expert framework integrating motion and appearance generation.

  • Music-to-Dance Generation: Music-to-dance generation has largely focused on 3D motion using GAN-based, autoregressive, and diffusion-based models.The passage identifies these three broad families and gives examples of GAN-based methods including CoheDancers and DeepDance.
  • Human-Centric Image Animation: Human-centric image animation uses a reference image and driving signals to generate visually high-quality motion videos, including pose-driven methods based on 2D keypoints.Representative systems include Animate-X, Animate Anyone, and Wan-Animate.
  • Limitations and Motivation: Music-conditioned keypoint methods remain limited by limb occlusions and complex full-body locomotion, while existing dance-video approaches fail to capture dance’s inherently 3D nature.These limitations compromise both motion quality and visual appearance.
  • Limitations and Motivation: MACE-Dance introduces a cascaded expert framework that synergistically integrates motion and appearance generation for dance videos.The framework is presented as a response to the limitations of existing approaches.

3 Methodology

MACE-Dance uses cascaded Motion and Appearance Experts, with music converted into 3D motion before reference-conditioned dance-video synthesis. Its design combines a BiMamba–Transformer diffusion motion model with Guidance-Free Training and decoupled kinematic–aesthetic appearance fine-tuning.

  • Overall Framework: 3D motion bridges the experts because it preserves full-body geometry, global translation, and orientation needed for large-amplitude locomotion and complex spatial choreography.The method uses 3D motion instead of 2D keypoints as the intermediate representation.
  • Overall Framework: The cascaded framework maps music M to 3D motion X, then combines X with reference image I to synthesize spatiotemporally coherent dance video D.The Motion Expert enforces kinematic plausibility and artistic expressiveness, while the Appearance Expert preserves appearance during animation.
  • Motion Expert: Guidance-Free Training adds reconstruction, 3D joint, velocity, and foot contact losses to improve physical plausibility and aesthetic expressiveness.The overall objective is a weighted sum of these losses, with weights chosen to balance their magnitudes.
  • Motion Expert: The Motion Expert uses a BiMamba–Transformer hybrid: BiMamba models local intra-modal dependencies, while the Transformer captures cross-modal global musical context.Their complementary roles support short-range movement continuity and alignment with long-term musical phrasing, rhythm, and structure.
  • Appearance Expert: The Appearance Expert builds on Wan-Animate and uses decoupled kinematic–aesthetic fine-tuning for motion-guided, reference-conditioned video synthesis.During the aesthetic stage, kinematic pathways are frozen and lightweight LoRA adapters refine textures, clothing, accessories, and camera choreography.

4 Experiment … 4.4 Qualitative Analysis

MACE-Dance is evaluated on a newly curated large-scale dance-video dataset using a motion–appearance protocol, with comparisons showing SOTA motion and appearance quality. Qualitative analyses further demonstrate strong cross-domain, genre-specific, and long-sequence generation.

  • 4.1 Dataset: MA-Data contains 70k clips totaling 116 hours across more than 20 dance genres, combining complementary motion-centric and appearance-oriented sources.The dataset includes 5–10-second clips and genres such as Jazz, Latin, and Eastern Folk.
  • 4.2 Evaluation: The evaluation protocol separately measures motion quality and visual appearance to address kinematic plausibility, artistic expressiveness, fidelity, and spatiotemporal consistency.The protocol extracts 2D keypoint sequences with ViTPose for motion evaluation.
  • 4.3 Comparison: MACE-Dance is compared with 3D-dance-generation pipelines using Wan-Animate and general human-motion video generators on MA-Data.The baselines include EDGE, Lodge, MEGA, EchoMimic-V3, and WAN-S2V.
  • 4.3.1 Music-Driven 3D Dance Generation: The Motion Expert attains overall SOTA on FineDance, including best FID_k=17.83, DIV_k=10.30, DIV_g=8.09, and BAS=0.229, plus competitive FID_g=25.09 and FSR.These results indicate high fidelity, strong diversity, and physical plausibility.
  • 4.3.2 Pose-Driven Image Animation: The Appearance Expert achieves SOTA on all pose-driven image-animation metrics: FVD=274.94, SSIM=0.739, LPIPS=0.066, and PSNR=22.40.It is compared against Animate-Anyone, Magic-Animate, and Wan-Animate on MA-Data.
  • 4.4.1 Effect Comparison: Qualitatively, MACE-Dance avoids the blurring, artifacts, abrupt discontinuities, abnormal movements, and repetitive motions observed in competing methods across real-person and anime references and multiple music genres.The comparison includes Hallo2, EDGE, Lodge, MEGA, WAN-S2V, and Echomimic-V3.
  • 4.4.2 Cross-Genre Generation: MACE-Dance generalizes across genres by producing distinct motion signatures, such as Uyghur upper-body rotations, Dunhuang circular arm patterns, Dai flowing limb movements, and K-Pop crisp rhythmic poses.The qualitative examples demonstrate genre-specific motion rather than a single generic movement pattern.
  • 4.4.3 Long-Sequence Generation: For long-sequence synthesis, MACE-Dance combines a BiMamba–Transformer Motion Expert with pose-driven relay rendering and identity anchoring to mitigate motion drift and visual degradation.These designs target music tracks lasting roughly 30 seconds to 5 minutes.

4.5 Ablation Study · 4.6 Comparison with Video Foundation Models

Ablations show that MACE-Dance’s bidirectional motion modeling, guidance-free training, and two-stage appearance fine-tuning are important for quality. The complete system also outperforms expert-swapped variants and is evaluated against general-purpose video foundation models.

  • 4.5.1 Motion Expert.: Removing BiMamba weakens temporal understanding and degrades all dance-quality metrics, despite improving generation efficiency.The study identifies this efficiency–quality trade-off as unfavorable.
  • 4.5.2 Appearance Expert.: Removing the kinematic fine-tuning stage causes modest declines across all metrics, with kinematic errors and motion blur.This stage fine-tunes the Body Adapter while freezing other components.
  • 4.5.2 Appearance Expert.: Removing the aesthetic stage substantially degrades all metrics and produces obvious ghosting, while the Appearance Expert outperforms Wan-Animate.These findings support the Kinematic-Aesthetic fine-tuning strategy’s role in preserving video aesthetics.
  • 4.5.3 Motion Representation.: MACE-Dance compares 2D and 3D intermediate motion representations at both motion-generation and final-video levels, using matched Motion Expert training on FineDance.The design is motivated by the prevalence of 2D keypoints in pose-driven image animation, whereas MACE-Dance uses 3D motion.
  • 4.5.4 Role of Each Expert.: The full MACE-Dance consistently achieves the best performance across all evaluated metrics when either expert is replaced by its baseline counterpart.The cross-composition study uses EDGE as the baseline Motion Expert and WAN-Animate as the baseline Appearance Expert.
  • 4.6 Comparison with Video Foundation Models: MACE-Dance is compared with CogVideoX1.5-5B and WAN2.2-5B to assess a structured motion-to-appearance pipeline against general-purpose video foundation models.These models have broad video-generation ability but are not specifically designed for music-driven dance video generation, including beat, rhythm, and body-motion coherence.
  • 4.6 Comparison with Video Foundation Models: The comparison with general-purpose video foundation models is reported quantitatively and qualitatively through Table 6 and Figure 8.The cited materials identify the evaluation artifacts but do not provide their numerical results in the supplied passage text.

5 Conclusion

MACE-Dance is a cascaded Mixture-of-Experts framework for music-driven dance video generation. Its Motion Expert targets kinematic plausibility and artistic expressiveness, while its Appearance Expert preserves visual identity and spatiotemporal coherence.

  • Framework: MACE-Dance uses cascaded Mixture-of-Experts for music-driven dance video generation.The framework combines separate Motion and Appearance Experts.
  • Motion Expert: The Motion Expert enforces kinematic plausibility and artistic expressiveness.It uses a diffusion model with a BiMamba–Transformer hybrid backbone and Guidance-Free Training.
  • Appearance Expert: The Appearance Expert preserves visual identity with spatiotemporal coherence.It adopts a decoupled Kinematic–Aesthetic fine-tuning strategy.

6 Implementation Details

MACE-Dance decouples music-driven dance video generation into Motion and Appearance Experts trained on task-specific data. The Motion Expert uses a BiMamba-Transformer diffusion model with GFT, while the Appearance Expert uses kinematic-aesthetic decoupled fine-tuning with staged optimization.

  • Framework and Data: MACE-Dance separates generation into music-to-3D motion and pose-driven image animation, training the Motion Expert on motion-centric rendered data and the Appearance Expert on all MA-Data.This cascaded Mixture-of-Experts design assigns each expert task-specific training data.
  • Motion Expert: The Motion Expert combines a diffusion model with a BiMamba-Transformer hybrid architecture and Guidance-Free Training on FineDance.Training uses Adam with learning rate 4 × 10−4, weight decay 0.02, 4000 epochs, batch size 128, and eight NVIDIA H20 GPUs.
  • Motion Expert: The Motion Expert trains on 240-frame (8s) sequences and infers 1024-frame (34.13s) sequences, with EMA decay 0.9999 applied for stabilization.Distributed training uses the Accelerate library, and checkpoints are periodically saved for evaluation.
  • Appearance Expert: The Appearance Expert applies kinematic-aesthetic decoupled fine-tuning on MA-Data, first updating only the Body Adapter while freezing the DiT backbone and VAE.The Kinematic Stage uses Adam with learning rate 1×10−5, batch size 128, and 50k iterations with simple diffusion noise prediction loss.

7 User Study

A double-blind user study with 40 participants evaluated five methods across six motion and appearance dimensions using preference-based rankings. MACE-Dance achieved dominant user preference across all dimensions, and its automatic metrics aligned strongly with human judgments.

  • Study Design: The study used 30 eight-second music segments and a double-blind questionnaire completed by 40 participants with dance backgrounds.Generated sequences from the models in Sec. 4.3.1 were evaluated in the music-to-dance generation task.
  • Study Design: Participants ranked five methods side-by-side using a randomized, preference-based mechanism that allowed multiple selections per test case.This design was intended to capture subtle differences between methods rather than score videos in isolation.
  • Evaluation Dimensions: The evaluation covered six dimensions: Dance Synchronization, Dance Quality, Dance Creativity, Perceptual Quality, Temporal Consistency, and Identity Consistency.The first three assess human motion, while the latter three assess visual appearance.
  • User-Study Results: MACE-Dance achieved the highest preference across all six dimensions, including over 60% of votes for Dance Synchronization and Dance Quality.Users also strongly preferred its perceptual quality, identity consistency, and temporal consistency, while it led in Dance Creativity.
  • Protocol Validation: MACE-Dance’s automatic metrics aligned with human preferences, including BAS (0.523), FID𝑔 (0.28), DIV𝑘 (9.74), SC (93.97), BC (94.57), and TF (97.10).Human preference ratios ranged from 50% to 65.1%, with DQ and DS both at 65.1%, IC at 50.0%, TC at 56.2%, and PQ at 60.9%.

8 Qualitative Analysis

Qualitative comparisons show that MACE-Dance produces kinematically plausible, artistically expressive motion and preserves spatiotemporally coherent appearance across diverse references and musical styles. Competing methods exhibit blurred faces, artifacts, discontinuities, implausible configurations, repetitive motion, or instability in fast dance movements.

  • Dance Video Generation: MACE-Dance preserves spatiotemporally coherent appearance while generating kinematically plausible and artistically expressive movements across diverse reference images and musical styles.The comparisons cover diverse reference-image domains and music genres.
  • Dance Video Generation: Hallo2 produces blurred facial regions and substantial background artifacts, whereas EDGE suffers from abrupt motion discontinuities that degrade temporal smoothness.
  • 3D Dance Generation: EDGE exhibits abrupt motion discontinuities, Lodge generates physically implausible body configurations, and MEGA produces simplified, repetitive sequences with limited expressive diversity.
  • 3D Dance Generation: The Motion Expert generates kinematically plausible and artistically expressive motion with stable dynamics and rich stylistic detail.These results support its ability to model high-quality, music-driven 3D dance motion.
  • Pose-Driven Video Animation: Magic-Animate and Animate-Anyone often produce spatial distortions and temporal flickering in fast or large-amplitude motions, causing unstable body shapes and inconsistent textures.The comparison is conducted against Magic-Animate, Animate-Anyone, and Wan-Animate on the MA-Data test set.

9 Motion Editing

MACE-Dance’s Motion Expert supports inference-time motion editing by preserving user-specified 3D constraints while plausibly generating unconstrained regions. Its explicit 3D outputs also enable motion retargeting for character rigs and reuse in applications beyond video synthesis.

  • Motion editing: The Motion Expert performs masked denoising on structured 3D motion sequences, preserving fixed user constraints while completing unknown regions.At each reverse step, constrained elements are replaced with their forward-diffused known motion at the same noise level.
  • Motion editing: The unconstrained region is generated by the diffusion prior to maintain temporal smoothness, physical plausibility, and musical coherence.The editing mechanism is compatible with DDIM-based inference.
  • Editing modes: Different mask designs support temporal inpainting for motion in-betweening and smooth transitions, as well as joint-wise inpainting that fixes selected body parts.Temporal inpainting preserves motion at sequence boundaries while synthesizing missing middle segments; joint-wise inpainting completes remaining joints.
  • Motion reuse: Because the Motion Expert outputs explicit 3D motion, its results can be transferred to standard character rigs through conventional motion retargeting pipelines.The generated motion can be rendered by the Appearance Expert or reused as a structured asset for CG animation, VR avatars, and interactive applications.

10 Further Discussion about MACE-Dance

MACE-Dance’s cascaded MoE factorizes music-driven dance video generation into specialized motion and appearance tasks, reducing complexity while enabling interpretability and control. Its long-sequence design combines motion length extrapolation with pose-anchored relay generation to support complete dance performances.

  • Cascaded MoE Design: MACE-Dance decouples generation into a music-to-3D Motion Expert and a pose-driven Appearance Expert.The cascaded MoE architecture is motivated by reducing learning complexity and improving data utilization.
  • Cascaded MoE Design: Task factorization lets the Motion Expert model music–kinematics relationships while the Appearance Expert synthesizes images from explicit poses.This specialization avoids requiring the motion model to learn texture or lighting and the appearance model to understand musical semantics.
  • Cascaded MoE Design: The intermediate 3D motion representation suppresses spurious music–visual correlations and provides an inspectable, editable, and replaceable control interface.The structured bottleneck filters irrelevant cues such as background or clothing, while supporting user-level control.
  • Long-Sequence Generation: Long-sequence generation is fundamental because complete performances typically span 30s-4min and require full choreographic, narrative, and artistic structure.Short clips cannot capture the full emotional arc or precise synchronization of a musical piece.
  • Long-Sequence Generation: The Motion Expert extrapolates beyond short training clips through BiMamba–Transformer modeling, while the Appearance Expert uses pose-anchored relay generation for coherent video chunks.Mamba propagates local dynamics beyond the training horizon, and relay generation anchors chunks to globally consistent 2D poses and prior frames.

11 Ethical Considerations

MACE-Dance presents ethical risks because it can generate realistic identity-preserving videos without consent and may reflect biases in its training data. The authors therefore emphasize research-only use, legal and ethical compliance, and safeguards for deployment.

  • Risks: The model could be misused to synthesize realistic videos of individuals without consent, enabling misleading or deceptive media.This risk is especially relevant because the Appearance Expert preserves identity-related cues from a reference image.
  • Risks: Training-data biases in dance style, body shape, clothing, scene composition, and cultural representation may produce uneven quality across subjects or styles.The outputs should not be interpreted as neutral or universally representative.
  • Safeguards: MACE-Dance is intended for research on controllable dance video synthesis, not identity manipulation or harmful content creation.Practical deployment should respect consent, portrait rights, and copyright constraints.
  • Safeguards: Future releases should consider usage restrictions, provenance disclosure, or watermarking mechanisms.These safeguards are proposed to support responsible practical deployment.

12 Limitations and Future Work

MACE-Dance is limited by music as a fixed-form control signal that cannot fully express diverse user intentions. Future work targets free-form textual control and real-time interaction, with the Appearance Expert remaining the main efficiency bottleneck despite SOTA motion-generation efficiency and animation quality.

  • Music is a fixed-form carrier that cannot fully capture diverse user intentions.
  • Future work will incorporate free-form textual descriptions for lower-cost, more flexible, semantically rich, personalized, and expressive choreographic control.Text is envisioned as a rich, hierarchical control mechanism for articulating choreographic requirements.
  • Real-time interaction is constrained primarily by the Appearance Expert, although the Motion Expert achieves SOTA generation efficiency in 3D motion synthesis.The fine-tuned Appearance Expert is based on the 14B-parameter Wand-Animate model and delivers SOTA quality in pose-driven image animation, but its computational demands preclude real-time use.
Loading 2512.18181v3…