PaperlayerView exact revision on arXiv

Source-linked AI summary

CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation

Jinting Wang, Chenxing Li, Dong Yu, Li Liu

arXiv:2609.13118v1cs.AIcs.SD

TL;DR

D2M models must infer dense musical composition from sparse dance cues, while existing approaches typically supervise only final audio. CMA-OT progressively transfers hierarchical knowledge from a pre-trained music expert and aligns representations with scale-aware FGW; it outperforms state-of-the-art approaches on two datasets.

  • Problem

    Dance provides sparse rhythm and style cues, while music composition requires dense structural and timbral information; existing methods typically rely on sparse cues and final-audio supervision.

    Source: §I · p. 1

  • Method

    CMA-OT progressively transfers multi-scale music-expert supervision from coarse to fine and uses scale-aware FGW to align expert representations with generator latents.

    Source: §I · p. 2

  • Results

    CMA-OT outperforms state-of-the-art approaches on two datasets, supporting the effectiveness of hierarchical expert supervision for high-quality D2M generation.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that hierarchical expert supervision bridges the semantic gap and supports robust representations and coherent, expressive, well-aligned music generation.

    Source: §I · p. 2

Abstract

from arXiv · show

Dance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics. Existing methods typically rely on these sparse cues and supervise only the final audio output, resulting in poorly learned music representations and generated music with limited musicality and structural coherence. To address these issues, we propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning. To effectively incorporate hierarchical supervision, we introduce a curriculum-guided multi-scale learning strategy that progressively transfers musical knowledge from the expert to the music generator, enabling stable and effective representation learning. Moreover, to accommodate the semantic and structural variations across different expert scales and achieve fine-grained alignment under temporal mismatch, we propose a scale-aware optimal transport alignment mechanism, which models soft correspondences between hierarchical expert representations and the generator's latent features. Extensive experiments on two datasets demonstrate that CMA-OT achieves state-of-the-art performance in rhythmic synchronization, perceptual quality, and overall music generation.

I. INTRODUCTION

D2M must translate sparse dance cues into dense musical structure, but existing methods generally rely on final-audio supervision. CMA-OT instead uses hierarchical music-expert supervision, progressive multi-scale learning, and scale-aware alignment; the paper reports gains over state-of-the-art approaches on two datasets.

I. INTRODUCTION

  • Dance offers sparse rhythm and style cues, whereas music composition requires dense information about structure, melody, rhythm, and timbre.Source: §I · p. 1
  • Existing methods supervise mainly final audio, leaving generators to learn rhythmic alignment and composition together from scarce paired data, which can limit musicality and structural coherence.Source: §I · p. 1
  • CMA-OT uses a pre-trained multi-scale music expert to guide generator latents hierarchically, bridging dance–music mismatch and supporting coherent, expressive, well-aligned generation.Source: §I · p. 2
  • A curriculum progressively transfers supervision from coarse to fine, while scale-aware FGW models soft correspondences that preserve semantic consistency and temporal structure.Source: §I · p. 2
  • CMA-OT outperforms state-of-the-art approaches on two datasets, supporting the effectiveness of hierarchical expert supervision for high-quality D2M generation.Source: §I · p. 2

A. Dance-to-Music Generation

  • For D2M, hierarchical representation supervision provides structured guidance across semantic levels, in contrast to dance-centric representations that lack explicit modeling of rich musical structures.Source: A. Dance-to-Music Generation · p. 2

B. Representation Alignment for Generative Models

  • Existing alignment methods often target global semantic consistency or point-wise matching without explicit temporal or hierarchical structure; this work introduces scale-aware alignment for multi-scale music experts.Source: B. Representation Alignment for… · p. 2

C. Optimal Transport for Cross-Space Alignment

FGW combines feature similarity with structural consistency, and this work extends it to scale-aware alignment across semantic levels. The framework also uses representation-level curriculum learning and a multi-scale music expert whose codes range from global musical semantics to local temporal detail.

C. Optimal Transport for Cross-Space Alignment

D. Curriculum Learning for Structured Training

  • The representation-level curriculum progressively transfers supervision from coarse semantic structure to fine-grained temporal dynamics, helping the model learn global organization before local details.Source: D. Curriculum Learning for Structured… · p. 3

III. PRELIMINARY

  • The framework's theoretical foundations cover hierarchical music representations and FGW for structured representation alignment.Source: §III · p. 3

A. Multi-Scale Music Expert

  • A pre-trained multi-scale VQ-VAE music expert maps audio into top-level codes for global style and genre, middle-level codes for melodic and harmonic phrases, and bottom-level codes for rhythmic textures and timbre.Source: A. Multi-Scale Music Expert · p. 3

B. Fused Gromov-Wasserstein Distance

CMA-OT uses a multi-scale music expert to guide a flow-based generator conditioned on dance style and rhythm. It aligns expert and generator representations at coarse, medium, and fine levels, using curriculum learning and FGW to handle temporal and structural differences.

B. Fused Gromov-Wasserstein Distance

  • FGW compares feature similarity and sequence structure, unlike standard OT’s focus on global feature similarity, making it suitable for unequal-length sequences.The transport plan couples sequences while preserving their intra-sequence relations.Source: B. Fused Gromov-Wasserstein Distance · p. 4

A. Overview

  • I3D extracts global dance-style features, while a rhythm encoder captures fine-grained temporal dynamics to condition the flow-based music generator.A pre-trained music expert provides hierarchical structural guidance for musical coherence.Source: A. Overview · p. 4

B. Curriculum-Guided Multi-Scale Alignment

  • The expert’s top, middle, and bottom representations encode global style, melodic and harmonic structures, and local rhythmic patterns, respectively.These levels are aligned with deep, middle, and shallow DiT representations.Source: B. Curriculum-Guided Multi-Scale… · p. 4
  • A smooth three-phase curriculum with cosine-annealed weights progressively activates multi-scale alignment to mitigate conflicts between scales.The curriculum moves from top-level alignment to middle-level and then bottom-level alignment; the full objective is retained in the final phase to maintain optimization stability.Source: B. Curriculum-Guided Multi-Scale… · p. 4
  • The multi-scale alignment loss is computed at the final denoising step, when DiT representations most closely approximate clean data.This applies supervision in a space comparable to clean expert features.Source: B. Curriculum-Guided Multi-Scale… · p. 5

C. Scale-Aware Optimal Transport Alignment

CMA-OT uses scale-aware FGW alignment to address semantic and temporal discrepancies between expert representations and DiT states. Its adaptive weighting combines semantic correspondence with structural preservation at each scale.

C. Scale-Aware Optimal Transport Alignment

  • Scale-aware FGW jointly models semantic correspondence and structural relations to address temporal misalignment between expert representations and DiT internal states.The scale-aware strategy adaptively balances these constraints across levels.Source: C. Scale-Aware Optimal Transport… · p. 5
  • At each scale, the transport plan P(k) and intra-sequence relation matrices encode alignment and the sequences’ internal topological structure.The matrices are denoted C̃hk and Czk.Source: C. Scale-Aware Optimal Transport… · p. 5
  • The adaptive factor αk is computed from expected semantic and structural costs under the current transport plan, dynamically weighting both forms of consistency.The formulation preserves FGW’s soft-alignment nature; σ is the sigmoid function and ε supports numerical stability.Source: C. Scale-Aware Optimal Transport… · p. 5

D. Conditional Music Generation

The music generator uses conditional flow matching to learn a trajectory from Gaussian noise toward music latents, conditioned on dance-derived visual and rhythm features. Its training objective combines the generation loss with hierarchical expert alignment.

D. Conditional Music Generation

  • Conditional flow matching trains a continuous trajectory from Gaussian noise to music latents using dance-derived visual-style and temporal-rhythm features.The input music signal is encoded as Z0, with Zt denoting its noisy version at time t.Source: D. Conditional Music Generation · p. 5
  • The conditional flow matching loss trains the velocity field to map noisy latent trajectories back to the target music latent.The field is conditioned on dance-derived visual and rhythm features.Source: D. Conditional Music Generation · p. 5
  • The final objective jointly optimizes flow-matching generation and hierarchical expert alignment to support perceptually realistic, rhythmically synchronized music with hierarchical semantic consistency.These properties are stated for the combined loss.Source: D. Conditional Music Generation · pp. 5–6

V. EXPERIMENT

Experiments use AIST++ and TikTok under standard dataset splits and evaluation protocols from prior work. The datasets differ in size, genre coverage, and recording context.

A. Dataset

  • Experiments evaluate CMA-OT on AIST++ and TikTok using standard dataset splits and evaluation protocols from prior work.The cited protocols are from and [37].Source: A. Dataset · p. 6
  • AIST++ contains 1,020 dance videos across 10 genres, paired with music and recorded in professional studios with clean backgrounds.The dataset is cited as.Source: A. Dataset · p. 6
  • TikTok contains 445 dance videos covering 85 different songs, and the dataset is cited as.Source: A. Dataset · p. 6

B. Implementation Details

CMA-OT is reported to outperform existing methods across two datasets, while ablations associate its gains with hierarchical supervision, curriculum learning, and scale-aware alignment. The section also specifies the implementation and evaluation measures used.

B. Implementation Details

  • Training uses a 12-layer DiT with VAE music latents; a three-stage cosine curriculum shifts emphasis from top-level to mid-level and then bottom-level alignment.The model is trained for 200 epochs and uses a 32-step Euler ODE sampler at inference.Source: B. Implementation Details · p. 6

C. Evaluation Metrics

  • Evaluation covers beat coverage and hit scores, FAD-based music similarity, four aesthetic dimensions, and human ratings of overall quality and dance relevance.Human ratings use a five-point Likert scale; the metrics passage defines the measures and feature extractors.Source: C. Evaluation Metrics · p. 6

D. Comparison with SOTA Methods

  • CMA-OT consistently outperforms baselines on two datasets across rhythmic alignment, aesthetic quality, and music quality; qualitative comparisons report stronger rhythmic alignment and tonal coherence.Figure 4 contrasts CMA-OT with competing methods and ground-truth music.Source: D. Comparison with SOTA Methods · pp. 6–7, Figure 4 · p. 7

E. Ablation Study

  • Global CLAP and MERT supervision yields only marginal gains, while single-scale expert alignment—particularly at the bottom level—produces the largest rhythmic gains among single-scale variants.The ablation compares external supervision, hierarchical alignment, curriculum learning, and scale-aware adaptation on AIST++.Source: E. Ablation Study · p. 7, Table II · p. 7
  • Multi-scale alignment without curriculum degrades under conflicting gradients, whereas curriculum-guided alignment reduces FADp by 40.81% versus Base.The comparison attributes more stable cross-scale knowledge transfer to coarse-to-fine curriculum learning.Source: E. Ablation Study · p. 7, Table II · p. 7
  • FGW improves all metrics over cosine similarity, and scale-aware adaptive weighting adds gains; CMA-OT achieves the best overall performance and converges faster.Standard OT improves distribution-level alignment but does not preserve hierarchical structural consistency across scales.Source: Figure 5 · p. 7, Curriculum Learning Strategy · p. 8, Table IV · p. 8
  • Among the curriculum strategies, the cosine schedule achieves the highest rhythmic and aesthetic scores and the lowest FADp.The comparison attributes improved stability and music quality to smooth, continuous emphasis during training.Source: Curriculum Learning Strategy · p. 8, Table III · p. 8

F. User Study

In a user study, CMA-OT receives the highest average scores for synchronization and musical quality among the compared methods. The conclusion presents hierarchical expert supervision, curriculum learning, and scale-aware FGW alignment as the framework’s central components.

F. User Study

  • The user study evaluates generated music for both overall perceptual quality and relevance to the input dance video.The evaluation uses a five-point Likert scale.Source: F. User Study · p. 8
  • CMA-OT achieves the highest average scores on both user-study metrics and outperforms all baselines in synchronization and perceived musical coherence.Forty volunteers rated 30 AIST++ test samples generated by six D2M methods.Source: F. User Study · p. 8

VI. CONCLUSION

  • CMA-OT combines a pre-trained multi-scale music expert, progressive coarse-to-fine supervision, and scale-aware FGW alignment for generator latents.The conclusion reports experiments on two datasets and consistent gains in synchronization, perceptual quality, and expressiveness.Source: §VI · p. 8

APPENDIX

The appendix describes the multi-scale expert features and two alternative curriculum schedules. Fixed weights activate all alignment levels together, while hard switch changes emphasis abruptly across three training phases.

APPENDIX

  • The expert VQ-VAE supplies bottom, middle, and top latent levels that guide the generator from rhythmic details toward global musical style.The levels are derived from Jukebox 1B Lyrics2 using its VQ-VAE encoder.Source: APPENDIX · p. 10

A. Fixed Weights

  • The fixed-weight baseline keeps top, middle, and bottom alignment weights at 0.5, 0.3, and 0.2 throughout training.It activates all alignment levels simultaneously without curriculum scheduling.Source: A. Fixed Weights · p. 10, APPENDIX · p. 10

B. Hard Switch

  • Hard switch divides training into phases beginning at epochs 30 and 100, with abrupt transitions that can cause optimization shocks and hinder stable convergence.Each phase emphasizes a different alignment level.Source: B. Hard Switch · p. 10

C. Linear Schedule

The section contrasts linear and cosine-annealed alignment-weight schedules, with cosine annealing producing the smoothest, most stable transitions. At inference, the music expert is not used.

C. Linear Schedule

  • Linear interpolation gradually adapts alignment emphasis but remains limited in smoothness near transition boundaries.The schedule interpolates weights between phases.Source: C. Linear Schedule · p. 10

D. Cosine Schedule (Ours)

  • Cosine annealing progressively shifts emphasis from top-level alignment to mid-level structure and finally fine-grained rhythm correspondence.Top-level weight decays from 1.0 to 0.2; mid-level weight rises from 0 to 0.6 and then falls to 0.3; bottom-level weight rises to 0.5.Source: D. Cosine Schedule (Ours) · p. 10

E. Visualization

  • The cosine schedule produces the smoothest, most stable transitions; hard-switch and linear schedules transition more sharply, while fixed weights remain constant.Figure 7 compares all four strategies.Source: E. Visualization · p. 10, Figure 7 · p. 11
  • At inference, dance-derived visual and rhythm features condition the generator, and the external music expert is not used.The generator synthesizes a music latent that a pretrained VAE decodes into a waveform, with no additional computational overhead compared with the base diffusion model.Source: E. Visualization · p. 11
  • The representation-similarity analysis seeks correspondences between hierarchical Jukebox features and internal DiT layers.The analysis is used to determine a mapping between expert levels and generator layers.Source: E. Visualization · p. 11

F. Analysis Method

The analysis finds that CMA-OT representations overlap closely with ground-truth music, unlike the baseline, and uses representation similarity to select DiT layers for each expert scale.

F. Analysis Method

  • CKNNA compares representations from all 12 DiT layers at the final denoising step with three Jukebox feature levels on 1,000 clean AIST++ training samples.The analysis uses k = 10.Source: F. Analysis Method · p. 11

G. Analysis Results

  • The CKNNA heatmap shows bottom-level Jukebox features most similar to shallow DiT layers, middle-level features to intermediate layers, and top-level features to deeper layers.The authors interpret this diagonal pattern as a correspondence from fine-grained temporal details to global semantic structures.Source: G. Analysis Results · pp. 11–12, Figure 8 · p. 12

H. Selected Layer

  • The analysis selects DiT Layer 2 for bottom-level alignment, Layer 6 for middle-level alignment, and Layer 10 for top-level alignment.This principled mapping aligns semantically compatible representations at each scale for hierarchical knowledge transfer from the Jukebox expert to the DiT generator.Source: H. Selected Layer · p. 12
  • CMA-OT features form a compact cluster whose mean nearly coincides with ground-truth music, unlike the baseline’s offset distribution.CMA-OT’s confidence ellipse almost fully overlaps with ground truth; the baseline ellipses overlap only slightly.Source: Figure 9 · p. 12, H. Selected Layer · p. 12
Loading 2609.13118v1…