PaperlayerView exact revision on arXiv

Source-linked AI summary

Where the Body Keeps the Beat: Structured Motion Conditioning and Music Dynamics Supervision for Dance-to-Music Generation

Changchang Sun, Lu Cheng, Yan Yan

arXiv:2610.00726v1cs.SD

TL;DR

D2M methods do not explicitly separate motion cues across body parts and frequency bands, while standard flow matching lacks dedicated supervision for local music-latent dynamics. Dyna2Music combines structured motion conditioning with latent-dynamics consistency; experiments on AIST++ and TikTok report improved rhythmic alignment and audio quality over representative baselines.

  • Problem

    Representative D2M methods do not explicitly distinguish motion cues by body part or frequency, and standard flow matching lacks a dedicated objective for supervising local music-latent dynamics.

    Source: §1 · p. 1

  • Method

    Dyna2Music hierarchically fuses part-wise slow-fast motion energy with pretrained joint features and uses LDC to match adjacent-frame change magnitudes between an estimated clean latent and its paired reference.

    Source: §1 · pp. 2–3

  • Results

    Experiments on AIST++ and TikTok report improved rhythmic alignment and audio quality over representative D2M baselines, with higher onset synchrony and lower FAD than evaluated baselines.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Dyna2Music supports variable-length generation, and its experiments show consistent BAS advantages across motion-reference configurations.

    Source: §1 · p. 3, §5 · p. 9

    Shifting motion conditions by about a quarter or half beat leaves onset peaks near their original phase, consistent with a clip-onset phase prior and limiting claims of local motion-dependent phase tracking.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Dance-to-music (D2M) generation aims to synthesize musically plausible soundtracks whose temporal structure aligns with a given dance performance. Representative D2M methods do not explicitly distinguish motion cues across body parts and frequency bands, while standard flow matching lacks a dedicated objective for supervising local music-latent dynamics. To address these issues, we propose Dyna2Music, a latent flow-matching framework that combines structured motion conditioning with explicit supervision of music-latent dynamics. An empirical analysis on AIST++ quantifies how spatial partitioning and frequency separation affect raw music-to-kinematic beat alignment and motion-reference density, informing the conditioning design. Accordingly, Dyna2Music decomposes joint velocities into slow and fast components and hierarchically fuses the resulting part-wise motion energy with pretrained joint features to condition music generation. To complement this representation, we introduce latent dynamics consistency (LDC), an auxiliary objective that matches adjacent-frame change magnitudes between a single-step clean-latent estimate and the paired reference. LDC makes local music-latent variation an explicit training target without adding trainable parameters or inference computation. Dyna2Music supports variable-length music generation, and experiments on AIST++ and TikTok demonstrate improved rhythmic alignment and audio quality over representative D2M baselines.

1 INTRODUCTION

D2M must align soundtrack structure with dance dynamics, but existing methods do not explicitly separate motion cues by body part or frequency, and standard flow matching does not directly supervise local music-latent dynamics. Dyna2Music combines structured motion conditioning with latent-dynamics supervision; experiments report improved rhythmic alignment and audio quality over representative baselines.

1 INTRODUCTION

  • Existing D2M conditioning does not explicitly distinguish body-part and frequency cues, which can obscure localized events such as a wrist accent alongside torso motion.D2M also needs soundtracks to respond to sustained motion, brief limb articulations, and rhythmic accents.Source: §1 · p. 1
  • Five-part references score higher raw BAS than coarser references, and slow-fast separation further raises the score, while also increasing motion-reference density.With no frequency separation, BAS rises from 0.307 for whole-body to 0.569 for five-part references; with two bands, the five-part score is 0.661.Source: Figure 1 · p. 2, §1 · p. 2
  • Dyna2Music decomposes joint velocities into slow and fast energy for five body parts, then fuses these signals with pretrained MotionBERT features to condition music generation.LDC matches adjacent-frame change magnitudes between a single-step clean-latent estimate and the paired music latent, adding no trainable parameters or inference computation.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Experiments on AIST++ and TikTok report improved rhythmic alignment and audio quality over representative D2M baselines, and the framework supports variable-length generation.The empirical motion-reference analysis measures raw music-to-kinematic beat alignment and beat density on AIST++.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans general video-to-music generation and two D2M families: autoregressive methods that decode sequentially, and non-autoregressive methods that use adversarial synthesis or iterative refinement. Dyna2Music is positioned among non-autoregressive approaches as a latent flow-matching method with structured motion conditioning and explicit supervision of local music-latent dynamics.

General Video-to-Music Generation

  • General video-to-music methods address semantic compatibility, temporal alignment, or controllability using visual features, musical controls, and multimodal or language-model interfaces.Examples include CMT (Di et al., 2021), VidMuse (Tian et al., 2025), GVMGen (Zuo et al., 2025), and MTCV2M (Wu et al., 2025).Source: §2 · pp. 3–4

Autoregressive Dance-to-Music Generation

  • Autoregressive D2M methods predict symbolic events or discrete audio tokens sequentially from motion and prior outputs, retaining dependencies between output time steps.Examples include Dance2Music (Aggarwal & Parikh, 2021), RhythmicNet, and DanceComposer.Source: §2 · p. 4

Non-autoregressive Dance-to-Music Generation

  • Non-autoregressive D2M methods use adversarial synthesis or iterative refinement; Dyna2Music combines part-specific slow-fast conditioning with explicit LDC supervision in latent flow matching.The passage also describes diffusion-based approaches including CDCD (Zhu et al., 2022b), LORIS (Yu et al., 2023), and PN-Diffusion (Sun et al., 2025).Source: §2 · p. 4

3 METHODOLOGY

Dyna2Music encodes paired music in a frozen VAE latent space and conditions generation on dance motion. Its pipeline decomposes motion into part-wise slow and fast signals, hierarchically fuses those signals with joint features, and trains a conditional flow-matching model with an auxiliary local-dynamics objective.

3.1 PROBLEM FORMULATION

  • The task is formulated as learning a conditional distribution over music latents given dance joint positions, with generated latents decoded into waveforms by the frozen VAE.The music latent has T frames and dimensionality da.Source: §3.1 · p. 4

3.2 SPATIO-SPECTRAL MOTION DECOMPOSITION

  • SSMD computes per-part motion energy by filtering joint velocities into slow and fast components before averaging squared speed within each body part.Slow and fast refer to temporal frequency of velocity variation; frozen MotionBERT features are pooled over the same parts and resampled to music-latent frames.Source: §3.2 · pp. 4–5

3.3 HIERARCHICAL SLOW-FAST CONDITIONING ENCODER

  • At each latent frame, band-specific energy embeddings query joint-feature context across five body parts, and energy-dependent gates aggregate attention outputs within each band.The encoder retains part and band tokens during context integration.Source: §3.3 · p. 5
  • A learned scalar gate mixes slow and fast representations, while the final projection also receives their normalized concatenation to form conditioning tokens.The resulting sequence conditions the music generator.Source: §3.3 · p. 5

3.4 LATENT FLOW MATCHING WITH DYNAMICS SUPERVISION

  • The conditional flow-matching model interpolates Gaussian noise and the paired music latent, training a transformer to predict the target velocity.The transformer predicts velocity conditioned on the HSF sequence.Source: §3.4 · pp. 5–6
  • LDC matches adjacent-frame change magnitudes between a single-step clean-latent estimate and the paired reference, complementing velocity regression without constraining change direction or beat locations.It adds neither trainable parameters nor inference computation.Source: §3.4 · p. 6
  • At inference, the model integrates the conditional velocity field from Gaussian noise and decodes the resulting latents; sequence length follows dance duration within the configured limit.Integration uses Euler updates and a frozen VAE decoder.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate Dyna2Music on controlled-studio AIST++ and in-the-wild TikTok data against video-to-music and dance-to-music baselines. The results cover onset synchrony, audio quality, beat alignment across motion references, beat-relative rhythm, and cross-dance output variation.

4.1 DATASETS

  • AIST++ contributes 8,819 training and 180 test pairs from controlled studio performances; TikTok contributes 391 training and 53 test pairs from in-the-wild videos.The two benchmarks provide different recording conditions for evaluating rhythmic alignment and audio quality.Source: §4.1 · p. 6

4.2 BASELINES AND EVALUATION PROTOCOLS

  • Comparisons include general video-soundtracking and D2M baselines; an AIST++ user study assesses overall musical quality and dance-music relevance.The subjective-study protocol and results are in Appendix E; implementation uses frozen MotionBERT and waveform VAE components.Source: §4.2 · pp. 6–7

4.3 MODEL COMPARISON

  • F1 reaches 55.76 on AIST++ and 29.99 on TikTok, exceeding the strongest baselines by 17.70 and 4.36 percentage points, respectively.Dyna2Music also achieves the highest BCS, BHS, and F1 on both datasets; its CSD/HSD are not the lowest.Source: Table 1 · p. 7, §4.3 · p. 7
  • Dyna2Music obtains the lowest FAD in all three feature spaces on both datasets, supporting closer agreement with reference music distributions.The strongest competing method varies by feature extractor.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 BEAT ALIGNMENT ACROSS MOTION GRANULARITIES

  • With the five-part, two-band reference, BAS is 0.684 on AIST++ and 0.654 on TikTok, 0.023 above the strongest baseline on each.Both scores remain below paired ground-truth music, and Dyna2Music ranks first under four additional reference configurations.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 BEAT-RELATIVE RHYTHM ANALYSIS

  • Dyna2Music shows more pronounced reference-beat onset peaks than representative baselines, but quarter- or half-beat condition shifts leave peaks near their original phase.The observed phase behavior is consistent with a clip-onset phase prior and limits claims of local motion-dependent phase tracking.Source: §4.5 · p. 8, Figure 4 · p. 9

4.6 OUTPUT DIVERSITY ACROSS DANCES

  • Mean cross-dance CLAP similarity is 0.58 versus GT's 0.62 on AIST++ and 0.47 versus 0.66 on TikTok; intervals overlap only on AIST++.The statistic measures variation across inputs, not repeated-sample diversity for a fixed dance or novelty relative to training music.Source: §4.6 · p. 8, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents Dyna2Music as a variable-length D2M framework combining structured motion cues with local music-latent dynamics supervision. It reports higher onset synchrony and lower FAD than evaluated baselines, and identifies finer phase control and long-range musical structure as future work.

5 CONCLUSION

  • Dyna2Music combines hierarchical fusion of part-specific slow-fast motion cues and pretrained joint features with latent dynamics consistency to supervise local music-latent variation.The conclusion reports higher onset-level synchrony and lower FAD than evaluated baselines, with consistent BAS advantages across motion-reference configurations.Source: §5 · p. 9
  • Future work will address finer motion-dependent phase control and long-range musical structure.Source: §5 · p. 9

APPENDIX

  • The appendices cover implementation, evaluation protocols and BAS comparisons, LDC-weight sensitivity, the user study, limitations, and broader impact.Source: APPENDIX · p. 13

A IMPLEMENTATION DETAILS OF DYNA2MUSIC

Motion preprocessing computes slow and fast energy for five body parts, then aligns those signals and pooled joint features to the music-latent timeline.

A.1 MOTION PREPROCESSING AND SSMD

  • Joint velocities are split into low- and high-frequency components, and mean squared speed is computed separately for each of five body parts.The parts are torso, left and right arms, and left and right legs; velocities use consecutive pose differences scaled by video frame rate.Source: §A.1 · p. 13
  • Energy signals are resampled to the target latent length, while pose-only inference derives that length from video duration and frame rate.Energy is capped per clip, part, and band at the 99.5th percentile, then clamped to nonnegative values.Source: §A.1 · p. 13
  • Frozen MotionBERT features are averaged within each part and Fourier-resampled to align with the slow and fast energy sequences.The resulting joint features have shape T × 5 × 512; energy sequences have shape T × 5.Source: §A.1 · p. 13

A.2 HSF-ENCODER ARCHITECTURE

The appendix specifies how HSF motion tokens are formed and describes the music transformer and supported sequence lengths.

A.2 HSF-ENCODER ARCHITECTURE

  • The HSF encoder standardizes energy per sample, part, and band, then uses band-specific queries, part gating, and a learned fusion gate to combine motion signals.Energy inputs use two-layer MLPs; part weights are softmax-normalized, and the final projection maps 768 dimensions to 256.Source: §A.2 · pp. 13–14

A.3 MUSIC BACKBONE AND SEQUENCE LENGTHS

  • The music backbone is a 16-block noncausal transformer with cross-attention to HSF tokens, and inference supports variable durations up to a configurable cap.The default cap is 1200 latent frames; training caps are 1200 for AIST++ and 700 for TikTok, and five TikTok training pairs fall below the 100-frame minimum.Source: §A.3 · p. 14

A.4 TRAINING OBJECTIVE AND SAMPLING

Training combines flow-matching velocity regression with latent-dynamics consistency, while generation integrates the learned velocity field to produce audio latents.

A.4 TRAINING OBJECTIVE AND SAMPLING

  • The flow-matching loss averages squared velocity error over valid sample-frame pairs and latent channels, with flow times sampled independently per sample.A shared minibatch decision zeros HSF tokens with probability 0.2 for classifier-free guidance training.Source: §A.4 · p. 14
  • LDC reuses the flow-matching velocity prediction to compare adjacent-frame latent changes, excluding padded pairs and detaching the reference latent.Its computation remains in latent space and requires neither beat annotations nor waveform decoding.Source: §A.4 · p. 14
  • Generation uses Euler integration from Gaussian noise, with optional classifier-free guidance and 31 updates before the frozen VAE decodes the endpoint.The scripts set guidance strength to γ = 0, and neither LDC nor the VAE encoder runs during sampling.Source: §A.4 · pp. 14–15

B EVALUATION DETAILS

The evaluation defines onset synchrony and beat-relative rhythm comparisons, then tests cross-dance similarity and alternative motion-reference granularities.

B.1 ONSET-LEVEL SYNCHRONY

  • Onset synchrony uses one-to-one onset matching within ±50 ms, reporting clip-averaged BCS, BHS, F1, CSD, and HSD as percentages.These scores alone do not establish dance-specific rhythmic alignment because chance matches depend on onset density.Source: §B.1 · p. 15

B.2 BEAT-RELATIVE ONSET STRUCTURE AND PERIODICITY

  • Dyna2Music has the largest generation-method contrasts for on-beat onset strength (0.379; 95% CI [0.256, 0.487]) and periodicity (0.272; [0.232, 0.315]).The two measures offer distinct evidence: CMT has positive periodicity contrast but an on-beat interval containing zero.Source: §B.2 · p. 15, Figure A1 · p. 16

B.3 CROSS-DANCE OUTPUT SIMILARITY

  • Cross-dance output similarity averages cosine similarity between clip representations, with uncertainty bootstrapped over music tracks or source videos.The statistic describes variation across inputs, not repeated-sample diversity for a fixed dance or novelty relative to training music.Source: §B.3 · p. 16

C MOTION GRANULARITY ANALYSES

C.1 ABLATION OF MOTION CONDITIONING

The conditioning ablation favors retaining both five-part spatial and two-band frequency distinctions, although the gains depend on combining them and are not isolated from training-budget differences.

C.1 ABLATION OF MOTION CONDITIONING

  • The full five-part, two-band model achieves the highest BCS, BHS, and F1 on both AIST++ and TikTok.The table reports onset-level metrics for both datasets.Source: Table A1 · p. 17, §C.1 · p. 18
  • Merging frequency bands at five-part resolution reduces F1 by 8.38 points on AIST++ and 9.54 on TikTok.Merging five parts into upper/lower body while retaining both bands reduces F1 by 12.18 and 9.98 points, respectively.Source: §C.1 · p. 18
  • Band separation at two-part resolution does not improve F1, and finer partitioning alone gives no consistent gains among one-band variants.Differing training budgets limit attribution of the reported benefits to conditioning structure alone.Source: §C.1 · p. 18

D SENSITIVITY TO THE LDC WEIGHT

The LDC-weight sweep shows that the weight maximizing onset-level F1 differs from those minimizing FADc, so stronger LDC supervision does not improve synchrony monotonically.

D SENSITIVITY TO THE LDC WEIGHT

E SUBJECTIVE EVALUATION

In an AIST++ user study, Dyna2Music receives higher overall musical-quality and dance-music-relevance ratings than the selected baselines.

E SUBJECTIVE EVALUATION

  • Ten participants each completed 10 randomized, blinded trials comparing soundtracks from Dyna2Music, AudioLDM-ControlNet, and SONIQUE for the same dance video.This yielded 100 ratings per method for each dimension.Source: E SUBJECTIVE EVALUATION · p. 18
  • Participants rated overall musical quality (OVL) and dance-music relevance (REL) on a five-point scale, with higher scores indicating better perceived quality or relevance.REL assesses perceived rhythmic correspondence with the dance.Source: E SUBJECTIVE EVALUATION · p. 18, Table A2 · p. 19
  • Dyna2Music’s mean OVL of 4.3 and REL of 4.5 exceed the highest selected-baseline scores by 1.0 and 1.1 points, respectively.AudioLDM-ControlNet has the highest baseline OVL, and SONIQUE the highest baseline REL.Source: E SUBJECTIVE EVALUATION · p. 18, Table A2 · p. 19
Loading 2610.00726v1…