PaperlayerView exact revision on arXiv

Source-linked AI summary

V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation

Yan-Bo Lin, Jonah Casebeer, Long Mai, Aniruddha Mahapatra, Gedas Bertasius, Nicholas J. Bryan

arXiv:2603.11042v2cs.CVcs.AIcs.LGcs.MMcs.SD

TL;DR

Existing text-to-music models lack fine-grained temporal control for synchronizing music with video, while paired video–music training data is commonly required for video-to-music methods. V2M-ZERO uses within-modality event curves to transfer temporal structure from music during training to video at inference, while text controls semantics. Across three datasets, it reports state-of-the-art results without paired video–music data, with gains in audio quality, semantic alignment, temporal synchronization, and beat alignment.

  • Problem

    Existing text-to-music models lack fine-grained temporal control, and many video-to-music methods rely on paired video–music datasets.

  • Method

    V2M-ZERO fine-tunes a text-to-music model on music-event curves computed from intra-modal similarity, then substitutes video-event curves at inference while text controls semantic content.

  • Results

    5–9% higher audio quality, 13–15% better semantic alignment, 21–52% better temporal synchronization, and 28% higher beat alignment on dance videos are reported across three datasets.

  • Takeaways & Limitations

    Within-modality temporal features enable zero-pair video-to-music generation with separate timing and semantic controls.

  • Takeaways & Limitations

    Automatic metrics incompletely capture complex temporal structure, macro-level phrasing, and professional scoring quality.

Abstract

from arXiv · show

Generating music that temporally aligns with video events is challenging for existing text-to-music models, which lack fine-grained temporal control. We introduce V2M-ZERO, a video-to-music generation approach that generates time-aligned music with disentangled time synchronization and semantic control (e.g., genre, mood) from video while requiring zero video-music pairs at training time. Our method is motivated by a key observation: temporal synchronization requires matching when and how much change occurs, not what changes. While musical and visual events differ semantically, they exhibit shared temporal structure that can be captured independently within each modality. We capture this structure through event curves computed from intra-modal similarity using pretrained music and video encoders. By measuring temporal change within each modality independently, these curves provide comparable representations across modalities. This enables a simple training strategy: fine-tune a text-to-music model on music-event curves, then substitute video-event curves at inference without cross-modal training or paired data. Across OES-Pub, MovieGenBench-Music, and AIST++, V2M-ZERO achieves state-of-the-art performance without any paired music-video data, surpassing the strongest prior baselines per metric with 5-9% higher audio quality, 13-15% better semantic alignment, 21-52% improved temporal synchronization, and 28% higher beat alignment on dance videos. We find similar results via a large crowd-source subjective listening test. Our results validate that temporal alignment through within-modality features is not only effective for video-to-music generation but also leads to better performance than paired cross-modal supervision. Furthermore, our approach enables independent controls for timing and music style (e.g., genre, mood) for more controllable generation.

1 Introduction

V2M-ZERO addresses the lack of fine-grained temporal control in text-to-music generation by transferring within-modality temporal structure from music to video without paired training data. It separates timing from semantic control and reports strong gains across quality, alignment, and synchronization metrics.

  • Motivation: Text-to-music models typically lack video-conditioned temporal synchronization, forcing creators to manually edit videos to fit generated music.Existing video-to-music methods often use paired video–music data, while prompt-based approaches capture semantics but not explicit temporal correspondence.
  • V2M-ZERO: V2M-ZERO represents perceptually relevant temporal changes as event curves computed from intra-modal encoder similarity in music and video.The method relies on when and how much change occurs rather than on semantic identity of events.
  • V2M-ZERO: Timing and semantic control are decoupled: event curves specify when musical change occurs, while text conditioning determines how the music sounds.The two controls are designed to operate together, with neither being sufficient alone.
  • V2M-ZERO: V2M-ZERO fine-tunes a pretrained text-to-music model on music-event curves, then swaps in video-event curves at inference without paired video–music data.This test-time transfer is enabled by shared temporal structure across modalities.
  • Results: 5–9% higher music quality, 13–15% better semantic alignment, 21–52% better temporal synchronization, and 28% higher beat alignment on dance videos are reported across three datasets.The authors state that the method surpasses the strongest prior baseline per metric and that crowdsourced listening tests corroborate the gains.

2 Related work

Prior video-to-music work commonly relies on paired internet videos and music, while zero-pair prompt-based methods improve semantic control but remain limited in fine-grained temporal alignment.

  • Text-to-music generation: Text-to-music systems commonly generate audio from text using autoregressive or diffusion-style modeling paradigms.Autoregressive systems predict discrete audio tokens conditioned on prior tokens and text prompts.
  • Paired video-to-music: Recent video-to-music methods use paired internet videos and music tracks to align soundtracks with visual rhythm and motion.Earlier symbolic approaches used MIDI or ABC notation but were limited in scale and expressivity.
  • Zero-pair methods: Prompt-based zero-pair methods translate visual content into text before generating music with pretrained text-to-music models.These methods can capture high-level semantics such as genre and mood.
  • Zero-pair methods: Prompt-based methods struggle with fine-grained temporal alignment because text prompts lack sufficient expressiveness for timing and dynamics.V2M-ZERO instead directly conditions on music or video event curves.

3 Technical approach

V2M-ZERO fine-tunes a pretrained text-conditioned rectified-flow music model with standardized event curves, then transfers from music curves during training to video curves at inference. The curves encode temporal change from pretrained modality-specific features, while text supplies semantic content.

  • Model fine-tuning: The model is fine-tuned from a pretrained text-to-music latent rectified-flow model with temporal event-curve conditioning.Text prompts provide semantic content while event curves provide timing.
  • Rectified flow: The rectified-flow model transports Gaussian noise to the music-data distribution through a learned velocity field conditioned on text and temporal signals.Sampling solves the model’s ODE from t = 1 to t = 0 using 96 steps and classifier-free guidance.
  • Temporal event curves: Event curves measure temporal change from consecutive feature vectors produced by pretrained music or visual encoders.Higher dissimilarity values indicate stronger changes such as beat onsets, scene cuts, motion, or musical-structure changes.
  • Temporal event curves: The procedure standardizes, resamples, and smooths each event curve to reduce modality-specific distribution differences before transfer.Standardization makes the curve a relative temporal-change signal, while smoothing preserves larger structure and suppresses modality-specific details.
  • Model fine-tuning: Event curves are concatenated as an additional channel to the rectified-flow latent, requiring only input-projection parameters in the DiT.The same conditioning mechanism supports event curves computed at different temporal scales.
  • Inference: Music-event curves are used during training, and video-event curves replace them at inference for zero-pair synchronized generation.The model weights remain fixed during this test-time transfer, while an LLM generates a music-appropriate text prompt from video captions.

4 Experimental setup

The evaluation uses diverse video-to-music benchmarks, multiple baselines, and metrics covering audio fidelity, semantic alignment, temporal synchronization, and subjective preference. Results are compared across OES-Pub and MovieGenBench-Music, with additional beat-alignment measures for dance videos.

  • Evaluation protocol: The evaluation follows a standard protocol with multiple baselines and dataset-specific metrics.The protocol measures audio fidelity, semantic alignment, and temporal synchronization, customizing metrics to each task.
  • Evaluation data: Three benchmarks span general, cinematic, and dance videos: OES-Pub, MovieGenBench-Music, and AIST++.OES-Pub contains 115 clips, MovieGenBench-Music 527 pairs, and AIST++ 20 dance video–music pairs across 10 genres.
  • Evaluation metrics: FAD measures audio fidelity, CLAP measures semantic alignment, and scene cut hit measures whether beats occur within ±100 ms of scene cuts.Lower FAD is better, while higher CLAP and scene cut hit scores are better.
  • Comparative evaluation: Table 1 compares methods on OES-Pub and MovieGenBench-Music using FAD, CLAP, and scene cut hit, with V2M-ZERO outperforming competitors across datasets and metrics.Because OES-Pub reference data includes non-musical sounds, SongDescriber is used for the FAD reference distribution.
  • Evaluation metrics: Dance-video evaluation additionally measures beat coverage, beat hit score, F1, and temporal deviation against motion beats and ground-truth tempo.Beat coverage and beat hit score capture recall and precision, F1 is their harmonic mean, and temporal deviation uses a 0.2-second tolerance.

5 Results and analysis

V2M-ZERO performs strongly across benchmarks, with event-curve conditioning delivering high temporal alignment without paired training data. Results also examine human preference, dance-video rhythm metrics, encoder choices, smoothing trade-offs, and model generalization.

  • General results: V2M-ZERO outperforms paired and unpaired baselines across audio fidelity, semantic alignment, and temporal synchronization without paired video-music training data.The analysis highlights cross-domain generalization and the utility of event curves.
  • General and cinematic video: 4.95 FAD* on OES-Pub and 2.68 FAD on MovieGenBench are the best audio-quality results on the two benchmarks.V2M-ZERO is more consistent across datasets, whereas VidMuse varies from 10.4 to 2.98 across the benchmarks.
  • General and cinematic video: 0.61 SCH on OES-Pub and 0.58 SCH on MovieGenBench show stronger temporal alignment than SONIQUE’s 0.27 and 0.21, respectively.The result indicates that explicit event-curve conditioning can surpass paired supervision for fine-grained synchronization.
  • Human evaluation: 1403 human ratings favor V2M-ZERO in music quality and temporal alignment, with the method best in 6/6 temporal-alignment comparisons and significant in 5/6.On scene-cut videos, average win-rate increases and is significant in all 12/12 comparisons.
  • Model generalization: 0.28 to 0.38 SCH improves when video-event curves are substituted into Stable-Audio-ControlNet at inference, with comparable FAD* and CLAP.This supports model-agnostic zero-shot integration of event-curve conditioning.
  • Ablation studies: 31 is selected as the smoothing kernel because larger kernels improve FAD from 8.17 to 3.12 but reduce SCH from 0.61 to 0.27.The ablation identifies a trade-off between audio quality and fine-grained temporal alignment.
  • Ablation studies: 4.52 FAD is achieved by AVSiam, but SCH falls from 0.61 to 0.35, indicating a quality-alignment trade-off for shared encoders.The authors hypothesize that larger shared encoders could reduce this gap.

6 Conclusions

V2M-ZERO generates video-conditioned music by transferring temporal structure between modalities without paired video–music training data. It achieves strong alignment and quality across multiple benchmarks, supported by human evaluation.

  • V2M-ZERO uses intra-modal event curves to transfer temporal structure from video to music without paired supervision.The method fine-tunes text-to-music models on music-event curves and substitutes video-event curves at inference.
  • V2M-ZERO provides disentangled control over music timing and semantic attributes from video.The paper identifies timing and semantics as separately controllable aspects of generation.
  • V2M-ZERO achieves state-of-the-art results on OES-Pub, MovieGenBench-Music, and AIST++ without paired video–music data.It outperforms paired-data methods in audio quality, semantic alignment, and temporal synchronization.
  • Direct human evaluation reinforces the reported performance of V2M-ZERO.

Appendix overview

The appendix covers limitations and broader impact, implementation details, additional quantitative results, and qualitative event-curve results.

  • The appendix includes a section on limitations and broader impact.
  • The appendix provides implementation details for V2M-ZERO.
  • The appendix reports additional quantitative results.
  • The appendix presents qualitative results on event curves.

A Limitations and broader impact

The paper notes that current automatic metrics incompletely evaluate video-to-music generation, while describing potential accessibility benefits and broader social risks.

  • Limitations: Current automatic metrics assess audio quality, coarse semantic alignment, and local synchronization but omit complex temporal structure, macro-level phrasing, and professional scoring quality.
  • Limitations: Because longer-term musical improvements cannot yet be fairly evaluated, the main system focuses on reliably measurable aspects.
  • Broader impact: V2M-ZERO may reduce effort and improve accessibility for creating time-aligned background music.
  • Broader impact: Commercial systems may benefit because licensed video–music pairs are difficult and expensive to obtain, whereas licensed text–music data and silent videos are more feasible.
  • Broader impact: Potential negative impacts include increased media consumption and screen time, with concerns about digital addiction and social isolation.

B Implementation details

The implementation combines pretrained encoders, lightweight text-to-music fine-tuning, multimodal prompt generation, and scene-cut evaluation to condition music generation on video.

  • Feature encoder: The system uses MusicFM and DINOv2-L for default music and video feature extraction, with CoTracker reported for dance videos.
  • Training: Fine-tuning adds event-curve conditioning to a pretrained text-to-music model using approximately 25k hours of licensed instrumental music-text pairs.The procedure uses 32-second clips and requires 192–768 GPU hours.
  • Text-only baseline: Table 5 compares V2M-ZERO with the same text-to-music backbone without event-curve conditioning on OES-Pub.
  • Scene cut hit: The SCH metric measures whether generated music contains an onset near each detected scene cut.The implementation uses a temporal tolerance of ±0.1 seconds and computes the ratio of matched cuts to all scene cuts.
  • Music prompt generation: Music prompts are generated by combining an ASR transcript with aggregated frame-level visual descriptions.The multimodal process produces prompts describing mood, instrumentation, intensity, and emotional character.

C Additional quantitative results.

Additional analyses show that event-curve conditioning improves synchronization while preserving semantic alignment, but curve-based distances do not consistently predict human judgments. The results also identify encoder choice, temporal scale, prompting granularity, and benchmark design as important evaluation factors.

  • CoTracker improves all reported AIST++ metrics over DINOv2, whose baseline scores are BCS 0.5522, BHS 0.5748, F1 0.5750, and TD 17.23.The passage presents inference-time encoder selection as a source of performance gains for dance videos.
  • SCH rises from 0.35 to 0.61 with event-curve conditioning, while CLAP remains 0.23 for both the conditioned model and text-only baseline.The comparison attributes the synchronization gain to explicit temporal structure while reporting similar music quality.
  • Human temporal-alignment preferences do not strongly correlate with event-curve Fréchet distances, so low distributional distance does not guarantee human preference.The analysis distinguishes event curves as useful generative intermediates from their suitability as evaluation metrics.
  • OES-Pub and MovieGenBench-Music are not specifically curated for temporal synchronization, and many clips lack strong or consistent rhythmic structure.This limits the signal available to curve-based metrics.
  • DINOv2 features remain similar after non-semantic frame perturbations, with cosine similarity µ = 0.983 and σ = 0.025.The tested perturbations include translation, rotation, brightness, and gamma changes.
  • Video-event curves support 68.2% accuracy in a three-way cinematic, natural, and dance video classification task.A 1-layer MLP evaluated the curves using a 90%/10% train–test split.
  • At 30 fps, offsets 1, 15, and 60 represent approximately 33 ms, 500 ms, and 2 s; offset 1 yields the strongest measured SCH synchronization.Longer and multiple offsets preserve FAD and CLAP, but their longer-term dependencies lack an appropriate evaluation metric.
  • Chunked 5 s prompting is closer to V2M than whole-video prompting, with cosine similarity 0.86 versus 0.63, suggesting time-varying captions may capture macro-level phrasing.The comparison uses MusicFM features averaged across 5 s music chunks.

D Qualitative results on event curves

Event curves encode relative temporal dynamics, while text supplies complementary semantic guidance such as content, energy, and mood. Qualitative examples show that normalized curves retain distinct timing patterns and precise salient-event localization despite smoothing.

  • Event curves represent when changes occur, while text provides complementary semantic information including content, energy, and mood.Normalization makes the curves encode relative temporal structure rather than absolute scale.
  • Videos with frequent scene cuts and slower camera motion produce visibly distinct normalized temporal patterns.Figure 4 uses blue and orange curves to illustrate these different dynamics.
  • Hann-window smoothing suppresses high-frequency noise while preserving precise localization of salient event peaks.The equivalent bandwidth is roughly 21 samples at 30 fps, or about 700 ms.
Loading 2603.11042v2…