Source-linked AI summary
MVWeaver: A Hierarchical Music Video Generation Agent with a Learned Song-to-Visual Bridge
Sifei Li, Minyan Luo, Xu Li, Guodong Qi, Xincan Wang, Hanwen Wang, Chen Zhang, Pengfei Wan, Oliver Deussen, Weiming Dong
TL;DR
Automatic music video systems can produce plausible shots but still face challenges in long-form coherence and song-grounded visual development. MVWeaver learns song-to-visual bridges from real-MV-derived annotations and uses them to guide hierarchical planning; experiments report improvements in grounded translation, visual development, and coherence.
Problem
Automatic music video generation can produce visually plausible shots but continues to face challenges in long-form coherence and song-grounded visual development.
Source: §1 · pp. 1–2Method
MVWeaver learns global and local song-to-visual bridge decisions from real-MV-derived rationales and uses them to condition hierarchical visual planning.
Source: §1 · p. 2, §2.2 · p. 3Results
Takeaways & Limitations
The paper frames MV planning as developing plausible visual associations from song evidence into a coherent plan, rather than merely illustrating the song.
Source: §1 · p. 1More explicit modeling of cross-shot camera-motion continuity is left for future work incorporating professional editing principles.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Existing automatic music video (MV) generation systems can generate visually plausible shots, yet often struggle with long-form coherence and song-grounded visual development. We present MVWeaver, a music video generation agent that integrates hierarchical planning with a learned song-to-visual bridge that translates song understanding into executable shot plans. The MVWeaver architecture comprises a comprehensive song analysis module, a visual planner that constructs hierarchical plans, and downstream image and video generation models that render the planned content. To equip a general-purpose LLM with MV-specific song-to-visual knowledge, we learn a bridge between song analysis and visual planning from real-MV-derived supervision and curate 1,861 real-world song--MV pairs with structured song-side, MV-side, and teacher-inferred song-to-visual rationale annotations. Using these annotations, we perform LoRA-based supervised fine-tuning (SFT) of a large language model to predict song-to-visual bridges that guide hierarchical visual planning. Our experiments demonstrate stronger song-grounded visual translation, richer visual development, and greater conceptual and shot-to-shot coherence, while ablations support the benefits of learned bridge conditioning.
1 Introduction
MV planning is framed as developing plausible visual associations from song evidence into a coherent plan, rather than merely illustrating a song. Existing systems address multi-shot production, music-aware editing, and song-to-MV generation, but the paper identifies a gap in MV-specific song-to-visual knowledge for planning.
1 Introduction
- Professional MVs extend a song through associated situations, motifs, and visual metaphors, motivating plans that develop song-grounded associations into coherent visual concepts.The task is to analyze a song, develop a visual concept, and generate a full-length video that is musically aligned and visually consistent.Source: §1 · p. 1
- Automatic MV work spans agentic multi-shot production, music-guided editing of existing footage, and direct generation of MVs from songs.Examples include AniME [Zhang et al. 2025], AniMaker [Shi et al. 2025], VISTA [Long et al. 2026], and STAGE [Zhang et al. 2026]; GLANCE [Lin et al. 2026] and BEAT [Wang et al. 2026c]; YingVideo-MV [Chen et al. 2025], AutoMV [Tang et al. 2025], and AllocMV [Wang et al. 2026b].Source: §1 · p. 1
- Existing visual planning relies largely on general-purpose LLMs rather than knowledge distilled from real-MV song-to-visual associations, while creator-facing platforms rely heavily on user materials and guidance.The cited platforms are VidMuse and Seko.Source: §1 · p. 2
- MVWeaver couples hierarchical production with a learned song-to-visual bridge, using teacher-inferred rationales from real MVs to connect song evidence to visual choices.The paper reports a dataset of 1,861 real-world song–MV pairs and improvements in song-grounded translation, visual development, and long-range conceptual and shot-level coherence.Source: §1 · p. 2
2 Method
MVWeaver analyzes songs, predicts global and local song-to-visual bridge decisions, and expands them through hierarchical planning into renderable shots. Its bridge is learned from real-MV annotations and guides the planning stages without supplying song evidence or association chains in planner prompts.
2.1 Hierarchical Music Video Agent
- Song analysis combines an eight-dimensional whole-song profile with timestamped lyrics, musical sections, and section-level normalized, smoothed log-RMS energy.Beats, downbeats, and strong onsets are combined into rhythmic candidates that provide temporal anchors for scene and shot boundaries.Source: §2.1 · p. 2
- The planner progressively expands bridge decisions through an MV brief, visual developments, scenes, and shots, carrying reusable assets and aligning scene and shot boundaries to rhythmic candidates.Global decisions concern the overall MV concept; ordered local decisions concern visual developments.Source: §2.1 · p. 2
- Reference-conditioned image and video generators render reusable assets and first frames, then animate non-singing and singing shots through separate generators.GPT Image 2 renders assets and first frames; Kling v3 animates non-singing shots, while Kling Avatar generates singing shots from first frames and vocal segments.Source: §2.1 · p. 2, Figure 1 · p. 3
2.2 Real-MV-Derived Song-to-Visual Bridge Dataset
- The bridge dataset pairs songs with real MVs and represents each MV through an overall visual design and temporally organized visual developments.Its global bridge covers six design dimensions, while each local development records song evidence, visual progression, and an association chain.Source: §2.2 · pp. 2–3
- Teacher-inferred five-stage association chains connect song evidence to observed MV decisions through conceptual abstraction, cross-domain mapping, and visual concept formation.This process supervision is intended to encourage transferable song-to-visual relations rather than reproduction of isolated decisions from individual MVs.Source: §2.2 · p. 3
2.3 Bridge Learning and Agent Integration
- LoRA-based supervised fine-tuning trains Qwen3.6-27B on song-side information to jointly predict global and local bridges retrospectively inferred from paired real MVs.Joint prediction encourages local developments to remain compatible with the global bridge concept; at inference, song evidence and association chains are not appended to planner prompts.Source: §2.3 · p. 3
3 Experiments
Experiments compare MVWeaver with AutoMV and a no-bridge ablation on song-grounded planning, long-form coherence, and generated-MV preferences. Results favor MVWeaver across these evaluations, while qualitative comparisons illustrate differences in visual development and bridge outputs.
3 Experiments
- The evaluation scores plans for 20 test songs against AutoMV and w/o Bridge using blinded five-point ratings across song alignment, visual development, and long-form coherence.Higher scores are better; the study uses ISA, GDE, CC, and SSC.Source: §3 · pp. 3–4
- With identical song-side inputs, full SFT translates the song’s cues into protagonist-controlled magical actions, unlike abstract or surface-level bridge outputs.The figure shows only each global bridge output’s MV summary.Source: Figure 2 · p. 4
- MVWeaver consistently outperforms AutoMV on song-grounded translation and long-form planning coherence, and improves ISA, GDE, and SSC over w/o Bridge, with a smaller CC gain.The shared hierarchical planner and reusable assets may account for the smaller CC gain relative to the bridge ablation.Source: §3 · p. 4, Table 1 · p. 4
- AutoMV often uses generic situations and drifts across settings, whereas MVWeaver develops song-specific premises with reusable locations and planned montage transitions.The qualitative comparison concerns final MVs; the supplementary video compares MVWeaver with AutoMV.Source: §3 · p. 4
- MVWeaver receives higher preference rates across song interpretation, long-range coherence, and overall effectiveness in a blind A/B study of final generated MVs.The study compared ten randomly selected pairs, produced 30 questions, and collected votes from 34 participants.Source: §3 · p. 4, Table 1 · p. 4
4 Conclusion
MVWeaver combines hierarchical visual planning for full-song music videos with a learned bridge from song analysis to visual planning. The paper reports improved song-grounded visual development and coherence, and identifies explicit cross-shot camera-motion continuity as future work.
4 Conclusion
- MVWeaver combines long-form visual planning with a learned song-to-visual bridge for full-song music-video generation.The bridge model predicts global and local decisions from song-side information.Source: §4 · p. 4
- The authors report that bridge supervision improves song-grounded visual development and coherence across the generated sequence.Source: §4 · p. 4
- Future work will incorporate professional editing principles to model cross-shot camera-motion continuity more explicitly.Source: §4 · p. 4