PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoMV: An Automatic Multi-Agent System for Music Video Generation

Xiaoxuan Tang, Xinping Lei, Chaoran Zhu, Shiyun Chen, Ruibin Yuan, Yizhi Li, Changjae Oh, Ge Zhang, Wenhao Huang, Emmanouil Benetos, Yang Liu, Jiaheng Liu, Yinghao Ma

arXiv:2512.12196v1cs.MMcs.CVcs.SDeess.AS

TL;DR

Full-song music-to-video generation remains challenging because existing methods produce short, disconnected clips that may miss musical alignment and temporal consistency. AutoMV combines music-aware preprocessing with multi-agent planning, generation, and verification, and the authors report stronger results than commercial systems across multiple dimensions while narrowing the gap to professional MVs.

  • Problem

    Existing music-to-video approaches often produce short, disconnected clips that fail to align with musical structure and lack temporal consistency.

    Source: §1 · p. 4

  • Method

    AutoMV combines music-aware preprocessing, agent-led script planning and video generation, and verifier-guided refinement for full-length music videos.

    Source: §1 · p. 5

  • Results

    AutoMV outperforms strong commercial systems across semantic, technical, editorial, and artistic dimensions, significantly narrowing the gap to professional MVs.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Expert evaluation confirms AutoMV’s gains, while LLM-based evaluation offers a scalable complement to human judgment.

    Source: §6 · p. 13

    AutoMV cannot replace human creators in the short term, and its quality remains below large commercial productions.

    Source: Ethics and Social Impact · p. 14

Abstract

from arXiv · show

Music-to-Video (M2V) generation for full-length songs faces significant challenges. Existing methods produce short, disjointed clips, failing to align visuals with musical structure, beats, or lyrics, and lack temporal consistency. We propose AutoMV, a multi-agent system that generates full music videos (MVs) directly from a song. AutoMV first applies music processing tools to extract musical attributes, such as structure, vocal tracks, and time-aligned lyrics, and constructs these features as contextual inputs for following agents. The screenwriter Agent and director Agent then use this information to design short script, define character profiles in a shared external bank, and specify camera instructions. Subsequently, these agents call the image generator for keyframes and different video generators for "story" or "singer" scenes. A Verifier Agent evaluates their output, enabling multi-agent collaboration to produce a coherent longform MV. To evaluate M2V generation, we further propose a benchmark with four high-level categories (Music Content, Technical, Post-production, Art) and twelve ine-grained criteria. This benchmark was applied to compare commercial products, AutoMV, and human-directed MVs with expert human raters: AutoMV outperforms current baselines significantly across all four categories, narrowing the gap to professional MVs. Finally, we investigate using large multimodal models as automatic MV judges; while promising, they still lag behind human expert, highlighting room for future work.

1. Introduction

Full-song music-to-video generation remains underexplored, while existing approaches struggle with musical alignment, long-term coherence, and evaluation. AutoMV combines music-aware planning and multi-agent generation with a benchmark intended to assess long-form music videos.

1. Introduction

  • M2V has received less attention than other AIGC tasks, despite music-video production involving substantial labor, time, and expense.The introduction frames automated generation as a way to lower production barriers for independent musicians and low-budget creators.Source: §1 · p. 4, Figure 2 · p. 5
  • Full-song M2V methods often produce short, disjointed clips that miss musical structure and beats, lack temporal consistency, and are difficult to evaluate with existing metrics.The stated evaluation gap includes music-video alignment, musicality, and creativity.Source: §1 · p. 4
  • AutoMV uses music information retrieval to extract beats, song structure, and time-stamped lyrics, which inform a visual script and shared character profiles.A verifier assesses clip alignment and physical feasibility; the system also uses specialized APIs for visual creation and lip-syncing.Source: §1 · pp. 4–5
  • The paper introduces an open-source multi-agent workflow, an M2V evaluation benchmark with LLM judges, and ablations of music analysis, character consistency, and verification components.The benchmark evaluates long-form music video generation; the listed ablations quantify the impact of the MIR tools, character bank, and Verifier agent.Source: §1 · p. 5

2. Related Work

Prior work advances long-form text-to-video generation and audio-visual alignment, but the paper identifies full-song, story-driven music-to-video generation as a distinct challenge. AutoMV targets song-level visual alignment with audio structure, beats, and lyrics.

2.1. Video generation

  • Text-to-video systems improve long-form consistency and multi-shot planning, but typically remain limited to short clips or lack the thematic continuity needed for full-length stories.Related work also describes multi-agent and LLM-guided frameworks for planning and refining long-form video.Source: §2.1 · pp. 5–6
  • AutoMV differs from text-driven systems by targeting long, song-level generation aligned to audio structure, beats, and lyrics.The paper positions these audio-specific requirements as distinct from existing text-driven video generation.Source: §2.1 · p. 6

2.2. Audio-visual alignment

  • Prior music-video generation systems address scripting, assembly, or beat synchronization, but the paper describes generated visuals as still challenging to produce directly from audio and often short or disconnected.Other audio-visual work instead generates music from video, using visual cues to guide musical attributes or soundtracks.Source: §2.2 · p. 6
  • Multi-agent strategies have also been applied to audio generation and long-form video dubbing, including workflows that decompose dubbing into scenes, scripts, sound design, and synthesis.The cited LVAS-Agent workflow uses discussion, correction, and retrieval to maintain temporal coherence across scenes.Source: §2.2 · pp. 6–7

3. Methodology

AutoMV turns a full song into a long-form video through music-aware preprocessing, shot-level script planning, per-shot video generation, and verifier-guided selection. Its planning uses song structure and timestamped lyrics to shape scenes and character continuity.

3. Methodology

  • AutoMV’s four stages are music-aware preprocessing, Director-led shot planning, per-shot rendering with selected video or lip-sync APIs, and verification for alignment and physical feasibility.The Verifier can prompt regeneration or fallback when needed.Source: §3 · p. 7, Figure 3 · p. 7

3.1. Preprocessing on music informatics

  • Music preprocessing extracts song structure, vocals, lyrics, and captions, providing contextual information for later generation stages.The described tools include SongFormer for song sections, htdemucs for vocal separation, and Whisper plus Gemini for lyric transcription and timestamp refinement.Source: §3.1 · pp. 7–8

3.2. Screenwriter and director

  • The screenwriter segments songs at lyric boundaries and section transitions into user-controllable 3–15 second shots, with clip times quantized to 1/24 second.These timing constraints ensure audio–video length parity and prevent accumulation drift.Source: §3.2 · p. 8
  • For each segment, the screenwriter derives scene descriptions from lyric meaning, themes, and emotional tone; a character bank supplies appearance details to stabilize identity across shots.The Director produces shot-level scripts and keyframe prompts, retrieving matching character descriptions from the bank.Source: §3.2 · p. 8

3.3. Video generation

  • The Director selects video backends per shot, using Doubao for general cinematic rendering and Qwen-Wan-2.2 for salient lip-sync moments.Doubao subclips are typically 3–8 seconds and can be merged into videos up to 15 seconds; prior clip frames can support continuity.Source: §3.3 · pp. 8–9

3.4. Gemini Verifier

  • The Verifier checks keyframes for physical realism and instruction adherence, then evaluates rendered clips for feasibility, script alignment, and character continuity.When realism passes, the highest-scoring candidate is accepted; video verification focuses on physical feasibility.Source: §3.4 · p. 9

4. Experiment

The experiment describes objective, LLM-based, and expert-human evaluation for a 30-song benchmark, compares commercial systems with professional MVs, and specifies component ablations. These passages establish evaluation procedures and variants, not ablation outcomes.

4.1. Evaluation metrics

  • The evaluation combines ImageBind cross-modal similarity with LLM ratings across 12 criteria and four categories, plus expert ratings using the same rubric.LLM criterion scores use a 1–5 scale; category scores are averaged and combined into a weighted score.Source: §4.1 · pp. 9–10

4.2. Experiment setup and bselines

  • The benchmark covers 30 songs and compares AutoMV with two commercial systems and professional ground-truth MVs, using shared audio inputs and evaluation measures.Commercial products are used without human editing; the professionally created MVs serve as an upper bound.Source: Table 1 · p. 10, §4.2 · p. 10

4.3. Ablation study

  • The ablations remove lyric transcription and timestamps, the character bank, or the Gemini Verifier to examine those components’ roles in the pipeline.Without the Verifier, first-round results are accepted, including clips that may be physically unrealistic or inconsistent with script requirements.Source: §4.3 · p. 11

5. Results and Discussion

AutoMV outperforms commercial generators across the four evaluation categories, while professional MVs remain stronger overall; its gap to them narrows on music-alignment and creative-coherence measures. Ablations and a case study highlight the contributions of lyrics, character conditioning, and verifier-guided refinement.

5.1. Main results

  • At 10–20 USD per song, AutoMV costs about as much as Revid.ai-base and less than OpenArt story mode, but takes around 30 minutes or more per song.The reported runtime reflects available hardware and single-node inference; the authors say more parallelism, faster GPUs, and production-grade infrastructure could reduce it.Source: §5.1 · p. 11
  • AutoMV matches expert-directed MVs on Music Content and ranks behind them in Technical and Post-production, while commercial systems’ highest Art scores mainly reflect AI-styled artifacts.The commercial Art advantage is concentrated in “AI Feature Showcase,” where the LLM interprets those artifacts as artistic novelty.Source: §5.1 · p. 11
  • The gap to professional MVs persists but narrows on ImageBind alignment and human-rated creative coherence; experts score innovation and AI-specific artistry conservatively.The expert rubric covers 12 sub-criteria across four aspects, and its weighted average forms the expert score reported in Table 1.Source: §5.1 · pp. 11–12, Table 2 · p. 12
  • AutoMV substantially outperforms commercial generators across all four categories, while expert MVs remain the upper bound, especially in Technical (3.82) and Content (2.95).AutoMV is the strongest automatic system in semantic audio–visual alignment, LLM-based quality scores, and expert ratings.Source: §5.1 · pp. 11–12

5.2. Ablation results

  • Removing lyrics and timestamps most sharply reduces Musical Theme, Storytelling, Emotional Expression, and Audio–Visual Correlation, while technical scores remain largely stable.The authors report that shot-transition consistency also remains similar, while Art scores decrease.Source: §5.2 · pp. 12–13
  • Without the character bank, Character Consistency falls from 3.07 to 1.22, alongside mismatched faces, clothing changes, and weaker Musical Theme and Storytelling scores.The authors connect stable, well-defined character design with narrative coherence across long-form MVs.Source: §5.2 · p. 13
  • Removing Gemini verifiers lowers Physical Authenticity, Visual Harmony, Visual Quality, and AI Novelty; Visual Quality drops from 3.28 to 2.61.Experts report more physically implausible motion and off-script details without verifier-driven rejection and regeneration.Source: §5.2 · p. 13

5.3. Case study

  • In a shared-input case study, AutoMV produces more dynamic, varied narratives than the baselines while maintaining character identity across shots and camera angles.The examples include singing, dancing, and interactions with the environment; OpenArt-story shows limited character action, while Revid.ai-base mostly generates static images.Source: §5.3 · p. 13

6. Conclusion

AutoMV is a training-free multi-agent system for generating full-length music videos from audio, combining music-aware preprocessing, script-driven planning, character design, and verifier-guided refinement. Across 30 songs, it outperforms commercial systems and narrows the gap to professional MVs, though expert-directed videos remain stronger overall.

6. Conclusion

  • AutoMV combines music-aware preprocessing, script-driven planning, character design, and verifier-guided refinement to generate coherent, full-length music videos directly from audio.The authors describe it as a training-free multi-agent system.Source: §6 · p. 13
  • Across 30 songs, AutoMV consistently outperforms commercial systems across semantic, technical, editorial, and artistic dimensions, significantly narrowing the gap to professional MVs.A human study with industry experts confirms the gains, while LLM-based evaluation provides a scalable complementary metric.Source: §6 · p. 13
  • Human-directed MVs still achieve higher overall quality, and current multimodal LLMs cannot fully replace expert judgment.The authors identify longer-sequence character consistency, creative reasoning, and inference efficiency as future-work areas.Source: §6 · p. 13

Ethics and Social Impact

The paper discusses safeguards and social risks of music-video generation, including transparency, copyright, representation, and the limits of current production quality. It also describes the benchmark’s human-evaluation ethics and the system’s intended support for independent creators.

Ethics and Social Impact

  • Generated music videos should carry AI-generated tags and metadata to distinguish them from human-produced works and support public trust and accountability.The authors say labeling becomes increasingly critical as generative quality improves.Source: Ethics and Social Impact · p. 14
  • The benchmark releases YouTube URLs rather than commercial audio, allowing evaluation under the same protocol without redistributing copyrighted music.The authors identify this as a way to avoid violating music copyright law.Source: Ethics and Social Impact · p. 14
  • The authors flag under-representation of minority music styles, visual identities, and low-resource lyric languages, and encourage more diverse curation and system design.Source: Ethics and Social Impact · p. 14
  • Human expert evaluation follows institutional ethics review requirements and uses informed consent and voluntary participation.Source: Ethics and Social Impact · p. 14
  • The MIT license lowers barriers for independent creators, but AutoMV still has a quality gap from large commercial productions and cannot replace human creators in the short term.The authors also call for continued monitoring of long-term labor and creative-displacement risks.Source: Ethics and Social Impact · p. 14
  • More realistic visual generation raises portrait-rights and identity-misuse risks; proposed safeguards include watermarking, attribution guidance, and community audits.Source: Ethics and Social Impact · p. 14

A. Demo Comparison

The demo comparison reports that AutoMV’s full-length videos improve on baselines in several aspects while costing less, with different weaknesses distinguishing Revid.ai-base and OpenArt-Story.

A. Demo Comparison

  • AutoMV produces better identity consistency, content diversity, story flow, and music–lyrics–visual alignment than the compared baselines, while maintaining lower cost.Source: A. Demo Comparison · p. 19
  • Revid.ai-base aligns lyrics and visuals well but produces videos with almost no narrative structure.Source: A. Demo Comparison · p. 19
  • OpenArt-Story maintains character identity and coherent storylines, but relies mainly on static shots and has weaker lyrics–image correspondence.Source: A. Demo Comparison · p. 19

B. Detailed Protocol of LLM and Human Experts Scoring

The scoring protocol applies a shared, rubric-based evaluation to full-length music videos and compares two commercial baselines with AutoMV. The protocol covers technical, post-production, content, and artistic qualities.

B. Detailed Protocol of LLM and Human Experts Scoring

  • LLM judges receive the full generated video and original audio, score 12 sub-criteria on a five-level rubric, and use the same rubric as human evaluators.Ratings are integers from 1 to 5.Source: B. Detailed Protocol of LLM and Human… · p. 19
  • The four weighted categories are Technical and Post-Production at 20% each, and Content and Art at 30% each.Technical averages four sub-points, Post-Production two, and Content and Art three each.Source: B. Detailed Protocol of LLM and Human… · pp. 19–21
  • The rubric spans character consistency, physical authenticity, lip-sync accuracy, visual harmony, shot continuity, audio–visual correlation, musical theme, storytelling, emotional expression, visual quality, creativity, and AI novelty.Source: B. Detailed Protocol of LLM and Human… · pp. 19–20

C. Details of Baseline

  • The comparison uses OpenArt-Story and Revid.ai-base because no open-source music-video generation pipeline was publicly available.Source: C. Details of Baseline · p. 22
  • OpenArt’s narrative mode generates images before video clips, but its characters come from a fixed bank that cannot adapt to the input music.The experiments use Doubao for image generation and Hailuo for video generation; the current cost is around $20 per song.Source: C. Details of Baseline · p. 22
  • Only Revid.ai’s base version is evaluated; its Pro version costs more than three times as much as AutoMV, and its internal models are undisclosed.Source: C. Details of Baseline · p. 23
  • Manual editing is disabled for both commercial baselines so their outputs depend solely on the input music.Source: C. Details of Baseline · p. 23

D. Detail Results of LLM-score Per-categary

The results distinguish the judges’ ability to track expert assessments: Gemini models preserve meaningful differences between methods, whereas Qwen models are less discriminative under limited temporal context and score saturation.

D. Detail Results of LLM-score Per-categary

  • Gemini-2.5/3-Pro are the most aligned and discriminative LLM judges, closely tracking expert assessments while preserving meaningful gaps between methods.Source: Table 3 · p. 23, Table 4 · p. 23, Table 5 · p. 24
  • Gemini-2.5 Flash and Pro rank humans highest in Technical and Post-Production, AutoMV second, and OpenArt last; AutoMV approaches or slightly exceeds human scores in Content.The judges differ in Art: AI-styled visuals can raise OpenArt’s AI Novelty scores, while AutoMV is closer to human videos on more classical criteria.Source: D. Detail Results of LLM-score… · pp. 23–24, Table 3 · p. 23, Table 4 · p. 23
  • Gemini-3-Pro-Preview shows a similar ordering, with lower absolute scores and narrower spread; AutoMV remains better than OpenArt in Content and Post-Production.The authors report that it has better sample-level correlation to human experts than Gemini-2.5.Source: Table 5 · p. 24, D. Detail Results of LLM-score… · p. 24
  • Qwen-Omni-3 gives methods scores often within 0.1–0.3, but its 30-second mid-song input mainly reflects a few shots rather than full-song structure.The authors say it consequently struggles to capture long-term narrative, rhythm-aware editing, and identity consistency.Source: Table 6 · p. 24, D. Detail Results of LLM-score… · p. 24
  • Qwen-Omni-2.5 assigns nearly all methods scores between 4.7 and 5.0 across most criteria, making relative rankings unreliable despite a roughly reasonable ordering.The paper characterizes it as closer to a generous pass/fail checker than a fine-grained evaluator under current prompting and context constraints.Source: D. Detail Results of LLM-score… · p. 24, Table 7 · p. 25

E. Failure Case Study and Future Work

AutoMV's output quality remains inconsistent because of foundation-model limitations, with failures spanning physical plausibility, beat-aligned motion, text, and lip-sync. The section describes where these failures occur and notes source separation's benefit for lip-sync.

Failure overview

  • Foundation-model limitations leave AutoMV outputs inconsistent across physical plausibility, off-beat movement, textual errors, and lip-sync mismatch.The reported sources include video-generation APIs and ASR modules.Source: E. Failure Case Study and Future Work · p. 25

Physically Implausible

Off-Beat Dance Movements

  • Beat-aligned dance remains challenging because beat-tracking and rhythm-alignment methods are not robust enough across music styles and temporal patterns.The system therefore does not use beat-aligned motion generation as a module.Source: E. Failure Case Study and Future Work · pp. 25–26

Textual Inconsistencies

  • Video models struggle with consistent glyph shapes and character progression, while Whisper transcription errors can propagate into distorted or misaligned on-screen text.Figure 6 illustrates inconsistent handwritten glyphs and temporal progression.Source: E. Failure Case Study and Future Work · p. 26, Figure 6 · p. 26

Lip-Sync mismatch without source separation

  • Source separation significantly improves lip-sync accuracy and naturalness compared with processing mixed audio, so the main experiments use source separation.The mismatch is especially pronounced in Jay Chou’s songs; Figure 7 contrasts approaches with and without source separation.Source: Figure 7 · p. 26, E. Failure Case Study and Future Work · p. 27

F. Human eval v.s. Gemini eval v.s. Rule-based eval

Model-based evaluation scores correlate with human expert judgments, with stronger video-understanding models showing higher correlations on several categories. The reported relationships vary across models and evaluation criteria.

Model-based evaluation

  • Figure 8 heatmaps Pearson correlations across models, 12 sub-criteria, four category scores, and the Weighted Total; darker blue indicates stronger correlation.A zero denotes no correlation or an evaluation method that is not applicable.Source: Figure 8 · p. 27
  • Gemini-series models correlate more highly with human judgments than Qwen-Omni models on Technical, Post-Production, and Content categories.Gemini-3-Pro-Preview aligns notably with human preferences on Physical Authenticity, Shot Continuity, and Emotional Expression.Source: F. Human eval v.s. Gemini eval v.s.… · p. 27, Figure 8 · p. 27
  • ImageBind's holistic score has a 0.274 correlation with overall human-rated scores.ImageBind jointly encodes visual, auditory, and textual data.Source: F. Human eval v.s. Gemini eval v.s.… · p. 27
Loading 2512.12196v1…