Source-linked AI summary
Video Background Music Generation with Controllable Music Transformer
Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, Shuicheng Yan
TL;DR
Video background music generation remains underexplored, and prior music-generation work does not address rhythmic consistency between video and music. The paper maps video timing and motion cues to musical features and uses them to guide a Controllable Music Transformer. The authors report compatible music with high musical quality, without paired video–music training data.
Problem
Prior music-generation work does not address rhythmic consistency between a video and its background music, despite the real-world significance of generating music for video.
Source: §1 · p. 2Method
Results
The authors report that the method generates melodious, compatible music without paired video–music training data, and that its generated music surpasses matched music overall.
Source: §6 · p. 8, §5.4 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
In this work, we address the task of video background music generation. Some previous works achieve effective music generation but are unable to generate melodious music tailored to a particular video, and none of them considers the video-music rhythmic consistency. To generate the background music that matches the given video, we first establish the rhythmic relations between video and background music. In particular, we connect timing, motion speed, and motion saliency from video with beat, simu-note density, and simu-note strength from music, respectively. We then propose CMT, a Controllable Music Transformer that enables local control of the aforementioned rhythmic features and global control of the music genre and instruments. Objective and subjective evaluations show that the generated background music has achieved satisfactory compatibility with the input videos, and at the same time, impressive music quality. Code and models are available at https://github.com/wzk1015/video-bgm-generation.
1 INTRODUCTION
The paper frames video background music generation as an underexplored task because prior music-generation work does not address video–music rhythmic consistency. It proposes a transformer-based approach that uses video rhythm and motion to guide music generation, with controls for musical features, genre, and instruments.
1 INTRODUCTION
- The proposed relations pair video timing with beat timing, motion speed with simu-note density, and motion saliency with simu-note strength.Video timing supplies positional encoding; motion speed is measured using average optical flow, while density and strength count simu-notes per bar and notes per simu-note, respectively.Source: §1 · p. 2
- The music representation adds rhythmic attributes for local control and genre and instrument tokens for global customization, using a linear transformer backbone.Training uses music features from the Lakh Pianoroll Dataset; video-derived features guide generation at inference.Source: §1 · p. 2
- CMT generates video background music from video–music rhythmic relations without requiring paired, annotated video–music training data.The paper also reports that its approach matches music to a video's rhythm and mood while achieving high musical quality.Source: §1 · p. 2
- The paper reports that CMT matches video rhythm and mood while achieving high musical quality.It introduces simu-note density and strength representations, which the authors associate with better-generated music and more controllable multi-track generation.Source: §1 · p. 2
2 RELATED WORK
Prior symbolic music representations provide structures for rhythmic control, while music-generation models use several modeling approaches. Earlier video-to-music work focuses on clips of people playing instruments; this paper instead targets arbitrary videos.
Representations of music
- The paper builds on REMI's explicit bars, beats, chords, and tempo, then uses compound words to group neighboring tokens and shorten sequences.It adds genre and instrument tokens for global information, and density and strength attributes for local control.Source: §2 · p. 2
Music generation models
- Prior music-generation systems use autoencoders, convolutional models, transformers, or direct and indirect waveform generation; this work uses a linear transformer.The cited linear attention mechanism reduces time complexity.Source: §2 · p. 2
Composing music from silent videos
- Earlier video-to-music work targets clips of people playing instruments, whereas this paper aims for non-determined music generation from general videos.The paper notes that no dataset of paired arbitrary videos and music exists specifically for this task, and that existing audiovisual datasets may contain speech or no music.Source: §2 · p. 2
3 ESTABLISHING VIDEO-MUSIC RHYTHMIC RELATIONS
The section establishes rhythmic links between video and music, mapping video timing, motion speed, and motion saliency to musical beat, simu-note density, and simu-note strength.
3.1 Video Timing and Music Beat
- Each video frame is mapped to a musical beat using tempo and frame rate, with an inverse mapping from beats back to video frames.These mappings form the basic block for relating video timing and music.Source: §3.1 · p. 3
3.2 Motion Speed and Simu-note Density
- Video motion speed is estimated from average optical-flow magnitude over clips aligned to four beats, then connected to musical simu-note density.A simu-note groups notes sharing an onset, and density describes the distribution of these groups across a bar.Source: §3.2 · pp. 3–4
- The speed-to-density mapping uses 16 classes, matching video-speed percentiles to density levels observed in the Lakh Pianoroll Dataset.Video clips and music bars are matched by length for this mapping.Source: §3.2 · pp. 3–4
3.3 Motion Saliency and Simu-note Strength
- Visual beats are frames with local-maximum motion saliency at near-constant tempo, represented by frame index and saliency.Motion saliency is the average positive change in optical flow between consecutive frames.Source: §3.3 · p. 4
- Simu-note strength counts notes in a group and is positively correlated with visual-beat saliency, linking distinct motion to a clear music beat.Greater strength is described as producing more auditory impact.Source: §3.3 · p. 4
4 CONTROLLABLE MUSIC TRANSFORMER
CMT trains on MIDI rhythmic features and replaces them with video-derived features at inference to control generated background music. Its token representation separates rhythmic and note attributes, while beat-based position and timing encodings support control over musical structure and video progress.
4 CONTROLLABLE MUSIC TRANSFORMER
- CMT trains using rhythmic features from MIDI, then replaces them with video-derived features at inference for controllable background-music generation.The framework also uses video-music rhythmic relations without requiring paired annotated video and music for training.Source: §4 · p. 4
4.1 Music Representation
- Music tokens group seven attributes into rhythm-related and note-related types, with density and strength encoding remaining simu-notes and following note counts.Genre and instrument types are supplied as initial tokens.Source: §4.1 · pp. 4–5
4.2 Control
- At inference, CMT replaces bar-token density with optical-flow video density and visual-beat strength to control the generated number of beat and note tokens.Control degree C ranges from generation from scratch at 0 to full compatibility at 1, with greater constraint trading off against melodiousness.Source: §4.2 · p. 5
- Beat-based position encoding gives tokens within the same beat a shared position signal, reflecting that their notes map to the same audio segment regardless of order.Beat-timing encoding also represents current beat progress relative to the video's total beats and is added to token embeddings.Source: §4.2 · p. 5
4.3 Sequence Modeling
- A linear transformer models dependencies among music tokens, and a two-stage output module predicts token type before its six other attributes.Temperature-controlled stochastic sampling is used to increase token diversity.Source: §4.3 · pp. 5–6
- During inference, generation starts with genre and instrument tokens, uses video beat timing, and substitutes visual beats and density attributes when applicable.The generated token list is returned after the end-of-sequence token is predicted.Source: §4.3 · p. 6
5 EXPERIMENTS
The experiments evaluate music quality and video compatibility through objective metrics, a user study, control-error measures, and loss curves. CMT’s rhythmic controls improve objective melodiousness, while subjective evaluations show a trade-off between compatibility and melodiousness.
5 EXPERIMENTS
- The evaluation combines objective music-quality metrics and subjective ratings of generated music quality and video compatibility.Objective metrics include pitch-class histogram entropy, grooving-pattern similarity, and structureness; participants also rate melodiousness and compatibility.Source: §5 · p. 6, §5.3 · p. 6, §5.4 · p. 7
5.1 Dataset
- The training set contains 3,038 selected MIDI pieces across six genres, drawn from the cleansed Lakh Pianoroll Dataset.The cleansed dataset merges tracks into five categories: Drums, Piano, Guitar, Bass, and Strings.Source: §5.1 · p. 6
5.2 Implementation Details
- The model uses concatenated attribute embeddings, 12 self-attention layers, and trains for 100 epochs on four RTX 1080Ti GPUs.The hidden size is 512, the feed-forward inner size is 2,048, and training takes approximately 28 hours.Source: §5.2 · p. 6
5.3 Objective Evaluation
- Combining density, strength, and beat-timing features improves every reported objective metric over the baseline, while video-controlled variants show structural degradation.Experiment No. 5 has the highest overall rank; the authors attribute the structural trade-off to aligning generated music with video rhythm.Source: §5.3 · p. 7
5.5 Controlling Accuracy
- Control errors are about 0.1 for density against 9.9 simu-notes per bar, and strength control usually matches the video’s number of notes per simu-note.The evaluation normalizes density, strength, and timing errors by average simu-notes per bar, notes per simu-note, and total video time, respectively.Source: §5.5 · p. 8, Table 4 · p. 8
5.6 Visualization
- CMT’s loss curves converge faster than the baseline’s across attributes, especially for beat.The curves use exponential smoothing with α=0.9.Source: §5.6 · p. 8, Figure 5 · p. 8
6 CONCLUSION
The paper presents CMT for video background music generation, using rhythmic relations between video and music to enable local and global control. It does not require paired video–music training data and generates music described as melodious and compatible with the video.
6 CONCLUSION
- CMT uses three rhythmic relations between video and music to control background-music generation locally and globally.The method does not require paired video and music data for training.Source: §6 · p. 8
- The authors report that the method generates melodious music compatible with the given video without paired video–music training data.Source: §6 · p. 8
- Future work may explore abstract visual–music connections, waveform music, and unsupervised audiovisual representation learning from paired data.Source: §6 · p. 8