Source-linked AI summary
Video-to-Music Generation for Gameplay Videos
Felipe Marra, Lucas N. Ferreira
TL;DR
Video-to-music models have mainly been studied for film and music videos, leaving open whether they transfer to gameplay with rendered graphics and looping soundtracks. This paper builds an SNES gameplay-and-music dataset and compares video encoders connected directly to a MusicGen decoder. The frozen ViViT model achieves the best overall results and compares favorably with baselines in objective evaluation and listening tests.
Problem
It remains unclear whether video-to-music models built on real-world-pretrained encoders transfer to games, where graphics and audio are synthetic and soundtracks loop across levels rather than following on-screen events.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Video-to-music models have advanced considerably in the last few years, particularly in film and music video applications. In this paper, we investigate this problem in the video game domain, which introduces new challenges for these models: video frames are rendered graphics, music is mostly synthetic audio, and soundtracks loop across entire levels rather than following on-screen events. We introduce a new dataset of 217.6 hours of Super Nintendo (SNES) gameplay video paired with 485 hours of clean soundtracks, free of sound effects and voice-overs, matched to gameplay audio via audio fingerprinting. With this dataset, we train a simple encoder-decoder transformer that passes video features directly to a MusicGen decoder, comparing different encoding strategies: textual descriptions (T5), independent frames (ViT), or spatiotemporal patches (ViViT). Each encoder is tested both frozen and fine-tuned, while the decoder is always fine-tuned. Frozen encoders match or outperform their fine-tuned counterparts on every metric, and the frozen ViViT achieves the best overall results. We compare this model with state-of-the-art baselines using both objective metrics and a listening study (N = 96). Despite having up to 18% fewer parameters, our model outperforms all baselines on objective metrics, surpasses GVMGen in the listening study, and performs comparably to OSSL.
1. INTRODUCTION
Video-to-music research has largely focused on film and music videos, while video games differ in their rendered graphics, synthetic audio, and level-spanning looping soundtracks. This work introduces an SNES gameplay-and-music dataset and tests video-to-music models in that setting.
1. INTRODUCTION
- The study asks whether video-to-music models pretrained on real-world data transfer to games, where rendered visuals and looping soundtracks differ from film and music-video settings.Unlike those domains, game soundtracks need not follow on-screen events.Source: §1 · p. 1
- 217.6 hours of SNES gameplay video are paired with 485 hours of clean soundtrack audio, matched through audio fingerprinting and balanced across 11 genres.The soundtracks exclude voice-overs and sound effects.Source: §1 · p. 1
- The direct-mapping transformer compares T5, ViT, and ViViT encoders with a fine-tuned MusicGen decoder; frozen encoders match or outperform fine-tuned ones on every metric.Frozen ViViT achieves the best overall results.Source: §1 · p. 1
- The model outperforms Herrmann-1, OSSL, and GVMGen on objective metrics and, in a listening study (N = 96), beats GVMGen and performs comparably to OSSL with up to 18% fewer parameters.The authors interpret these results as evidence that frozen real-world-pretrained Transformer encoders can generalize to retro-game videos.Source: §1 · p. 1
2. RELATED WORK
Prior video-to-music work commonly uses text to bridge visual and musical representations, while video-game music research has largely focused on symbolic generation. The paper positions its work as audio generation conditioned on gameplay video for SNES games.
2. RELATED WORK
- Video-to-music systems are grouped by how text bridges visual and musical modalities: text as interface, text embedding as interface, or direct mapping.MusicGen is a widely adopted text-to-music base model for video-to-music generation.Source: §2 · pp. 1–2
- Text-as-interface systems turn visual features into descriptions or tags for a text-to-music generator; Herrmann-1 targets film, while CoT-VTM and SONIQUE use related prompt-based strategies.These components can remain frozen, allowing use of unpaired video and music data.Source: §2 · p. 2
- Text-embedding methods map visual features into a decoder’s text space, as in OSSL, while direct-mapping models send visual features to the decoder and typically train it.GVMGen argues that spatiotemporal fusion can degrade frame-level alignment; VidMuse and V2Meow are other direct-mapping examples.Source: §2 · p. 2
- Video-game music work has mostly used symbolic generation: NES-MDB and LakhNES lack gameplay conditioning, while NES-MVDB pairs gameplay video with symbolic music.This paper instead generates audio for SNES games, whose sample-based audio is richer than the NES’s synthesized sound.Source: §2 · p. 2
3. SNES VIDEO-TO-MUSIC DATASET
The dataset pairs SNES gameplay clips with clean original soundtracks by matching gameplay audio to fingerprinted tracks. Genre balancing yields a large collection that retains coverage of unique background tracks.
Collection & Pre-Processing
- The collection starts with soundtracks from 1,604 games and keeps YouTube gameplay videos retrieved by game title, partitioned into 10-second clips to reduce track-transition capture.Clips were sized to match the ViViT encoder’s input.Source: §3 · p. 2
Noisy to Clean Soundtracks
- Fingerprint matching assigns a clean original track when its confidence exceeds 0.01; assigned tracks reached 77.36% average accuracy against manual annotations of 10 hours of video.The annotations recorded music-track start and end timestamps and were balanced by genre.Source: §3 · p. 2
Downsampling
- Balancing clips across 11 genres while retaining at least one clip for every unique background track produces 78,344 clips, 217.6 hours of video, and 16,808 tracks totaling 485 hours of clean audio.Genre labels were mapped using Wikidata metadata and an LLM.Source: §3 · p. 2
4. VIDEO ENCODERS
The models share a MusicGen decoder and differ in how they represent video: captions, independent frames, or spatiotemporal regions. Encoder outputs are projected to the decoder dimension and passed through cross-attention.
4. VIDEO ENCODERS
- Each model linearly projects its encoder output to the shared MusicGen decoder dimension and supplies it through cross-attention.The comparison holds the decoder constant across T5, ViT, and ViViT models.Source: §4 · p. 2
- A T5 encoder maps each caption to a sequence of contextual embeddings, representing raw visual semantics without an intermediate LLM translation of scene descriptions into musical prompts.For gameplay, captions cover actions, environments, movement speed, and mechanics.Source: §4 · p. 3
- ViT represents each clip with 32 evenly spaced frames, retaining each frame’s [CLS] vector as a compact temporal sequence.Each frame is encoded with ViT-B/16; the [CLS] vector summarizes that frame.Source: §4 · p. 3
- ViViT jointly projects the 32 frames into non-overlapping spatiotemporal tubelets, retaining local region embeddings and omitting the global [CLS] token.This preserves local spatiotemporal detail rather than a single clip-level summary.Source: §4 · p. 3
5. EXPERIMENTS AND RESULTS
The frozen ViViT model achieves the strongest objective results against the baselines and is nearly 18% smaller than OSSL. In the listening study, ViViT and OSSL perform comparably on video-music and genre alignment, while ViViT leads GVMGen on SNES aesthetic alignment.
Experimental Setup
- Evaluation uses genre-split data and reports FAD for audio quality, KLD for reference-track semantic similarity, and ImageBind cosine similarity for video-music alignment.ImageBind audio and video encoders were tuned on the validation split, which was not used for testing.Source: §5 · p. 3
Encoders Comparison
- All encoder variants outperform MusicGen Base on every metric, while frozen encoders perform at least as well as their fine-tuned counterparts.ViViT has the lowest FAD mean and variance (10.53 ± 3.60) and is the only model ranked in the top two on every metric; ViT and ViViT tie for the highest mean and lowest variance IB score (0.49 ± 0.02).Source: §5 · p. 3, Table 1 · p. 3
SOTA Comparison
- ViViT achieves the best mean on all objective metrics and ranks first or tied in at least 9 of 11 genres per metric, using approximately 1.93B parameters.It uses fewer parameters than OSSL (2.35B) and GVMGen (2.22B).Source: §5 · pp. 3–4, Figure 1 · p. 4
- Direct mapping with a trained decoder outperforms OSSL on all metrics, and the simpler model also beats GVMGen despite its added Feature Transformation module.Fine-tuning substantially improves GVMGen but barely changes OSSL, consistent with the reported importance of adapting the decoder.Source: §5 · p. 4
Listening Study
- On video-music and game-genre alignment, ViViT and OSSL perform comparably and both significantly outperform GVMGen; ViViT also scores highest on SNES aesthetic alignment, significantly beating GVMGen.Differences were significant only for Q1, Q2, and Q5; 96 participants completed the listening test.Source: §5 · p. 4, Table 2 · p. 4
6. CONCLUSION
The paper introduces an SNES video-to-music dataset and compares visual encoding strategies for autoregressive generation. ViViT performs best overall, frozen encoders match or outperform fine-tuned ones, and the model performs strongly against state-of-the-art baselines.
6. CONCLUSION
- The paper introduces an SNES video-to-music dataset for comparing visual encoding strategies in autoregressive generation.Source: §6 · p. 4
- ViViT achieved the best overall results among T5, ViT, and ViViT encoders, while frozen encoders matched or outperformed fine-tuned ones.The paper suggests real-world pre-trained features transfer to rendered game graphics when only the decoder is adapted.Source: §6 · p. 5
- The direct-mapping model achieved the best objective results against state-of-the-art baselines and performed comparably to OSSL in a listening study, with up to 18% fewer parameters.Source: §6 · p. 5