Source-linked AI summary
Foley Music: Learning to Generate Music from Videos
Chuang Gan, Deng Huang, Peihao Chen, Joshua B. Tenenbaum, Antonio Torralba
TL;DR
The paper addresses how to generate plausible, synchronized music from silent videos of musicians. It translates body keypoints into MIDI with a Graph-Transformer, then synthesizes audio; human evaluations report significantly better performance than baselines, while MIDI supports flexible style generation. The system’s current synthesis stage relies on an off-the-shelf synthesizer, with end-to-end neural waveform synthesis left for future work.
Problem
Generating music from performance videos requires recognizing body–instrument interactions, selecting a predictable musical representation, and associating visual motion with audio events.
Method
The system extracts body and hand keypoints, translates them into MIDI events with a Graph-Transformer, and converts MIDI into audio using a standard synthesizer.
Results
Human evaluation shows the system performs significantly better than baselines, establishing correlations between visual signals and music through body keypoints and MIDI representations.
Takeaways & Limitations
Interpretable MIDI representations support flexible generation of different music styles and music editing.
Takeaways & Limitations
The system is not yet end-to-end trainable; future work proposes a neural synthesizer that generates waveforms directly from MIDI events.
Abstract
from arXiv · showhide
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the demo video with audio turned on to experience the results.
1 Introduction
Foley Music addresses video-to-music generation by translating musicians’ body movements into interpretable MIDI events and synthesizing plausible music. The system uses body keypoints, MIDI representations, and a Graph-Transformer to generate synchronized music, outperforming prior systems and supporting flexible style editing.
- Motivation: The paper asks whether silent videos of musicians can support automatic generation of plausible music aligned with their body movements.The proposed capability could automate sound addition and support auditory immersive experiences.
- Challenges: Video-to-music generation requires recognizing body–instrument interactions, choosing a predictable audio representation, and associating visual and musical modalities.The paper identifies these as central challenges in learning from music-performance videos.
- Representations: Body and hand keypoints provide visual representations, while MIDI encodes each note event’s timing and loudness for motion-to-MIDI translation.MIDI is symbolic, interpretable, flexible, and convertible to realistic audio with a standard synthesizer.
- Model: The Graph-Transformer combines a graph-convolutional encoder with a Transformer decoder to map temporally structured body motions to MIDI event sequences.The model captures spatial-temporal relationships among body keypoints and long-term dependencies between motion and MIDI events.
- Results: Human evaluations across nine instruments show that the generated music significantly outperforms several strong baselines on correctness, noise, synchronization, and overall preference.The evaluated instruments include accordion, bass, bassoon, cello, guitar, piano, tuba, ukulele, and violin.
- Applications: MIDI representations enable controllable generation of different music styles and flexible music editing through an interpretable intermediate representation.The paper presents this flexibility as supporting applications that were difficult with waveform or spectrogram representations.
2 Related Work
The related work spans cross-modal vision-audio learning, motion-sound correlations, neural music generation, and automated Foley. Foley Music differs by emphasizing body-part motion and MIDI representations for music generation.
- Cross-modal learning: Prior cross-modal research learns visual or audio representations from synchronized or correspondence-based vision-audio data.These studies include sound-supervised visual learning, scene-based audio representation learning, and joint visual-audio correspondence.
- Motion and sound: Earlier work demonstrates correlations between sound and motion for facial animation, talking-face generation, speech separation, lip-reading, and visual sound generation.Some studies use optical-flow-like motion representations to improve sound separation or generation.
- Music generation: Neural music-generation research includes models for melodies, chorales, speech and music waveforms, songs with multiple components, and expressive music sequences.The cited systems use approaches including recurrent, hierarchical, and Transformer-based models.
- Automated Foley: Automated Foley research has explored predicting or retrieving sounds from visual interactions, including objects struck with drumsticks.These approaches commonly use visual appearance or optical flow with spectrogram or waveform audio representations.
3 Approach
The approach converts video body motion into MIDI events through a Graph-Transformer, then renders those events as audio with a standard synthesizer.
- Representations: Human pose features capture fine-grained body and hand movements, while MIDI represents note timing, pitch, and loudness symbolically.MIDI includes note-on and note-off events, pitch, and velocity information, and can be rendered by a standard synthesizer.
- System Overview: The system uses a visual encoder, MIDI decoder, and audio synthesizer to transform video frames into music.The visual encoder extracts keypoints and body dynamics, the decoder predicts MIDI events, and the synthesizer converts them into waveforms.
- Visual Encoder: A spatial-temporal GCN encodes keypoint coordinates by modeling body-part connectivity and aggregating motion across time.The graph nodes represent body keypoints, while spatial graph convolution and temporal convolution produce the encoded pose feature P.
- MIDI Decoder: An autoregressive Transformer decoder predicts MIDI events from encoded pose features and previously generated MIDI tokens.Masked self-attention preserves autoregressive generation, cross-attention incorporates pose features, and relative positions represent token distances.
- Training and Rendering: The model is trained end-to-end with cross-entropy and generates test-time MIDI sequences using beam search with beam size 5.The predicted MIDI sequence is finally rendered into a waveform by a standard synthesizer.
4 Experiments
The experiments compare Foley Music with prior systems across multiple music-performance datasets using human judgments, automatic diversity metrics, ablations, and MIDI-based editing. Results consistently favor the proposed representations and model for alignment, sound quality, diversity, and controllability.
- Experimental setup: Experiments use MUSIC and AtinPiano for comparisons with prior systems, while URMP supports ablation studies with ground-truth MIDI.The comparison covers nine instruments, and URMP evaluations use five instruments.
- Human evaluation: Human forced-choice evaluations show that Foley Music outperforms baseline systems across instrument categories and correctness, noise, synchronization, and overall preference.The evaluation uses 450 clips, with three independent judgments per test video and majority voting.
- Qualitative analysis: Generated MIDI events are reasonably similar to ground truth, while generated spectrograms contain more structured harmonic components than those of competing methods.These visual comparisons assess both intermediate MIDI predictions and resulting sound representations.
- Human evaluation: In real-versus-fake judgments, generated music from the proposed method is harder to distinguish from real recordings than music from other systems.Workers selected which video they believed contained real audio, with 50% representing perfect confusion between real and generated audio.
- Automatic evaluation: Lower NDB demonstrates that Foley Music generates more diverse sound than the compared systems.NDB is computed by clustering training spectrograms into 50 Voronoi cells and measuring statistically different bins.
- Music editing with MIDI: Manipulating predicted MIDI keys produces different musical styles, demonstrating flexible and interpretable music editing.The paper presents key editing as an application enabled by MIDI rather than waveform or spectrogram representations.
5 Conclusions and Future Work
Foley Music generates expressive music from videos by detecting human skeletons, recognizing instrument interactions, and predicting corresponding MIDI files. Human evaluations found significantly better performance than baselines, while MIDI representations support stylistic extension.
- The system detects human skeletons, recognizes interactions with musical instruments over time, and predicts corresponding MIDI files from video.
- Human evaluation showed that the algorithm performed significantly better than baseline systems.
- Body keypoints and MIDI representations establish correlations between visual and musical signals.
- Future Work: MIDI representations allow the framework to be extended to generate music in different styles.
- Future Work: Future work plans a WaveNet-like neural synthesizer that generates waveforms from MIDI events, enabling end-to-end training.