SonicVisionLM: Playing Sound with Vision Language Models

• On-screen sounds correspond to visible actions, whereas off-screen sounds are not directly observable and can provide ambience or additional narrative information.The framework combines automatic recognition of the former with user editing of the latter.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2
• Existing methods struggle with incorrect sound meanings, mismatched timing, monotonous effects, complex scenarios, and limited off-screen editability.These difficulties arise from challenging alignment between video and audio features.Source: §1 · p. 1
• The framework uses a VLM to identify on-screen sound descriptions, a timestamp network to locate events, and text-timestamp conditions for audio generation.This decomposes video-to-audio generation into video-to-text, temporal detection, and text-to-audio stages.Source: §1 · pp. 1–2
• Users can edit generated text and timestamps or add new text-timestamp pairs to personalize off-screen sound design.The text-based interaction component supports customization beyond automatically detected on-screen events.Source: §1 · p. 2, Figure 2 · p. 2
• CondPromptBank contains over ten thousand text-to-single-sound examples across 23 categories for training the time-controllable adapter.The dataset provides sound-effect descriptions and timing cues.Source: §1 · p. 2
• SonicVisionLM achieves state-of-the-art results on conditional and unconditional video-sound generation, with conditional IoU improving from 22.4 to 39.7.The authors report improvements across all conditional-task metrics.Source: §1 · p. 2

Audio Generation

• Text-to-audio research includes TTS, text-to-music, and universal audio generation, while SonicVisionLM focuses on text-to-single-sound data with semantic and temporal annotations.CondPromptBank is designed for video sound effects, where timing descriptions require precise control.Source: §2 · p. 2

Video-to-Audio Generation

• Prior video-to-audio methods, including Diff-foley, use contrastive audio-visual pretraining to align audio and visual features, but generated sounds often suffer from poor synchronization, noticeable noises, and limited editability.Source: §2 · p. 3

Diffusion Models

• Diffusion models support audio generation in mel-spectrogram or waveform spaces, but controlling latent diffusion for audio remains challenging.SonicVisionLM addresses this by introducing time control for audio-visual synchronization.Source: §2 · p. 3

Vision-Language Models

• Vision-language models provide multimodal alignment and visual grounding capabilities that SonicVisionLM applies to audio generation.The paper positions VLMs as the bridge from visual events to sound descriptions.Source: §2 · p. 3

3.1. Overview

• SonicVisionLM combines visual-to-audio event understanding, timestamp detection, and a time-controllable latent diffusion model for synchronized sound generation.The overview presents these as the framework’s main stages, producing semantically coherent and temporally aligned multiple sounds.Source: §3.1 · p. 3

3.2. Preliminaries

• The latent diffusion backbone generates audio from text guidance through forward noising and reverse denoising, with a U-Net noise estimator and audio VAE decoder.The reverse process samples an audio latent from Gaussian noise, reconstructs a mel-spectrogram, and passes it to a vocoder for final audio.Source: §3.2 · p. 3

3.3. Visual-to-Audio Event Understanding Module

• Four temporally prompted video frames are processed by an adapted multimodal VLM to generate descriptions of sounds associated with visual events.The adaptation addresses the limitation of single-image understanding for dynamic video information.Source: §3.3 · pp. 3–4

3.4. Sound Event Timestamp Detection Module

• A ResNet(2+1)-D18 network predicts frame-level sound presence from silent-video sequences, deriving event start and end times for temporal conditioning.Audio detected within 0.02 seconds across consecutive frames is treated as one event; the resulting binary timestamp vector marks sound presence and absence.Source: §3.4 · p. 4

3.5. Time-controllable Latent Diffusion Model

• The time-controllable adapter constructs an Audio Time-condition Embedding by masking Mel-spectrogram features with detected timestamps and encoding the result.The embedding is integrated with text and target-audio representations while the base Tango elements remain frozen.Source: §3.5 · pp. 4–5

4.1. Experiment Settings

• CondPromptBank contains 10,276 single-sound-effect entries with titles, timestamps, 23 categories, and manually filtered audio for training time-controlled adapters.The dataset uses sounds typically 10 seconds or shorter and adds fine-grained descriptions of sound characteristics and actions.Source: §4.1 · p. 5

4.2. Conditional Generation Task Results

• 36.8% and 42.8% CLAP-top scores exceed every CondFoleyGen model by more than 20%, while Onset AP, IoU, onset accuracy, and timing precision also improve.SonicVisionLM-full is reported to outperform CondFoleyGen across all conditional-generation metrics.Source: §4.2 · p. 5, Table 2 · p. 6
• SonicVisionLM produces conditional sounds that more closely match target sound shapes and achieve higher visual-audio synchronization than CondFoleyGen.The qualitative comparison reports closer positional matches and more similar mel-spectrograms, while CondFoleyGen sometimes produces blurred or mismatched sounds.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3. Unconditional Generation

• SonicVisionLM performs strongly on unconditional generation, with improved Greatest Hits MKL, IS, and IoU and nearly 10-point higher IS than baselines on CountixAV.On CountixAV, the model has slightly lower MKL and FID than baselines, which the authors relate to the dataset’s noisy, mixed-source audio and their focus on specific sound events.Source: §4.3 · p. 7, Table 3 · p. 7
• Qualitative CountixAV results show more precise sounds and finer-grained responses to visual motion, whereas SpecVQGAN is noisy and DIFF-FOLEY can lack synchronization.The comparison uses denoised data to make timing differences clearer.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4. Ablation Study

• Both SonicVisionLM versions outperform Tango in IoU, supporting the effectiveness of the time-controllable adapter.The ablation also finds the best balance of sound quality and steadiness at conditioning scale 2.0; control declines substantially at 3.0.Source: §4.4 · p. 8, Table 5 · p. 8, Figure 6 · p. 8

4.5. Multi-soundtracks generation

• SonicVisionLM separates visible-event sounds from optional off-screen tracks and places each soundtrack on the timeline to match visible actions.The multi-soundtrack example describes automatic recognition of sounds such as cooking meat and pots, followed by user-directed additions such as distant chopping or machine noise.Source: §4.5 · p. 8, Figure 7 · p. 8

5. Conclusion

• SonicVisionLM uses VLMs to identify events in silent videos and suggest matching sounds, achieving strong results in conditional and unconditional generation.The conclusion also describes automatic on-screen sound recognition and personalized off-screen sound editing for post-production.Source: §5 · p. 8
• Visual understanding and timestamp detection still require refinement, while broader multimedia coverage and more diverse audio-generation control remain future needs.These limitations define the paper’s stated scope boundary for improving applicability.Source: §5 · p. 8

Supplementary Material

• The unconditional-generation evaluation page requires participants to enter three scores for each presented video before clicking “Next.”The interface screenshot documents the participant workflow rather than reporting evaluation outcomes.Source: Figure 8 · p. 11

A.1. Subjective Results Details

• Figure 8 supplies a screenshot of the main evaluation page shown to participants during the test.The passage identifies the page but does not describe its scoring fields or report study results.Source: §A.1 · p. 11

• An AP of 0.80 and accuracy of 0.72 are achieved on the Greatest Hits test set.Source: §A.2 · p. 11
• Validation performance is higher, reaching an AP of 0.92 and accuracy of 0.82.Source: §A.2 · p. 11
• CountixAV performance drops to an AP of 0.52 and accuracy of 0.52, likely because complex sound sources create inaccurate ground truth labels, especially for non-static footage.Source: §A.2 · p. 11

Conditional Generation Results

• Conditional generation results provide six examples comparing target videos, control conditions, and generated outputs.Source: §A.3 · p. 11, Figure 9 · p. 11

Unconditional Generation Results

• Unconditional generation results compare GT, SpecVQGAN, DIFF-FOLEY, and SonicVisionLM on six CountixAV examples.Source: §A.3 · p. 11, Figure 10 · p. 12

Multi-soundtracks Generation Results

• For multi-soundtrack generation, videos are first segmented into shots before processing clear-action shots with VLMs.Source: §A.3 · p. 12, Figure 11 · p. 12
• VLM-generated descriptions and spatial positions are used to produce on-screen sounds for shots with clear actions.Source: §A.3 · p. 12
• Other shots support user editing before being processed by the LDM to generate off-screen sounds.Source: §A.3 · p. 12
• The multi-soundtrack demonstrations include one simple case and one complex case.Source: §A.3 · p. 12