PaperlayerView exact revision on arXiv

Source-linked AI summary

V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models

Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, Weidong Cai

arXiv:2308.09300v4cs.CVcs.AIcs.MMcs.SDeess.AS

TL;DR

Open-domain vision-to-audio generation lacks lightweight foundation-model solutions, while existing systems often require complex training on modest datasets. The paper connects frozen CLIP, CLAP, and AudioLDM through a V2A-Mapper, achieving stronger fidelity and relevance with substantially fewer parameters, while retaining limits in fine control and temporal alignment.

  • Problem

    Open-domain vision-to-audio generation remains under-researched with foundation models, and existing methods often use complex systems trained on modestly sized datasets.

    Source: Introduction · pp. 1–2

  • Method

    A V2A-Mapper translates CLIP visual features into CLAP auditory embeddings that condition pretrained AudioLDM, while the foundation models remain otherwise intact.

    Source: Introduction · p. 2, AudioLDM · p. 3

  • Results

    The method uses 86% fewer parameters than Im2Wav while improving fidelity (FD) by 53% and relevance (CS) by 19%, with strong objective and subjective performance.

    Source: Introduction · p. 2, Objective Results · p. 6, Subjective results · p. 6

  • Takeaways & Limitations

    Foundation models can support efficient open-domain V2A generation by requiring optimization of only a mapper between visual and auditory representations.

    Source: Introduction · p. 2, Conclusion · p. 8

    The system lacks fine-grained controllability, can fail on unclear visual subjects, and does not explicitly handle audio-visual temporal alignment.

    Source: Limitation and Future Work · p. 8

Abstract

from arXiv · show

Building artificial intelligence (AI) systems on top of a set of foundation models (FMs) is becoming a new paradigm in AI research. Their representative and generative abilities learnt from vast amounts of data can be easily adapted and transferred to a wide range of downstream tasks without extra training from scratch. However, leveraging FMs in cross-modal generation remains under-researched when audio modality is involved. On the other hand, automatically generating semantically-relevant sound from visual input is an important problem in cross-modal generation studies. To solve this vision-to-audio (V2A) generation problem, existing methods tend to design and build complex systems from scratch using modestly sized datasets. In this paper, we propose a lightweight solution to this problem by leveraging foundation models, specifically CLIP, CLAP, and AudioLDM. We first investigate the domain gap between the latent space of the visual CLIP and the auditory CLAP models. Then we propose a simple yet effective mapper mechanism (V2A-Mapper) to bridge the domain gap by translating the visual input between CLIP and CLAP spaces. Conditioned on the translated CLAP embedding, pretrained audio generative FM AudioLDM is adopted to produce high-fidelity and visually-aligned sound. Compared to previous approaches, our method only requires a quick training of the V2A-Mapper. We further analyze and conduct extensive experiments on the choice of the V2A-Mapper and show that a generative mapper is better at fidelity and variability (FD) while a regression mapper is slightly better at relevance (CS). Both objective and subjective evaluation on two V2A datasets demonstrate the superiority of our proposed method compared to current state-of-the-art approaches - trained with 86% fewer parameters but achieving 53% and 19% improvement in FD and CS, respectively.

Introduction

Open-domain vision-to-audio generation remains underexplored with foundation models, despite its relevance to immersive media, video editing, and visual assistance. The paper addresses complex, resource-intensive existing systems with a lightweight mapper connecting pretrained visual, audio, and generative models.

  • Open-domain vision-to-audio generation remains unclear as an application of foundation models, unlike better-established vision-language and text-to-audio adaptation.Source: Introduction · p. 1
  • Natural sound generation from diverse real-world visual inputs supports applications including virtual reality, video editing, and assistance for people with visual impairment.Source: Introduction · p. 2
  • Existing open-domain V2A methods train complex, separately optimized submodules on limited datasets, increasing resource demands and potentially restricting generalization.Source: Introduction · p. 2
  • The proposed system trains only a V2A-Mapper to translate CLIP visual representations into CLAP auditory representations for pretrained AudioLDM generation.Source: Introduction · p. 2
  • The method uses 86% fewer training parameters while improving fidelity (FD) by up to 53% and relevance (CS) by up to 19% in objective and subjective evaluations on two V2A datasets.Source: Introduction · p. 2

Related Works

Earlier V2A systems progressed from controlled sound types to label-free and CLIP-based generation, but commonly retained multiple trainable modules. This paper instead adapts frozen foundation models by adding a mapper without altering their internal architectures.

Vision-to-Audio Generation.

  • Earlier V2A methods handled controlled or individual sound types, while SpecVQGAN introduced label-free generation of diverse sounds with visual features and an autoregressive synthesizer.Source: Vision-to-Audio Generation · p. 2
  • Im2Wav and CLIPSonic-IQ adopted CLIP but still trained multiple modules from limited data, whereas the proposed approach optimizes only a V2A-Mapper to connect foundation models.Source: Vision-to-Audio Generation · p. 2

Foundation Model Adaptation.

  • Unlike prior multimodal adaptations that add auxiliary layers or tokens, this method keeps foundation models frozen and intact, simplifying deployment and updating.Source: Foundation Model Adaptation · p. 2

Method

The method combines frozen CLIP, CLAP, and AudioLDM with a trainable V2A-Mapper that translates visual embeddings into audio space for vision-to-audio generation. It first measures and bridges the CLIP–CLAP domain gap, then compares regression and diffusion-based mapper designs.

Method

  • Frozen CLIP, CLAP, and AudioLDM are combined with a trainable V2A-Mapper to generate visually aligned sound while leaving the foundation models unchanged.CLIP provides visual features, CLAP supervises audio-space translation, and AudioLDM synthesizes the waveform.Source: Method · p. 2, AudioLDM · p. 3

Selected Foundation Models

  • CLIP extracts semantically rich visual features, while CLAP supplies a joint text-audio embedding space that can supervise translation from vision to audio.The shared text modality between CLIP and CLAP is assumed to facilitate this cross-modal translation.Source: CLIP · p. 3, CLAP · p. 3

AudioLDM.

  • AudioLDM is retained as an off-the-shelf conditional generator that converts translated CLAP embeddings into audio waveforms.It generates mel-spectrogram latent codes conditioned on CLAP audio embeddings, which are then decoded into waveforms.Source: AudioLDM · p. 3

Bridge the Domain Gap between Vision and Audio

  • The authors find a substantial CLIP–CLAP domain gap: paired visual and audio embeddings have average cosine similarity near 0 before mapping.They estimate the distributions using 5,000 VGGSound samples, 512-dimensional features, and UMAP visualization.Source: Bridge the Domain Gap between Vision… · p. 4, Figure 3 · p. 4
  • V2A-Mapper converts frozen CLIP video features into pseudo-CLAP embeddings using paired visual-audio data and mean squared error supervision.Frame features are aggregated by random selection, middle-frame selection, or temporal averaging; temporal averaging performs best in fidelity and relevance.Source: Bridge the Domain Gap between Vision… · p. 4
  • After training, translated embeddings overlap the target CLAP space, indicating that the mapper closes the visual-audio domain shift.The visual representation is formed by aggregating frozen CLIP frame features, while paired audio is encoded with frozen CLAP.Source: Bridge the Domain Gap between Vision… · p. 4

Diffusion-based V2A-Mapper

  • The diffusion mapper models a one-to-many visual-to-audio mapping by recovering target audio embeddings from noisy embeddings conditioned on timestep and visual features.The authors compare MLP and Transformer implementations; the Transformer incorporates the condition better than simple MLP concatenation.Source: Diffusion-based V2A-Mapper · p. 4
  • The method achieves the best objective metrics on both VGGSound and ImageHear and outperforms prior methods in subjective sound quality and visual relevance.The objective comparison also reports fewer trainable parameters for the proposed method.Source: Table 1 · p. 5, Table 2 · p. 5

Experiments

The method is evaluated on in-distribution and out-of-distribution visual-audio data using objective and subjective measures of fidelity and relevance. It outperforms prior systems while requiring substantially fewer parameters and faster inference.

Datasets.

  • The experiments train on 183,730 VGGSound videos, evaluate on 15,446 held-out videos, and test generalization on 101 ImageHear images.All evaluations generate 10-second audio samples.Source: Datasets · p. 5

Metrics.

  • Evaluation uses FD as the main sound-quality metric, FAD for comparison, and CS for visual-audio relevance.FD compares synthesized and real embedding distributions, while CS compares CLIP visual embeddings with Wav2CLIP audio embeddings.Source: Metrics · p. 5

Subjective Testing.

  • Human evaluation asks 20 listeners to rate fidelity and relevance for audio generated from 20 randomly selected visual samples.Ratings are summarized with MOS and paired comparisons using Just Meaningful Difference scores.Source: Subjective Testing · p. 5

Compare with SOTA

  • The approach trains only a V2A-Mapper on top of pretrained foundation-model components, unlike prior systems that train multiple audio-generation modules.Im2Wav uses two Transformer decoders, a VQ-VAE, and a vocoder, while CLIPSonic-IQ also trains a diffusion model and BigVGAN.Source: Compare with SOTA · p. 6

Objective Results.

  • 53% higher FD and 19% higher CS than Im2Wav, with 86% fewer trained parameters and 24× faster inference.The method also outperforms CLIPSonic-IQ across objective metrics with fewer parameters and faster inference.Source: Objective Results · p. 6

Subjective results.

  • The proposed method exceeds previous works in both fidelity and relevance in subjective evaluations on videos and images.CLIPSonic-IQ particularly fails on relevance for video inputs, while a gap remains to ground truth.Source: Subjective results · p. 6

Ablation Study

The ablations show that directly translating visual representations into the auditory latent space is preferable to caption-based or unmapped conditioning. Generative mapping, temporal feature aggregation, compatible vision encoders, and stronger audio generators each materially affect performance.

Different Ways of Utilizing FMs.

  • Caption-based conditioning remains inferior to V2A-Mapper conditioning in fidelity and relevance, while skipping the mapper prevents high-fidelity, visually relevant sound.The mapper preserves visual information in latent form and explicitly translates CLIP features into CLAP space.Source: Different Ways of Utilizing FMs · pp. 6–7

Inside the Mapper: Generative vs. Regression.

  • Generative mapping improves fidelity and diversity, whereas regression mapping is slightly better for relevance but produces much worse FD scores.The generative mapper learns a one-to-many mapping, and Transformer attention integrates visual conditions better than linear projections.Source: Inside the Mapper: Generative vs.… · p. 7

Different Aggregator Methods σ.

  • Averaging CLIP features across all video frames outperforms selecting a random or middle frame because it summarizes temporal dynamics for 10-second audio generation.The averaged representation retains semantic content throughout the video rather than relying on a single frame.Source: Different Aggregator Methods σ · p. 7

Different Pretrained Vision FMs.

  • The mapper improves both CLIP- and BLIP-based systems, with similar performance despite differences in the vision-language model or domain gap.This supports the mapper’s generalization to alternative pretrained vision models.Source: Different Pretrained Vision FMs · p. 7

Different Pretrained Audio FMs.

  • Scaling AudioLDM or training it for more steps improves performance to some extent, suggesting that stronger audio generators could further improve quality and relevance.The ablation compares audioldm-s, audioldm-s-v2, and audioldm-l.Source: Different Pretrained Audio FMs · p. 7

Latent Space Interpolation

The translated CLAP-space representation supports linear interpolation between audio concepts guided by either images or text. The resulting smooth transitions indicate that the mapper learns a meaningful translation from visual to auditory space.

Latent Space Interpolation

  • Linear interpolation can transition from a frog sound toward either an image-guided flute sound or a text-specified target.The paper reports relatively smooth auditory transitions, with vision, text, and audio semantically gathered in one space without three-modality training.Source: Latent Space Interpolation · pp. 7–8, Figure 6 · p. 8

Discussion

The discussion identifies limitations in controllability, visual ambiguity, and temporal alignment, while noting misuse risks and the importance of shared detection resources.

Limitation and Future Work.

  • The system lacks fine-grained control, can fail on unclear visual subjects, and does not explicitly model temporal alignment.Suggested future directions include text-conditioned control and multimodal foundation models spanning vision, language, and audio.Source: Limitation and Future Work · p. 8
  • Text conditioning is proposed as a starting point for explicit controllability over generated sound.Source: Limitation and Future Work · p. 8

Ethical Statement.

  • Malicious misuse on social platforms could produce negative societal outcomes, motivating continued work on audio deepfake detection.The authors note that ample datasets remain important for improving detection accuracy and commit to releasing synthesized samples.Source: Ethical Statement · p. 8

Conclusion

The conclusion presents V2A-Mapper as a lightweight bridge between visual and auditory foundation models for open-domain vision-to-audio generation. It reports visually aligned, high-fidelity synthesis and superior evaluation against more complex approaches.

  • V2A-Mapper translates CLIP visual features into CLAP auditory space so AudioLDM can synthesize high-fidelity, visually aligned sound.Source: Conclusion · p. 8
  • The method is relatively lightweight because training optimizes only the V2A-Mapper.Source: Conclusion · p. 8
  • Subjective and objective evaluations show superior performance to current state-of-the-art approaches with more complex training regimes.Source: Conclusion · p. 8

Supplementary Material

The supplementary material expands the experimental documentation and explains the procedure used for subjective testing.

Implementation Details

The implementation uses four RTX A6000 GPUs, 100 training epochs, and Transformer or MLP V2A-Mapper variants. Foundation-model pretraining draws on large image-text, text-audio, and sound-clip datasets.

  • All models are trained for 100 epochs on four NVIDIA RTX A6000 GPUs, with training times ranging from 40 minutes to about 14 hours.The best diffusion-with-transformer model takes around 14 hours, while ablated models take 40 minutes to 12 hours.Source: Implementation Details · p. 11
  • V2A-Mapper is implemented with either a Transformer or a simple MLP, with depth and expansion ablations used to select configurations.The Transformer uses 12 attention heads of dimension 64, feedforward expansion rate 4, and empirically best depth 12.Source: Implementation Details · p. 11
  • CLIP, CLAP, and AudioLDM are pretrained on 400M image-text pairs, 2.5M text-audio pairs, and 3.3M ten-second sound clips, respectively.Source: Implementation Details · p. 11

Hyper-parameter Tuning

The study tunes mapper architectures and diffusion guidance scales, finding that guidance settings materially affect fidelity and relevance. Joint tuning reaches 23.468 FD on VGGSound and 9.967/12.370 CS on VGGSound/ImageHear.

Hyper-parameter Tuning

Architectures

  • Regression-based MLPs achieve their best reported results with depth 1 and expansion rate 4, although different metrics favor different hyper-parameters.This configuration is also the third-best setting on ImageHear and is used for main-paper reporting.Source: Architectures · p. 11, Table 8 · p. 11

Transformer.

  • Both regression-based and diffusion-based Transformer mappers are evaluated across layer depths, with depths 8 and 12 selected respectively.The regression-based Transformer uses depth 8, while the diffusion-based Transformer uses depth 12.Source: Transformer · p. 11, Table 10 · p. 11, Table 11 · p. 12

Guidance Scale

  • 23.468 FD on VGGSound and 9.967/12.370 CS on VGGSound/ImageHear result from jointly tuning AudioLDM and V2A-Mapper guidance scales.These settings provide the reported best performance in the guidance-scale experiments.Source: Guidance Scale · p. 12

Guidance scale of AudioLDM.

  • AudioLDM guidance initially improves fidelity and relevance, but performance slightly drops as the guidance scale increases further.The comparison varies AudioLDM guidance while holding V2A-Mapper guidance fixed.Source: Guidance scale of AudioLDM · p. 12

Guidance scale of the V2A-Mapper.

  • A V2A-Mapper guidance scale of 0.9 performs best when AudioLDM guidance is fixed at 4.5.The authors associate relaxing the mapper’s conditioning with greater diversity and visual relevance.Source: Guidance scale of the V2A-Mapper · p. 12

Human Evaluation

Human evaluation measures audio quality and visual relevance through listening tests for video-to-sound and image-to-sound generation. Ratings are analyzed with both MOS and paired-comparison JMD scores, which yield comparable standings.

Human Evaluation

  • Twenty listeners rate 20 visual samples for audio quality and relevance on a five-point scale in separate video-to-sound and image-to-sound tests.Video tests include ground truth, Im2Wav, CLIPSonic-IQ, and the proposed method; image tests omit ground-truth sound.Source: Human Evaluation · p. 12, Figure 9 · p. 13
  • MOS averages listeners’ five-category ratings, while JMD preserves each listener’s pairwise ranking through indirect scaling.The paired analysis counts wins and ties between algorithms before deriving normalized scale values.Source: Human Evaluation · p. 12
  • MOS and JMD standings are highly correlated, with 1 JMD corresponding to about 0.5 MOS.The authors therefore regard the two analyses as producing comparable results.Source: Human Evaluation · p. 13
  • The image-to-sound survey presents three generated audio clips for participants to rate on quality and relevance to the image.The video-to-sound survey instead presents four video conditions, including ground truth.Source: Figure 9 · p. 13, Figure 10 · p. 14

Related papers

Loading 2308.09300v4…