Source-linked AI summary
Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap
KiHyun Nam, Jongmin Choi, Hyeongkeun Lee, Jungwoo Heo, Joon Son Chung
TL;DR
The modality gap in contrastive audio–language embeddings is a concern for coupling multimodal encoders with LLMs. Diffusion-Link maps audio embeddings toward the text distribution and improves modality-gap measures and AAC outcomes, reaching state-of-the-art without external knowledge.
Problem
A structural modality gap in contrastive multimodal encoders is linked in prior work to zero-shot and cross-modal task performance.
Source: §1 · p. 1Method
Diffusion-Link trains on paired audio–text embeddings and uses a text-targeted reverse diffusion process to map audio embeddings into the text-embedding distribution.
Source: §1 · p. 1Results
Diffusion-Link reduces the modality gap most among prior methods and reaches state-of-the-art AAC, with up to 52.5% zero-shot and 7.5% supervised relative gains without external knowledge.
Source: §1 · pp. 1–2Takeaways & Limitations
The reported AAC results position modality bridging as a performance source beyond external-knowledge retrieval.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Contrastive audio-language pretraining yields powerful joint representations, yet a persistent audio-text modality gap limits the benefits of coupling multimodal encoders with large language models (LLMs). We present Diffusion-Link, a diffusion-based modality-bridging module that generatively maps audio embeddings into the text-embedding distribution. The module is trained at the output embedding from the frozen multimodal encoder and implemented as a lightweight network with three residual MLP blocks. To assess the effect of Diffusion-Link on multimodal encoder-LLM coupling, we evaluate on Automatic Audio Captioning (AAC); to our knowledge, this is the first application of diffusion-based modality bridging to AAC. We report two results. (1) Modality-gap analysis: on similarity and geometric criteria, Diffusion-Link reduces the modality gap the most among prior diffusion-based methods and shows a collective migration of audio embeddings toward the text distribution. (2) Downstream AAC: attaching Diffusion-Link to the same multimodal LLM baseline achieves state-of-the-art on AudioCaps in both zero-shot and fully supervised captioning without external knowledge, with relative gains up to 52.5% and 7.5%, respectively. These findings show that closing the modality gap is pivotal for effective coupling between multimodal encoders and LLMs, and diffusion-based modality bridging offers a promising direction beyond knowledge-retrieval-centric designs. Code will be released upon acceptance https://github.com/DevKiHyun/Diffusion-Link
1. INTRODUCTION
Contrastive audio–language encoders and LLMs enable strong multimodal systems, but a structural modality gap remains a concern for zero-shot and cross-modal performance. Diffusion-Link bridges audio embeddings to the text distribution and improves modality-gap measures and AAC performance.
1. INTRODUCTION
- Prior work links modality-gap magnitude to zero-shot performance and fairness, while other studies report benefits from reducing the gap for cross-modal tasks.The introduction frames addressing the gap as important for zero-shot and cross-modal performance.Source: §1 · p. 1
- Diffusion-Link treats embeddings as data and moves audio through a shared isotropic Gaussian waypoint before mapping it into the text-embedding distribution.The paper motivates diffusion as a way to learn a trajectory between modality embedding distributions.Source: §1 · p. 1
- Prior embedding-generation work covered speaker recognition and image-to-text bridging; Diffusion-Link builds on these precedents for audio–text mapping.The introduction identifies this as the first application of diffusion-based modality bridging to audio captioning.Source: §1 · p. 1
- The method trains on paired audio–text embeddings, noises both toward a shared isotropic Gaussian, and learns a text-targeted reverse process with a topology loss.At inference it produces text-like embeddings; the encoder is frozen, and the module uses three residual MLP blocks.Source: §1 · p. 1
- Diffusion-Link yields the largest gap reduction among prior methods and improves the same multimodal LLM baseline by up to 52.5% zero-shot and 7.5% supervised, reaching state-of-the-art without external knowledge.The evaluation reports gains on AudioCaps across modality-gap analysis and downstream AAC.Source: §1 · pp. 1–2
2. METHOD
Diffusion-Link uses paired audio and text embeddings from a multimodal encoder to train a denoising process that maps embeddings toward the text distribution. At inference, its text-like audio embedding can serve as a soft prefix for an LLM.
2.1. Background on Diffusion Probabilistic Models
- DDPM progressively corrupts samples with forward diffusion and trains a neural denoiser to predict the clean sample z0 at s=0.The reverse process gradually denoises samples toward the data distribution.Source: §2.1 · p. 2
2.2. Modality Gap Bridging via Diffusion-Link
- Diffusion-Link trains an encoder-output denoiser to map noised audio and text embeddings into the text-embedding distribution.The method uses paired embeddings from a multimodal encoder.Source: §2.2.1 · p. 2, §2.2 · p. 2, §2 · p. 2
2.2.1. Training Objective
- The sample-prediction objective reconstructs text-like embeddings and pushes audio embeddings toward the text distribution; a batch topology loss preserves text-distribution geometry.The topology loss is the squared Frobenius distance between text and text-like embedding similarity matrices.Source: §2.2.1 · p. 2
2.2.2. Inference to generate text-like embedding
- At inference, optional forward noising starts from an audio embedding at s*, then DDIM reverse sampling yields a text-like embedding.The figure visualizes audio–text pairs and text-like–original-text pairs on AudioCaps using UMAP.Source: §2.2.2 · pp. 2–3, Figure 2 · p. 3
2.3. LLM-based Text Decoding
- A projection head turns the text-like embedding into soft-prefix tokens for the LLM; zero-shot evaluation uses text-only decoder training.Fully supervised decoder training instead uses audio-driven text-like embeddings without an instruction prompt.Source: §2.3 · p. 3
3. EXPERIMENTAL SETTINGS
Experiments use AudioCaps to assess modality bridging and audio captioning, with a frozen LAION-CLAP encoder and a lightweight Diffusion-Link module.
3.1. Datasets
- AudioCaps provides 48,595 training clips and 944 test clips, with one training caption per clip and up to five test captions.Audio is resampled to 48 kHz; preprocessing uses 1,024-sample STFT windows and a 480-sample hop, followed by 64-bin mel-spectrograms.Source: §3.1 · p. 3
- Table 1 compares average cosine similarity for matched and non-matched embedding pairs across CLAP and four transformations.The transformations are C3, Diffusion-Bridge, DiffGap, and Diffusion-Link.Source: Table 1 · p. 3
- Table 2 reports average cosine similarity across inference forward timesteps s* on AudioCaps.Source: Table 2 · p. 3
3.2. Implementation Details and Metrics
- The frozen LAION-CLAP encoder feeds a three-residual-MLP Diffusion-Link, trained with cosine noise scheduling and sampled using five-step DDIM inference.The implementation uses batch size 128, learning rate 1×10−4, and EMA decay 0.995; AAC is evaluated with METEOR, CIDEr, SPICE, and SPIDEr.Source: §3.2 · p. 3
4. RESULTS
Diffusion-Link improves modality-bridging measures and yields state-of-the-art AudioCaps captioning results in zero-shot and supervised settings without external knowledge. Ablations indicate that excessive forward noising degrades semantic preservation and downstream performance.
4.1. Main Results
- Diffusion-Link attains the highest cosine similarity on matched pairs and the lowest on non-matched pairs, while visualizations show embeddings moving toward the text distribution.The authors contrast this with DiffGap, which underperforms because generation from pure Gaussian noise weakens information reconstruction; Diffusion-Link instead treats the input as an intermediate reverse step.Source: §4.1 · pp. 3–4
- Up to 52.5% relative improvement in zero-shot CIDEr and up to 7.3% in fully supervised captioning accompany state-of-the-art AudioCaps results without external knowledge.The comparison uses the same multimodal LLM backbone; the paper reports substantial gains on other metrics as well.Source: §4.1 · p. 4, Table 3 · p. 4, Table 4 · p. 4
4.2. Ablation Studies
- As inference forward timestep s* increases past a threshold, similarity drops sharply as over-noising erases information and harms semantic preservation.Diffusion-Bridge has similarity comparable to Diffusion-Link at s*=300–400; the authors associate this with over-noising and limited AAC improvement.Source: §4.2 · p. 4
- Across the ablation comparisons, excessive forward noising weakens bridging and downstream AAC, whereas an appropriate s* preserves content and strengthens LLM conditioning alignment.Source: §4.2 · p. 4
5. CONCLUSION
Diffusion-Link bridges audio embeddings to the text-embedding distribution with a lightweight residual MLP diffusion module. It improves the same multimodal LLM baseline on AAC without external knowledge.
5. CONCLUSION
- 52.5% and 7.3%: Diffusion-Link improves the same multimodal LLM baseline on zero-shot and fully supervised AAC, respectively, without external knowledge.Source: §5 · p. 4
- Diffusion-Link is a lightweight residual MLP diffusion module that bridges audio embeddings to the text-embedding distribution without keeping the multimodal encoder frozen.Source: §5 · p. 4
- The method aligns its conditioning input by increasing matched similarity and decreasing mismatched similarity.Source: §5 · p. 4
- The authors expect this plug-in-play approach to generalize beyond audio captioning and enable effective zero-shot performance in various multimodal LLMs.Source: §5 · p. 4