Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners

1. Introduction

• Single-modality generation leaves videos without accompanying audio and audio without synchronized visuals, limiting users’ ability to create content such as films.Earlier V2A and A2V methods are described as domain-limited or lower-performing, while prior Joint-VA work is described as limited in domain and semantic control.Source: §1 · p. 1
• The paper proposes using pretrained ImageBind to connect strong, pretrained single-modality generators through a shared semantic space, avoiding expensive training to synthesize each modality.The authors identify ImageBind’s ability to connect modalities in a shared semantic space as the basis for this approach.Source: §1 · p. 1
• During denoising, an aligner uses a noisy latent and another modality’s condition to guide generation toward that condition in ImageBind space; Joint-VA applies guidance bidirectionally.The overview identifies Joint-VA, V2A, A2V, and I2A as supported tasks.Source: §1 · p. 1, Figure 1 · p. 2
• The authors present the approach as a versatile, flexible system that bridges pretrained single-modality models without requiring large-scale dataset training.They describe the system as an organic combination of pretrained generators and report evaluation on four tasks.Source: §1 · pp. 1–2
• The method reports superiority over baseline approaches across four tasks: V2A, I2A, A2V, and Joint-VA.The paper presents the approach as training-free with respect to large-scale datasets.Source: §1 · p. 2, Figure 1 · p. 2

2.1. Conditional Audio Generation

• Conditional audio work spans text-, image-, and video-conditioned generation, using autoregressive or diffusion approaches, language-model enhancements, and visual-feature-based audio synthesis.Examples include Im2Wav’s language-model audio representation and SpecVQGAN’s video-feature-conditioned spectrogram generation.Source: §2.1 · p. 2

2.2. Conditional Visual Generation

• Conditional visual work includes audio-to-image and text-to-video generation; cited audio-to-video approaches vary in synchronization and video variety.The section describes TATS as producing synchronized video with significantly constrained variety, while Tempotokens generates video from audio input.Source: §2.2 · pp. 2–3

2.3. Multimodal Joint Generation

• Prior joint-generation systems include unconditional MM-Diffusion, limited to its training-set domain, and MovieFactory, which retrieves audio rather than generating scene-specific audio.MovieFactory expands text prompts into sequential scripts used for visual generation and audio retrieval.Source: §2.3 · p. 3

3.1.1 Latent diffusion models

• Latent diffusion encodes data into z, runs diffusion and denoising in latent space, then decodes the denoised latent into a sample.The forward process adds noise over T timesteps; a trained denoiser estimates less noisy latents during the backward process.Source: §3.1.1 · p. 3

3.1.2 Classifier guidance

• Classifier guidance conditions an unconditional diffusion model on a desired category using a classifier-derived gradient during denoising.The time-aware classifier approximates the label distribution for each noisy sample.Source: §3.1.2 · p. 3

3.1.3 Linking multiple modalities

• ImageBind aligns modalities by embedding paired data in a shared semantic space, minimizing distances for matching pairs and maximizing them for different pairs.Its contrastive objective uses a temperature factor τ to control Softmax smoothness and treats data from another pair as a negative sample.Source: §3.1.3 · pp. 3–4

3.2.1 Problem formulation

• For each modality, the aligner takes a noisy latent and another modality’s condition, then returns a modified latent better aligned with that condition.For joint visual-audio generation, guidance should draw information from both modalities and guide both latents.Source: §3.2.1 · p. 4

3.2.2 Multimodal guidance

• At each denoising step, ImageBind-based guidance uses a clean prediction and the condition to adjust the latent, without retraining the external model.The approach adds a small amount of sampling time and requires no additional datasets or expensive network training.Source: §3.2.2 · pp. 4–5

3.2.3 Dual/Triangle loss function

• The joint-VA loss combines pairwise distances among video, audio, and text embeddings; the added text signal addresses audio’s often incomplete semantics.The authors report that their approach helps correct semantic errors in extracted captions and improves semantic alignment.Source: §3.2.3 · p. 5

3.2.4 Guided prompt tuning

• Because latent guidance has negligible effect for audio-to-visual generation and can reduce temporal consistency, the authors optimize the generative model’s text embedding.The updated embedding is shared across timesteps to provide consistent semantic guidance.Source: §3.2.4 · p. 5

4.1. Experimental Setup

• Evaluation uses VGGSound and Landscape: 3k samples each for video-to-audio, audio-to-video, and image-to-audio, plus 200 Landscape pairs for joint generation.AudioLDM is used for video-to-audio and image-to-audio, AnimateDiff for audio-to-video, and both for joint generation; denoising uses 30, 25, and 25 steps, respectively.Source: §4.1 · pp. 5–6

4.2. Baselines

• SpecVQGAN, Im2Wav, and TempoTokens serve as video-to-audio, image-to-audio, and audio-to-video baselines; joint generation uses MM-Diffusion on Landscape and guided-versus-vanilla comparison on the open domain.Vanilla pipelines caption the conditioning video or audio before synthesis; vanilla joint generation separately feeds the test prompt to AudioLDM and AnimateDiff.Source: §4.2 · pp. 6–8

4.3. Visual-to-Audio Generation

• Training-free visual-to-audio generation outperforms the large-scale-trained baseline and improves audio-video relevance and audio quality over text-to-audio; guidance reduces irrelevant sound and background noise.Figure 3 contrasts SpecVQGAN's failure to generate realistic, aligned audio with the method's rhythm-aligned audio.Source: Figure 3 · p. 6, §4.3 · p. 8, Figure 6 · p. 8

4.4. Audio-to-Video Generation

• The training-free method outperforms the training-based baseline in semantic alignment and video quality, and achieves better audio-video alignment than text-to-video with comparable visual quality.Figure 5 contrasts TempoToken's poor visual quality and audio alignment with the method's visually better, semantically aligned output.Source: Figure 5 · p. 7, §4.4 · p. 8

4.5. Joint Video and Audio Generation

• Against MM-Diffusion, the aligner boosts audio quality while maintaining video performance; text-conditioned joint generation improves text-video, text-audio, and video-audio alignment over separate models.Figure 4 qualitatively compares the generated visual alignment and audio quality with the vanilla model.Source: Figure 4 · p. 7, §4.5 · p. 8

4.6. Limitations

• Performance is limited by AudioLDM and AnimateDiff; for A2V and Joint-VA, visual quality, complex concept composition, and complex motion remain improvement targets.The authors note that the method can adopt more powerful generative models in the future.Source: §4.6 · p. 9

• Experiments on several evaluation datasets show advantages, especially improved audio generation fidelity and audio-visual alignment.Source: §5 · p. 9
• The method supports video-to-audio, audio-to-video, joint video-audio, image-to-audio, and audio-to-image generation.Source: §5 · p. 9
• It uses ImageBind’s shared multimodality embedding space to bridge pretrained visual and audio diffusion models instead of training giant models from scratch.Source: §5 · p. 9