High-Quality Visually-Guided Sound Separation from Diverse Categories

1 Introduction

• Mask-regression methods struggle when sounds interact in complex ways and competing sounds heavily suppress the desired sound.The paper frames this as a challenge for recovering complex time-frequency patterns from hidden sound patterns.Source: §1 · pp. 1–2
• Figure 1 illustrates DAVIS recovering suppressed time-frequency structures that mask-regression methods fail to recover.The comparison includes DAVIS, iQuery, and CCoL.Source: Figure 1 · p. 2
• DAVIS iteratively generates a magnitude spectrogram from Gaussian noise, using a Separation U-Net and visual-feature interaction to model audio-visual correspondences.The Convolution-Attention block captures local and non-local contexts, while the Feature Interaction module injects visual cues.Source: §1 · pp. 2–3
• DAVIS uses a robust L1 training loss and silence mask-guided sampling to refine separation during inference.Near-zero silent frames can indicate overlap between the mixture and target sounds, but make conventional diffusion L2 loss susceptible to error.Source: §1 · p. 3
• DAVIS consistently outperforms state-of-the-art methods in separation quality on AVE and MUSIC.The paper also reports competitive or better performance on datasets with diverse sound categories.Source: §1 · p. 3

2 Related Work

• Audio-visual separation methods commonly train on mixtures of two videos and use visual features to condition a U-Net that predicts a mask.Prior work spans speech, musical instruments, and universal sound sources; some methods also require text queries, motion cues, or class labels.Source: §2 · p. 3
• DAVIS starts from a standard-distribution latent and uses a Separation U-Net conditioned on the mixture, visual features, and timestep to denoise toward the separated sound.The visual feature is used in the Feature Interaction Module, while the timestep conditions the denoising process throughout.Source: Figure 2 · p. 4
• Diffusion has been applied across several domains, but prior audio-visual diffusion work focused on speech separation or enhancement rather than broader sound separation.The paper identifies architecture for audio-visual correspondence and adaptation to magnitude-spectrogram distributions as challenges.Source: §2 · p. 4

3 Method

• Section 3 introduces DAVIS and outlines diffusion preliminaries, task setup, the separation framework, training, and silence mask-guided sampling.Source: §3 · pp. 4–5

3.1 Preliminaries: Diffusion Models

• Diffusion adds scheduled noise to spectrograms and learns a reverse Markov chain conditioned on mixture and visual context, trained to predict injected noise.The reverse process gradually removes noise to recover a separated sample.Source: §3.1 · p. 5

3.2 Task Setup and Method Overview

• Using the “mix and separate” strategy [27], training maps an STFT mixture plus one video's visual stream to that source's magnitude spectrogram.The audio magnitudes are transformed to a log-frequency scale, scaled, and clipped to [0, 1].Source: §3.2 · p. 6, §3.4 · p. 8

3.3 Proposed DAVIS Framework

• The Separation U-Net pairs encoder/decoder CA blocks for local and long-range audio modeling with temporally aggregated visual cues and a bottleneck interaction module.The interaction module combines audio and visual feature maps to support learning their associations.Source: §3.3 · pp. 7–8

3.4 Training Pipeline

• The authors use an L1 noise-prediction loss, hypothesizing greater robustness to spectrograms with many near-zero regions.Both mixed videos contribute training examples, with noise and a timestep sampled for the denoising objective.Source: §3.4 · pp. 8–9

3.5 Silence Mask-Guided Sampling

• At each reverse step, silence-mask-guided sampling replaces mixture-silent regions with mixture values, using 0.002 as the silence threshold.The refined sample is passed to the next iteration; the authors state this reduces hallucinated content and improves separation.Source: §3.5 · p. 9

4.1 Experimental Setup

• The experiments use AVE's diverse, noisy open-domain videos and MUSIC's clean solo-instrument videos, comparing DAVIS with discriminative baselines using SDR, SIR, and SAR.The baselines include iQuery, described as the current state-of-the-art; SAR measures artifact absence.Source: §4.1 · pp. 9–11

4.2 Comparisons with State-of-the-art

• On MUSIC, DAVIS consistently outperforms existing methods across metrics; on AVE, it matches iQuery's SDR and improves SIR by 0.9 dB.AVE metrics may be affected by off-screen sounds and background noise.Source: Table 1 · p. 10, §4.2 · pp. 11–12

4.3 Experimental Analysis

• Ablations favor CA blocks combining local and global context and feature interaction over naive fusion; Fig. 5 illustrates accurate association in a baby-and-truck mixture.The baby video also contains a running-car sound, yet DAVIS extracts crying and removes irrelevant sounds.Source: §4.3 · pp. 12–13, Figure 5 · p. 12
• Silence-mask sampling can improve post-training separation when its threshold is selected carefully, and DAVIS wins 82.5% of 176 subjective judgments.The analysis also presents a challenging real-world three-source example.Source: §4.3 · pp. 13–14, Figure 6 · p. 13

4.4 Application: Zero-Shot Text-guided Separation

• DAVIS supports qualitative zero-shot text-guided separation by replacing visual conditioning with text embeddings, including an example filtering out speech.The text description corresponds to the image, using CLIP's shared image-text embedding space.Source: §4.4 · p. 14, Figure 7 · p. 14

• DAVIS uses a T-step diffusion model to iteratively synthesize separated magnitude spectrograms conditioned on visual input.The framework pairs a specialized Separation U-Net with a novel sampling strategy.Source: §5 · p. 14
• Experiments on MUSIC and AVE validate DAVIS's effectiveness in separating sounds in specific and open domains.Source: §5 · p. 14
• The authors conclude that DAVIS can handle diverse time-frequency structures while producing high-quality sound separation results.Source: §5 · p. 14