Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing

1. INTRODUCTION

• Audio in everyday videos can contain multiple sources, background noise, reverberation, and other recording artifacts that post-processing must address.Rebalancing, noise suppression, and reverberation mitigation can improve viewing experience and intelligibility.Source: §1 · p. 1
• Visually guided acoustic highlighting focuses primarily on loudness, while reconstruction-based evaluation can penalize perceptually valid remixes that differ from one ground truth.The paper frames remixing as a creative task with multiple valid outputs.Source: §1 · p. 1
• VisAH’s spectral-mask prediction struggles to replace heavily degraded target audio, while both prior approaches jointly train the codec and backbone and lack user control beyond video.VisAH-FM’s rollout loss also requires full generation for every training sample.Source: §1 · p. 1
• SSE combines multimodal, user-guided generation with the ability to replace severely degraded target-source regions, and evaluates remixes by perceptual quality and effectiveness.The evaluation also uses controlled degradations to test challenging acoustic conditions.Source: §1 · p. 1
• SSE outperforms VisAH in controllability and quality under generative and reconstruction-based metrics, though results are mixed under single-ground-truth metrics.The paper also introduces DegradedMix and an evaluation framework centered on perceptual quality and remix effectiveness.Source: §1 · p. 1

2. METHOD

• SSE separately encodes modalities, aligns and fuses audio and visual features, then uses text-guided generation to produce waveform audio for remixing and enhancement.It can rebalance the mix, remove unwanted sources, and generate replacement audio for degraded content.Source: §2 · p. 1

2.1. Neural Audio Codec – SkipDACVAE

• SkipDACVAE adds encoder-to-decoder skip connections to DACVAE, giving the decoder direct access to original audio features at matching temporal resolutions.Each connection uses a 1D convolution to map encoder features to the decoder’s channel dimension.Source: §2.1 · pp. 1–2

2.2. Visual Encoder

• The visual encoder is a pretrained Perception Encoder that produces per-frame features representing video content.Its contrastive vision-language training supports representations of actions and scene context.Source: §2.2 · p. 2

2.3. Text Encoder

• A pretrained T5-base encoder produces text features, which are projected to the generative model’s channel dimension before cross-attention.The text sequence comes from the encoder’s last hidden layer.Source: §2.3 · p. 2

2.4. Generative Conditional Flow Matching Model

• The conditional flow-matching model learns a vector field that transports a Gaussian prior sample to a target audio-and-residual sample, conditioned on mixture audio, video, and text.Visual features are aligned to 25 Hz and added through a gated sum; the model uses a Diffusion Transformer with flow-time modulation.Source: §2.4 · p. 2
• Training samples one of four flow timesteps, and inference uses Euler integration with the same number of steps; the model starts from pretrained SAM-Audio weights.The reported best results used selective training of attention projections, the modality aligner, and the text projection.Source: §2.4 · p. 3

3.1. Dataset – DegradedMix

• DegradedMix extends MuddyMix with background noise and reverberation and adds audio-based captions; evaluation on both datasets supports standard and more challenging degradation settings.MuddyMix itself does not support background-noise removal because its degraded and ground-truth mixes contain the same stems.Source: §3.1 · p. 3

3.2. Compared Methods

• The comparisons include VisAH, a discriminative spectral-mask method, and VisAH-FM, a conditional-flow-matching method that was unavailable for evaluation.VisAH-FM is described as a closed-source concurrent work.Source: §3.2 · p. 3

3.3. Implementation and Training Details

• Training uses 25 FPS video, 48 kHz audio, and a 10-second context; degraded audio is generated on the fly with gain changes, noise, and reverberation.The model was trained for approximately 67K steps with batch size 26 on two NVIDIA RTX PRO 6000 Blackwell GPUs.Source: §3.3 · p. 3

3.4. Evaluation Metrics

• Evaluation replaces single-target reconstruction distances with generative-modeling metrics for distribution matching, audio quality, semantic alignment, synchronization, and language-audio alignment.Distribution matching uses PANNs and PaSST features; PANNs is also used for Inception Score.Source: §3.4 · p. 3

3.5. Results

• On proposed metrics, SSE beats retrained VisAH except PaSST FD on DegradedMix and all compared MuddyMix methods except PaSST FD, with the best overall balance.VisAH-FM could not be evaluated for these comparisons.Source: Table 1 · p. 3, §3.5 · pp. 3–4, Table 3 · p. 4
• SSE-L was preferred in 81% of UGC trials; VGAH metrics favor SSE over VisAH on DegradedMix but favor VisAH-FM on three MuddyMix measures.On MuddyMix, SSE has the best KLPaSST score, and subjective preference favors SSE-L over VisAH despite marginally better or comparable VGAH scores.Source: Table 2 · p. 3, Table 5 · p. 4, §3.5 · p. 4

3.6. Ablations

• Ablations find the 3B model performs best on most metrics and skip connections improve all metrics; caption variants include audio descriptions intended to mimic user guidance.The tested model sizes are 500M, 1B, and 3B parameters; the SkipDACVAE ablation was not retrained because the codecs share a latent space.Source: §3.6 · p. 4, Table 6 · p. 4, Table 7 · p. 4

• SSE outperforms VisAH in controllability and quality under both generative and reconstruction-based metrics.Source: §4 · p. 4
• Results are more mixed under single-ground-truth metrics, underscoring remixing’s one-to-many nature and the need for evaluation that embraces it.Source: §4 · p. 4
• The authors hope DegradedMix and their evaluation framework will support this shift in how remixing is evaluated.Source: §4 · p. 4