Source-linked AI summary
Decoupled Latent Flow Matching for Few-Step Joint Vocal-Accompaniment Separation
Lishi Zuo, Youzhi Tu, Lu Yi, Zezhong Jin, Chongxin Gan, Man-Wai Mak, KongAik Lee
TL;DR
The paper asks whether joint latent flow matching can efficiently model mixture-conditioned vocal-accompaniment separation despite the cost of iterative generation for long music signals. It uses a pretrained VAE, semantic-acoustic decoupling, and latent Flow2GAN-style adversarial post-training, with experiments showing improved perceptual and separation metrics under reduced sampling budgets.
Problem
Iterative diffusion and flow matching are costly for long music signals, motivating an efficient generative formulation for mixture-conditioned vocal-accompaniment separation.
Method
The method jointly generates vocal and accompaniment latents in a pretrained VAE space, decouples separation encoding from velocity decoding, and applies latent adversarial post-training for few-step generation.
Results
Latent adversarial refinement improves perceptual and separation metrics under a reduced sampling budget.
Takeaways & Limitations
Latent refinement provides a few-step generation strategy while jointly modeling vocal and accompaniment sources.
Abstract
from arXiv · showhide
Generative modeling provides a flexible way to model mixture-conditioned source distributions, but iterative diffusion and flow matching models are costly for long music signals. This paper studies joint vocal-accompaniment separation through latent flow matching, where a pretrained variational autoencoder (VAE) maps mixtures and sources into a compact latent space and a flow matching model generates vocal and accompaniment latents jointly. The proposed framework decouples semantic separation from acoustic velocity prediction through a Separation Encoder and a Velocity Decoder. To reduce sampling cost, we further apply latent adversarial post-training inspired by Flow2GAN for few-step generation. Experiments show that latent adversarial refinement can improve perceptual and separation metrics under a reduced sampling budget.
I. INTRODUCTION
The paper frames vocal-accompaniment separation as joint generative modeling because the sources share musical structure, then moves generation into VAE latents to address waveform-level cost. It separates source representation extraction from acoustic velocity prediction and adds latent adversarial post-training for few-step generation.
- Motivation: Vocal and accompaniment are treated as correlated sources because they share temporal, rhythmic, harmonic, and structural patterns.The joint formulation is intended to preserve cross-source dependencies and expose quality trade-offs between output streams.
- Few-step generation: Latent adversarial post-training extends Flow2GAN-style refinement to reduce the iterative sampling cost of latent flow matching.A latent discriminator targets real source-latent structure while latent reconstruction preserves correspondence to target sources and source balance.
- Joint latent modeling: Joint latent flow matching generates vocal and accompaniment simultaneously within a shared pretrained VAE latent space.The VAE maps mixtures and target sources into latent representations before joint generation and waveform decoding.
- Semantic-acoustic decoupling: Semantic-acoustic decoupling separates source-specific representation extraction from acoustic velocity prediction using a Separation Encoder and a Velocity Decoder.The design keeps both sources jointly modeled while assigning different roles to source information separation and acoustic state evolution.
II. RELATED WORK
Prior music separation work is dominated by discriminative predictors, while diffusion, flow matching, and latent modeling provide generative alternatives for complex distributions and more efficient audio generation.
- Discriminative separation: Discriminative separators such as Open-Unmix, Conv-TasNet, and Demucs directly estimate target sources from mixtures.These methods span spectrogram-domain, time-domain convolutional, and hybrid or waveform-domain approaches.
- Generative modeling: Diffusion and score-based models represent complex distributions, while flow matching learns continuous-time velocity fields.The paper positions flow matching as a generative framework for audio source modeling.
- Efficient generation: Latent generative modeling reduces generation cost by moving the process into a compact latent space.Few-step methods further target inference cost through consistency or adversarial distillation and related formulations.
III. METHODOLOGY
The proposed framework combines latent encoding, joint source-latent generation, semantic-acoustic decoupling, and latent adversarial post-training.
- The framework includes latent encoding, joint source-latent generation, semantic-acoustic decoupling, and latent adversarial post-training.
A. Joint Latent Modeling for Vocal-Accompaniment Separation
The method formulates separation as mixture-conditioned joint generation of vocal and accompaniment latents in a pretrained VAE space, starting from Gaussian noise and decoding generated latents into waveforms.
- Input representation: The mixture waveform is represented as the sum of vocal and accompaniment waveform vectors.
- Latent representations: A pretrained VAE encodes the mixture and each target source into latent representations for joint source modeling.The VAE encoder and decoder define the latent-space representation and subsequent waveform reconstruction.
- Joint flow: The joint flow transforms initial Gaussian noise into concatenated vocal-and-accompaniment target latents.The source latents share a joint representation with dimensions 2C × L.
- Conditional generation: The velocity field is conditioned on the mixture latent, and numerical integration produces source latents that DecVAE decodes into waveforms.
B. Semantic-Acoustic Decoupling
The framework separates source-specific representation extraction from acoustic velocity prediction while keeping vocal and accompaniment generation within one shared latent flow. Representation Alignment for Generation further encourages the separation representations to capture vocal-relevant semantics.
- Semantic-acoustic decoupling: The Separation Encoder extracts source-specific representations from the mixture for conditioning joint source generation.The Velocity Decoder uses these representations to predict the acoustic velocity field.
- Semantic-acoustic decoupling: The Velocity Decoder predicts the acoustic velocity field conditioned on Separation Encoder representations.
- Semantic representation alignment: Frozen MERT embeddings from the isolated vocal stem supervise Separation Encoder hidden representations through Representation Alignment for Generation.
A. Latent Encoding
A frozen VAE provides the latent representation used by the framework, while the model aligns separation features with vocal MERT embeddings and combines flow matching with representation alignment.
- Latent encoding: A frozen VAE encodes the mixture and target sources into mixture and joint source latents.
- Latent decoding: The framework uses a frozen VAE decoder to map generated source latents back to vocal and accompaniment waveforms.
- Representation alignment: The Separation Encoder hidden representation H is projected to the MERT feature dimension by gψ and normalized for alignment.
- Training objective: The final training objective combines flow matching with representation alignment, whose contribution is controlled by λREPA.
C. Adversarial Post-Training in the Latent Space
Latent adversarial post-training adapts an already trained conditional flow separator for designated few-step sampling budgets. It combines adversarial latent refinement with reconstruction and uses a shared discriminator for both sources.
- Post-training strategy: Flow2GAN-style post-training fine-tunes the flow-matching generator with an adversarial objective without redesigning the underlying flow-matching target.
- Latent adversarial training: A single shared one-dimensional convolutional discriminator processes both vocal and accompaniment source latents, with source-specific weights balancing their contributions.
- Generator objective: The generator objective combines adversarial and latent ℓ1 reconstruction terms, with coefficients λadv and λℓ1 controlling their contributions.
- Few-step generation: For each target sampling step N, an N-step generator is optimized end-to-end when N > 1.This jointly refines intermediate states and the final output for the designated sampling budget.
- Few-step generation: Each N-step generator is optimized for the trade-off between generation quality and sampling efficiency at its designated number of steps.
IV. EXPERIMENTAL SETUP
The experiments evaluate generative vocal-accompaniment separation on held-out song segments using perceptual, spectral, and conventional separation metrics. Training includes staged flow matching and latent or waveform-domain adversarial post-training comparisons.
- Data: Evaluation uses 50 non-overlapping 20-second segments held out from the training set.Backing vocals are treated as accompaniment in the training data.
- Evaluation metrics: ViSQOL measures perceptual quality, MR-STFT loss measures spectral fidelity, and SDR provides a conventional separation reference.
- Evaluation metrics: Higher ViSQOL and SDR indicate better quality, whereas lower MR-STFT loss indicates better spectral reconstruction.
- Training setup: The initial flow-matching model is pretrained for 135K steps and then further trained for 40K steps under alternative freezing settings.
- Training setup: Latent-domain adversarial post-training freezes the Separation Encoder and optimizes the Velocity Decoder for 40K steps.
- Training setup: The study compares latent-domain adversarial training with a waveform-domain Flow2GAN baseline using waveform discriminators after VAE decoding.
V. RESULTS
The results show that latent adversarial refinement reduces sampling steps while preserving or improving separation quality, with outcomes depending on refinement domain and training schedule.
- Main comparison: One-step Latent Flow2GAN raises vocal ViSQOL from 3.717 to 3.826 and vocal SDR from 4.693 dB to 5.895 dB versus 20-step Flow Matching.Accompaniment ViSQOL decreases slightly from 3.692 to 3.676.
- Main comparison: Four-step Latent Flow2GAN improves ViSQOL for both sources, although its SDR gain is not consistent.The comparison indicates a quality–efficiency trade-off under reduced sampling budgets.
- Latent versus waveform-domain refinement: Latent-domain adversarial refinement consistently outperforms waveform-domain refinement for both sources across ViSQOL, MR-STFT loss, and SDR.The comparison uses the same decoupled latent flow matching model under one-step generation.
- Decoupled training strategy: A 40K-step schedule that freezes the Separation Encoder improves vocal ViSQOL from 3.693 to 3.717, vocal SDR from 4.169 dB to 4.693 dB, and accompaniment SDR from 6.951 dB to 7.296 dB.The schedule continues training only the Velocity Decoder after flow matching pretraining.
- Decoupled training strategy: Continuing joint training degrades ViSQOL and MR-STFT loss for both sources and substantially lowers accompaniment SDR.These results indicate interference with the learned source representation during prolonged joint updates.
VI. CONCLUSION
The paper formulates vocal-accompaniment separation as joint latent flow matching with decoupled source representation and acoustic velocity prediction, followed by latent Flow2GAN-style post-training for few-step generation. Experiments show that latent adversarial post-training improves performance with fewer sampling steps.
- The framework combines joint latent flow matching, decoupled source representation and acoustic velocity prediction, and latent Flow2GAN-style post-training for few-step generation.