Source-linked AI summary
Music Source Separation via Stem Discovery
V. Valtteri Kallinen, Eloi Moliner, Lauri Juvela, Vesa Välimäki
TL;DR
MuS3D addresses the limits of fixed stem taxonomies and the need for manual queries in query-based separation. It recursively discovers stems from the mixture, and on correctly detected sources its generated queries match ground-truth-query separation; subjective results also identify the audio autoencoder as a limiting factor in perceived quality.
Problem
Fixed stem taxonomies do not scale to diverse sources, manual queries constrain query-based separation, and prior blind-separation work does not address music mixtures.
Source: §1 · p. 1Method
MuS3D recursively generates stem embeddings from the mixture and previously generated queries, then uses them to condition a query-based separator without user-supplied text or audio examples.
Source: §5 · p. 4Results
On correctly detected sources, generated-query separators perform on par with ground-truth-query separators in objective metrics and formal listening, while SEG outperforms a language-model source-identification baseline.
Source: §5 · p. 4Takeaways & Limitations
Audio queries can be generated automatically without loss in separation quality, supporting music separation when mixture stems are not known in advance.
Source: §5 · p. 4Evaluation is limited to 10-second segments, leaving query consistency across whole songs open; SEG also misses or misclassifies some stems, and inference is computationally expensive.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Music source separation (MSS) methods aim to extract stems from music mixtures, which is important, for example, in karaoke, music remixing, and pedagogical applications. While earlier research on MSS systems has been dominated by models targeting narrow sets of general stems, there have recently been attempts to support broader source definitions. One of these methods uses audio queries to provide direct and descriptive control over the desired separation targets based on the sound itself. However, query-based separation remains cumbersome due to the need to provide audio examples with features matching the sources contained within mixtures. This paper proposes Music Source Separation via Stem Discovery (MuS3D), a query-based source separation framework that iteratively discovers active sources from the mixture. On correctly detected sources, our model matches manually queried baselines and surpasses state-of-the-art text-based models. Subjective evaluation indicates encoding artifacts as the limiting factor in current generative separation. The findings suggest that audio-based query representations offer an effective and automatable interface for source separation.
1. INTRODUCTION
Music source separation is useful for applications such as karaoke and remixing, but fixed stem taxonomies do not scale well and manual queries hinder automation. MuS3D automatically discovers stem embeddings from a mixture and uses them as separation queries.
1. INTRODUCTION
- Static MSS systems commonly target four fixed stems, while expanding the vocabulary creates target sparsity and still misses sources that do not fit instrument taxonomies.The four targets are vocals, drums, bass, and other (VDBO).Source: §1 · p. 1
- Existing query-based systems can target broader sources, but suitable manual queries are difficult to provide and limit large-scale automation [14].Source: §1 · p. 1
- Blind separation seeks all mixture sources without knowing their identities or count, but prior approaches do not address music’s strong temporal and harmonic source correlations.Source: §1 · p. 1
- MuS3D generates embeddings for constituent stems and uses them as conditioning queries, separating fine-grained instruments without external side information or manual queries.Source: §1 · p. 1
2. METHODS
Query-based separation uses a source-descriptive embedding to condition a separator. MuS3D’s SEG recursively generates stem queries from the mixture and prior queries, with a residual signal that supports termination.
2. METHODS
- Source separation decomposes a mixture into underlying components, which in music may be vocal or instrumental tracks or groups of stems.Source: §2 · p. 1
2.1. Query-based source separation
- A query-based separator takes the mixture and a source-descriptive query, then estimates the target; queries are time-invariant feature embeddings, with target specificity as a design principle.The ideal query would maximize information about its target while minimizing information about interfering sources, though this criterion is not explicitly enforced.Source: §2.1 · p. 2
2.2. Recursive stem embedding generation
- SEG recursively estimates one query embedding at a time from the mixture and previously generated queries, producing a query and residual embedding at each step.The flow model obtains this pair by solving an ODE from t = 1 to t = 0.Source: §2.2 · p. 2
- When no sources remain, SEG emits a fixed stop symbol in the residual slot and halts when its normalized residual meets the stopping threshold.Source: §2.2 · p. 2
- Training samples a recursion depth uniformly and supervises the target query and remaining-source residual, or the stop symbol at the final depth.This covers every recursion stage and termination without unrolling; step cost is independent of source count, and shuffled stem order makes inference order random.Source: §2.2 · p. 2
3. EXPERIMENTS
The experiments use a music-stem dataset organized at group and instrument levels, with PaSST embeddings as query descriptors. They describe the SEG and separator implementations and their training and inference settings.
3. EXPERIMENTS
- Experiments use MoisesDB group and instrument categories, supplemented with MedleyDB and AlbumDB; 279 songs train the models, 48 test them, and 26 validate them.Training uses processed stems and excludes MoisesDB songs with bleeding tracks; segments are augmented, while the 1,062 fixed-grid test segments are not.Source: §3 · p. 2
3.1. Query descriptor
- Query descriptors are PaSST embeddings reduced by PCA to 128 dimensions, retaining 0.96 of the explained variance.PaSST uses 10-second mono audio downsampled to 32 kHz.Source: §3.1 · p. 2
3.2. Stem embedding generator
- SEG is a roughly 37M-parameter, 12-layer diffusion transformer conditioned on mixture representations, previous queries, flow time, recursion step, and hierarchy level.It is trained for 400k iterations and uses a 32-step stochastic flow sampler at inference; recursion is capped at 16 steps.Source: §3.2 · pp. 2–3
3.3. Query-based separator models
- The query-based separator models are DiT-Sep, adapted from SAM-Audio and fine-tuned on the training data, and a query-conditioned BS-Locoformer.DiT-Sep has roughly 2.2B trainable parameters; the models use different batch sizes and training iteration counts.Source: §3.3 · p. 3
4. EVALUATION
The evaluation measures stem discovery and query-conditioned separation with objective metrics and listening tests. MuS3D performs strongly on correctly detected sources, while the listening results identify the autoencoder as a limit on perceived quality.
4.1. Stem discovery
- SEG has the highest precision and lowest ΔK at both levels, with F1 of 0.88 vs. 0.89 for groups and 0.74 vs. 0.75 for instruments.MusicFlamingo’s F1 falls from 0.82 at group level to 0.33 at instrument level; conditioning SEG on the previous residual degrades all metrics, attributed to exposure bias.Source: Table 1 · p. 3, §4.1 · p. 3
4.2. Source separation objective evaluation
- MuS3D (DiT-Sep) achieves the highest Judge scores at both levels (4.7 and 4.6), on par with ground-truth stems.Both separators perform nearly as well with SEG-generated as with oracle queries; MERT-MSE increases by at most 0.01, and HTDemucs trades a lower Judge score for higher SNR.Source: §4.2 · pp. 3–4, Table 2 · p. 4
4.3. Source separation subjective evaluation
- The subjective test compares separated vocals, bass, drums, guitar, and piano using 4-s excerpts, with 13 experienced participants.All excerpts were from stems detected by both SEG and MusicFlamingo, so separation errors were attributable to the separator.Source: §4.3 · p. 4
- MuS3D (DiT-Sep) significantly outperforms HTDemucs and SAM-Audio in listening tests, matches oracle and autoencoded ground-truth stems, but loses to original stems.The authors identify the autoencoder, rather than separation or query generation, as the limiting factor.Source: Figure 2 · p. 4, §4.3 · p. 4
5. CONCLUSION
MuS3D automatically discovers mixture stems with SEG-generated audio queries and uses them to condition a separator, removing the need for user-provided text or audio examples. For correctly detected sources, generated queries match ground-truth-query separation in objective metrics and formal listening; the audio autoencoder remains the primary perceived-quality limit for DiT-Sep.
5. CONCLUSION
- SEG recursively generates one embedding per stem, conditioned on the mixture and previously generated embeddings, to query a separator without user-supplied text or audio examples.Source: §5 · p. 4
- Generated-query separation matches GT-query separation on correctly detected sources in objective metrics and formal listening, while SEG outperforms a language-model source-identification baseline.Source: §5 · p. 4
- Listening tests identify the audio autoencoder as the primary limit on perceived DiT-Sep quality.Source: §5 · p. 4
- SEG still misses or misclassifies stems, evaluation covers only 10-second segments, and the sampling-plus-separation inference pipeline is computationally expensive.Whole-song consistency of queries remains open; flow maps are noted as a possible acceleration route.Source: §5 · p. 4