Source-linked AI summary
Retrieving Individual Stems from Music Mixtures with Slot Embeddings
David Braun, Junyi Fan, Pranay Manocha, Donald S. Williamson, Adam Finkelstein
TL;DR
Music retrieval systems search stem libraries using audio embeddings, but a single mixture embedding combines information from its different stems and CIR works best with a specified instrument family. Stembed represents a mixture with stem-specific slot embeddings trained through matching to isolated stems. On MoisesDB, it outperforms CIR–MuQ when searching all stems, including when predicting stem count without family labels.
Problem
A single embedding aggregates a mixture’s stems, while CIR benefits from a specified instrument family that requires skilled human input.
Source: §1 · p. 1Method
Stembed learns stem-specific mixture slots by matching them to solo embeddings and training contrastive representations of the assigned stem identities.
Source: §1 · p. 1, §2 · pp. 1–2Results
56.8% All R@1 for Stembed versus 23.6% for CIR–MuQ; without a supplied stem count, Stembed reaches 56.0 ± 1.3% All R@1.
Source: §4 · p. 4, §4.1 · p. 4Takeaways & Limitations
On MoisesDB, Stembed’s slot embeddings achieve recall comparable to family-guided CIR–MuQ without instrument-family labels.
Source: §5 · p. 4Evaluating perceptual similarity when the target is absent from the database remains future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Music producers search libraries of isolated instrument recordings, called stems, for sounds resembling parts of an existing song. Neural retrieval systems address this by mapping audio to embeddings and ranking library stems by their similarity to the query. The leading method, Contrastive Instrument Retrieval (CIR), encodes the mixture as a single embedding, but it works best when a user specifies the target's instrument family. We introduce Stembed, which encodes a mixture as several slot embeddings representing candidate stems. During training, we construct mixtures from stems of the same song and match their slot embeddings to those of the isolated stems. The slot embeddings from mixtures inherit the stem identities of their assigned solo embedding, enabling a contrastive loss. On mixtures from held out MoisesDB artists, Stembed outperforms a CIR-style baseline when both search the full stem library. Even when predicting the stem count itself without family labels, Stembed exceeds the baseline's family-filtered R@1. Our website demonstrates how users can select a slot by inspecting the tags of its retrieved stems.
1. INTRODUCTION
Music retrieval models rank library stems by comparing audio embeddings, but a single mixture embedding combines information from all its stems. Stembed instead produces stem-specific slots, aiming to retrieve without requiring users to specify an instrument family.
1. INTRODUCTION
- Earlier mixture-retrieval approaches include source separation followed by retrieval, SMTI’s prediction of frozen solo embeddings, and slot-based audio models.Prior slot models address speaker mixtures, synthesized notes, or source-centric audio representations.Source: §1 · p. 1
- CIR’s single mixture embedding competes across multiple instruments, so retrieval benefits from restricting searches to a specified instrument family.Choosing that family requires skilled human intervention and limits automation.Source: §1 · p. 1
- Stembed learns stem-specific slot embeddings from multitrack recordings and matches mixture slots to solo embeddings so assigned slots inherit stem identities for contrastive training.Its demo suggests users can inspect retrieved tags or filenames to choose a slot rather than name an instrument family.Source: §1 · p. 1
- On MoisesDB, Stembed achieves higher All R@1 than family-guided CIR–MuQ without instrument-family guidance.The paper also compares against SMTI-style objectives and pipelines that separate sources before encoding.Source: §1 · p. 1
2. METHOD
Stembed trains mixture slots to represent individual stems by matching them to solo embeddings and applying contrastive and presence-prediction objectives. Its slot decoder produces candidate retrieval embeddings and scores slots for selection at inference.
2. METHOD
- Each training batch forms ten audio nodes from four same-song stems—four solos, two pairs, and four triples—so shared stems connect nodes in a graph.Each node uses its own 5-second song window; all nodes are normalized to −18 LUFS with a 24 dB gain cap.Source: §2 · pp. 1–2
2.1. Transformer slot decoder
- The decoder returns four slot embeddings, while the solo path uses only the first slot to encode isolated stems during training and database encoding.The shared backbone and slot decoder map frame tokens to retrieval embeddings and presence logits; inference selects slots using their presence scores.Source: §2 · p. 2, §2.1 · p. 2, Figure 2 · p. 2
2. METHOD
- For each mixture, minimum-cost assignment matches solo embeddings to distinct slots by normalized-embedding distance, leaving K − m slots unassigned.Assigned slots inherit their matched stem identities, while unmatched slots do not enter the contrastive embedding list.Source: §2 · p. 2
- Training combines supervised contrastive learning over solo and matched-slot embeddings with binary cross-entropy presence prediction for assigned versus unassigned slots.The loss uses λc=1.0 and λp=0.1; only matched slots contribute to the contrastive list.Source: §2 · p. 2
3. EXPERIMENTAL SETUP
The experiments use artist-separated MoisesDB splits and a fixed retrieval evaluation protocol. They compare Stembed with trained and frozen baselines, source-separation pipelines, and isolated-stem references.
3. EXPERIMENTAL SETUP
- Training, validation, and test splits contain 166, 37, and 35 songs respectively, with 20, 13, and 9 artist groups kept separate.The dataset is filtered to 238 songs with at least four usable stems, and sampled mixtures require all constituent stems to be active together.Source: §3 · pp. 2–3
3.1. Retrieval protocol
- Retrieval ranks gallery stems by cosine similarity, with Stembed returning one ranking per selected slot and pooled encoders one per mixture.For each target, an oracle matches the ground-truth stem to a distinct selected slot; R@k measures whether its gallery stem appears in the first k results.Source: §3.1 · p. 3
- The fixed MoisesDB evaluation uses 1536 mixtures and 4608 target retrievals, with gallery excerpts disjoint from query windows when available.The evaluation cycles through 35 test songs; reported standard deviations reflect training-seed variation, not the limited number of test songs.Source: §3.1 · p. 3
- The “All” protocol searches the full gallery, while “Family” restricts each query to the target stem’s instrument family.The MoisesDB gallery contains 2068 stems, and pooled encoders can retrieve at most one of a mixture’s m targets at rank one under “All.”Source: §3.1 · p. 3, Table 1 · p. 3
3.2. Baselines
- The baseline suite includes CIR–MuQ, SMTI–MuQ, frozen general-purpose encoders, CIR–AST, and source-separation pipelines using HT-Demucs.CIR–MuQ applies a full-triplet objective to mean-pooled MuQ embeddings; SMTI–MuQ predicts embeddings from a frozen solo encoder using minimum-cost matching.Source: §3.2 · pp. 3–4
4. RESULTS
Stembed outperforms CIR–MuQ in full-library MoisesDB retrieval, and it remains competitive without supplied stem counts. Ablations show that both slot-based training choices and mixture construction affect retrieval outcomes.
4. RESULTS
- 56.8% All R@1 for Stembed versus 23.6% for CIR–MuQ on MoisesDB.With family guidance, Stembed has higher R@1 (58.9% versus 52.4%), while CIR–MuQ has slightly higher R@5 (81.7% versus 81.2%).Source: §4 · p. 4
- On Slakh2100, count-supplied and predicted-count Stembed exceed CIR–MuQ and SMTI–MuQ in All R@1 and All R@5.With family guidance, Stembed and CIR–MuQ have close R@1 values, while CIR–MuQ has higher Family R@5.Source: §4 · p. 4, Table 2 · p. 4
4.1. Ablations
- 56.0 ± 1.3% All R@1 and 76.3% R@5 result when Stembed predicts stem count without a supplied count.It undercounts 56.7% of four-stem mixtures; training omits four-stem mixtures, which may explain the undercounting.Source: §4.1 · p. 4
- 58.0±0.5% All R@1 with cosine-triplet training compares with 56.8± 0.9% for InfoNCE, suggesting InfoNCE is not essential to Stembed’s gains.The slot-based triplet control also exceeds pooled CIR–MuQ on All R@5 (79.2% versus 60.7%) and Family R@1 (60.1% versus 52.4%), though the models differ in other settings.Source: §4.1 · p. 4
- Cross-song negatives matter: using only same-song negatives gives 7.9 ± 0.4% All R@1, versus 45.3 ± 1.3% using only cross-song negatives.Both controls fall below the full model, particularly the control without cross-song negatives.Source: §4.1 · p. 4
- Independent windows improve R@5 but not R@1, while random cross-song mixtures retain similar mean All R@1/R@5 to same-song mixtures.Sharing windows changes mean All R@1 from 56.8% to 57.0% and reduces R@5 from 78.1% to 75.0%; random cross-song mixtures change the means from 56.8%/78.1% to 57.5%/76.7%.Source: §4.1 · p. 4
5. CONCLUSION
Stembed represents a mixture with multiple stem-specific embeddings, using slot decoding, minimum-cost matching, and contrastive learning. The conclusion highlights recall comparable to family-guided CIR–MuQ without family labels, while identifying perceptual similarity for absent targets as future evaluation.
5. CONCLUSION
- Stembed combines slot decoding, minimum-cost matching, and contrastive learning to encode mixtures as several stem-specific embeddings.On MoisesDB, it reaches recall comparable to family-guided CIR–MuQ without instrument-family labels.Source: §5 · p. 4
- Future work could evaluate perceptual similarity when the target is absent from the database.The paper also proposes training on denser mixtures, increasing slot counts as needed, and using slots to condition generative or retrieval-and-resynthesis systems.Source: §5 · p. 4