Source-linked AI summary
A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine
Amirhossein Hajavi, Hanhee Lee, Pushya Jain, Sky Qiao, Emmanuel Ko, Yuanhao Yu, Irina Kezele
TL;DR
Application-specific LASS adaptation is challenged by noisy or insufficiently representative training data and CLAP controls with limited semantic coverage. The paper combines an automated data engine with class-specific CLAP control-signal optimization. Across seven classes, objective separation metrics improve, and subjective comparisons favor the optimized model over baseline and commercial options.
Problem
Adapting LASS to application-specific classes is challenging because performance depends on training-data representation and CLAP embeddings may not preserve class hierarchies as effective controls.
Source: §1 · p. 1Method
A multistage data engine curates training examples, while class-specific CLAP controls are formed from selected prompts and optimized using SGD.
Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 3Results
39.07% of baseline comparisons preferred the optimized model, 24.63% disfavored it, and 36.30% tied; SDRi and SI-SDRi improved from baseline to refined to optimized.
Source: §4 · p. 4Takeaways & Limitations
The framework is effective for adapting pretrained LASS models to defined general sound classes and outperforms baseline and commercial options.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Language-queried audio source separation (LASS) enables extracting any sound source using natural language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.
1. INTRODUCTION
Application-specific LASS adaptation matters because target-class performance depends on training-data representation and the suitability of CLAP control signals. This work addresses underexplored data curation and control-signal optimization, reporting improvements across seven sound classes.
1. INTRODUCTION
- Application-specific separation prioritizes selected sound classes, and CLAP lets downstream applications specify arbitrary classes with natural language rather than predefined labels.Source: §1 · p. 1
- LASS performance depends on how well a target class is represented in training data, while CLAP embeddings may not preserve class hierarchies well enough to serve as effective controls.Source: §1 · p. 1
- The proposed data engine and optimized control signals significantly improve source separation for seven target sound classes.The work addresses both training-data curation and class-specific control signals.Source: §1 · p. 1
- Prior work primarily modifies model architecture, leaving useful training-data curation and language-based control-signal optimization underexplored.Source: §1 · p. 1
2. RELATED WORK
Earlier source-separation benchmarks focused on music or speech with predefined classes or reference speakers, while universal and language-queried approaches broadened the target classes. Prior adaptation work addressed output refinement or model adaptation, but not training-data curation or control-signal optimization.
2. RELATED WORK
- Universal audio source separation allows arbitrary target classes, unlike earlier music or speech tasks with predefined labels or reference speakers [1].Source: §2 · p. 1, Figure 1 · p. 2
- Language-queried models use natural-language controls with FiLM conditioning [21, 22, 23], and later CLAP-based controls improved separation through semantically aligned embeddings [3, 4, 5, 6, 7, 8, 9, 10, 11].Source: Figure 1 · p. 2
- Prior subset-focused adaptation refined outputs with a GAN or combined domain partitioning and PEFT, but did not target training-data curation or control-signal optimization.Source: §2 · p. 2
3. METHOD
The framework fine-tunes a pretrained LASS model using curated training data and refined class-specific CLAP control signals. Its data engine filters samples for relevance and single-class content, while control-signal optimization selects and refines prompts to represent target classes and their subcategories.
3.1. Data Generation and Filtering Pipeline
- An audio-captioning model and LLM-based judge filter samples for single-class content that belongs to the target set.Captions can be generated for samples from datasets without detailed descriptions, broadening the available data sources.Source: §3.1 · pp. 2–3
3.2. Control Signal Optimization
- The first optimization stage retains the largest prompt subset whose mean text embedding stays within a cosine-distance threshold of representative audio embeddings.The threshold balances the number of retained prompts against audio-text alignment; retained prompts feed the second stage.Source: §3.2 · p. 3
- In the second stage, SGD optimizes each class embedding using orthogonal normalization, deviation regularization, and LINCE, with the objectives combined by a weighted sum.Orthogonal normalization aims to increase angular distance between embeddings, while deviation regularization limits movement from the original embeddings.Source: §3.2 · p. 3
4. EXPERIMENTS AND RESULTS
Experiments across seven sound classes compare data refinement and control-signal optimization using objective metrics and subjective listening tests. Both stages improve separation results, and the optimized model receives favorable subjective evaluations.
4. EXPERIMENTS AND RESULTS
- Seven classes were targeted at 20K samples each; Crowd and Wind required generated or handcrafted supplements when dataset totals fell short.Speech and Music exceeded 20K samples, so 20K were randomly selected for each class to maintain balance.Source: §4 · pp. 3–4
- Across target classes, both SDRi and SI-SDRi improve from baseline to refined and improve further with optimized control signals.Table 1 compares baseline, refined, and optimized signals across the target classes.Source: Table 1 · p. 4, §4 · p. 4
- Against baseline, the optimized model is preferred in 39.07% of trials, has a 24.63% loss rate, and ties in 36.30%.The study also reports an overall preference for the optimized model over the Google and Samsung commercial options.Source: Figure 2 · p. 4, §4 · p. 4
- AudioSep served as the baseline, and evaluation combined signal-level and semantic metrics with a 17-participant listening test.Objective comparison with commercial options was not possible because of heavy processing in the mobile apps.Source: §4 · p. 4
- CLAPScore varied minimally, while CLAPScoreA improved from baseline through refined and optimized models, indicating greater semantic similarity to target classes.Source: §4 · p. 4
5. CONCLUSION
The conclusion presents an approach to adapting pretrained LASS models to defined sound classes and reports improved performance against baseline and commercial options. It identifies further control-signal optimization and prompt-selection refinement as future directions.
5. CONCLUSION
- The framework combines an automated data engine with optimization of CLAP-based control signals to address training-data procurement and control-signal challenges.Source: §5 · p. 4
- Subjective and objective tests support the framework's effectiveness in adapting pretrained LASS models, with reported performance above baseline and commercial options.Source: §5 · p. 4
- Future work will explore other CLAP-based control-signal optimization methods and further refine text-prompt selection.Source: §5 · p. 4