Source-linked AI summary
PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment
Yutong Wen, Ke Chen, Prem Seetharaman, Haonan Chen, Oriol Nieto, Paris Smaragdis, Justin Salamon, Zeyu Jin
TL;DR
Target speaker extraction often relies on one cue family, although no single cue is both broadly available and reliably discriminative. PromptSep-Speaker uses latent diffusion with four interchangeable cues and an auxiliary alignment loss. It outperforms specialized TSE and universal separation baselines across four evaluation sets and a listening test.
Problem
TSE systems typically rely on a single cue family, although no single cue is both broadly available and reliably discriminative.
Source: §1 · p. 1Method
PromptSep-Speaker combines latent diffusion with four interchangeable cues and a WavLM-based auxiliary alignment loss that supervises cue attention.
Source: §1 · p. 1Results
Across four evaluation sets and a listening test, PromptSep-Speaker outperforms specialized TSE systems and universal separation baselines.
Source: §5 · p. 4Takeaways & Limitations
Users can provide speaker order, transcript, enrollment audio, or a caption, whichever cue their setting affords.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
1. INTRODUCTION
Prior TSE often relies on one cue family, while individual cues can be unavailable or ambiguous; mask-prediction systems are also described as less robust in separation quality and across acoustic conditions. PromptSep-Speaker uses latent diffusion with four interchangeable cues and an auxiliary alignment loss to supervise cue attention.
1. INTRODUCTION
- Enrollment audio can confuse similar-sounding speakers or be too short, while textual descriptions may identify a class rather than a specific speaker.A concurrent approach uses the target’s spoken words to pinpoint a speaker.Source: §1 · p. 1
- Most TSE approaches use a single cue, and mask-prediction models can be less robust in separation quality and across acoustic conditions.Prior multi-cue systems often remain within one modality or treat audio and text as complementary.Source: §1 · p. 1
- PromptSep-Speaker accepts speaker temporal order, transcript, enrollment audio, or speaker caption as interchangeable target cues.Users can provide whichever cue their setting affords.Source: §1 · p. 1
- A frozen WavLM representation and auxiliary alignment loss supervise how the model attends to a cue, encouraging speech features to help locate the target in the mixture.The loss requires no extra labeling, and ablations identify it as decisive.Source: §1 · p. 1
- The paper evaluates all four cues across four evaluation sets and a listening test, reporting state-of-the-art results against TSE and universal source-separation baselines.The authors describe improvements in separation accuracy and quality across cue modalities.Source: §1 · p. 1
2. METHODOLOGY
PromptSep-Speaker encodes mixture audio and target cues for a latent-diffusion separator. Transcript and speaker-ID cues use a cross-attention alignment module, with an auxiliary loss supervising that attention.
2.1. Model Architecture
- A frozen DAC-VAE encodes the 48 kHz mixture and enrollment audio into latents; a DiT generates target latents that the DAC-VAE decodes into the separated waveform.The mixture also receives frozen WavLM features through an alignment module.Source: §2.1 · p. 2
2.2. Condition Injection
- Caption text and enrollment latents join the diffusion sequence, while transcript and speaker-ID tokens enter the alignment module as cues.Enrollment audio is a different-utterance reference clip of up to 8 seconds; speaker ID encodes speaking order.Source: §2.2 · p. 2
2.3. Alignment Module
- A two-layer bidirectional transformer uses mixture-frame WavLM features as queries and transcript or speaker-ID tokens as keys and values to produce a frame-level target map.The map is projected and added to the diffusion latent, modulating the separator toward the target time footprint.Source: §2.3 · p. 2
2.4. Auxiliary Alignment Loss
- The auxiliary loss supervises cue-to-frame cross-attention using target-active frames derived from oracle-target RMS energy pooling.The objective combines diffusion MSE with a task-specific alignment loss; speaker-ID supervision directs target frames to the tag and other frames to the null token.Source: §2.4 · p. 2
3. EXPERIMENTS
The experiments describe PromptSep-Speaker’s training setup, four evaluation datasets, baseline and metric choices, and a 700-item subjective listening test. The supplied passages specify evaluation scope and protocol but do not provide benchmark outcomes.
3.1. Training Data Construction and Hyper-parameters
- Training uses synthesized 16-second, 48 kHz mono mixtures with two or three speakers, varied acoustic conditions, and time-layout templates covering partial overlap, turn-taking, and full overlap.The source collection includes 4.8M speech, 1.1M sound-effects, 10K music, and 2K ambient-noise samples; the model has roughly 0.8B parameters and uses 24 inference steps.Source: §3.1 · p. 2
3.2. Evaluation Datasets
- Four evaluation sets provide 5k mixtures per task, expanded to 10k extraction instances or 15k for three-speaker cases; all sources are disjoint from training.Libri2Mix supports transcript and enrollment-audio cues only, while LibriTTS-R-Mix does not support caption evaluation; Table 1 marks unsupported cues with dashes.Source: §3.2 · pp. 2–3, Table 1 · p. 3
3.3. Baselines
- The comparison includes six baselines selected for cue overlap, while several concurrent systems were excluded because their checkpoints were unavailable.The listed systems include transcript- and enrollment-conditioned TSE models and caption-based source separators.Source: §3.3 · p. 3
3.4. Objective Metrics
- Objective evaluation reports SI-SDRi, mel-spectrogram L2 distance, DNSMOS, and F1 Decision Error for target temporal-region identification.Audio is resampled to match each model’s evaluation setting.Source: §3.4 · p. 3
3.5. Subjective Listening Test
- The subjective test covers 700 items across four tasks and three ablated configurations, with ratings for speaker accuracy, speech quality, and content consistency.Each item presents a mixture, a ground-truth target reference, and one system output; the three questions use a five-point scale.Source: §3.5 · p. 3
4. RESULTS
PromptSep-Speaker performs strongly across cue types and evaluation settings, with transcript, enrollment, speaker-ID, and caption results supported by objective and listening-test findings. Stress tests and model ablations show where performance degrades and how cue alignment relates to target selection.
Transcript
- On three out-of-domain sets, PromptSep-Speaker improves SI-SDRi by 6.6–11.3 dB over DAE-TSE; DAE-TSE leads only on in-domain Libri2Mix.PromptSep-Speaker also has lower mel-L2 and higher target-selection accuracy measured by F1 error on the out-of-domain sets.Source: §4 · p. 4
Enrollment Audio
- On CapSpeech, LibriTTS-R, and internal sets, PromptSep-Speaker outperforms SoloSpeech across SI-SDRi, mel-L2, and F1.Listening results show SoloSpeech and WeSep have similar speaker accuracy and speech-quality scores despite widely differing SI-SDRi; PromptSep-Speaker remains consistent across objective and subjective evaluations.Source: §4 · p. 4
Speaker Id
- With speaker ID, SI-SDRi drops only 0.41, 0.81, and 1.10 dB versus full transcript on CapSpeech-Mix, LibriTTS-R-Mix, and the internal set; F1 remains 0.93, 0.94, and 0.95.DNSMOS differs by at most 0.07; no baseline supports this temporal-order cue.Source: §4 · p. 4
Caption
- On the internal set, PromptSep-Speaker gains at least 12 dB SI-SDRi and 0.17 F1 over the strongest caption-based baseline, outperforming all three baselines on both sets and every metric.Both sets are out-of-domain for all models.Source: §4 · p. 4
- SAM-Audio separates speakers more reliably than the other two caption baselines but trails PromptSep-Speaker on every metric, attributed to limited exposure to overlapping-speech data and conditions.PromptSep and FlowSep, universal separators not substantially trained on speaker data, struggle with speaker extraction by design and data scarcity.Source: §4 · p. 4
Task Ablations
- Partial transcripts under 60% of full words (3–8 words) yield 6.57 dB mean SI-SDRi and 0.89 mean F1; full-overlap transcripts yield 6.10 dB mean SI-SDRi with unchanged F1.Each costs about 4–5 dB SI-SDRi relative to the original transcript; adding a third speaker lowers results to 4.72 dB SI-SDRi and 0.83 F1.Source: §4 · p. 4
Model ablations
- Without auxiliary alignment, only 1% of attention mass falls on condition tokens; both model ablations substantially degrade SI-SDRi and F1 despite relatively strong DNSMOS.With alignment, transcript tokens attend to corresponding speech frames and localize the target speaker’s utterances around the interferer.Source: §4 · p. 4
Subjective listening test
- PromptSep-Speaker exceeds 4.0 on 16 of 21 listening-test cells and on speaker accuracy, quality, and content consistency for six of seven tasks.Transcript variants remain within 0.3 MOS of the unstressed cue; neither baselines nor ablations exceed 4.0 on any criterion except WeSep’s 4.23 content-consistency score.Source: §4 · p. 4
5. CONCLUSION
PromptSep-Speaker is a generative latent-diffusion separator that unifies four interchangeable cues and introduces an auxiliary alignment loss. Across four evaluation sets and a listening test, it outperforms specialized TSE systems and universal separation baselines.
5. CONCLUSION
- PromptSep-Speaker unifies speaker ID, transcript, enrollment audio, and caption cues in one generative latent-diffusion separation model, with an auxiliary alignment loss to condition speech representation.Across four evaluation sets and a listening test, it outperforms specialized TSE systems and universal separation baselines.Source: §5 · p. 4