NECTAR: Automatic Negative Clue Estimation via Centroid-Space Mapping for Target Speaker Extraction

1. INTRODUCTION

• An interferer's enrollment can act as an effective negative clue, motivating the question of whether interferer-related information can be derived without additional enrollment.The proposed route analyzes the relationship between mixture and constituent-speaker centroids.Source: §1 · p. 1
• NECTAR automatically derives an interferer-related negative clue from the mixture and uses both target and interferer information.Because the centroid-space relationship does not directly hold for individual utterance embeddings, a Centroid-Space Mapper estimates centroid-space representations from input segments.Source: §1 · p. 1
• 16.18 dB SI-SDRi and 97.1% extraction accuracy on Libri2Mix; NECTAR outperforms all compared systems.The method is also reported effective in in-domain, out-of-domain, and noisy settings, with negligible computational overhead.Source: §1 · p. 1
• The proposed centroid-space relationship places the mixture centroid near the midpoint of the target and interferer centroids.The contributions also report effectiveness across in-domain, out-of-domain, and noisy settings, with negligible computational overhead.Source: §1 · p. 1

2.1. Speaker Centroid Estimation

• Speaker-centroid work aggregates multiple utterance embeddings for speaker-discriminative learning or more robust target representations; this paper instead studies centroid geometry for negative-clue derivation.The cited examples include GE2E, metric-learning approaches, and Heo et al..Source: §2.1 · p. 1

2.2. Negative Clues for Source Extraction

• NECTAR differs from query-based extraction and embedding de-mixing by using mixture-speaker centroid geometry to derive a negative clue without interferer enrollment at inference.Prior query approaches include class-based queries, positive/negative queries, and multimodal queries; embedding de-mixing is cited as, and interfering-speaker enrollment as.Source: §2.2 · p. 1
• Fig. 1 validates the centroid-space relationship at N = 100, while Fig. 2 depicts centroid deviation across scaling factors λ as centroid size N increases.Source: Figure 1 · p. 2, Figure 2 · p. 2

3.1. Centroid-Space Relationship

• For both encoders, the centroid-space relationship is characterized by the mixture centroid lying near the midpoint of the two speaker centroids.At N = 100, cosine similarity is maximized at α = 0.5 and MSE is minimized at β = 0.5.Source: §3.1 · p. 2
• Centroid deviation decreases and gradually saturates as centroid size N increases, while deviation remains substantially higher at N = 1.Because practical inference has only one mixture and one target enrollment, the relationship cannot be applied reliably to individual segments without CSM.Source: §3.1 · p. 2

3.2. Centroid-Space Mapper

• CSM maps frozen speaker-encoder frame features from a single-speaker segment or two-speaker mixture to a centroid-space representation.Its parameters are shared between enrollment and mixture branches, and training uses single-speaker signals and mixtures with centroid-based supervision.Source: §3.2 · p. 2
• CSM markedly reduces interferer-centroid deviation relative to N = 1 and approaches the N = 100 reference using a single mixture and target enrollment.The estimated interferer centroid is used as the negative clue in the subsequent extraction stage.Source: Figure 3 · p. 2, §3.2 · p. 3

3.3. Negative Clue for Target Speaker Extraction

• NECTAR concatenates the target enrollment embedding and estimated interferer centroid as positive and negative clues for the TSE extractor.The negative clue is derived from the mixture and target enrollment without requiring interferer enrollment at inference.Source: §3.3 · p. 3, Figure 4 · p. 3

4.1. Training

• The extractor was trained on LibriSpeech train-clean-100 with dynamic speaker mixing, while CSM training used LibriSpeech train-clean-100 and Libri2Mix train-100.The extractor used interferer enrollment embeddings during training and centroid-based clues at inference; extractor and CSM were trained separately.Source: §4.1 · p. 3
• BSRNN served as the TSE backbone and frozen CAM++ as the speaker encoder; CSM training took about 1 hour.The extractor was trained for 150 epochs and the CSM for 50 epochs.Source: §4.1 · p. 3

4.2. Evaluation

• Evaluation covered Libri2Mix mix clean, VCTK-2Mix, and noisy Libri2Mix mix both, using SI-SDRi, PESQ, ESTOI, and extraction accuracy.Extraction accuracy is the percentage of samples with SI-SDRi greater than 1 dB.Source: §4.2 · p. 3
• Table 2 compares auxiliary clues on Libri2Mix with target enrollment as the positive clue; Table 3 reports out-of-domain VCTK-2Mix performance.Source: Table 2 · p. 4, Table 3 · p. 4

5.1. TSE Performance

• 16.18 dB SI-SDRi and 97.1% extraction accuracy make NECTAR the strongest compared system on Libri2Mix, ahead of Multi-Level BSRNN at 15.91 dB SI-SDRi.Source: §5.1 · p. 4
• NECTAR improves SI-SDRi by 1.37 dB over the target-enrollment Baseline and outperforms Mixture Clue on Libri2Mix.The comparison isolates the negative clue while keeping the target enrollment embedding as the positive clue.Source: §5.1 · p. 4
• 2.36 dB SI-SDRi for the single-embedding approximation contrasts with 16.62 dB for the oracle centroid approximation, supporting centroid-level rather than utterance-level midpoint geometry.Oracle interferer-centroid and clean interferer-utterance clues reach 17.44 dB and 17.45 dB SI-SDRi, respectively.Source: §5.1 · p. 4
• 14.67 dB SI-SDRi versus 12.90 dB on VCTK-2Mix without fine-tuning accompanies consistent gains across metrics and Baseline improvements in noisy Libri2Mix.The noisy-condition comparison reports gains across all metrics.Source: §5.1 · p. 4

5.2. Does the centroid-space relationship hold for 3 speakers?

• 0.89 mean cosine similarity and 4 × 10−4 MSE show that the equal-weight speaker-centroid approximation resembles the mixture centroid in three-speaker mixtures.The analysis used 10 speaker triplets and fixed 3-second segments; Figure 5 visualizes the centroids and approximation.Source: Figure 5 · p. 4, §5.2 · p. 4

• NECTAR derives interferer information from the observed mixture and target enrollment through centroid-space structure, with CSM enabling the relationship to be used from individual input segments.Source: §6 · p. 4
• 16.18 dB SI-SDRi on Libri2Mix is the reported NECTAR result.Source: §6 · p. 4
• Performance gains extend to noisy conditions and out-of-domain VCTK-2Mix evaluation without fine-tuning, supporting effective, generalizable conditioning from derived interferer information.Source: §6 · p. 4

• A theoretical explanation of the midpoint relationship remains underexplored despite empirical evidence in CAM++ and ECAPA-TDNN.Source: §7 · p. 4
• The TSE experiments explore only two-speaker mixtures, although the three-speaker centroid analysis suggests potential applicability.Source: §7 · p. 4
• Extending NECTAR to mixtures with more speakers and settings with a non-speech primary interferer remains future work.Source: §7 · p. 4