PaperlayerView exact revision on arXiv

Source-linked AI summary

RemixIT-TSE: Progressive Synthetic-to-Real Adaptation for Target Speech Extraction via Target-Aware Supervision and Remixing

Yu Wang, Haixin Guan, Shuang Wei, Yanhua Long

arXiv:2609.35118v1cs.SD

TL;DR

Real-world TSE faces a domain gap from synthetic training, and conversational recordings typically lack clean target references. RemixIT-TSE progressively adapts a synthetic-trained model using weakly supervised real data and then real-data-only pseudo-target training. On the REAL-TSE Challenge evaluation sets, it outperforms the source-domain baseline across seen and unseen conditions.

  • Problem

    Most TSE systems train on synthetic mixtures with clean targets, whereas real conversational recordings have acoustic mismatch and typically lack signal-level ground truth.

    Source: §1 · p. 1

  • Method

    RemixIT-TSE progressively combines synthetic supervision with region-level weak supervision on real recordings, then adapts using real-only teacher pseudo-targets and target-aware constraints.

    Source: §1 · p. 1, §2.4 · p. 3, §5 · p. 4

  • Results

    RemixIT-TSE outperforms the source-domain baseline across seen and unseen REAL-TSE Challenge conditions, reducing EVAL-2 TER from 0.763 to 0.713.

    Source: §4.2 · p. 4

  • Takeaways & Limitations

    The framework adapts synthetic-trained TSE to real recordings without requiring signal-level clean target references.

    Source: §1 · p. 1, §5 · p. 4

Abstract

from arXiv · show

Target Speech Extraction (TSE) in real-world conversational scenarios suffers from severe performance degradation due to the domain gap between synthetic training data and complex acoustic environments, where signal-level ground truth is typically unavailable. To address this challenge, we make the first attempt to extend RemixIT from speech enhancement to TSE and propose a progressive synthetic-to-real adaptation framework for real-world TSE with two fine-tuning stages. The first stage leverages region-wise speaker similarity and silence constraints within a target-aware adaptation framework to jointly optimize the model using synthetic and weakly supervised real-world data, injecting real-world traits while preserving synthetic-learned capabilities. The second stage further adapts the model using only real-world data through our RemixIT-TSE, where quality-filtered teacher pseudo targets, which guarantee reliable student training, provide signal-level supervision via SI-SNR loss. Experiments on the real conversational evaluation set (EVAL-2) of the SLT 2026 REAL-TSE Challenge, the proposed method achieves a 6.53% relative TER reduction, together with relative improvements of 21.84% in speaker similarity, 9.89% in DNSMOS-P808, and 4.10% in target-activity F1 over the source-domain baseline, demonstrating its effectiveness under unseen real-world conditions. Source code at https: //github.com/YuWang-Speech/RemixIT-TSE.

1 Introduction

TSE systems commonly rely on synthetic mixtures with clean-target supervision, while real conversations introduce a domain gap and generally lack signal-level ground truth. RemixIT-TSE addresses this gap with progressive adaptation that uses weak real-world annotations and quality-filtered pseudo-targets.

1 Introduction

  • Real conversational recordings add reverberation, noise, irregular overlap, and recording mismatch, while generally lacking signal-level ground truth despite often providing speaker labels and aligned transcripts.Most TSE systems instead train on synthetic mixtures such as WHAMR! and Libri2Mix, where clean targets are available.Source: §1 · p. 1
  • Prior RemixIT adaptation addressed speech enhancement, while the closest prior TSE work mainly considered noise-free conditions and required clean speech for noisy scenarios.The paper identifies this as limiting applicability to real-world recordings without clean target signals.Source: §1 · p. 1
  • The paper extends RemixIT from speech enhancement to real-world TSE without requiring signal-level clean target references, using progressive synthetic-to-real adaptation.Its stages combine mixed-domain weakly supervised fine-tuning with region-wise speaker similarity and silence constraints, then real-data-only adaptation.Source: §1 · p. 1
  • The framework progresses from supervised synthetic-data pretraining to joint synthetic-real adaptation, then real-data-only RemixIT-TSE adaptation.Stage II uses region-wise speaker and silence constraints; Stage III uses quality-filtered targets and SI-SNR-based student optimization.Source: Figure 1 · p. 2

2 Proposed Method

RemixIT-TSE progressively adapts a synthetic-trained TSE model through joint synthetic-real training and then real-only teacher-student adaptation. Weak annotations guide target-aware training, while filtered pseudo targets provide supervision without clean real-world references.

2.1 Overview of the Framework

  • Training progresses from synthetic-data pretraining to joint target-aware adaptation and finally real-only RemixIT-TSE adaptation.The three stages are SDP, TAA, and RTA.Source: §2.1 · p. 2, §2.4 · p. 2

2.2 Enrollment and Pseudo-Target Generation

  • Five non-overlapped utterances from the target speaker are encoded and averaged to form the enrollment speaker centroid.The real-world annotations provide speaker-attributed, time-aligned intervals for constructing these enrollments.Source: §2.2 · p. 2

2.3 Quality-Aware Pseudo-Label Filtering

  • Quality filtering addresses interfering-speaker leakage in pseudo targets, using speaker similarity as the primary criterion and TER as a complementary content criterion.Only pseudo targets are filtered, retaining a small fraction of high-confidence recordings for adaptation.Source: §2.3 · p. 2

2.4 Progressive Synthetic-to-Real Adaptation

  • Synthetic pretraining uses mixtures, enrollments, and clean target references with SI-SNR, compressed-magnitude reconstruction, and centroid-based speaker-consistency objectives.The centroid-based loss is an advanced variant of target speaker similarity loss.Source: §2.4 · pp. 2–3
  • Target-aware adaptation initializes from the synthetic model and jointly optimizes synthetic supervision with real-branch speaker-similarity and silence constraints.The real branch uses target activity masks derived from time-aligned annotations.Source: §2.4 · p. 3
  • Real-only RemixIT-TSE uses the Stage-II model as teacher and student initialization, then remixes selected target and non-target pairs after quality filtering.Non-target components are permuted within the batch while avoiding pairing with the same speaker.Source: §2.4 · p. 3
  • The student is trained against the detached teacher pseudo target using negative SI-SNR on target speech, and teacher parameters follow the student by exponential moving average.Unlike standard RemixIT, the optimization focuses only on target supervision rather than aligning both speech and residual noise.Source: §2.4 · p. 3

3 Experimental Setup

The experiments use synthetic mixtures for supervised pretraining and conversational recordings with weak annotations for real-world adaptation. Evaluation covers seen and unseen challenge scenarios using intelligibility, speaker similarity, perceptual-quality, and target-activity metrics.

3.1 Datasets

  • The real adaptation corpus contains approximately 51.03 hours of annotated mixtures from AMI, AliMeeting, AISHELL-4, and CHiME-6, with 4.2 hours retained as high-confidence teacher-target data.Mixtures were retained when target-speech and overlap durations each exceeded five seconds; the evaluation sets include unseen conversational scenarios in EVAL-2.Source: §3.1 · p. 3

3.2 Configurations

  • USEF-TFGridNet is the TSE backbone, trained in three stages with Adam and stage-specific learning rates and adaptation settings.The models use a causal architecture and 16-kHz audio; RTA applies SIM and TER thresholds of 0.7.Source: §3.2 · p. 3

3.3 Evaluation Metrics

  • Evaluation measures TER, SIM, DNSMOS P.835 and P808, and target-speaker activity precision, recall, and F1.TER is computed with Zipformer ASR, while FireRedVAD measures target-speaker presence.Source: §3.3 · p. 3

4 Results and Analysis

The progressive adaptation strategy improves results on seen and unseen REAL-TSE evaluation conditions, while development-set comparisons and ablations show that adaptation choices and pseudo-target quality matter. The strongest reported unseen-set gains include lower TER and higher P808.

4.1 Comparison of Adaptation Strategies

  • TAA preserved baseline TER while raising P808 from 2.897 to 3.103 and F1 from 0.837 to 0.848; RTA then reached TER 0.621, SIM 0.501, and F1 0.854.By comparison, SAMoM (synthetic clean) worsened TER to 0.807 and P808 to 2.612; SAMoM (real single-spk) raised SIM to 0.505 but worsened TER to 0.753 and P808 to 2.726.Source: §4.1 · p. 4

4.2 Results on REAL-TSE Evaluation Sets

  • TER fell from 0.763 to 0.713 and P808 rose by 0.268 on unseen EVAL-2; the method also improved across perceptual and tracking metrics on seen EVAL-1.Across the combined evaluation set, the authors report comprehensive performance improvements.Source: Table 2 · p. 4, §4.2 · p. 4

4.3 Analysis of Modifications for RemixIT-TSE

  • Using 51.03 hours of unfiltered real data degraded all metrics, whereas 4.2 hours of filtered high-confidence data achieved the best performance with 471 iterations.Restoring residual supervision also degraded performance; the authors therefore use quality-filtered teacher targets with target-only supervision.Source: Table 3 · p. 4, §4.3 · p. 4

5 Conclusion

RemixIT-TSE extends the RemixIT paradigm to real-world TSE using quality-aware pseudo-target filtering and target-only supervision. Its progressive adaptation combines weak speaker-attributed annotations with target-aware adaptation and filtered pseudo-target learning, and the challenge experiments report improvements across extraction, speaker similarity, perceptual quality, and target-activity prediction.

5 Conclusion

  • RemixIT-TSE extends RemixIT to real-world target speech extraction with quality-aware pseudo-target filtering and target-only supervision.The progressive framework adapts a synthetic-trained model without signal-level clean target references and improves reported extraction, speaker-similarity, perceptual-quality, and target-activity measures.Source: §5 · p. 4

Related papers

Loading 2609.35118v1…