PaperlayerView exact revision on arXiv

Source-linked AI summary

Downstream-Task-Aware Unified Source Separation

Yoshiki Mitsui, Ryo Aihara, Tatsuhiko Saito, Yoshiki Masuyama, Christoph Boeddeker, Julius Richter, Gordon Wichern, Jonathan Le Roux

arXiv:2609.11092v1eess.AS

TL;DR

Conventional TUSS does not account for whether enhanced speech is intended for listening or ASR. This paper extends prompts and switches losses according to them, so one model can produce task-appropriate outputs. Joint training maintains comparable general enhancement performance while improving ASR robustness, with similar ASR trends reported across English and Japanese evaluations.

  • Problem

    Conventional TUSS does not account for downstream requirements, even though enhancement objectives for human listening and ASR can differ.

    Source: §1 · p. 1

  • Method

    The framework adds downstream-task prompts and switches between regularized ASR and conventional SNR losses according to the selected prompt.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    Jointly trained TUSS maintains comparable speech-enhancement performance while improving ASR robustness, and Japanese CER trends follow the LibriSpeech English evaluation.

    Source: §4.2 · pp. 3–4, §5 · p. 4

  • Takeaways & Limitations

    The framework can jointly support ASR-oriented enhancement and reverberation control within a single model.

    Source: §4.2 · p. 4, §5 · p. 4

    The formulation assumes the observed signal is a mixture of N source signals.

    Source: §2 · p. 1

Abstract

from arXiv · show

Task-aware unified source separation (TUSS) enables a single model to handle diverse separation tasks by conditioning on input prompts. However, conventional TUSS does not account for downstream task requirements, such as whether the enhanced speech will be used for human listening or automatic speech recognition (ASR). In this paper, we propose a prompt extension framework for TUSS that incorporates downstream task information into the input prompts and switches the loss function according to the given prompt during training, enabling outputs with different signal characteristics at inference time. Specifically, we introduce an ASR-dedicated prompt paired with a regularized loss function that reduces speech artifacts to improve ASR robustness, while the standard prompt is paired with the conventional SNR loss function. Experiments on the LibriSpeech and JNAS corpora demonstrate that the proposed joint-training scheme enables a single model to improve ASR performance over noisy input across a wide range of SNR conditions by selecting the ASR-dedicated prompt, while maintaining general speech enhancement quality when the standard prompt is used.

1. INTRODUCTION

Conventional enhancement objectives can produce different outcomes for human listening and ASR, motivating a unified model that adapts to downstream requirements. The proposed TUSS prompt extension conditions both prompts and training losses on the task.

1. INTRODUCTION

  • Speech enhancement for listening targets residual-noise reduction, whereas enhancement can degrade ASR performance relative to noisy input.ASR-oriented objectives can sacrifice perceptual quality, while inference-only observation adding cannot incorporate task requirements into the training objective.Source: §1 · p. 1
  • A unified training framework is motivated to adapt output characteristics to downstream-task requirements.This follows from the mismatch between listening and ASR objectives.Source: §1 · p. 1
  • The proposed TUSS framework extends prompts with downstream-task information and applies prompt-dependent losses to generate task-appropriate outputs.Experiments are reported to demonstrate more flexible separation across purposes and conditions.Source: §1 · p. 1

2. CONVENTIONAL TUSS

Conventional TUSS uses prompts to select source types within one model that handles multiple separation tasks. Its pipeline extracts prompt-conditioned features and decodes them into separated signals.

2. CONVENTIONAL TUSS

  • For speech enhancement, source separation estimates speech and noise from the observed mixture.The general formulation treats the observed signal as a mixture of N source signals.Source: §2 · p. 1
  • TUSS handles multiple source-separation tasks with one model by changing prompts that represent source types, such as speech, sound effects, or music stems.The passage assumes an observed signal is a mixture of N sources; speech enhancement typically treats speech and noise as the two sources.Source: §2 · p. 1
  • The model transforms the mixture and prompt embeddings into prompt-conditioned features, refines them with target source extraction, then decodes separated signals using negative SNR loss.The described pipeline uses STFT, a band-split encoder, cross-prompt processing, and a band-split decoder with inverse STFT.Source: §2 · p. 1

3. PROMPT EXTENSION FOR MULTIPLE CONDITIONS

The framework extends TUSS with downstream-task prompts and prompt-dependent losses, letting one model switch between ASR-oriented and general enhancement. It also adds a prompt for reverberation control, with training targets that depend on whether reverberation should be retained.

3.1. Overview of Prompt Extension

  • New prompts trained with downstream-task-specific losses let TUSS switch among task-appropriate outputs at inference.Existing prompts retain the conventional TUSS objective.Source: §3.1 · p. 2
  • Source-conditioned prompts encode a source type and downstream-task requirement, whereas task-only prompts encode conditions independent of source identity.Multiple task-only prompts can be combined to represent several conditions simultaneously.Source: §3.1 · p. 2

3.2. Application to ASR Frontend

  • For ASR front-end use, preserving target speech with minimal distortion is prioritized over aggressive noise reduction, which conventional TUSS SNR loss does not explicitly enforce.The paper notes that this mismatch can degrade ASR performance.Source: §3.2 · p. 2
  • The regularized loss uses a noisy reference whose SNR is raised by α dB relative to the original mixture.λ is the regularization weight.Source: §3.2 · p. 2
  • The regularization term trades off matching the clean reference sn against the noise-reduced mixture yn, which can improve ASR performance by avoiding overly aggressive noise suppression.Source: §3.2 · p. 2
  • Selecting an ASR-specific prompt applies regularized loss, while the standard prompt uses the conventional loss to prioritize general waveform reconstruction.This lets one TUSS model switch between ASR-oriented and general enhancement at inference time.Source: §3.2 · p. 2

3.3. Application to Reverberation Control

  • The <ex-KeepRev> prompt controls reverberation according to use, since ASR may favor its removal while human listening may favor retaining some.The prompt is downstream-task-only.Source: §3.3 · p. 2
  • With <ex-KeepRev>, training uses reverberant speech as the reference; without it, the target uses the RIR’s earlier 50 ms to exclude late reverberation.Mixtures are generated by convolving dry speech with room impulse responses and then adding noise.Source: §3.3 · p. 2

4. EXPERIMENTS

Experiments test prompt- and loss-conditioned speech enhancement on LibriSpeech and JNAS, then extend the model to reverberation control. Joint training preserves the distinct enhancement and ASR-oriented output profiles, with Japanese results following similar trends subject to a difficult-noise exception.

4.1. Experimental Conditions

  • A3/A4 jointly train standard SNR and regularized-SNR objectives with prompt-specific outputs; A5 independently toggles ASR and reverberation prompts during training.A5 uses four prompt combinations sampled uniformly, and trains on an equal mixture of anechoic and reverberant noisy data.Source: §4.1 · pp. 2–3
  • Evaluation comprised 18,340 LibriSpeech mixtures from 2,620 speech samples and seven test noise types; JNAS used male and five female speakers, with SNRs from 5 to −15 dB in 5-dB steps.JNAS used the same seven ATRANS-II noise types reserved for evaluation.Source: §4.1 · p. 3
  • Models were evaluated with SI-SDR, UTMOS, and Whisper-turbo WER, using 16 kHz inputs and 32 ms STFT windows with 8 ms hops.Training ran for 150 epochs; A5 included reverberant speech simulated from SLR26 room impulse responses.Source: §4.1 · p. 3

4.2. Results

  • Joint models A3/A4 retain the single-objective enhancement and ASR profiles, while ASR-oriented WER remains higher than OA’s; the high-quality branch stays comparable to A1.A2 and the joint ASR branches also have lower WER than A1, despite lower SI-SDR and UTMOS.Source: Table 1 · p. 3, §4.2 · pp. 3–4
  • On JNAS, CER trends broadly match LibriSpeech, but at −15 dB on harder noises B1 leads while regularized-objective models show more repetition errors.At SNRs above 0 dB, standard enhancement can also degrade ASR relative to no enhancement.Source: Table 2 · p. 4, §4.2 · p. 4
  • A5-HD performs best on SI-SDRNR and A5-HR on SI-SDRR; A5-AD has better WER, supporting joint ASR enhancement and reverberation control.Combining the ASR and reverberation prompts modestly improves SI-SDRR while leaving SI-SDRA and SI-SDRNR nearly unchanged.Source: Table 3 · p. 4, §4.2 · p. 4

5. CONCLUSION

The extended-prompt TUSS model achieves comparable speech enhancement while improving ASR robustness, and the framework can handle multiple downstream tasks simultaneously. Its regularized objective still underperforms the baseline OA method.

5. CONCLUSION

  • The extended-prompt TUSS model achieves comparable speech enhancement performance while improving ASR robustness.The conclusion presents this as validation of the proposed method.Source: §5 · p. 4
  • The framework can handle multiple downstream tasks simultaneously, including ASR-oriented speech enhancement and reverberation control.Source: §5 · p. 4
  • The regularized objective underperforms the baseline OA method, motivating investigation of better losses for ASR-oriented speech enhancement.Source: §5 · p. 4
Loading 2609.11092v1…