PaperlayerView exact revision on arXiv

Source-linked AI summary

One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications

Szu-Wei Fu, Rong Chao, Xuesong Yang, Sung-Feng Huang, Ante Jukić, Yu Tsao, Yu-Chiang Frank Wang

arXiv:2606.25621v2cs.SD

TL;DR

Real-time speech enhancement must meet diverse latency budgets, but existing approaches may require separate models or fix one latency dimension. This paper presents a single model controlling algorithmic and computational latency, supporting 30 configurations while narrowing the gap with specialized models.

  • Problem

    Real-time speech enhancement requires operation under strict latency budgets, while prior open-source work lacks a universal model for complex degradations and varying sampling rates.

  • Method

    The framework combines configurable look-ahead frames, parallel convolutional layers, early exits, and two-stage shared-to-multiple decoder training in one streamable model.

  • Results

    The model supports 30 latency configurations, and two-stage training consistently improves most evaluation metrics while narrowing the gap with specialized models.

  • Takeaways & Limitations

    A single real-time enhancement model can be deployed across diverse latency budgets without retraining separate models.

Abstract

from arXiv · show

Different real-time speech applications impose distinct latency budgets, often requiring separately trained enhancement models for each scenario. In this paper, we propose a one-for-all, real-time universal speech enhancement model that provides explicit control over both algorithmic and computational latency. Algorithmic latency is flexibly adjusted via configurable look-ahead frames. To avoid learning inefficiency caused by varying padding configurations, we introduce parallel convolutional layers corresponding to different look-ahead settings. Computational latency is controlled through an early-exit mechanism, enabling inference at different network depths. To narrow the performance gap between specialized and flexible models, we propose a two-stage training strategy with a shared-to-multiple decoder transition. Overall, the proposed framework enables a single model to be deployed across diverse latency budgets without retraining separate models. Model weights are available for download at: https://huggingface.co/nvidia/Real-time_RE-USE

I. INTRODUCTION

Real-time speech enhancement must meet application- and hardware-dependent latency budgets, whereas existing causal models do not readily fit these varied constraints. The paper proposes a one-for-all streamable universal speech enhancement model with explicit algorithmic-latency control through flexible look-ahead frames.

  • Latency decomposition: Total latency comprises algorithmic and computational latency, with frequency-domain algorithmic latency determined by STFT window size, look-ahead frames, and hop size.Algorithmic latency is the input required for the first output unit; computational latency is the time required to generate an output afterward.
  • Motivation: Real-time speech enhancement operates under strict latency budgets unlike prerecorded-speech restoration, which prioritizes output quality.The paper distinguishes real-time constraints from offline restoration priorities.
  • Motivation: Interactive conversational VoIP typically tolerates 50–150 ms, while streaming ASR generally operates within 100–200 ms.Computational latency also depends on deployment-hardware capability, further diversifying requirements.
  • Contribution: The proposed one-for-all, real-time, streamable USE model controls algorithmic latency through flexible look-ahead frames while handling diverse degradation conditions.This explicit control enables adaptable total latency and deployment across a wide range of latency budgets.

II. FROM OFFLINE TO REAL-TIME SPEECH ENHANCEMENT

Real-time speech enhancement models must satisfy causality, application-specific latency budgets, and processing-time constraints. Offline RTF measurements can underestimate streaming latency by exploiting utterance-level parallelism and optimized large-tensor CUDA kernels.

  • Causality requires a causal architecture or only a limited number of look-ahead frames.
  • The total latency must not exceed the application-specific latency budget, l_budget.
  • Real-time processing requires computation per step to finish within the corresponding hop size h, making the real-time factor less than 1.
  • Offline RTF measurements can substantially underestimate streaming RTF because whole-utterance processing exploits time-dimension parallelism and optimized large-tensor CUDA kernels.

III. PROPOSED METHOD · A. Adjustable Algorithmic Latency

The method adjusts algorithmic latency by configuring convolutional padding to control look-ahead frames. It uses parallel, look-ahead-specific convolutional layers sampled during training, with user-selected expert inference based on the latency budget.

  • A. Adjustable Algorithmic Latency: Algorithmic latency is controlled by the number of configurable look-ahead frames during inference.The approach favors controlling look-ahead frames over modifying the STFT window size or hop length.
  • A. Adjustable Algorithmic Latency: The model supports latency-aware inference paths selected according to the application’s latency budget.Figure 2 depicts adjustable algorithmic latency through look-ahead frames and low-latency inference via a designated path.
  • A. Adjustable Algorithmic Latency: Varying padding configurations create a learning-efficiency issue addressed by the proposed architecture.The supplied result passage references the green UTMOS validation learning curve and directs readers to Section IV for experimental settings.
  • A. Adjustable Algorithmic Latency: Parallel convolutional layers are assigned to specific look-ahead-frame counts and padding configurations.This design is inspired by the mixture-of-experts paradigm and lets each latency setting use its corresponding convolutional layer.
  • A. Adjustable Algorithmic Latency: During training, one look-ahead-specific convolutional layer is randomly sampled to construct the computational graph.Unlike conventional MoE models, the framework does not require learned routing because users explicitly select the expert based on the latency budget.

B. Two-Stage Training for Early-Exit Optimization · IV. EXPERIMENTS · A. Dataset

The paper improves early-exit optimization with a two-stage shared-to-multiple decoder training framework, while evaluating on multilingual, multi-condition speech data spanning diverse sampling rates and degradations.

  • B. Two-Stage Training for Early-Exit Optimization: Intermediate early exits compromise performance because they must accommodate the requirements of subsequent layers.Their performance therefore lags behind models optimized for a fixed output depth.
  • B. Two-Stage Training for Early-Exit Optimization: The proposed remedy assigns separate decoders to different intermediate output layers.This allows each output layer to learn layer-specific parameters.
  • B. Two-Stage Training for Early-Exit Optimization: Stage one trains the model with a shared decoder while randomly sampling exit layers at each training step.The shared decoder is trained until convergence before independent decoders are introduced.
  • B. Two-Stage Training for Early-Exit Optimization: Stage two initializes independent decoders for each output layer from the shared decoder.The encoder and sequence modeling modules preceding the decoders are then fine-tuned with a smaller learning rate.
  • A. Dataset: The training dataset contains multi-condition speech recordings in English, German, French, Spanish, and Chinese.It follows the URGENT 2025 Challenge setup.
  • A. Dataset: The dataset covers sampling rates of 8, 16, 22.05, 24, 32, 44.1, and 48 kHz.It includes clean speech, noise samples, and room impulse responses.
  • A. Dataset: Seven degradation types are considered, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, and packet-related degradation.The supplied passage lists these degradation categories as part of the dataset design.

B. Model Architecture

The model uses a causal, Mamba-based architecture with configurable look-ahead and a maximum depth of 12 layers for real-time deployment. It supports varying sampling rates through SFI STFT and reduces refinement cost using two-stage adversarial training.

  • Core architecture: The architecture follows USEMamba and RE-USE, uses Mamba for RNN-like real-time inference, and has 12 layers with 3.7M parameters.The maximum depth is set to 12 Mamba layers to meet computational latency constraints.
  • Latency control: Causal convolutions, unidirectional temporal Mamba, and channel-only layer normalization enforce causality or limited look-ahead.Look-ahead is controlled through different amounts of left padding in the first convolutional layer.
  • Training strategy: The two-stage training strategy pretrains with regression loss and then fine-tunes with adversarial loss guided by discriminators, avoiding an additional generative refinement model.This approximates transporting the MMSE estimate toward the true data distribution while reducing computational latency relative to RE-USE.
  • Sampling-rate flexibility: SFI STFT enables operation across sampling rates by adjusting FFT window and hop size while maintaining fixed time duration.The model uses a 40 ms window and 20 ms hop for all sampling rates, ensuring an integer number of frequency bins.
  • Latency control: The resulting algorithmic latency is 40 ms plus the number of look-ahead frames multiplied by 20 ms.This follows from the fixed 40 ms window and 20 ms hop used across sampling rates.

C. Evaluation Metrics · D. Results on the non-Blind URGENT 2025 Test Set

The evaluation combines perceptual, intelligibility, fidelity, downstream-task, and non-intrusive quality metrics with explicit latency measurement. On the non-blind URGENT 2025 test set, the one-for-all model supports 30 latency configurations, balancing algorithmic-latency flexibility with depth-controlled computational latency.

  • C. Evaluation Metrics: Evaluation uses PESQ and ESTOI for perceptual quality and intelligibility, respectively.These are reference-based metrics.
  • C. Evaluation Metrics: Downstream evaluation includes task-independent SBERT and LPS metrics plus ASR character accuracy.SBERT denotes SpeechBERTScore, while LPS denotes Levenshtein Phoneme Similarity.
  • C. Evaluation Metrics: Non-intrusive perceptual quality is measured using DNSMOS and NISQA.The supplied passage identifies both metrics for non-intrusive perceptual-quality assessment.
  • C. Evaluation Metrics: Algorithmic latency is computed as 40 ms + (# look-ahead × 20 ms), while computational latency is measured per frame on an NVIDIA A100 GPU with 16 kHz input.Measurements on NVIDIA 3090 and 4090 GPUs were reported to fall within a similar range; torch.compile and CUDA graphs were not applied.
  • D. Results on the non-Blind URGENT 2025 Test Set: The model provides 30 latency configurations through 10 exit layers from 3 to 12 and 3 look-ahead settings from 0 to 2.This jointly varies computational latency through exit depth and algorithmic latency through look-ahead.
  • D. Results on the non-Blind URGENT 2025 Test Set: The evaluation compares the proposed method with noisy speech, non-causal TF-GridNet, specialized models, and early-exit.The specialized model is described as a performance upper bound, while TF-GridNet uses early-reflected speech as its learning target.
  • D. Results on the non-Blind URGENT 2025 Test Set: Parallel conv. (MoE) achieves performance comparable to early-exit while additionally controlling algorithmic latency through parallel convolutional layers.The passage contrasts this flexibility with early-exit’s computational-latency control and notes early-exit generally falls short of the specialized model.
  • D. Results on the non-Blind URGENT 2025 Test Set: For UTMOS, increasing model depth yields larger gains than adding look-ahead, whereas ASR accuracy improves substantially with one look-ahead frame and only marginally with a second.These observations come from evaluation across the finer grid of total latency configurations.

E. Comparison with Other Real-Time Speech Enhancement Models

The paper compares its universal speech enhancement model with existing real-time systems on the VoiceBank-DEMAND benchmark because no prior open-source model addressed the full universal setting. Baselines include DEMUCS, DeepFilterNet3, Diffusion Buffer, and Stream.FM, evaluated under a training–testing dataset mismatch.

  • Comparison setup: No prior open-source real-time universal speech enhancement model was identified for complex degradations and varying sampling rates in the URGENT Challenge setting.The authors therefore use the VoiceBank-DEMAND benchmark for comparison with existing real-time enhancement models.
  • Comparison setup: The comparison uses DEMUCS, DeepFilterNet3, Diffusion Buffer, and Stream.FM as real-time speech enhancement baselines.Most baseline results are taken directly from reference [19].
  • Comparison setup: None of the compared models except DEMUCS were trained on the VoiceBank-DEMAND training set, creating a training–testing dataset mismatch.For the proposed model, the comparison begins with exit layer 8 and look-ahead 0.

F. Practical Deployment

The model can be evaluated and configured on users’ hardware for specific latency budgets by selecting early-exit layers and look-ahead settings. After selection, retaining only the needed layers and convolutional branch yields a specialized-model-sized deployment without additional footprint.

  • Deployment configuration: Users can evaluate total latency across early-exit layers and look-ahead configurations on their own hardware, then choose a setting matching their latency budget.The selected configuration determines the retained network depth and look-ahead branch.
  • Deployment configuration: Retaining layers through the selected exit point and the convolutional branch for the chosen look-ahead produces a model the same size as a specialized model.This deployment has no additional footprint.
  • Latency constraints: Computational latency must satisfy both Equations (1) and (2), while algorithmic latency is governed only by Equation (1).With limited computational resources, users can increase the number of look-ahead frames.

V. FUTURE WORK · VI. CONCLUSION

The paper presents a single real-time speech enhancement model with explicit control over algorithmic and computational latency. Future work targets faster inference and a smaller performance gap between shallow and deep outputs.

  • V. FUTURE WORK: The framework is designed as a flexible one-for-all model deployable under diverse conditions.The paper’s stated focus is deployment flexibility across diverse conditions.
  • V. FUTURE WORK: Pruning and quantization are identified as promising directions for accelerating inference and can be combined with the framework.These techniques are presented as natural extensions rather than reported results.
  • V. FUTURE WORK: Future work also aims to further reduce the performance gap between shallow and deep outputs.The passage specifically points to knowledge distillation strategies inspired by large-to-small language model compression.
  • VI. CONCLUSION: The proposed framework explicitly controls both algorithmic and computational latency within a single real-time universal speech enhancement model.This is the central contribution stated in the conclusion.
  • VI. CONCLUSION: Parallel convolutional layers enable flexible adjustment of look-ahead frames for algorithmic latency control.The conclusion associates configurable look-ahead with algorithmic latency adjustment.
  • VI. CONCLUSION: An early-exit mechanism enables dynamic computational latency control through variable network depth.The mechanism provides computational-latency flexibility during inference.
Loading 2606.25621v2…