PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Buffer for Online Generative Speech Enhancement

Bunlong Lay, Rostislav Makarov, Simon Welker, Maris Hillemann, Timo Gerkmann

arXiv:2510.18744v1eess.AScs.LGcs.SD

TL;DR

Online speech enhancement needs low-latency, low-computation processing, while diffusion enhancement typically requires many neural-network evaluations. The Diffusion Buffer aligns physical and Diffusion time, uses a look-ahead-matched UNet and DP loss, and reduces latency while improving performance, including on unseen noise.

  • Problem

    Online enhancement requires one low-latency computation per incoming frame, whereas diffusion models generally need many costly evaluations and predictive models can distort speech under unseen acoustic conditions.

    Source: §I · p. 1

  • Method

    The Diffusion Buffer progressively denoises frames through physical time using one neural-network call per incoming frame, with a look-ahead-aligned UNet and Data Prediction loss.

    Source: §I · p. 2, §IV · p. 4

  • Results

    The extended Diffusion Buffer reduces latency from 320–960 ms to 32–176 ms while outperforming the original method and predictive enhancement on unseen noisy speech data.

    Source: §I · p. 1, §VII · p. 9, §VIII · p. 11

  • Takeaways & Limitations

    Online generative diffusion enhancement can run on consumer-grade GPUs while retaining an advantage over predictive enhancement on unseen noise types.

    Source: §I · p. 2, §VII · p. 9, §VIII · p. 11

Abstract

from arXiv · show

Online Speech Enhancement was mainly reserved for predictive models. A key advantage of these models is that for an incoming signal frame from a stream of data, the model is called only once for enhancement. In contrast, generative Speech Enhancement models often require multiple calls, resulting in a computational complexity that is too high for many online speech enhancement applications. This work presents the Diffusion Buffer, a generative diffusion-based Speech Enhancement model which only requires one neural network call per incoming signal frame from a stream of data and performs enhancement in an online fashion on a consumer-grade GPU. The key idea of the Diffusion Buffer is to align physical time with Diffusion time-steps. The approach progressively denoises frames through physical time, where past frames have more noise removed. Consequently, an enhanced frame is output to the listener with a delay defined by the Diffusion Buffer, and the output frame has a corresponding look-ahead. In this work, we extend upon our previous work by carefully designing a 2D convolutional UNet architecture that specifically aligns with the Diffusion Buffer's look-ahead. We observe that the proposed UNet improves performance, particularly when the algorithmic latency is low. Moreover, we show that using a Data Prediction loss instead of Denoising Score Matching loss enables flexible control over the trade-off between algorithmic latency and quality during inference. The extended Diffusion Buffer equipped with a novel NN and loss function drastically reduces the algorithmic latency from 320 - 960 ms to 32 - 176 ms with an even increased performance. While it has been shown before that offline generative diffusion models outperform predictive approaches in unseen noisy speech data, we confirm that the online Diffusion Buffer also outperforms its predictive counterpart on unseen noisy speech data.

I. INTRODUCTION

Online speech enhancement processes streaming noisy speech frame by frame under tight latency and computational constraints. Diffusion models offer better robustness to unseen noise but ordinarily require too many neural-network evaluations for online use.

I. INTRODUCTION

  • Online speech enhancement outputs one enhanced frame for each incoming noisy frame, with up to 200 ms input-to-output delay allowed.This supports live applications such as video conferencing and VoIP, but requires low computational latency and models capable of handling unseen real-world data.Source: §I · p. 1
  • Predictive models learn noisy-to-clean mappings from fixed noise types and SNRs, so mismatched real-world conditions can produce unpleasant distortions.Source: §I · p. 1
  • Diffusion models learn clean-speech distributions and can perform well on unseen data, but reverse-process enhancement requires many evaluations of a large neural network.Source: §I · p. 1
  • More than 30 score-model evaluations may be needed for good diffusion performance, making the computational complexity too high for many consumer-grade GPUs.Source: §I · p. 1
  • The Diffusion Buffer extends prior work that achieved online generative enhancement on an NVIDIA RTX 4080 Laptop GPU with 320–960 ms latency.Source: §I · p. 1
  • The extended method aligns a 2D convolutional UNet with Diffusion Buffer look-ahead, replaces DSM with DP loss, lowers latency to 32–192 ms, and outperforms predictive enhancement on unseen data.Source: §I · p. 2

D. Online Processing

Online processing evaluates an enhancement model on each newly arriving STFT frame together with a bounded history of past frames. Frames-lag controls latency and look-ahead, while real-time operation requires processing faster than frames arrive.

D. Online Processing

  • Each incoming STFT frame is processed with available past frames, typically limiting history to a few hundred milliseconds through chunk-based processing.Source: D. Online Processing · pp. 2–3
  • The model stores the last M frames, then sends the d-th last output frame to the listener; larger Frames-Lag increases algorithmic latency.Source: D. Online Processing · p. 3

1) Latency:

  • Algorithmic latency is the hardware-independent delay intrinsic to the processing system, while total latency adds one frame-hop duration.Source: 1) Latency · p. 3
  • Increasing Frames-Lag trades higher algorithmic latency for greater maximal look-ahead in the enhanced output.Source: 1) Latency · p. 3

2) Real-time-factor:

  • Online enhancement requires real-time factor below 1, because RTF ≥1 causes processing delays to accumulate as new frames arrive.Source: 2) Real-time-factor · p. 3

III. DIFFUSION MODELS

The diffusion formulation models speech enhancement as a stochastic process in the STFT domain that evolves from clean speech toward noisy observations. A reverse process is then used to recover clean speech.

III. DIFFUSION MODELS

1) Denoising Score Matching:

The Diffusion Buffer applies multiple reverse steps across successive physical frames while calling the neural network only once per incoming frame. Its DP loss enables adjustable latency, and its receptive-field design aligns processing with available look-ahead.

B. Latency considerations for streaming data

IV. DIFFUSION BUFFER

  • The proposed 2D convolutional UNet has a receptive field aligned with the Diffusion Buffer look-ahead constraint, avoiding unavailable future-frame inputs.Source: §IV · p. 4
  • The buffer places newer frames at larger Diffusion time-steps and progressively denoises older frames, outputting a frame further in the past as each new frame arrives.Source: §IV · p. 4
  • DSM fixes the output delay, whereas DP permits the listener-directed frame to vary from 1 through B during streaming, trading enhancement performance for latency.Source: §IV · p. 4
  • The perturbed input contains buffer frames at different Diffusion time-steps, while the current streamed frame is placed at the final buffer position.Source: §IV · pp. 4–5

A. Training

Training samples streamed-data conditions by padding and cropping STFT segments, then applies either DSM or DP optimization to the Diffusion Buffer network.

A. Training

  • Training pads clean and noisy STFT files with K−1 leading zeros and randomly crops paired segments of K frames.The padding mimics streamed-data initialization before optimization.Source: A. Training · p. 5
  • The procedure samples an ascending B-step Diffusion-time sequence beginning at ϵ and constructs the noisy inputs across frequencies and frames.Noise is arranged in matrices Z and Σ for the Diffusion Buffer inputs.Source: A. Training · p. 5
  • The network can optimize either the Denoising Score Matching loss or the Data Prediction loss.The DSM loss uses elementwise division by Σ, while the DP loss targets clean speech over the final B frames.Source: A. Training · p. 5

B. Online inference for streamed data

Online inference maintains a K-frame Diffusion Buffer, adds noise to each incoming frame, and performs one reverse update across the buffer before selecting an output frame.

B. Online inference for streamed data

  • Each incoming frame is appended to the most recent K-frame stream context, initialized with zero frames, and processed by one reverse step.Algorithm 2 operates on fixed Diffusion time-steps and returns an enhanced stream output.Source: B. Online inference for streamed data · p. 5
  • The new frame is perturbed to Diffusion time-step t_B, after which the trained network reverses all frames in the buffer by one step.DSM uses EuM for the reverse process, whereas DP uses the DP reverse update.Source: B. Online inference for streamed data · p. 5
  • The algorithmic latency is determined by the selected frame position and differs between DSM and DP processing.The supplied passage states separate latency expressions for DSM and DP, but their displayed terms are incomplete.Source: B. Online inference for streamed data · p. 5
  • DSM outputs the frame that has completed B reverse steps, while DP can output the d-th-last frame because each evaluation provides a clean-speech estimate.DP therefore permits flexible output-frame selection during inference instead of fixing the output to the B−1-th-last frame.Source: B. Online inference for streamed data · p. 5

C. Look-ahead Constraint of the Diffusion Buffer

The Diffusion Buffer requires output frames to depend on a bounded future context, motivating an architecture whose receptive field aligns with the buffer’s look-ahead.

C. Look-ahead Constraint of the Diffusion Buffer

  • The Diffusion Buffer’s output frame O[i+K−B] depends on input frames Y[i+K−B:K], defining its look-ahead constraint.The authors hypothesize that satisfying this constraint benefits low-latency output frames.Source: C. Look-ahead Constraint of the… · p. 6

V. ARCHITECTURE DESIGN

  • The architecture-design section develops modifications to NCSN++ because its standard receptive field induces excessive future dependence.The section presents a starting architecture, analyzes its receptive field, and designs a constraint-compatible alternative.Source: §V · p. 6

A. NCSN++

  • NCSN++ uses a multi-resolution 2D U-Net for complex spectrograms, with FIR-filter downsampling and upsampling, self-attention, GroupNorm, and progressive growing.These components form the baseline architecture considered for speech enhancement.Source: A. NCSN++ · p. 6

B. Receptive field of NCSN++

  • Self-attention and GroupNorm are non-causal, so the receptive field is at least the full input sequence when they are retained.The receptive-field discussion therefore excludes these modules and focuses on the remaining operations.Source: B. Receptive field of NCSN++ · p. 6
  • Symmetric downsampling and upsampling create a symmetric receptive field requiring approximately r/2 future frames for an output.For the 18M-parameter model, r=200 yields about 100 future tokens, or roughly 1.6 seconds of look-ahead; unavailable future inputs are zero-padded.Source: B. Receptive field of NCSN++ · p. 6

C. Modification of NCSN++

The modified UNet removes standard sources of future dependence and uses block-causal convolutional design choices so its receptive field matches the Diffusion Buffer’s look-ahead.

C. Modification of NCSN++

  • The proposed modifications replace fixed FIR resampling with learnable 2D convolutions, remove attention and progressive growing, and replace GroupNorm with cumulative GroupNorm.This intermediate Sym-NCSN++ still has a symmetrical receptive field.Source: C. Modification of NCSN++ · pp. 6–7
  • Prior causal convolutions are insufficient when downsampling and upsampling are composed with stride greater than one, motivating three design choices for block causality.The proposed architecture is intended to fulfill the Diffusion Buffer look-ahead constraint.Source: C. Modification of NCSN++ · p. 7, D. Designing the Block-Causal… · p. 7

D. Designing the Block-Causal Architecture

  • For the illustrated stride-2 example, the output preserves input length, and the final outputs have one-frame and zero-frame look-ahead respectively.The construction crops excess upsampling output and avoids future dependence for the newest output frame.Source: D. Designing the Block-Causal… · p. 7
  • With global stride g equal to buffer length B, block-causal dependencies match the Diffusion Buffer look-ahead constraint.Each output block depends only through its corresponding input block, while look-ahead is bounded by g.Source: D. Designing the Block-Causal… · pp. 7–8
  • The three choices omit right padding, add exact left zero-padding during downsampling, and left-crop upsampling outputs.Together, these choices yield the block-causal BC-NCSN++ architecture and were experimentally verified by gradient backpropagation.Source: D. Designing the Block-Causal… · pp. 7–8

E. Modification for Diffusion Buffer

The Diffusion Buffer uses BC-NCSN++ with diffusion-specific time embeddings and compares it with predictive NCSN++ variants for online enhancement.

E. Modification for Diffusion Buffer

  • BC-NCSN++ adapts the network to the Diffusion Buffer by expanding vector-valued diffusion times through Fourier embeddings and 2D convolutions before adding them to sampling operations.The embeddings are transformed separately for each downsampling and upsampling operation.Source: E. Modification for Diffusion Buffer · p. 8

A. Data representation

  • Each audio input is represented as a compressed complex STFT using 32 ms windows, 16 ms hops, and randomly cropped 128-frame training sequences.The approximately 2-second crops use β = 0.5 and α = 0.15 for magnitude compression.Source: A. Data representation · p. 8

1) Diffusion Models:

  • DB-BBED experiments vary the online or alternative training objective and use either NCSN++ or BC-NCSN++, with global stride set to buffer length for the latter.This stride choice ensures the Diffusion Buffer look-ahead constraint is fulfilled.Source: 1) Diffusion Models · p. 8

2) Predictive:

  • Predictive NCSN++ and BC-NCSN++ directly estimate the clean target from degraded observations using MSE, while SEMamba is trained causally with its official streaming pipeline.The predictive diffusion-network baselines omit diffusion states and time embeddings.Source: 2) Predictive · p. 8

C. Parameterization and Training of BC-NCSN++, NCSN++

The study trains and evaluates BC-NCSN++ and NCSN++ variants on matched and mismatched noisy-speech data using standard enhancement metrics. Online results show block-causal and generative advantages under low-latency or unseen-noise conditions.

D. Datasets

  • The experiments use EARS-WHAM-v2 with 54 training hours, 1.1 validation hours, and 3.5 test hours, plus a 1.5-hour EARS-General impulsive-noise test set.EARS-General mixes selected impulsive noises with clean test files at -7 to 5 dB SNR.Source: D. Datasets · p. 9

E. Metrics

  • Evaluation reports PESQ, ESTOI, SI-SDR, SI-SIR, and DistillMOS, covering speech quality, intelligibility, distortion, interference, and predicted opinion score.PESQ ranges from 1 to 4.5, while ESTOI ranges from 0 to 1.Source: E. Metrics · p. 9

VII. RESULTS

  • During online evaluation, generative methods use the chunk-based Diffusion Buffer while predictive methods use chunk-based processing, with inference context reduced to 64 frames without reduced performance.SEMamba uses its official streaming pipeline.Source: §VII · p. 9
  • At small Frames-Lag values, BC-NCSN++ degrades less than Sym-NCSN++, while both reach similar maximal PESQ when Frames-Lag is sufficiently large.The comparison uses chunk-based online processing on EARS-WHAM-v2.Source: §VII · p. 9, Figure 4 · p. 9
  • On unseen impulsive noise, generative methods outperform the predictive method when using more reverse steps, while the predictive method sometimes cannot remove the impulsive noise.On the matched test set, the predictive counterpart has a small advantage in most metrics.Source: §VII · p. 9

A. Analysis of Block-Causal NCSN++

The section analyzes block-causal architectures, improves the Diffusion Buffer for lower latency, and compares generative and predictive enhancement on matched and mismatched noise conditions.

A. Analysis of Block-Causal NCSN++

  • BC-NCSN++ retains similar maximal PESQ to Sym-NCSN++ while avoiding much of the low-delay degradation caused by symmetric zero-padding.At d = 0, BC-NCSN++ with g = 16 exceeds Sym-NCSN++ by more than 0.16 PESQ.Source: A. Analysis of Block-Causal NCSN++ · p. 10

B. Ablation on Diffusion Buffer in matched conditions

C. Predictive versus Generative

  • On mismatched EARS-General data, generative DB-BBED outperforms its predictive counterpart, including approximately 7 dB higher SI-SIR at larger d.The SI-SIR result indicates that the predictive baseline struggles to remove unseen impulsive noises.Source: Figure 5 · p. 10, C. Predictive versus Generative · p. 11

VIII. CONCLUSION

  • The redesigned Diffusion Buffer reduces latency from 320–960 ms to 32–176 ms, improves over the original system, and runs online on an NVIDIA RTX 2080Ti.It combines a look-ahead-aligned 2D convolutional UNet with Data Prediction loss.Source: B. Ablation on Diffusion Buffer in… · p. 10, §VIII · p. 11

X. BIOGRAPHY SECTION

The biography section presents the authors’ educational backgrounds, research positions, and research interests across signal processing, machine learning, speech enhancement, and related fields.

X. BIOGRAPHY SECTION

  • Bunlong Lay researches diffusion-based models for speech enhancement for real-time applications at the University of Hamburg.He previously worked in radar signal processing at Fraunhofer FKIE.Source: §X · p. 12
  • Rostislav Makarov combines experience in production-grade speaker recognition, anti-spoofing, and speech processing with current doctoral research on diffusion-based speech enhancement.He worked as a Machine Learning Research Engineer at ID R&D from 2019 to 2025.Source: §X · p. 13
  • Simon Welker researches machine learning methods for inverse problems in speech processing and X-ray imaging at the University of Hamburg and DESY.He is a PhD student in the labs of Timo Gerkmann and Henry N. Chapman.Source: §X · p. 13
  • Maris Hillemann is a University of Hamburg computer-science master’s student and student assistant in its Signal Processing group.Source: §X · p. 13
  • Timo Gerkmann is a University of Hamburg professor whose research spans statistical signal processing and machine learning for speech and audio applications.His applications include communication devices, hearing instruments, audiovisual media, and human-machine interfaces.Source: §X · p. 13

Related papers

Loading 2510.18744v1…