PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Streamable Generative Speech Restoration with Flow Matching

Simon Welker, Bunlong Lay, Maris Hillemann, Tal Peer, Timo Gerkmann

arXiv:2512.19442v3eess.SPcs.LGcs.SD

TL;DR

Stream.FM tackles the computational and latency barriers that have limited generative speech restoration in real-time communication. It combines frame-causal flow matching, buffered streaming inference, an optimized DNN, and learned few-step solvers across several restoration tasks. The authors report high-quality real-time processing on consumer GPUs, with 32 ms algorithmic and 48 ms total latency, and state-of-the-art generative streaming performance.

  • Problem

    Generative speech restoration requires repeated evaluations of large DNNs, while streaming systems must operate with fixed latency and limited future context.

  • Method

    Stream.FM combines a frame-causal flow-matching model, buffered streaming inference, an optimized causal DNN, and learned Runge-Kutta solvers.

  • Results

    Stream.FM supports six streaming speech-restoration tasks and achieves 32 ms algorithmic latency and 48 ms total latency on consumer hardware.

  • Takeaways & Limitations

    The work demonstrates that high-quality multi-step generative speech restoration can be realized in real time on available consumer GPUs.

Abstract

from arXiv · show

Diffusion-based generative models have greatly impacted the speech processing field in recent years, exhibiting high speech naturalness and spawning a new research direction. Their application in real-time communication is, however, still lagging behind due to their computation-heavy nature involving multiple calls of large DNNs. Here, we present Stream$.$FM, a frame-causal flow-based generative model with an algorithmic latency of 32 milliseconds (ms) and a total latency of 48 ms, paving the way for generative speech processing in real-time communication. We propose a buffered streaming inference scheme and an optimized DNN architecture, show how learned few-step numerical solvers can boost output quality at a fixed compute budget, explore model weight compression to find favorable points along a compute/quality tradeoff, and contribute a model variant with 24 ms total latency for the speech enhancement task. Our work looks beyond theoretical latencies, showing that high-quality streaming generative speech processing can be realized on consumer GPUs available today. Stream$.$FM can solve a variety of speech processing tasks in a streaming fashion: speech enhancement, dereverberation, codec post-filtering, bandwidth extension, STFT phase retrieval, and Mel vocoding. As we verify through comprehensive evaluations and a MUSHRA listening test, Stream$.$FM establishes a state-of-the-art for generative streaming speech restoration, exhibits only a reasonable reduction in quality compared to a non-streaming variant, and outperforms our recent work (Diffusion Buffer) on generative streaming speech enhancement while operating at a lower latency.

I. INTRODUCTION

Stream.FM addresses the gap between high-quality generative speech restoration and practical low-latency streaming by combining flow matching with buffered inference and causal architectures. It targets multiple restoration tasks while validating real-hardware feasibility and quality against prior approaches.

  • Generative speech methods offer perceived-quality and generalization advantages, especially when restoration requires reconstructing missing information.
  • Multiple DNN evaluations inside numerical solvers make generative methods computationally intensive, challenging real-time deployment.
  • Reported real-time claims can be misleading because offline real-time factors exploit temporal parallelism and optimized large-tensor processing unavailable in streaming inference.
  • Stream.FM aims to close this gap with a flow-matching model operating below 50 ms total latency, building on prior 48 ms consumer-hardware streaming inference.
  • Stream.FM extends streaming generative restoration beyond speech enhancement to dereverberation, bandwidth extension, codec post-filtering, STFT phase retrieval, and Mel vocoding.
  • Prior streaming approaches remain limited by higher latency, missing real-hardware validation, task-specific coverage, proprietary data, or speech-enhancement-only demonstrations.

B. Latency definitions

The paper separates latency imposed by the causal signal-processing design from latency caused by computation, then combines them into total latency. Achieving total latency requires streaming processing faster than real time.

  • Algorithmic latency is the attainable latency on infinitely fast hardware, while total latency equals algorithmic latency plus per-frame processing time.The paper denotes these quantities as ℓalg and ℓtot=ℓalg+ℓproc.
  • For frame-causal STFT processing, algorithmic latency is determined by the synthesis window, while processing latency is bounded by the synthesis hop.
  • With matched analysis and synthesis configurations, total latency is expressed using the analysis window length and hop.
  • Attaining the reported total latency on real hardware requires a streaming real-time factor below 1.

C. Explicit Runge-Kutta ODE solvers

The streaming method makes multi-step flow inference feasible by caching frame-causal convolution states, so each new frame requires only incremental layer evaluations. Learned Runge-Kutta solvers further target quality under fixed few-step budgets.

  • Explicit Runge-Kutta ODE solvers: Euler integration uses N DNN evaluations but has relatively large approximation error for a fixed evaluation budget, motivating explicit Runge-Kutta solvers.
  • Buffered streaming inference: A frame-causal DNN can process one new output frame by evaluating only the latest required positions while storing past activations in buffers.The figure illustrates a receptive field of 9 where only 3 frames are evaluated per layer.
  • Explicit Runge-Kutta ODE solvers: Learned Runge-Kutta coefficients are optimized from speech data to improve quality at fixed low NFE without retraining the flow model.
  • Multi-step streaming diffusion: The method performs the full numerical denoising process frame by frame, whereas lower-latency approximations such as Diffusion Buffer can incur quality degradation.
  • Buffered streaming inference: The buffered scheme extends to multiple solver calls by maintaining independent cache buffers for every DNN call and layer.
  • Buffered streaming inference: Streaming execution reproduces offline causal-model computation up to floating-point error, avoiding redundant computation and allowing standard batched training.

C. DNN architecture

Stream.FM uses a modified NCSN++-style complex-STFT CNN tailored for frame-causal streaming. The architecture preserves frequency processing while replacing temporal downsampling with causal dilation and removing attention.

  • The model operates on complex-valued STFTs represented by separate real and imaginary channels using a modified NCSN++ architecture.
  • Downsampling and upsampling occur only along frequency, while time context comes from time-dilated causal convolutions.The temporal dilation is 2.
  • The main streaming models remove temporal FIR filtering and attention layers to simplify the architecture while retaining streaming capability.Causal fixed-window attention is possible in principle but is omitted from the main models.
  • For speech enhancement, a predictor first estimates an initial signal representation that is then used as the input target for the generative model.
  • The predictor is trained with a multi-resolution magnitude-STFT L1 loss using four Hann-window configurations with 50% overlap.

E. Custom learned low-NFE ODE solvers

The paper learns specialized low-NFE Runge–Kutta solvers for pretrained flow-matching models, optimizing speech-specific quality without retraining the model or materially increasing inference computation. The design constrains the learned coefficients for consistency and first-order convergence, while also exploring compressed network weights to trade quality against compute.

  • Custom learned low-NFE ODE solvers: Learned Runge–Kutta solvers optimize speech quality at a fixed DNN-evaluation budget without retraining or finetuning the flow model.The pretrained vector field remains frozen while solver coefficients are learned by backpropagating through the solved ODE path.
  • Custom learned low-NFE ODE solvers: The solver loss combines negative SpeechBERTScore with multi-resolution log-magnitude STFT MSE to reduce phonetic hallucinations and recover high-frequency detail.The authors also consider a differentiable PESQ-oriented alternative for signal fidelity, using analysis windows of 320, 512, and 640 samples with 75% overlap.
  • Custom learned low-NFE ODE solvers: Coefficient constraints ensure all model calls contribute to the estimate and guarantee consistency with at least first-order convergence.The learned schemes do not guarantee convergence order above one because their coefficients need not satisfy higher-order algebraic conditions.
  • Custom learned low-NFE ODE solvers: For tasks without a predictive network, the method uses five flow-model evaluations, initialized with one Ralston-2 step followed by one Ralston-3 step.The learned Runge–Kutta parameters are task-specific.
  • Weight compression: Weight compression decomposes selected 3×3 convolutions using depthwise and pointwise factors derived from per-channel SVD, followed by fine-tuning.The compression is applied to layers with at least nine output channels, and the singular values are distributed across the two factors for fine-tuning stability.

IV. EXPERIMENTS

The experiments evaluate Stream.FM across six streaming speech-restoration tasks using corruption-specific datasets, representations, and baselines. The setup includes enhancement, dereverberation, codec post-filtering, bandwidth extension, STFT phase retrieval, and Mel vocoding, with task-specific signal models and latency-oriented STFT choices.

  • Task coverage: The study covers speech enhancement, dereverberation, codec post-filtering, bandwidth extension, STFT phase retrieval, and Mel vocoding.Table I defines the corruption and feature-representation operators used for these restoration tasks.
  • Speech enhancement and dereverberation: Speech enhancement uses additive uncorrelated noise on EWv2, while dereverberation uses convolution with sampled room impulse responses from EARS-Reverb v2.EWv2 clean utterances also provide the basis for the other tasks except dereverberation.
  • Codec post-filtering: Codec post-filtering uses Stream.FM after the streaming Lyra V2 codec at 3.2 kbit/s, replacing the non-causal and computationally expensive DAC setup.The configuration is motivated by streaming speech coding on consumer devices.
  • Bandwidth extension: Bandwidth extension randomly downsamples clean speech to 8 or 4 kHz, producing 4 or 2 kHz frequency cutoffs for reconstruction.The two downsampling conditions define the BWE corruption model.
  • Phase retrieval and Mel vocoding: STFT phase retrieval adapts DiffPhase to streaming with 50% overlap, while Mel vocoding applies lossy Mel compression to STFT magnitudes before reconstruction.Mel vocoding uses 32 ms windows and 16 ms hops to reduce latency relative to the referenced configuration.
  • Common experimental configuration: The common preprocessing uses 16 kHz audio, 32 ms Hann windows, 16 ms hops, 50% overlap, magnitude compression, and zero-padding after discarding the Nyquist band.Normalization differs by task: enhancement normalizes input and target independently, whereas other tasks normalize them jointly.

C. DNN configuration and training

The experimental system uses a four-level Stream.FM U-Net, task-specific training and baselines, and end-to-end latency measurements on a consumer laptop GPU. Evaluation combines objective speech metrics with listening tests comparing streaming and non-streaming methods.

  • DNN configuration and training: The Stream.FM backbone has four U-Net levels, channel sizes [128, 256, 256, 256], two residual blocks per level, and 27.9M parameters.The speech-enhancement initial predictor has 24.6M parameters after reducing its input channels and removing time conditioning.
  • DNN configuration and training: Flow models train for 150,000 steps on two NVIDIA RTX A6000 GPUs using 2-second snippets, batch size 12 per GPU, SOAP optimization, cosine annealing, warmup, and gradient clipping.These settings define the main training procedure for each task.
  • DNN configuration and training: The enhancement system also includes a joint predictive-generative variant with 24 ms total latency using 16 ms frames and 8 ms hops.The initial predictor is trained first and frozen during flow-model training.
  • Evaluation: Objective evaluation reports PESQ, ESTOI, SI-SDR, LSD, WER, NISQA, WVMOS, and DiMOS across model outputs sampled at 16 kHz.WER uses QuartzNet15x5Base-En with transcripts from clean audio as references.
  • Evaluation: Listening tests compare Stream.FM with non-streaming flow matching, Diffusion Buffer, and DEMUCS for enhancement and with corresponding baselines for bandwidth extension.Each experiment has 12 participants rating eight randomly sampled utterances on a 0–100 overall-quality scale.
  • Runtime evaluation: Runtime is measured on a single NVIDIA RTX 4080 Laptop GPU using PyTorch FLOP counting and CUDA event wall-clock timings.The setup is intended to assess practical rather than purely theoretical runtime.

V. RESULTS AND DISCUSSION

Stream.FM performs competitively across streaming speech enhancement and restoration tasks, with learned solvers improving selected metrics and quality remaining close to non-streaming performance. Results also show task-dependent solver trade-offs, latency-quality trade-offs, and limitations for bandwidth extension.

  • A. Speech enhancement: Stream.FM attains the best or second-best values among streaming speech-enhancement methods in almost all metrics, with especially strong DistillMOS, WVMOS, and NISQA improvements.At matched low latency, it outperforms Diffusion Buffer with four Euler steps and performs similarly to or better than the higher-latency d=9 variant.
  • A. Speech enhancement: The non-streaming FM baseline outperforms streaming models, but streaming quality degradation is considered acceptable, and the 16 ms algorithmic-latency variant reaches approximately 24 ms total latency with minor additional reduction.Solver choice changes the quality profile: LRK4 improves PESQ, WVMOS, and WER over Euler4, while the PESQ-trained solver improves PESQ, WER, and LSD at some non-intrusive-metric cost.
  • A. Speech enhancement: SFM achieves the best ESTOI and DistillMOS among streaming methods on VoiceBank-DEMAND and is second-best in PESQ, while causal SEMamba degrades SI-SDR below the noisy input.Both learned Runge-Kutta variants transfer reasonably across datasets despite training on EWv2.
  • A. Speech enhancement: SFM is clearly preferred over low-latency baselines in listening tests and receives median ratings similar to higher-latency Diffusion Buffer.The non-streaming FM baseline receives an excellent median score around 90.
  • B. Dereverberation: For dereverberation, all solvers except Euler1 improve reverberant audio; the PESQ-trained LRK solver gives the best PESQ, NISQA, and WER, while SFM Euler5 slightly exceeds SGMSE+ and uses 12× fewer NFE.The non-causal FM baseline performs best overall, benefiting from future information for estimating room-response characteristics and suppressing reverberation.
  • D. Bandwidth extension: For bandwidth extension, learned RK5 performs best on all metrics except PESQ and SI-SDR and receives the highest listening-test scores, but its PESQ-trained variant reaches PESQ 3.71 while losing non-intrusive metrics and high-frequency detail.AnyEnhance performs well on non-intrusive metrics but trails SFM and FM on all intrusive metrics and WER.

E. STFT phase retrieval / Mel vocoding

Stream.FM performs strongly on STFT phase retrieval and Mel vocoding while retaining streaming-oriented runtime and memory characteristics. Learned Runge–Kutta solvers improve quality especially for these tasks, although their benefits vary with task curvature and solver transferability.

  • Results: PESQ exceeds 4.1 and ESTOI reaches at least 0.96, with Stream.FM outperforming HiFi-GAN on all Mel-vocoding metrics except LSD.For STFT phase retrieval, Stream.FM and the non-causal FM baseline outperform DiffPhase except in WER and LSD, using six times fewer NFE.
  • Learned solvers: Learned solvers show a clear advantage over Euler on STFT phase retrieval and Mel vocoding, particularly for WER and LSD.Euler1 produces unusable estimates for these ill-posed nonlinear inverse problems, while learned solvers are especially beneficial on the reported metrics.
  • Runtime: Streaming RTF is the relevant runtime measure: offline RTF consistently and severely underestimates it, while FLOPs have no simple relation to streaming RTF.For example, Diffusion Buffer [22] has lower RTF than Stream.FM at 5 NFE but uses substantially more FLOPs.
  • Runtime: On an AMD Ryzen 7 9800X3D, Stream.FM has streaming RTF N×0.883, permitting real-time operation with one DNN evaluation but not more than one model call.This motivates future work on more efficient DNN architectures.
  • Learned solvers: The LRK benefit depends on task: it is strongest for bandwidth extension, STFT phase retrieval, and Mel vocoding, but weaker for enhancement, dereverberation, and codec-artifact removal.The authors relate this variation to the curvature of the learned velocity field; speech enhancement has the lowest curvature and already works well with one Euler step.
  • Learned solvers: Solver parameters transfer reasonably within enhancement, dereverberation, codec-artifact removal, and bandwidth-extension tasks, but transfer poorly to phase retrieval and Mel vocoding.Conversely, solvers trained for phase retrieval and Mel vocoding appear most universally applicable.

H. Model weight compression for Mel vocoding

Weight compression can trade a small amount of model capacity for additional solver evaluations within a fixed runtime budget. In Mel vocoding, this tradeoff favors slight compression when it enables higher NFE.

  • Model compression: At maximum hardware-supported NFE, the compressed T=6 model with NFE=7 outperforms the uncompressed T=K=9 model with NFE=5 across all metrics while slightly reducing GFLOPs per frame.With NFE fixed at 5, stronger compression reduces all metrics but linearly decreases GFLOPs per frame.
  • Model compression: Slight model compression is preferable to no compression when the saved computation allows additional solver evaluations.This result supports using compression to optimize the compute–quality tradeoff rather than minimizing model size alone.
  • Architecture ablations: Non-causal bottleneck attention strongly improves dereverberation quality, whereas causal windowed attention provides only a minor increase.The non-causal variant is slightly better in PESQ and DistillMOS than the non-causal FM model, while the causal variant preserves streaming capability.
  • Architecture ablations: Removing attention simplifies and improves streaming efficiency because future information is especially helpful for dereverberation.The authors therefore omit attention in their streaming-oriented architecture despite the quality advantage of non-causal attention.
  • Architecture ablations: Channel concatenation for skip connections gives no improvement and costs 62.9N additional GFLOPs per second, supporting simple addition instead.One residual block also decreases all three reported metrics, supporting the use of two residual blocks.

S.VIII. MODEL IMPLEMENTATION AND OPTIMIZATION

The implementation supports frame-wise streaming with explicit state management and optimizations aimed at meeting tight per-frame budgets. Learned Runge–Kutta solvers are deliberately fixed-budget methods rather than general-purpose numerical integrators.

  • Implementation: The streaming API separates stateless layers from explicit state using init_state() and forward_step(frame, state).This supports one shared DNN instance with multiple state collections, one for each solver step.
  • Implementation: In-place state updates improve compatibility with torch.compile and CUDA graphs, while further optimization remains future work.The design differs from purely functional APIs that construct a new state object for every call.
  • Optimization: Initial processing exceeded the 16 ms per-frame budget even at NFE=3 because CPU–GPU overhead dominated runtime.Model compilation reduced this overhead only partially, motivating additional implementation optimization.
  • Numerical properties: All learned Runge–Kutta methods are first-order consistent and have convergence order at least 1 by construction, with zero-stability inherited from consistent one-step schemes.The imposed coefficient constraints ensure these numerical properties.
  • Numerical properties: Higher convergence orders are not attained, but this is acceptable because the solvers target a fixed NFE and fixed step Δt=1 rather than general-purpose integration.The learned solvers are trained for one call over the fixed horizon τ∈[0,1] for each frame.
  • Numerical properties: The absolute-stability function R(z) is evaluated from the learned Runge–Kutta coefficients A and b, with |R(z)|<1 defining the stability region.The unusually large stability regions for phase retrieval and Mel vocoding may relate to their cross-task reliability, though deeper investigation is deferred.

C. Speech enhancement

The cross-task LRK evaluation tests whether solvers learned for one speech-restoration task transfer to others, while trajectory analysis examines curvature as a possible explanation for solver quality differences.

  • The mismatched experiment forms task–model–dataset and solver combinations, then measures each combination's empirical metric difference from its matched pairing.
  • LRK solvers trained for speech enhancement, dereverberation, codec artifact removal, and bandwidth extension transfer reasonably well within that task group, unlike those trained for phase retrieval or Mel vocoding.
  • Phase-retrieval and Mel-vocoding solvers appear most universally applicable, sometimes improving mismatched tasks, whereas the speech-enhancement solver is least widely applicable.
  • The Lyra model performs better with phase-retrieval or Mel-vocoding solvers than with its task-specific solver, although LSD increases slightly.
  • Curvature is computed from the acceleration component perpendicular to velocity, because acceleration parallel to existing velocity does not bend the trajectory.

S.XI. SNR-SPLIT METRIC EVALUATION ON EARS-WHAM V2

The EARS-WHAM v2 evaluation splits inputs into four SNR ranges and compares metric changes over noisy inputs across speech-enhancement methods. Relative method rankings remain mostly stable across SNR bins and metrics.

  • Across the four SNR bins, all evaluated methods show consistent behavior, with relative rankings mostly preserved for PESQ, SI-SDR, DistillMOS, and WER.
  • The cross-task solver analysis represents metric differences relative to matched task–solver pairings, with rows for task and pretrained model and columns for solvers fitted to listed tasks.Red denotes metric worsening and blue denotes improvement; for WER and LSD, lower values are better.
  • The evaluation reports improvement or degradation over noisy inputs separately by SNR bin for PESQ, SI-SDR, DistillMOS, and WER.The bins are [-2.5, 0], [0, 6], [6, 12], and [12, 17.5] dB, with small gray bars marking 95% confidence intervals.
Loading 2512.19442v3…