Source-linked AI summary
StreamWSR: Streamable and Lightweight Waveform-Domain Neural Speech Super-Resolution
Yuan Tian, Yang Ai, Hui-Peng Du, Zhen-Hua Ling
TL;DR
Speech SR must restore missing high-frequency content while meeting low-latency needs, but spectral approaches involve vocoder-based reconstruction or challenging phase modeling. StreamWSR uses a causal waveform-domain design with compact frame-level modeling and achieves competitive or superior quality and intelligibility against representative baselines.
Problem
Speech SR seeks to restore missing high-frequency components, while practical communication applications call for low-latency operation and spectral approaches involve vocoder-based reconstruction or challenging phase modeling.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
StreamWSR supports zero-look-ahead streaming without vocoder-based reconstruction or explicit phase prediction.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper proposes StreamWSR, a Streamable neural Waveform-domain model for speech Super-Resolution (SR). By adopting a fully causal architecture with compact frame-level waveform representation, the proposed StreamWSR supports zero-look-ahead streaming inference while avoiding vocoder-based reconstruction and explicit phase prediction. Specifically, StreamWSR downsamples the input waveform into a compact frame-level representation using strided causal convolutions. Then, a lightweight causal long-short-term modeling backbone is employed to capture both local waveform structures and long-range historical dependencies under causal constraints. Finally, the modeled output is converted back to the waveform domain through a causal transposed-convolution and combined with the input waveform via a residual connection to generate the final high-resolution speech. Experimental results on 16 kHz speech SR show that StreamWSR achieves competitive or superior speech quality and intelligibility compared with representative waveform- and spectrum-based baselines, while maintaining a zero-look-ahead streaming advantage with only 9M parameters and 2G FLOPs.
1. Introduction
Speech SR must restore missing high-frequency content while meeting low-latency demands. StreamWSR addresses limitations of spectral pipelines with a causal waveform-domain model and reports competitive or superior results against representative baselines.
1. Introduction
- Speech SR restores missing high-frequency components to improve speech quality, intelligibility, and naturalness, while practical applications require low-latency generation from current and past inputs.The motivating use cases include real-time communication, codec-related scenarios, and downstream speech tasks.Source: §1 · p. 1
- Mel-spectrogram systems need vocoders to reconstruct phase, while STFT systems face challenging phase modeling or complex explicit amplitude-phase prediction.These trade-offs motivate waveform-domain SR that avoids vocoder reconstruction and explicit phase prediction under zero-look-ahead constraints.Source: §1 · p. 1
- StreamWSR is a lightweight, fully causal waveform-domain model with compact frame-level representation, and achieves competitive or superior speech quality and intelligibility with 9M parameters and 2G FLOPs.It uses spectral supervision only during training, without inference cost, vocoder reconstruction, or explicit phase prediction.Source: §1 · pp. 1–2
2. Proposed Method
StreamWSR maps an upsampled low-resolution waveform to a high-resolution waveform through causal frame-level modeling. Its waveform generator is paired with training-only spectral adversarial and reconstruction supervision.
2.1. Model Structure
- Causal convolutions compress the upsampled waveform into frame-level features for long-short-term blocks, then a causal transposed convolution produces a residual added to the input waveform.The blocks use causal dilated convolution for local history and masked self-attention for long-range history, supporting zero-look-ahead inference.Source: §2.1 · p. 2
2.2. Training Strategies
- The generator operates directly on waveforms, while spectral representations provide training supervision without adding inference computation.The training framework uses GAN-based supervision to improve perceptual quality and spectral fidelity.Source: §2.2 · p. 2
2.2.1. Multi-resolution Spectral Adversarial Training
- A multi-resolution MDCT discriminator evaluates generated and reference speech at multiple time-frequency scales using a hinge adversarial objective.Feature matching on intermediate discriminator features is also used to stabilize adversarial training and improve perceptual consistency.Source: §2.2.1 · p. 2
2.2.2. Spectral Reconstruction Losses
- The generator loss combines adversarial and feature-matching terms with MDCT and mel-spectrogram reconstruction losses.The MDCT loss weights higher frequency bins slightly more, while the mel loss uses L1 and L2 distances.Source: §2.2.2 · pp. 2–3
3.1. Experimental Setup
Experiments use VCTK speech at a 16 kHz target rate across three upsampling settings. Baselines are compared for speech quality, intelligibility, and computational complexity.
3.1.1. Dataset
- The VCTK-0.92 corpus provides approximately 44 hours of 48 kHz speech from 110 English speakers, resampled to 16 kHz references for three SR settings.Inputs are downsampled to 8, 4, or 2 kHz, corresponding to extension factors of 2, 4, and 8.Source: §3.1.1 · p. 3
3.1.2. Implementation
- Training uses random 16,000-sample crops, AdamW, 600k steps, batch size 16, and a single NVIDIA RTX 3090 GPU.The initial learning rate is 2 × 10^-4 with an exponential decay factor of 0.999.Source: §3.1.2 · p. 3
3.1.3. Baselines
- The four baselines are waveform-based UDM+ and TRAMBA, and spectrum-based FLowHigh and AP-BWE, trained and evaluated on the same dataset.Their approaches include diffusion denoising, Transformer-Mamba modeling, flow matching with vocoder reconstruction, and amplitude-phase prediction, respectively.Source: §3.1.3 · p. 3
3.1.4. Evaluation Metrics
- Evaluation covers LSD and ViSQOL for speech quality, Whisper-derived WER and CER plus STOI for intelligibility, and parameters and FLOPs for complexity.FLOPs are measured for generating one second of 16 kHz speech.Source: §3.1.4 · p. 3
3.2. Results and Analysis
Across baseline comparisons, StreamWSR balances speech quality, intelligibility, complexity, and zero-look-ahead streamability. Ablations show the roles of spectral adversarial guidance and compact frame-level processing.
3.2.1. Comparison with Baseline Speech SR Methods
- StreamWSR outperforms UDM+ and TRAMBA in speech quality, remains comparable in intelligibility to TRAMBA, and compares favorably with FLowHigh and AP-BWE while supporting zero-look-ahead streaming.Against UDM+, it has lower LSD and higher ViSQOL in all three settings; against FLowHigh it has lower LSD in all settings and better intelligibility at the most extreme extension factor.Source: §3.2.1 · pp. 3–4, Table 2 · p. 4
3.2.2. Ablation Studies
- In the 2 kHz→16 kHz ablations, replacing the spectral discriminator degrades quality and intelligibility, while removing strided convolutions worsens LSD and ViSQOL and raises FLOPs to 78.4 G.The full-resolution variant has intelligibility metrics close to StreamWSR; the experiments identify spectral guidance and strided modules as important design components.Source: Table 3 · p. 4, §3.2.2 · p. 4
4. Conclusion
StreamWSR is a fully causal waveform-domain speech super-resolution model designed for zero-look-ahead streaming. On 16 kHz speech SR, it achieves competitive or superior quality and intelligibility versus representative waveform- and spectrum-based baselines, with 9M parameters and 2G FLOPs.
4. Conclusion
- StreamWSR achieves competitive or superior speech quality and intelligibility versus representative waveform- and spectrum-based baselines on 16 kHz speech SR, using 9M parameters and 2G FLOPs.The model is fully causal and supports zero-look-ahead streaming inference.Source: §4 · p. 4
- StreamWSR directly maps low-resolution waveforms to high-resolution waveforms without vocoder-based reconstruction or explicit phase prediction.Its compact frame-level waveform representation and causal long-short-term modeling capture local waveform structures and long-range historical dependencies.Source: §4 · p. 4
- Ablations confirm the effectiveness of the multi-resolution spectral discriminator and compact frame-level waveform representation.Source: §4 · p. 4
- Future work will improve StreamWSR's robustness and generalization and explore applications to more real-time speech and audio tasks.Source: §4 · p. 4