Source-linked AI summary
Towards Balanced Spectral Reconstruction: Spectrally Adaptive Loss for Streaming Speech Enhancement
Haixin Zhao, Nilesh Madhu
TL;DR
Lightweight streaming speech enhancement must balance noise suppression with spectral fidelity because phase-aware losses can over-attenuate mid-to-high frequencies. The paper introduces sigmoid-weighted and signal-dependent spectrally adaptive STFT losses, evaluated with the HyST-Net streaming backbone. Both improve high-frequency reconstruction, while the adaptive loss additionally improves mid-frequency reconstruction and yields more balanced spectral reconstruction overall.
Problem
Phase-aware STFT losses can cause magnitude over-attenuation in mid-to-high frequencies, while uniform magnitude-loss weighting trades noise suppression against perceptual fidelity.
Method
The paper proposes frequency- and signal-dependent spectrally weighted STFT losses and evaluates them using HyST-Net, a lightweight hybrid MHA-GRU streaming backbone.
Results
Both proposed losses consistently improve high-frequency spectral reconstruction, while LAdp further enhances mid-frequency reconstruction and maintains overall enhancement quality.
Takeaways & Limitations
The spectrally adaptive loss achieves more balanced reconstruction across the full frequency range while alleviating the trade-off between noise suppression and over-attenuation.
Abstract
from arXiv · showhide
This paper proposes two spectrally weighted STFT loss functions for lightweight streaming speech enhancement, addressing the magnitude over-attenuation in mid-to-high frequency regions caused by the magnitude-phase compensation effect. The proposed sigmoid-weighted loss applies a smooth frequency-dependent modulation to the phase-aware contribution, while the signal-dependent spectrally adaptive loss further conditions the modulation on the ground-truth log-magnitude spectrogram. To evaluate the proposed objectives, we additionally design HyST-Net, a lightweight and competitive backbone with hybrid MHA-GRU spectral-temporal modelling for low-latency streaming scenarios. Experimental results exhibit consistent improvements in high-frequency spectral reconstruction for both losses. The spectrally adaptive loss further enhances the mid-frequency region, resulting in a more balanced spectral reconstruction across the full frequency range.
1. INTRODUCTION
Lightweight streaming speech enhancement faces a trade-off between noise suppression and spectral fidelity because phase-aware losses can over-attenuate mid-to-high frequencies. The paper proposes spectrally weighted objectives and evaluates them with a lightweight streaming backbone.
- Motivation: Lightweight streaming speech enhancement models face stringent latency and computational constraints that limit model capacity.These constraints make the training objective especially important because compact models have limited capacity to compensate for loss-induced biases.
- Motivation: Phase-aware STFT losses can suppress magnitudes where phase predictions are unreliable, causing pronounced mid-to-high-frequency energy attenuation.This magnitude-phase compensation effect produces inaccurate magnitude estimates and degrades spectral fidelity.
- Motivation: Uniformly adding a magnitude-domain term trades noise suppression against perceptual fidelity because attenuation is spectrally concentrated rather than uniform.The paper characterizes this weighting as an inherent compromise rather than a principled solution.
- Contributions: The paper proposes sigmoid-weighted and signal-dependent spectrally adaptive STFT losses that modulate phase-aware contributions by frequency and signal characteristics.It also introduces HyST-Net as a lightweight, low-latency backbone for streaming speech enhancement evaluation.
- Motivation: A larger phase-aware loss weight improves noise suppression but causes greater mid-to-high-frequency over-attenuation.Figure 1 compares λ values of 0, 0.3, and 1.
2. METHODS
The paper develops frequency- and signal-conditioned STFT losses alongside HyST-Net, a causal hybrid spectral-temporal network for low-latency streaming enhancement. The losses alter phase-aware weighting, while the network estimates a compressed-domain complex mask from noisy spectrograms.
- Spectrally Weighted STFT Loss Functions: The baseline combines compressed magnitude and phase-aware losses, with λ controlling the trade-off between noise suppression and spectral reconstruction.The paper reports λ = 0.3 as yielding the best instrumental metric performance, while excessive λ causes mid-to-high-frequency over-attenuation.
- Spectrally Weighted STFT Loss Functions: LSig applies a smooth frequency-dependent sigmoid weight to reduce phase-aware contribution where mid-to-high-frequency over-attenuation is more severe.Its empirical settings are r = 0.4, λsig = 0.5, and β = -20; the transition avoids spectral banding artefacts.
- Spectrally Weighted STFT Loss Functions: LAdp derives a signal-dependent spectral weight from the ground-truth log-magnitude spectrogram to modulate the phase-aware loss contribution.The procedure includes time averaging, min-max normalisation, and one-dimensional spectral smoothing.
- Streaming HyST-Net: HyST-Net takes channel-concatenated real and imaginary noisy spectrogram components as input and estimates a compressed-domain complex ideal ratio mask.The enhanced complex spectrogram is recovered by applying the estimated mask and then decompressing magnitude.
- Streaming HyST-Net: HyST-Net uses a causal convolutional U-Net with streaming buffer caches and interleaved MHA spectral modelling plus GRU temporal modelling.The hybrid bottleneck assigns different modules to the spectral and temporal dimensions rather than reusing one sequential module for both.
3. EXPERIMENTS
Experiments evaluate HyST-Net as a causal streaming backbone and compare the proposed spectrally weighted losses with LMR Mix. Both losses improve high-frequency reconstruction, while LMR Adp provides the strongest mid-frequency reconstruction and more balanced spectral recovery.
- 140 hours of synthesised wideband speech were used for training, with SNRs from −5 dB to 20 dB, and evaluation used the DNS Challenge public synthetic test set.
- HyST-Net matches FTF-Net across enhancement metrics while reducing MACs by 16.3% and parameters by 21%.It attains RTF 0.22 versus FTF-Net's RTF 1.05 on a single CPU thread, and uses 96% fewer parameters than CRUSE at similar RTF.
- Both proposed losses achieve overall metrics on par with LMR Mix under the same HyST-Net configuration.The losses retain comparably high phase-aware weighting at low frequencies, where the instrumental metrics are dominated.
- Approximately 9.5% lower C-RMSE and 15.2% lower M-RMSE, with gains of 1.18 dB in LSD and 0.43 dB in SI-SDR, occur in the HF region for both proposed losses.
- LMR Adp consistently outperforms LMR Sig and LMR Mix across all MF metrics.Its signal-dependent weighting adapts the phase-aware contribution to frequency-wise spectral energy, unlike LMR Sig's fixed sigmoid modulation.
- Fig. 4 shows that LMR Adp reconstructs more mid-to-high-frequency energy while preserving low-frequency harmonic structures and effective noise suppression.The example spectrogram comparison reports reduced over-attenuation without noticeable artefacts.
4. CONCLUSIONS
The paper proposes spectrally weighted STFT losses and evaluates them with HyST-Net for lightweight streaming speech enhancement. Both losses improve high-frequency reconstruction, while LAdp additionally improves mid-frequency reconstruction and achieves more balanced spectral recovery.
- The proposed LSig and LAdp losses modulate the phase-aware contribution according to frequency and signal characteristics.
- HyST-Net serves as a competitive lightweight backbone for evaluating the proposed objectives in streaming speech enhancement.
- Both losses consistently improve high-frequency reconstruction metrics, while LAdp further enhances mid-frequency reconstruction through signal-dependent weighting.
- The spectrally adaptive loss achieves more balanced spectral reconstruction across the full frequency range while alleviating the trade-off between noise suppression and over-attenuation.