Source-linked AI summary
Differentiable Consistency Constraints for Improved Deep Speech Enhancement
Scott Wisdom, John R. Hershey, Kevin Wilson, Jeremy Thorpe, Michael Chinen, Brian Patton, Rif A. Saurous
TL;DR
Masking approaches can neglect STFT consistency and mixture consistency, including for complex-valued masks. This paper adds differentiable projection layers to enforce both constraints, and reports a 0.7 dB increase in mean SI-SDR improvement over its baseline.
Problem
Masking approaches may produce STFTs that do not represent time-domain signals and may fail to ensure separated sources sum to the mixture; mixture consistency has not been leveraged for complex-valued masks.
Source: §1 · p. 1, §3.2 · p. 2Method
The paper adds differentiable layers that impose STFT and mixture consistency in masking-based enhancement, including with complex-valued masks.
Source: §1 · p. 1, §3.2 · p. 2Results
0.7 dB: the best system’s mean SI-SDR improvement is 10.6 dB, compared with 9.9 dB for the baseline.
Source: §4.3 · p. 4Takeaways & Limitations
The constraints are compatible with STFT-based enhancement systems, including those using complex-valued masks.
Source: §5 · p. 4Weighted mixture projections with different weights per time-frequency bin are not orthogonal to STFT consistency, and jointly imposing both is more complicated.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
In recent years, deep networks have led to dramatic improvements in speech enhancement by framing it as a data-driven pattern recognition problem. In many modern enhancement systems, large amounts of data are used to train a deep network to estimate masks for complex-valued short-time Fourier transforms (STFTs) to suppress noise and preserve speech. However, current masking approaches often neglect two important constraints: STFT consistency and mixture consistency. Without STFT consistency, the system's output is not necessarily the STFT of a time-domain signal, and without mixture consistency, the sum of the estimated sources does not necessarily equal the input mixture. Furthermore, the only previous approaches that apply mixture consistency use real-valued masks; mixture consistency has been ignored for complex-valued masks. In this paper, we show that STFT consistency and mixture consistency can be jointly imposed by adding simple differentiable projection layers to the enhancement network. These layers are compatible with real or complex-valued masks. Using both of these constraints with complex-valued masks provides a 0.7 dB increase in scale-invariant signal-to-distortion ratio (SI-SDR) on a large dataset of speech corrupted by a wide variety of nonstationary noise across a range of input SNRs.
1. INTRODUCTION
Masking-based enhancement can produce STFTs that do not correspond to any time-domain signal, while failing to ensure that separated sources sum to the input mixture. The paper proposes adding differentiable consistency layers to address both constraints, including with complex-valued masks.
1. INTRODUCTION
- Masking an STFT does not necessarily yield a consistent STFT, so its values may not correspond to any real-valued time-domain signal.The introduction also notes that inverting and recomputing such an STFT can change its magnitudes and phases.Source: §1 · p. 1
- Existing approaches using complex-valued masks do not leverage the assumption that estimated sources should sum to the original mixture.Source: §1 · p. 1
- Differentiable end-to-end layers can enforce STFT and mixture consistency in masking-based enhancement models, including models with complex-valued masks.The paper presents these constraints as compatible with any masking-based speech enhancement model.Source: §1 · p. 1
2. RELATION TO PRIOR WORK
Prior work addressed STFT or mixture consistency through model-based or iterative methods, while this paper combines consistency constraints with DNNs and complex-valued masking.
2. RELATION TO PRIOR WORK
- Prior STFT-consistency methods generally used model-based iterative algorithms, whereas this work combines STFT consistency with DNNs without iterations.Source: §2 · p. 1
- Earlier mixture-consistency methods included MISI, unfolded MISI iterations, and PPR; unlike these, this work avoids multiple phase-estimation iterations and tests complex-valued masks.Source: §2 · p. 1
- Lee et al. used a soft squared-error penalty to promote mixture consistency, while this work uses an exact projection layer.Source: §2 · p. 1
3. CONSISTENCY
The section defines STFT and mixture consistency and describes projections for imposing them, including a weighted mixture projection. It also illustrates the effect of STFT consistency and discusses when the two projection operations commute.
3.1. STFT consistency
- A consistent STFT is one representable as the STFT of a real-valued time-domain signal; measuring loss on an inconsistent masked STFT may misstate the reconstructed signal’s spectral magnitude.Source: §3.1 · pp. 1–2
3.1.1. Illustration of STFT consistency
- 9.57e-3 versus 1.58e-2: in the illustration, the consistent STFT has lower magnitude-squared error to the true STFT than the masked STFT.The figure contrasts the masked spectrogram with the projected consistent spectrogram.Source: §3.1.1 · p. 2, Figure 1 · p. 2
3.1.2. Backpropagating through STFTs
- Applying the inverse and then forward STFT to an estimate projects it toward consistency, and the linear STFT operations can be implemented as a network layer.Source: §3.1.2 · p. 2
3.2. Mixture consistency
- Mixture consistency requires estimated source STFTs to sum to the mixture; the projection layer supports real- or complex-valued masks without constraining masks to sum to one.Constraining complex masks to sum to one can be too restrictive when sources need different phase modifications in a time-frequency bin.Source: §3.2 · p. 2
3.2.1. Backpropagating through mixture-consistent projection
- The mixture projection moves masked source estimates to the nearest mixture-consistent points for each time-frequency bin, yielding an update that can be added as a network layer and backpropagated through.Source: §3.2.1 · pp. 2–3
3.2.2. Weighted mixture-consistent projection with uncertainty
- The weighted projection uses source-estimate variances as uncertainty weights; when all sources have equal uncertainty, it reduces to the unweighted projection.The illustrated weighted projection selects the intersection of the mixture constraint and Gaussian contours; equal variances give circular contours and an orthogonal projection.Source: §3.2.2 · p. 3
3.3. Order of consistency operations
- The unweighted STFT- and mixture-consistency projections commute, but per-bin weighted mixture projections are not orthogonal to STFT consistency, making joint enforcement more complicated.The paper defers jointly imposing STFT consistency and weighted mixture consistency to future work.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments evaluate consistency-constrained speech enhancement on mixed speech and nonstationary noise, using real- and complex-valued masks. The best overall mean SI-SDR improvement comes from complex masking with both consistency constraints and learned mixture weights.
4.1. Dataset
- The 134.2-hour training set mixes LibriSpeech speech with varied freesound.org noise; validation and test sets each span 6.2 hours, with nonoverlapping speaker sets.Mixing SNRs are normally distributed with mean 5 dB and standard deviation 10 dB; audio is sampled at 16 kHz.Source: §4.1 · p. 3
4.2. Model architecture and training
- The model uses a convolutional spectral front end, a 400-wide unidirectional LSTM with residual connections, and two 600-unit fully connected layers.It trains on fixed three-second clips with Adam, batch size 8, and learning rate 3 × 10^-5; the loss weights speech at 0.8 and noise at 0.2.Source: §4.2 · p. 3
4.3. Results
- 10.6 dB mean SI-SDR improvement for complex masking with STFT consistency and learned weighted mixture consistency, versus 9.9 dB for the baseline, a 0.7 dB gain.Models using both constraints almost always outperform those without them; the best models tend to use learned weighted mixture consistency.Source: Figure 4 · p. 4, §4.3 · p. 4
4.2. Model architecture and training
- Real-valued masks use noisy phase, while complex-valued masks predict real and imaginary components; weighted mixture consistency uses estimated magnitudes or learned weights.Estimated squared source magnitudes avoid corrections in low-magnitude bins; learned weights are complementary sigmoid outputs for speech and noise.Source: §4.2 · p. 4
5. CONCLUSION
The conclusion presents differentiable layers as a way to impose STFT and mixture consistency on source estimates, with improved speech-enhancement performance on varied nonstationary noise and SNRs.
5. CONCLUSION
- Differentiable neural-network layers enforce STFT and mixture consistency on source estimates.The constraints are compatible with STFT-based enhancement systems, including those using complex-valued masks.Source: §5 · p. 4
- Adding the constraints improves speech-enhancement performance on a dataset spanning varied nonstationary noise and SNR levels.Source: §5 · p. 4
- Future work proposed includes combining the constraints with other phase-estimation and generative models, extending them to general audio separation, and exploring weighted STFT consistency.Source: §5 · p. 4