Source-linked AI summary
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
Zhong-Qiu Wang, Jonathan Le Roux, DeLiang Wang, John R. Hershey
TL;DR
Magnitude-based separation systems reuse mixture phase, creating phase inconsistency, while post-hoc phase reconstruction has generally produced only small gains. This paper trains T-F masking through iterative phase reconstruction and reports 12.6 dB SI-SDR on wsj0-2mix, close to the oracle IRM result.
Problem
Magnitude-domain separation commonly reuses mixture phase, yielding an inconsistent STFT that is not guaranteed to correspond to a time-domain signal.
Source: §1 · p. 1Method
The approach optimizes waveform-level separation while training the masking network through iterative phase reconstruction.
Source: §3.2 · p. 2, §5 · p. 4Results
12.6 dB SI-SDR on wsj0-2mix, surpassing the previous best by a large margin and comparable to the 12.7 dB oracle IRM result.
Source: §1 · p. 1Takeaways & Limitations
On a large open dataset, deep-learning-based phase reconstruction combined with state-of-the-art magnitude-domain separation yields tangible, large improvements.
Source: §1 · p. 1The waveform-approximation initialization still uses mixture phase, before any phase reconstruction is performed.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
This paper proposes an end-to-end approach for single-channel speaker-independent multi-speaker speech separation, where time-frequency (T-F) masking, the short-time Fourier transform (STFT), and its inverse are represented as layers within a deep network. Previous approaches, rather than computing a loss on the reconstructed signal, used a surrogate loss based on the target STFT magnitudes. This ignores reconstruction error introduced by phase inconsistency. In our approach, the loss function is directly defined on the reconstructed signals, which are optimized for best separation. In addition, we train through unfolded iterations of a phase reconstruction algorithm, represented as a series of STFT and inverse STFT layers. While mask values are typically limited to lie between zero and one for approaches using the mixture phase for reconstruction, this limitation is less relevant if the estimated magnitudes are to be used together with phase reconstruction. We thus propose several novel activation functions for the output layer of the T-F masking, to allow mask values beyond one. On the publicly-available wsj0-2mix dataset, our approach achieves state-of-the-art 12.6 dB scale-invariant signal-to-distortion ratio (SI-SDR) and 13.1 dB SDR, revealing new possibilities for deep learning based phase reconstruction and representing a fundamental progress towards solving the notoriously-hard cocktail party problem.
1. Introduction
Recent separation advances still rely on magnitude-domain masking and mixture-phase resynthesis, which can produce an inconsistent STFT. The paper trains separation through iterative phase reconstruction and reports 12.6 dB SI-SDR on wsj0-2mix.
1. Introduction
- Magnitude-domain separation systems reuse mixture phase for resynthesis, risking an inconsistent STFT that may not correspond to any time-domain signal.Speech-frame overlap makes the STFT highly redundant, contributing to phase inconsistency.Source: §1 · p. 1
- Post-hoc iterative phase reconstruction usually yields only small gains, which the authors suggest may reflect masking that ignores the later reconstruction steps.They propose training the masking network with iterative phase reconstruction in the loop.Source: §1 · p. 1
- 12.6 dB SI-SDR on wsj0-2mix, surpassing the previous best by a large margin and approaching the 12.7 dB oracle IRM result.The authors report this as the first large-open-dataset demonstration of tangible, large improvements from deep-learning-based phase reconstruction combined with state-of-the-art magnitude-domain separation.Source: §1 · p. 1
2. Chimera++ Network
The Chimera++ network combines deep clustering and mask inference through multi-task learning, pairing speaker-structured embeddings with a permutation-free masking objective. Its embedding network uses bidirectional context, while mask output conventions allow values beyond one.
2. Chimera++ Network
- A recent multi-task approach combines deep-clustering regularization with end-to-end mask inference, and obtaining a good magnitude estimate is a prerequisite for phase reconstruction.The cited work reports improvements over the individual models.Source: §2 · p. 1
- Deep clustering learns embeddings that bring T-F units dominated by the same speaker closer together and units dominated by different speakers farther apart.An alternative loss that whitens the embedding in a k-means objective is reported to improve separation.Source: §2 · pp. 1–2
- A bidirectional LSTM models context from past and future frames when learning the embeddings.The deep-clustering embedding layer is fully connected, followed by a nonlinearity and unit-length normalization for each frequency.Source: §2 · p. 2
- The mask-inference network minimizes loss over source permutations and uses the phase-sensitive mask as its target, with unbounded values truncated for the tPSA objective.The paper notes sigmoid output with truncation factor γ = 1 as common; larger factors are also discussed.Source: §2 · p. 2
- Chimera++ combines deep clustering and mask inference with a weighted sum of their losses; only the mask-inference output is needed at run time.The combination is presented as a multi-task learning approach.Source: §2 · p. 2
3. Proposed Algorithms
The proposed algorithms train speech separation on reconstructed waveforms and unfold MISI phase reconstruction within the network. They also use output activations that permit masks above one, accommodating magnitudes that may exceed the mixture magnitude.
3.1. Iterative Phase Reconstruction
- MISI iteratively reconstructs each source phase from its estimated magnitude and the mixture phase while constraining reconstructed sources to sum to the mixture.The magnitudes remain fixed during iterations, while the phases are updated.Source: §3.1 · p. 2
3.2. Waveform Approximation
- Waveform approximation (WA) trains on permutation-minimized L1 error between reference sources and waveforms reconstructed by iSTFT.The initial reconstruction combines estimated magnitudes with the mixture phase; phase reconstruction has not yet occurred.Source: §3.2 · p. 2
3.3. Unfolded Iterative Phase Reconstruction
- Unfolding K MISI iterations as STFT and iSTFT layers lets end-to-end training adapt estimated magnitudes to later phase reconstruction and produce better phase estimates.The resulting model is denoted WA-MISI-K, with K ≥ 1.Source: §3.3 · pp. 2–3
3.4. Activation Functions with Values Beyond One
- Because source magnitudes can exceed the mixture magnitude, the authors consider output activations that allow mask values above one, including doubled sigmoid, clipped ReLU, and convex softmax.The convex softmax models modes at 0, 1, and 2; the study truncates PSM values to [0, 2].Source: §3.4 · p. 3
4. Experimental Setup
Experiments use the wsj0-2mix corpus and evaluate separation with SI-SDR and SDR. The setup uses a staged training curriculum that adds unfolded MISI iterations.
4. Experimental Setup
- wsj0-2mix contains 20,000 training, 5,000 validation, and 3,000 test two-speaker mixtures, with unseen speakers in the open-speaker test set.The corpus is sampled at 8 kHz.Source: §4 · p. 3
- The network uses four bidirectional LSTM layers with 600 units per direction and trains on 400-frame segments using Adam.Dropout of 0.3 is applied after each BLSTM layer except the last.Source: §4 · p. 3
- Training progresses from chimera++ through LWA and then LWA-MISI models, with the MISI iteration count increasing to five after performance saturated.The authors report that this curriculum learning strategy was helpful.Source: §4 · p. 4
- Evaluation reports SI-SDR and SDR, and the setup includes Table 1's SI-SDR results and a figure tracking SI-SDR against test-time MISI iterations.The STFT uses a 32 ms window, an 8 ms hop, and a 256-point DFT.Source: §4 · p. 4, Table 1 · p. 4, Figure 2 · p. 4
5. Evaluation Results
On wsj0-2mix, the proposed approach improves both SI-SDR and SDR over the previous state of the art. Results also show how end-to-end training through phase reconstruction and mask activation choices affect SI-SDR.
5. Evaluation Results
- 11.5 dB SI-SDR follows five MISI iterations on estimated magnitudes with mixture phase, versus 11.3 dB with five Griffin-Lim iterations.The chimera++ network obtains 11.2 dB, versus 10.4 dB and 10.0 dB for the individual models.Source: §5 · p. 4
- 12.2 dB SI-SDR is reached by LWA-MISI-5, compared with 11.6 dB for end-to-end LWA without phase-reconstruction post-processing.Applying five MISI iterations after LWA does not improve its result.Source: §5 · p. 4
- 12.6 dB SI-SDR is reached by convex softmax, the best of the three proposed activation functions.Activations allowing values beyond one become beneficial only when training through phase reconstruction.Source: §5 · p. 4
- Models trained for more MISI iterations start with lower SI-SDR but reach higher values as test-time MISI iterations increase.Figure 2 tracks SI-SDR against zero to five test-time iterations for convex-softmax models trained with different objectives.Source: §5 · p. 4
- 1.1 dB better SI-SDR and SDR than the previous state-of-the-art approach is reported on wsj0-2mix.Table 2 compares systems on the same corpus.Source: §5 · p. 4, Table 2 · p. 4
6. Concluding Remarks
The paper concludes that training the T-F masking network through iterative phase reconstruction improves single-channel speech separation. It identifies speech enhancement, improved initial-phase estimation, and alternatives to waveform-level loss as future directions.
6. Concluding Remarks
- Training the T-F masking network through iterative phase reconstruction yields significant improvements in single-channel speech separation.The authors propose an end-to-end approach.Source: §6 · p. 4
- Future work will apply the methods to speech enhancement and jointly estimate magnitude with an initial phase that improves on the mixture phase..Source: §6 · p. 4
- The authors also plan to improve ideal amplitude-mask estimation and consider losses computed on reconstructed-signal magnitude spectrograms instead of waveform-level loss..Source: §6 · p. 4