Source-linked AI summary
End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation
Yi Luo, Zhuo Chen, Nima Mesgarani, Takuya Yoshioka
TL;DR
The paper addresses the challenge of making end-to-end multi-channel speech separation robust to microphone permutations and varying microphone counts. It introduces TAC within FaSNet to transform, pool, and recombine channel features for invariant global processing, and reports improved separation in both ad-hoc and fixed-geometry arrays. The supported scope includes arrays with arbitrary geometry and a variable channel count between 2 and a predefined maximum.
Problem
End-to-end neural beamformers may assume a fixed microphone count and channel indexing, limiting permutation and number invariance in multi-channel speech separation.
Method
TAC applies shared per-channel transformations, averages them for global pooling, and concatenates the pooled representation with channel features before producing channel-dependent outputs in FaSNet.
Results
TAC improves FaSNet separation performance across microphone counts in ad-hoc arrays and across conditions in fixed-geometry arrays.
Takeaways & Limitations
TAC provides end-to-end microphone permutation- and number-invariant processing while using information from all microphones for global decisions.
Takeaways & Limitations
The evaluation considers arbitrary-geometry arrays with a channel count between 2 and a predefined maximum, and FaSNet may remain unstable in ad-hoc settings when microphone acoustics differ substantially.
Abstract
from arXiv · showhide
An important problem in ad-hoc microphone speech separation is how to guarantee the robustness of a system with respect to the locations and numbers of microphones. The former requires the system to be invariant to different indexing of the microphones with the same locations, while the latter requires the system to be able to process inputs with varying dimensions. Conventional optimization-based beamforming techniques satisfy these requirements by definition, while for deep learning-based end-to-end systems those constraints are not fully addressed. In this paper, we propose transform-average-concatenate (TAC), a simple design paradigm for channel permutation and number invariant multi-channel speech separation. Based on the filter-and-sum network (FaSNet), a recently proposed end-to-end time-domain beamforming system, we show how TAC significantly improves the separation performance across various numbers of microphones in noisy reverberant separation tasks with ad-hoc arrays. Moreover, we show that TAC also significantly improves the separation performance with fixed geometry array configuration, further proving the effectiveness of the proposed paradigm in the general problem of multi-microphone speech separation.
1. INTRODUCTION
End-to-end neural beamformers struggle to remain invariant to microphone permutations and varying microphone counts. The paper proposes TAC, which combines shared channel transforms, averaging, and concatenation to make global, permutation- and number-invariant decisions within FaSNet.
- Optimization-based beamforming is invariant to microphone permutation and count, but end-to-end neural beamformers often lack these guarantees.
- Fixed-size outputs can assume a known microphone count and encode channel indexing, so permuting channel indexes may produce inconsistent outputs.
- FaSNet avoids explicitly specifying microphone number or permutation, but its pre-separation dependence and pair-wise features limit reliable global filter estimation, especially for ad-hoc arrays.
- TAC transforms every channel with shared parameters, averages the transformed features, concatenates the global result with channel features, and generates channel-dependent outputs.
- Parameter sharing and permutation-invariant averaging give TAC microphone permutation and number invariance while allowing global decisions across channels.
2. TRANSFORM-AVERAGE-CONCATENATE PROCESSING
This section describes TAC processing and its integration with FaSNet for microphone-invariant multi-channel beamforming. TAC combines shared per-channel transforms, global averaging, and channel-specific outputs, while FaSNet estimates time-domain filters from framed multi-channel signals.
- Transform-average-concatenate processing: TAC transforms each channel with shared parameters, averages channel features globally, concatenates the pooled result with each channel feature, and produces channel-dependent outputs.The transform and channel-output sub-modules are shared across channels, while averaging provides a permutation-invariant global representation.
- Transform-average-concatenate processing: Averaging across channels makes TAC invariant to microphone permutation and supports processing arrays with variable numbers of microphones.The design treats channel features as a set and uses parameter sharing plus pooling to preserve invariant processing.
- FaSNet recap: FaSNet frames each microphone signal, adds future and past context, and estimates time-domain beamforming filters for target sources.Each context frame contains L + 2W samples, and the filters have length 2W + 1.
- FaSNet recap: In its two-stage design, FaSNet estimates reference-microphone filters first, then estimates filters for remaining channels using NCC features and channel embeddings.The reference embedding is combined with pooled cross-channel features for initial filter estimation; remaining-channel filters use pair-wise features with the pre-separation output.
- TAC-FaSNet integration: TAC is integrated into FaSNet by replacing pair-wise second-stage estimation with global filter estimation and by adding TAC to intermediate neural-network blocks.A single-stage variant jointly estimates filters for all channels so reference-channel pre-separation can also benefit from TAC.
3. EXPERIMENTAL PROCEDURES
The experiments evaluate multi-channel two-speaker noisy speech separation using ad-hoc and fixed-geometry microphone arrays, comparing several TasNet-based and FaSNet configurations.
- Dataset: The dataset contains 20,000 training, 5,000 validation, and 3,000 test four-second utterances constructed from Librispeech speech and nonspeech noise.Two speakers and one nonspeech noise source are randomly selected, with speaker overlap uniformly sampled from 0% to 100%.
- Compared models: Models are evaluated in both ad-hoc and fixed-geometry array configurations using multiple TasNet-filter and FaSNet variants.The comparisons include pure single-channel processing, mean-pooled NCC features, index-concatenated NCC features, and four FaSNet variants.
- Training: All models use DPRNN blocks and are trained with negative SI-SNR and utterance-level permutation-invariant training.DPRNN blocks are used because prior results showed they outperform TCN with a smaller model size.
- Window settings: The context window W is fixed at 16 ms, while the center window L is normally set equal to W and also evaluated at smaller values.Varying L while keeping W fixed preserves the dimensions of the beamforming filters and NCC features.
4. RESULTS AND DISCUSSIONS
TAC improves FaSNet separation across microphone counts in ad-hoc arrays and also improves performance in fixed-geometry arrays. The results additionally support global processing during filter estimation as an explanation for performance with smaller center windows.
- Ad-hoc arrays: Adding TAC to single-stage FaSNet improves performance across all ad-hoc-array conditions and microphone numbers.The improvement is particularly significant when the speaker overlap ratio is high.
- Ad-hoc arrays: TAC ensures that adding more microphones does not worsen single-stage FaSNet performance in the ad-hoc configuration.This contrasts with TasNet-based models, for which increasing microphone count does not necessarily improve performance.
- Ad-hoc arrays: Single-stage FaSNet without TAC already outperforms TasNet-based and two-stage FaSNet models in the ad-hoc configuration.The results indicate that the pre-separation stage is unnecessary for this configuration.
- Fixed geometry arrays: Single-stage FaSNet with TAC greatly outperforms original FaSNet across all fixed-geometry-array conditions.The authors suggest TAC may learn geometry-dependent information without explicit geometry-related features.
- Window-size analysis: TAC yields better or on-par performance with a 4 ms center window, supporting the authors’ claim that global filter-estimation processing matters.This observation challenges the explanation that smaller-window degradation is primarily caused by insufficient frequency resolution.
5. CONCLUSION
The paper concludes that TAC provides permutation- and number-invariant multichannel speech separation while using all channel information for global decisions. Integrated into FaSNet, it improves separation in both ad-hoc and fixed-geometry configurations.
- Conclusion: TAC is a set-function method whose shared transformations and averaging provide microphone permutation- and number-invariant processing.The design averages transformed channel features, concatenates the pooled output with channel representations, and generates channel-dependent outputs.
- Conclusion: TAC uses the full information within the microphone set to make global decisions during multichannel processing.The conclusion presents this property as a central advantage of the proposed design.
- Conclusion: When inserted into FaSNet, TAC greatly improves separation performance in both ad-hoc and fixed-geometry array configurations.The authors suggest the paradigm may inform model designs for other multichannel processing problems.