MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition

1. INTRODUCTION

• Monaural end-to-end multi-speaker recognition still has a large performance gap from single-speaker ASR, limiting its readiness for widespread real-world use.The introduction notes that significant improvements have not closed this gap.Source: §1 · p. 1
• Multi-channel microphone arrays provide spatial information that can help locate speakers and separate their speech, but prior multi-channel multi-speaker work mainly focused on separation.Earlier approaches include multi-channel PIT beamforming and DPCL using interchannel and spectral features.Source: §1 · p. 1
• MIMO-Speech maps simultaneous multi-channel speech directly to a separate text sequence for each speaker.Its redesigned neural-beamformer front end attends to multiple beams at different directions.Source: §1 · pp. 1–2
• The system extracts log filterbank features from separated signals and recognizes each stream with a joint CTC/attention encoder-decoder; beamformer outputs can also serve as separation signals.The authors report that the intermediate signals have good audibility even without explicitly optimizing for separation.Source: §1 · p. 2
• A curriculum-learning data schedule provides an additional performance boost, and the authors evaluate the approach in anechoic and preliminary reverberant settings.The schedule uses curriculum learning during training.Source: §1 · p. 2

2.1. Model Architecture

• Arbitrary-channel input maps directly to one text sequence per speaker through integrated masking and beamforming, with the whole model optimized solely by a speech-recognition objective.Source: §2.1 · p. 2

2.1.1. Monaural Masking Network

• The monaural network independently processes each channel’s complex STFT to estimate time-frequency masks for every speaker and an extra noise source.Source: §2.1.1 · p. 2

2.1.2. Multi-source Neural Beamformer

• Per-channel speaker and noise masks form source PSD matrices; MVDR filters treat other speakers alongside noise as interference and produce enhanced single-channel speaker signals.Source: §2.1.2 · pp. 2–3

2.1.3. End-to-End Speech Recognition

• Beamformed STFT magnitudes become normalized log-filterbank features before each speaker stream enters the end-to-end recognition module.Source: §2.1.3 · p. 3
• Because speaker streams can match references in multiple orders, PIT selects an assignment using CTC loss, and the final ASR loss interpolates CTC with attention cross-entropy.Source: §2.1.3 · p. 3

2.2. Data Scheduling and Curriculum Learning

• Training mixes multi-speaker batches with single-speaker WSJ batches that bypass separation; single-speaker batches optimize recognition only and regularize the ASR model.Source: §2.2 · p. 3
• Curriculum orders mixtures by ascending SNR between the loudest and quietest speaker, and single-speaker utterances from short to long.Source: §2.2 · p. 3

3. EXPERIMENT

• 98.5 h of two-utterance mixtures formed the multi-speaker training set, with source SNR sampled uniformly from [−5, 5] dB.Source: §3 · p. 3
• The validation and evaluation sets were generated from WSJ Dev93 and Eval92, with durations of 1.3 h and 0.8 h, respectively.Source: §3 · p. 3
• The curriculum sorts clean utterances by increasing length and mixtures by ascending SNR, then alternates minibatches from the clean and noisy sets.Source: §3 · p. 4

• Training continues while the model has not converged.Source: §11 · p. 4
• The procedure randomly shuffles clean and noisy training data and divides them into minibatch sets.Source: §11 · p. 4
• A minibatch is selected randomly, and its noisy data is fed into the model to update it iteratively.Source: §11 · p. 4

3.1. Configurations

• Experiments use two-speaker spatialized wsj1-2mix with WSJ1 si284 training data for regularization; results report two channels, though the model supports arbitrary channel counts.The evaluation includes anechoic and reverberant scenarios.Source: §15 · p. 4

3.1.1. Neural Beamformer

• The mask estimation network is a three-layer BLSTMP; a small epsilon is added to the PSD matrix in the MVDR formula to guarantee an inverse exists.The reference microphone vector dimension is 512.Source: §3.1.1 · p. 4

3.1.2. Encoder-Decoder Network

• The recognition encoder combines two VGG-motivated CNN blocks with three BLSTMP layers, while the decoder is a single-layer unidirectional LSTM.The model extracts 80-dimensional log filterbank features and uses a loss interpolation factor of 0.2.Source: §3.1.2 · p. 4

3.2. Performance of Multi-Speaker Speech Recognition

• 3.75% CER and 7.55% WER are the final evaluation results; MIMO-Speech improves over the best baseline by over 60% on both metrics, while curriculum learning adds over 12% versus MIMO-Speech without it.The final results are reported on the spatialized anechoic wsj1-2mix evaluation set.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3. Performance of Multi-Speaker Speech Separation

• Mask examples distinguish speakers, and separated-stream spectrograms show less overlap than the input, suggesting MIMO-Speech separates speech to some extent.The masks are not constrained to sum to one at each time-frequency unit.Source: §3.3 · p. 5, Figure 2 · p. 5
• At 500, 1000, 2000, and 4000 Hz, the two speakers’ beam patterns have different crests and troughs, indicating source-distinguishing beamforming behavior.The beam values at different degrees quantify the reduction of received speech signals.Source: §3.3 · p. 5, Figure 3 · p. 6
• 3.6 average PESQ and 23.1 dB average SI-SDR quantify separated-speech quality; curriculum learning slightly degrades both metrics, but quality remains high.The evaluation reconstructs separated waveforms from beamformer outputs using inverse STFT and compares them with reference signals.Source: §3.3 · p. 5, Table 2 · p. 6

3.4. Evaluation on the spatialized reverberant data

• Multi-conditioned training improves WER by over 60% on reverberant data, while inference-stage Nara-WPE dereverberation leads to further improvement; the study is preliminary.The model does not incorporate dereverberation techniques in its design.Source: §3.4 · p. 6, Table 4 · p. 6, Table 5 · p. 6

• MIMO-Speech takes multi-speaker microphone-array recordings as input and outputs text sequences for each speaker.Source: §4 · p. 6
• Its neural beamformer learns speech separation without an explicit signal reconstruction criterion, while the differentiable model is optimized with an ASR loss.Source: §4 · p. 6
• Single-channel, single-speaker speech and a curriculum learning strategy are used to make training easier and improve performance.Source: §4 · p. 6
• Experiments on spatialized wsj1-2mix show fairly good performance, but reverberant data still has a large gap against the anechoic case.The authors identify incorporating dereverberation and further integrating masking and beamforming as future work.Source: §4 · p. 6