Source-linked AI summary
VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking
Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, Ignacio Lopez Moreno
TL;DR
Speech recognition remains vulnerable to interfering speakers in crowded recordings, motivating speaker-dependent voice filtering. VoiceFilter uses a reference recording to condition spectrogram masking on a target-speaker embedding, reducing recognition WER in multi-speaker scenarios while keeping single-speaker WER approximately unchanged.
Problem
Speech recognition systems still perform poorly when the speaker of interest is recorded with foreground or background interfering speakers.
Method
VoiceFilter separately trains a speaker encoder and a time-frequency masking system conditioned on a target-speaker embedding and noisy multi-speaker audio.
Results
VoiceFilter reduces speech recognition WER from 55.9% to 23.4% in two-speaker scenarios while WER stays approximately the same on single-speaker scenarios.
Takeaways & Limitations
The system is more applicable to real scenarios because it does not require prior knowledge of the number of speakers and avoids the permutation problem.
Takeaways & Limitations
Future work includes training on larger and more challenging datasets, adding more interfering speakers, and improving speaker embeddings from multiple utterances.
Abstract
from arXiv · showhide
In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.
1. Introduction
Voice filtering isolates selected speakers from multi-speaker audio using a reference recording, avoiding several challenges of speaker-independent separation. VoiceFilter conditions a masking system on a speaker embedding to suppress interference and target the selected voice.
- Crowded environments with foreground or background speakers remain difficult for speech recognition systems.
- Classical separation must handle an unknown number of speakers and permutation-invariant output assignment.It may also require selecting which separated output corresponds to the target speaker.
- Voice filtering isolates a subset of speakers of interest from other speakers and noises using prior reference recordings.The paper calls speaker-dependent speech separation “voice filtering.”
- VoiceFilter frames the task as binary classification, with target speech positive and interfering speech and noises negative.A reference speaker embedding conditions the masking system that removes interference and outputs the target voice.
- The process can be repeated for each reference recording to filter multiple speakers of interest.
2. Approach
VoiceFilter uses two separately trained components: a speaker encoder that produces target-speaker embeddings and a spectrogram-masking network that uses those embeddings with noisy audio. The network predicts a mask to produce an enhanced spectrogram and waveform.
- The architecture separates a speaker encoder from the VoiceFilter network, which receives the encoder output as an additional input.
- Speaker encoder: The speaker encoder is a 3-layer LSTM that produces fixed 256-dimensional d-vectors from log-mel filterbank energies.Sliding windows with 50% overlap are averaged after L2 normalization to compute one utterance-level d-vector.
- VoiceFilter system: VoiceFilter takes a target-speaker d-vector and a noisy-audio magnitude spectrogram, then predicts a soft mask.The mask is multiplied element-wise with the noisy spectrogram, and the noisy phase is reused to reconstruct the waveform by inverse STFT.
- VoiceFilter system: The VoiceFilter network contains 8 convolutional layers, 1 LSTM layer, and 2 fully connected layers.The d-vector is concatenated with convolutional outputs at every time frame before the following LSTM layers.
3. Experimental setup
The experiments separately train the system components, construct target-speaker training triplets, and evaluate VoiceFilter with WER and SDR. The setup tests whether VoiceFilter improves noisy-speech recognition while preserving clean-speech performance.
- Experimental setup: The two system components are trained separately, and evaluation uses speech recognition WER and Source to Distortion Ratio (SDR).The experimental setup describes separate training data for the speaker encoder and VoiceFilter system, alongside the two evaluation metrics.
- Data generation: Each training step uses clean target audio, multi-speaker noisy audio, and a separate reference recording from the target speaker.The reference recording supplies the d-vector, while the clean target audio serves as ground truth.
- Data generation: Reference audio is randomly selected from another utterance by the target speaker, while noisy audio is created from the clean and interfering signals.The supplied passages state that the reference differs from the clean audio and that the noisy signal is generated by mixing an interfering speaker.
- Data generation: Changing the interfering-audio weight with uniform ranges [0, 1] or [0, 2] did not affect VoiceFilter performance in the experiments.This tests sensitivity to the amplitude weighting used when generating interference.
- Evaluation criteria: Noisy-enhanced WER is significantly lower than Noisy WER, while Clean-enhanced WER remains very close to Clean WER.These are the two stated properties of a good VoiceFilter model: improvement for multi-speaker scenarios and minimal impact for single-speaker scenarios.
4. Results
VoiceFilter lowers speech-recognition WER on noisy multi-speaker data while preserving near-clean performance, with results varying by dataset and training set size. The bi-directional LSTM variant performs best on LibriSpeech, generalizes to VCTK, and outperforms a permutation-invariant separation baseline.
- LibriSpeech: 58.1% relative WER reduction is achieved on noisy LibriSpeech data with the bi-directional LSTM VoiceFilter model.On clean data, WER changes from 10.9% to 11.1%.
- LibriSpeech: The bi-directional LSTM achieves the best WER on noisy LibriSpeech data, while sequential LSTM processing is important relative to the convolution-only model.
- VCTK: 38.9% relative WER reduction is obtained on noisy VCTK data when VoiceFilter is trained on VCTK.Clean-data WER is significantly higher in this setting, which the paper attributes mostly to the 99-speaker training set.
- VCTK: Training VoiceFilter on LibriSpeech reduces VCTK clean-data WER to 5.9% and further decreases noisy-data WER, indicating cross-dataset generalization.The LibriSpeech training set contains 2,338 speakers versus 99 in VCTK.
- Source to distortion ratio: The bi-directional LSTM VoiceFilter achieves the highest SDR on LibriSpeech.Table 4 reports SDR in dB and compares VoiceFilter with a permutation-invariant-loss system.
- Source to distortion ratio: VoiceFilter outperforms the permutation-invariant-loss system, while avoiding the need to estimate speaker count and select an output in practical use.The comparison system uses the same bi-directional-LSTM architecture but does not receive speaker embeddings.
5. Conclusions and future work
The paper demonstrates that discriminatively trained speaker encoders can condition VoiceFilter effectively, improving recognition in two-speaker mixtures while preserving single-speaker performance. Future work targets broader data, more interference, more robust embeddings, joint enhancement, and recognition training.
- VoiceFilter reduces speech recognition WER from 55.9% to 23.4% in two-speaker scenarios.
- WER stays approximately the same on single-speaker scenarios.
- The system uses a discriminatively trained speaker encoder to condition speech separation.
- Future improvements include larger, more challenging datasets, additional interfering speakers, and d-vectors computed over several utterances.These changes are intended to obtain more robust speaker embeddings.
- Future directions include joint voice separation and speech enhancement, plus joint training with speech recognition.The enhancement direction would remove interfering speakers and ambient noise.