Source-linked AI summary
Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
Fadi Biadsy, Ron J. Weiss, Pedro J. Moreno, Dimitri Kanevsky, Ye Jia
TL;DR
Parrotron addresses speech conversion by mapping an input spectrogram directly to an output spectrogram without an intermediate symbolic representation. The model normalizes speech, improves adapted deaf-speaker speech, and also performs speech separation.
Problem
The paper investigates whether modern end-to-end speech conversion can improve intelligibility and naturalness for deaf speakers with highly atypical speech.
Source: §1 · p. 1, §3.2 · p. 3Method
Results
Parrotron normalizes speech, adapts to improve deaf-speaker speech, and separates overlapping speech; adapting all parameters reduced WER from 89.2% to 32.7%.
Source: §4 · p. 4, §3.2.1 · p. 4Takeaways & Limitations
The model's applications include improving vocal communication for people with speech disorders and serving as a front end to voice-enabled systems.
Source: §3.2 · p. 3The proposed advantage of direct speech conversion over ASR followed by TTS has not yet been quantified.
Source: §3.2.1 · p. 4
Abstract
from arXiv · showhide
We describe Parrotron, an end-to-end-trained speech-to-speech conversion model that maps an input spectrogram directly to another spectrogram, without utilizing any intermediate discrete representation. The network is composed of an encoder, spectrogram and phoneme decoders, followed by a vocoder to synthesize a time-domain waveform. We demonstrate that this model can be trained to normalize speech from any speaker regardless of accent, prosody, and background noise, into the voice of a single canonical target speaker with a fixed accent and consistent articulation and prosody. We further show that this normalization model can be adapted to normalize highly atypical speech from a deaf speaker, resulting in significant improvements in intelligibility and naturalness, measured via a speech recognizer and listening tests. Finally, demonstrating the utility of this model on other speech tasks, we show that the same model architecture can be trained to perform a speech separation task
1. Introduction
Parrotron directly transduces speech spectrograms into spectrograms without an intermediate discrete representation, targeting speech normalization and adaptation to atypical speech.
1. Introduction
- Parrotron maps an input spectrogram directly to an output spectrogram, without an intermediate discrete representation.Source: §1 · p. 1
- The normalization task projects away speaker and other non-linguistic information while preserving speech content in a canonical target voice.The paper evaluates preservation of content and removal of other information using ASR and listening studies.Source: §1 · p. 1
- Adapting the pretrained normalizer converts highly atypical deaf-speaker speech into fluent speech with significantly improved intelligibility and naturalness.Source: §1 · p. 1
- The same network is also evaluated for speech separation.The introduction frames this as a test of whether the model can perform the task.Source: §1 · p. 1
- Unlike, Parrotron focuses on many-to-one conversion, and unlike [18], it uses no auxiliary alignment or auto-encoding losses.It also multitask-trains the model to predict source speech phonemes.Source: §1 · p. 1
2. Model Architecture
Parrotron is an end-to-end sequence-to-sequence model that converts paired source speech into synthesized target speech. Its encoder and attention-based decoders predict spectrograms, while an auxiliary transcript decoder encourages retention of speech content.
2. Model Architecture
- Parrotron maps source speech to synthesized target speech using an attention encoder-decoder followed by a vocoder.The output in Figure 1 is from a different gender and has higher pitch and formants, with a slightly slower speaking rate.Source: §2 · p. 1, Figure 1 · p. 2
2.1. Spectrogram encoder
- The encoder extracts 80-dimensional log-mel frames, downsamples the sequence fourfold with convolutional layers, then computes a 512-dimensional representation using bidirectional recurrent layers.Source: §2.1 · pp. 1–2
2.2. Spectrogram decoder
- The autoregressive decoder predicts target spectrogram frames with attention and a residual post-net; Griffin-Lim or, for listening tests, WaveRNN synthesizes audio.Source: §2.2 · p. 2
2.3. Multitask training with an ASR decoder
- An auxiliary transcript decoder trains the shared encoder to retain linguistic content, biasing spectrogram predictions toward the underlying speech.Table 1 compares WER across architecture variations and auxiliary ASR losses.Source: §2.3 · p. 2, Table 1 · p. 2
3. Applications
Parrotron is applied to voice normalization, atypical deaf-speaker speech, and speech separation. The experiments show benefits from adapting the normalizer to atypical speech and demonstrate separation on overlapping-speaker mixtures.
3.1. Voice normalization
- Voice normalization maps varied input speech to a canonical speaker using synthetic TTS targets, enabling a large paired corpus with consistent speaker and accent.Training used about 24 million English utterances, approximately 30,000 hours, with targets synthesized in a female voice.Source: §3.1 · p. 2, §3.1.1 · p. 2
3.1.1. Experiments
- 17.6% WER was achieved with slower learning-rate decay; auxiliary grapheme prediction reduced WER from 27.1% to 19.9%.Location-sensitive attention further improved results, while phoneme targets showed consistent improvements; removing the CLSTM hurt performance.Source: §3.1.1 · p. 3
- On heavily accented, noisy speech, Parrotron WER was comparable to real speech despite slightly lower naturalness, and listeners confirmed consistent use of the TTS voice.The voice-consistency study presented 20 utterances to 8 native English subjects.Source: §3.1.1 · p. 3, Table 2 · p. 3, Table 3 · p. 3
3.1.2. Error analysis
- Plosives and other short phonemes are difficult to model under framewise L2 loss, which gives vowel and /t/ frames equal weight despite their differing frequency in training.In the top 5% of increased confusions, 26% were plosives and 12% were vowel exchanges; Arabic and Spanish inputs sometimes produced American-accented echoes of the original content.Source: §3.1.2 · p. 3
3.2. Normalization of hearing-impaired speech
- The hearing-impaired speech application targets fluent conversion for accessibility, using a case study of a profoundly deaf speaker who learned English as a teenager after being born in Russia.The authors also describe possible use for people with other speech disorders or as a front-end to voice-enabled systems; adaptation used 15.4 hours of read movie quotes.Source: §3.2 · p. 3
3.2.1. Experiments
- Full-model adaptation reduced deaf-speaker WER from 89.2% to 32.7% and increased MOS naturalness by over 1.4 points versus the original speech.All three finetuning strategies improved intelligibility and naturalness; the unadapted model produced speech-like sounds but nonsense words.Source: §3.2.1 · p. 4
3.3. Speech separation
- The separation experiment reconstructs a target speaker from instantaneous mixtures of up to eight voices; Parrotron reduced WER, especially insertions, but increased deletions.The mixtures used randomly selected background utterances; the reported average SNR was 12.15 dB, and the authors attribute increased deletions likely to early end-of-utterance prediction.Source: §3.3 · p. 4, Table 5 · p. 4
4. Conclusion
Parrotron directly converts input spectrograms into output spectrograms without intermediate symbolic representations. It normalizes speech, adapts to improve deaf-speaker speech, and separates overlapping speakers to improve ASR performance.
4. Conclusion
- Parrotron directly maps input spectrograms to output spectrograms without an intermediate symbolic representation.The same model can identify, separate, and reconstruct the loudest speaker in overlapping speech, improving ASR performance.Source: §4 · p. 4
- The model normalizes speech from different speakers into one target voice while preserving linguistic content and projecting away non-linguistic content.Source: §4 · p. 4
- Adapting Parrotron improves word error rate and naturalness for a deaf speaker.Source: §4 · p. 4
- Potential applications include converting heavily accented or otherwise atypical speech into standard speech.The authors plan to test other speech disorders and adopt techniques to preserve speaker identity.Source: §4 · p. 4