Source-linked AI summary
Listening while Speaking: Speech Chain by Deep Learning
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura
TL;DR
ASR and TTS have largely developed independently despite their linked roles in speech communication. This paper builds a deep-learning closed-loop speech chain that trains both models with paired and unpaired data, improving ASR and TTS performance in single- and multi-speaker experiments.
Problem
ASR and TTS have progressed largely independently despite the close relationship between speech perception and production.
Method
A closed-loop sequence-to-sequence architecture jointly connects ASR and TTS so each reconstructs speech or text from the other using labeled and unlabeled data.
Results
The approach improved ASR and TTS performance in both single-speaker and multi-speaker experiments using unpaired data.
Takeaways & Limitations
The machine speech chain integrates speech perception and production and enables the two models to teach each other using unpaired data.
Abstract
from arXiv · showhide
Despite the close relationship between speech perception and production, research in automatic speech recognition (ASR) and text-to-speech synthesis (TTS) has progressed more or less independently without exerting much mutual influence on each other. In human communication, on the other hand, a closed-loop speech chain mechanism with auditory feedback from the speaker's mouth to her ear is crucial. In this paper, we take a step further and develop a closed-loop speech chain model based on deep learning. The sequence-to-sequence model in close-loop architecture allows us to train our model on the concatenation of both labeled and unlabeled data. While ASR transcribes the unlabeled speech features, TTS attempts to reconstruct the original speech waveform based on the text from ASR. In the opposite direction, ASR also attempts to reconstruct the original text transcription given the synthesized speech. To the best of our knowledge, this is the first deep learning model that integrates human speech perception and production behaviors. Our experimental results show that the proposed approach significantly improved the performance more than separate systems that were only trained with labeled data.
1 Introduction
Speech perception and production are tightly linked in human communication, but ASR and TTS have largely developed separately. The paper uses deep learning to build a closed-loop speech chain that connects both processes and learns from labeled and unlabeled data.
- Human speech communication forms a chain from speech production through waveform transmission to auditory perception.
- ASR and TTS have progressed largely independently despite their close relationship.
- Auditory feedback lets speakers monitor volume, articulation, and comprehensibility while planning subsequent speech.
- Deep learning makes a joint sequence-to-sequence approach feasible for linking speech perception and production.
- The proposed closed-loop model connects ASR and TTS, using unlabeled speech and text through reciprocal transcription and reconstruction.
2 Machine Speech Chain
The machine speech chain jointly trains sequence-to-sequence ASR and TTS models in both directions. Reconstruction on unpaired speech or text supplies feedback that lets the models teach each other alongside supervised learning.
- The architecture contains sequence-to-sequence ASR and TTS models connected in both directions and trained jointly.
- Supervised training uses paired speech-text data to optimize ASR and TTS against their ground-truth target sequences.
- For speech-only data, ASR generates text and TTS reconstructs the original speech waveform from that text.
- For text-only data, TTS generates speech and ASR reconstructs the original text transcription from the synthesized waveform.
- Reconstruction losses from the two autoencoder-like processes allow ASR and TTS to teach each other using unpaired data.
3 Sequence-to-Sequence Model for ASR
The ASR system is an attention-based encoder-decoder that models the conditional probability of transcription sequences given framed speech features. Its attention mechanism supplies decoder context from relevant encoder information.
- A sequence-to-sequence ASR model directly models conditional probability p(y|x) from speech-feature sequences to label sequences.
- The attention-based encoder-decoder consists of encoder, decoder, and attention modules.
- The encoder transforms the input sequence into representative information used by the decoder.
- Attention provides context at each decoding step by locating relevant encoder information using encoder and decoder hidden states.
- The decoder predicts target-sequence probabilities from prior outputs and attention context, with speech transcriptions represented as phoneme or grapheme sequences.
4 Sequence-to-Sequence Model for TTS
The TTS component models speech synthesis as sequence-to-sequence reconstruction from text, using a modified Tacotron architecture with multiple spectrogram and end-of-frame outputs.
- Sequence-to-sequence formulation: TTS models the conditional probability p(x|y), generating framed speech features x from a character sequence y.The speech features are represented as a sequence of log Mel-scale or related spectrogram vectors.
- Architecture: The modified Tacotron encoder embeds characters, processes them through fully connected layers and CBHG, and supplies representations to the decoder.CBHG uses convolution banks, highway layers, and bidirectional GRUs.
- Decoder outputs: The decoder predicts a log Mel-scale spectrogram, reconstructs a log magnitude spectrogram, and estimates frame termination.The second spectrogram is generated from the first, while the decoder uses stacked LSTM layers and monotonic attention.
- Frame termination: The binary termination output marks the final speech frame because real-valued spectrogram outputs cannot use an end-of-sentence token.The authors report that this modification determined speech endings more robustly than forcing trailing zero frames.
- Training and waveform reconstruction: TTS training uses a loss over predicted spectrograms and frame-ending probabilities, while Griffin-Lim reconstructs the waveform from estimated phase information.The loss compares predicted log Mel and log magnitude spectrograms with ground truth and includes binary frame-ending prediction.
5 Experiment on Single-Speaker Task
The single-speaker experiment evaluates the speech-chain models on paired and generated speech data, measuring ASR transcription error and TTS spectrogram and endpoint quality. Training with unpaired data improved both ASR and TTS over labeled-data-only training.
- Dataset: The single-speaker study used 10,000 paired speech utterances for supervised ASR and TTS training, with additional development and test data.The speech dataset was generated from English BTEC sentences using Google TTS.
- Features and text: Speech features were extracted with STFT-based log magnitude and Mel-scale spectrogram processing, while text was normalized to lowercase symbols and special tags.The text inventory included letters, punctuation, and start, end, and space tags.
- Models: The ASR system used an attention-based encoder-decoder, while the TTS system modified Tacotron with LeakyReLU, reduced CBHG filters, stacked LSTM layers, and multi-frame prediction.The ASR encoder processed log-Mel spectrograms with stacked bidirectional LSTMs and sequence subsampling.
- Evaluation: Evaluation used character error rate for ASR, spectrogram MSE for TTS, and accuracy of predicting the final speech frame.The experiment also varied the supervised-loss coefficient and greedy versus beam-search decoding during unsupervised learning.
- Results: 4.6% lower CER was achieved by ASR than by the labeled-data-only system, while TTS also reduced spectrogram MSE and improved end-of-speech prediction accuracy.The reported gains followed additional training with unpaired data after initial training on a small paired dataset.
6 Experiment on Multi-Speaker Task
The multi-speaker experiment tested the speech chain on natural speech from multiple speakers using paired and unpaired subsets. Both ASR and TTS improved with additional unpaired-data training, with the larger ASR coefficient producing the stronger improvement.
- Data partitioning: The data split used 20 utterances per speaker for validation, 20 for testing, 80 paired speech-text examples for training, and remaining speech and text as unsupervised data.The paired, unpaired, validation, and test sets did not overlap.
- Model adaptation: The multi-speaker TTS system used modified Tacotron with speaker embeddings transferred from a pretrained single-speaker model except for the speaker-embedding layer.The ASR model was unchanged from the preceding experiment.
- Results: Both ASR and TTS improved after additional training on unpaired datasets, and α = 0.5 produced a larger ASR improvement than α = 0.25.The authors hypothesize that the weaker baseline required greater weighting of the paired-training loss and gradient.
7 Related Works
Prior speech technologies developed ASR and TTS largely separately, while related neural machine translation work used dual or semi-supervised learning with monolingual data. This paper adapts the closed-loop idea to spoken language by connecting ASR and TTS across speech and text domains.
- Dual learning: Dual-learning translation systems connect source-to-target and target-to-source models in a closed loop and use monolingual data for feedback.The systems use language-model likelihood and reconstruction error after training on a small parallel corpus.
- Semi-supervised translation: Semi-supervised NMT reconstructs monolingual corpora with an autoencoder built from source-to-target and target-to-source translation models.This approach trains on concatenated labeled parallel and unlabeled monolingual data.
- Speech-chain contribution: Unlike prior translation work, this paper connects ASR and TTS across different domains: speech is transcribed into text and text is synthesized into speech.The loop reconstructs unlabeled speech through ASR-to-TTS and unlabeled text through TTS-to-ASR.
- Research gap: The authors report that no prior studies had addressed analogous closed-loop problems in spoken language processing tasks.They describe the proposed mechanism as integrating human speech perception and production behaviors.
8 Conclusion
The paper demonstrates a deep-learning machine speech chain that jointly connects ASR and TTS through a closed-loop architecture. Experiments across single-speaker synthetic and multi-speaker natural speech show improved performance using unpaired data.
- The proposed machine speech chain uses a closed-loop sequence-to-sequence architecture to jointly connect ASR and TTS.
- Experiments covered both single-speaker synthetic speech and multi-speaker natural speech.
- ASR and TTS improved performance by teaching each other using only unpaired data.