PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

Murali Karthick Baskar, Shinji Watanabe, Ramon Astudillo, Takaaki Hori, Lukáš Burget, Jan Černocký

arXiv:1905.01152v2eess.AScs.CLcs.IRcs.LGcs.SD

TL;DR

Seq2seq ASR needs substantial paired speech and text, a constraint for under-resourced domains where unpaired data is abundant. This paper combines cycle-consistency losses for unpaired speech and text; experiments report gains from each modality and additional gains when combined.

  • Problem

    Seq2seq ASR requires considerable paired speech and text to learn alignment and classification, limiting its use in under-resourced domains.

    Source: §1 · p. 1

  • Method

    The paper derives cycle-consistency losses for unpaired speech and text, using ASR→TTS and TTS→ASR pipelines and combining them with a shallow integrated RNNLM.

    Source: §1 · p. 1

  • Results

    Experiments on WSJ and Librispeech support that unpaired speech and text individually improve ASR, while combining them provides additional gains.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The reported experiments support using both unpaired speech and text to improve ASR performance across the WSJ and Librispeech corpora.

    Source: §1 · p. 1

    The x-vectors retain speaker characteristics but not general speech structure, so the model cannot directly copy the input speech to its output.

    Source: §2.2 · p. 2

Abstract

from arXiv · show

Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.

1. Introduction

Seq2seq ASR needs substantial paired speech and text, while unpaired data is widely available and relevant to low-resource domains. This paper uses cycle-consistency to build losses for unpaired speech and text, then combines them with an integrated RNNLM.

1. Introduction

  • Seq2seq ASR requires considerable paired speech and text, limiting its use in under-resourced domains despite the availability of abundant unpaired data.The model learns alignment and classification from paired data.Source: §1 · p. 1
  • Prior low-resource methods use unpaired speech, unpaired text, or both, including ASR–TTS/TTE cycles, decoder or encoder approaches, and adversarial training.Speech-chain systems use ASR→TTS and TTS→ASR pipelines, with backpropagation only in the secondary stage.Source: §1 · p. 1
  • Cycle-consistency has also supported end-to-end differentiability in machine translation and improvements in image-processing tasks.The cited machine-translation approach uses an N-best method; the image-processing work pairs cycle-consistency with an adversarial objective.Source: §1 · p. 1
  • The paper applies cycle-consistency to derive a loss that uses both unpaired speech and text by comparing an input with its reconstruction.The motivating principle is that a model's output should allow another model to reconstruct a similar input.Source: §1 · p. 1
  • The proposed speech-only ASR→TTS and text-only TTS→ASR losses are combined with a shallow integrated RNNLM, with experiments on WSJ and Librispeech.The authors report that each unpaired modality can individually improve ASR and that combining them provides additional gains.Source: §1 · p. 1

2. Cycle consistency training

The paper uses seq2seq ASR and TTS models as the components of cycle-consistency training, then extends that training to unpaired speech and text. Its speech-only pipeline reconstructs speech from sampled ASR outputs, while its text-only pipeline trains ASR on TTS-generated speech.

2.1. ASR and TTS with seq2seq models

  • Seq2seq ASR uses an encoder-decoder with attention to predict characters from speech, while TTS maps character sequences to speech features using Tacotron2.With paired data, the models use cross-entropy for ASR; TTS loss includes regression and end-of-utterance terms.Source: §2.1 · pp. 1–2

2.2. Unsupervised training with unpaired speech data

  • The ASR→TTS speech-only cycle uses the TTS loss instead of TTE loss to reconstruct speech from ASR outputs, with speaker vectors carrying speaker characteristics through the text bottleneck.ASR outputs are sampled, and policy gradient updates ASR parameters using the reconstruction loss as a reward.Source: §2.2 · pp. 2–3

2.3. Unsupervised training with unpaired text data

  • The TTS→ASR text-only cycle generates speech from unpaired text and uses its transcription to train ASR without backpropagating into TTS.The point estimate uses randomly chosen x-vectors; the pipeline is described as akin to back-translation.Source: §2.3 · p. 3

X {pTTS(X | Y)}. (12)

The TTS→ASR pipeline uses synthetic speech to train ASR from unpaired text, while the combined loss jointly trains ASR→TTS and TTS→ASR objectives.

X {pTTS(X | Y)}. (12)

  • Synthetic speech generated from unpaired text is transcribed by ASR, making the TTS→ASR pipeline an autoencoder-like training route for text-only data.The pipeline is non-differentiable and is described as akin to back-translation [23].Source: X {pTTS(X | Y)}. (12) · p. 3

2.4. Unsupervised training with both unpaired speech and text data

  • When paired speech and text are available, Lboth jointly uses the ASR→TTS and TTS→ASR losses through linear interpolation.The interpolation weight α defaults to 0.5.Source: §2.4 · p. 3

3. Experimental Setup

Experiments use WSJ and Librispeech subsets to simulate unpaired speech and text, with conventional supervised training followed by unsupervised training.

3. Experimental Setup

  • Experiments use Librispeech and WSJ subsets to simulate unpaired speech and text conditions.For WSJ, 81 hours are divided into 2, 5, 10, and 14 paired hours, with the remaining 67 hours unpaired; Librispeech uses 100 paired and 360 unpaired hours.Source: §3 · p. 3
  • The ASR encoder has eight bidirectional LSTM layers and the decoder has one unidirectional LSTM layer, each with 320 units.Inputs are 83-dimensional filter-bank features with pitch; training uses AdaDelta, batch size 30, and 20 epochs.Source: §3 · p. 3
  • Unsupervised training follows conventional supervised training, and a small amount of paired data regularizes the unsupervised stage.Cycle-consistency uses five samples in Eq. 10; evaluations use WSJ eval92 and Librispeech test-clean.Source: §3 · p. 3, Table 1 · p. 3

4. Analysis and Results

Across WSJ and Librispeech, the paper evaluates unpaired-data quantity and modality, compares its TTS loss with TTE, and reports results against prior approaches. The results vary with supervision and corpus: text-only data helps in very low-resource WSJ, while speech-only data is more beneficial on large Librispeech.

4. Analysis and Results

  • 2.5% relative lower %WER with TTS than TTE on 360 hours of unpaired data (20.7% to 20.1%), and 2.9% on 180 hours (19.9% to 19.4%).These comparisons use Librispeech with 100 hours of paired data.Source: §4 · p. 3

4.1. Impact of amount of paired data (semi-supervision)

  • With 2 hours of paired data, %WER falls from 68.2 to 49.8 with 14 hours of unpaired speech, while 67 hours yields 51.9.The comparison is on WSJ eval-92 and varies paired and unpaired data quantities and modality.Source: §4.1 · p. 3, Table 2 · p. 4
  • Under very low-resource conditions, text-only data improved consistently from 63 (pair 14 hours) to 39.6 (pair 67 hours), while overfitting was observed with unpaired speech.With 14 hours of supervision and 67 hours of unpaired speech, text, and both, %WER was 28.0, 27.1, and 26.2, respectively.Source: §4.1 · pp. 3–4
  • With 14 hours of supervision, increasing unpaired speech from 14 to 67 hours improves WER by ∼1% absolute.At 67 hours of unpaired data, speech, text, and both yield 28.0, 27.1, and 26.2 %WER, respectively.Source: §4.1 · p. 4

4.2. Model comparison with literature

  • 20.3 %WER with RNNLM and 26.1 %WER without RNNLM on WSJ; on Librispeech, joint unpaired audio and text achieves 17.5 %WER.The WSJ result is 37.1% relatively different from the 16.4% oracle; Librispeech is 32.5% relatively different from its oracle. On Librispeech, text-only training yields an 18.0% relative improvement over backtranslation-TTE, with a further absolute 0.9% gain from RNNLM integration.Source: §4.2 · p. 4, Table 3 · p. 4
  • On Librispeech, unpaired speech alone yields 16.8% WER, and combining unpaired speech and text changes it to 16.6%.The authors infer that speech-only training has major benefits over text-only training on a large corpus such as Librispeech.Source: §4.2 · p. 4

5. Related Work

The paper positions its loss as combining and extending earlier approaches, joining direct-speech TTS→ASR training with an end-to-end differentiable ASR→TTS loss.

5. Related Work

  • Unlike the speech-chain, the proposed approach jointly combines TTS→ASR pipeline loss with end-to-end differentiable ASR→TTS loss.The speech-chain alternates the losses to update both models and is not end-to-end differentiable.Source: §5 · p. 4
  • The TTS→ASR loss resembles TTE back-translation but uses speech directly and is reported to perform better overall.The paper extends TTE to TTS with x-vectors and combines it with a TTS→ASR point-estimate loss.Source: §5 · p. 4
  • The ASR→TTS component extends the speech-only differentiable TTE loss to TTS with x-vectors, and is related to the straight-through approach [8].The paper describes [8] as constructing ASR→TTS losses with straight-through estimators.Source: §5 · p. 4

6. Conclusions

The paper proposes using unpaired speech and text to improve seq2seq ASR. It reports relatively larger gains under low-resource conditions and additional gains from combining modalities and shallow RNNLM integration.

6. Conclusions

  • The approach uses unpaired speech and/or text to improve seq2seq ASR performance.The conclusion describes both pipeline loss and shallow RNN language-model integration as ways to combine the modalities.Source: §6 · p. 4
  • Performance improvements are relatively higher on low-resource WSJ than on Librispeech, which has sufficient data.The comparison is framed in terms of corpus data quantity.Source: §6 · p. 4
  • Combining unpaired speech and text through pipeline loss and shallow RNNLM integration provides additional gains and competitive results.Future work targets cycle-consistency when ASR and TTS conditions do not match; preliminary experiments identify this as a current limitation.Source: §6 · p. 4

http://arxiv.org/ps/1905.01152v2

The supplied passages identify a figure and its caption, which concerns % WER on the eval92 test set across varying a.

% WER

The supplied passages indicate eval92 %WER results involving varying amounts of paired data, but provide no readable values.

% WER

  • The passages label eval92 %WER results for varying amounts of paired data and list a baseline and speech/text additions, without readable values.The supplied labels include +14-hour and +67-hour additions, but their truncated text does not establish complete conditions or comparisons.Source: % WER · pp. 13, 14, 15, 19

% WER

The section fragment identifies a WSJ %WER figure and mentions 2, 5, and 67 hours, but gives no performance comparison.

% WER

  • The WSJ %WER figure is identified as wsj_chart.png.Source: % WER · p. 21
  • Neither passage provides a WER value or a comparative result.Source: % WER · pp. 21–22
  • The fragment mentions 2hrs, 5hrs, and 67 hours, without specifying their comparisons or outcomes.Source: % WER · p. 22

Related papers

Loading 1905.01152v2…