PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling

Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, Takaaki Hori

arXiv:1810.03459v1cs.CLcs.LGcs.SDeess.AS

TL;DR

Seq2seq ASR integrates acoustic modeling, alignment, and language modeling but requires more data than conventional hybrid systems. The paper trains a multilingual model, transfers it to target languages, and evaluates a multilingual character RNNLM; the experiments report fine-tuning gains and improved %WER with RNN-LM decoding.

  • Problem

    Seq2seq ASR integrates acoustic modeling, alignment, and language modeling but requires more data than conventional hybrid systems.

    Source: §1 · p. 1

  • Method

    The paper applies multilingual training and transfer to joint CTC-attention seq2seq models, and evaluates a multilingual character-based RNNLM during decoding.

    Source: §1 · p. 1

  • Results

    Stage-2 fine-tuning yields absolute %CER gains of 1.6% after epoch 4 and 4.3% after epoch 15, while RNN-LM decoding adds an average absolute gain of ∼6% across target languages.

    Source: §4.3 · p. 4, §4.4 · p. 5

  • Takeaways & Limitations

    The experiments show that transfer learning is important in this seq2seq setting and that character-based RNNLMs boost %WER across the available target-data sizes.

    Source: §5 · p. 5

    The naive multilingual approach includes target-language data in training, so it does not keep target languages unseen.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Sequence-to-sequence (seq2seq) approach for low-resource ASR is a relatively new direction in speech research. The approach benefits by performing model training without using lexicon and alignments. However, this poses a new problem of requiring more data compared to conventional DNN-HMM systems. In this work, we attempt to use data from 10 BABEL languages to build a multi-lingual seq2seq model as a prior model, and then port them towards 4 other BABEL languages using transfer learning approach. We also explore different architectures for improving the prior multilingual seq2seq model. The paper also discusses the effect of integrating a recurrent neural network language model (RNNLM) with a seq2seq model during decoding. Experimental results show that the transfer learning approach from the multilingual model shows substantial gains over monolingual models across all 4 BABEL languages. Incorporating an RNNLM also brings significant improvements in terms of %WER, and achieves recognition performance comparable to the models trained with twice more training data.

1. INTRODUCTION

Seq2seq ASR combines acoustic modeling, alignment, and language modeling, but requires more data than conventional hybrid systems. This paper brings multilingual approaches into joint CTC-attention models and investigates transfer to unseen languages and multilingual character RNNLMs.

1. INTRODUCTION

  • Prior multilingual CTC work used universal phonesets and FST or language-specific adaptation, while attention-based multilingual models pooled languages seen during training without transfer.The paper frames transfer to languages unseen during training as an unresolved need.Source: §1 · p. 1
  • The work incorporates multilingual approaches into a joint CTC-attention seq2seq framework with beam-search decoding.It also investigates transferring the multilingual model across different target-language data sizes.Source: §1 · p. 1
  • The low-resource strategy combines transfer learning with a multilingual character-based RNNLM.The RNNLM is included during decoding, as described in the paper’s contribution statement.Source: §1 · p. 1

2. SEQUENCE-TO-SEQUENCE MODEL

  • CTC loss supplements attention because attention has limitations in monotonic alignment; the two objectives are unified with a linear interpolation weight.The paper describes the unified model as combining monotonicity with sequence-level training.Source: §2 · p. 1
  • A shared encoder is trained with both CTC and attention objectives, and the joint decoder predicts labels using CTC, the attention decoder, and an RNN-LM.The network architecture is presented as a hybrid attention/CTC model with an LM extension.Source: §2 · p. 1, Figure 1 · p. 2

Encoder

The encoder supplies shared output states to both CTC and attention, with recurrent and convolutional-recurrent alternatives explored.

Encoder

  • Both CTC and attention use the same encoder function to produce encoder output states.The encoder is denoted Encoder(X), with h_t as its output state at time t.Source: Encoder · p. 2
  • The encoder alternatives are a bidirectional LSTM or a deep CNN followed by BLSTMs.The paper investigates convolutional layers in the joint CTC-attention multilingual setting.Source: Encoder · p. 2
  • The VGG-based option uses the initial six VGG layers to produce 128 feature maps downsampled to one-quarter along time and frequency.Two max-pooling layers with stride 2 perform the downsampling.Source: Encoder · p. 2

Attention Decoder:

The attention decoder uses location-aware attention to form a context vector from encoder states, then predicts the next character from that context and its previous label and hidden state.

Attention Decoder:

  • Equation (5) denotes the output of location-aware attention, with a_l^t acting as an attention weight.Source: Attention Decoder · p. 2
  • Location-aware attention convolves the previous label's attention weights into a multichannel representation for improved representation.Source: Attention Decoder · p. 2
  • Attention weights are normalized, then used to compute a context vector as a weighted sum of encoder outputs across frames.Source: Attention Decoder · p. 2
  • The LSTM decoder predicts the next character from the previous label, decoder hidden state, and attention output.Source: Attention Decoder · p. 2

Connectionist temporal classification (CTC):

CTC combines an RNN-based encoder with a language model and state-transition module to obtain character-level recognition probabilities.

Connectionist temporal classification (CTC):

Joint decoding:

The paper jointly uses CTC and attention-based seq2seq models during decoding, combining their scores in beam search.

Joint decoding:

  • CTC and attention-based seq2seq models are jointly used for decoding.Source: Joint decoding · p. 2
  • The joint decoding score interpolates the CTC and attention log probabilities using α.Source: Joint decoding · p. 2
  • During beam search, α weights the CTC and attention models, and α and the multi-task learning weight λ are set differently across experiments.Source: Joint decoding · p. 2

3. DATA DETAILS AND EXPERIMENTAL SETUP

The experiments use BABEL speech data and 80-dimensional filterbank features; character error rate is used for seq2seq experiments, with word error rate used when integrating an RNNLM.

3. DATA DETAILS AND EXPERIMENTAL SETUP

  • The BABEL corpus consists mainly of conversational telephone speech, with some scripted and far-field recordings.Source: §3 · p. 3
  • The speech samples are represented by 80-dimensional Mel-filterbank features extracted with 25 ms windows and a 10 ms stride.Source: §3 · p. 3
  • Seq2seq experiments use character error rate, while the RNNLM-integrated evaluation in section 4.4 uses word error rate; experiments are implemented in ESPnet.Source: §3 · p. 3

4. MULTILINGUAL EXPERIMENTS

The section describes a staged multilingual seq2seq approach for low-resource speech recognition. Its naive model shares training across languages, but includes target-language data during training.

4. MULTILINGUAL EXPERIMENTS

  • The transfer sequence starts with naive combined-language training, then retrains both decoders, and finally retrains the encoder and decoder.Stage 1 initializes from Stage 0 and retrains the CTC and attention decoders; Stage 2 retrains the encoder and decoder.Source: §4 · p. 3
  • Table 4 compares the naive approach with training only the last layer for Assamese.Source: Table 4 · p. 3

4.1. Stage 0 - Naive approach

  • The approach trains a single seq2seq model using data from 10 languages totaling approximately 600 hours, with a character-label set drawn from all languages in both the training and target sets.Table 3 reports recognition performance of the naive multilingual approach on an evaluation set of 10 BABEL training languages, trained using the training set of those same languages.Source: §4.1 · p. 3, Table 3 · p. 4
  • The naive model can improve robustness and recognition for limited-data languages, but it does not keep target-language data unseen during training.Source: §4.1 · p. 4

Comparison of VGG-BLSTM and BLSTMP

The experiments compare multilingual architectures, target-language retraining strategies, and RNNLM-assisted decoding. Stage-2 retraining improves over monolingual models across languages, while RNNLM decoding gives consistent WER gains across target-data sizes.

Comparison of VGG-BLSTM and BLSTMP

  • VGG-BLSTM gives an 8.8% absolute average gain over monolingual models except Pashto, Georgian, and Tokpisin; BLSTMP gives a 5.0% gain except Pashto and Georgian.VGG-BLSTM could not be used for stage-1 and stage-2 retraining because of time constraints, so retraining experiments used multilingual BLSTMP.Source: Comparison of VGG-BLSTM and BLSTMP · p. 4

4.2. Stage 1 - Retraining decoder only

  • Stage 1 retrains the target-language classification layer; both the CTC and attention decoder layers are updated during this stage.The learning rate decays by a factor of 1e−1 when validation accuracy drops.Source: §4.2 · p. 4

4.3. Stage 2 - Finetuning both encoder and decoder

  • Stage-2 retraining across the encoder and decoder gives an absolute %CER gain over monolingual models across all languages using the full target-language data.Decoder-only retraining degraded %CER from 45.6 to 61.3; stage-2 fine-tuning yielded absolute gains of 1.6% after epoch 4 and 4.3% after epoch 15. Gains from 5 to 10 hours exceeded those from 20 hours to the full set.Source: §4.3 · p. 4, Table 5 · p. 4

4.4. Multilingual RNNLM

  • Adding an RNNLM yields an average absolute gain of ∼6% across target languages; 5-hour Assamese with an RNNLM is nearly comparable to stage-2 retraining with 20 hours.The reported RNNLM gains are consistent across target-data sizes, with more distinctive improvements in %WER than %CER.Source: §4.4 · p. 5, Table 6 · p. 5, Figure 3 · p. 5

5. CONCLUSION

The conclusion highlights stage-2 multilingual retraining and character-based RNNLM decoding as effective components of the seq2seq approach. It also points to the breadth of recognition performance across selected BABEL train and target languages.

5. CONCLUSION

  • Stage-2 multilingual retraining is an important transfer-learning approach in the seq2seq setting.The authors report that their character-based RNNLM boosts %WER across the different amounts of target-language data available for transfer learning.Source: §5 · p. 5
  • The selected BABEL training and target languages provide a wide variety in recognition performance (%CER), helping assess seq2seq model efficacy.Source: §5 · p. 5
  • The methods are reported to work with both attention- and CTC-based seq2seq models without compromising performance.Source: §5 · p. 5

6. FUTURE WORK

The authors propose several directions for extending multilingual transfer and language-model-assisted decoding. These include architectural and language-conditioning changes, as well as alternative RNNLM integration strategies.

6. FUTURE WORK

  • Future work could use VGG-BLSTM as a multilingual prior before stage-2 retraining and add language vectors as input or target to reduce confusion.Source: §6 · p. 5
  • The authors propose investigating multilingual bottleneck features, word-level RNNLM decoding, and cold fusion for attention decoders.They aim to incorporate these techniques toward performance comparable with state-of-the-art hybrid DNN/RNN-HMM systems.Source: §6 · pp. 5–6
Loading 1810.03459v1…