Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning

1. INTRODUCTION

• End-to-end recognition maps acoustic frames to characters in one step without predefined alignments, unlike separately trained DNN-HMM components and their conditional-independence assumptions [1, 2, 3, 4, 5, 6, 7, 8, 9].The traditional hybrid system comprises acoustic, context-dependent phone transducer, pronunciation, and language models.Source: §1 · p. 1
• CTC and attention-based encoder-decoder models are two main end-to-end approaches [2] [13] [4] [6]; CTC uses repeated and blank labels, but assumes framewise conditional independence.CTC loss is efficiently computed with the forward-backward algorithm, while attention has often achieved better CER than CTC without an external language model [7].Source: §1 · p. 1
• Attention conditions each output character on the input and previous target history, but noisy speech can corrupt its alignments and long sequences make training from scratch difficult.The paper notes that windowing is commonly used to constrain attention, but its parameters must be set manually for the training data.Source: §1 · p. 1
• The joint CTC-attention system outperforms CTC and attention models in CER and learning speed on WSJ and CHiME-4.The introduction reports these gains qualitatively without giving numerical results.Source: §1 · p. 1
• Jointly training a shared encoder with CTC and attention objectives uses CTC’s forward-backward loss to rectify alignment problems and accelerate learning.The paper attributes attention’s weakness to lacking the left-to-right constraints available in CTC, especially with noisy data or long input sequences.Source: §1 · p. 1

2.1. Connectionist temporal classification (CTC)

• CTC computes the output distribution P(y|x) efficiently with the forward-backward algorithm; its conditional-independence assumption limits character-level language modeling, so lexicons or language models are commonly incorporated [2].Source: §2.1 · p. 2

2.2. Attention-based encoder-decoder

• Attention uses an RNN encoder-decoder that predicts each character from previous characters and attention-weighted encoder outputs, with content-based or location-based attention mechanisms.Location-based attention also uses convolutional features derived from the previous attention weights.Source: §2.2 · p. 2
• Attention models can misalign on noisy speech and are difficult to train from scratch on long inputs; restricting attention to a narrow input range can limit useful information from long character sequences.These issues arise because attention lacks a monotonic alignment constraint such as those in DNN-HMM and CTC.Source: §2.2 · p. 2

2.3. Proposed model: Joint CTC-attention (MTL)

• The proposed model trains a shared encoder with both CTC and attention objectives, using CTC’s monotonic alignment constraints to improve robustness and speed alignment learning without manual rough alignments.A tunable parameter λ, with 0 ≤ λ ≤ 1, combines the objectives.Source: §2.3 · pp. 2–3, Figure 1 · p. 3

3.1. Data

• Experiments covered clean WSJ1 and WSJ0 and noisy CHiME-4, using 120-dimensional filterbank features and no language model or lexicon.Evaluation used eval92 for WSJ and the et05 real isolated 1ch track for CHiME-4; development sets were dev93 and dt05 multi isolated 1ch track. Attention used 32 labels; CTC used a blank instead of sos/eos, while MTL used both.Source: §3.1 · p. 3

3.3. Results

• MTL (λ=0.2) significantly outperformed CTC and attention in CER on CHiME-4 and WSJ, with 5.4–14.6% relative evaluation-set and 6.0–8.4% validation-set gains.The authors report λ=0.2 as the best task weight across the noisy CHiME-4 and clean WSJ tasks.Source: Table 1 · p. 3, §3.3 · p. 4

3.2. Training and Decoding

• The encoder used a four-layer bidirectional LSTM with 320 cells per layer and direction, and the decoder used a one-layer LSTM with 320 cells.The encoder reduced utterance length by a factor of four; attention and MTL decoding used beam size 20, while CTC used the most likely output sequence. MTL tested λ values of 0.2, 0.5, and 0.8.Source: §3.2 · pp. 3–4

3.3. Results

• The authors suggest CTC may improve generalization because its training does not explicitly use character interdependencies, but say this needs further verification.They raise this possibility after observing that MTL also outperformed both baselines on clean WSJ1 and WSJ0.Source: §3.3 · p. 4
• MTL learned the desired alignment by epoch 5, while attention had not learned it by epoch 9; greater CTC-loss weight also led to quicker convergence.The authors interpret this as CTC loss guiding MTL alignments to be monotonic.Source: Figure 2 · p. 3, §3.3 · p. 4, Figure 3 · p. 4

• The method outperformed both CTC and attention models on speech recognition tasks in real-world noisy and clean conditions.Source: §4 · p. 4
• An auxiliary CTC objective trains a shared encoder as part of the multi-task method for end-to-end speech recognition.Source: §4 · p. 4
• The method significantly speeds alignment learning, even for longer sequences, without manually restricting the input range.Source: §4 · p. 4
• The approach may also apply to other sequence-to-sequence learning tasks.Source: §4 · p. 4