Source-linked AI summary
Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
Suyoun Kim, Takaaki Hori, Shinji Watanabe
TL;DR
Attention-based speech recognition can struggle with alignment in noisy conditions and during early training on long inputs. The paper addresses this with a joint CTC-attention multi-task model, which outperforms CTC and attention baselines on clean and noisy tasks while improving convergence.
Problem
Attention-based recognition can produce poor alignments in noisy speech and is difficult to learn from scratch on longer input sequences.
Method
The method jointly trains a shared encoder with CTC and attention objectives in a multi-task learning framework.
Results
The model outperformed both CTC and attention baselines on noisy CHiME-4 and clean WSJ, with 5.4 - 14.6% relative CER improvements on evaluation sets.
Takeaways & Limitations
The joint framework improves recognition performance and speeds learning of the desired alignment without manually restricting the input range.
Takeaways & Limitations
The proposed explanation that CTC improves generalisation through its training procedure requires additional experiments for verification.
Abstract
from arXiv · showhide
Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments. One approach is the attention-based encoder-decoder framework that learns a mapping between variable-length input and output sequences in one step using a purely data-driven method. The attention model has often been shown to improve the performance over another end-to-end approach, the Connectionist Temporal Classification (CTC), mainly because it explicitly uses the history of the target character without any conditional independence assumptions. However, we observed that the performance of the attention has shown poor results in noisy condition and is hard to learn in the initial training stage with long input sequences. This is because the attention model is too flexible to predict proper alignments in such cases due to the lack of left-to-right constraints as used in CTC. This paper presents a novel method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4-14.6% relative improvements in Character Error Rate (CER).
1 Mitsubishi Electric Research Laboratories (MERL) 2 Carnegie Mellon University (CMU)
The paper concerns end-to-end speech recognition, CTC, attention, and multi-task learning.
- End-to-end speech recognition is a central topic.
- Connectionist temporal classification is one of the named methods.
- Attention and multi-task learning are also central concepts.
1. INTRODUCTION
End-to-end speech recognition maps acoustic frames directly to characters without predefined alignments, unlike separately trained hybrid systems. The paper combines CTC and attention objectives to address attention-model alignment problems in noisy or long-input conditions and evaluates the approach on WSJ and CHiME-4.
- End-to-end recognition directly maps acoustic frames to characters without predefined alignments.
- CTC and attention-based encoder-decoder models are the two main end-to-end approaches discussed.
- Attention predicts each character using the input and previous target-character history, but noise and long inputs can produce misalignment and difficult training.
- The proposed method jointly trains a shared encoder with CTC and attention objectives in a multi-task learning framework.
- The model is evaluated on WSJ and CHiME-4 against both CTC and attention baselines in CER and learning speed.
2. JOINT CTC-ATTENTION MECHANISM
The joint CTC-attention framework trains a shared encoder with both CTC and attention objectives, combining CTC’s alignment constraints with attention-based character prediction. This multi-task design addresses noisy-speech misalignment and long-sequence training difficulties.
- 2.1. Connectionist temporal classification (CTC): CTC represents labels with repetitions and blanks, while computing sequence probabilities over possible alignments with the forward-backward algorithm.The blank symbol represents no output label, and the resulting network can be trained by backpropagation.
- 2.2. Attention-based encoder-decoder: The attention decoder predicts each character conditioned on encoder features and previously generated characters, using attention weights to select input information.Location-based attention additionally uses convolutional features derived from the previous attention weights.
- 2.2. Attention-based encoder-decoder: Attention-based recognition is vulnerable to noisy-speech misalignment and is difficult to train from scratch on long input sequences because it lacks monotonic alignment constraints.Restricting attention to a narrow input range can accelerate training but may limit information extraction from long character sequences.
- 2.3. Proposed model: Joint CTC-attention (MTL): The proposed method uses CTC as an auxiliary task to train an encoder shared by the CTC and attention models.The shared encoder transforms x into high-level features h, and the attention decoder generates the character sequence y.
- 2.3. Proposed model: Joint CTC-attention (MTL): CTC’s forward-backward algorithm enforces monotonic alignment, which the authors expect to improve noisy-condition robustness and accelerate network learning.The joint objective combines the CTC and attention losses with a tunable parameter λ, where 0 ≤ λ ≤ 1.
3. EXPERIMENTS
Experiments evaluate the proposed model on clean WSJ and noisy CHiME-4 speech, using specified architectures, decoding settings, and learning-curve analyses. MTL outperformed CTC and attention in CER, accelerated alignment learning, and achieved its best performance with λ = 0.2.
- Data: Experiments used clean WSJ1 and WSJ0 corpora and the noisy CHiME-4 corpus, with 120-dimensional filterbank-based input features.Evaluation used eval92 for WSJ and the et05 real isolated 1ch track for CHiME-4.
- Model and setup: The encoder was a four-layer bidirectional LSTM, while attention and MTL decoding used a one-layer LSTM decoder.The encoder reduced utterance length by a factor of four, with L = T/4.
- Model and setup: Decoding used beam search for attention and MTL and greedy most-likely outputs for CTC, without lexicon or language-model information.The attention and MTL beam size was 20.
- Results: 6.0–8.4% relative validation-set and 5.4–14.6% evaluation-set CER improvements were obtained over CTC and attention on noisy CHiME-4 and clean WSJ.The proposed MTL model significantly outperformed both baselines on both task conditions.
- Results: The best joint CTC-attention performance used λ = 0.2 on both noisy CHiME-4 and clean WSJ tasks.The clean-corpus advantage was also observed on WSJ1 and WSJ0.
- Results: MTL learned the desired alignment by the fifth epoch, whereas attention had not learned it by the ninth epoch in the analyzed CHiME-4 utterance.The authors attribute the monotonic alignment guidance to the CTC loss.
4. CONCLUSIONS
The paper introduces a joint CTC-attention multi-task method that improves end-to-end speech recognition and accelerates alignment learning. It outperforms both CTC and attention baselines in clean and noisy conditions.
- The method jointly trains a shared encoder with CTC and attention encoder-decoder objectives.
- It improves performance while significantly speeding alignment learning, including for longer sequences without manually restricting input ranges.
- The approach outperforms both CTC and attention models in real-world noisy and clean speech recognition conditions.
- The general method could potentially extend to other sequence-to-sequence learning tasks.