Source-linked AI summary
Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM
Takaaki Hori, Shinji Watanabe, Yu Zhang, William Chan
TL;DR
The paper tackles the complexity and linguistic-resource requirements of traditional modular ASR. It extends joint CTC-attention recognition with joint decoding, a VGG-based deep CNN encoder, and an RNN-LM, achieving state-of-the-art performance against hybrid systems on Japanese and Chinese tasks.
Problem
Traditional ASR depends on complex modules, pronunciation dictionaries, language models, and linguistic preprocessing, making systems difficult for non-experts and new languages.
Method
The model jointly trains CTC and attention objectives, adds a VGG-based deep CNN encoder, combines CTC and attention probabilities during decoding, and integrates a character-level RNN-LM.
Results
The end-to-end joint CTC-attention model outperforms hybrid systems without an explicit language model on the Japanese task and achieves state-of-the-art performance with a pretrained character-level language model on Chinese and Japanese.
Takeaways & Limitations
The approach significantly improves end-to-end ASR without linguistic resources such as morphological analyzers and pronunciation dictionaries, while remaining competitive with conventional hybrid systems.
Abstract
from arXiv · showhide
We present a state-of-the-art end-to-end Automatic Speech Recognition (ASR) model. We learn to listen and write characters with a joint Connectionist Temporal Classification (CTC) and attention-based encoder-decoder network. The encoder is a deep Convolutional Neural Network (CNN) based on the VGG network. The CTC network sits on top of the encoder and is jointly trained with the attention-based decoder. During the beam search process, we combine the CTC predictions, the attention-based decoder predictions and a separately trained LSTM language model. We achieve a 5-10\% error reduction compared to prior systems on spontaneous Japanese and Chinese speech, and our end-to-end model beats out traditional hybrid ASR systems.
1. Introduction
The paper addresses the complexity of modular ASR by extending joint CTC-attention end-to-end recognition with joint decoding, a deep CNN encoder, and an RNN-LM. These extensions achieve superior performance to several state-of-the-art hybrid systems on spontaneous Japanese and Mandarin Chinese tasks.
- Motivation: Traditional ASR combines acoustic, lexicon, and language models with pronunciation dictionaries, text preprocessing, and WFST decoding, making new systems difficult for non-experts to develop.These components also require linguistic knowledge and language-specific resources.
- Motivation: End-to-end ASR simplifies the modular architecture into a single network trained from paired acoustic and language data without linguistic knowledge.The approach potentially enables building ASR systems without expert knowledge.
- Contributions: The paper extends prior joint CTC-attention ASR by using CTC probabilities during decoding through rescoring and one-pass methods.The prior work used the CTC objective only for training; the new decoding methods combine CTC and attention probabilities.
- Contributions: The encoder incorporates a VGG-based deep CNN consisting of 4 convolutional and 2 max-pooling layers.This deep CNN is introduced as an encoder extension for end-to-end ASR.
- Contributions: The model adds an RNN-LM in parallel with the attention decoder, trained separately or jointly on character sequences.This language model is one of the paper’s introduced extensions.
- Results: The extended joint CTC-attention system achieves performance superior to several state-of-the-art hybrid ASR systems on spontaneous Japanese and Mandarin Chinese tasks.The experiments evaluate the efficacy of the introduced techniques and compare the resulting model with traditional hybrid systems.
2. Joint CTC-attention
The joint CTC-attention framework combines CTC’s monotonic alignment benefits with attention-based sequence modeling, using shared encoder representations and jointly optimized objectives.
- Connectionist Temporal Classification (CTC): CTC monotonically maps an input sequence to a shorter output sequence and introduces framewise labels with an additional blank symbol.Its posterior uses conditional independence assumptions and can be marginalized efficiently with dynamic programming.
- Connectionist Temporal Classification (CTC): CTC models framewise posterior distributions conditioned on all inputs using BLSTM hidden representations and a softmax output over characters plus blank.The output dimension is |U|+1 because of the blank symbol.
- Attention-based encoder-decoder: Attention-based encoding estimates p(C|X) through the chain rule without CTC’s conditional independence assumptions, conditioning predictions on previous outputs.Content-based attention forms letter-wise hidden vectors from weighted encoder states before recurrent decoding.
- Joint CTC-attention: The framework uses both CTC and attention during training, sharing a BLSTM encoder between the CTC and attention decoder networks.CTC’s forward-backward algorithm encourages monotonic alignments and helps estimate alignments in long sequences.
- Multi-task learning: The joint objective is a logarithmic linear combination of the CTC and attention objectives, while CTC training reduces irregular alignments and improves performance.The earlier multi-task formulation used CTC as an auxiliary task for training the attention-model encoder.
3. Extended joint CTC-attention
The paper extends joint CTC-attention ASR with CTC-aware decoding, a deep VGG-based CNN encoder, and an integrated character-level RNN language model. These components combine CTC, attention, and language-model information during end-to-end recognition.
- Overview: Three extensions comprise joint decoding, a deep CNN encoder, and an RNN-LM network.The extended architecture combines these components within the joint CTC-attention ASR framework.
- Joint decoding: CTC-aware decoding combines attention and CTC probabilities through rescoring or one-pass beam search.Rescoring evaluates complete attention-generated hypotheses with CTC, whereas one-pass decoding scores partial hypotheses using CTC prefix probabilities.
- Joint decoding: CTC prefix probabilities aggregate the probabilities of all label sequences sharing a partial hypothesis prefix.The CTC score is computed efficiently by retaining forward probabilities over input frames for each partial hypothesis and combining it with the attention score using λ.
- Deep CNN encoder: The shared encoder uses initial VGG layers followed by BLSTM layers, with a six-layer CNN architecture preceding recurrent processing.The CNN uses convolution and max-pooling layers; the input channels contain spectral, delta, and delta-delta features.
- RNN-LM integration: A character-level RNN-LM runs in parallel with the attention decoder and can be trained separately or jointly.Its probabilities are combined with decoder logits or pre-activations, allowing language-model states to depend purely on output-label sequences.
4. Experiments
Experiments evaluate the extended joint CTC-attention ASR on Japanese lecture speech and Mandarin Chinese conversational telephone speech, using character error rate (CER). The proposed system improves over attention and prior joint-training baselines and surpasses reported hybrid-system results on both tasks.
- Datasets: The evaluation uses CSJ Japanese lecture speech and HKUST MTS Mandarin Chinese conversational telephone speech benchmarks.CSJ contains 581 hours of training data and 5 hours of evaluation data across three evaluation sets.
- Configuration: The encoders use 80 mel-scale filterbank coefficients with pitch features, while the CNN-BLSTM variant additionally uses delta and delta-delta features.The encoder is a 4-layer BLSTM with 320 cells in each layer and direction, and subsampling reduces utterance length by a factor of 4.
- Results: Joint decoding improves CER over the baseline attention model and prior multi-task-learning work, especially with one-pass decoding and RNN-LM integration.In MTS, joint training further improved performance to 32.1% CER.
- Results: The extended joint CTC-attention system achieves lower CER than reported hybrid-system results on the CSJ task.The comparison uses conventional systems trained with linguistic resources, including Kaldi and syllable-based CTC systems.
- Results: The final MTS model with the VGG network and RNN-LM achieves 28.0% CER without linguistic resources and defeats state-of-the-art systems, including lattice-free MMI methods.Speed perturbation generated additional training data by scaling audio lengths by 0.9 and 1.1.
5. Conclusion
The paper proposes joint CTC-attention decoding with an RNN language model and a deep CNN encoder for end-to-end ASR. This approach improves end-to-end ASR without linguistic resources required by conventional Mandarin Chinese and Japanese systems.
- The approach jointly integrates CTC-attention decoding and an RNN language model for end-to-end ASR.The paper presents this integration as a central contribution.
- A deep CNN encoder is used to improve extracted acoustic features.
- The combined approach significantly improves the current best end-to-end ASR system without morphological analyzers or pronunciation dictionaries.These resources are described as essential components of conventional Mandarin Chinese and Japanese ASR systems.