Source-linked AI summary
Intermediate Loss Regularization for CTC-based Speech Recognition
Jaesong Lee, Shinji Watanabe
TL;DR
CTC ASR can be efficient but often trails encoder-decoder systems, motivating methods that improve performance without restoring autoregressive decoding or external language models. The paper adds an intermediate CTC loss, combines it with stochastic depth, and reports WER 9.9% on WSJ and CER 5.2% on AISHELL-1 using greedy CTC decoding. The intermediate loss improves Transformer and Conformer architectures with small training overhead and no test-time overhead.
Problem
CTC ASR is computationally efficient but often underperforms encoder-decoder models because of its conditional-independence assumption, while language models and beam search add computational cost.
Method
The paper attaches an auxiliary CTC loss to an intermediate encoder representation and combines it with stochastic depth training for CTC-based Transformer and Conformer models.
Results
The combined method reaches WER 9.9% on WSJ and CER 5.2% on AISHELL-1 using CTC greedy decoding without an external language model.
Takeaways & Limitations
Intermediate CTC loss improves Transformer and Conformer CTC models with small training overhead and no test-time overhead, while stochastic depth further improves training.
Takeaways & Limitations
The intermediate layer is mainly placed at ⌊L/2⌋ as a safe choice between lower and higher layers.
Abstract
from arXiv · showhide
We present a simple and efficient auxiliary loss function for automatic speech recognition (ASR) based on the connectionist temporal classification (CTC) objective. The proposed objective, an intermediate CTC loss, is attached to an intermediate layer in the CTC encoder network. This intermediate CTC loss well regularizes CTC training and improves the performance requiring only small modification of the code and small and no overhead during training and inference, respectively. In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network. We evaluate the proposed method on various corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy search without a language model. Especially, the AISHELL-1 task is comparable to other state-of-the-art ASR systems based on auto-regressive decoder with beam search.
1. INTRODUCTION
CTC offers compact, fast, non-autoregressive ASR, but its conditional-independence assumption often weakens performance. The paper introduces intermediate CTC loss and combines it with stochastic depth to improve CTC models with minimal training and no inference overhead.
- CTC avoids a separate decoder and supports fast parallel greedy decoding, enabling compact and efficient ASR models.
- CTC performance is often weaker than encoder-decoder models because of its strong conditional-independence assumption.
- Intermediate CTC loss applies an auxiliary CTC objective to an intermediate representation, adding very small training overhead and no inference overhead.
- The method improves Transformer and Conformer models and can be combined with stochastic depth for further enhancement.
- WER 9.9% on WSJ and CER 5.2% on AISHELL-1 are achieved using CTC modeling with greedy decoding only.
2. ARCHITECTURE
The architecture uses multi-layer Transformer or Conformer encoders with CTC training and greedy decoding. Transformer captures global representations through self-attention, while Conformer combines global self-attention with local convolution; stochastic depth regularizes deep residual networks by randomly skipping layers.
- Connectionist Temporal Classification: CTC computes target-sequence likelihood by summing over compatible alignments of length T, including the special blank token.
- Connectionist Temporal Classification: CTC factorizes alignment probabilities under a conditional-independence assumption, then minimizes the induced negative log-likelihood during training.
- Connectionist Temporal Classification: Greedy search selects the most probable alignment at test time for fast inference.
- Encoder: Transformer encoder layers use self-attention for global representations together with layer normalization, residual connections, and feed-forward layers.
- Encoder: Conformer combines Transformer and convolution layers to learn global and local representations efficiently.
- Stochastic Depth: Stochastic depth randomly skips residual-network layers during training and uses all layers during testing.
3. INTERMEDIATE CTC LOSS
Intermediate CTC regularizes the lower layers by applying a CTC loss to an intermediate sub-model, while retaining the full model’s final decoding path. The method adds little training overhead, no inference overhead, and can be combined with stochastic depth and extended to related configurations.
- Intermediate CTC loss: Skipping all layers after an intermediate layer creates a sub-model that relies on lower layers and regularizes the lower part of the full model.The paper mainly uses the middle layer, ⌊L/2⌋, as the intermediate position.
- Intermediate CTC loss: The intermediate representation is naturally available during full-model computation, so its CTC loss can be added with very small training overhead.The training objective combines the full-model and intermediate losses; experiments use w = 0.3.
- Intermediate CTC loss: During testing, decoding uses only the final representation, so the intermediate prediction adds no inference overhead.The intermediate prediction is not used during testing.
- Combination with stochastic depth: Intermediate CTC can be combined with stochastic depth because the two methods are expected to regularize lower and higher layers, respectively.The paper reports that experiments evaluate this combined approach.
- Position and loss variants: The method supports lower intermediate positions, multiple sub-models, random positions, and a stochastic loss variant.The deterministic loss is argued to be more effective than the stochastic variant, and experiments report that the stochastic variant is less effective.
- Related methods and applications: Unlike hierarchical CTC, intermediate CTC requires no additional low-level labels or tokenization and no additional network.The paper also applies the method to Mask CTC by improving the encoder’s intermediate CTC probability.
4. EXPERIMENTS
Experiments evaluate intermediate CTC across Transformer and Conformer architectures, regularization combinations, design variants, and Mask CTC. Intermediate CTC consistently improves CTC performance, while combining it with stochastic depth yields strong results without language models or beam search.
- Experimental setup: Experiments use WSJ, TED-LIUM2, and AISHELL-1 with Transformer and Conformer models, comparing baseline, intermediate CTC, stochastic depth, and their combination.Evaluation uses WER for WSJ and TED-LIUM2 and CER for AISHELL-1; decoding is greedy CTC without external language models or beam search.
- Transformer results: Intermediate CTC improves every experiment over the baseline, while combining it with stochastic depth performs better than either method alone.The authors conclude that the two methods have complementary effects.
- Transformer results: WER improved from 21.1% to 18.3% on WSJ when intermediate CTC was applied to a 6-layer Transformer.The result suggests that intermediate CTC remains beneficial for smaller networks.
- Conformer results: Intermediate CTC consistently improves Conformer over baseline, while stochastic depth improves WSJ and AISHELL-1 but not TED-LIUM2.The Conformer experiments use a 12-layer model.
- Headline results: The combined method achieves WER 9.9% on WSJ and CER 5.2% on AISHELL-1 using CTC with greedy decoding.These results use neither an external language model nor beam search.
- Intermediate-loss design: Design-variant results favor the default intermediate-loss design: multiple positions help only the 48-layer model, and the stochastic variant is worse than the deterministic one.The stochastic variant remains better than no regularization, while the overall result is described as mixed across depths.
- Mask CTC: In Mask CTC, intermediate CTC reduces CTC-prediction WER from 13.5% to 11.6% and overall WER from 12.9% to 11.3%.The overall result is lower than Align-Refine's 11.4% on WSJ.
5. CONCLUSION
The paper introduces intermediate CTC loss as a lightweight auxiliary task for CTC-based speech recognition. It improves Transformer and Conformer models, and its combination with stochastic depth reaches strong WSJ and AISHELL-1 results without autoregressive decoding or an external language model.
- Conclusion: Intermediate CTC loss is an easy-to-implement auxiliary task with small training overhead and no test-time overhead.It is applied to both Transformer and Conformer architectures.
- Conclusion: Combining intermediate CTC with stochastic depth further improves training, reaching WER 9.9% on WSJ and CER 5.2% on AISHELL-1.The results use neither an autoregressive decoder nor an external language model.