Source-linked AI summary
Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, Tetsunori Kobayashi
TL;DR
End-to-end ASR needs faster decoding than autoregressive models provide, while CTC can lose accuracy because of conditional independence. Mask CTC combines CTC with mask prediction, initializing outputs from greedy CTC and refining low-confidence tokens using surrounding high-confidence context. Across speech-recognition tasks, it outperforms standard CTC, remains fast, and approaches autoregressive performance.
Problem
Autoregressive ASR requires one decoding iteration per output token, while CTC’s conditional-independence assumption can reduce recognition accuracy.
Method
Mask CTC jointly trains a Transformer encoder-decoder with CTC and mask prediction, then iteratively refines low-confidence greedy CTC tokens conditioned on unmasked tokens and speech features.
Results
Mask CTC outperformed standard CTC while maintaining fast decoding and approached autoregressive results, matching or exceeding them especially on CSJ.
Takeaways & Limitations
Refining CTC outputs with mask prediction provides a non-autoregressive ASR framework that combines fast decoding with improved recognition accuracy.
Takeaways & Limitations
The framework was especially effective for languages with small-unit tokens, leaving its effectiveness with BPEs for such languages as future work.
Abstract
from arXiv · showhide
We present Mask CTC, a novel non-autoregressive end-to-end automatic speech recognition (ASR) framework, which generates a sequence by refining outputs of the connectionist temporal classification (CTC). Neural sequence-to-sequence models are usually \textit{autoregressive}: each output token is generated by conditioning on previously generated tokens, at the cost of requiring as many iterations as the output length. On the other hand, non-autoregressive models can simultaneously generate tokens within a constant number of iterations, which results in significant inference time reduction and better suits end-to-end ASR model for real-world scenarios. In this work, Mask CTC model is trained using a Transformer encoder-decoder with joint training of mask prediction and CTC. During inference, the target sequence is initialized with the greedy CTC outputs and low-confidence tokens are masked based on the CTC probabilities. Based on the conditional dependence between output tokens, these masked low-confidence tokens are then predicted conditioning on the high-confidence tokens. Experimental results on different speech recognition tasks show that Mask CTC outperforms the standard CTC model (e.g., 17.9% -> 12.1% WER on WSJ) and approaches the autoregressive model, requiring much less inference time using CPUs (0.07 RTF in Python implementation). All of our codes will be publicly available.
1. Introduction
Non-autoregressive ASR reduces decoding iterations by generating tokens in parallel, but CTC’s conditional-independence assumption can hurt recognition accuracy. Mask CTC addresses this trade-off by refining greedy CTC outputs with masked prediction conditioned on high-confidence tokens.
- 1. Introduction: Autoregressive decoding requires L iterations for an L-token sequence, whereas non-autoregressive models use K iterations with K ≪ L.This makes non-autoregressive generation attractive for reducing inference time.
- 1. Introduction: Non-autoregressive sequence generation has been explored through iterative refinement, insertion or editing, masked language modeling, and generative-flow approaches.These approaches seek alternatives to fully autoregressive decoding.
- 1. Introduction: CTC efficiently models speech-to-token alignment but its strong conditional-independence assumption causes poorer performance than autoregressive models.CTC represents alignment through frame-wise latent variables and dynamic programming.
- 1. Introduction: Mask CTC jointly trains a Transformer encoder-decoder with CTC and mask-predict objectives.At inference, greedy CTC outputs initialize the sequence, and low-confidence tokens are masked and predicted using past and future high-confidence context.
2. Mask CTC framework
Mask CTC combines CTC with masked-token prediction to generate speech transcriptions non-autoregressively, refining greedy CTC outputs through bidirectional context. Its decoding avoids separately predicting output length and progressively improves uncertain tokens while retaining constant-iteration decoding options.
- Mask CTC decoding: Masked tokens are predicted from the input speech and high-confidence tokens across both past and future context.The decoder attends to the entire sequence, unlike autoregressive decoding that conditions only on previously generated tokens.
- Joint CTC-CMLM training: Mask CTC jointly trains a Transformer encoder-decoder with CTC and conditional masked language modeling objectives.Joint training addresses skipping and repeating errors observed when applying the original masked language model directly to non-autoregressive ASR.
- Mask CTC decoding: Mask CTC uses CTC outputs to initialize sequence length, avoiding a separate output-length prediction step that is challenging for speech recognition.The CTC sequence is obtained through a single encoder calculation and then refined by the decoder.
- Mask CTC decoding: During inference, greedy CTC outputs initialize the target sequence, and low-confidence tokens are masked using approximate frame-level CTC probabilities.A threshold determines which tokens are masked before refinement.
- Mask CTC decoding: Iterative easy-first decoding can control computation in a constant K iterations by predicting the highest-confidence masked tokens at each step.With C = [L/K], the number of iterations remains independent of target length.
3. Experiments
Experiments across English, Italian, and Japanese evaluated Mask CTC against CTC and autoregressive models using error rates and decoding speed. Mask CTC improved over CTC across tasks, approached autoregressive performance in several settings, and retained fast decoding, while its ability to recover errors depended on the error type and tokenization.
- WSJ results: Mask CTC improved WSJ performance over the standard CTC model, approached autoregressive results, and was at most 116 times faster in decoding.Refinement improved performance progressively with more decoding iterations, while the reported speed used CPU real-time factors.
- WSJ results: Mask CTC recovered substitution errors caused by incomplete word spelling by modeling conditional dependence between characters.The example decoding process initializes from CTC outputs, masks low-confidence tokens, and iteratively predicts them from unmasked tokens.
- Voxforge results: On Voxforge, Mask CTC yielded better scores than standard CTC, supporting adoption across languages with relatively small training datasets.The Voxforge evaluation used Italian speech and character-based tokenization.
- CSJ results: On CSJ, Mask CTC was close to or better than the autoregressive model in CER and clearly improved sentence-level error rates across evaluation sets.Character-level gains over simple CTC were smaller, while sentence-level improvements were clear.
- Cross-task findings: Mask CTC was especially effective for languages whose tokens have small units, including Latin-alphabet and other phonemic scripts.The authors identify BPEs for such languages as future work.
4. Conclusions
Mask CTC refines greedy CTC outputs through mask prediction in a non-autoregressive ASR framework, improving recognition while preserving fast decoding. It approaches autoregressive performance, with comparable or better results on CSJ.
- Mask CTC refines greedy CTC outputs by iteratively predicting low-confidence masked tokens from unmasked tokens and speech features.
- Mask CTC outperformed the standard CTC model while maintaining fast decoding speed.
- Mask CTC approached autoregressive-model results, with comparable or better performance on CSJ.
- The authors identify reducing the training–inference masking mismatch as future work.Training uses random masking, whereas inference uses CTC outputs.
- The authors also plan to integrate external language models such as BERT into Mask CTC.