Source-linked AI summary
Improving CTC-based speech recognition via knowledge transferring from pre-trained language models
Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, Pengyuan Zhang
TL;DR
CTC-based speech recognition decodes quickly but its conditional-independence assumption limits text-context use, and external autoregressive language models reduce that speed advantage. The paper transfers pre-trained language-model knowledge to CTC systems through BERT-based representation learning and GPT2-based joint classification learning, reporting a 4.2% CER on AISHELL-1 and a 16.1% relative reduction without an external LM.
Problem
CTC’s conditional-independence assumption limits its use of text-modal contextual information, while using an external autoregressive LM sacrifices non-autoregressive decoding speed.
Source: §1 · p. 1Method
The paper transfers pre-trained LM knowledge to CTC using BERT representations as an auxiliary target or GPT2 text modeling in a hybrid CTC/attention architecture.
Source: §1 · p. 1Results
Takeaways & Limitations
The proposed CTC system without an external LM can surpass the wav2vec2.0 CTC baseline that uses an autoregressive LM.
Source: §4.3 · p. 3KT-RL-CIF training takes longer than KT-RL-ATT because of its serial structure.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
Recently, end-to-end automatic speech recognition models based on connectionist temporal classification (CTC) have achieved impressive results, especially when fine-tuned from wav2vec2.0 models. Due to the conditional independence assumption, CTC-based models are always weaker than attention-based encoder-decoder models and require the assistance of external language models (LMs). To solve this issue, we propose two knowledge transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to improve CTC-based models. The first method is based on representation learning, in which the CTC-based models use the representation produced by BERT as an auxiliary learning target. The second method is based on joint classification learning, which combines GPT2 for text modeling with a hybrid CTC/attention architecture. Experiment on AISHELL-1 corpus yields a character error rate (CER) of 4.2% on the test set. When compared to the vanilla CTC-based models fine-tuned from the wav2vec2.0 models, our knowledge transferring method reduces CER by 16.1% relatively without external LMs.
1. INTRODUCTION
CTC models decode quickly in parallel but their conditional-independence assumption limits use of text context, while external autoregressive language models reduce that speed advantage. The paper transfers pre-trained language-model knowledge to CTC systems and reports a 4.2% CER on AISHELL-1.
1. INTRODUCTION
- CTC’s conditional-independence assumption limits its use of text context; adding an external autoregressive LM can help but sacrifices the speed advantage of non-autoregressive decoding.The paper seeks to improve CTC itself without adding inference-time parameters.Source: §1 · p. 1
- Mask CTC and Imputer refine CTC predictions within non-autoregressive systems but require extra computation and parameters.Source: §1 · p. 1
- 4.2% CER on the AISHELL-1 test set, with a 16.1% relative CER reduction over the wav2vec2.0 CTC baseline without an external LM.The paper proposes BERT-based representation learning and GPT2-based joint classification learning.Source: §1 · p. 1
- The BERT-based method uses BERT representations as an auxiliary learning target, while the GPT2-based method combines text modeling with a hybrid CTC/attention architecture.The paper reports that the BERT-based method performs better, transferring bidirectional contextual knowledge.Source: §1 · p. 1
2. RELATED WORKS
Related work describes self-supervised speech representations and language-model knowledge transfer to speech recognition. Prior BERT transfer methods target sequence-to-sequence systems, while this paper focuses on CTC systems.
2. RELATED WORKS
- BERT and GPT2 are described as distinct language models: BERT uses masked-learning-pretrained Transformer encoders, whereas GPT2 uses unidirectional Transformer decoders.Source: §2 · p. 1
- wav2vec learns raw-speech representations through self-supervised context prediction, and wav2vec2.0 combines contextual representations with discrete speech-unit learning.The passage states that wav2vec2.0 can outperform prior work with less labeled data.Source: §2 · pp. 1–2
- Prior work transfers BERT knowledge to AED sequence-to-sequence ASR, but the paper frames CTC as a distinct target because it lacks an autoregressive decoder for text modality.The cited AED transfer approaches use soft labels, multi-objective training, or transfer of BERT parameters.Source: §2 · p. 2
3. PROPOSED METHOD
The paper proposes two wav2vec2.0-based knowledge-transfer methods for CTC recognition: BERT-based representation learning and GPT2-based joint classification. Representation learning aligns speech and linguistic representations, while joint classification combines GPT2 text context with speech features.
3. PROPOSED METHOD
- KT-RL transfers BERT knowledge through representation learning, while KT-CL uses GPT2 for joint classification learning under the wav2vec2.0 framework.The BERT and GPT2 parameters are fixed in their respective methods.Source: §3 · p. 2, §3.1 · p. 2, §3.2 · p. 2
3.1. Knowledge transferring based on representation learning
- KT-RL extracts linguistic representations aligned to the target sequence and trains them toward contextual embeddings produced by BERT.BERT parameters remain fixed during this process.Source: §3.1 · p. 2, §3.1.1 · p. 2
3.1.1. KT-RL based on CIF (KT-RL-CIF)
- KT-RL-CIF derives frame weights with a fully connected layer and sigmoid, then accumulates them to fire monotonic-aligned linguistic representations.The CIF process divides weights when their accumulated value passes 1, keeping the number of output representations consistent with the target length.Source: §3.1.1 · p. 2
3.1.2. KT-RL based on attention (KT-RL-ATT)
- KT-RL-ATT uses target-length positional embeddings and parallel multi-head cross-attention to extract linguistic representations.Its cosine and multitask losses follow the same formulation as the CIF-based method.Source: §3.1.2 · p. 2
3.2. Knowledge transferring based on classification learning
- KT-CL combines GPT2 representations of previous tokens with speech-encoder outputs through masked cross-attention, then trains with joint classification.The subsequent mask prevents access to future tokens.Source: §3.2 · pp. 2–3
4. EXPERIMENTS
The experiments evaluate knowledge transfer on AISHELL-1 and compare representation-learning and joint-classification variants against CTC baselines. They report lower test CER without an external LM and examine accuracy, training-time, and language-model trade-offs.
4.1. Corpus
- AISHELL-1 evaluation uses about 120,000 training, 14,000 development, and 7,000 test utterances, with a wav2vec2.0 model pre-trained on unlabeled AISHELL-2 speech.Source: §4.1 · p. 3
4.2. Model descriptions
- The models use raw speech, 4,233 output units, fixed pre-trained BERT and GPT2 parameters, and CTC-only decoding with beam size 10.Training used 20 epochs with early stopping; external-LM decoding was also evaluated.Source: §4.2 · p. 3
4.3. Experimental results
- Without an external AR language model, the proposed non-autoregressive system surpasses the wav2vec2.0 CTC baseline using an AR language model, while training for 20 rather than 50 epochs saves training costs.Source: §4.3 · p. 3
- 14.3% and 16.1% relative test-set CER reductions with and without an external LM, respectively, versus the vanilla wav2vec2.0 CTC baseline.The comparison is on AISHELL-1.Source: Table 1 · p. 3, §4.3 · p. 3
4.4. Ablation studies on KT-RL
- KT-RL-CIF outperforms KT-RL-ATT, and cosine embedding loss outperforms MSE; the authors attribute CIF’s advantage to monotonic alignment and more accurate knowledge transfer.The results also imply that vector angle may matter more than norm during representation learning.Source: §4.4 · pp. 3–4
- KT-RL-CIF takes longer to train because of its serial structure, whereas KT-RL-ATT has greater parallel efficiency and reduces training time.The reported comparison measures forward, backward, and total time per iteration.Source: §4.4 · p. 4, Table 3 · p. 4
4.5. Ablation studies on KT-CL
- KT-CL further improves on vanilla CTC/attention training, but KT-RL outperforms KT-CL by transferring bidirectional rather than unidirectional contextual knowledge.Joint CTC/attention decoding can achieve higher accuracy, with slower inference.Source: §4.5 · p. 4
4.6. Ablation studies on pre-trained LM
- KT-RL-CIF with BERT outperforms its GPT2 version, while unidirectional BERT is only slightly worse than GPT2 for KT-CL; KT-RL outperforms KT-CL with either LM.Source: §4.6 · p. 4
5. CONCLUSION
The paper transfers pre-trained language-model context to CTC-based ASR through representation learning and joint classification learning. On AISHELL-1, the methods achieve 4.2% test CER and a 16.1% relative CER reduction without an external LM.
5. CONCLUSION
- 4.2% test-set CER and a 16.1% relative CER reduction over the wav2vec2.0 CTC baseline without an external LM are reported on AISHELL-1.The paper also reports that its method surpasses AED-based ASR systems on the AISHELL-1 benchmark.Source: §5 · p. 4
- The BERT-based representation-learning method uses extracted representations as auxiliary targets to distill bidirectional contextual knowledge into ASR systems.Source: §5 · p. 4
- The GPT2-based joint-classification method combines text modeling with a hybrid CTC/attention structure to use unidirectional contextual knowledge.Source: §5 · p. 4