Source-linked AI summary
CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
Linhao Dong, Bo Xu
TL;DR
ASR attention models face difficulties with streaming recognition, acoustic timestamping, and unnecessary computation, motivating a more efficient monotonic alignment. The paper proposes Continuous Integrate-and-Fire (CIF), which accumulates weighted encoder states to locate boundaries and emit label representations, with supporting strategies for training and inference. CIF achieves competitive ASR results, including 2.86% WER on Librispeech test-clean and a new state-of-the-art result on the HKUST Mandarin telephone benchmark.
Problem
Attention-based ASR requires the entire encoded sequence, lacks frame-synchronous timestamps, and performs unnecessary computation on irrelevant encoder steps.
Method
CIF is a soft, monotonic encoder-decoder alignment that accumulates weighted encoder states to a threshold, locates acoustic boundaries, and shares boundary-step information between adjacent labels.
Results
2.86% WER on Librispeech test-clean, while CIF also creates a new state-of-the-art result on the HKUST Mandarin telephone benchmark.
Takeaways & Limitations
CIF supports concise alignment calculation, online recognition, acoustic boundary positioning, and application across ASR datasets with different languages and speech types.
Abstract
from arXiv · showhide
In this paper, we propose a novel soft and monotonic alignment mechanism used for sequence transduction. It is inspired by the integrate-and-fire model in spiking neural networks and employed in the encoder-decoder framework consists of continuous functions, thus being named as: Continuous Integrate-and-Fire (CIF). Applied to the ASR task, CIF not only shows a concise calculation, but also supports online recognition and acoustic boundary positioning, thus suitable for various ASR scenarios. Several support strategies are also proposed to alleviate the unique problems of CIF-based model. With the joint action of these methods, the CIF-based model shows competitive performance. Notably, it achieves a word error rate (WER) of 2.86% on the test-clean of Librispeech and creates new state-of-the-art result on Mandarin telephone ASR benchmark.
1. INTRODUCTION
The paper addresses limitations of attention-based ASR by introducing a soft, monotonic alignment inspired by integrate-and-fire dynamics. CIF accumulates weighted acoustic information to locate boundaries, while supporting strategies improve label-count matching and inference handling.
- Core motivation and approach: Attention-based ASR requires the entire encoded sequence, lacks frame-synchronous timestamps, and performs unnecessary computation on acoustically irrelevant encoder steps.
- Core motivation and approach: The proposed alignment is designed for efficient monotonic calculation, online recognition, and acoustic boundary positioning in ASR.
- Core motivation and approach: Integrate-and-fire supplies the inspiration because it monotonically integrates input until a threshold is reached, then emits an event that can mark an acoustic boundary.
- Core motivation and approach: CIF forwardly accumulates weighted encoder information until a threshold identifies an acoustic boundary, then shares the boundary step between adjacent labels.The mechanism uses continuous functions so it remains trainable by back-propagation.
- Supporting strategies: CIF-based models use scaling, quantity loss, and tail handling to address unequal label lengths, supervise label quantity, and process residual inference information.The paper reports that these strategies jointly improve performance across datasets covering different languages and speech types.
2. RELATION TO PRIOR WORK
CIF differs from prior soft and monotonic alignment methods by learning directly from acoustic data without a prescribed attention window or decoder-state prediction. It also combines boundary locating and acoustic integration in one calculation rather than using separate alignment stages or CTC-based pre-partitioning.
- Model architecture: The CIF-based ASR architecture uses a training-stage path distinct from the inference-stage path through a switch before the CIF module.
- Comparison with prior alignment methods: CIF avoids predefined Gaussian or heuristic attention windows and does not use decoder state, encouraging alignment patterns to be learned from acoustic data.
- Comparison with prior alignment methods: CIF locates boundaries and integrates acoustic information simultaneously, unlike methods that first apply hard monotonic attention and then soft attention.
- Comparison with ACS: CIF differs from ACS by processing firing phenomena at finer time granularity rather than dynamically selecting a block of frames for each linguistic output.
3. METHOD
CIF provides soft, monotonic alignment by accumulating weighted encoder states until an acoustic boundary is reached, then producing integrated embeddings for decoding. Supporting strategies address length mismatch, label-quantity prediction, residual tail information, and model-structure choices.
- 3.1. Continuous Integrate-and-Fire: CIF accumulates weighted encoder states until a threshold identifies an acoustic boundary, splitting boundary-crossing weights between adjacent labels to produce integrated embeddings.The calculation takes encoder outputs and corresponding weights as input and returns integrated embeddings for output labels.
- 3.2. Supporting Strategies for CIF-based Model: Scaling weights to sum to the target length makes the number of integrated embeddings match the targets for one-to-one cross-entropy training.The strategy multiplies the predicted weights by a scalar whose sum equals the target length.
- 3.2. Supporting Strategies for CIF-based Model: Quantity loss supervises the predicted number of integrated embeddings and promotes acoustic boundary positioning while reducing degradation when scaling is removed during inference.It encourages the embedding quantity to approach the target-label quantity.
- 3.2. Supporting Strategies for CIF-based Model: Tail handling performs an additional firing when residual weight exceeds 0.5 and adds an <EOS> target to reduce incomplete outputs from unused tail information.The method addresses residual information that is insufficient to trigger a normal firing.
- 3.3. Model Structure: The model supports autoregressive decoding with previous-label information and non-autoregressive decoding that can parallelize offline inference from precomputed integrated embeddings.The non-autoregressive decoder directly processes each currently integrated embedding.
- 3.3. Model Structure: Training combines cross-entropy with optional quantity and auxiliary losses, while inference uses beam search followed by second-pass language-model rescoring.The loss weights are tunable, and the final transcript is selected after language-model rescoring.
4. EXPERIMENTAL SETUP
The experiments evaluate CIF on Librispeech, AISHELL-2, and HKUST, covering English and Mandarin read and telephone speech. The setup uses standardized feature extraction and augmentation, dataset-specific model configurations, and beam-search inference with language-model rescoring.
- 4. EXPERIMENTAL SETUP: Experiments use Librispeech, AISHELL-2, and HKUST, spanning English read speech, Mandarin read speech, and Mandarin telephone ASR.Librispeech training uses 960 hours, with separate development, test, and language-model data.
- 4. EXPERIMENTAL SETUP: All datasets use a shared feature-extraction setup, fixed ±10% speed perturbation, and specified frequency and time masking except for the Librispeech base model.Librispeech uses 3722 BPE word pieces plus blank and end-of-sentence special labels.
- 4. EXPERIMENTAL SETUP: Model configurations vary by dataset and Librispeech model size, with SAN dimensions and encoder settings specified separately for Mandarin, base, and big models.The encoder uses the cited pyramid configuration, while chunk-hopping supports forward encoding for online recognition.
- 4. EXPERIMENTAL SETUP: Training applies SAN dropout, label smoothing, and Mandarin scheduled sampling, while inference uses beam search of size 10 and dataset-specific language-model weights.The language-model rescoring hyperparameter γ differs across datasets and model settings.
- 4. EXPERIMENTAL SETUP: Aligned CIF boundary results are made available through an online visualization.The paper provides a dedicated web page for inspecting the located boundaries.
5. RESULTS
CIF-based models achieve competitive results across Librispeech, AISHELL-2, and HKUST, including strong offline and online recognition performance. Ablations show that auto-regression and the quantity loss are especially important, while the proposed support strategies improve performance.
- 5.1. Results on Librispeech: 2.86% WER is achieved on Librispeech test-clean, with 8.08% on test-other, outperforming other soft and monotonic models and matching or surpassing most published end-to-end results.
- 5.1. Results on Librispeech: 3.25% WER on test-clean and 9.63% on test-other show that chunk-hopping enables the trained big model to support online Librispeech recognition.
- 5.2. Ablation Study on Librispeech: Ablating auto-regression causes the largest performance degradation, while scaling, quantity loss, and tail handling each provide clear improvements.
- 5.2. Ablation Study on Librispeech: The quantity loss is the most important support strategy in the ablation, and removing it causes the largest performance loss and learning instability.
- 5.3. Results on AISHELL-2: On AISHELL-2, CIF performs competitively across all test sets and significantly improves on the Chain model results.
- 5.4. Results on HKUST: 23.09% CER is reported for CIF + SAN and 23.60% with online chunk-hopping on HKUST, establishing a new state-of-the-art result on Mandarin telephone ASR.
6. DISCUSSION AND CONCLUSION
CIF models integrate-and-fire dynamics with continuous values and supports concise acoustic embedding extraction for ASR and other sequence transduction tasks.
- CIF simulates integrate-and-fire dynamics by continuously processing encoder-step information and modeling changes in accumulated weights.Unlike the discrete IF model, CIF treats information within an encoder step as a whole and uses continuous values.
- CIF can extract acoustic embeddings concisely and support sequence transduction tasks such as TTS with suitable encoding resolution.
- CIF-based models achieve competitive ASR benchmark performance, while larger datasets and additional tasks remain future evaluation targets.