Source-linked AI summary
E-Branchformer: Branchformer with Enhanced merging for speech recognition
Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu J. Han, Shinji Watanabe
TL;DR
Existing ASR architectures combine convolution and self-attention, but alternatives have struggled to match Conformer-level performance. E-Branchformer improves Branchformer’s merging mechanism and adds point-wise modules, achieving 1.81% and 3.65% WER on LibriSpeech test-clean and test-other without external data.
Problem
Existing convolution–self-attention integration approaches have not consistently matched Conformer’s ASR performance.
Method
E-Branchformer enhances Branchformer with an improved merge module combining self-attention and convolution sequentially and in parallel, plus additional point-wise modules.
Results
1.81% and 3.65% WER on LibriSpeech test-clean and test-other set a new state of the art without external data.
Takeaways & Limitations
E-Branchformer outperforms both Conformer and Branchformer on LibriSpeech test sets without using external data.
Abstract
from arXiv · showhide
Conformer, combining convolution and self-attention sequentially to capture both local and global information, has shown remarkable performance and is currently regarded as the state-of-the-art for automatic speech recognition (ASR). Several other studies have explored integrating convolution and self-attention but they have not managed to match Conformer's performance. The recently introduced Branchformer achieves comparable performance to Conformer by using dedicated branches of convolution and self-attention and merging local and global context from each branch. In this paper, we propose E-Branchformer, which enhances Branchformer by applying an effective merging method and stacking additional point-wise modules. E-Branchformer sets new state-of-the-art word error rates (WERs) 1.81% and 3.65% on LibriSpeech test-clean and test-other sets without using any external training data.
1. INTRODUCTION
E-Branchformer extends Branchformer by improving how local and global branches are merged, targeting stronger end-to-end ASR encoders. The paper reports improved baselines, extensive merge studies, and new LibriSpeech state-of-the-art results without external data.
- Motivation and approach: E-Branchformer enhances Branchformer by improving the merging mechanism for convolutional and self-attention branches.The enhanced design studies ways to combine local and global information more effectively.
- Contributions: The paper introduces a Conformer baseline whose attention-based encoder-decoder accuracy matches Google’s Conformer.
- Contributions: 0.2% and 0.5% absolute WER improvements are reported for the improved Branchformer baseline.
- Contributions: The authors conduct an extensive study of methods for merging local and global branches.
- Contributions: 1.81% and 3.65% WER on LibriSpeech test-clean and test-other establish new state-of-the-art performance without external data.
2. RELATED WORK
Prior work combines convolution and self-attention either sequentially or in parallel to capture local and global patterns. Examples span speech, language, question answering, and vision, with Conformer demonstrating strong ASR performance from sequential integration.
- ASR encoder background: End-to-end ASR models include CTC, transducer, and attention-based encoder-decoder systems that share acoustic-encoder backbones.
- ASR encoder background: Convolution captures local information, while self-attention models long-range interactions in acoustic encoders.
- Combining local and global context: Prior models combine convolution and self-attention sequentially or in parallel to capture local and global patterns.
- Sequential integration: QANet pioneered sequential convolution and self-attention, while Conformer showed that placing convolution after self-attention performs best among tested ASR arrangements.
- Parallel integration: Parallel-branch approaches such as Lite Transformer, ConvBERT, and Inception Transformer combine local and global processing across language and vision tasks.
3. PRELIMINARY: BRANCHFORMER
Branchformer uses parallel global and local extractor branches, then merges their outputs. The global branch uses self-attention, while the local branch uses convolutional gating to encode local information.
- Each Branchformer block contains global and local extractor branches plus a merge module.
- The global extractor applies layer normalization, multi-head self-attention, and dropout in sequence.
- The local extractor projects features, applies GELU and convolutional spatial gating, then projects them back to the hidden dimension.
- The original merge module concatenates global and local outputs and applies a linear projection.
- A weighted-average alternative assigns scalar weights to the two branches and can set one branch weight to 0.
4. E-BRANCHFORMER
E-Branchformer enhances Branchformer by making the merge operation sensitive to temporal information and by adding point-wise feed-forward modules. Its main merge enhancement uses lightweight depth-wise convolution to refine information from both branches.
- 4.1. Revisiting the Merge Module: Branchformer’s point-wise linear merge is considered suboptimal because it combines branch outputs without temporal information.
- 4.1.1. Depth-wise Convolution: The proposed merge adds depth-wise convolution, enabling adjacent features to contribute while having negligible speed impact.
- 4.2. Revisiting the Point-wise Feed-Forward Network: The authors add feed-forward modules because Transformer FFNs refine point-wise information after temporal features are aggregated.
- 4.1. Revisiting the Merge Module: The study evaluates multiple merge designs, including multi-kernel convolutions, squeeze-and-excitation, and a Conformer-style convolution module.
5. EXPERIMENTS
Experiments use ESPnet and attention-based encoder-decoder models on LibriSpeech, with specified acoustic features, encoder configurations, augmentation, and decoding components. Training includes SpecAug, speed perturbation, joint CTC-attention loss, and external-language-model shallow fusion with internal language-model estimation.
- Experiments mainly use LibriSpeech, which contains approximately 1000 hours of transcribed speech plus an additional text-only corpus.
- ESPnet provides the experimental toolkit and enables comparisons with released recipes and downloadable models.
- The base system is an attention-based encoder-decoder using 80-dimensional log Mel features, 5K BPE units, convolutional subsampling, and Base or Large encoders.
- Training applies SpecAug, speed perturbation, joint CTC-attention loss, label smoothing, and Adam optimization.
- Decoding combines joint CTC-attention scores with a Transformer external language model and internal language-model estimation.
6. RESULT
Experiments compare E-Branchformer with published models and baselines using WER on LibriSpeech test-clean and test-other, alongside FFN, kernel-size, merge-module, and complexity analyses.
- 6.1. Main result: 1.85% and 3.71% WER were obtained by E-Branchformer (L) with an external LM on test-clean and test-other.The model uses merge-module depth-wise convolution with kernel size 31 and a narrowed macaron-style FFN.
- 6.1. Main result: 0.1% and 0.2% WER improvements over Branchformer were achieved by E-Branchformer (B) with an external LM on test-clean and test-other.Without the LM, E-Branchformer (B) showed similar performance despite being smaller than Branchformer.
- 6.1. Main result: 1.81% and 3.65% WER were obtained on test-clean and test-other after applying Internal Language Model Estimation, establishing a reported no-external-data state of the art.
- 6.2.1. FFN module: FFN and macaron-style FFN variants were evaluated under similar encoder sizes, with 17-layer Branchformer plus FFN outperforming 25-layer Branchformer-only and 13-layer macaron-style FFN performing better.The study also measures MACs because similar parameter counts can yield different computational complexity.
- 6.2.1. FFN module: FFN together with Branchformer was judged a reasonable way to improve accuracy over deeply stacking only Branchformer blocks.
- 6.2.3. Merge module: 0.1% to 0.3% WER improvements occurred across evaluation sets when depth-wise convolution was added to the merge module, whereas an additional kernel-size-3 convolution was harmful.Additional linear projections before each depth-wise convolution produced even worse results; the authors characterize simple depth-wise convolution as parameter- and computation-efficient.
7. CONCLUSION
E-Branchformer enhances Branchformer by merging self-attention and convolution sequentially and in parallel, outperforming Conformer and Branchformer on LibriSpeech without external data.
- 7. CONCLUSION: E-Branchformer is a new ASR encoder architecture with an enhanced merging mechanism for hybrid self-attention and convolution.
- 7. CONCLUSION: The architecture outperforms both Conformer and Branchformer on LibriSpeech test sets without using external data.
- 7. CONCLUSION: Future work considers applying E-Branchformer to Transducer models and other speech tasks, including self-supervised learning and speech enhancement.