Source-linked AI summary
Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
Yifan Peng, Siddharth Dalmia, Ian Lane, Shinji Watanabe
TL;DR
Existing speech encoders combine local and global context but can be difficult to interpret, modify, or make computationally efficient. Branchformer uses parallel attention and cgMLP branches, and experiments show competitive or superior results to Transformer, cgMLP, and Conformer while enabling flexible inference complexity. The paper also uses learned branch weights to analyze layer-wise use of local and global dependencies.
Problem
Conformer’s sequential single-branch design makes local and global dependency use across layers difficult to interpret and modify, while self-attention has quadratic sequence-length complexity.
Method
Branchformer uses parallel branches, with attention or an efficient attention variant for long-range dependencies and cgMLP for local dependencies, merged by concatenation or weighted averaging.
Results
Branchformer outperforms Transformer and cgMLP, matches or outperforms Conformer across reported ASR and SLU benchmarks, and supports reduced-computation variants and multiple inference speeds in one trained model.
Takeaways & Limitations
Learned branch-merging weights expose how local and global dependencies are utilized across layers, while branch dropout enables faster inference without retraining or fine-tuning.
Takeaways & Limitations
The reported LibriSpeech numbers are worse than the best Conformer-Transducer result because Branchformer uses a Transformer decoder and a different codebase.
Abstract
from arXiv · showhide
Conformer has proven to be effective in many speech processing tasks. It combines the benefits of extracting local dependencies using convolutions and global dependencies using self-attention. Inspired by this, we propose a more flexible, interpretable and customizable encoder alternative, Branchformer, with parallel branches for modeling various ranged dependencies in end-to-end speech processing. In each encoder layer, one branch employs self-attention or its variant to capture long-range dependencies, while the other branch utilizes an MLP module with convolutional gating (cgMLP) to extract local relationships. We conduct experiments on several speech recognition and spoken language understanding benchmarks. Results show that our model outperforms both Transformer and cgMLP. It also matches with or outperforms state-of-the-art results achieved by Conformer. Furthermore, we show various strategies to reduce computation thanks to the two-branch architecture, including the ability to have variable inference complexity in a single trained model. The weights learned for merging branches indicate how local and global dependencies are utilized in different layers, which benefits model designing.
1. Introduction
Conformer combines convolution and self-attention for local and global context, but its sequential single-branch design is difficult to interpret and modify. Branchformer addresses this with parallel branches and achieves competitive speech-processing results while enabling flexible computation and analysis.
- Conformer captures local and global contextual information using convolution and self-attention, respectively.
- Its sequential, static single-branch architecture makes it difficult to interpret or modify how local and global relationships are used across layers.
- Branchformer uses two parallel branches to model various ranged context in end-to-end ASR and SLU tasks.
- One branch uses self-attention for long-range dependencies, while the other uses a gated MLP to capture local dependencies.
- Across three ASR and two SLU datasets, Branchformer outperforms Transformer and cgMLP and matches or outperforms state-of-the-art Conformer results.The two-branch design also supports efficient attention variants, branch-weight analysis, and two inference speeds within one trained model.
2. Related Work
Prior speech architectures trade off local modeling, global interactions, efficiency, and flexibility. Branchformer combines attention, convolution, and MLP components in parallel to address these limitations while supporting analysis and customizable inference complexity.
- RNNs model temporal dependencies but are difficult to parallelize and do not capture long-range interactions well.
- Self-attention effectively models long-range global context, but its time and memory complexity is quadratic in sequence length.
- MLP-based models can perform comparably with Transformers, but fixed-length inputs prevent direct application to speech processing.
- Convolutional gating enables cgMLP to process speech, and cgMLP achieved the best performance among the referenced MLP-based speech variants.
- Branchformer combines self-attention, convolution, and MLP modules in a unified model because these modules have complementary capacities.
- Unlike Conformer’s sequential single-branch design, Branchformer uses two branches to improve flexibility, interpretability, and customization across layers.Conformer also fixes an interleaving pattern and retains quadratic self-attention complexity.
- Branchformer integrates channel projections into the cgMLP branch and omits an explicit feed-forward network after branch merging.The authors describe this design as novel compared with most prior models.
3. Branchformer
Branchformer uses parallel attention and cgMLP branches to model global and local context, with flexible merging and pruning options. Its design supports efficient alternatives and interpretable branch weighting.
- Architecture: Branchformer blocks use parallel branches sharing one input, with attention modeling global dependencies and cgMLP modeling local dependencies.The branches are followed by dropout, merging, and a residual connection.
- Attention branch: Fastformer can replace standard self-attention, providing global-context modeling with attention-based pooling whose complexity is linear in sequence length.Standard self-attention has quadratic complexity in sequence length, whereas Fastformer uses linear-complexity attention-based pooling.
- MLP branch: The cgMLP branch uses depth-wise convolution and linear gating to capture local dependencies from the sequence.The input is split into two feature-wise sequences; the gated output is their element-wise product.
- Branch merging: The two branches are merged by concatenation or weighted average; weighted averaging makes branch contributions interpretable through learned weights.Concatenation projects the combined features back to the original dimension, while weighted averaging produces a representation capturing both dependency ranges.
- Inference flexibility: Branch dropout enables pruning the entire attention branch at inference by setting its weight to zero after training with branch removal.This supports faster inference while retaining a single trained two-branch model.
4. Experiments
Experiments across ASR and SLU benchmarks show that Branchformer generally surpasses Transformer and cgMLP, while matching or exceeding Conformer. Its two-branch design also supports efficient variants, flexible inference modes, and interpretable branch weighting.
- Evaluation: Branchformer is evaluated on three ASR datasets and two SLU tasks, with reproduced Transformer, cgMLP, and Conformer baselines.The ASR datasets are Aishell, Switchboard 300h, and LibriSpeech 960h; SLURP provides the SLU evaluation.
- ASR Results: 0.7% absolute CER reduction versus cgMLP and Transformer, and 0.2% versus Conformer, gives Branchformer the best Aishell performance.These results are reported on the Aishell test set without a language model.
- ASR Results: Branchformer matches Conformer on Switchboard swb and outperforms it by 0.4% on CallHome while outperforming cgMLP and Transformer by a large margin.The comparison uses WER without language-model rescoring.
- ASR Results: On LibriSpeech, Branchformer achieves 2.4/5.5 WER without an LM and 2.1/4.5 with an LM on test clean/other, comparable with Conformer.Its results are better than those reported by other open-source toolkits but worse than the best reported Conformer-Transducer result.
- SLU Results: Branchformer achieves the best reported intent-classification accuracy and entity-prediction SLU-F1 on SLURP, including across model scales.Figure 3 reports that it outperforms Conformer, cgMLP, and Transformer at all scales.
- Training Stability: Branchformer is reported to be more stable to train than Conformer on short utterances and limited data, including Google Speech Commands.The reproduced vanilla Conformer diverged on that dataset, whereas Branchformer achieved similar performance to other models.
- Efficiency and Analysis: Replacing self-attention with Fastformer yields linear complexity with only minor performance degradation, while branch dropout enables quadratic and linear inference modes from one trained model.Weighted-average merging is slightly worse than concatenation but remains comparable with Conformer; learned weights indicate local/global branch importance by layer.
5. Conclusion
Branchformer uses parallel attention and cgMLP branches to model global and local context. The architecture supports customization, flexible inference complexity, and branch-level interpretability while performing competitively across ASR and SLU benchmarks.
- Architecture: Branchformer has parallel attention and cgMLP branches for global interactions and localized context, respectively.The architecture is designed for end-to-end speech recognition and spoken language understanding.
- Flexibility: Replacing self-attention with efficient attention, weighted-average merging, and branch dropout provides customization and two inference-time complexity modes in one trained model.The two modes trade effectiveness for speed and complexity.
- Interpretability: Learned branch weights reveal how local and global dependencies are used across layers, supporting model design.This interpretability follows from the weighted branch-merging mechanism.
- Performance: Branchformer outperforms Transformer and cgMLP across ASR and SLU benchmarks and is comparable with or better than Conformer.The conclusion also reports greater training stability in extreme data regimes.
A. Fastformer Architecture
Fastformer is an efficient attention variant used to model global dependencies with linear sequence-length complexity. It pools global information from queries and keys, applies it through element-wise multiplication, and produces the block output.
- Design: Fastformer has linear complexity with respect to sequence length and serves as an efficient alternative to standard self-attention.It is used as the global-dependency mechanism in Branchformer variants.
- Global Pooling: The block first transforms the input into query, key, and value sequences, then attention-pools the queries into one global contextual vector.That vector is multiplied element-wise with every key vector to form a new sequence.
- Output Formation: A second attention pooling summarizes the transformed key sequence and multiplies the result into every value vector before a final linear transform.The query is added to the output in a residual-like connection.
B. Implementation Details
The paper presents the hyper-parameters used in its main experiments in Table 8.
- Reproducibility: Table 8 summarizes the hyper-parameters used in the main experiments.
C. Full Results on LibriSpeech 960h
This section reports the full LibriSpeech 960h results and the implementation configurations used for the main experiments.
- Table 9 gives the full LibriSpeech 960h results with and without language models.
- Table 8 documents the Branchformer configurations used across tasks and datasets.
- The section notes that T denotes the input sequence length.
D. Diagonality of Attention Weight Matrices
The diagonality analysis compares attention patterns in Branchformer and Transformer to assess how locally or globally their attention operates.
- Diagonality is computed from attention weights in a T × T matrix, where each row sums to one.
- Branchformer attention has fewer diagonal patterns than Transformer, indicating more global relationships.
- Transformer attention has more diagonal weights, associated with greater capture of local context.
E. Results of Two-Stage Mixed Models
The two-stage experiments combine Conformer and Branchformer blocks sequentially, while branch dropout enables alternative inference modes with different computational costs.
- Conformer-first, Branchformer-later ordering outperforms vanilla Conformer and approaches Branchformer performance.
- Reversing the block order degrades performance, supporting interleaving local and global blocks earlier and specialization later.
- Branch dropout allows one trained model to run in linear- or quadratic-time inference modes without fine-tuning or retraining.
- Increasing dropout harms the original two-branch model but improves the pruned model, which can approach a same-architecture model trained from scratch.
- Inference time for the branch-dropout modes is reported in Figure 7.
G. Preliminary Results on Machine Translation
Preliminary machine-translation experiments evaluate Branchformer on IWSLT 14 De-En using ESPnet, alongside broader experimental context from the paper.
- Branchformer achieves higher BLEU scores than the standard Transformer on IWSLT 14 De-En.
- The machine-translation experiments use the IWSLT 14 De-En dataset, a widely used corpus for evaluating NMT systems.
- The paper also reports that Branchformer converged faster during these preliminary experiments.
- Table 9 provides WER results on LibriSpeech 960h, while Table 10 reports two-stage encoder results on Aishell.
- Table 11 reports Aishell CERs under different self-attention-branch dropout rates for original and pruned Branchformer models.