Source-linked AI summary
Self-Attentional Acoustic Models
Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, Alex Waibel
TL;DR
Acoustic modeling requires handling long sequences while representing both local and global dependencies, and standard self-attention introduces computational and positional challenges. The paper addresses these issues with downsampling, hybrid position modeling, and Gaussian context biases, obtaining performance close to a strong LSTM baseline at much faster computation speed while revealing interpretable head specialization.
Problem
Self-attention is promising for sequence modeling, but acoustic modeling is challenging because sequences are long, positional information is difficult to encode, and frame-level context relevance is harder to estimate.
Method
The paper applies self-attention within a listen-attend-spell acoustic encoder using sequence downsampling, hybrid self-attention/RNN position modeling, and learnable Gaussian attention biases.
Results
The model approaches a strong LSTM-based baseline at much faster computation speed, while Gaussian biasing improves word error rates by up to 1.59%.
Takeaways & Limitations
Self-attentional acoustic models offer interpretability because their heads learn a linguistically plausible division of labor across acoustic representations.
Abstract
from arXiv · showhide
Self-attention is a method of encoding sequences of vectors by relating these vectors to each-other based on pairwise similarities. These models have recently shown promising results for modeling discrete sequences, but they are non-trivial to apply to acoustic modeling due to computational and modeling issues. In this paper, we apply self-attention to acoustic modeling, proposing several improvements to mitigate these issues: First, self-attention memory grows quadratically in the sequence length, which we address through a downsampling technique. Second, we find that previous approaches to incorporate position information into the model are unsuitable and explore other representations and hybrid models to this end. Third, to stress the importance of local context in the acoustic signal, we propose a Gaussian biasing approach that allows explicit control over the context range. Experiments find that our model approaches a strong baseline based on LSTMs with network-in-network connections while being much faster to compute. Besides speed, we find that interpretability is a strength of self-attentional acoustic models, and demonstrate that self-attention heads learn a linguistically plausible division of labor.
1. Introduction
Self-attention offers efficient access to both local and long-range context, but acoustic modeling introduces sequence-length, positional, and context-relevance challenges. The paper addresses these challenges with downsampling, hybrid position modeling, and learnable Gaussian attention biases, while finding interpretable specialization among attention heads.
- Self-attention relates sequence positions through pairwise similarities and can condition directly on both short- and long-range context.
- Quadratic memory growth is problematic for long acoustic sequences, so the model reshapes sequences before self-attentional layers to downsample them.Training utterances contain up to 2026 frames, with 800 frames on average.
- Acoustic position modeling is difficult because self-attention lacks inherent sequence order, and standard positional encodings are unsuitable for fixed speech features.The paper therefore explores concatenated positional embeddings and a hybrid self-attention/RNN architecture.
- Learnable diagonal Gaussian masks give attention heads explicit control over context range and improve word error rates by up to 1.59%.Bottom-layer heads develop diverse context ranges, whereas higher layers use long-range context.
- Different attention heads measure similarity along linguistically plausible dimensions such as phoneme clusters, indicating a division of labor.The analysis suggests that heads establish averaged versions of matching acoustic events across an utterance.
2. Attentional Models for ASR
The ASR system uses a listen-attend-spell encoder that downsamples long acoustic sequences. Its baselines include a pyramidal LSTM and a stronger LSTM/NiN encoder.
- The listen-attend-spell encoder serves as the acoustic model, taking speech features as input and downsampling long acoustic sequences.The pyramidal LSTM halves the number of states between layers by concatenating pairs of consecutive outputs.
- The LSTM/NiN baseline stacks LSTM, network-in-network projection, and batch-normalization blocks, followed by a final LSTM layer.The network-in-network projection is applied at every time step and can downsample by concatenating adjacent inputs.
3. Self-Attentional Acoustic Models
The self-attentional acoustic model transforms state sequences using multi-head attention, feed-forward processing, and layer normalization. Queries, keys, and values are linearly projected for multiple attention heads.
- Self-attention transforms each state into a weighted average of sequence states, with relevance determined by projected dot-product similarity.The intended representation conditions each time step on the most relevant states across the sequence.
- The model applies self-attention in a core architecture specified for the listen-attend-spell encoder.
- Inputs are linearly transformed into query, key, and value representations across eight attention heads.X has sequence length l and hidden dimension d, while each head uses representations of dimension d/n.
- Each attention block adds a position-wise feed-forward network and layer normalization to introduce depth, nonlinearities, and normalized residual processing.The feed-forward network uses two learned projections with a ReLU nonlinearity.
4. Tailoring Self-Attention to Speech
The model adapts self-attention to acoustic sequences through downsampling, hybrid recurrent layers for positional information, and explicit context-range biases. These changes address memory growth, position modeling, and the need to balance local and long-range acoustic context.
- Downsampling: Downsampling before each self-attention block reduces sequence length and lowers attention-matrix memory consumption by a factor of a^2.The reshaping operation reduces sequence length by a factor a while increasing vector-state dimensionality.
- Position modeling: Because self-attention is position-agnostic and standard positional encodings perform poorly on fixed speech features, the model explores concatenated position representations and recurrent hybrids.The hybrid alternatives use LSTM layers either above or within self-attention blocks, trading speed against access to position information.
- Context biasing: Soft Gaussian masks bias attention toward nearby frames while allowing each head to learn its own context range through a trainable variance σ.Separate variances provide different attention heads with different context ranges and make those ranges inspectable.
- Context biasing: Hard band masks restrict attention to a local region of size b by setting weights outside the band to zero.The band width can be selected before training to approximate a desired temporal context range.
5. Experimental Setup
Experiments use TEDLIUM speech data and an XNMT-based sequence-to-sequence setup, with normalized Mel filterbank inputs and training choices adapted to sequence length. Baseline comparisons include reported WER and GPU training speed.
- Data and framework: Experiments use the 200-hour TEDLIUM corpus, with its development split serving as validation data.The implementation is based on the XNMT toolkit.
- Features and normalization: The acoustic input consists of 40-dimensional Mel filterbank features with per-speaker mean and variance normalization.Normalization is performed using Kaldi.
- Training constraints: Utterances longer than 1500 frames are excluded to keep memory requirements manageable.Batch size is dynamically adjusted according to input sequence size.
6. Quantitative Results
The experiments find that hybrid self-attention models achieve competitive WER while improving training speed, and that Gaussian attention biasing is particularly effective. Position modeling favors hybrid architectures, while learned variances reveal differentiated context ranges across heads and layers.
- Baseline comparison: The interleaved hybrid outperforms the stacked hybrid and the pyramidal LSTM on development WER, but not on test WER, while LSTM/NiN remains strongest overall.The stacked hybrid is fastest by a large margin, followed by the interleaved model and the LSTM/NiN model.
- Baseline comparison: The stacked hybrid loses 1.25% absolute WER when attention scores outside the diagonal are set to −∞.This sanity check indicates that the hybrid model uses more than purely diagonal attention.
- Position modeling: Additive positional encodings diverge, concatenated embeddings converge to poor optima, and only hybrid models obtain results comparable to the baselines.Adding positional embeddings to hybrid models does not improve results.
- Attention biasing: Learnable Gaussian masks are the strongest biasing variant, and initializing them with a large variance is important for performance.The local diagonal mask constrains context to five time steps, while Gaussian variances are initialized to 9 or 100.
- Attention biasing: In the first layer, attention heads learn diverse local and longer contexts, whereas the second layer shows less benefit from limiting context.The result supports increasing modeling granularity across layers while retaining some long-range context in lower layers.
7. Interpretability of Attention Heads
The authors correlate attention patterns with phoneme activations and find a linguistically plausible division of labor across heads, with some heads associated with specific acoustic contexts.
- Phoneme activations are obtained from encoder-decoder attention scores and correlated with summarized self-attention sequences using Pearson correlation.The analysis retrains the stacked hybrid model with phoneme targets and concatenates sequences across utterances.
- Table 4 identifies the most highly correlated phonemes for each attention head and manually groups them into linguistic categories.The categories are neither exhaustive nor disjunct.
- The heads exhibit a linguistically plausible division of labor, despite some category overlap and incompleteness.
- Head 2 focuses on utterance-final regions associated with silence, while head 8 is mostly unfocused.The authors interpret these patterns as possibly establishing channel and speaker context.
8. Conclusion
The paper addresses computational and modeling challenges in acoustic self-attention with hybrid architectures and Gaussian biases for controlling context range. The resulting model is nearly as effective as a strong LSTM baseline, computes faster, and offers interpretability through its attention heads.
- The authors address acoustic self-attention challenges with a hybrid architecture and Gaussian biases that control context range.
- The best model is almost as good as a strong LSTM-based baseline while computing much faster.
- Interpretability is presented as an advantage, with attention heads showing a linguistically plausible division of labor.