Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition

1. INTRODUCTION

• CTC sidesteps the alignment procedure of HMM-NN systems, while encoder-decoder ASR often needs extensive data, careful schedules, pretraining, or auxiliary CTC losses to approach state-of-the-art results.Encoder-decoder decoders are typically autoregressive at prediction time, restricting inference speed.Source: §1 · p. 1
• Recurrent layers hinder parallelization, and convolutional models need greater depth to achieve the same temporal receptive field.Self-attention uses the whole sequence to model feature interactions at arbitrary temporal distances.Source: §1 · p. 1
• Prior self-attention work in ASR includes layer replacements in HMM-NN systems, hybrid self-attention/LSTM encoders, and Transformer adaptations, rather than fully self-attentional CTC.These approaches include the lattice-free MMI model, LAS, and encoder-decoder systems.Source: §1 · p. 1
• SAN-CTC outperforms existing CTC models and most encoder-decoder models in CER on WSJ and LibriSpeech, with fewer parameters or less training time.The authors also evaluate label alphabets, WFST-decoded WERs, and learned attention heads.Source: §1 · p. 1

2. MODEL ARCHITECTURES FOR CTC AND ASR

• CTC represents an output through paths of intermediate labels, collapsing repeated labels and blanks, then marginalizes over paths corresponding to that output.The intermediate alphabet adds a blank symbol to the label alphabet.Source: §2 · pp. 1–2
• CTC models each path probability as conditionally independent per-time outputs, enabling parallel computation followed by beam-search prediction or gradient-based training.The monotonicity of the path-to-output mapping permits efficient dynamic programming.Source: §2 · p. 2

2.1. Recurrent and convolutional models

• Recurrent CTC models use stacked bidirectional units, while convolution-only models need many layers to cover broad temporal context.Broad context can matter because some label alphabets violate CTC’s conditional-independence assumption.Source: §2.1 · p. 2

2.2. Motivating the self-attention layer

• SAN-CTC replaces recurrent and convolutional layers with self-attention, plus downsampling, position-aware embeddings, and framewise projection to logits over the intermediate alphabet.Each self-attention position attends to all others, giving an immediate full receptive field at O(T^2) inner-product cost.Source: §2.2 · p. 2

2.3. Formulation

• The self-attention layer combines multi-head scaled dot-product attention with a position-wise feed-forward network, using residual connections and layer normalization.The heads are concatenated before the feed-forward sublayer.Source: §2.3 · pp. 2–3

2.4. Downsampling

• Because speech inputs can be much longer than outputs and attention uses a T × T matrix, SAN-CTC considers subsampling, pooling, and reshaping to reduce input length.These options range from least to most preserving of the input data, and CTC still requires U ≤ T/k.Source: §2.4 · p. 3

2.5. Position

• The model compares content-only, additive, and concatenative positional encodings, since self-attention is content-based and CTC’s monotonicity may provide a positional inductive bias.Concatenative encodings use d_emb = 40; additive encodings match d_emb to d_h.Source: §2.5 · p. 3

3. EXPERIMENTS

• The fixed architecture used 10 layers, 512-dimensional representations, 8 attention heads, and 2048-dimensional feed-forward layers, for about 30 million parameters; inputs were downsampled by a factor of 3.Utterances longer than 1800 frames, less than 1% of each training set, were excluded.Source: §3 · p. 3

3.1. Wall Street Journal (WSJ)

• The WSJ experiments trained character- and phoneme-label systems on 80 hours of data, using 40-dimensional mel filter-bank features, dropout of 0.2, and a schedule change at epoch 40.The WSJ models trained in one day on a Tesla V100.Source: §3.1 · p. 3
• Position encoding contributed relatively little to WSJ character-model accuracy, while lossy downsampling preserved CER but degraded WER as frame-transition information was lost.The authors relate these observations to CTC’s monotonicity and independence assumptions.Source: §3.1 · p. 4
• Character models showed forward- and backward-attending heads at all layers, suggesting context expanded with depth; phoneme models had a sharp backward-attending head and more diffuse heads.The authors suggest character labels are more context-dependent than phonemes; they also identify directed or restricted attention as possible routes to faster training.Source: §3.1 · p. 4

3.2. LibriSpeech

• 2.8% test-clean CER on LibriSpeech followed 70 epochs in slightly over a week on one Tesla V100; this outdid all previous end-to-end results except OCD training.For WSJ, SAN-CTC reached 4.7% eval92 CER and outdid all previous CTC-like results except the 4.6% trainable-frontend result; training took one day on a Tesla V100.Source: §3.1 · p. 3, §3.2 · p. 4
• The LibriSpeech experiments used 39-dimensional MFCC features, dropout of 0.1, and a schedule change at epoch 30; the selected test epoch had the best validation score.The paper reports comparison WERs using the provided 4-gram LM via WFST decoding.Source: §3.2 · p. 4
• With 300 BPE merges, the LibriSpeech subword model reached 7.4% CER, 8.7% WER without an LM, and 5.2% WER with a subword WFST LM.Using 10k merge operations was unstable.Source: §3.2 · p. 4

• SAN-CTC is competitive with or outperforms existing end-to-end models on WSJ and LibriSpeech.Source: §4 · p. 5
• The framework integrates a fully self-attentional network with a connectionist temporal classification loss.Source: §4 · p. 5
• Future work includes multitasking SAN-CTC with other decoders or objectives and streamlining its structure with directed or restricted attention.Source: §4 · p. 5