Source-linked AI summary
Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding
Pan Zhou, Ruchao Fan, Wei Chen, Jia Jia
TL;DR
Transformer ASR’s parallel decoder makes scheduled sampling inefficient, while long sequences with similar acoustic or semantic segments present a generalization challenge. The paper combines parallel scheduled sampling and relative positional embedding, reporting relative gains for short and long utterances.
Problem
Transformer’s nonrecurrent parallel decoder makes scheduled sampling inefficient, and long utterances can trigger deletions among similar acoustic or semantic segments.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3Method
The paper combines parallel scheduled sampling, which simulates inference errors while retaining parallel decoder inputs, with relative positional embedding to restrict attention position ranges.
Source: §1 · p. 1, §3.1 · p. 2, §3.1.1 · p. 3Results
7% relative improvement for short utterances and 70% relative gain for long utterances are reported for the combined methods.
Source: §5 · p. 6Takeaways & Limitations
PSS alleviates the training-testing discrepancy, while RPE eases model collapse on unseen long sentences, according to the authors.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Transformer has shown promising results in many sequence to sequence transformation tasks recently. It utilizes a number of feed-forward self-attention layers to replace the recurrent neural networks (RNN) in attention-based encoder decoder (AED) architecture. Self-attention layer learns temporal dependence by incorporating sinusoidal positional embedding of tokens in a sequence for parallel computing. Quicker iteration speed in training than sequential operation of RNN can be obtained. Deeper layers of the transformer also make it perform better than RNN-based AED. However, this parallelization ability is lost when applying scheduled sampling training. Self-attention with sinusoidal positional embedding may cause performance degradations for longer sequences that have similar acoustic or semantic information at different positions as well. To address these problems, we propose to use parallel scheduled sampling (PSS) and relative positional embedding (RPE) to help the transformer generalize to unseen data. Our proposed methods achieve a 7% relative improvement for short utterances and a 70% relative gain for long utterances on a 10,000-hour Mandarin ASR task.
1. INTRODUCTION
The introduction identifies two generalization challenges for Transformer ASR: scheduled sampling disrupts decoder parallelism, and longer-than-training sentences are especially difficult. It proposes parallel scheduled sampling (PSS) and relative positional embedding (RPE), reporting gains for both short and long utterances.
1. INTRODUCTION
- Transformer’s parallel decoder is faster to train than RNN-based AED, but its nonrecurrent decoding makes scheduled sampling inefficient and it is more sensitive to longer-than-training sentences.The introduction contrasts Transformer’s parallel training with the sequential predictions that make scheduled sampling efficient for RNNs.Source: §1 · p. 1
- PSS mixes ground-truth labels with outputs from a Kaldi-based hybrid model or the Transformer to simulate inference errors while preserving decoder parallelization.The authors report about a 7% relative gain over teacher-forcing training.Source: §1 · pp. 1–2
- Experimental results report about a 7% relative gain over teacher-forcing training, while relative position embedding yields a 30% absolute gain on test utterances longer than the training set.Source: §1 · p. 2
- RPE is proposed to restrict attention positions because the authors attribute long-utterance deletions to confusion among similar acoustic or semantic segments.The introduction names deletions beyond the maximum training position as tail deletion and deletions between similar segments as internal deletion.Source: §1 · p. 2
2. TRANSFORMER-BASED E2E ASR
The section describes Transformer as an encoder-decoder ASR model built from attention and feed-forward layers. Multi-head attention provides parallel attention across representation subspaces, while absolute positional encoding supplies sequence order to a model without recurrence or convolution.
2. TRANSFORMER-BASED E2E ASR
- Transformer follows the AED architecture, with stacked multi-head attention and feed-forward networks in both encoder and decoder; the encoder transforms spectral features and the decoder predicts speech units autoregressively.The predicted units may be phones, syllables, characters, or sub-words.Source: §2 · p. 2
2.1. Multi-head Attention
- Multi-head attention performs parallel attention across representation subspaces using learned projections and multiple attention heads.Each head has dimension d_k, and its output H_i is a head-specific representation.Source: §2.1 · p. 2
2.2. Absolute Positional Embedding
- Absolute positional encoding is added to input representations before the encoder and decoder stacks to supply sequence order; its dimensions use sine for even indices and cosine for odd indices.The position variable denotes absolute sequence position, while i denotes an input-feature dimension.Source: §2.2 · p. 2
3. PROPOSED METHODS FOR IMPROVING GENERALIZATION OF TRANSFORMER
The proposed methods preserve transformer parallelism during scheduled sampling and restrict relative-position modeling to address generalization challenges on long speech. They use simulated decoding errors for decoder inputs and bounded relative-position embeddings in attention.
3.1. Parallel Scheduled Sampling
- Parallel scheduled sampling prepares decoder inputs in advance from simulated inference errors, preserving transformer parallelism during training.The paper proposes two ways to simulate transformer output errors: decoding results from a chain model or from the transformer itself.Source: §3.1 · p. 2
3.1.1. PSS with simulation from chain model
- The chain-model variant mixes Kaldi chain-model hypotheses with ground-truth labels according to a scheduled teacher-force rate, using sentence-level or token-level mixing.Token-level mixing selects between true and hypothesized tokens at each step; positions beyond a short hypothesis are padded.Source: §3.1.1 · pp. 2–3
- Despite the added cost of obtaining hybrid-system outputs, chain-model PSS trains faster than traditional scheduled sampling for the transformer.The authors characterize hybrid-system outputs as error-containing data augmentation that should help generalization, while noting no evidence that its error distribution relates to AED errors.Source: §3.1.1 · p. 3
3.1.2. PSS with simulation from self-decoding
- The self-decoding variant generates training hypotheses batch by batch with the newest model, using greedy search with ground truth as decoder input.It uses the same scheduled strategies as the chain-model variant.Source: §3.1.2 · p. 3
- With N=1, self-decoding PSS is the paper’s token-mixing case; repeating the process for N≥u is equivalent to conventional scheduled sampling, with approximately the same time cost.The paper identifies teacher-forcing training as the N=0 special case.Source: §3.1.2 · p. 3
3.2. Relative Positional Embedding
- RPE targets tail deletion and internal deletion in long utterances; longer training sentences may address the former but not necessarily the latter.The paper links the latter problem to confusion among similar segments in global self-attention.Source: §3.2 · p. 3
- RPE caps relative offsets at k using 2×k+1 embeddings, then incorporates those embeddings into attention’s softmax-input similarity.Different RPEs are used across layers, while attention heads within a layer share the same RPE.Source: §3.2 · p. 4
4. EXPERIMENT
Experiments on 10,000-hour Chinese dictation data compare transformer performance on short and long utterances. PSS and RPE improve results, with the combined system reporting relative CER gains on both test sets.
4. EXPERIMENT
- The evaluation uses about 33K short and 5.6K long test utterances after excluding sentences longer than 40 Chinese characters.The transformer is compared with LAS and a streaming TDNN-LSTM chain model; the base transformer performs better than LAS on SU but degrades on LU.Source: §4 · p. 4
4.1. Improvements by PSS
- Online PSS experiment E3 reaches 8.88% CER on SU, about a 7.2% relative reduction, and performs better than earlier PSS variants.E3 generates hypotheses during training; E1 uses a chain model trained with 50,000 hours, and E2 uses transformer-generated hypotheses in advance.Source: §4.1 · p. 4, Table 2 · p. 4
4.2. Improvements by RPE
- With RPE replacing APE in encoder and decoder, LU CER falls from 42.41% to 12.73%.The encoder setting uses k = 10, attending to about 40 frames on each side of the query.Source: §4.2 · p. 5, Table 3 · p. 5
4.3. Combination of PSS and RPE
- Compared with base transformer B1, relative CER improvements of 7.0% on SU and 70% on LU are reported; this passage does not attribute them to the combined methods.For E8+E3, combining RPE and PSS with the settings in E3’s first row improves SU to 8.9%, while LU stays unchanged.Source: §4.3 · p. 5
- The combined system significantly reduces LU deletions from B1 to E8+E3, and the decoding examples show recovery of tail and internal deletions.B1 outputs are about 40 characters long, with later characters deleted; RPE also alleviates AED self-loops.Source: §4.3 · pp. 5–6, Figure 2 · p. 5, Figure 1 · p. 5
5. CONCLUSIONS AND FUTURE WORK
PSS and RPE improve transformer generalization in speech recognition, with reported gains on short and long utterances. The authors identify source-attention restriction as future work.
- 7% relative improvement for short utterances and 70% relative gain for long utterances were achieved with PSS and RPE.Source: §5 · p. 6
- PSS simulates the real sampling process of scheduled sampling and alleviates the training–testing discrepancy, while RPE distinguishes similar acoustic and semantic segments and eases model collapse on unseen long sentences.Source: §5 · p. 6
- Attention restriction in source attention is identified as a next direction for improving transformer generalization in speech.Source: §5 · p. 6