PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Modeling with Deep Transformers

Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney

arXiv:1905.04226v2cs.CLcs.LG

TL;DR

Speech-recognition language modeling needs effective deep architectures and a way to handle sequence position in autoregressive self-attention. This paper revisits Transformer configurations across LibriSpeech word- and BPE-level tasks and tests models with and without positional encoding. Deep Transformers outperform shallow LSTM stacks, while removing positional encoding slightly improves performance.

  • Problem

    The paper addresses how to configure deep autoregressive Transformers for speech-recognition language modeling and whether they need explicit positional encoding.

  • Method

    The authors evaluate deep Transformer language models on LibriSpeech word-level and BPE-level tasks, applying them through lattice rescoring and shallow fusion and analyzing positional encoding.

  • Results

    Deep Transformer language models outperform shallow LSTM stacks in speech recognition, and removing positional encoding slightly improves their performance.

  • Takeaways & Limitations

    Deep autoregressive self-attention can exploit the increasing information available across positions without an explicit positional-encoding model.

  • Takeaways & Limitations

    Universal Transformer comparisons require separately tuned hyperparameters, including a lower gradient-clipping threshold that may slow convergence.

Abstract

from arXiv · show

We explore deep autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.

1. Introduction

The paper studies deep autoregressive Transformers for speech-recognition language modeling, revisiting their configurations and questioning the need for positional encoding. Experiments cover word-level and BPE-level LibriSpeech models and their speech-recognition applications.

  • The work revisits Transformer configurations specifically for autoregressive language modeling in speech recognition.
  • Word-level models are evaluated through hybrid speech-recognition lattice rescoring, while BPE-level models are combined with encoder-decoder systems by shallow fusion.
  • Well-configured Transformer language models outperform shallow-stack LSTM baselines in perplexity and word error rate.
  • Deep autoregressive self-attention models do not require explicit positional encoding because accessible information increases from left to right and can provide positional information.
  • Removing sinusoidal positional encoding slightly improves the performance of deep Transformer language models.

2. Related Work

Related work established strong Transformer language-modeling performance and explored larger, deeper models and positional-encoding variants. This paper extends that line by testing whether explicit positional encoding is needed in deep autoregressive models.

  • Previous studies reported impressive language-modeling performance using Transformer decoder components.
  • Prior Transformer language-modeling work systematically used learned or sinusoidal positional encoding, with similar performance reported for both.
  • Earlier research also investigated positional-encoding variants to improve self-attention.
  • This work reports that deep autoregressive self-attention models achieve their best performance without an explicit position-encoding model.

3. Autoregressive Self-Attention

The language model uses the Transformer decoder as stacked autoregressive self-attention and feed-forward layers. Its next-token distribution is computed from token embeddings, with positional encoding included in the described input formulation.

  • The language model is based on the Transformer decoder component and stacks self-attention with feed-forward modules.
  • Each autoregressive self-attention layer transforms the representation at position t using the preceding layer's input.
  • Query, key, and value projection matrices, layer normalization, self-attention, and residual projection define the layer transformation.
  • The transformed representation is then passed to a feed-forward module whose activation may be ReLU, GELU, or GLU.
  • The network input sums token embeddings with sinusoidal positional encodings, and the output softmax predicts the next-token probability distribution.
  • During inference, hidden states from predecessor positions are stored to avoid redundant computation; training parallelizes computation across positions.

4. LibriSpeech Dataset

The LibriSpeech language-modeling corpus combines large text-only data with audio transcriptions and evaluates sentence-bounded perplexity. Baselines include a 4-gram model and tuned LSTM language models.

  • The training data combines 800M-word text-only data with 10M words from 960 hours of audio transcriptions.
  • The resulting training corpus averages 21 words per sentence and has a maximum sentence length of 600 words.
  • The word-level vocabulary contains 200K words, and perplexities are reported without contexts beyond sentence boundaries.
  • The baseline set includes the official LibriSpeech 4-gram model and LSTM-RNN language models.
  • The LSTM baseline uses two 2048-node layers with a 128-dimensional input projection, while stacking layers saturates at four layers.
  • LSTM improvements over the 4-gram language model exceed 58% relatively.

5. Text based Experiments

Experiments identify depth, attention-head count, and standard Transformer components as important configuration choices for language modeling, while parameter sharing improves with recurrence but remains behind standard deep Transformers. The resulting word-level models improve perplexity over LSTM models and yield consistent WER gains in hybrid speech recognition.

  • Hyper-parameter Tuning: Increasing Transformer depth from 1 to 42 layers gradually improves perplexity, and a 12-layer model outperforms a similarly parameterized 6-layer model.The compared configurations are (12, 4096, 512, 8) and (6, 8192, 512, 8).
  • Hyper-parameter Tuning: Wide but shallow Transformer models do not perform well, while adding a large projection before the output layer yields no improvement.The shallow wide model has configuration (2, 8192, 2048, 8); the additional projection was tested with (12, 2048, 512, 8).
  • Hyper-parameter Tuning: 16 attention heads give the best performance in the tested setup, whereas activation-function choice has limited overall impact on perplexity.Other models therefore use the standard ReLU activation.
  • Hyper-parameter Tuning: Layer normalization and residual connections are needed for the Transformer models, and removing or reorganizing these components produced no improvement.The experiments also tried removing both components and modifying the feed-forward module.
  • Speech Recognition Results: Longer-trained Transformer models achieve better perplexities than the LSTM-based language models, and lattice rescoring produces consistent WER improvements over LSTM baselines.The word-level models are evaluated on hybrid LibriSpeech recognition using lattice rescoring.
  • Parameter Tying: Parameter-shared Universal Transformers remain behind standard Transformers, although increasing recurrence from 3 to 12 layers consistently improves perplexity without additional parameters.The comparison is subject to different training requirements, including reducing gradient clipping from 1 to 0.1, which may slow convergence.

7. Analysis

The analysis compares Transformer language models with and without positional encoding and shows how autoregressive structure supplies positional information. Removing sinusoidal encoding improves perplexity, while attention patterns reveal distinct layer behaviors.

  • 7.1. Transformer LM without positional encoding: Removing sinusoidal positional encoding gives better perplexities than using it in deep autoregressive Transformer language models.The comparison uses 24-layer models specified as (L, 2048, 512, 8) and evaluated after 5 epochs.
  • 7.1. Transformer LM without positional encoding: In autoregressive modeling, increasing information from left to right provides positional information that deeper layers can recognize without explicit encoding.A new token arrives at each time step, so the lowest network level has progressively more accessible context across positions.
  • 7.2. First layer: With positional encoding, the first layer learns roughly 2- or 3-gram features, indicating direct use of positional information.The first layer is especially interpretable because each position's feature corresponds directly to the word at that position.
  • 7.2. First layer: Without positional encoding, the first layer focuses on the newly input token, forming a diagonal pattern that demonstrates positional awareness.Some functional words are ignored and may instead be represented through attention to the beginning-of-sentence token.
  • 7.2. First layer: The remaining layers behave similarly across models, including blur layers that average positions and window layers that focus on local n-grams.The second and third layers are blur layers, while layers 4 to 9 are window layers.

8. Conclusion

The paper applies deep Transformer language models to speech recognition and finds they outperform shallow LSTM-RNN stacks at both word and BPE levels. It also reports no observed overfitting across tested word-level model sizes, while identifying scaling and regularization as possible improvements.

  • 8. Conclusion: Deep Transformer language models outperform shallow LSTM-RNN stacks on both word-level and BPE-level speech-recognition language modeling.The conclusion summarizes successful application of these models to speech recognition.
  • 8. Conclusion: No regularization is applied because no overfitting is observed across the tested word-level model sizes.The authors suggest that scaling model size and adding regularization may further improve performance.
Loading 1905.04226v2…