Source-linked AI summary
Character-Level Language Modeling with Deeper Self-Attention
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, Llion Jones
TL;DR
Character-level language modeling has relied heavily on recurrent models and truncated backpropagation, despite questions about their use of long-term context. This paper evaluates a fixed-context causal transformer with auxiliary losses across positions and layers. A 64-layer model achieves state-of-the-art results on text8 and enwik8, while those auxiliary losses support convergence and deeper training.
Problem
Character-level modeling requires learning vocabulary from scratch and handling long sequences, while recurrent models trained with truncated backpropagation may not strongly use long-term context.
Method
The paper trains deep causal transformer self-attention models on fixed-length character sequences, adding auxiliary losses at intermediate positions, layers, and future targets.
Results
The 64-layer transformer achieves state-of-the-art results with 1.13 bits per character on text8 and 1.06 bpc on enwik8.
Takeaways & Limitations
Auxiliary losses at intermediate layers and positions are critical for reaching the reported performance and training much deeper transformer networks.
Abstract
from arXiv · showhide
LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.
Introduction
Character-level language modeling requires learning vocabulary from scratch while handling long, computationally demanding sequences. The paper tests whether fixed-context causal self-attention can replace recurrent processing and finds that auxiliary losses help deep transformers converge and train effectively.
- Challenges: Character-level models must learn word vocabulary from scratch and handle dependencies spanning hundreds or thousands of time steps.Character sequences also require more computation steps than word sequences.
- Existing approach: Strong character-level RNNs commonly use truncated backpropagation through time, passing hidden states between sequential mini-batches.Gradients stop at the boundary of each batch.
- Approach: A causal transformer processes fixed-length inputs from random corpus positions, predicts upcoming characters, and passes no information between batches.Each prediction uses only backward-looking self-attention within its fixed context.
- Findings: The transformer is reported as well-suited to long-sequence language modeling and potentially able to replace RNNs in this domain.The authors speculate that self-attention quickly propagates information across arbitrary distances, unlike step-by-step recurrent transmission.
- Auxiliary training: Three auxiliary losses predict characters at intermediate positions, from intermediate representations, and at future target positions.These losses speed convergence and make deeper networks trainable.
Character Transformer Model
The model uses causal self-attention to factor character-sequence probabilities while adding auxiliary prediction tasks throughout a deep transformer. These tasks provide additional training signals, with some losses decayed or used only during training.
- Character Transformer Model: Language-model probability is factored into conditional probabilities for each character given its preceding sequence.The conditional distribution Pr(t_i|t_0:i−1) is modeled from the preceding characters.
- Character Transformer Model: The 64-layer transformer uses causal attention so each character position attends only to earlier positions.This preserves left-to-right prediction while processing fixed-length character sequences.
- Character Transformer Model: Training deeper than ten layers initially produced slow convergence and poor accuracy, while auxiliary losses enabled substantially deeper networks.The proposed network is described as deeper than prior transformer networks discussed by the authors.
- Auxiliary Losses: Auxiliary losses are added at intermediate positions, intermediate layers, and non-adjacent targets, with discounted weights and decay schedules during training.Only the final-position prediction at the final layer is used during evaluation and inference.
- Multiple Positions: Predicting every sequence position supplies additional training examples, including predictions from contexts as short as one or two characters.These losses are kept at full weight during training and improve convergence and results.
- Intermediate Layers: Intermediate-layer predictions are made at all sequence positions, with lower-layer losses progressively reduced until all intermediate losses disappear halfway through training.For n layers, the lth intermediate layer stops contributing after l/2n of training.
- Non-adjacent Targets: Each position can predict multiple future characters using separate classifiers, with extra-target losses weighted by 0.5.This design is illustrated as adding two predictions per position.
- Positional Embeddings: Learned per-layer positional embeddings replace the standard input timing signal because depth may cause positional information to be lost.The model learns a distinct 512-dimensional embedding for each context position in each layer.
Experimental Setup
Experiments evaluate the model on text8 and enwik8 using deep transformer configurations, fixed-length contexts, and checkpoint-based validation. The setup includes byte- or character-level inputs and reports performance with bits per character.
- Datasets: Evaluation focuses mainly on text8, a 100M-character English Wikipedia corpus reduced to 27 lowercase-character and space symbols.The data are split into 90M training, 5M development, and 5M test characters.
- Datasets: The study also evaluates enwik8, a 100M-byte unprocessed Wikipedia corpus containing markup, non-Latin characters, and 205 unique bytes.Like text8, enwik8 is split into 90M training, 5M development, and 5M test bytes.
- Model configuration: The main model uses 64 transformer layers with two attention heads per layer, hidden size 512, filter size 2048, and sequences of 512 bytes or characters.Each input item is replaced by a 512-dimensional embedding.
- Optimization: The model contains approximately 235 million parameters and is regularized with dropout probability 0.55 in attention and ReLU layers.Training uses momentum 0.99, a fixed learning rate of 0.003, and batches of 16 randomly selected sequences.
- Inference: At inference, the final position of the final layer predicts each character from a 512-character context without passing state between predictions.Consequently, each prediction processes its context from scratch.
- Evaluation protocol: Checkpoints are evaluated roughly every 10,000 steps using bits per character over the validation set, with the best parameters retained.The best model appears after around 2.5 million steps, requiring 175 hours on one Google Cloud TPU v2.
Results
The 64-layer transformer achieves state-of-the-art character-level language-modeling results, while auxiliary losses substantially improve deep-network training. Analyses show strong long-distance copying and diminishing returns from context beyond 512 characters.
- Benchmark performance: 1.13 bpc on text8 test establishes a new state of the art for T64.The smaller 12-layer T12 model also outperforms previous models at 1.18 bpc.
- Context length: 512 characters gives the best reported context result at 1.06 bpc, while increasing context to 1024 provides no further improvement.Performance improves as context grows, then levels off beyond 512 characters.
- Benchmark performance: State-of-the-art performance on enwik8 is achieved without retuning the text8 hyperparameters or training procedure.The paper notes that some prior enwik8 bpc reports actually measured bits per byte.
- Ablation experiments: Multiple-position and intermediate-layer losses provide the largest ablation gains by increasing effective predictions and speeding convergence.These auxiliary losses also help the model utilize greater depth.
- Qualitative analysis: The model copies a deliberately corrupted name across 434 characters, indicating context-based copying rather than memorization from training.It immediately reproduces the remainder of the fake name after observing its initial characters.
Related Work
Character-level language modeling builds on recurrent approaches and related non-recurrent sequence models, alongside applications across several language tasks.
- Character-level modeling has been applied to sentiment analysis, question answering, and classification.
- RNNs and their variants became the dominant language-modeling architecture after advances addressing vanishing gradients.The cited advances include LSTMs, GRUs, Recurrent Highway Networks, and Unitary RNNs.
- Related alternatives include attention-based models, dilated-convolution ByteNet, and Gated Convolutional Networks.These approaches were studied for language modeling at character, byte, or word level.
- Transformer architecture enabled substantially deeper language-modeling networks than were usual for recurrent models.The paper describes a 64-layer transformer, while a 29-layer CNN was considered deep in NLP.
Conclusion
The paper reports state-of-the-art character language modeling with stacked transformers, with further gains from depth and auxiliary losses. The resulting network exploits dependencies spanning more than 400 characters.
- A 12-layer transformer achieves state-of-the-art results in character language modeling.
- Deepening the network to 64 layers improves quality, while auxiliary losses are critical for training at that depth.The losses operate at intermediate layers and sequence positions.
- The network exploits structural and content dependencies over distances exceeding 400 characters.