Multi-scale Transformer Language Models

1. Introduction

• Word choice is constrained by both local grammatical correctness and longer-range information such as topic, motivating representations that reflect language’s simultaneous organization across scales.The passage relates this structure to hierarchies in visual perception.Source: §1 · p. 1
• Multi-resolution representations have also been used for image generation and compression, and for speech and audio generation.The cited examples include Laplacian pyramids and multi-scale audio models.Source: §1 · p. 1
• Language models typically learn representations only at word or subword scale and rely on next-word prediction to capture consistency across scales, while transformer memory grows quadratically with context length.Earlier multi-scale sequence-model efforts had not seen widespread adoption on current large-scale language-modeling benchmarks.Source: §1 · p. 1
• 23% smaller memory footprint and better perplexity were achieved by a 30-layer multi-scale model than by a vanilla transformer with less than half as many layers on the Toronto BookCorpus.The comparison is reported as an example of the models’ benchmark advantages.Source: §1 · p. 1
• Multi-scale models combine fine-grained representations with coarser-scale context, and a short fine-scale context plus longer coarse-scale context can yield perplexity similar to using the full longer context at fine scale.The paper presents this as an advantage of combining representations at multiple scales.Source: §1 · p. 2

Memory-Efficient Transformers

• As transformer models grow, computational resources—especially memory—grow quickly, motivating work on memory-efficient architectures.Source: §2 · p. 2
• Memory-efficient transformer approaches include adaptive attention spans, compressed memories, sparse attention, reversible layers, and infinite-depth weight-tied networks.Some approaches trade additional computation for memory savings; reported overhead for root-finding methods is typically at least 50%.Source: §2 · p. 2

Multi-scale architectures for sequential data

• Multi-scale sequence models build representations at multiple resolutions to preserve high-level structure while retaining fine-scale flexibility, with prior applications in conversation, audio, speech, music, and language.Approaches include fixed time scales and learned clocking functions.Source: §2 · p. 2
• Transformer hierarchies based on explicit sentence or paragraph boundaries can miss hierarchies within sentences or across sentences in a paragraph; the authors identify generic alternatives as unexplored for transformer language models.The passage frames this as a limitation of boundary-dependent models and a gap in prior work.Source: §2 · p. 2

3.1. Language Modeling & Transformer Preliminaries

• Language models factorize sequence probability autoregressively; multi-scale models use downsampled representations alongside the finest scale, where k_1=1.For example, scales (1, 4, 16) make each scale four times coarser than the preceding one.Source: §3.1 · pp. 2–3

3.2. Top-down Model

• A 26-layer Top-down model is about 30% faster at inference than a 14-layer vanilla transformer.It builds representations from coarse to fine using transformer layers and convolutional upsampling, with predictions at the finest scale; coarse-scale layers run only once every k_i steps.Source: §3.2 · p. 4

3.3. Bottom-up Model

• The Bottom-up model builds representations from fine to coarse, then aggregates coarser-scale representations with word embeddings before finest-scale blocks make word-level predictions.Its aggregation layer uses attention heads that access representations from different scales, with masks that prevent looking at future tokens.Source: §3.3 · p. 4

3.4. Retina Model

• The Retina model uses finer representations for nearby tokens and progressively coarser ones for distant context, selecting scales through different attention-head subsets.Each scale has its own context window, with coarser scales assigned larger, more distant windows.Source: §3.4 · pp. 4–5

3.5. Space & Time Complexity for Transformer Layers

• At 4x, 16x, and 64x scales, transformer layers are more efficient in time and memory, and coarser layers process fewer positions.Transformer attention has quadratic complexity in sequence length, while position-wise activation storage dominates memory when N ≈ H.Source: §3.5 · p. 5

4.1. Datasets

• Wikitext-103, BookCorpus, and CC-news use standard splits for Wikitext-103 and random splits for the other two; dataset statistics are reported in Table 1.Wikitext-103 uses 32k BPE replacements and word-level perplexity; BookCorpus and CC-news use 30k replacements and subword-level perplexity.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Experimental Details

• Wikitext-103 and BookCorpus use 12-, 14-, or 16-layer vanilla Transformer baselines, while CC-news uses a larger model dimension and feedforward layer.The Wikitext-103 and BookCorpus baseline has dmodel = 768 and dff = 3,072; CC-news uses dmodel = 1,024 and dff = 4,096.Source: §4.2 · p. 6
• At inference, models process contiguous text with a 512-token sliding window, stride 256, and predictions restricted to each window’s last 256 tokens.Training uses batches of 256 randomly sampled 512-token chunks; CC-news uses gradient accumulation to simulate that batch size.Source: §4.2 · p. 6
• Multi-scale models retain the baseline configuration per Transformer layer and add upsampling and downsampling modules, with scales increasing fourfold at each step.The best-performing models use average pooling for downsampling and a transpose convolution for upsampling; strided convolutions are examined in an ablation.Source: §4.2 · p. 6
• Training uses mixed precision and Adam, with a linearly warmed-up learning rate followed by cosine annealing over one million steps.The learning rate reaches 2.5 × e−4 over 40,000 warmup steps; Wikitext-103 dropout rates were tuned from 0.1 to 0.5 in increments of 0.05.Source: §4.2 · p. 6

5.1. Perplexity vs Memory Footprint trade-off

• Multi-scale transformers match strong baselines with less memory or achieve lower perplexity at the same memory footprint.Figure 3 plots test perplexity against training memory footprint on BookCorpus.Source: §5.1 · pp. 6–7, Figure 3 · p. 7
• On Wikitext-103, multi-scale transformers overfit faster than vanilla transformers, while modeling rare tokens better than the baseline.The authors attribute faster overfitting to greater capacity at the same memory footprint and propose investigating better regularization.Source: §5.1 · p. 7

5.2. Analyzing Transformer LM behavior to context perturbations

• When shuffled context is more than 50 tokens away, the change in likelihood is minimal, suggesting limited use of word order beyond that distance.The test shuffled the first 256 tokens of a 512-token sequence and measured likelihood on the subsequent 256 tokens.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3. Ablations

• Adding a coarser scale lowers perplexity when moving from (4,1) to (16,4,1) at the same memory footprint, but adding the 64x scale yields no observed gain.Source: §5.3 · p. 8
• The best layer allocation places more layers at finer scales, trading perplexity against memory because the finest scale accounts for most memory use.Source: §5.3 · p. 8
• Skinnier, deeper vanilla transformers performed worse, indicating that multi-scale gains are not entirely due to increased depth.Source: §5.3 · p. 8
• With the same memory footprint, Retina improves over a model attending to all 512 words at the finest scale by assigning different context ranges across scales.The reported mask gives the finest scale 0:8, the next scale 8:256, and the coarser scale 256:512.Source: §5.3 · p. 8

5.4. Sample based evaluation

• Average pooling outperformed strided convolutions while max-pooling was unstable; multi-scale and baseline models showed no significant difference in generation quality.Source: §5.3 · p. 8, §5.4 · p. 8

• Three multi-scale architectures achieve competitive or better perplexities than vanilla transformers at the same model memory footprint across three language-modeling benchmarks.Source: §6 · pp. 8–9
• Combining representations from shorter-sequence coarser scales with the finest-scale representation reduces computational and memory costs.Source: §6 · p. 9
• Future work will study combining these architectures with adaptive attention-head spans and how topic and grammatical information respond to scale choices.Source: §6 · p. 9

6.1. NLL vs Word Frequency

• The performance gap between models with smaller and larger context windows stems from low-frequency tokens, examined through frequency-binned test NLL on Wikitext-103 and BookCorpus.Tokens were grouped into 5 or 10 equally sized cumulative-frequency bins, ordered from rarest to most frequent.Source: §6.1 · p. 11, Figure 5 · p. 11, Figure 6 · p. 11

6.2. Coarse-only Transformer LMs

• At scale 4, the coarse-only model averages embeddings of non-overlapping 4-gram chunks and predicts the subsequent chunk using a cross-entropy objective against a uniform distribution.Source: §6.2 · p. 11
• Adding a criterion to predict the subsequent bag-of-words distribution at every scale produced no observed improvements and served as a sanity check on scale-specific representations.Source: §6.2 · p. 11
• Nearest neighbors from Wikitext-103 representations at different scales typically returned segments within highly related topics.Source: §6.2 · p. 11, Table 6 · p. 13

6.3. Sample-Level Evaluation Metrics

• Sample-based evaluations generated 256-token sequences from 64-token contexts using top-k sampling at temperature 0.7, and included likelihood, reference-BLEU, and n-gram-repeat measures.Source: §6.3 · pp. 11–12
• The authors observed no significant sample-quality differences between models and also reported generation time for 256 tokens given a 64-token context.Source: §6.3 · p. 12, Table 5 · p. 12