Source-linked AI summary
The Evolved Transformer
David R. So, Chen Liang, Quoc V. Le
TL;DR
The paper asks whether neural architecture search can find a better feed-forward alternative to the Transformer for sequence tasks. It searches a Transformer-inclusive space with evolutionary methods and Progressive Dynamic Hurdles, finding the Evolved Transformer, which consistently improves over the Transformer across four language tasks and reaches 29.8 BLEU on WMT’14 English-German.
Problem
The paper examines whether neural architecture search can design better feed-forward architectures for sequence-to-sequence tasks than the Transformer.
Method
The authors use evolutionary search with Transformer warm starting, a large encoder–decoder search space, and Progressive Dynamic Hurdles to allocate more resources to promising models.
Results
The Evolved Transformer consistently improves over the Transformer on four language tasks and achieves 29.8 BLEU on WMT’14 English-German.
Takeaways & Limitations
The Evolved Transformer matches the original big Transformer with 37.6% fewer parameters and exceeds it by 0.7 BLEU at a mobile-friendly model size of ∼7M parameters.
Takeaways & Limitations
PDH may discard good candidates that become strong only toward the latter part of training.
Abstract
from arXiv · showhide
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.
1. Introduction
The paper applies neural architecture search to design a better feed-forward sequence-to-sequence architecture, motivated by the speed and trainability advantages of feed-forward networks over RNNs. It warm-starts evolution with the Transformer and introduces Progressive Dynamic Hurdles to search directly on WMT’14 English-German, producing consistent improvements across four language tasks.
- Feed-forward networks are viable sequence-to-sequence models because they are faster and easier to train than RNNs.
- The work uses tournament-selection architecture search, warm-started with the Transformer, to evolve a better and more efficient feed-forward architecture.
- Progressive Dynamic Hurdles dynamically allocates more training resources to promising candidate architectures during search on WMT’14 English-German.
- The Evolved Transformer consistently improves over the Transformer on WMT’14 English-German, WMT’14 English-French, WMT’14 English-Czech, and LM1B.
- 29.8 BLEU is achieved by the Evolved Transformer at big model size on WMT’14 English-German.
2. Related Work
Related work established feed-forward alternatives to RNNs for sequence modeling and advanced both Transformer-based architectures and neural architecture search. This paper combines convolution and attention in its search space and introduces PDH as an adaptive alternative to fixed candidate-pool resource allocation.
- RNNs were historically standard for sequence modeling, but convolutional models and the Transformer showed that state-of-the-art sequence models need not be recurrent.
- The search space combines convolutions and attention to leverage strengths of both layer types.
- Prior work improved feed-forward sequence models through pre-training, larger batch sizes, relative positions, and weighted multi-head attention.
- PDH trains promising models to completion while discarding unpromising models early, unlike Hyperband and Successive Halving, which establish candidate pools a priori.
3. Methods
The methods define a Transformer-inclusive encoder–decoder search space and evolve architectures with tournament selection. Progressive Dynamic Hurdles progressively increases training for candidates that exceed dynamically computed fitness thresholds, reducing wasted computation while preserving adaptive candidate generation.
- Evolution with Progressive Dynamic Hurdles: Tournament selection encodes architectures as genes, trains candidates, assigns fitness using validation negative log perplexity, and repeatedly selects models for reproduction or removal.
- Search Space: The initial population is seeded with the Transformer to provide a strong starting point in a minimally tuned, large search space.
- Search Space: The search space contains encoder and decoder cells with NASNet-style blocks and branch, block, and cell-level architectural choices.
- Search Space: Each encoder cell has six blocks and each decoder cell has eight blocks, allowing the Transformer to be represented exactly.
- Evolution with Progressive Dynamic Hurdles: PDH creates hurdles from population mean fitness and grants additional training steps to candidates that exceed each hurdle.
- Evolution with Progressive Dynamic Hurdles: PDH can misclassify candidates that become strong only late in training, although early discarding saves resources and improved overall search quality in the experiments.
4. Experiment Setup
Experiments evaluate translation on three WMT datasets and language modeling on LM1B using Tensor2Tensor-derived data and Transformer-based TPU training settings. Searches use WMT’14 English-German validation fitness, standardized hardware configurations, and specified decoding and regularization procedures.
- Machine Translation: The translation experiments use WMT English-German, English-French, and English-Czech datasets from Tensor2Tensor implementations.
- Machine Translation: The WMT’14 English-German setup uses 4.5 million sentence pairs, newstest2013 for development, and newstest2014 for testing.
- Data Processing: Datasets are packed into uniform 256-token inputs and targets, with examples longer than 256 discarded.
- Language Modeling: Language modeling uses the 1 Billion Word Language Model Benchmark with Tensor2Tensor’s packed implementation and approximately 32K word-piece vocabulary.
- Training Details and Hyperparameters: Training follows Tensor2Tensor Transformer TPU hyperparameters with the memory-efficient Adafactor optimizer and a 10^-2 warmup learning rate over 10K steps.
- Training Details and Hyperparameters: BLEU uses case-sensitive tokenization, while WMT’14 English-German additionally uses the compound splitting procedure from Vaswani et al.
5. Results
The proposed search procedure performs best on average and most consistently, while the resulting Evolved Transformer improves over the Transformer across tasks and model sizes. Its strongest advantages appear at smaller parameter scales, with a 29.8 BLEU state-of-the-art result on WMT’14 En-De.
- Search technique ablation: PDH with Transformer seeding achieved the best average search performance and the lowest standard deviation among tested setups.Only one control run produced a model better than the proposed search’s best models, but that setup also produced substantially worse models.
- Search technique ablation: Random seeding performed considerably worse than every other configuration and was the only setup whose top model exceeded the Transformer’s 4.75 ± 0.01 perplexity.
- Main search: The main search evaluated 15K child models using 979M train steps, while hurdles avoided spending 3.6B steps to evaluate the same models to 240K steps.Over 13K models failed the first hurdle, reducing the resources needed to assess promising models at later training stages.
- Main search: The Evolved Transformer uses wide depth-wise separable convolutions, Gated Linear Units, branching structures, and swish activations, while retaining Transformer-like later portions.Both encoder and decoder independently developed branched lower portions of wide convolutions.
- Performance and analysis: The Evolved Transformer outperformed the Transformer at every tested size, with a 0.7 BLEU advantage at approximately 7M parameters.It matched the base Transformer using 78.4% of its parameters and exceeded the big Transformer with 37.6% fewer parameters.
- Performance and analysis: ET is much more effective at smaller model sizes, while its advantage narrows at big size as BLEU performance saturates.The authors suggest overfitting may occur at big sizes and report that depth should be tuned together with embedding size.
- Performance and analysis: On additional language tasks, ET performed at least one standard deviation above the Transformer on each task and improved BLEU by 0.6 on both En-Fr and En-Cs at base size.LM1B compares test-set perplexity, whereas translation tasks use validation perplexity and test-set BLEU.
- Performance and analysis: 29.8 BLEU established a new state-of-the-art result for the Evolved Transformer on WMT’14 En-De.The result came from the ET Deep model evaluated under a TPU training setup with checkpoint averaging and beam-width-5 decoding.
6. Conclusion
The search found the Evolved Transformer, which consistently outperformed the Transformer on translation and language modeling. It achieved 29.8 BLEU on WMT’14 En-De and strong parameter efficiency at smaller sizes.
- The Evolved Transformer showed consistently stronger performance than the Transformer on translation and language-modeling tasks.
- 29.8 BLEU was achieved on WMT’14 En-De, establishing a new state-of-the-art result.
- 37.6% fewer parameters matched the original “big” Transformer’s quality at smaller model sizes.
- 0.7 BLEU improvement over the Transformer was achieved at a mobile-friendly model size of approximately 7M parameters.
A. Search Algorithms
Progressive Dynamic Hurdles evaluates candidate models incrementally and stops models that fail dynamically established fitness thresholds. This allocates more training to promising architectures while reducing wasted computation.
- Fitness with Hurdles: Each child model is initially trained for s0 steps and evaluated on the validation set before hurdle-based continuation decisions.
- Fitness with Hurdles: Models that fall below the hurdle for their training stage are stopped immediately and return their current fitness.
- Fitness with Hurdles: A model passing every hurdle receives a final training interval before its fitness is returned.
- Progressive Dynamic Hurdles: PDH begins with fixed-step training, then creates hurdles from the mean fitness of the current population after a set number of child models.
- Algorithm Inputs: PDH takes a step-increment vector and child-model count as inputs, while its hurdle queue records established thresholds.
B. Search Space Information
The search space represents encoder and decoder cells built from configurable branches, layers, dimensions, activations, and combiners. Relative dimensions and parameter constraints keep candidate architectures comparable while permitting varied structures.
- Encoding: The encoding uses 14 blocks across encoder and decoder cells, plus separate cell-count fields for repeated architectures.
- Layers: Layer choices include standard, depthwise-separable, and lightweight convolutions, multi-head attention, GLUs, encoder attention, identity, and dead branches.
- Constraints: Decoder convolutions shift inputs by (w − 1)/2 so positions cannot see later predictions.
- Dimensions: Relative output dimensions preserve ratios between layers and are scaled to obtain absolute dimensions within the parameter constraint.
- Dimensions: Relative sizes increase valid configurations under fixed parameter bounds compared with absolute size choices.
- Cell Composition: Each block has two branches that transform hidden-state inputs and combine their outputs into one block output.
- Constraints: Candidate models are constrained to approximately 59.1M–64.1M parameters, centered around the Transformer’s roughly 61.1M parameters.
- Constraints: The search excludes decoder architectures without encoder-attention layers and architectures lacking an unbroken residual path.
C. Ablation Study of the Evolved Transformer
Ablations show that Evolved Transformer variants outperform corresponding Transformer variants, but the performance gap is not attributable to a single mutation. Some parameter-reducing changes were necessary to accommodate more expensive changes.
- Ablation Results: In every ablation comparison, augmented Evolved Transformer models outperformed augmented Transformer models.
- Mutation Effects: Increasing decoder cells from 3 to 4 had the seemingly strongest individual impact, but did not explain the full performance gap.
- Mutation Effects: In half of the cases, both model families indicated that the same mutation was helpful.
- Parameter Trade-offs: Parameter-reducing mutations enabled parameter-expensive changes such as adding a decoder cell within the allowed range.
- Limitations: The ablation study only approximates mutation usefulness because effects depend on the model and interactions among encoding fields.