PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Tips for the Transformer Model

Martin Popel, Ondřej Bojar

arXiv:1804.00247v2cs.CL

TL;DR

The paper addresses limited machine-translation evidence on Transformer hyperparameters and training behavior. It runs broad English-to-Czech experiments in Tensor2Tensor, finding practical benefits from larger batches, the BIG model, and intensive multi-GPU training. The best 8-GPU, 8-day model outperformed the WMT17 winner on several automatic metrics.

  • Problem

    Prior hyperparameter studies were mostly theoretical or drawn from image recognition, leaving Transformer behavior in machine translation insufficiently explored.

  • Method

    The authors run broad English-to-Czech experiments with the Transformer model in Tensor2Tensor, varying practical training parameters and reporting recommendations.

  • Results

    The best model, trained on 8 GPUs for 8 days, outperformed the WMT17 winner on several automatic metrics.

  • Takeaways & Limitations

    Larger batch sizes improve both training speed and translation quality, and the BIG setup is preferred with at least a day and an 11GB GPU.

  • Takeaways & Limitations

    The experiments focus on basic parameter settings and English-to-Czech translation, while stopping criteria remain difficult to compare fairly across systems.

Abstract

from arXiv · show

This article describes our experiments in neural machine translation using the recent Tensor2Tensor framework and the Transformer sequence-to-sequence model (Vaswani et al., 2017). We examine some of the critical parameters that affect the final translation quality, memory usage, training stability and training time, concluding each experiment with a set of recommendations for fellow researchers. In addition to confirming the general mantra "more data and larger models", we address scaling to multiple GPUs and provide practical tips for improved training regarding batch size, learning rate, warmup steps, maximum sentence length and checkpoint averaging. We hope that our observations will allow others to get better results given their particular hardware and data constraints.

1. Introduction

The paper empirically studies Transformer NMT training in Tensor2Tensor, focusing on hyperparameters and practical training behavior in machine translation. It reports both expected effects and surprising findings, including strong benefits from larger data, models, and multi-GPU training.

  • The study explores important Transformer hyperparameters in Tensor2Tensor because their effects on large-data news translation were not yet fully characterized.
  • The investigation focuses exclusively on machine translation and the Transformer model, extending prior hyperparameter research beyond theory and image-recognition experiments.
  • Larger training datasets generally improve performance, confirming a common expectation in English-to-Czech translation experiments.
  • Two GPUs are more than three times faster than one GPU in the reported experiments.
  • The paper examines interactions among maximum sentence length, learning rate, and batch size as potentially surprising determinants of training behavior.

2. Evaluation Methodology

The paper evaluates translation quality and training speed while emphasizing reproducible stopping criteria, wall-clock learning curves, and terminology for comparing training behavior. It also documents practical measurement limitations and the instability of conventional early stopping.

  • BLEU is used as the main automatic translation-quality metric, despite acknowledged problems and available alternatives.
  • Considerations on Stopping Criterion: NMT training is often nondeterministic and may neither converge nor overfit on large datasets, making conventional final-test evaluation after overfitting impractical.
  • Considerations on Stopping Criterion: Unspecified or approximate stopping criteria can make comparisons unfair because different systems may be evaluated at different training stages.
  • Considerations on Stopping Criterion: Fixed iteration or epoch budgets remain imperfect because models may be insufficiently converged and training-step durations differ across systems.
  • Considerations on Stopping Criterion: Early stopping produced substantial variance in training time and final BLEU across runs sharing hyperparameters but using different random seeds.
  • The study reports full learning curves against wall-clock hours rather than isolated scores, while acknowledging hardware and process-related measurement variation.
  • Terminology: Training throughput equals computation speed multiplied by effective batch size, while convergence speed is BLEU increase divided by time.
  • T2T’s default approx-bleu uses internal subwords and is usually 1.2–1.8 times higher than real BLEU.

3. Data Selection and Preprocessing

The experiments use predominantly CzEng English-to-Czech parallel data and Tensor2Tensor’s built-in subword preprocessing. The authors provide practical guidance on vocabulary training and sentence-length filtering to support larger batches.

  • Most training data comes from CzEng 1.7, with 57M sentence pairs, while 1M additional pairs come from Europarl, News Commentary, and Common Crawl.
  • Most experiments use 58M sentence pairs, while selected experiments substitute the 15M-pair CzEng 1.0 dataset.
  • WMT newstest2013 serves as the development set, and the best model is evaluated on WMT newstest2017 against state-of-the-art systems.
  • T2T trains and applies its own subword vocabulary to raw text, without requiring input tokenization.
  • The default shared English-Czech vocabulary contains 32k subwords, and the main training data contain 992 million subwords after preprocessing.
  • The authors recommend training the subword vocabulary on a sufficiently large sample of the training data.
  • Excluding sentences longer than a threshold can permit higher batch sizes, including through preprocessing before TFRecord creation.

4. Experiments

The experiments examine how model size, batch size, GPU count, training-data size, and sentence-length limits affect Transformer training speed, throughput, memory use, and BLEU. Results generally favor larger models and datasets, but scaling and configuration choices involve important trade-offs.

  • Computation Speed and Training Throughput: BASE provides approximately twice the computation speed and training throughput of BIG.
  • Computation Speed and Training Throughput: 3.2 times more training data is processed per hour with 6 GPUs than with 1 GPU, despite synchronization overhead.Ideal linear scaling would yield 6 times more data per hour.
  • Computation Speed and Training Throughput: 1.3 times higher throughput results from increasing single-GPU batch size from 1000 to 6000, compared with 3.2 times from scaling to 6 GPUs.Throughput grows sub-linearly with batch size, while computation speed decreases as batch size increases.
  • Model Size: 1.8 BLEU separates BIG with batch size 2000 from BASE with batch size 4500 after three days on a single GPU.BIG becomes clearly better after four hours at equal batch size, and on 8 GPUs BIG with batch size 1500 surpasses BASE with batch size 4500 after 18 hours.

Tips on max_length

Choose max_length and batch size together: shorter limits enable larger batches or models, but overly restrictive limits can exclude data and harm training. For batch size, maximize it within GPU memory while accounting for model-specific convergence behavior.

  • Maximum sentence length: Set a reasonably low max_length to enable a higher batch size or a bigger model and reduce out-of-memory risk.Also monitor excluded sentences and early BLEU drops or stagnation.
  • Batch size: For the BASE model, increasing batch size improves BLEU up to 4500, with diminishing returns thereafter.Batch sizes 1000, 1500, 3000, 4500 and 6000 were compared on one GPU.
  • Batch size: The BIG model needs a minimum batch size to converge, while larger batches show almost no BLEU difference after that threshold.Batch size 1450 trained well, whereas 1400 dropped after two hours; lower sizes could diverge permanently or temporarily.
  • Batch size: Set batch size as high as possible while reserving memory to avoid out-of-memory errors.The largest feasible batch size should be established before long training.

4.6. Learning Rate and Warmup Steps on a Single GPU

On a single GPU, the BIG Transformer tolerates a broad range of learning rates and warmup settings, but extremes can slow or destabilize training. Warmup increases can prevent divergence from overly high learning rates, with default values remaining effective.

  • Learning rate: Learning rates from 0.05 to 0.25 make almost no difference, whereas 0.01 slows convergence and 0.30 causes divergence.At divergence, the learning curve drops almost to zero and remains there.
  • Warmup steps: Increasing warmup from 16k to 32k steps enabled learning rates 0.30 and 0.50 without divergence.Learning curves were similar to the baseline using learning rate 0.20 and 16k warmup steps.
  • Warmup steps: Twelve thousand warmup steps caused divergence, while 48k slowed initial convergence but matched the baseline after several hours.The fixed learning rate was 0.20.
  • Recommendation: For the BIG model on one GPU, a relatively large range of learning-rate and warmup-step values achieves optimal results.The defaults are learning_rate=0.20 and learning_rate_warmup_steps=16000.
  • Recommendation: When training diverges, try gradient clipping or more warmup steps, then decrease the learning rate if needed.Decreasing warmup steps can increase the maximum actual learning rate under the noam schedule.

Tips on the Number of GPUs

Scaling the effective batch size across GPUs changes how the Transformer’s learning-rate schedule is experienced over training examples. Experiments show that multi-GPU training can improve convergence substantially, while learning-rate and warmup adjustments require care.

  • Recommendation: Use as many GPUs as available for fastest BLEU convergence, up to the eight GPUs tested.The authors prefer sequential 8-GPU experiments over parallel lower-GPU runs.
  • Motivation: Large-batch training remains an open question for NMT because prior evidence largely comes from image recognition and convolutional networks.The paper tests whether those scaling recommendations transfer to Transformer sequence-to-sequence models.
  • Learning-rate scaling: On 8 GPUs, learning rate 0.30 diverged, but gradient clipping enabled 0.40; increasing to 0.60 still caused divergence.The experiment used the default multi-GPU settings before tuning stabilization measures.
  • Learning-rate scaling: With 8 GPUs, keeping the learning_rate parameter unchanged increases the actual learning rate by 8 times when warmup is ignored.This behavior follows from the linear_warmup_rsqrt_decay schedule and global_step being measured in steps rather than examples.

Tips on Learning Rate and Warmup Steps on Multiple GPUs

When scaling to multiple GPUs, retain the single-GPU optimal learning-rate parameter and adjust warmup cautiously. Reducing warmup may be attempted, but it should not be expected to improve final BLEU.

  • Recommendation: Keep the learning_rate parameter at the optimal value found in single-GPU experiments.Warmup steps may be decreased, but less than linearly and without expecting higher final BLEU.

4.9. Resumed Training

Resumed Tensor2Tensor training preserves optimizer state and supports continuing or modifying experiments, but it does not preserve the training-data position. The framework also supports changing selected hyper-parameters and switching datasets during resumed runs.

  • Resuming from checkpoints: T2T resumes training from an existing checkpoint and restores Adam momentum, allowing training to continue almost as if uninterrupted.The checkpoint is selected through the output_dir parameter and checkpoint file.
  • Resuming from checkpoints: Resumed training starts from a random position in the training data, and stopping changes relative and wall-clock times shown in TensorBoard.
  • Changing training settings: Resumed runs can change hyper-parameters such as effective batch size and GPU count when those changes cannot be parameterized by training steps.
  • Changing training settings: Resumed training can switch from general-domain data to target-domain data for late-stage domain adaptation.The learning rate or schedule may also need adjustment so it does not become too small.

4.10. Checkpoint Averaging

Checkpoint averaging reduces training-curve variance and usually improves BLEU over a non-averaged baseline. The experiments favor hour-spaced checkpoints in later training, while acknowledging that the best averaging window was not tested for statistical significance.

  • Checkpoint selection: One-hour checkpoint intervals achieve slightly better results than averaging the last 20 checkpoints saved every 10 minutes.The longer interval also reduces time spent saving checkpoints.
  • Effect of averaging: About 0.2 BLEU is the usual improvement from checkpoint averaging over the non-averaged baseline, with some setups improving by over 1 BLEU.Averaging also lowers checkpoint-to-checkpoint flickering.
  • Checkpoint selection: After 4.5–7.5 days of training, averaging 16 checkpoints covering the last 16 hours appears slightly better on average than averaging 8 checkpoints.The authors did not perform a proper significance evaluation for this comparison.
  • Model averaging: Forking resumed training creates semi-independent models whose final checkpoints can be averaged, with preliminary results showing a small additional improvement.
  • Practical recommendation: Averaging 8 checkpoints takes about 5 minutes, making it a low-cost way to improve BLEU relative to total training time.

5. Comparison with WMT17 Systems

The best Transformer run was compared with WMT17 English-to-Czech systems using the same automatic-metric implementation. It outperformed the best system across several automatic metrics despite using fewer common enhancements.

  • Experimental setup: The best Transformer model was a BIG configuration trained on 8 GPUs for 8 days with 8 checkpoints averaged.
  • Automatic evaluation: The Transformer outperformed the best WMT17 system in BLEU, TER, CharacTER and BEER.The comparison used a subset of WMT17 constrained training data, so the results were comparable.
  • System differences: The Transformer achieved this comparison without back-translated data, reranking with other models or ensembling.

6. Conclusion

The article reports broad Transformer experiments for English-to-Czech translation and extracts practical recommendations for training. Larger batches improved both speed and translation quality, while the best 8-GPU, 8-day model surpassed the WMT17 winner on multiple automatic metrics.

  • Scope: The study covers a broad range of basic Transformer parameter experiments for English-to-Czech neural machine translation.The experiments consumed about 4 years of GPU time.
  • Practical observations: Larger batch sizes produced faster training and better translation quality for the Transformer model.
  • Practical observations: The authors recommend intense training with as many GPUs as possible rather than several concurrent single-GPU experiments.
  • Main comparison: The best model trained on 8 GPUs for 8 days outperformed the WMT17 winner on several automatic metrics.
Loading 1804.00247v2…