Scaling Laws for Neural Machine Translation

• NMT scaling is harder to characterize than decoder-only language modeling because it is conditional, uses separate components, and permits multiple valid translations.The paper links differing test-set difficulty to translation direction and contrasts translated target text with natural target text.Source: §1 · p. 2
• A bivariate law using encoder and decoder parameter counts describes NMT scaling, whereas total parameter count alone is insufficient.The study identifies total capacity and its allocation between the encoder and decoder as relevant to scaling behavior.Source: §1 · p. 2
• Target-natural evaluation continues to benefit from capacity scaling, while cross-entropy improvement on source-natural evaluation saturates even for moderately sized models.Source: §1 · p. 2
• The study investigates whether cross-entropy improvements track generation quality across encoder-versus-decoder scaling and source-natural-versus-target-natural evaluation.Source: §1 · p. 2
• Model scaling predictably improves validation cross-entropy across multiple language pairs and training/test data compositions.Source: §1 · p. 2

2.1 Experimental setting

• Experiments vary Transformer depth while holding width and other internal settings fixed, testing encoder-only, decoder-only, and symmetric scaling.Encoder-only and decoder-only runs vary one side while holding the other at six layers; symmetric runs increase both together.Source: §2.1 · p. 3

2.2 Results

• A power law using only total parameters misses NMT scaling differences; the proposed law models loss as a function of separate encoder and decoder parameter counts.Its exponents are test-set-specific, with separate terms for encoder and decoder size.Source: §2.2 · p. 4
• The proposed law captures both encoder- and decoder-scaling behavior, explaining more than 99% of the variation in English→German log-perplexity.The law was jointly fitted to empirical losses from both scaling experiments.Source: §2.2 · p. 4, Figure 2 · p. 5
• The law fitted on encoder- and decoder-scaling data predicts symmetric-scaling test loss with 99.8% of its variation captured out of sample.The reported out-of-sample comparison uses English→German models.Source: §2.2 · p. 4, Figure 3 · p. 5
• For German→English as well as English→German, the proposed functional form closely captures observed scaling behavior.The reported German→English fit is for the Web-Domain test set.Source: §2.2 · p. 5, Figure 4 · p. 6

2.3 Analysis

• Across all considered test sets, the decoder exponent exceeds the encoder exponent, making decoder growth more effective for reducing test loss.The paper notes that this contrasts with the common practice of using deep encoders and shallow decoders.Source: §2.3 · pp. 5–6
• Proportional encoder-decoder scaling achieves the optimal scaling exponent when the parameter ratio stays fixed, while symmetric scaling is nearly optimal and heavily lopsided allocations perform worse.The symmetric allocation is reported as N_d/N ≈ 0.55.Source: §2.3 · pp. 6–7

• The paper defines composition bias as deviation from naturally generated text in source-target data, arising when the paired text is human-translated or machine-translated.The authors note that machine-generated content can also appear in web-harvested corpora and data-augmentation pipelines.Source: §3 · pp. 7–8
• Source- and target-original test sets have drastically different fitted scaling powers, unlike the scaling-penalty pattern reported for language-model evaluations.Source: §3 · p. 7
• On target-original test sets, reducible loss falls more slowly than on source-original sets: scaling the baseline 6L-6L model by 11× is estimated to bring it below 0.05, versus 2.5× for source-original sets.The comparison uses fitted scaling laws and symmetric encoder/decoder scaling.Source: §3 · p. 8, Figure 7 · p. 8
• The test-set composition changes the practical value of model size: moderate scaling can nearly eliminate reducible loss on source-original sets, while target-original sets need larger models.Source: §3 · p. 8
• For models trained on target-original back-translated data, test loss stops improving and deteriorates beyond roughly 400M parameters, despite continued training-loss reduction; the authors hypothesize this threshold reflects the back-translation model’s capacity.They leave that capacity hypothesis for future verification.Source: §3 · pp. 8–9, Figure 8 · p. 9
• In the source-original training experiment, even the smallest models reach roughly 0.16 loss, with an apparent overfitting pattern on a target side described as composition-biased.The experiment compares target-original and source-original training datasets generated from web documents and translated with a hybrid model.Source: §3 · pp. 8–9

• The analysis uses beam search with the baseline model’s decoding configuration rather than tuning its hyperparameters for each model.Source: §4 · p. 9
• On target-original test sets, lower cross-entropy loss accompanies consistent BLEU increases; on source-original sets, loss improves while BLEU deteriorates beyond a threshold.Figure 10 plots BLEU against cross-entropy loss across models and training checkpoints.Source: §4 · pp. 9–10, Figure 10 · p. 10
• A simple power law captures the BLEU–cross-entropy relationship for high-quality models, with reported deviations for smaller models and early checkpoints.Source: §4 · p. 9
• BLEURT scores closely mirror BLEU’s scaling behavior, supporting the same contrast across test-set constructions.Source: §4 · p. 10
• At similar test-loss values, encoder-scaled models yield better generation quality than decoder-scaled models; the cause remains unresolved.The authors leave insufficient search versus different architectural priors as alternatives for future work.Source: §4 · p. 10

5 Conclusion and Limitation

• A bivariate law using encoder and decoder parameter counts describes NMT scaling across strategies, language pairs, and evaluation sets, unlike a univariate law based only on total parameters.Whether this behavior is intrinsic to encoder-decoder architecture or to the NMT task requires further study.Source: §5 · p. 10
• Target-original evaluation sets continue benefiting from scaling, while reducible error on source-original sets quickly saturates to zero.The authors suggest this may reflect lower diversity in translated text, but present that explanation as a possibility.Source: §5 · pp. 10–11
• When training on target-original back-translated text, quality improves with scale before saturating near the capacity of the model used for back-translation.The authors say the synthetic-text generator’s capacity may play a role, but requires further investigation.Source: §5 · p. 11
• On target-original evaluation, lower cross-entropy is associated with improved BLEU and BLEURT, but on source-original data that relationship breaks down.At the same cross-entropy, encoder-scaled models perform better on BLEU/BLEURT than decoder-scaled models; the explanation requires further investigation.Source: §5 · p. 11, Figure 11 · p. 11
• The study’s scaling findings are empirical, and whether their formulation generalizes beyond the studied architecture or task remains unresolved.Source: §5 · pp. 10–11
• The authors raise concerns that machine-generated text may affect scaling and evaluation, including in future models trained on web text.They note that machine-translated text already makes up a significant portion of web text in low-resource languages.Source: §5 · pp. 11–12

A Architecture and Hyper-Parameter Details

• All models use a 1024-dimensional model, 8192-dimensional feed-forward layers, 16 attention heads, 1024 attention hidden dimensions, and a 32,000-piece vocabulary.Source: A Architecture and Hyper-Parameter… · p. 15
• Training uses dropout and label smoothing of 0.1, logit clipping of 10, and Adafactor with 40,000 linear warm-up steps and an inverse-square-root learning-rate schedule.Adafactor uses momentum of 0.9 and a factored second moment to save memory.Source: A Architecture and Hyper-Parameter… · p. 15
• Tables 1–3 report parameter decompositions for encoder-scaling, decoder-scaling, and symmetric-scaling models, respectively.The symmetric-scaling table covers models trained for English→German.Source: Table 1 · p. 15, Table 2 · p. 16, Table 3 · p. 16

B Scaling Laws for Other Test Sets

• The fitted law closely captures empirical scaling behavior across the additional test sets and domains for English→German and German→English.The English→German fits use joint fits on final losses from encoder- and decoder-scaling models; final loss is the median over steps 450K–500K.Source: B Scaling Laws for Other Test Sets · p. 17, Figure 12 · p. 18, Figure 15 · p. 21
• The fitted law also nearly perfectly fits both training-data cross-entropy and training loss with regularization in the illustrated cases.Source: B Scaling Laws for Other Test Sets · p. 17, Figure 12 · p. 18
• On log-log plots, the reducible final loss is plotted against encoder or decoder parameter count for the corresponding scaling models, with uncertainty regions based on Table 4.The plotted confidence region quantifies expected uncertainty from initialization and training-pipeline randomness.Source: B Scaling Laws for Other Test Sets · p. 17, Figure 13 · p. 19, Figure 16 · p. 22
• The empirical encoder- and decoder-scaling results appear linear against their respective parameter counts, as predicted by Eq. (2).Source: B Scaling Laws for Other Test Sets · p. 17, Figure 13 · p. 19
• The scaling law fitted only on encoder- and decoder-scaling models accurately predicts final test loss for symmetric models across the test sets.This tests predictions out of sample, across a different scaling approach.Source: B Scaling Laws for Other Test Sets · p. 17, Figure 14 · p. 20

C Quantifying the Random Variations in the Results

• Eight symmetric models, from 2L-2L through 32L-32L, were retrained using four different seeds to measure final-test-loss fluctuations.The authors identify initialization, data order, and hardware failures or preemptions as sources of training-pipeline randomness.Source: C Quantifying the Random Variations in… · p. 23
• Figure 17 reports standard deviation and maximum difference in final test loss for each retrained model.Source: Figure 17 · p. 23, C Quantifying the Random Variations in… · p. 23
• Table 4 reports variability of final test loss for each test dataset, averaged over all models.Source: Table 4 · p. 23

D.1 Proof of proposition 1

• Taking the logarithm of the objective and changing variables converts the sizing problem into a convex optimization problem.The transformed problem is then solved in the changed variables.Source: §D.1 · p. 24
• The proof uses the convexity of the transformed problem to apply KKT conditions and form a Lagrangian.Source: §D.1 · p. 24
• First-order conditions, the binding constraint, and successive substitutions yield optimal encoder/decoder sizes and their substitution into the scaling law.Source: §D.1 · p. 24

E Curve Fitting Details

• Curve fitting uses scipy.optimize.least_squares with the soft_l1 loss to provide robustness to outliers.Source: E Curve Fitting Details · p. 25
• The fitting function takes encoder and decoder parameter counts as inputs and fits four parameters to log perplexity.The parameter array corresponds to α, p_e, p_d, and c.Source: E Curve Fitting Details · p. 25
• The fitting routine rejects inputs containing NaNs or mismatched observation and feature shapes.Source: E Curve Fitting Details · p. 25
• The four fitting parameters are initialized to 0.2, 0.4, 0.6, and 1.0 before optimization.Source: E Curve Fitting Details · p. 25
• The fitting-scale parameter is selected from 25 geometrically spaced values between 0.001 and 0.025.At 0.025 the fit effectively becomes least squares; smaller values increase robustness to outliers.Source: E Curve Fitting Details · p. 25

F Analysis of the Generation Quality

• All models use beam search with four beams and length normalization 1.0, without per-model hyperparameter tuning.Source: F Analysis of the Generation Quality · p. 26
• For large, well-trained models on target-original data, encoder-scaling fits have larger BLEU exponents than decoder or symmetric scaling fits.The authors report that encoder-scaling models are more successful at improving BLEU-measured generation quality.Source: F Analysis of the Generation Quality · p. 26
• Shallow-decoder models and early-training checkpoints deviate from the Eq. (5) trend.The shallow-decoder examples are 6L2L, 6L4L, and 6L6L; the authors defer detailed analysis of both deviations.Source: F Analysis of the Generation Quality · p. 26, Figure 19 · p. 28, Figure 20 · p. 28
• On target-original data, BLEU improves consistently as cross-entropy loss falls, whereas on source-original data the relationship diverges beyond a threshold.Source: F Analysis of the Generation Quality · p. 26, Figure 18 · p. 27
• On English-to-German and German-to-English Web Domain sets, the largest models score below mid-sized models on source-original data, while target-original BLEU rises with model size.Source: Figure 21 · p. 29, Figure 22 · p. 30
• BLEURT follows the same composition-dependent pattern: lower cross-entropy accompanies consistent gains on target-original data, but the relationship breaks down on source-original data.Source: Figure 23 · p. 31