Source-linked AI summary
Lessons on Parameter Sharing across Layers in Transformers
Sho Takase, Shun Kiyono
TL;DR
Transformers can benefit from larger parameter sizes, but these models occupy substantial GPU memory, and one-layer sharing can require more computation. This paper assigns parameters for M layers across N Transformer layers using SEQUENCE, CYCLE, and CYCLE (REV). The strategies achieve slightly better BLEU scores with less computation at almost equal parameter counts and outperform Universal under the same computational budgets.
Problem
Larger Transformer parameter sizes can improve performance but occupy substantial GPU memory, while Universal Transformer can require twice the vanilla Transformer's training time on WMT English-to-German.
Source: §1 · p. 1Method
Results
The strategies achieve slightly better BLEU scores with less computation at almost equal parameter counts and outperform Universal under the same computational budgets.
Source: §6 · p. 8Takeaways & Limitations
The strategies are efficient in parameter size and computational time, with reported efficiency in high-resource settings, other language pairs, and speech-to-text.
Source: §6 · p. 8The study lacks theoretical justification and does not establish efficiency in pre-training followed by fine-tuning, which was not tested because of computational-budget limits.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all layers such as Universal Transformers (Dehghani et al., 2019), to increase the efficiency in the computational time. We propose three strategies: Sequence, Cycle, and Cycle (rev) to assign parameters to each layer. Experimental results show that the proposed strategies are efficient in the parameter size and computational time. Moreover, we indicate that the proposed strategies are also effective in the configuration where we use many training data such as the recent WMT competition.
1 Introduction
The paper seeks parameter-efficient Transformers that retain performance while reducing memory and computation demands. It proposes sharing parameters across M layers to construct N layers and reports slightly better machine-translation scores with less computation than sharing one layer across all layers.
1 Introduction
- Transformer models with many parameters occupy substantial GPU memory, motivating parameter-efficient approaches that preserve performance at a fixed parameter size.The introduction notes that larger parameter sizes can improve performance (Brown et al., 2020).Source: §1 · p. 1
- Universal Transformer shares one layer's parameters across all layers, but larger weight matrices make it computationally slower; on WMT English-to-German it takes twice the vanilla Transformer's training time.The comparison concerns models with almost the same parameter count.Source: §1 · p. 1
- The method prepares parameters for M layers to construct an N-layer encoder-decoder, relaxing earlier all-layer sharing so smaller matrices can be stacked to improve expressiveness and speed.The paper proposes simple assignment strategies and evaluates them empirically.Source: §1 · pp. 1–2
- The proposed strategies slightly improve machine-translation scores with less computation than one-layer sharing, and outperform it under almost equal training-time budgets.The paper also reports efficiency in high-resource settings, other language pairs, speech-to-text, automatic speech recognition, and language modeling.Source: §1 · p. 2
2 Proposed Method
The method assigns parameters for M independent layers across an N-layer encoder-decoder using three strategies: SEQUENCE, CYCLE, and CYCLE (REV). Each strategy defines a different pattern for reusing those parameters across layers.
2 Proposed Method
- The encoder construction takes the total layer count N, independent-layer count M, and a sharing strategy, then assigns parameters to produce enc1 through encN.The decoder uses the same assignment manner.Source: §2 · p. 2, Figure 2
2.1 SEQUENCE
- SEQUENCE assigns the same parameters to each run of sequential floor(N/M) layers; with M = 3 and N = 6, each pair shares parameters.The paper describes this as the simplest strategy.Source: §2.1 · p. 2
2.2 CYCLE
- CYCLE stacks M independently parameterized layers, then repeats the same order until reaching N layers; with M = 3 and N = 6, it repeats the three-layer stack twice.Source: §2.2 · p. 2
2.3 CYCLE (REV)
- CYCLE (REV) reuses lower-layer parameters in higher layers by reversing the order for the final stack, so the lowest and highest layers share parameters.The motivation is that higher decoder layers have been reported to show larger gradient norms, suggesting lower layers may have redundant parameters.Source: §2.3 · pp. 2–3
3 Experiments on Machine Translation
The machine-translation experiments compare parameter-sharing strategies with Universal and non-sharing baselines across standard and high-resource English-to-German settings, then test additional language directions. Across these settings, the proposed strategies generally improve BLEU over Universal at similar parameter sizes while using less training time; the best strategy can depend on the language pair.
3.1.1 Datasets / 3.1.2 Methods
- The standard English-to-German setup uses 4.5M WMT 2016 sentence pairs; proposed models use M = 6 with N = 12 or 18, alongside Vanilla, Universal, and larger non-sharing baselines.BLEU is measured with SacreBLEU; the proposed strategies are compared with Universal and with models using larger parameter counts.Source: §3.1.1 · p. 3, §3.1.2 · p. 3
3.1.3 Results
- With M = 6 and N = 12, SEQUENCE, CYCLE, and CYCLE (REV) are faster and slightly better-scoring than Universal.On newstest2013, their NLL becomes better than Universal's and Vanilla's after 15,000 training steps and remains better at training end.Source: Table 1 · p. 4, §3.1.3 · p. 4, Figure 3 · p. 5
- The proposed strategies also achieve higher averaged BLEU than models with more parameters, supporting their effectiveness beyond parameter efficiency.The comparison uses averaged BLEU on newstest2010–2016.Source: Table 1 · p. 4, §3.1.3 · pp. 4–5
3.2.1 Datasets / 3.2.2 Methods
- The high-resource English-to-German setup uses 44.2M genuine and 284.3M synthetic training pairs; CYCLE (REV) uses M = 6, N = 12.The comparison includes a reported Kiyono et al. (2020) model with nine layers and considerably more parameters.Source: §3.2.1 · p. 5, §3.2.2 · p. 5
3.3.1 Datasets / 3.3.2 Methods
- The additional language-direction experiments use German-to-English data from the standard setup and 35.8M WMT 2014 English-to-French pairs, with proposed models set to M = 6, N = 18.The English-to-French and French-to-English data use the same preprocessing; proposed strategies are configured for training times almost equal to Universal's.Source: §3.3.1 · pp. 5–6, §3.3.2 · p. 6
3.2.3 Results
- The proposed strategies achieve better BLEU than Vanilla and Universal with almost the same parameter count in the high-resource setting.With synthetic data, CYCLE (REV) matches Kiyono et al. (2020)'s best system except on newstest2019, despite using fewer parameters.Source: §3.2.3 · p. 5, §3.3.1 · p. 5, Table 2 · p. 6
3.3.3 Results
- At almost the same parameter size, all three proposed strategies outperform Universal across the tested language datasets.CYCLE and CYCLE (REV) exceed SEQUENCE on German-to-English, but their superiority over SEQUENCE is unclear for English-to-French and French-to-English.Source: Table 3 · p. 6, §3.3.3 · p. 6
4 Experiments on Automatic Speech Recognition
The ASR experiments evaluate the proposed strategies on LibriSpeech with a Pre-LN speech-to-text Transformer. All three strategies outperform Universal on every set while running faster, though the strongest strategy varies by task and architecture.
4.1 Datasets
- Experiments use LibriSpeech's standard splits, all available training data, clean and other development and test sets, and word error rate for evaluation.LibriSpeech contains 1,000 hours of English audiobook speech.Source: §4.1 · pp. 6–7
4.3 Results
- Across all LibriSpeech sets, SEQUENCE, CYCLE, and CYCLE (REV) achieve better word error rates than Universal while running faster.The table reports word error rates, for which lower is better.Source: Table 4 · p. 7, §4.3 · p. 7
4.2 Methods
- The speech-to-text Transformer uses Pre-LN, with 16 encoder layers and 8 shared parameter sets, and 8 decoder layers and 4 shared parameter sets for the proposed strategies.Vanilla and Universal use six encoder and decoder layers.Source: §4.2 · p. 7
4.3 Results
- SEQUENCE outperforms CYCLE and CYCLE (REV) in ASR; the authors suggest task and layer-normalization differences may explain the contrast with machine translation.Language-modeling results also imply SEQUENCE may be more appropriate with Pre-LN; the authors leave theoretical analysis of training dynamics for future work.Source: §4.3 · p. 7
5 Related Work
Prior work uses full cross-layer parameter sharing to reduce model size, but larger shared weight matrices can increase computation time. The paper positions its partial-sharing strategies alongside work on training acceleration and deep Transformer architectures.
Parameter Sharing
- Parameter-efficient models matter because large training corpora and larger parameter counts are associated with stronger NLP performance, while GPU memory is limited.The cited WMT 2020 news translation system used about 10 times as many parameters as the widely used Transformer-base setting.Source: §5 · p. 7
- Full sharing uses one layer's parameters across N layers, but larger weight matrices increase computation time; the paper proposes assigning M layers' parameters across N layers instead.The proposed strategies are described as more efficient than one-layer sharing and potentially complementary to other parameter-sharing techniques.Source: §5 · p. 8
- Prior work reports that full layer sharing can improve performance at the same parameter size as the original Transformer, but requires more computation time.The section also names shared encoder-decoder parameters and shared attention weights as techniques that can be combined with this method.Source: §5 · pp. 7–8
Training Acceleration
- Separate training-acceleration work on layer stacking and regularization is presented as applicable to accelerating training of the proposed strategies.The cited examples include growing a model from shallow to deep and using simpler regularization such as word dropout.Source: §5 · p. 8
Deep Transformers
- Deep Transformer training stability depends on architecture: Pre-LN is described as more stable than Post-LN, while Admin stabilizes Post-LN models used in machine translation experiments.The paper uses Pre-LN for its other experiments and reports effectiveness of its strategies with both Post-LN and Pre-LN architectures.Source: §5 · p. 8
6 Conclusion
The paper proposes SEQUENCE, CYCLE, and CYCLE (REV), which assign parameters for M layers across N Transformer layers. Across translation settings and other evaluations, the strategies are reported to improve efficiency relative to Universal, with results varying by configuration.
6 Conclusion
- SEQUENCE, CYCLE, and CYCLE (REV) assign parameters for M layers across N layers, using smaller weight matrices than Universal while stacking layers.This relaxes the prior strategy of sharing parameters from only one layer across all layers.Source: §6 · p. 8
- With almost equal parameter counts (M = 6, N = 12), the strategies achieve slightly better BLEU scores than Universal with small computational time; at the same computational budgets (M = 6, N = 18), they outperform Universal.The conclusion also reports greater efficiency in high-resource settings, other language pairs, and speech-to-text.Source: §6 · p. 8
Limitations
The paper's evidence is empirical and does not establish a theoretical justification for the proposed strategies. Experiments also leave pretraining-and-fine-tuning configurations untested and do not identify a universally best strategy.
Limitations
- The proposed parameter-sharing strategies lack theoretical justification; the study relies on empirical observations.The authors frame their purpose as testing alternatives to one-layer sharing across all layers.Source: Limitations · p. 9
- Experiments focus mainly on sequence-to-sequence tasks and train models from scratch; pretraining followed by fine-tuning was not tested because of computational-budget limits.The authors therefore say efficiency in configurations such as BERT cannot be claimed.Source: Limitations · p. 9
- The best strategy cannot be concluded from the experiments; results suggest it depends on task and Transformer architecture, and budget limits prevent exploring all assignments.The authors nevertheless state that all three strategies are more efficient than Universal.Source: Limitations · p. 9
Ethics Statement
The authors explore parameter-efficient Transformers partly in hopes of reducing energy consumption, while noting that evaluating their strategies required substantial computational costs.
Ethics Statement
- The authors believe their proposed strategies are effective at reducing energy consumption.They frame this as addressing substantial energy use by recent neural models.Source: Ethics Statement · p. 9
- Investigating the strategies across various settings incurred substantial computational costs.Appendix B reports the GPUs used and number of updates corresponding to those costs.Source: Ethics Statement · p. 9
A.1 Dataset
The language-modeling experiments use Wikitext-103 to test the proposed strategies with decoder-only Transformers and describe the model configuration used.
A.1 Dataset
- The language-modeling experiment tests decoder-only Transformer strategies on Wikitext-103, which contains a large amount of training data.The study measures validation and test perplexity.Source: §A.1 · p. 12
A.2 Methods
- The language-modeling Transformer uses a Pre-LN configuration; Vanilla and Universal use N = 6, while the proposed strategies use N = 12 and M = 6.Source: §A.2 · p. 12
A.3 Results
In language modeling, the proposed strategies outperform Vanilla and are reported as more efficient than Universal; SEQUENCE achieves the best perplexity among the proposed strategies and remains strong at a smaller parameter size.
A.3 Results
- The proposed strategies outperformed Vanilla and were more efficient than Universal in language modeling.Source: §A.3 · p. 12, Table 5 · p. 12
- SEQUENCE achieved the best perplexity among the proposed strategies.The authors suggest it may be more appropriate for Transformers with Pre-LN and identify theoretical analysis during training as future work.Source: §A.3 · p. 12
- SEQUENCE achieved better perplexities than the other reported scores despite having a smaller parameter size.The lower part of Table 5 includes the Baevski and Auli reported score, the authors’ reproduction, and SEQUENCE with more parameters.Source: §A.3 · p. 12, Table 5 · p. 12
B Details of Experimental Settings
The experimental settings specify the GPUs used and point to a table of task-specific training hyperparameters.
B Details of Experimental Settings
- All experiments used NVIDIA Tesla V100 GPUs, and Table 6 lists the training hyperparameters for each task.Source: B Details of Experimental Settings · p. 12, Table 6 · p. 13