Source-linked AI summary
On the Effect of Dropping Layers of Pre-trained Transformer Models
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Preslav Nakov
TL;DR
The paper asks whether all layers of pre-trained models are necessary for maintaining task-specific performance while improving efficiency. It proposes practical layer-dropping strategies and finds that models can be substantially reduced while retaining most performance and matching distilled models.
Problem
The paper examines whether all layers of a pre-trained model are necessary to maintain good task-specific performance while achieving efficiency.
Method
The paper proposes practical strategies that drop top, bottom, middle, or alternate layers in pre-trained models, including contribution-based dropping.
Results
Models can be reduced by up to 40% while maintaining up to 98% of their original downstream-task performance, and pruned models perform on par with distilled models.
Takeaways & Limitations
Layer dropping offers a practical path to improve model efficiency while retaining task-specific performance and matching knowledge-distilled models.
Takeaways & Limitations
The reported results are limited to the five most stable tasks: SST-2, MNLI, QNLI, QQP, and STS-B.
Abstract
from arXiv · showhide
Transformer-based NLP models are trained using hundreds of millions or even billions of parameters, limiting their applicability in computationally constrained environments. While the number of parameters generally correlates with performance, it is not clear whether the entire network is required for a downstream task. Motivated by the recent work on pruning and distilling pre-trained models, we explore strategies to drop layers in pre-trained models, and observe the effect of pruning on downstream GLUE tasks. We were able to prune BERT, RoBERTa and XLNet models up to 40%, while maintaining up to 98% of their original performance. Additionally we show that our pruned models are on par with those built using knowledge distillation, both in terms of size and performance. Our experiments yield interesting observations such as, (i) the lower layers are most critical to maintain downstream task performance, (ii) some tasks such as paraphrase detection and sentence similarity are more robust to the dropping of layers, and (iii) models trained using a different objective function exhibit different learning patterns and w.r.t the layer dropping.
1. Introduction
This paper asks whether all layers of pre-trained Transformer models are necessary for downstream tasks and proposes practical layer-dropping strategies to improve transfer-learning efficiency. Across models and tasks, dropping top layers can substantially reduce size while preserving performance, with reduced models comparable to distilled models.
- The study examines whether every pre-trained model layer is needed for downstream performance and which layers should be retained for efficient transfer learning.
- The authors propose dropping top, bottom, middle, or alternate layers, and identifying low-contribution layers from activation patterns and weights.
- 40% reduction in model size preserves up to 98.2% of performance across tasks and pre-trained models when top layers are dropped.
- Reduced models perform on par with knowledge-distilled models in accuracy, size, and inference speed without costly new-model training.
- One-third of a distilled model can also be pruned with an average loss of 0.75 points, while ALBERT tolerates pruning despite cross-layer parameter sharing.
- Certain downstream tasks require as few as 3 layers, and RoBERTa and XLNet learn task-specific knowledge earlier than BERT.
- The work recommends top-layer dropping as a baseline for distilled models and reports smaller models that are memory- and speed-efficient.
2. Related Work
Related work reduces Transformer models through pruning, factorization, distillation, quantization, and dynamic inference, but often requires training or leaves model memory unchanged. This paper instead evaluates layer importance directly across architectures and tasks, enabling training-free layer selection.
- Prior compression methods include pruning, weight factorization and sharing, knowledge distillation, and quantization.
- Existing pruning work removes weights or attention heads, while LayerDrop trains models to tolerate layer removal during inference.
- The paper shows layers can be removed safely from pre-trained models without additional training.
- Its results are comparable to greedy layer pruning, but it requires no additional training to find an optimal layer set.
- Dynamic exiting speeds inference but leaves the model memory footprint unchanged.
- Layer-level representation analyses describe captured linguistic properties but do not identify which layers matter for downstream transfer.
- This work gauges subsets of layers by downstream-task performance across four pre-trained models and many tasks.
- The reported finding that higher layers can be safely pruned resonates with analyses locating core linguistic knowledge in lower layers.
3. Methodology
The methodology evaluates six ways to remove encoder layers from pre-trained models, then fine-tunes each reduced network on a downstream task. Strategies range from fixed positional removal to parameter- and representation-based selection.
- The authors define a model with an embedding layer and L encoder layers, then evaluate six layer-dropping strategies followed by task-specific fine-tuning.
- Top-Layer Dropping: Top-layer dropping removes K upper layers and adds a task-specific layer above the remaining network.
- Alternate Dropping: Alternate dropping removes N odd or even layers, passing the previous layer's output into the next retained layer.
- Parameter-Based Dropping: Parameter-based methods rank layers by aggregated weight magnitude or weight variance and remove those with the lowest scores.
- The parameter-based experiments use only final weights, and STS-B results are reported on the development set because its test set is unavailable.
- Contribution-Based Dropping: Contribution-based dropping uses cosine similarity between each layer's input and output CLS representations, averaged over the development set.
- Symmetric Dropping: Symmetric dropping retains X layers at both ends and removes K middle layers satisfying 2X + K = L.
- Bottom-Layer Dropping: Bottom-layer dropping removes the first K encoder layers, sending the embedding output directly to layer lK+1.
4. Experimental Setup
The experiments evaluate layer-dropping strategies on selected GLUE tasks using base pre-trained models, followed by task-specific fine-tuning and official-devset evaluation.
- Datasets: The study evaluates SST-2, MNLI, QNLI, QQP, RTE, MRPC, and STS-B, while excluding WNLI because of dataset irregularities.
- Models: Experiments use 12-layer BERT, RoBERTa, and XLNet models, with additional comparisons involving 12-layer ALBERT and six-layer DistilBERT.
- Models: The model selection compares auto-regressive and auto-encoder models, large models with distilled versions, and different pre-training objective functions.
- Implementation and scope: The experiments use default transformer-library settings without parameter optimization and restrict evaluation to base models because larger models exceeded available memory.
- Procedure: The procedure drops layers from a pre-trained model, fine-tunes it on GLUE training sets for three epochs, and evaluates it on official development sets.
5. Evaluation Results
Across GLUE evaluations, layer-dropping performance depends on which layers and strategy are removed: higher layers are generally safer to drop than lower layers, with task- and model-specific exceptions.
- Compared strategies: K = 2, 4, or 6 layers are dropped from 12-layer BERT, RoBERTa, and XLNet, while DistilBERT uses K = 1, 2, or 3.
- Parameter-based strategy: The variance-based parameter strategy is the second-best strategy at K = 6 and outperforms the aggregation-based strategy, selecting different layers for each model.
- Model-specific patterns: The last 2–3 and middle layers can often be removed with small performance loss, but BERT suffers a huge loss when variance-based pruning removes its first two layers.
- Layer location: Dropping alternate top layers outperforms dropping consecutive top layers, indicating that retaining the final layer can matter more than retaining the second-to-last layer.
- Model-specific patterns: XLNet is sensitive to middle-layer removal, while BERT remains competitive after removing four middle layers; contribution-based dropping works better for XLNet than BERT.
- Layer location: Top-layer dropping consistently performs well, whereas bottom-layer dropping performs worst across models because removing lower-layer information is more damaging.
- Task-wise results: QQP and STS-B are least affected by layer dropping: with half the layers removed, QQP loses at most 0.72 points and STS-B decreases by 0.56, 0.95, and 0.34 points for XLNet, RoBERTa, and BERT.
- DistilBERT: Dropping one-third of DistilBERT's layers produces an average loss of 0.75 points, but removing half drastically degrades performance on several tasks.
6. Discussion
The discussion evaluates task-specific layer dropping, model learning dynamics, ALBERT robustness, and comparisons with other compressed models. Results support preserving lower layers while dropping top layers, with substantial robustness across tasks and architectures.
- Task-specific optimal number of layers to drop: Task-specific optimization identifies the minimum layers needed to remain within 1%, 2%, or 3% performance on development sets.The experiments split development data into dev and hold-out sets and evaluate generalization.
- Task-specific optimal number of layers to drop: Most development-set optima aligned with hold-out performance; BERT QNLI dropped one layer with a 0.84-point absolute loss.The reported hold-out loss is evaluated relative to the full model.
- Task-specific optimal number of layers to drop: RoBERTa and XLNet were most robust: dropping the top 9 XLNet or 8 RoBERTa layers kept performance within 1 point, leaving three layers.Dropping nine BERT layers instead caused a 3% loss on QQP.
- Comparing Pre-trained Models: Average layer-wise results show RoBERTa and XLNet mature near layer 7, whereas BERT continues improving through higher layers.RoBERTa learns task-level information earlier, and XLNet learns it faster than BERT in lower-middle layers.
- Pruning the ALBERT Model: ALBERT-6 showed relatively small performance degradation, suggesting parameter sharing may produce representations robust to layer dropping.The authors present this explanation as a hypothesis and report further gains in runtime efficiency from pruning.
- Comparing against Distilled Models: The simple top-layer strategy matched or exceeded GLP6 on QQP, STS-B, MNLI, and QNLI, but underperformed it on SST-2.The comparison indicates no clear advantage for greedy task-optimized pruning over the six-layer model.
7. Conclusion
The paper shows that dropping layers from diverse pre-trained models can substantially reduce size while preserving downstream performance. It also finds that layer importance and learning patterns vary across tasks and training objectives.
- Up to 40% model-size reduction retained up to 98% of original downstream-task performance.
- Pruned models performed on par with knowledge-distilled models.
- The pruning approach does not require retraining and applies to a large set of pre-trained models, including distilled models.
- Lower layers are most critical for maintaining downstream-task performance.
- Models trained with different objectives show different learning patterns, with XLNet and RoBERTa learning task-specific information earlier than BERT.