Source-linked AI summary
What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
Jaejun Lee, Raphael Tang, Jimmy Lin
TL;DR
The paper asks how many final layers of pretrained transformers must be fine-tuned for strong downstream performance. It evaluates BERT and RoBERTa while freezing progressively more early layers, finding that one fourth of the layers generally suffices for 90% of original quality, with full fine-tuning sometimes reducing quality.
Problem
The paper asks how many final layers must be fine-tuned to retain high downstream quality, given evidence that later layers change most during fine-tuning.
Method
The study evaluates BERT and RoBERTa across tasks while varying the number of final layers fine-tuned and freezing earlier layers.
Results
Across tasks, fine-tuning only one fourth of the layers achieves 90% of the original quality.
Takeaways & Limitations
Fine-tuning all layers is not always beneficial, since large BERT and RoBERTa improve on SST-2 when 12–16 layers are frozen.
Takeaways & Limitations
The comprehensive all-datasets analysis is limited mainly to base BERT, while large variants and RoBERTa are analyzed comprehensively only on smaller datasets because they are more computationally intensive.
Abstract
from arXiv · showhide
Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.
1 Introduction
The paper asks how many final transformer layers must be fine-tuned, motivated by evidence that later layers change most and by the computational cost of tuning many parameters. Across models and datasets, it finds that fine-tuning only one fourth of the final layers usually reaches within 10% of full-model quality, while tuning every layer can sometimes hurt.
- Motivation: Recent studies indicate that only a few attention heads or final transformer layers may be necessary for acceptable downstream effectiveness.These findings motivate testing how many final layers actually need fine-tuning.
- Research objective: The paper evaluates how the number of fine-tuned final layers affects model quality across multiple pretrained transformers and datasets.The study treats the relationship between fine-tuned-layer count and task quality as its central research objective.
- Importance: A reasonable cutoff can save computational memory across multiple fine-tuning tasks and support parameter-saving methods.The paper also frames this relationship as useful for guiding future modeling work.
- Main contribution: One fourth of the final layers is sufficient to achieve within 10% parity with the full model on most tasks.This is the paper’s main reported cross-task finding.
- Main contribution: On SST-2, not fine-tuning all layers improves quality compared with fine-tuning the full model.The result is reported as a surprising exception to the usual expectation that tuning more layers helps.
2 Background and Related Work
The paper builds on pretrained language models such as BERT and RoBERTa, alongside evidence that transformer layers have different functional roles and that later layers change most during fine-tuning. It positions its contribution as a comprehensive examination of how many layers are necessary, beyond prior partial analyses.
- Pretrained language models: Pretrained language models are trained on large text collections and then fine-tuned for specific downstream tasks.ELMo established this paradigm before deeper transformer models such as BERT extended it.
- Pretrained language models: BERT uses deep bidirectional transformers pretrained on Wikipedia and BooksCorpus and achieves state-of-the-art results across GLUE and question answering.BERT introduced a prominent transformer-based instance of the pretrained language-modeling paradigm.
- Related models: XLNet and RoBERTa pursue the more-data-plus-better-models approach, with differences in attention objectives, pretraining duration, data, and next-sentence prediction.These models represent prominent alternatives to BERT in the related work discussed.
- Layerwise interpretability: Prior analyses suggest that earlier layers extract more general features while later layers perform more task-specific modeling.In pretrained transformers, bottom layers attend broadly, top layers capture syntax, and the last few layers change most after fine-tuning.
- Research gap: The paper distinguishes its contribution from prior work by comprehensively examining the number of necessary fine-tuned layers.The supplied passages identify earlier studies as partial evidence rather than a full layer-count analysis.
3 Experimental Setup
The experiments compare BERT and RoBERTa across GLUE tasks while progressively freezing earlier layers and fine-tuning the remainder. The setup uses reproduced full-model hyperparameters, averages development results across runs, and limits comprehensive coverage for computational reasons.
- Models: The study evaluates BERT and RoBERTa base and large variants, containing 12 and 24 layers respectively.The models share the same architecture and represent state-of-the-art systems selected for the study.
- Datasets: The datasets come from GLUE and cover natural language inference, sentiment classification, linguistic acceptability, and semantic similarity.The benchmark includes tasks such as MNLI, QNLI, RTE, WNLI, MRPC, STS-B, QQP, and SST-2.
- Evaluation: Development results in the reported tables are averaged across five runs for BERT and RoBERTa model variants.The tables compare none, some, and all nonoutput layer weights being fine-tuned.
- Fine-tuning procedure: The procedure freezes embeddings and the first N layers, then fine-tunes the remaining layers using the full model’s best hyperparameters.This directly varies how many final layers remain trainable.
- Scope: The comprehensive all-datasets exploration is restricted to base BERT, while both models receive broader analysis on the smaller CoLA, SST-2, MRPC, and STS-B datasets.The paper attributes this design to the greater computational cost of large variants and RoBERTa.
4 Analysis
The analysis compares operating points and layer-freezing patterns, finding that relatively few final layers can preserve most full-model quality while additional fine-tuning yields diminishing returns. Freezing 12–16 layers also improves SST-2 quality for large BERT and RoBERTa.
- Operating Points: Fine-tuning only the output and task-specific layers is insufficient across all tasks.The analysis examines the operating points corresponding to 0, 12, and 24 frozen layers.
- Operating Points: 3–5 of 12 layers suffice for base models to reach 90% of the original quality, while large models require 6 of 24 for BERT and 7 for RoBERTa.The intermediate operating point measures the layers needed to reach at least 90% of full-model quality, excluding CoLA as an outlier.
- Per-Layer Study: 64% of original quality remains when every component except the output and task-specific layers is frozen.As more layers are fine-tuned, effectiveness often improves sharply, with gains decomposing nonadditively across frozen initial layers.
- Per-Layer Study: Fine-tuning subsequent layers shows diminishing returns, with models rapidly approaching baseline quality after fine-tuning half of the network.The paper therefore treats half the network as a reasonable cutoff for characterizing the models.
- Per-Layer Study: Freezing 12–16 layers consistently increases quality for large BERT and RoBERTa on SST-2.The finding suggests these models may be overparameterized for SST-2.
5 Conclusions and Future Work
The paper concludes that only a fraction of pretrained transformer layers need fine-tuning for near-original quality and identifies finer-grained attention-head analysis as future work.
- Only a fourth of the layers necessarily need fine-tuning to obtain 90% of the original quality.
- Future work could conduct a more fine-grained analysis of attention-head contributions.