PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Few-sample BERT Fine-tuning

Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, Yoav Artzi

arXiv:2006.05987v3cs.CLcs.LG

TL;DR

Few-sample BERT fine-tuning is unstable, and the paper examines whether optimizer bias, pre-trained-layer initialization, and short fixed training schedules contribute to that instability. Through empirical studies across benchmarks and training settings, it evaluates remedies and revisits previously proposed stabilization methods. The authors report more stable fine-tuning after addressing these practices, with the effects of prior methods diminishing substantially.

  • Problem

    Few-sample BERT fine-tuning can produce substantially different or degenerate models across random seeds, motivating study of optimization, initialization, and training-iteration choices.

  • Method

    The paper empirically studies optimizer bias, BERT-layer initialization, and training duration across few-sample benchmarks, then re-evaluates proposed stabilization methods under revised practices.

  • Results

    Fine-tuning becomes more stable and degenerate runs are eliminated after addressing the identified practices, while previously proposed methods show substantially reduced impact.

  • Takeaways & Limitations

    Bias correction, re-initializing selected top layers, and allocating more training time are practical remedies examined for stabilizing few-sample BERT fine-tuning.

  • Takeaways & Limitations

    The reported numbers are test performance and are not directly comparable with previously published validation results; the study emphasizes relative improvements.

Abstract

from arXiv · show

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.

1 INTRODUCTION

Few-sample BERT fine-tuning can be unstable, producing substantially different or degenerate models across random seeds. The paper investigates optimization, initialization, and training-duration practices and tests remedies across benchmarks.

  • Few-sample BERT fine-tuning can yield significantly different and sometimes degenerate models from identical processes with different random seeds.The instability is especially observed for BERTLarge on small datasets.
  • The study examines how the optimizer, model initialization, and number of training iterations affect few-sample fine-tuning.Experiments cover multiple common benchmarks and focus on small-data settings.
  • Common practices identified as suboptimal include biased gradient estimation, detrimental top-layer initialization, and a predetermined small training-iteration budget.These practices are investigated together with alternative remedies.
  • Addressing these practices eliminates degenerate runs and makes performance more stable, reducing the need for numerous random restarts.The tested remedies can have overlapping effects, and longer training can compensate for the biased optimizer.
  • The paper re-evaluates proposed stabilization methods and finds their impact decreases substantially under the modified fine-tuning process.The revised optimization practices also identify directions for future method development.

2 BACKGROUND AND RELATED WORK

BERT transfers contextual representations from self-supervised pre-training by using a Transformer encoder and a task-specific fine-tuning layer. Prior work links instability and layer transferability to how fine-tuning modifies the pre-trained model.

  • BERT: BERT is a Transformer encoder pre-trained with masked language modeling and next-sentence prediction objectives.It produces contextualized embeddings for input tokens.
  • BERT: For classification, BERT prepends a [CLS] token whose pooled embedding summarizes the input for a task-specific output layer.The pooler aggregates the [CLS] representation through a fully connected layer with tanh non-linearity.
  • Fine-tuning: Fine-tuning replaces BERT’s original output layer and updates both the new task-specific parameters and the pre-trained weights.The modified weights include word embeddings, Transformer blocks, and the pooler.
  • Fine-tuning Instability: Earlier methods stabilize few-sample fine-tuning by using an intermediate task or regularizing the fine-tuned model toward pre-trained weights.These approaches address the instability documented since BERT’s introduction.
  • BERT Representation Transferability: Probing studies find intermediate-layer representations more transferable or applicable to new tasks than later-layer features, which change more after fine-tuning.The paper builds on these findings while focusing on how pre-trained weights influence optimization.

3 EXPERIMENTAL METHODOLOGY

The experiments study few-sample BERT fine-tuning across GLUE tasks using controlled datasets, optimizer analyses, and repeated training runs. The setup evaluates stability and performance under different data sizes, initialization choices, and training durations.

  • Data: The study uses eight GLUE datasets spanning natural language inference, paraphrase detection, sentiment classification, and linguistic acceptability.It focuses primarily on four datasets with fewer than 10k training samples and also downsamples all eight to 1k examples.
  • Optimization Algorithm: The paper compares BERTADAM with debiased ADAM because BERTADAM omits ADAM’s bias-correction step.The algorithm passages define the moment estimates, correction terms, and parameter update used for comparison.
  • Data: The evaluation creates held-out test sets by splitting validation data for the four primary few-sample datasets.This differs from earlier studies that primarily reported validation performance.
  • Experimental Setup: The default setup fine-tunes uncased 24-layer BERTLarge with batch size 32, dropout 0.1, peak learning rate 2 × 10^-5, and three epochs.Training uses gradient clipping, warm-up followed by linear decay, mixed precision, validation checks, early stopping, and 20 random seeds.
  • Experimental Setup: Optimizer effects are evaluated over 50 random seeds on RTE, MRPC, STS-B, and CoLA, while other experiments use repeated controlled fine-tuning runs.The methodology emphasizes performance distributions and stability across random trials.

4 OPTIMIZATION ALGORITHM: DEBIASING OMISSION IN BERTADAM

BERTADAM omits ADAM’s bias-correction step, producing an overestimated update early in training; this disproportionately affects small datasets and destabilizes fine-tuning. Experiments show debiasing reduces variance, improves convergence, and can yield reliable performance with moderate random-trial budgets.

  • BERTADAM omits ADAM’s bias-correction step, directly using biased moment estimates in parameter updates.The standard ADAM correction compensates for moment estimates initialized at zero.
  • The bias is highest early in training and converges toward one with sufficient iterations, so its effect is concentrated in short fine-tuning runs.Small datasets typically use fewer than 1k iterations, while the bias ratio remains significantly above one throughout fine-tuning.
  • Bias correction significantly reduces performance variance across 50 random seeds and four datasets, eliminating many degenerate runs.Without correction, 48% of RTE runs achieved accuracy below 55%, near random guessing.
  • Bias correction speeds convergence and reduces the range of training loss across random trials.Biased BERTADAM produces worse average training loss and higher maximum training loss, indicating underfitting linked to optimization.
  • Debiased ADAM reliably achieves good test results with 5 or 10 random trials, while excessive trials can overfit validation-based model selection.The authors therefore recommend a moderate random-trial budget.

5 INITIALIZATION: RE-INITIALIZING BERT PRE-TRAINED LAYERS

The paper tests re-initializing BERT’s pooler and top Transformer layers because later pre-trained layers can be poor fine-tuning initialization points. Re-initialization generally improves performance, reduces variability, and accelerates convergence, but its benefit depends on how many layers are replaced.

  • Method: Re-initialization replaces the pooler and top L Transformer blocks with the original BERT initialization and compares this setting with standard fine-tuning.The number of re-initialized blocks is selected from L ∈ {1, ..., 6} using mean validation performance.
  • Performance: Re-initialization consistently improves mean performance and usually decreases variance across the evaluated datasets.The same benefits are also reported for pre-trained models other than BERT.
  • Sensitivity to L: Re-initialization improves worst-case performance more than best-case performance, suggesting greater robustness to unfavorable random seeds.Improvements are already visible when only the pooler is re-initialized, and further layers can help.
  • Sensitivity to L: Performance plateaus or decreases when too many layers are re-initialized, because some pre-trained layers contain generally important features.The best value of L varies across datasets.
  • Optimization effects: Re-initialization leads to faster convergence during fine-tuning.The paper also tracks L2 parameter changes across Transformer blocks to examine how re-initialization affects optimization.

6 TRAINING ITERATIONS: FINE-TUNING BERT FOR LONGER

The paper examines the common three-epoch schedule and finds that a fixed, short training duration is sub-optimal. Longer training often improves performance and stability, but the best iteration count differs across tasks.

  • Motivation: The commonly adopted three-epoch practice for BERT fine-tuning is sub-optimal.The standard slanted triangular learning-rate schedule requires choosing the total number of iterations in advance.
  • Results: Longer fine-tuning can improve both training stability and model performance.The improvement is more pronounced for the 1k downsampled datasets.
  • Iteration sensitivity: Different tasks generally require different numbers of training iterations, making a one-size-fits-all schedule difficult to identify.The experiments tune iteration counts across 200, 400, 800, 1600, and 3200 steps for the 1k datasets.
  • Interaction with Re-init: Training longer reduces the gap between standard fine-tuning and Re-init, although Re-init still improves final performance on datasets such as MRPC and MNLI.This indicates that additional iterations can help models recover from bad initializations.
  • Interaction with Re-init: Re-init requires fewer iterations to achieve the best performance on most datasets.The paper reports this as evidence that Re-init provides a better initialization point for fine-tuning.

7 REVISITING EXISTING METHODS FOR FEW-SAMPLE BERT FINE-TUNING

The paper re-evaluates previously proposed few-sample fine-tuning methods using debiased ADAM and test-set evaluation. Their benefits are generally smaller than previously reported, while intermediate-task training remains consistently effective and several regularization methods show inconsistent or negligible gains.

  • Experimental setup: The evaluation covers intermediate-task transfer, Mixout, pre-trained weight decay, conventional weight decay, longer training, and Re-init on RTE, MRPC, STS-B, and CoLA.Models use BERTLarge with debiased ADAM, and methods are tuned with comparable search spaces.
  • Overall comparison: Previously proposed methods show less pronounced benefits when re-evaluated with debiased ADAM.The paper focuses on relative improvement over standard baselines or simply training longer.
  • Method comparisons: Intermediate-task training with additional annotated data continues to produce consistent improvement over average performance across datasets.The intermediate task is MNLI.
  • Method comparisons: LLRD and Mixout have less consistent performance impacts, while pre-trained and conventional weight decay show no noticeable improvement in these experiments.The observed differences suggest the surveyed methods may partly address optimization difficulty caused by omitting debiasing in BERTADAM.
  • Overall comparison: Several methods do not outperform standard baselines or simply training longer.This comparison is central to the paper’s reassessment of proposed stability techniques.

8 CONCLUSION

The paper finds that optimization choices strongly affect few-sample BERT fine-tuning, and recommends debiasing ADAM, re-initializing top layers, and training beyond three epochs.

  • Debiasing omission in BERTADAM is identified as the main cause of commonly observed degenerate models on small datasets.
  • Re-initializing the top pre-trained BERT layers speeds learning and improves model performance by avoiding a detrimental initialization.
  • The common one-size-fits-all three-epochs practice is sub-optimal, while allocating more training time can stabilize fine-tuning.
  • The positive effects of previously proposed stabilization methods are reduced when fine-tuning uses debiased ADAM.
  • Future work will examine different pre-training objectives, model architectures, and parameter evolution during fine-tuning.

A DATASETS

The experiments use GLUE datasets spanning entailment, paraphrase, similarity, acceptability, and sentiment tasks. On RTE, fine-tuning randomness is controlled across initialization, data order, and Dropout to assess instability and Re-init 5.

  • Datasets: The evaluation covers GLUE tasks including entailment, paraphrase, sentence similarity, grammatical acceptability, question equivalence, and sentiment classification.RTE, MRPC, STS-B, CoLA, MNLI, QQP, and SST-2 provide varied classification and regression settings.
  • Randomness control: BERT fine-tuning randomness comes from weight initialization, data order, and Dropout regularization.The RTE experiment separates initialization randomness from data-order and Dropout randomness using two random-number generators.
  • Re-initialization: Re-init 5 usually improves RTE validation accuracy and consistently reduces instability across the controlled randomness sources.The figure reports reduced instability regardless of whether initialization, data order, or Dropout is controlled.

C MIXED PRECISION TRAINING

Mixed precision preserves BERT fine-tuning performance while accelerating computation, whereas bias correction improves convergence and validation behavior in few-sample settings. On full MNLI, bias correction does not significantly change average accuracy.

  • Mixed precision: Mixed precision matches single-precision test performance, with no statistically significant difference.The experiments use 16-bit computation through Apex O1-level optimization in Transformers.
  • Large-data setting: On the full MNLI dataset, bias correction produces no significant difference in average accuracy.The paper reports no disadvantage to applying the proposed procedure on large datasets.
  • Bias correction: Bias correction speeds convergence and reduces the range of training loss during fine-tuning.This pattern is shown across 50 random trials.
  • Bias correction: Bias correction consistently produces faster convergence and improved expected validation performance as the number of random trials increases.The expected validation performance rises monotonically with additional random trials.

F SUPPLEMENTARY MATERIAL FOR SECTION 5

The supplementary experiments examine Re-init across datasets, model architectures, parameters, and training durations. They find dataset-dependent layer choices, stronger worst-case gains, and reduced differences after longer training, while Re-init remains better on MNLI.

  • Effect of L on Re-init: Re-init improves worst-case performance across random trials, but the best number of re-initialized layers differs by dataset.These trends are observed across eight downsampled datasets.
  • Effect of Re-init on Model Parameters: The supplementary analysis tracks how different Transformer blocks change during fine-tuning on RTE, MRPC, STS-B, and CoLA.This evaluates the parameter-level effect of Re-init.
  • Effect of Re-init on Other Models: Re-init is evaluated on XLNetLarge, RoBERTaLarge, BARTLarge, and ELECTRALarge, covering diverse architectures, objectives, data, and training strategies.These models extend the analysis beyond BERTLarge.
  • Hyperparameters and performance: The supplementary hyperparameter comparison reports official fine-tuning settings for BERT and its variants.Average test performance with standard deviation is compared across four small datasets and four pretrained models.
  • Training duration: Longer training generally improves fine-tuning performance and narrows the gap between standard fine-tuning and Re-init.On MNLI, Re-init still outperforms standard fine-tuning after longer training.

H EXPERIMENTAL DETAILS IN SECTION 7

The experiments specify search spaces for three fine-tuning methods and track L2 distance from initialization across four tasks.

  • The experimental hyperparameter search space is explicitly allocated for each evaluated method.
  • Layerwise Learning Rate Decay searches initial learning rates of 2×10−5, 5×10−5, and 1 × 10−4 with decay rates of 0.9 and 0.95.
  • Mixout tunes the mixout probability p across 0.1, 0.3, 0.5, 0.7, and 0.9.
  • Weight decay toward the pre-trained weight tunes regularization strength λ across 10−3, 10−2, 10−1, and 100.
  • Figures 15–18 report L2 distance to initialization during BERT fine-tuning on RTE, MRPC, STS-B, and CoLA, respectively.
Loading 2006.05987v3…