PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

Cheolhyoung Lee, Kyunghyun Cho, Wanmo Kang

arXiv:1909.11299v2cs.LGstat.ML

TL;DR

Finetuning large pretrained language models can become unstable on small downstream datasets. The paper introduces mixout, which stochastically mixes current parameters with a target model and adaptively regularizes deviation toward that target. Experiments report improved finetuning stability and average accuracy for BERT on GLUE tasks, with additional wall-clock cost.

  • Problem

    Finetuning large pretrained language models can fail when downstream datasets contain fewer than 10,000 training instances.

  • Method

    Mixout stochastically mixes current parameters with parameters from a target model and regularizes learning toward that target.

  • Results

    Mixout reduces unusable models and improves average development scores and finetuning stability for BERTLARGE across small GLUE tasks.

  • Takeaways & Limitations

    Mixout provides a target-centered regularization approach for finetuning pretrained language models, including models such as BERT and RoBERTa.

  • Takeaways & Limitations

    Mixout takes longer to finetune than original dropout, although the increase is described as not significant relative to failed dropout runs.

Abstract

from arXiv · show

In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

1 INTRODUCTION

Finetuning large pretrained language models can fail on small downstream datasets. The paper introduces mixout, a dropout-motivated regularizer that mixes current parameters with a target model’s parameters and improves BERT finetuning stability and scores.

  • Finetuning BERTLARGE sometimes fails when downstream datasets contain fewer than 10,000 training instances.
  • Mixout generalizes dropout by replacing randomly eliminated outgoing parameters with corresponding parameters from a target model.
  • Mixout(wpre) reduces unusable BERTLARGE models and increases average development scores across small GLUE tasks.
  • Dropout and related prior analyses characterize regularization toward the origin, whereas this work extends the target point beyond zero.
  • Prior work used pretrained-parameter penalties or tuneout, but did not establish their theoretical justification or empirical significance for finetuning stability.

2 PRELIMINARIES AND NOTATIONS

The preliminaries define the paper’s norm and loss notation, strong convexity, weight decay, and dropout conventions. Dropout randomly removes neurons, while inverted dropout rescales retained weights during training.

  • Unless otherwise stated, ∥·∥ denotes the L2-norm, and L(w) denotes the neural network loss over trainable parameters w.
  • A differentiable function is strongly convex when it satisfies a positive quadratic lower-curvature bound.
  • Usual weight decay with coefficient λ is equivalent to penalization toward the origin, written as wdecay(0, λ).
  • Dropout(p) randomly drops a neuron with probability p, while dropconnect(p) randomly drops an individual parameter with probability p.
  • Inverted dropout uses w/(1−p) during training so separate expected-output computation at test time is unnecessary.

3 ANALYSIS OF DROPOUT AND ITS GENERALIZATION

The analysis generalizes dropout through random parameter mixtures around a target model. Under strong convexity, the resulting objective behaves like adaptive L2 regularization toward that target, motivating mixout for pretrained models.

  • Dropout generalization: The analysis begins with dropconnect and represents its SGD dynamics using independent Bernoulli masks with drop probability p.
  • Random mixtures: Mixconnect forms a random mixture of current parameters w and target parameters u using a mask matrix with specified mean and variance.
  • Theoretical result: Under strong convexity, mixconnect minimizes an objective equivalent to an L2 regularizer with coefficient mσ2/µ2.
  • From mixconnect to mixout: Mixout is a correlated Bernoulli-mask special case of mixconnect, replacing outgoing parameters neuron-wise and allowing application to selected layers.
  • Adaptive regularization: For mixout(u,p), the L2 coefficient mp/(1−p) grows without bound as p approaches 1, while its coefficient depends on the current model through m.
  • Mixout for pretrained models: Because pretrained parameters are typically far from the origin, ordinary dropout penalizes deviation toward zero; mixout(wpre) instead targets the pretrained parameters.

4 VERIFICATION OF THEORETICAL RESULTS FOR MIXOUT ON MNIST

The MNIST-to-EMNIST experiment verifies that mixout keeps finetuned parameters closer to the pretrained model while retaining target-task performance and source-task accuracy better than dropout.

  • Verification of theoretical results: Mixout keeps the finetuned solution closer to wpre in the L2 sense than dropout.The experiment compares parameter deviation, MNIST accuracy, and EMNIST Digits accuracy across 10 random restarts.
  • Target-task performance: Both mixout and dropout achieve high MNIST validation accuracy for p ∈{0.1, 0.2, 0.3}.
  • Target-task performance: Mixout is more robust to the choice of p than dropout on the MNIST target task.
  • Source-task retention: On EMNIST Digits, dropout accuracy drops by 0.041, 0.074, and 0.105 for p ∈{0.1, 0.2, 0.3}.Mixout accuracy drops from 0.992 to approximately 0.723 regardless of p, with smaller declines than dropout at those probabilities.

5 FINETUNING A PRETRAINED LANGUAGE MODEL WITH MIXOUT

The paper evaluates mixout for unstable BERTLARGE finetuning on small GLUE training sets. Across repeated restarts, mixout reduces degenerate runs and increases average dev scores, while combining it with pretrained-weight decay can further improve stability and scores.

  • Experimental setup: The study finetunes BERTLARGE with mixout on the unstable small-data GLUE tasks RTE, MRPC, CoLA, and STS-B.The comparison includes standard dropout and weight decay, pretrained-weight decay, mixout, and their combination.
  • Finetuning stability: Mixout significantly decreases degenerate BERTLARGE configurations that fail at chance-level accuracy across the evaluated tasks.The analysis uses dev-score distributions from 20 random restarts with shared random initializations.
  • Finetuning stability: On CoLA, mixout(wpre, 0.7) yields one degenerate model configuration versus seven for Devlin et al.’s and six for Wiese et al.’s strategies.
  • Dev-score performance: Mixout(wpre, {0.7, 0.8, 0.9}) increases mean dev scores for all tasks; on CoLA, mixout(wpre, 0.8) reaches 57.9 versus 38.8.The reported CoLA result is a 49.2% increase over the baseline using dropout and zero-target weight decay.
  • Dev-score performance: Mixout improves best dev scores for all tasks over the standard dropout and zero-target weight-decay baseline.For MRPC, mixout(wpre, 0.9) improves the maximum dev score by 0.9.
  • Combined regularization: Combining mixout(wpre) with wdecay(wpre, 0.01) improves both average and best dev scores over the compared baseline strategies.The combination further improves stability in some tasks, although its effect on degenerate configurations varies by task.

6 ABLATION STUDY

The ablations examine mixout with larger training sets, additional output-layer regularization, and varying mix probabilities. Mixout is most beneficial in low-data settings, while its advantage narrows with sufficient data and incurs extra computation.

  • 6.1 MIXOUT WITH A SUFFICIENT NUMBER OF TRAINING EXAMPLES: On a larger SST-2 training set, mixout and dropout produce little difference in mean and maximum dev scores across 20 restarts.
  • 6.2 EFFECT OF A REGULARIZATION TECHNIQUE FOR AN ADDITIONAL OUTPUT LAYER: Mixout(w0, 0.7) on the randomly initialized output layer improves both average and best dev scores on RTE, CoLA, and STS-B.The pretrained layers use mixout(wpre, 0.7).
  • 6.2 EFFECT OF A REGULARIZATION TECHNIQUE FOR AN ADDITIONAL OUTPUT LAYER: The highest mean and best dev scores on RTE, CoLA, and STS-B use mixout(wpre, 0.7) for pretrained layers and mixout(w0, 0.7) for the output layer.MRPC uses different settings for its highest mean and best scores.
  • 6.3 EFFECT OF MIX PROBABILITY FOR MIXOUT AND DROPOUT: As p increases, mixout mean dev scores rise whereas dropout mean dev scores fall; mixout(wpre, {0.7, 0.8, 0.9}) outperforms dropout for every p.These mixout settings also significantly reduce runs failing at chance-level accuracy.
  • 6.3 EFFECT OF MIX PROBABILITY FOR MIXOUT AND DROPOUT: Mixout takes longer than dropout because it performs additional computation with the pretrained parameter.

7 CONCLUSION

The conclusion presents mixout(wpre) as an adaptive regularizer toward pretrained parameters that improves finetuning stability and accuracy in low-data settings.

  • Mixout(wpre) adapts its L2 regularization coefficient along the optimization path, unlike fixed wdecay(wpre).
  • Experiments show improved finetuning stability, average accuracy, and best accuracy for large pretrained language models with few target-task examples.
  • The approach is architecture-independent and could be applied to pretrained models such as RoBERTa and XLNet.The paper leaves those applications as future work.

A PROOFS FOR THEOREM 1

The proof analyzes random parameter mixing under strong convexity and extends the interpretation from all parameters to selected layers. It characterizes mixout as adaptive weight decay toward a target parameter.

  • Theorem 1 assumes a strongly convex loss and a random mixture of parameters from u governed by a diagonal mask M.
  • The mask entries satisfy E M_i = µ and Var(M_i) = σ^2 for every parameter coordinate.
  • The proof uses strong convexity and the mixture moments to derive the theorem's regularized-loss expression.
  • When applied to selected layers, mixconnect and mixout correspond to adaptive weight decay toward the target parameters of those layers.

C.3 TEST RESULTS ON GLUE TASKS

The test evaluation compares mixout against the original dropout and weight-decay strategy for BERTLARGE on GLUE tasks. Mixout achieves higher test scores on every listed task except MRPC.

  • The evaluation uses best models selected on dev sets after 20 random restarts with BERTLARGE finetuning.
  • The proposed mixout applies mixout(wpre, 0.7) to pretrained layers and mixout(w0, 0.7) to the additional output layer.
  • Mixout improves test scores over Devlin et al.'s original regularization strategy on all evaluated tasks except MRPC.

D VERIFICATION OF COROLLARY 1.1 WITH LEAST SQUARES REGRESSION

Least squares experiments support Corollary 1.1: increasing mixout probability pulls the fitted parameter toward the target model parameter rather than the true parameter. The mixout configuration using the pretrained parameter also incurs additional finetuning time compared with dropout.

  • Theoretical prediction: Increasing p strengthens mixout’s regularization toward the target model parameter u under a strongly convex loss.The corollary states that the deviation from u is minimized more strongly as p increases.
  • Synthetic verification: As p increases from 0.0 to 0.9, the fitted regression line converges toward the target model’s line rather than the true data-generating line.This behavior is shown on a synthetic least-squares regression dataset.
  • Computational cost: Mixout(wpre) took about 843 seconds versus about 636 seconds for dropout when finetuning BERTLARGE, or 32.5% more time.The extra computation uses the pretrained model parameter wpre.

F EXTENSIVE HYPERPARAMETER SEARCH FOR DROPOUT

The dropout hyperparameter was evaluated across four GLUE tasks using mean dev scores from repeated BERTLARGE finetuning runs. Dropout(0.1) was highest on MRPC and STS-B and nearly optimal across all tasks.

  • Overall conclusion: Across RTE, MRPC, CoLA, and STS-B, the experiments identify dropout(0.1) as almost optimal for average dev score.The evaluation compares dropout probabilities from 0.0 through 0.5.
  • Results by task: Dropout(0.1) achieved the highest average dev scores on MRPC and STS-B.The scores were measured across 20 random restarts.
  • Results by task: Dropout(0.1) achieved the second-highest average dev scores on RTE and CoLA, behind dropout(0.0).This makes 0.1 nearly optimal for those tasks in terms of mean dev score.
Loading 1909.11299v2…