PaperlayerView exact revision on arXiv

Source-linked AI summary

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Iulia Turc, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

arXiv:1908.08962v2cs.CL

TL;DR

Large pre-trained models are costly to deploy, while direct pre-training and fine-tuning of compact models had been overlooked. The paper combines compact-model pre-training with teacher distillation and optional fine-tuning, finding competitive or improved performance across model sizes and unlabeled-data conditions, including a compounding effect when both operations use the same data.

  • Problem

    The paper addresses how to obtain the gains of large pre-trained models under restricted memory and latency budgets, where direct pre-training and fine-tuning of compact models had been overlooked.

  • Method

    Pre-trained Distillation sequences masked LM pre-training, task-specific distillation from a large fine-tuned teacher, and optional fine-tuning.

  • Results

    Pre-trained Distillation is competitive with or better than more elaborate approaches, improves over the pre-training+fine-tuning baseline, and shows compounding benefits when pre-training and distillation are applied sequentially.

  • Takeaways & Limitations

    Pre-training remains important for compact students, while combining it with distillation can use teacher expertise and labeled, unlabeled transfer, and language-model data.

  • Takeaways & Limitations

    Task-relevant unlabeled transfer text is not readily available for some paired-text and domain-specific tasks, and deployed inputs can shift over time.

Abstract

from arXiv · show

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

1 INTRODUCTION

The paper argues that compact models benefit from the standard pre-training-plus-fine-tuning pipeline and introduces Pre-trained Distillation, which combines pre-training, distillation, and optional fine-tuning for further gains.

  • Motivation: Pre-training and fine-tuning compact architectures directly is a competitive baseline for building small models.This baseline had been overlooked in prior work on compact language representations.
  • Method: Pre-trained Distillation sequences masked LM pre-training, task-specific distillation, and optional fine-tuning.Distillation transfers predictions from a highly accurate teacher to the compact student, including on unlabeled data.
  • Results: Pre-trained Distillation outperforms the pre-training+fine-tuning baseline, especially with a large distillation transfer set.
  • Results: Pre-trained Distillation outperforms or matches more elaborate approaches using sophisticated distillation or pre-training methods.The comparison uses controlled data and model-architecture settings from concurrent work.
  • Experiments: Across 24 models ranging from 4m to 110m parameters, pre-trained students leverage depth better than width, unlike randomly initialized models.Pre-trained Distillation is also more robust than standard distillation to transfer-set amount and similarity variations.
  • Experiments: Applying pre-training and distillation sequentially to the same dataset performs better than either operation alone.The authors describe this compounding effect as evidence that the two operations learn complementary aspects of the data.

2 PROBLEM STATEMENT

The problem is to build accurate compact Transformer models within memory and latency budgets using labeled, unlabeled transfer, and language-model data under realistic resource constraints.

  • Goal: The goal is to build accurate models that fit a specified memory and latency budget.The study fixes the architecture to bidirectional Transformers while varying training resources and model configurations.
  • Models: Teachers are highly accurate large end-task models that do not satisfy the target resource constraints.The experiments use a single teacher for faster experimentation without claiming it is architecturally optimal.
  • Models: Students are compact models designed to satisfy resource constraints, spanning 24 sizes from TransformerTINY with 4m parameters to TransformerBASE with 110m.Most figures focus on five representative models, while conclusions are checked across the full grid.
  • Data: Labeled training data consists of input-label pairs and is generally restricted because labels are difficult to produce.
  • Data: Unlabeled transfer data supports teacher-to-student distillation but may be unavailable for paired-text tasks and may become stale under distribution shift.It is generally easier to produce and larger than labeled data, but task-relevant inputs are not always readily available.
  • Data: Unlabeled language-model data enables masked-language-model pre-training and often consists of vast general-domain text collections.Strong domain similarity is not required for these corpora.

3 PRE-TRAINED DISTILLATION

Pre-trained Distillation (PD) builds compact models by combining language-model pre-training, knowledge distillation, and optional end-task fine-tuning. The approach is intended to remain simple and architecture-general while improving on pre-training plus fine-tuning.

  • Design: PD is presented as a general algorithm that can be applied to any architecture while leveraging language-model data, teacher predictions, and labeled task data.Its sequence combines standard training operations rather than architecture-specific mechanisms.
  • Algorithm: PD applies three standard operations: pre-training on DLM, distillation on transfer data, and optional fine-tuning on labeled data.The two-step version is called PD; adding final fine-tuning yields PDF.
  • Algorithm: Pre-training initializes compact students to capture linguistic phenomena before they learn teacher predictions through soft-label distillation.The pre-training step can mitigate negative effects from an imperfect unlabeled transfer set.
  • Motivation: Student pre-training remains important even when millions of transfer samples are available, despite teachers already being produced through pre-training and fine-tuning.This importance is described as surprising because the teacher might transfer pre-training knowledge through distillation.

4 COMPARISON TO CONCURRENT WORK

The paper compares PD with concurrent compression methods under matched architecture, size, and dataset settings. In that controlled setting, PD performs best on average, while the simpler PF baseline remains surprisingly competitive.

  • Compared methods: The compared concurrent methods use either intermediate teacher activations for task distillation or teacher-assisted language-model pre-training before task fine-tuning.These approaches make additional initialization, architectural, or pre-training choices beyond PF.
  • Experimental setup: Under the concurrent-work settings, PD uses a 6-layer BERT student and a 12-layer BERTBASE teacher with 768-dimensional embeddings.The transfer set coincides with the labeled set, so DT = DL.
  • Results: On six selected GLUE tasks, PD performs best on average, while PF is surprisingly competitive with more elaborate alternatives.The comparison uses the same model architecture, sizes, and dataset settings as the cited concurrent studies.
  • Results: PF preserves generality and simplicity without compromising quality in this setting, but cannot use unlabeled task data or teacher predictions.Its limitation matters when additional transfer data or teacher supervision is available.

5 ANALYSIS SETTINGS

The analysis varies model architecture and training resources across compact Transformer students, using supervised, distillation, and pre-training-plus-fine-tuning baselines on several task and dataset types. It examines how model size and unlabeled data shape these strategies.

  • Model analysis: The study evaluates 24 Transformer students spanning 4m to 110m parameters and depth/width trade-offs, reporting five representative models while checking conclusions across the full grid.The teacher is BERTLARGE with 340M parameters; the largest and smallest students differ substantially in size and latency.
  • Baselines: Basic Training directly trains a compact model on labeled data, Knowledge Distillation transfers teacher soft labels, and PF pre-trains on unlabeled general-domain text before fine-tuning.For classification, soft labels are teacher-produced class probabilities whose temperature controls distribution smoothness.
  • Baselines: The distillation baseline trains teachers by fine-tuning pre-trained BERTLARGE and trains students on teacher soft labels with temperature 12.Teachers do not learn from the transfer set.
  • Baselines: Following BERT, PF uses masked-language-model and next-sentence objectives before end-task fine-tuning.The paper studies this pre-training procedure systematically for compact architectures.
  • Tasks and datasets: The tasks include sentiment classification, natural-language inference, and textual entailment, with Amazon reviews, SST-2, MNLI, SNLI, QQP, and RTE datasets.MNLI is supplemented with SNLI and QQP because strictly in-domain proxy data is difficult to obtain; RTE tests robustness to limited transfer data.
  • Tasks and datasets: The analysis distinguishes labeled data, unlabeled transfer data, and unlabeled language-model data, while allowing the same corpus to serve multiple functional roles.DLM supports masked-language-model pre-training and generally does not require labels or strong domain similarity.

6 ANALYSIS

The analysis shows that full language-model pre-training is necessary for compact students, while distillation adds further gains and robustness. Pre-trained students exploit depth and complementary unlabeled-data signals more effectively than isolated or shortcut strategies.

  • 6.1 THERE ARE NO SHORTCUTS: WHY FULL PRE-TRAINING IS NECESSARY: Full language-model pre-training is necessary to unlock compact students’ potential; pre-trained representations alone or truncated stacks are suboptimal.Earlier strategies feeding pre-trained embeddings or truncating deeper models underperform direct LM pre-training.
  • 6.1 THERE ARE NO SHORTCUTS: WHY FULL PRE-TRAINING IS NECESSARY: Less than 24% of PD’s gains over distillation on MNLI come from pre-trained word embeddings, falling to 5% for TransformerTINY.The remaining benefit comes from additionally pre-training Transformer layers; embeddings remain trainable during distillation.
  • 6.1 THERE ARE NO SHORTCUTS: WHY FULL PRE-TRAINING IS NECESSARY: Pre-trained students favor depth over width: 6L/512H with 35.4m parameters outperforms 2L/768H with 39.2m parameters.This depth advantage is visible for PD and PF but not for randomly initialized models, which poorly exploit extra parameters.
  • 6.2 UNDER THE HOOD: DISSECTING PRE-TRAINED DISTILLATION: Pre-trained Distillation outperforms isolated basic training, distillation, and pre-training+fine-tuning across student sizes and tasks.On Amazon Book Reviews, TransformerMINI matches teacher accuracy with a 31x smaller model and 16x speed-up; on RTE, PD improves TransformerTINY by more than 5% absolute.
  • 6.2 UNDER THE HOOD: DISSECTING PRE-TRAINED DISTILLATION: PD is more robust to transfer-set size and domain mismatch: it matches teacher performance with TransformerMINI using 5m instances, versus distillation’s 8m.On out-of-domain data, distillation degrades more sharply, while PD retains smaller losses and can gain from final fine-tuning.
  • 6.3 BETTER TOGETHER: THE COMPOUND EFFECT OF PRE-TRAINING AND DISTILLATION: Applying pre-training and distillation sequentially on the same data outperforms either operation alone, indicating complementary learned aspects.For MNLI, PD is 2.2% better than PF on average across student sizes.

7 RELATED WORK

The paper situates its approach among work on pre-training, compact-model compression, and methods combining distillation with unsupervised pre-training.

  • Pre-training: Pre-training research progressed from word embeddings to contextual representations and fine-tuning entire large pre-trained models for downstream tasks.
  • Learning compact models: Model-compression work includes knowledge distillation and methods that transfer information through intermediate layer activations.
  • Learning compact models: Intermediate-activation approaches produced only slight gains in the authors’ experiments, dominated by gains from pre-training and not complementary to them.
  • Distillation with unsupervised pre-training: Some approaches combine distillation with unsupervised pre-training by providing pre-trained word embeddings or initializing Transformer students from BERT layers.

8 CONCLUSION

The paper uses extensive experiments to study pre-training and knowledge distillation separately and jointly. It finds that their benefits compound and presents Pre-trained Distillation as a way to use a powerful teacher and multiple data sources.

  • Extensive experiments examine pre-training and knowledge distillation in isolation and their interaction.
  • The benefits of pre-training and distillation compound when combined.
  • Pre-trained Distillation uses a powerful teacher together with labeled sets, unlabeled transfer sets, and unlabeled language-model sets.
Loading 1908.08962v2…