Source-linked AI summary
Knowledge Inheritance for Pre-trained Language Models
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
TL;DR
Training large PLMs is computationally demanding, and existing PLMs are often trained individually from scratch. This paper introduces knowledge inheritance, which uses distillation from existing PLMs during pre-training; it reports benefits for training larger models, knowledge transfer, and domain adaptation.
Problem
Training large PLMs requires substantial computation, while existing well-trained PLMs are generally not reused to train larger models.
Source: §1 · p. 1Method
Results
The paper reports KI’s feasibility for training larger PLMs, knowledge transfer across growing model sizes, and benefits from smaller domain teachers during domain adaptation.
Source: §5 · pp. 8–9Takeaways & Limitations
KI offers a way to share and exchange knowledge learned by different PLMs and support their continued performance improvement.
Source: §5 · p. 9The paper identifies more efficient absorption of teacher knowledge and KI across different vocabularies, languages, objectives, and modalities as future directions.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.
1 Introduction
Training larger PLMs can improve performance but demands substantial computation, while existing models are usually trained independently from scratch. The paper proposes knowledge inheritance (KI), which distills an existing PLM during pre-training, and investigates its feasibility, sequential transfer, teacher settings, and domain adaptation.
1 Introduction
- Training large-scale PLMs requires tremendous computational resources, while many well-trained PLMs are available but typically not reused for training larger models.The paper frames this as both a cost concern and an opportunity to use existing models.Source: §1 · p. 1
- KI applies knowledge distillation from an existing small PLM during pre-training to help train a larger PLM.The paper relates this intuition to reversed KD in computer vision.Source: §1 · p. 1
- The paper investigates whether KI benefits large-model pre-training, supports transfer across successively larger PLMs, depends on teacher settings, and helps domain adaptation.The questions cover architecture, training data, and strategies as possible teacher settings.Source: §1 · p. 2
- The paper formulates KI and reports that previously trained PLMs can help train larger models efficiently, with learned knowledge passed down across generations.It also analyzes how teacher pre-training settings may affect KI and inform teacher selection.Source: §1 · p. 2
2 Related Work
Prior work accelerates PLM pre-training through training techniques and progressive model growth, while knowledge distillation has been studied across training phases. This paper turns to distillation as a flexible way to transfer knowledge during pre-training.
2 Related Work
- Prior efficiency methods include mixed-precision and distributed training, large-batch optimization, and progressive pre-training.Progressive approaches first train a small PLM, then increase network depth or width using parameter recycling.Source: §2 · p. 2
- Progressive pre-training methods grow a model through parameter recycling, but typically require compatible architectures and do not support multiple teachers or domain adaptation as KI does.The paper instead uses KD without architecture constraints.Source: §2 · p. 2
- PLM knowledge distillation has been explored during pre-training, fine-tuning, or both, and can transfer logits or intermediate hidden representations.Conventional KD treats the teacher as pivotal and generally expects students not to match teacher performance; KI considers larger students and reversed KD.Source: §2 · p. 2
3 Knowledge Inheritance
KI trains a larger PLM on its own pre-training task while using a well-trained smaller PLM as an auxiliary source of knowledge. The method balances self-learning with teacher matching and decreases teacher guidance over training.
3 Knowledge Inheritance
- The setup takes textual inputs and task labels, with the number of classes set by the pre-training task, such as vocabulary size for masked language modeling.The larger model is trained on its corpus using a well-trained smaller PLM as the knowledge source.Source: §3 · pp. 2–3
- The larger PLM is trained using both its self-learning objective and a teacher-matching objective based on KL divergence between their output distributions.The two losses are balanced by an inheritance rate.Source: §3 · p. 3
- The method gradually reduces the inheritance rate so the student increasingly learns on its own and eventually switches to pure self-learning.The paper motivates this schedule by the larger model's expected progress and eventual surpassing of its teacher.Source: §3 · p. 3
- Teacher logits for the training corpus can be computed once and cached offline, avoiding repeated teacher inference while training the larger model.The passage describes this as a one-time process.Source: §3 · p. 3
4 Empirical Analysis
The empirical analysis tests whether knowledge inheritance (KI) improves large-model pre-training, accumulates knowledge across generations, depends on teacher settings, and helps domain adaptation. Across these settings, the reported results favor KI, with transfer shaped by teacher-student training choices.
4.1 RQ1: How Could Knowledge Inheritance Benefit Large PLMs’ Training?
- 3.41 validation PPL for BASE → LARGE versus 3.58 for LARGE; KI also improves downstream performance over baseline at each step.After teacher guidance stops, BASE → LARGE reaches 4.28 PPL versus 4.60 for LARGE, equivalent to the baseline’s 55k-step result with roughly 27.3% lower computational cost.Source: Figure 1 · p. 4, §4.1 · p. 4
- Linear annealing converges fastest; Heviside rises after teacher handoff, constant guidance becomes worse after the student surpasses its teacher, and top-K=10 retains most useful information.Saving and re-normalizing only top-K teacher probabilities reduces the memory footprint without much performance decrease.Source: §4.1 · pp. 4–5
4.2 RQ2: Could Knowledge Inheritance be Performed over Generations?
- Across three RoBERTa generations, KI outperforms self-training, and involving both earlier generations lets G3 benefit from accumulated knowledge.The tested generations grow roughly 1.7× in model size; G2 refines G1’s knowledge before passing it to G3.Source: §4.2 · p. 5
4.3 RQ3: How Could MS’s Pre-training Setting Affect Knowledge Inheritance?
- Deeper or wider teachers accelerate student convergence, and larger teachers require longer optimal periods of teacher guidance.The experiments vary teacher depth from 4 to 10 layers and width from 384 to 672 hidden units.Source: §4.3 · p. 6
- KI performance is not substantially undermined until the teacher uses only 1/16 of the original data, despite weaker teacher PPL with less data.The reported conclusion is that data-size impact can be ignored unless the larger model’s training data is extensively enlarged.Source: §4.3 · pp. 6–7
- KI benefits become more obvious as teacher data approaches the student’s domain, while zero data overlap is not a serious issue when both share a domain.The privacy experiment compares teachers trained on disjoint halves of the same-domain corpus.Source: §4.3 · p. 7
4.4 RQ4: How Could Knowledge Inheritance Benefit Domain Adaptation?
- For domain adaptation, KI yields lower final PPL and better domain-task performance than self-learning; simultaneous two-domain adaptation improves both domains but can overfit and trails one-domain-at-a-time adaptation.The reported advantage is especially relevant when domain-specific data is scarce.Source: §4.4 · p. 8, Table 2 · p. 8, Table 3 · p. 8
5 Conclusion and Future Work
The paper presents knowledge inheritance (KI) as a framework for training larger PLMs from existing ones, and extends it to knowledge transfer during domain adaptation. It identifies further work on KI efficiency, broader settings, and fair benchmarking.
5 Conclusion and Future Work
- KI leverages previously trained PLMs to train larger ones and supports knowledge transfer across a series of PLMs with growing sizes.The paper also analyzes teacher pre-training settings and applies KI to domain adaptation.Source: §5 · pp. 8–9
- During domain adaptation, a trained large PLM can benefit from smaller domain teachers.The paper frames this as a way to share and exchange knowledge learned by different models.Source: §5 · p. 9
- Future work includes improving knowledge absorption under limited computational budgets and testing KI across differing vocabularies, languages, objectives, and modalities.The paper suggests denoising teacher predictions, using more teacher information, and selecting representative data points.Source: §5 · p. 9
- The paper calls for fairer benchmarking through shared computation settings, detailed reporting, multiple evaluation seeds, diverse downstream tasks, frequent checkpoints, and open releases.Source: §5 · p. 9
Appendices
The appendix examines KI across four PLMs whose sizes grow by roughly 1.7× at each step, finding benefits across the tested transitions.
Appendices
- The four models are RoBERTaMEDIUM (73.5M), RoBERTaBASE (125M), RoBERTaBASE_PLUS (211M), and RoBERTaLARGE (355M).Source: §A.1 · p. 12
- KI is superior to self-learning across the tested model transitions, and its absolute PPL gap grows as the teacher and learner sizes increase.Source: §A.1 · p. 12
- Each teacher is pre-trained for 125k steps with batch size 2,048, then its knowledge is inherited by the next larger model and compared with self-learning from the start.Source: §A.1 · p. 12
A.2 Effects of MS’s Pre-training Steps
These experiments find that teacher training, student batch size, and teacher width affect KI’s training benefits. Longer-trained and wider teachers help, while KI accelerates convergence across batch sizes.
A.2 Effects of MS’s Pre-training Steps
- Longer teacher pre-training helps the student converge faster, but the benefit diminishes as teacher steps increase and validation PPL reaches a plateau.The tested RoBERTaMEDIUM teachers used 62.5k, 125k, 250k, or 500k steps; after 125k steps, the paper's models were well converged and further gains were limited.Source: §A.2 · p. 12, Figure 4 · p. 13
A.3 Effects of ML’s Batch Size
- KI accelerates convergence at every tested student batch size, with larger final validation PPL improvements at smaller batch sizes.The tested batch sizes were 256, 512, 1,024, and 2,048, while increasing batch size also improved final validation PPL.Source: §A.3 · p. 12, Figure 4 · p. 13
A.4 Additional Experiments of the Effects of Teacher Model MS’s architecture (width)
- A wider teacher improves the student PLM’s training efficiency across the tested hidden sizes.Figure 5’s left panel shows validation PPL for teacher hidden sizes 384, 480, 576, and 672.Source: §A.4 · pp. 12–13, Figure 5 · p. 13
A.5 Additional Experiments of Knowledge Inheritance for Domain Adaptation
In domain adaptation, KI outperforms self-learning across the tested target-domain settings, with larger gains when less domain data is available. The experiments also report greater source-domain forgetting under KI.
A.5 Additional Experiments of Knowledge Inheritance for Domain Adaptation
- KI consistently outperforms self-learning on target-domain validation PPL, and its advantage grows as domain data decreases, while self-learning overfits on smaller corpora.The experiments vary post-training from 1k to 5k steps and test corpora of 3,400M, 200M, 100M, and 40M tokens.Source: §A.5 · p. 13
- After adaptation, source-domain validation PPL increases, indicating forgetting of source knowledge that is more evident with KI than self-learning.Table 4 reports source-domain PPL after post-training RoBERTaBASE_WB on CS or BIO with either strategy.Source: §A.5 · pp. 13–14, Table 4 · p. 13
A.6 Detailed Downstream Performances on GLUE Tasks
Knowledge inheritance reaches near-final GLUE performance in fewer pre-training steps, while the architecture experiments show that teacher size relates to convergence and downstream performance.
A.6 Detailed Downstream Performances on GLUE Tasks
- RoBERTaBASE → RoBERTaLARGE downstream performance rose faster than the baseline across six GLUE tasks, requiring fewer steps to reach a high score.MNLI and SST-2 were the most stable tasks in these curves.Source: §A.6 · p. 14
- 42.9% FLOPs were saved when the baseline at 70k steps matched the method at 40k steps on average GLUE performance.The authors note that this exceeds the 27.3% savings estimated using pre-training PPL.Source: §A.6 · p. 14, Table 5 · p. 15
- 44% FLOPs were saved: RoBERTaBASE → RoBERTaLARGE reached almost the baseline’s final-step GLUE performance after 70k steps instead of 125k.Table 5 compares average GLUE performance at different pre-training steps.Source: §A.6 · p. 14, Table 5 · p. 15
A.7 Teacher Models’ Validation PPL Curves during Pre-training for “Effects of Model Architecture”
- At the same pre-training step, learning from teacher models with more parameters helped achieve better MNLI performance, aligning downstream gains with validation PPL.Figure 7 shows architecture-variation downstream performances and teacher validation PPL curves.Source: §A.6 · p. 14, Figure 7 · p. 15
- Larger teacher PLMs generally converged faster and achieved better final performance across the tested depth and width variations.The experiments varied teacher depth across 4, 6, 8, and 10 layers and width across 384, 480, 576, and 672.Source: §A.7 · p. 15
B Pre-training Hyper-parameters
The appendix records the pre-training setup, fine-tuning hyper-parameters, and vocabulary-overlap measure used to describe domain proximity.
B Pre-training Hyper-parameters
- Pre-training used batch size 2,048, a 199:1 training-validation data ratio, Adam, linear learning-rate decay, and 10% warmup.The appendix also points to Tables 6 and 7 for model architectures and teacher-guided training steps.Source: B Pre-training Hyper-parameters · p. 15
C Fine-tuning Hyper-parameters
- Fine-tuning hyper-parameters for ACL-ARC, CHEMPROT, and GLUE are listed in Table 8 and follow Liu et al. (2019) and Gururangan et al. (2020).The supplied passage describes the table and its sources, not the parameter values.Source: C Fine-tuning Hyper-parameters · p. 15
D Domain Proximity of WB, CS, and BIO
- Table 9 lists the domain proximity, measured as vocabulary overlap, for WB, CS, and BIO used in this paper.Source: D Domain Proximity of WB, CS and BIO · p. 15
E Comparison between Knowledge Inheritance and Parameter Recycling
The comparison presents parameter recycling as progressive size growth through parameter initialization, contrasting its architectural constraints with knowledge inheritance’s broader transfer settings.
E Comparison between Knowledge Inheritance and Parameter Recycling
- Parameter recycling progressively grows a small PLM’s depth or width using parameter initialization, whereas knowledge inheritance does not require matched teacher and student architectures.Existing methods including Gong et al. (2019) and Gu et al. (2021) impose architecture matching requirements.Source: E Comparison between Knowledge… · pp. 15–16
- Knowledge inheritance can absorb knowledge from multiple teachers simultaneously, unlike parameter recycling’s difficulty with jointly recycling multiple models’ parameters.Source: E Comparison between Knowledge… · p. 16
- Knowledge inheritance supports continual domain adaptation by letting larger models incorporate knowledge from locally trained domain experts.The described experts are trained on computer science and biomedical data.Source: E Comparison between Knowledge… · pp. 16–17
- Knowledge inheritance needs no access to teacher parameters: predictions can be precomputed and saved offline, supporting API-based online knowledge transfer.Source: E Comparison between Knowledge… · p. 17
F Comparing Label Smoothing and Knowledge Inheritance
The comparison tests whether knowledge inheritance’s gains might come from learning smoothed targets; the reported results distinguish it from label smoothing.
F Comparing Label Smoothing and Knowledge Inheritance
- To test whether KI’s success comes from learning a more-smoothed target, the authors compare label smoothing with KI; α=0 means no smoothing, and larger α yields a more-smoothed target.Source: F Comparing Label Smoothing and… · p. 17
- Label smoothing led to far worse validation PPL than the vanilla baseline, while the authors attribute KI’s improvement to inheriting teacher knowledge rather than smoothing.Table 10 compares validation PPL for RoBERTaBASE trained with different strategies, using RoBERTaMEDIUM as the KI teacher.Source: F Comparing Label Smoothing and… · p. 17, Table 10 · p. 17