Source-linked AI summary
Pre-Trained Models: Past, Present and Future
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, Jun Zhu
TL;DR
PTMs address the challenge of transferring learned knowledge to downstream tasks with limited samples, using large-scale pre-training over labeled and unlabeled data. This paper traces their history and relation to transfer learning and self-supervised learning, reviews advances across four directions, and identifies open problems for future development.
Problem
Training models from scratch for new tasks can require large amounts of data, motivating methods that capture reusable knowledge for targets with limited samples.
Method
The paper traces pre-training's history and position in AI, then comprehensively reviews PTM advances in architectures, contexts, efficiency, interpretation, and theory.
Results
Existing large-scale PTMs achieve promising results, including strong zero-shot and few-shot abilities, across language understanding and generation tasks.
Takeaways & Limitations
PTMs occupy a crucial position in AI research, while their continued development remains an open question requiring further study across multiple research directions.
Takeaways & Limitations
PTM robustness remains a serious security threat for real-world applications because irrelevant artifacts can mislead models into wrong predictions.
Abstract
from arXiv · showhide
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.
1 Introduction
Large-scale PTMs address data scarcity by learning from massive corpora and transferring stored knowledge to downstream tasks. The section presents their rapid progress, expanding scale, broad capabilities, and unresolved interpretive and computational challenges.
- Motivation: Limited task-specific datasets make manually annotating large-scale data expensive, time-consuming, and infeasible for every AI task.The passage notes that even large datasets cannot be built for all tasks.
- Transfer learning: Transfer learning pre-trains on source tasks and fine-tunes on target tasks, enabling models to handle new problems with limited samples.Its rationale is transferring previously acquired knowledge instead of training from scratch.
- Self-supervised learning: Self-supervised learning uses intrinsic correlations in unlabeled text as supervision, capturing versatile linguistic knowledge without labor-intensive annotation.Masked-word prediction is given as an example of this setting.
- Large-scale PTMs: Transformers combined with language-model objectives enabled GPT and BERT to capture lexical, syntactic, and factual knowledge in large-scale PTMs.Fine-tuning these models with few samples improves downstream NLP performance.
- Progress: Large-scale PTMs improve both language understanding and generation, while their parameters and pre-training data have expanded rapidly alongside growing research activity.Figure 2 reports parameter and data sizes increasing by 10 times per year.
- Open problems: Despite strong performance across AI tasks, the knowledge hidden in huge parameter sets and the computational cost of training remain fundamental open issues.The paper frames these challenges as motivating multiple future research directions.
- Paper scope: The paper traces pre-training’s history, reviews architectures, contexts, efficiency, and interpretation, and discusses open problems and future directions.Its organization follows these four lines of current PTM development.
2 Background
Pre-training developed from transfer learning and supervised representations toward self-supervised learning on large unlabeled datasets. Transformer-based PTMs then became reusable backbones across NLP and related AI tasks.
- Transfer learning: Early pre-training was framed as transfer learning: knowledge captured from source tasks is applied to target tasks through a pre-training and fine-tuning framework.Source and target tasks can differ in domains and settings while sharing useful knowledge.
- Transfer mechanisms: Feature transfer pre-trains representations, whereas parameter transfer reuses model parameters; both approaches laid foundations for modern PTMs.ELMo and BERT are presented as examples of representation and parameter transfer, respectively.
- Supervised pre-training: Supervised pre-training advanced computer vision, with ImageNet-pre-trained CNNs serving as backbones for classification, detection, segmentation, captioning, and related tasks.ResNet pre-training on ImageNet is identified as a crucial step for accurate results on most CV tasks.
- Self-supervised learning: Self-supervised learning extracts knowledge from unlabeled data by using the input itself as supervision, addressing the impracticality of annotating massive textual datasets.It remains within supervised-style paradigms such as classification and generation.
- Transformer-based PTMs: Transformer-based PTMs such as GPT and BERT became task backbones after pre-training on large textual corpora, achieving state-of-the-art results across almost all NLP tasks.Later models include XLNet, RoBERTa, BART, and T5.
- Cross-domain expansion: Self-supervised learning and Transformers also advanced computer vision, outperforming conventional supervised CNNs in preliminary efforts and motivating multimodal PTMs.The paper characterizes self-supervised pre-training as the focus of current AI research.
3 Transformer and Representative PTMs
Transformer provides the nonrecurrent encoder–decoder architecture underlying GPT and BERT, while their attention patterns and self-supervised objectives define distinct PTM designs. These models establish the pre-training and fine-tuning framework and demonstrate strong downstream performance.
- Transformer: Transformer is a nonrecurrent seq2seq architecture with stacked encoder and decoder blocks, combining multi-head attention, feed-forward layers, residual connections, and normalization.Decoder blocks additionally use cross-attention to incorporate encoder outputs as generation context.
- Transformer: Scaled dot-product attention compares queries with keys, applies masking when needed, and returns weighted combinations of value vectors.The matrix form produces an attention matrix A and output H; multi-head attention applies this mechanism across multiple projections.
- Transformer: Self-attention relates each word to all input words, masked self-attention restricts decoder access to later positions, and cross-attention aggregates encoder information for generation.These mechanisms support encoding, autoregressive decoding, and context-dependent sequence-to-sequence tasks such as translation and summarization.
- GPT: GPT combines the Transformer decoder with autoregressive language modeling, predicting each word from its previous words through masked multi-head self-attention.GPT removes the original Transformer decoder’s cross-attention because its pre-training objective is autoregressive language modeling.
4 Designing Effective Architectures
This section reviews architectures that unify language understanding and generation, incorporate cognitive-inspired memory, and address Transformer limitations. It highlights unified objectives, external memory, and explicit working-memory mechanisms as major directions.
- Unified Sequence Modeling: Transformer-based models seek unified task handling because the boundary between language understanding and generation is vague.The cited discussion notes that understanding tasks can be reformulated as generation tasks and that GPTs can match or exceed BERTs on some understanding benchmarks.
- Unified Sequence Modeling: After-BERT architectures pursue unified sequence modeling through autoregressive, autoencoding, and encoder-decoder objectives.XLNet, UniLM, GLM, T5, and BART each address tensions between understanding, generation, or variable-length span reconstruction.
- Unified Sequence Modeling: GLM autoregressively generates variable-length masked spans and uses 2D positional encoding to preserve mask-count information.The paper reports GLM as achieving the best performance across understanding, conditional generation, and unconditional generation tasks simultaneously.
- Unified Sequence Modeling: Encoder-decoder architectures solve variable-length blank filling but introduce more parameters and generally underperform strong encoder models on natural language understanding.The paper notes that well-trained RoBERTa or GLM encoders perform much better than encoder-decoder models despite improvements over similarly sized vanilla BERT.
- Cognitive-Inspired Architectures: Transformer architectures remain limited by fixed windows and quadratic space complexity, motivating efficient long-context and memory-based alternatives.Transformer-XL introduces segment-level recurrence and relative positional encoding, while CogLTX selects sentences for working memory.
- Cognitive-Inspired Architectures: Cognitive-inspired architectures extend Transformers with external long-term memory and mechanisms for maintaining working memory during reasoning.REALM retrieves Wikipedia sentences as pre-training context, while CogQA combines PTM-based System 1 processing with a GNN-based cognitive graph.
5 Utilizing Multi-Source Data
This section surveys PTMs that exploit multilingual, multimodal, and other heterogeneous data sources. It emphasizes cross-lingual objectives, image-text alignment, and larger or richer training corpora as routes to broader capabilities.
- Multilingual PTMs: Multilingual PTMs aim to share semantic knowledge across languages instead of training a separate large model for each language.Earlier task-specific multilingual methods had limited cross-task generalization, motivating general pre-training followed by downstream fine-tuning.
- Multilingual PTMs: mBERT uses multilingual masked language modeling on non-parallel Wikipedia in 104 languages and can generalize cross-lingual knowledge in zero-shot settings.The cited evidence attributes this result to research showing mBERT's zero-shot cross-lingual ability.
- Multilingual PTMs: XLM uses translation language modeling on bilingual sentence pairs to learn cross-lingual representations from parallel corpora.TLM concatenates semantically matched sentences in two languages and masks tokens in both parts.
- Multimodal PTMs: Vision-language PTMs primarily address grounding by aligning visual and textual content in a unified semantic space.Representative architectures use either separate visual and textual streams with co-attention or a single stream for joint processing.
- Multimodal PTMs: UNITER combines datasets into 5.6 million image-text pairs, and sufficient training data helps it achieve impressive downstream results.Additional text-only corpora can also help visual commonsense reasoning, especially with long and complex sentences.
- Multimodal PTMs: DALLE enables text-to-image generation with around 10 billion parameters, while CogView surpasses DALLE in FID on blurred COCO.CogView improves numerical precision and training stability using a sandwich Transformer and sparse attention.
6 Improving Computational Efficiency
The paper reviews system-level optimization, efficient learning algorithms, and model compression strategies for reducing the computational and memory costs of large-scale PTM training. It covers single-device and multi-device techniques, including precision reduction, memory offloading, and parallelism.
- Overview: Increasing PTM size improves accuracy but raises memory and computational requirements, motivating system optimization, efficient learning, and model compression.The review organizes computational-efficiency methods around these three aspects.
- Single-Device Optimization: FP16 reduces floating-point memory and computation with little precision loss, while mixed-precision training preserves critical weights in FP32 and uses dynamic loss scaling.Mixed precision addresses truncation and overflow problems that can make direct FP16 training fail.
- Single-Device Optimization: Activation states can consume more memory than model parameters, motivating strategies that store parameters and activations in CPU memory and schedule GPU–CPU swaps.These methods target redundant activation storage and models that do not fit in a single GPU.
- Multi-Device Optimization: Data parallelism splits batches across nodes and aggregates gradients with all-reduce, whereas model parallelism distributes parameters across nodes when a whole model cannot fit on one GPU.Data-parallel communication can add overhead, while model parallelism addresses the memory demands of billion- to trillion-parameter models.
- Multi-Device Optimization: Pipeline parallelism partitions layers across nodes and passes intermediate activations between adjacent stages, but incomplete batch overlap can create pipeline bubbles.The reviewed pipeline methods speed large-scale PTM training while requiring completion of gradient back-propagation before proceeding.
- Efficient Training and Compression: ELECTRA improves pre-training sample efficiency by replacing masked-token prediction with replaced-token detection, while pruning and quantization reduce model size with limited performance impact.Attention-head pruning can remove many heads with little accuracy loss; Q8BERT reports little impact on model performance, whereas ultra-low-bit methods require additional techniques.
7 Interpretation and Theoretical Analysis
The paper surveys methods for interpreting what PTMs learn, how their internal structures behave, and why pre-training helps. It reports evidence for linguistic and world knowledge, structural redundancy, robustness weaknesses, and theoretical links between pre-training and downstream performance.
- Overview: Interpretation studies examine PTM knowledge, robustness, structural sparsity or modularity, and theoretical explanations of pre-training.The review treats these as major directions for understanding PTM behavior.
- Linguistic Knowledge: Representation probing finds strong encoding of tokens, chunks, pairwise relations, and syntax, but comparatively little improvement on semantic tasks.Layer analyses further examine syntactic, semantic, local, and long-range information.
- Linguistic Knowledge: Attention analyses suggest lower layers encode positional information while higher layers capture hierarchical information.Researchers also use attention statistics and spanning-tree constructions to characterize internal structure.
- World Knowledge: PTMs encode commonsense and factual world knowledge, outperforming supervised baselines on some knowledge-generation tasks without fine-tuning.However, extracting knowledge can depend on constructing effective fill-in-the-blank statements or prompts.
- Robustness: PTMs remain vulnerable to synonym substitutions and irrelevant artifacts, making robustness a serious security concern for real-world deployment.Human-in-the-loop adversarial examples can be more natural, valid, and diverse than automatically generated attacks.
- Structural Sparsity: Transformer attention heads and parameters exhibit redundancy: removing attention heads or pruning 30–40% of weights can preserve or improve performance.The review links this redundancy to repeated attention patterns and a limited set of head behaviors.
- Theoretical Analysis: Pre-training has been explained through better optimization and regularization, while contrastive-learning theory proves that pre-training loss upper-bounds downstream loss under stated assumptions.These analyses connect pre-training objectives with optimization behavior and downstream generalization.
8 Future Directions
The paper identifies open directions for PTMs spanning architectures, pre-training, tuning, reliability, multimodal and multilingual learning, distributed efficiency, and modeledge management.
- Future PTM research should address architectures and pre-training methods, multilingual and multimodal pre-training, computational efficiency, theoretical foundations, modeledge learning, cognitive learning, and novel applications.
- Architectures and Pre-Training Methods: Transformers’ computational complexity limits many PTMs to sequences of at most 512 tokens, motivating efficient architectures for longer-range context and deployment on low-capacity or low-latency devices.
- Architectures and Pre-Training Methods: More efficient self-supervised tasks and training methods are needed because general-purpose PTMs require larger architectures, corpora, challenging objectives, and costly distributed or mixed-precision training.
- Beyond Fine-Tuning: Parameter-efficient adaptation can fix PTM parameters and add small task-specific modules, allowing one shared PTM to serve multiple downstream tasks.
- Reliability: PTM reliability and interpretability remain open concerns, with adversarial studies exposing vulnerabilities and defenses seeking improved robustness.
- Multimodal and Multilingual Pre-Training: Multimodal PTMs require efficient temporal modeling for video and audio, deeper interpretation of cross-modal effects, clearer real-world applications, and adaptation to unseen languages and direct audio transfer.
- Computational Efficiency: Distributed training must reduce data-movement bottlenecks and the complexity of manually programming communication, computation overlap, and parallelism strategies.
- Modeledge Learning: PTMs also create a modeledge storage and management challenge because differing architectures and training corpora produce diverse continuous knowledge.
9 Conclusion
The paper reviews PTM history, current advances, and future questions, emphasizing their role in AI and the need to represent and use machine-friendly knowledge more effectively.
- The paper traces pre-training history, positions PTMs within AI development, and reviews advances in architectures, contexts, efficiency, interpretation, and theory.
- Existing PTMs show promising results, including reported zero-shot and few-shot abilities, while their next development remains an open question.
- The paper names continuous machine-friendly knowledge stored in real-valued vectors “modeledge” and proposes capturing and stimulating it more effectively and efficiently.
Note and Contribution
The paper originated from a January 2021 Beijing workshop and records its organizational structure, section drafting responsibilities, commenting, and proofreading.
- The paper summarizes and extends discussions from a three-day closed-door workshop held in Beijing WTown from January 1 to January 3, 2021.
- Zhiyuan Liu and Xu Han designed the paper’s structure, while Xu Han also drafted the abstract and Sections 1 and 2.
- Different authors drafted Sections 3 through 8, with faculty authors assigned to specific topics in Section 8.
- Wayne Xin Zhao and Xipeng Qiu commented on the manuscript, while Xu Han, Ning Ding, and Zhengyan Zhang proofread the whole paper.