PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, Lichao Sun

arXiv:2302.09419v3cs.AIcs.CLcs.LG

TL;DR

The survey addresses the need for a comprehensive account of PFMs across data modalities, research aspects, and application domains. It synthesizes pretraining methods and models in NLP, computer vision, and graph learning, extends the review to unified and advanced PFMs, and discusses efficiency, security, privacy, challenges, and open problems. The survey concludes that PFMs span broad research opportunities while facing challenges in data, computation, multimodal dataset construction, and unified modeling.

  • Problem

    Existing surveys did not comprehensively review PFMs across NLP, computer vision, graph learning, speech, and video together with pretraining, efficiency, efficacy, and privacy.

  • Method

    The survey synthesizes PFM components, pretraining methods, models, advanced and unified architectures, efficiency and compression techniques, security, privacy, and open problems across modalities.

  • Results

    PFM research spans text, image, graph, speech, video, and multimodal settings, with BEIT-3 outperforming state-of-the-art models on vision and vision-language tasks.

  • Takeaways & Limitations

    The review highlights scalability, security, logical reasoning, cross-domain learning, and user-friendly interaction as important directions for PFMs.

  • Takeaways & Limitations

    PFM research still faces technical challenges across data, foundation, model design, and upstream and downstream tasks, while multimodal dataset construction remains urgent.

Abstract

from arXiv · show

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. BERT learns bidirectional encoder representations from Transformers, which are trained on large datasets as contextual language models. Similarly, the generative pretrained transformer (GPT) method employs Transformers as the feature extractor and is trained using an autoregressive paradigm on large datasets. Recently, ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few shot prompting. The remarkable achievements of PFM have brought significant breakthroughs to various fields of AI. Numerous studies have proposed different methods, raising the demand for an updated survey. This study provides a comprehensive review of recent research advancements, challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. The review covers the basic components and existing pretraining methods used in natural language processing, computer vision, and graph learning. Additionally, it explores advanced PFMs used for different data modalities and unified PFMs that consider data quality and quantity. The review also discusses research related to the fundamentals of PFMs, such as model efficiency and compression, security, and privacy. Finally, the study provides key implications, future research directions, challenges, and open problems in the field of PFMs. Overall, this survey aims to shed light on the research of the PFMs on scalability, security, logical reasoning ability, cross-domain learning ability, and the user-friendly interactive ability for artificial general intelligence.

1 Introduction

PFMs use large-scale pretraining to provide general representations that can be adapted across downstream tasks and modalities. This survey reviews their evolution, applications, unified models, efficiency, security, privacy, and future challenges.

  • PFMs and Pretraining: PFMs train general models on large amounts of data that can be fine-tuned for different downstream applications.
  • PFMs and Pretraining: Dynamic pretraining methods such as BERT and XLNet address the limited adaptability of earlier static methods such as NNLM and Word2vec.
  • PFMs and Pretraining: ChatGPT combines GPT-3.5 fine-tuning with reinforcement learning from human feedback to align large language models with human intent.
  • Contribution and Organization: The survey reviews PFMs for text, image, and graph data, while also covering speech, video, cross-domain, and multimodal models.
  • Contribution and Organization: Unified PFMs handle multiple data types such as text, images, and audio, with examples including OFA, UNIFIED-IO, FLAVA, and BEiT-3.
  • Contribution and Organization: The review additionally examines model efficiency and compression, security and privacy, evaluation resources, and future research challenges.

2 Basic Components

PFMs combine large-scale pretraining with downstream adaptation through architectures and learning mechanisms suited to different modalities. Their basic pipeline includes model construction, supervised or weakly supervised learning, self-supervision, reinforcement learning, and task-specific pretraining objectives.

  • PFM architecture: PFMs use large neural architectures, pretrained datasets, and downstream fine-tuning to meet efficacy, efficiency, and privacy requirements.Transformer is a mainstream architecture for PFMs in NLP and CV, while model designs vary by modality and task.
  • PFM architecture: Transformers rely on attention mechanisms, with masking controlling which input elements can attend to one another.Self-attention assigns weights to encoded representations and supports information transfer across neural units.
  • Learning mechanisms: Supervised, semi-supervised, weakly supervised, self-supervised, and reinforcement learning provide distinct routes for training PFM representations and policies.Self-supervised learning uses data-derived pseudo-labels or pretext tasks, while reinforcement learning optimizes sequential decisions through rewards.
  • Learning mechanisms: Pretraining initializes models with attributes, structure, and community information that can assist downstream tasks and accelerate convergence.The framework generally operates together with fine-tuning on downstream tasks.
  • Pretraining tasks: Pretraining objectives include MLM, DAE, RTD, NSP, and SOP, covering token reconstruction, noisy-input reconstruction, replacement detection, and sentence-order modeling.MLM masks words, DAE reconstructs noisy corpora, RTD detects replaced tokens, NSP tests sentence correlation, and SOP tests fragment order.

3 PFMs for Natural Language Processing

PFMs for NLP evolved from autoregressive, contextual, and permuted language modeling toward architectures and alignment methods that support generation, representation learning, and reasoning. The survey presents BERT, GPT, BART, SpanBERT, GPT-4, and chain-of-thought prompting as representative developments.

  • Word representation methods: Autoregressive LMs predict tokens from preceding context and perform well in natural language generation tasks such as summarization and translation.GPT uses stacked Transformer decoders with self-supervised pretraining followed by supervised fine-tuning; GPT-2 expanded this design to 48 layers and 1.5 billion parameters.
  • Word representation methods: Contextual LMs condition predictions on words across the sequence through Transformer encoders, enabling bidirectional representations.BERT combines a bidirectional Transformer with word-piece, segment, and position embeddings, while retaining substantial computational demands.
  • Word representation methods: Permuted LMs combine autoregressive prediction with bidirectional context by modeling tokens under sequence permutations and auxiliary position information.The approach addresses the masking mismatch between BERT-style pretraining and fine-tuning while avoiding strictly ordered autoregressive modeling.
  • Masking designing methods: BART corrupts text with five noise schemes and reconstructs the original document using a bidirectional encoder and autoregressive decoder.Its noise types are single-word masking, word deletion, span masking, sentence rearrangement, and document rearrangement.
  • Instruction-aligning methods: GPT-4 uses next-token pretraining followed by RLHF fine-tuning, and is reported to outperform GPT-3.5 in reliability, creativity, and nuanced instruction handling.Chain-of-thought prompting generates intermediate reasoning steps and has demonstrated benefits on arithmetic, symbolic, and common-sense reasoning tasks.

4 PFMs for Computer Vision

PFMs for computer vision use self-supervised pretext tasks to learn transferable representations from unlabeled data, spanning prediction, reconstruction, adversarial, contrastive, and memory-based methods. These approaches support downstream vision tasks while aiming to reduce annotation and training costs.

  • Self-Supervised Learning: Self-supervised learning reduces dependence on costly annotations by training encoders with automatically generated signals from unlabeled data.Pretext tasks match, contrast, or generate data-derived targets before supervised downstream learning.
  • Pretraining Strategies: Computer-vision PFMs use pretext tasks including specific prediction, frame order, generation, reconstruction, memory banks, sharing, and clustering.The survey organizes these approaches as major pretraining strategies for image representation learning.
  • Specific Pretext Tasks: Contrastive Predictive Coding predicts future observations in latent space by encoding inputs into representations and summarizing past latents with an autoregressive context.CPC can process modalities including speech, images, and text; CPC v2 transfers unsupervised representations to data-efficient downstream tasks.
  • Generative Methods: BigBiGAN adds an encoder to adversarial generation so data-latent pairs support unsupervised representation learning and auxiliary supervised discrimination tasks.The discriminator distinguishes pairs formed from raw data and latent distributions.
  • Reconstruction: Masked image modeling transfers language-style masked prediction to vision, with BEiT using discrete visual tokens and MAE predicting masked patches directly from unmasked ones.BEiT separates tokenizer training from masked prediction, whereas MAE provides an end-to-end solution and uses a 75% masking ratio.
  • Memory Bank: Memory-bank contrastive learning stores previous representations so additional negative pairs improve gradient scalability and produce stronger learned encoders.The method encourages transformed views of the same image to be similar and views from different images to be dissimilar.

5 PFMs for Graph Learning

PFM pretraining for graph learning addresses limited labeled data by learning representations from graph information, properties, and consistency across structural scales. The survey organizes these approaches around information completion, property prediction, and consistency-based objectives.

  • Large-scale graph labels are expensive and time-consuming to construct, especially when professional knowledge and timely data are required.
  • Graph Information Completion: Graph information completion masks graph information and reconstructs it from unmasked data to pretrain graph embeddings.
  • Graph Information Completion: GraphCompetion masks target-node features and uses unmasked features from local GCN neighborhoods to learn contextual representations.
  • Graph Consistency Analysis: Graph consistency analysis pretrains models by making semantically similar graph elements more consistent than unrelated elements.
  • Graph Consistency Analysis: Consistency-based methods compare context distributions, self-augmented views, and representations across scales using random walks, contrastive learning, or mutual-information objectives.
  • Graph Property Prediction: Graph property prediction creates self-supervised signals by using auxiliary graph properties as pretraining targets, including numerical property regression.

6 PFMs for Other Data Modality

PFMs have expanded beyond text, image, and graph learning into speech, video, code generation, multimodal learning, and unified architectures. These models use self-supervised, cross-modal, or multimodal pretraining to support varied downstream tasks.

  • Speech: Speech PFMs learn representations from large unlabeled datasets and can improve downstream tasks after fine-tuning on few samples.
  • Video: Video pretraining uses tasks such as odd-subsequence prediction and contrastive learning with positive and negative pairs to learn video representations.
  • Multimodal Learning: Multimodal text-image PFMs use either single-stream fusion or separate text and image encoders connected through attention.
  • Code Generation: Code-generation PFMs are trained on large natural-language corpora and fine-tuned for programming tasks, including web development, NLP, and data analysis.
  • Unified PFMs: Unified PFMs combine unimodal and multimodal data through single-transformer, multi-transformer, or comb-transformer architectures.
  • Unified PFMs: BEIT-3 contains 1.9B parameters and experimentally outperforms SOTA models on vision and vision-language tasks.

7 Other Advanced Topics on PFMs

The survey examines efficiency, compression, security, and privacy as foundational concerns for PFMs. It describes methods for reducing computational demands while highlighting vulnerabilities to adversarial inputs, backdoors, reasoning defects, and privacy leakage.

  • Efficiency and Compression: Growing parameter counts increase memory, computation, training cost, and deployment limits for resource-constrained devices.
  • Model Efficiency: Model-efficiency methods reduce pretraining cost through more effective objectives such as ELECTRA’s replaced-token detection and lower-complexity architectures.
  • Model Compression: Model compression reduces model size and resource requirements through parameter pruning, quantization, low-rank decomposition, parameter sharing, compact networks, and distillation.
  • Security: PFMs are vulnerable to adversarial samples, including tiny input changes that can mislead predictions across NLP, computer vision, and graph learning.
  • Security: Reasoning performance can be limited by false statistical information in datasets, while universal adversarial triggers can induce specific predictions.
  • Security: Backdoor attacks can inject poisoned pretrained weights or alter model corpora so attackers manipulate predictions through keywords or changed word meanings.
  • Privacy: PFMs may disclose private training examples, including personally identifiable information and IRC conversations, and larger models can be more vulnerable to such extraction.

8 Future Research Challenges and Open Problems

The survey identifies challenges spanning data, theoretical foundations, model design, and upstream–downstream transfer. It highlights open problems in multimodal and multilingual learning, efficiency, robustness, graph pretraining, and unified PFMs.

  • PFMs face technical challenges from large parameter counts, extensive training data, and high computational complexity.
  • Data: Pretraining datasets need broader multimodal, multilingual, and graph coverage, while graph data often lacks abundant unlabeled nodes and edges.
  • Data: Multimodal PFMs require new datasets linking different modalities, making multimodal dataset construction an urgent problem.
  • Data: Multilingual PFMs need suitable additional tasks, while their larger vocabularies sharply increase model parameters.
  • Foundations and transfer: PFM research still lacks comprehensive theoretical foundations, stable semantic understanding across domains, and consistent performance on downstream tasks.
  • Foundations and transfer: Upstream accuracy and additional data do not always improve downstream performance, revealing a nonlinear relationship between upstream and downstream tasks.
  • Unified PFMs: Future unified PFMs are expected to converge text, image, graph, and multimodal pretraining, but existing unified models largely neglect graphs and often excel in only one domain.

9 Conclusion

The survey comprehensively reviews PFMs across text, image, and graph domains, then examines advanced and unified models, efficiency, compression, security, privacy, challenges, and open problems.

  • The survey summarizes existing PFMs in text, image, and graph domains and reviews their pretraining model structures.
  • It also studies advanced and unified PFMs, model efficiency and compression, security and privacy, and future challenges.

A.1 Basic Components on NLP

This section introduces language-model foundations, neural and pretrained architectures, graph concepts, learning formulations, and supervision settings. It presents the notation and task structures used for NLP and graph learning.

  • Language models: Language models estimate the probability of text sequences and include statistical and neural-network approaches.
  • Language models: N-gram models trade stronger sequence constraints for sparse frequency information as N grows, while smaller N improves reliability and generalization but weakens constraints.
  • Language models: Neural language models use distributed word representations to address limitations of statistical models, including sparse data and weak long-term dependence.
  • Pretrained language models: Pretraining obtains parameters from pretraining tasks, initializes a model with them, and then improves performance through further training; GPT uses a two-stage process.
  • Graph learning: Graph learning represents graph attributes, structures, and related information in node representations through pretext tasks optimized for downstream tasks.
  • Graph learning: Graph learning supervision ranges from supervised and unsupervised settings to self-supervised learning, which mines pseudo-labels from unlabeled graph data.

B Traditional Learning Methods

Traditional learning methods developed from classical language, vision, and graph techniques toward neural, convolutional, attention-based, transformer, dimensionality-reduction, factorization, and kernel approaches.

  • Traditional text learning: Traditional NLP methods progressed from N-gram and word-vector representations toward neural models, but static vectors leave polysemy unresolved.
  • Traditional computer-vision learning: Computer-vision networks commonly combine data augmentation, convolution for spatial feature extraction, and deep architectures.
  • Transformer-based networks: Vision transformers use pretraining on large weakly labeled datasets followed by transfer and fine-tuning on downstream tasks.
  • Traditional graph learning: Graph dimensionality-reduction methods seek lower-dimensional representations while retaining important information, often under linear assumptions.
  • Traditional graph learning: Matrix-factorization methods reconstruct graph adjacency or proximity information to obtain lower-dimensional node representations.
  • Traditional graph learning: Graph kernels map graph representations into high-dimensional feature spaces or directly encode graph structure while supporting linear algorithms there.

D Pretext Task Taxonomy on CV

The survey organizes computer-vision pretext tasks into generation-, transformation-, context-, semantic-, and view-based methods, while also covering representation-learning pretraining for reinforcement learning. These approaches use pseudo-labels, augmentations, contextual relationships, semantic signals, or environmental representations to support downstream learning.

  • Pretext Task Taxonomy: Self-supervised pretext tasks are grouped into five categories: generation-based, transformation-based, context-based, semantic-based, and view-based methods.These tasks use pseudo-labels generated from the data itself to pretrain proxy models.
  • Generation-Based Methods: Generation-based methods use GAN-style objectives for image colorization, super-resolution, editing, translation, and video generation or representation learning.The listed video applications include future prediction and action recognition.
  • Transformation-Based Methods: Transformation-based methods treat transformed views of an image as positive or negative pairs for self-supervised pretraining.Common transformations include color changes, Gaussian blur, and brightness adjustment.
  • Context-Based Methods: Context-based methods create artificial tasks such as solving image jigsaws, comparing contextual similarity, or discriminating sequence order.Jigsaw tasks train encoders whose representations can transfer when the pretraining dataset is sufficiently large.
  • Semantic-Based Methods: Semantic-based methods use object detection, semantic segmentation, or depth prediction as pretext tasks for robust feature representation.These tasks provide pixel-based labels and are commonly established on video datasets.
  • View-Based Methods: View-based methods form positive pairs from alternate viewpoints in single-modal data and use cross-modal relationships for multimodal pretraining.Temporal slices may serve as negative pairs when scene content changes over time.
  • Pretraining in Reinforcement Learning: Reinforcement-learning pretraining includes reward-free, goal-conditioned, and representation-learning directions, with this survey focusing on representation learning.Model-based methods pretrain world models for use as state encoders or simulators, while contrastive-like methods pretrain encoders with auxiliary objectives or augmented data.

F Evaluation Metrics

The survey presents evaluation metrics for classification, ranking, generation, translation, summarization, and language modeling. These metrics quantify predictive correctness, label balance, sequence overlap, translation quality, or model uncertainty, with perplexity carrying explicit dataset- and punctuation-related caveats.

  • Classification Metrics: Classification evaluation uses accuracy, error rate, precision, recall, F1, Micro-F1, and Macro-F1 to measure overall or class-balanced performance.Macro-F1 weights labels equally, whereas Micro-F1 aggregates labels’ overall accuracy and recall.
  • Ranking Metrics: Mean Reciprocal Rank evaluates ranking algorithms for question answering and information retrieval using the ranks of ground-truth answers.Other listed metrics include EM, Hamming-loss, P@K, and NDCG@K.
  • Generation Metrics: Generation tasks use language models to predict the next likely word or sentence from input data.The survey discusses generation-oriented evaluation through translation and summarization metrics.
  • Translation Metrics: BLEU compares generated and reference sentences through weighted N-gram precision and a brevity penalty.The penalty factor addresses length gaps between machine and reference translations.
  • Summarization Metrics: ROUGE evaluates summaries using N-gram co-occurrence and longest-common-subsequence comparisons, commonly through ROUGE-N and ROUGE-L.ROUGE-N emphasizes recall, while ROUGE-L uses an F-value based on the longest common subsequence.
  • Translation Metrics: METEOR improves on BLEU by matching synonyms, roots, affixes, and definitions through WordNet while penalizing word-order differences.Its penalty depends on the number of adjacent matched chunks.
  • Language-Model Metrics: Perplexity scores candidate translations with a language model, and lower confusion indicates better translation quality.Its value decreases faster on larger datasets and is affected by punctuation and common-word interference.

G.1 Downstream Tasks and Datasets on NLP

NLP downstream evaluation is organized around text classification and text generation, covering sentiment, news, topic labeling, inference, entity recognition, question answering, dialogue, summarization, and translation. The survey pairs these tasks with datasets that vary in labels, domains, sizes, and annotation schemes.

  • Task Organization: NLP downstream datasets are divided mainly into text-classification and text-generation tasks.Classification includes sentiment analysis, news classification, topic labeling, inference, named entity recognition, question answering, and dialogue act classification; generation includes summarization and translation.
  • Sentiment Analysis: Sentiment-analysis datasets represent emotional polarity with dichotomous, trichotomous, or multiple-category labels.SST-1 has five categories, while SST-2 contains 9,613 movie reviews; IMDB reviews are balanced binary sentiment data with about 25,000 comments per split.
  • News Classification: News-classification datasets support topic identification and recommendation, including 20 Newsgroups with 20 classes and AG News with four categories.20 Newsgroups contains 18,846 articles, while AG News contains 120,000 training and 7,600 testing texts.
  • Topic Labeling: Topic-labeling datasets assign one or more topics to documents, with DBpedia providing 14 categories and Yahoo Answers providing 10 categories.DBpedia contains 560,000 training and 70,000 testing examples; Yahoo Answers contains 140,000 training and 5,000 testing examples.
  • Natural Language Inference: Natural-language-inference datasets determine whether one text’s meaning follows from another using sentence-pair annotations such as neutral, entailment, and contradiction.SNLI contains 570,152 human-annotated pairs, while MNLI contains 430,000 sentence pairs across three categories.
  • Named Entity Recognition: Named entity recognition identifies people, places, organizations, and other entities and supports information extraction, question answering, semantic parsing, and translation.CoNLL 2003 covers four entity types across English and German newswire, while OntoNotes 5.0 spans English, Chinese, and Arabic data.
  • Question Answering: Question-answering datasets include extractive and generation-oriented settings, with SQuAD 1.1 containing 100,000 samples and SQuAD 2.0 expanding to 150,000 with unanswered questions.MRPC instead evaluates whether sentence pairs are semantically equivalent using accuracy and F1.

G.2 Downstream Tasks and Datasets on CV

The survey groups computer-vision downstream datasets by classification, detection, and segmentation, spanning general-purpose benchmarks and specialized domains. These datasets cover digits, objects, scenes, fine-grained categories, actions, and pixel-level recognition.

  • Task Organization: Computer-vision datasets are organized into classification, detection, and segmentation tasks.Segmentation is treated as semantic pixel-level classification and is often reserved for downstream evaluation because annotation is difficult.
  • Large-Scale Datasets: JFT-300M contains 300M images and over 1000M labels for large-scale classification.Its labels are produced by algorithms using human-computer communications and multiple labels may attach to each image.
  • Detection: Detection benchmarks prominently include COCO and PASCAL VOC for object recognition, detection, segmentation, and captioning.COCO contains 330,000 RGB images and 1.5 million object instances across 80 categories; VOC uses standardized challenge datasets with 20 objects.

G.3 Downstream Tasks and Datasets on Graph

Graph-learning PFMs are evaluated on downstream tasks organized by nodes, links, and whole graphs, using datasets spanning citation, molecular, protein, social, movie, and other networks. The survey summarizes task-specific datasets for node, link, and graph classification settings.

  • Downstream graph tasks are organized into node-level, link-level, and graph-level analyses.The organization reflects whether the task analyzes nodes, edges, or properties of an entire graph.
  • Node-Level Tasks: Node-level tasks include node classification, node clustering, and Top-K search.Node classification may be transductive or inductive, clustering groups correlated nodes, and Top-K search retrieves highly associated nodes.
  • Link-Level Tasks: Link-level tasks include link classification, link prediction, and Top-K recommendation.Link classification labels edges while considering their endpoint nodes; link prediction identifies removed or potentially existing edges.
  • Graph-Level Tasks: Graph-level tasks infer whole-graph properties from node, edge, and attribute distributions, including graph classification.Graph classification is applied to social, molecular, and protein graphs to predict properties of communities, compounds, and proteins.
  • Data Source: Tables 7–9 provide dataset statistics for node-level tasks, link classification, and graph classification, distinguishing homogeneous and heterogeneous graphs.The graph-classification statistics appear in Table 9.
  • Data Source: The surveyed datasets cover citation, molecular, protein, social, movie, word, book, airline, and spatiotemporal networks.Examples include Cora, Citeseer, PubMed, MUTAG, PTC, Reddit, BlogCatalog, Wikipedia, Book-crossing, US-Airport, and METR-LA.
Loading 2302.09419v3…