Source-linked AI summary
A Survey of Knowledge Enhanced Pre-trained Language Models
Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, Juanzi Li
TL;DR
PLMs can lack external knowledge needed for strong symbolic reasoning and commonsense generation. This paper surveys KE-PLMs through separate NLU and NLG taxonomies, covering knowledge types and future directions. It concludes that the field must address knowledge integration, multimodality, and continual learning challenges.
Problem
PLMs can have limited symbolic reasoning and commonsense generation because their pre-training lacks explicit external knowledge.
Method
The paper provides a comprehensive review of KE-PLMs with separate taxonomies for NLU and NLG and discusses representative methods.
Results
NLU methods are categorized by linguistic, text, KG, and rule knowledge, while NLG methods are categorized as KG-based or retrieval-based.
Takeaways & Limitations
The survey identifies heterogeneous-source integration, multimodal knowledge, and continual learning as promising directions for KE-PLMs.
Takeaways & Limitations
Existing work usually uses a single knowledge source, leaving heterogeneous-source integration as an open research direction.
Abstract
from arXiv · showhide
Pre-trained Language Models (PLMs) which are trained on large text corpus via self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Research has been dedicated to incorporating knowledge into PLMs to tackle these issues. In this paper, we present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight these two main tasks of NLP. For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.
1 INTRODUCTION
PLMs achieve strong NLP performance but can lack explicit external knowledge and symbolic or commonsense reasoning. This survey organizes KE-PLMs across NLU and NLG and identifies future research directions.
- PLMs trained on massive unlabeled corpora are typically fine-tuned on smaller labeled datasets for downstream NLP tasks.
- Traditional pre-training can produce weak symbolic reasoning, while knowledge-enhanced models improve factual prediction in language understanding.For example, a knowledge-enhanced model predicts “Tiananmen” rather than “the” for a masked location.
- Conventional PLMs can generate grammatical but illogical sentences because they overlook external commonsense knowledge.Adding knowledge produces more natural and logically coherent generations from concept sets.
- The survey reviews KE-PLMs separately for NLU and NLG because knowledge injection supports both tasks but their focuses differ.
- For NLU, the survey categorizes knowledge as linguistic, text, KG, or rule knowledge, while NLG methods are retrieval-based or KG-based.
- The paper discusses potential future research directions based on existing problems and challenges in KE-PLMs.
2 BACKGROUND
PLMs evolved from pre-train-and-fine-tune systems toward prompt-based prediction, using large text corpora to form general-purpose representations. Prompt construction remains important because template and verbalizer choices can substantially affect performance.
- Early PLMs such as ELMo and ULMFiT introduced layer-wise fine-tuning of language models for downstream tasks.
- The pre-train-and-fine-tune paradigm learns general-purpose representations from raw text before applying task-specific objectives.
- Transformer self-attention enables models to capture long-range dependencies and learn more expressive representations.
- GPT is autoregressive, BERT is bidirectional masked language modeling, and BART and T5 use encoder-decoder architectures.
- Prompt learning reformulates downstream tasks with textual prompts so PLMs can predict desired outputs without task-specific fine-tuning objectives.
- Knowledge can serve as prompt information to improve prompt tuning and reduce the cost of constructing templates.Template and verbalizer selection can greatly affect model performance.
3 KE-PLMS FOR NLU
For NLU, KE-PLMs incorporate knowledge to help machines understand and interpret textual content for downstream applications. The survey organizes these models into four categories based on knowledge type.
- NLU enables machines to understand and interpret textual data, extract semantic information, and apply it to downstream tasks.Applications include text classification, relation extraction, named entity recognition, and dialogue systems.
- The survey divides NLU-oriented KE-PLMs into linguistic knowledge, text knowledge, knowledge graph, and rule knowledge categories.
- The survey discusses representative methods for each NLU knowledge category.
3.1 Incorporating Linguistic Knowledge into PLMs
Linguistic knowledge is a common auxiliary feature for PLMs, encompassing lexical information and syntax trees. Methods differ in how they encode syntax and when they fuse linguistic knowledge.
- Linguistic knowledge mainly comprises lexical knowledge and syntax trees and is the most common auxiliary feature incorporated into PLMs.
- Lexical knowledge includes POS tagging and sentiment tags, while LIBERT and SenseBERT pre-train models to capture semantic word information.
- Syntax-tree methods use new pre-training tasks, attention mechanisms, or model structures to incorporate syntactic information.LIMIT-BERT jointly trains across five linguistic tasks using POS tagging, semantic role labeling, dependency relations, and syntax trees.
- Some methods incorporate dependency-tree syntax information to improve PLMs.
- Most listed methods fuse linguistic knowledge during pre-training, whereas Sentiprompt, LET, and KEAR fuse it during fine-tuning.
3.2 Incorporating Text Knowledge into PLMs
Text knowledge is incorporated from general-domain collections, large corpora, and domain-specific scientific literature. These sources are integrated during either pre-training or fine-tuning.
- Text knowledge comes from general-domain collections such as WikiText and Wiktionary or large corpora such as Wikipedia.
- KNN-LM retrieves nearest neighbors from training samples, while REALM explicitly trains a retriever over external knowledge bases.
- BioBERT and SciBERT pre-train on large scientific-domain corpora for academic NLP tasks.
- S2ORC-BERT uses a larger multidisciplinary academic corpus and slightly improves performance on several downstream tasks.
KEPLM in NLU
The survey organizes knowledge-enhanced methods for language understanding around linguistic knowledge and downstream-task-oriented taxonomies. It also distinguishes knowledge fusion by training stage.
- NLU-oriented KE-PLMs are organized using a taxonomy of incorporated knowledge types and downstream language-understanding tasks.
- The survey reports that S2ORC-BERT slightly promotes performance on several downstream tasks.
- Text-knowledge methods are distinguished by whether knowledge is integrated during pre-training or fine-tuning.
3.3 Incorporating Knowledge Graph into PLMs
Knowledge graphs provide structured entity and relational information for enhancing PLMs. The survey categorizes KG methods by knowledge type and by whether they alter pre-training tasks, attention mechanisms, or model structure.
- Knowledge graph foundations: Knowledge graphs represent entities as nodes and relations as edges, providing structured information that differs from unstructured text knowledge.
- Taxonomy: KG knowledge is divided into entity knowledge and triplet knowledge, with each category further organized by its incorporation mechanism.
- Entity knowledge: Entity-knowledge methods introduce entity-related pre-training tasks, change attention mechanisms, or modify model structure.
- Triplet knowledge: Triplet-knowledge methods similarly design triplet-related pre-training tasks, alter attention mechanisms, or add knowledge-fusion modules.
- Attention mechanisms: K-BERT injects relevant KG triplets into input sentences and converts them into knowledge-rich sentence trees controlled by a visible matrix.
- Model structure: Knowledge-fusion architectures include fact memory, adapter modules, knowledge aggregators, GNN coupling, prompt templates, and ontology linearization.
- Knowledge reasoning: Some KG-enhanced models jointly learn knowledge embeddings while supporting entity classification, relation prediction, or knowledge graph completion.
- Fusion stage: The survey distinguishes pre-fusion methods that inject knowledge during pre-training from post-fusion methods that inject it during fine-tuning.
3.4 Incorporating Rule Knowledge into PLMs
Rule knowledge formalizes external knowledge as explicit logical reasoning processes. RuleBERT uses Horn rules to construct training data and probabilistic answer-set programming to predict events.
- Logic rules can formalize external knowledge and make reasoning paths more interpretable.
- RuleBERT uses Horn rules from an existing corpus to establish a training dataset before fine-tuning the model.
- RuleBERT applies probabilistic answer set programming to predict event probabilities and learn soft rules.
4 KE-PLMS FOR NLG
For NLG, the survey categorizes KE-PLMs by knowledge source into retrieval-based and KG-based methods, then refines each category by its processing or integration strategy.
- KE-PLMs for NLG use external knowledge beyond input sequences to improve text generation.The survey frames this knowledge as supporting generation models in producing more effective text.
- Retrieval-based methods: Retrieval-based methods obtain input-related knowledge from external sources and either use retrieved items directly or rerank them before rewriting.Sources include online search engines, large datasets, and training sets; Fig. 6 distinguishes retrieval augmented generation from retrieve, rerank and rewrite.
- KG-based methods: KG-based methods are divided by knowledge granularity into path-finding knowledge, triplet knowledge, and subgraph knowledge.Path-finding methods reason over relation paths, triplet methods convert knowledge into statements or use triplets, and subgraph methods model contextual concept structures.
- KG-based methods: Subgraph knowledge can be integrated into the encoder, decoder, or both, with decoder integration enabling tracing at each decoding step.Examples include JointGT and MoKGE for encoder integration, GRF for decoder integration, and KG-BART for both.
- Fusion timing: NLG KE-PLMs also differ by fusion timing: some fuse knowledge during pre-training, while others fuse it during fine-tuning.The survey labels these approaches pre-fusion and post-fusion methods, respectively.
5 FUTURE DIRECTIONS
The survey identifies future directions for KE-PLMs spanning heterogeneous and multimodal knowledge integration, commonsense reasoning, continual learning, efficiency, and diverse generation. It links these directions to improving knowledge coverage, interpretability, retention of old knowledge, computational practicality, and output variety.
- 5.1 Integrating Knowledge from Homogeneous and Heterogeneous Sources: Integrating heterogeneous knowledge sources can expand knowledge coverage and help open-domain QA models generate more reliable answers.UniK-QA combines text, tables, and relational triplets by linearizing heterogeneous sources into text.
- 5.2 Exploring Multi-modal Knowledge: Multimodal knowledge from images, videos, and audio remains underused despite textual and tabulated sources being dominant.The survey identifies multimodal integration as a direction that may further improve KE-PLM performance.
- 5.2 Exploring Multi-modal Knowledge: Combining visual and textual semantics can enhance PLMs on both natural language understanding and generation tasks.KB-VLP uses knowledge retrieved from external bases using text and images, while ERNIE-VIL structures image descriptions as scene graphs.
- 5.3 Providing Interpretability Evidence: Explicit commonsense reasoning paths can make generation more interpretable and predictions more rational.GRF performs dynamic multi-hop reasoning over multiple relational paths in external knowledge graphs.
- 5.4 Learning Knowledge in a Continuous Way: Continual learning is needed to acquire new knowledge while retaining previously learned knowledge and avoiding catastrophic forgetting.Adapters and extension modules are cited as approaches for storing or incorporating factual and linguistic knowledge over time.
- 5.5 Optimizing the Efficiency of Incorporating Knowledge into Large Models: Future work should improve knowledge acquisition and filtering efficiency while reducing the computational burden of knowledge incorporation.The survey notes that increasing model and knowledge-injection scales create challenges for computational efficiency and resources; ZeRO reduces redundant memory use by partitioning model states.
- 5.6 Increasing the Variety of Results Generated: Diversified commonsense knowledge reasoning can support alternative and more reasonable outputs in natural language generation.MoKGE associates concepts related to the input and uses a mixture-of-experts method to increase generation diversity.
6 CONCLUSION
The survey reviews KEPLMs across NLU and NLG, proposes taxonomies for their different focuses, discusses representative work, and identifies future research directions.
- The survey comprehensively reviews KEPLMs from both NLU and NLG perspectives.
- It proposes separate taxonomies for NLU and NLG to highlight their different focuses.
- The survey discusses representative work within these taxonomies.
- It identifies potential future research directions based on existing problems and challenges.