PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Knowledge-Enhanced Text Generation

Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, Meng Jiang

arXiv:2010.04389v4cs.CLcs.AIcs.LG

TL;DR

Text-generation models often lack sufficient knowledge beyond the input, limiting performance in real-world scenarios. This survey synthesizes methods for acquiring and incorporating diverse knowledge, finding the largest average improvement in generative commonsense reasoning tasks at +2.55% ΔBLEU.

  • Problem

    Input text alone often provides limited knowledge for desired outputs, leaving text-generation performance unsatisfactory in many real-world scenarios.

  • Method

    The survey reviews general knowledge-integration architectures and techniques organized by different sources of knowledge enhancement.

  • Results

    +2.55% ΔBLEU was the average improvement reported for KG-enhanced NLG on generative commonsense reasoning tasks, the largest across discussed tasks.

  • Takeaways & Limitations

    More effective knowledge representation and integration remains a central direction for advancing knowledge-enhanced NLG systems.

  • Takeaways & Limitations

    Existing knowledge-enhanced text-generation systems usually do not update their knowledge in real time.

Abstract

from arXiv · show

The goal of text generation is to make machines express in human language. It is one of the most important yet challenging tasks in natural language processing (NLP). Since 2014, various neural encoder-decoder models pioneered by Seq2Seq have been proposed to achieve the goal by learning to map input text to output text. However, the input text alone often provides limited knowledge to generate the desired output, so the performance of text generation is still far from satisfaction in many real-world scenarios. To address this issue, researchers have considered incorporating various forms of knowledge beyond the input text into the generation models. This research direction is known as knowledge-enhanced text generation. In this survey, we present a comprehensive review of the research on knowledge enhanced text generation over the past five years. The main content includes two parts: (i) general methods and architectures for integrating knowledge into text generation; (ii) specific techniques and applications according to different forms of knowledge data. This survey can have broad audiences, researchers and practitioners, in academia and industry.

1 INTRODUCTION

Knowledge-enhanced text generation addresses the limited knowledge available from input text alone by incorporating internal or external knowledge into neural generation. This survey reviews general integration methods and techniques and applications organized by knowledge source.

  • Motivation: Input text alone provides limited knowledge, while standard neural generation also struggles with language comprehension, memory, and reasoning over complex concepts and relations.These limitations motivate incorporating knowledge beyond the input text.
  • Knowledge Sources: Knowledge sources divide into internal knowledge from the input text and external knowledge from sources such as Wikipedia, ConceptNet, knowledge bases, knowledge graphs, and grounded text.Internal examples include keywords, topics, linguistic features, and internal graph structures; external examples include commonsense triples, reviews, and background documents.
  • Definition: Knowledge-enhanced text generation incorporates additional knowledge K to improve output Y given input X by leveraging dependencies among input, knowledge, and output.Knowledge may include information extracted internally or acquired from external sources.
  • Applications: Knowledge-enhanced systems have generated more informative, logical, and coherent texts, including topic-aware dialogue responses and structured summaries supported by knowledge graphs.For example, a topic-aware model generated “Then hydrate and moisturize your skin.” for the input “My skin is so dry.”
  • Survey Scope: The survey presents a comprehensive review covering general knowledge-integration methods and source-specific techniques and applications, including graph neural networks, reinforcement learning, and neural topic modeling.It is described as the first comprehensive survey of knowledge-enhanced text generation.
  • Challenges: Knowledge-enhanced NLG faces challenges in obtaining useful knowledge from diverse sources and effectively understanding and leveraging that knowledge during generation.The survey organizes its discussion around these challenges and distinguishes internal from external knowledge enhancement.

2 GENERAL METHODS OF INTEGRATING KNOWLEDGE INTO NLG · 2.1 The Basic Text Generation Models · 2.2 Knowledge-enhanced Model Architectures

Knowledge-enhanced NLG builds on encoder-decoder models that encode input sequences and autoregressively decode outputs, then incorporates external knowledge through specialized architectures. The survey organizes these architectures around attention, copy and pointing, memory, graph networks, and pretrained language models.

  • 2.1 The Basic Text Generation Models: Encoder-decoder models use RNN, CNN, or Transformer encoders to represent variable-length inputs and generate variable-length outputs through conditional sequence modeling.RNN encoders read sequentially, CNN encoders use local convolutional windows, and Transformer encoders use self-attention for global dependencies.
  • 2.1 The Basic Text Generation Models: The decoder generates one token per step autoregressively, using previously generated tokens, while training commonly optimizes negative log likelihood through maximum likelihood estimation.The generation process is treated as sequential multi-label classification.
  • 2.2 Knowledge-enhanced Model Architectures: Knowledge-enhanced architectures adapt general neural model designs to reflect the particular knowledge type being incorporated into text generation.The survey presents specialized architectures as the most popular approach for incorporating knowledge.
  • 2.2.1 Attention Mechanism.: Knowledge-related attention combines hidden and knowledge context vectors to produce knowledge-aware representations, including keyword, topic, knowledge-base, knowledge-graph, and grounded-text attention.Attention weights reflect the importance of input hidden states for deciding the next hidden state, while knowledge attention scores representations such as topic or graph-node vectors.
  • 2.2.2 Copy and Pointing Mechanisms.: CopyNet and pointer-generator models place input subsequences into outputs using differentiable generate-mode and copy-mode distributions over an extended vocabulary.Pointer-generator explicitly calculates a switch probability between the two modes and reuses attention as the copy distribution.
  • 2.2.2 Copy and Pointing Mechanisms.: Knowledge-related modes extend copying beyond the input by selecting subsequences from knowledge sources, adding entities and relations and combining generate-, copy-, and knowledge-base modes.These modes can produce words absent from both the global vocabulary V and source vocabulary V_X.
  • 2.2.3 Memory Network.: Memory-augmented encoder-decoders repeatedly retrieve external information during generation, supporting dialogue history and knowledge-base or topic modeling.Memory networks use recurrent attention over external memory and query vectors to read memories repeatedly.
  • 2.2.4 Graph Network / 2.2.5 Pre-trained Language Models.: Graph networks integrate knowledge graphs, dependency graphs, and other graph structures through message passing, while pretrained language models inject external entity representations into knowledge-intensive generation.Graph2Seq bridges graph representation learning and text generation, and PLM approaches include explicitly injecting representations pre-computed from external sources.

2.3 Knowledge-enhanced Learning and Inference

Knowledge-enhanced learning incorporates external knowledge through knowledge-informed tasks, conditioning, or constraints, while inference-time methods such as PPLM steer generation without necessarily changing the model architecture. These approaches support flexible knowledge integration across model architectures and data sources.

  • Knowledge-enhanced learning: Knowledge-enhanced learning designs supervised tasks or objectives that train models to acquire knowledge information, independently of a specific architecture.Knowledge can be expressed through diverse objective forms, making the approach flexible across model architectures.
  • Knowledge as target: Knowledge-as-target methods create knowledge-derived learning targets, using auxiliary tasks or typically noisy generated targets to supervise text generation.Examples include knowledge loss based on cross entropy, keyword extraction, template re-ranking, knowledge-graph link prediction, path reasoning, mode, and bag-of-word losses.
  • Knowledge as condition: Knowledge-as-condition methods learn p_θ(Y|X, K) by conditioning generation on externally provided, retrieved, or input-extracted knowledge.Knowledge may encode style or emotion, come from knowledge bases or Wikipedia, or be extracted from the input text.
  • Knowledge as constraint: Posterior regularization imposes knowledge constraints by restricting an auxiliary posterior distribution toward sequences with higher expected constraint values while keeping the model close through KL divergence.The framework augments a regular objective L(θ), can operate on unlabeled data, and introduces a slack variable ξ for constrained optimization.
  • Inference-time control: Plug and play language models steer generation during inference by repeatedly perturbing hidden histories and using differentiable attribute models, without requiring architectural modification or task-specific fine-tuning.PPLM is described as efficient and flexible for combining differentiable attribute models to control language generation.

3 NLG ENHANCED BY INTERNAL KNOWLEDGE

Topic knowledge enhances natural language generation by providing compressed semantic guidance for coherence and on-topic content. Existing approaches range from separate generative topic models to end-to-end neural and variational topic-generation frameworks, each with distinct strengths and limitations.

  • Topic-enhanced NLG: Topic modeling represents high-level document content as latent topics that can maintain semantic coherence and guide the NLG process.Latent Dirichlet allocation (LDA) is a classical method for inferring low-dimensional representations of latent word and document semantics.
  • Applications: Topics support dialogue, translation, and paraphrase by encouraging informative on-topic responses and preserving or determining cross-lingual and semantic content.They address trivial, non-committal dialogue responses and provide auxiliary guidance for translation and paraphrase generation.
  • Topic-enhanced NLG: Topic-enhanced NLG first discovers topics, commonly with LDA, then incorporates their representations into neural generation models.Topics can help interpret sentence meaning and constrain the semantic range of generated text.
  • Topic representations: Convolution-based topic extractors can outperform LDA-based models across dialogue, summarization, and translation applications, but their theoretical grounding and interpretability are weaker.The passage reports empirical superiority without providing a theoretical analysis guaranteeing convolution-captured topic quality.
  • Neural topic models: End-to-end neural frameworks jointly learn latent topics and generate outputs, while neural topic models enable backpropagation, joint optimization, scalability, and often better topic coherence than LDA.CNN-based frameworks remain limited by missing theoretical quality analysis and lower interpretability, whereas neural variational approaches assume isotropic-Gaussian topic distributions.

4 NLG ENHANCED BY EXTERNAL KNOWLEDGE · 4.1 NLG Enhanced by Knowledge Base

Knowledge bases enhance NLG by supplying factual and relational knowledge beyond the input, but effective use requires addressing discrete concept representation and knowledge selection. The survey organizes KB-enhanced methods around jointly optimized supervised tasks or selecting relevant KB facts, while noting noise as a remaining challenge.

  • 4.1 NLG Enhanced by Knowledge Base: Knowledge bases store large-scale information as subject–predicate–object triples that encode commonsense, factual events, and semantic relationships for NLG.These knowledge types help models discover dependencies within sequences and across input and output sequences.
  • 4.1 NLG Enhanced by Knowledge Base: KB-enhanced NLG applications include question answering, where generating proper answers may require completion, elaboration, reasoning, or inference beyond the question.Incorporating only the input question into neural generation models often fails because suitable answers can take different forms.
  • 4.1 NLG Enhanced by Knowledge Base: The survey categorizes KB-enhanced methods into joint optimization of supervised tasks around KBs and enhanced incorporation through selecting KB facts.These are designated methodologies M1 and M2, respectively.
  • 4.1.1 M1: Design Supervised Tasks around KB for Joint Optimization.: M1 jointly trains answer generation with question understanding and KB fact retrieval in a unified encoder–decoder framework.The additional tasks share knowledge with the target generation task and help address vanilla Seq2Seq limitations in representing discrete isolated concepts.
  • 4.1.2 M2: Enhance Incorporation by Selecting KB or Facts in KB.: M2 addresses inappropriate fact selection by approximating the posterior distribution with the prior distribution, enabling knowledge selection without posterior information.Lian et al. and Wu et al. introduce KLDivLoss to measure proximity between prior and posterior distributions.
  • 4.1.2 M2: Enhance Incorporation by Selecting KB or Facts in KB.: M2-based methods can retrieve more precise triples and improve generation performance compared with M1-based methods.The survey reports this comparison using entity recall scores, while also noting exact match and recall as retrieval-use measures.
  • 4.1.2 M2: Enhance Incorporation by Selecting KB or Facts in KB.: M2 training uses posterior context from ground-truth responses to supervise the prior fact-probability distribution, with total loss combining KLDivLoss and NLL generation loss.The posterior distribution p(k|X,Y) serves as labels for approximating the prior distribution p(k|X).
  • 4.1.2 M2: Enhance Incorporation by Selecting KB or Facts in KB.: Retrieved facts may contain noise, making generation unstable and posing particular risks for KB-based question answering and task-oriented dialogue systems.The problem is especially harmful when KB information supplies expected response entities.

4.2 NLG Enhanced by Knowledge Graph

Knowledge graphs enhance text generation by supplying structured entities, relations, and semantic information beyond input sequences. Methods retrieve or encode KG knowledge through embeddings, triplets, multi-hop reasoning, or graph neural networks, supporting applications such as commonsense reasoning and creative writing.

  • Knowledge Graph Foundations: A knowledge graph represents entities and relations as a structured, directed, multi-relational graph whose connections are central to the represented knowledge.Formally, a KG is G = (U, E, R), with typed edges E ⊆ U × R × U.
  • Applications: KG-enhanced generation supports commonsense reasoning and creative writing by exploiting semantic and structural information, multi-hop evidence chains, and paths representing scientific or story progression.Applications include abductive and counterfactual reasoning, scientific explanation, and story generation.
  • Knowledge Graph Foundations: Sequence-associated K-hop subgraphs collect linked entities and their neighbors, providing a graphical representation that enables graph-based integration with task sequences.The subgraph is formed using entity linking and neighboring nodes within K hops.
  • Integration Methodologies: Four methodologies incorporate KG semantics into generation: embeddings, triplet-based language-model transfer, path-based reasoning, and graph-neural-network-enhanced embeddings.These are labeled M1 through M4, respectively.
  • Methodological Tradeoffs: M1 and M2 preserve mainly one-hop knowledge, whereas M3 and M4 use multi-hop information but incur higher complexity and may retrieve subgraphs with limited useful-concept coverage.M3 can face a large path-finding action space, while M4 must attend to both the input sequence and KG.
  • Knowledge Sources: 57.1% of surveyed papers use ConceptNet as an external KG, while 42.9% construct domain-specific KGs from corpora.The survey reports eight papers using ConceptNet and six constructing their own KGs.

4.3 NLG enhanced by Grounded Text

Grounded text enhances NLG by adding relevant external textual knowledge, especially for dialogue and summarization. Methods either guide generation with retrieved information or model background knowledge into response generation.

  • Overview: Grounded text supplies relevant external knowledge from sources such as encyclopedias, social media, and shopping websites.This information may be absent from training corpora or structured databases and is used to better understand the input.
  • Applications: Knowledge-grounded dialogue systems retrieve background information to produce appropriate, factually meaningful responses.Fully data-driven systems often lack universal knowledge because it is not represented in their training corpora.
  • Applications: 3% of summaries contain fewer than three words, while 4% repeat a word more than 99 times, illustrating control problems in input-only Seq2Seq summarization.Such models also tend to copy source words in their exact order.
  • Methodologies: Grounded-text methods comprise retrieval-guided generation and background-knowledge modeling.The two methodologies are labeled M1 and M2, respectively.
  • Methodologies: Because grounded text is unstructured and noisy, systems retrieve relevant external documents, reviews, or templates and incorporate them during generation.This resembles knowledge acquisition and incorporation for knowledge bases and graphs, but uses noisier unstructured text.
  • M1.1: Retrieval-augmented generation (RAG): Retrieval-augmented generation first retrieves top-K passage distributions with p(Z|X), then generates tokens conditioned on the input, passage, and previous tokens.RAG therefore uses separate retrieval and generation stages.
  • M1.1: Retrieval-augmented generation (RAG): RAG variants that attend over concatenated representations of all retrieved passages can outperform RAG on the KILT benchmark.The cited methods use a pre-trained neural retriever and generator framework or related retrieval-and-generation designs.
  • M1.2: Retrieve, rerank and rewrite (R3): Retrieve-rerank-rewrite methods select a precise reference document for direct rewriting or editing and have succeeded in machine translation and summarization.Unlike RAG, they emphasize lightly editing one retrieved item rather than aggregating several retrieved contents.

4.4 M2: Modeling Background Knowledge into Response Generation

Background-based conversation models incorporate global background documents to keep dialogue on topic and generate more informative responses. They connect multiple information segments, sometimes through machine reading, to identify knowledge for response generation, but may still fragment semantic units and produce short replies.

  • 4.4 M2: Modeling Background Knowledge into Response Generation: Background documents provide global knowledge that helps generate informative responses and keep conversations from deviating from their topic.This setting is called background-based conversation (BBC).
  • 4.4 M2: Modeling Background Knowledge into Response Generation: Many BBC models connect multiple information pieces to identify appropriate background knowledge before generating the next response.This approach follows the idea of combining text segments to produce unified answers in machine reading comprehension.
  • 4.4 M2: Modeling Background Knowledge into Response Generation: BBC systems avoid generic dialogue responses by exploring related background information, but may break complete semantic units and generate shorter responses.These limitations remain unresolved in existing methods.
  • 4.4 M2: Modeling Background Knowledge into Response Generation: Wikipedia is the most commonly used evidence source in grounded text-enhanced work, while platforms such as Amazon provide product-review information.The surveyed datasets are public, with dataset links provided elsewhere in the survey.
  • 4.4 M2: Modeling Background Knowledge into Response Generation: 57.1% of grounded text-enhanced papers added retrieval supervision, while 42.9% used pretrained language models for document representations.Retrieval supervision included human-annotated or pseudo labels, and pretrained representations aimed to improve retrieval accuracy.

5 BENCHMARK, TOOLKIT AND LEADERBOARD PERFORMANCE

The section introduces a knowledge-enhanced NLG benchmark assembled by re-screening four existing text-generation benchmarks. It finalizes nine knowledge-centric datasets spanning diverse generation tasks and applications.

  • The benchmark re-screens GLGE, GEM, KilT, and GENIE to identify datasets specifically suited to knowledge-enhanced text generation.The selected datasets are drawn from four general NLG benchmarks or SemEval tasks.
  • Selection criteria require open-access datasets, diverse applications, at most three datasets per task, internal and external knowledge, and preferably multiple references.These criteria support broad coverage and more robust automatic evaluation.
  • Nine knowledge-centric tasks cover commonsense reasoning, text summarization, question generation, generative question answering, and dialogue.The benchmark selection aims to represent various NLG tasks and applications.
  • Wizard of Wikipedia grounds open-domain dialogue in retrieved Wikipedia knowledge, while CommonGen generates coherent everyday scenarios from supplied concepts.These datasets exemplify dialogue and generative commonsense reasoning within the benchmark.
  • PersonaChat evaluates engaging open-domain chitchat conditioned on profile information.The dataset represents dialogue generation using persona knowledge.

6 DISCUSSION ON FUTURE DIRECTIONS

Future work in knowledge-enhanced text generation should improve knowledge representation and integration while broadening the knowledge sources, learning settings, and applications addressed. Promising directions include visual-language generation, pre-trained language-model knowledge, few-shot adaptation, and lifelong learning.

  • Designing more effective ways to represent knowledge and integrate it into generation remains the most important trend in knowledge-enhanced NLG.
  • Visual-language generation: Knowledge-enhanced visual-language generation remains open to exploration through additional image/text retrieval, structured justifications, and audience- or context-specific captions.These directions target open-domain visual question answering and image/video captioning, among other tasks.
  • Multiple knowledge sources: Future systems should discover knowledge more broadly and combine multiple forms from sources such as network structures, dictionaries, and tables.Prior examples include citation graphs for scientific-paper intention detection and summarization, and retrieval for rare-word representations.
  • Pre-trained language models: Pre-trained language models offer substantial parameterized implicit knowledge, but directly fine-tuning them for story generation can still produce insufficient knowledge, repetition, and logical inconsistencies.
  • Few-shot learning: Few-shot adaptation is needed because real-world new-domain applications often provide only a few examples, limiting the knowledge learned for topics and syntactic structures.
  • Lifelong learning: Lifelong learning aims for systems that continuously accumulate knowledge from previous tasks and use it to assist future learning, but existing knowledge-enhanced systems remain limited.

7 CONCLUSIONS

The survey reviews representative research efforts and trends in knowledge-enhanced text generation, addressing how to acquire knowledge and incorporate it to facilitate generation. Its content is organized into three sections based on knowledge acquisition.

  • 7 CONCLUSIONS: The survey provides a comprehensive review of current representative research efforts and trends in knowledge-enhanced text generation.The authors expect this review to facilitate future research.
  • 7 CONCLUSIONS: It addresses how to acquire knowledge for knowledge-enhanced text generation.
  • 7 CONCLUSIONS: It also addresses how to incorporate knowledge to facilitate text generation.
  • 7 CONCLUSIONS: The survey’s main content is divided into three sections according to knowledge acquisition.

A APPENDIX

The appendix summarizes publication venues, paper categories, research trends, benchmark leaderboards, and representative open-source systems in knowledge-enhanced text generation. Among 160 selected papers, 87 are categorized as general methods.

  • Publication statistics: Most selected papers appeared in leading machine learning, artificial intelligence, and natural language processing conferences or high-impact journals.Examples include ACL, EMNLP, AAAI, ICLR, NeurIPS, TNNLS, JMLR, and TACL.
  • Research trends: Figure 9 summarizes selected papers by publication year, knowledge source, and method.It indicates that knowledge-enhanced text generation has gained emerging interest over the recent five years.
  • Benchmark results: Table 10 reports leaderboard performance on ten knowledge-enhanced generation benchmarks.The passage identifies the benchmark count but provides no individual performance values.
  • Open-source systems: Table 11 provides code links and programming languages for representative open-source knowledge-enhanced text generation systems.These systems were introduced in the survey.

A.1 Evaluation Metrics

This section introduces metrics for evaluating knowledge-enhanced text generation, covering n-gram precision, reference–hypothesis overlap, sequence matching, and normalized diversity. It also identifies the primary metrics used across summarization, commonsense, dialogue, question-answering, and explanation benchmarks.

  • BLEU-𝑚 (B-𝑚) measures a weighted geometric mean of 𝑛-gram precision scores.
  • ROUGE-𝑚 (R-𝑚) measures reference–hypothesis n-gram overlap, while ROUGE-L uses the longest matched words via longest common sub-sequence.
  • Distinct-𝑘 (D-k) measures unique 𝑘-grams normalized by generated 𝑘-gram tokens to avoid favoring long sentences.
  • ROUGE-2 and ROUGE-L are standard n-gram-based metrics for knowledge-enhanced summarization benchmarks.
  • B-4 and R-L are commonly used on αNLG-ART and ComVE, while SPICE is primary on CommonGen and R-L is primary on Holl-E.Wizard of Wikipedia reports seen and unseen test sets; Kilt R-L is primary on ELI5, with state-of-the-art results also reported for SQuAD and PersonaChat.
Loading 2010.04389v4…