PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretrained Language Models for Text Generation: A Survey

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, Ji-Rong Wen

arXiv:2201.05273v4cs.CL

TL;DR

Text generation research needs methods that produce readable text from diverse inputs despite limited labeled data and difficult output properties. This survey synthesizes PLM-based approaches across input representation, model architecture, optimization, challenges, resources, applications, and future directions. It concludes that PLM-based generation has advanced the state of the art, while substantial adaptation work remains and bias mitigation is still unresolved.

  • Problem

    Limited labeled data constrains text-generation methods, while diverse inputs, semantic fidelity, and difficult-to-optimize output properties create further challenges.

  • Method

    The survey organizes PLM-based text generation around input representation learning, model architecture design, parameter optimization, challenges, resources, applications, and future directions.

  • Results

    The survey reports that PLM-based text generation has greatly contributed to advances in the field’s state of the art.

  • Takeaways & Limitations

    The synthesis is intended to help researchers and practitioners learn core concepts, techniques, developments, resources, and applications for PLM-based text generation.

  • Takeaways & Limitations

    No general, unified approach currently reduces data bias in PLMs for text generation, and research on general mitigation remains preliminary.

Abstract

from arXiv · show

Text Generation aims to produce plausible and readable text in a human language from input data. The resurgence of deep learning has greatly advanced this field, in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as a promising approach in both academia and industry. In this paper, we provide a survey on the utilization of PLMs in text generation. We begin with introducing three key aspects of applying PLMs to text generation: 1) how to encode the input into representations preserving input semantics which can be fused into PLMs; 2) how to design an effective PLM to serve as the generation model; and 3) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we show the major challenges arisen in these aspects, as well as possible solutions for them. We also include a summary of various useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn the core concepts, the main techniques and the latest developments in this area based on PLMs.

1 INTRODUCTION

Text generation produces plausible, readable human-language text from varied inputs, but limited labeled data constrained earlier neural methods. This survey examines PLM-based generation through input representation, model design, and optimization.

  • Text generation produces plausible and readable human-language text from inputs including text, images, tables, and knowledge bases.
  • Large-scale labeled-data requirements limited applicability, while small datasets increased overfitting and weakened generalization.
  • PLMs pre-train on large unsupervised corpora before downstream fine-tuning, encoding linguistic knowledge in large-scale parameters.
  • Unlike application-specific surveys, this work provides a broader survey of the core text-generation techniques based on PLMs.
  • The survey organizes PLM-based text generation around input representation learning, model architecture design, and parameter optimization.

2 PRELIMINARY

Text generation models map input data to output text while satisfying desired properties such as fluency, relevance, accuracy, and coherence. PLM-based methods must address input encoding, architecture design, and optimization.

  • 2.1 Text Generation: Text generation models produce output y from input x while satisfying desired properties P, using a PLM-based generation function f_M.
  • 2.1 Text Generation: Different inputs yield distinct tasks, including unconditional generation, attribute-based generation, data-to-text, multimedia generation, translation, summarization, and dialogue.
  • 2.2 Pre-trained Language Models: PLMs are pre-trained on large unlabeled corpora and fine-tuned downstream, encoding linguistic knowledge into their parameters.
  • 2.3 PLM-based Text Generation Methods: Applying PLMs to generation requires encoding diverse inputs while preserving semantics, designing task-adapted architectures, and optimizing outputs for desired properties.
  • 2.3 PLM-based Text Generation Methods: General-purpose PLM architectures may not handle special generation cases, and some desired output properties are difficult to formulate or optimize.

3 ENCODING INPUT REPRESENTATIONS

The survey frames input representation learning as the first major component of PLM-based text generation. It categorizes inputs into unstructured, structured, and multimedia types.

  • Input representation learning encodes input data into meaningful representations that preserve semantics for fusion with PLMs.
  • The survey covers three main input categories: unstructured input, structured input, and multimedia input.

3.1 Unstructured Input

Unstructured-input representation learning addresses semantic modeling for paragraphs and documents, including hierarchy, graphs, critical-content selection, efficiency, and multilingual settings.

  • 3.1.1 Paragraph Representation Learning: Paragraph representation methods capture both word-level meanings and high-level topic semantics using hierarchical or graph-based approaches.
  • 3.1.1 Paragraph Representation Learning: Flat concatenation can miss semantic dynamics across utterances, motivating hierarchical encoders and utterance-level information-flow modeling.
  • 3.1.1 Paragraph Representation Learning: Graph-based methods represent words or phrases as connected nodes to aggregate relevant, potentially disjoint context and reduce redundant information.
  • 3.1.2 Document Representation Learning: Long-document encoding must model cross-sentence semantics and retain the most critical content for generation.
  • 3.1.2 Document Representation Learning: Because self-attention grows quadratically with sequence length, local attention, content selection, and divide-and-conquer methods improve long-document efficiency.
  • 3.1.3 Multilingual Representation Learning: English-focused pre-training makes multilingual generation difficult, especially for low-resource languages, prompting multilingual representation approaches.

3.2 Structured Input

Structured inputs create a semantic and representational mismatch with PLMs, while generation must also remain faithful to the source data. The survey organizes solutions around encoding structure, aligning representations, adding structural signals, and preserving input content.

  • Structured data poses semantic-gap, structural-encoding, and fidelity challenges because PLMs are typically pretrained on natural-language text.
  • Encoding Structured Input: Linearization converts structured inputs such as knowledge graphs into sequences that PLMs can process.Knowledge-graph triples may be concatenated, while templates can serialize attribute-value pairs.
  • Encoding Structured Input: Representation alignment projects structured representations into semantic spaces compatible with PLM word embeddings.One example uses graph neural networks to project knowledge-graph entities and minimizes Euclidean distance between representations.
  • Capturing Structural Information: Structural information can be captured through auxiliary objectives, explicit structural tokens or embeddings, and dedicated structural encoding modules.Examples include reconstructing table structure, prepending head-relation-tail markers, and adding graph convolutions or graph neural networks.
  • Maintaining Text Fidelity: Fidelity means that generated text adheres to structured input, a central requirement for accurate data-to-text generation.The survey discusses content-matching and fidelity-classification losses, copying input entities or values, and using target logical forms.

3.3 Multimedia Input

Multimedia inputs extend PLM-based text generation beyond text to images, video, and speech. The surveyed approaches use multimodal pretraining, modality-specific encoders, and unsupervised or semi-supervised learning to support these tasks.

  • Multimedia text generation uses image, video, and speech inputs in applications such as image captioning and speech recognition.
  • Image Captioning: Multimodal PLMs combine visual and textual modalities for image captioning by pretraining on image-to-text generation objectives.XGPT and VisualGPT are examples of image-captioning PLMs.
  • Image Captioning: Three pretraining tasks align text words, visual objects, and scene text representations for multimodal input.
  • Video Captioning: Video-language models support video captioning, with UniVL separating text and video encoding before sentence decoding.Earlier approaches commonly used a single encoder-decoder framework, whereas UniVL uses two single-modal encoders and a sentence decoder.
  • Speech Recognition: Speech recognition integrates PLMs through unsupervised and semi-supervised methods because human-transcribed supervised data is limited.Examples include pretraining encoder-decoder models with unpaired speech and transcripts and post-processing noisy recognition outputs.

4 DESIGNING PLMS FOR TEXT GENERATION

PLM-based text generation models factorize output probability over tokens and rely mainly on Transformer architectures. The survey compares single-Transformer variants with encoder-decoder designs and reviews improved embeddings, attention mechanisms, and multi-source fusion.

  • Text generation models represent output probability as token-level conditional probabilities given the input and previously generated tokens.The previous-token context is denoted y<ᵢ, and the input data is x.
  • Transformers became the backbone of large PLMs because attention captures long-range dependencies and supports efficient parallelization.
  • Transformer Architectures: Single-Transformer PLMs include masked, causal, and prefix language models with different attention-mask strategies.GPT-3 and UniLM are examples of single-Transformer models.
  • Transformer Architectures: Masked LMs use bidirectional masked-token prediction but are rarely used directly for generation because pretraining and generation objectives differ.They are more commonly used as encoders in generation models.
  • Transformer Architectures: Causal LMs generate the next word from previous words, while prefix LMs encode inputs bidirectionally and generate outputs left to right.Prefix LMs use mixture attention so input tokens attend mutually while output tokens attend to inputs and earlier outputs.
  • Transformer Architectures: Explicit encoder-decoder attention was found more effective than single-Transformer prefix attention for capturing conditional dependencies.
  • Transformer Architectures: Encoder-decoder LMs use separate encoder and decoder stacks with denoising-style pretraining objectives, as in MASS, ProphetNet, T5, and BART.
  • Improving Transformer Components: The survey reviews extended input embeddings, auxiliary embeddings, sparse or improved attention, and cross-attention strategies for multi-source inputs.These techniques address positional, dialogue, language, long-context, and input-aggregation requirements.

5 OPTIMIZING PLMS FOR TEXT GENERATION

PLM optimization for text generation includes fine-tuning, prompt-tuning, and property-tuning, with the supplied material focusing on fine-tuning variants. These methods incorporate task or domain knowledge while parameter-efficient approaches reduce the cost of updating large models.

  • The survey categorizes PLM optimization into fine-tuning, prompt-tuning, and property-tuning.
  • Fine-Tuning: Fine-tuning adapts pretrained linguistic knowledge to downstream generation by updating PLM weights on task-specific datasets.
  • Fine-Tuning: Fine-tuning variants include vanilla, intermediate, parameter-efficient, and multi-task fine-tuning.Intermediate and multi-task methods can alleviate overfitting on small generation datasets to some extent.
  • Intermediate Fine-Tuning: Intermediate fine-tuning uses labeled data from a related task or domain to infuse domain- or task-specific knowledge and improve small-data generation.DAIFT uses a related NLP task in the same domain, whereas TAIFT uses the same generation task in another domain.
  • Multi-Task Fine-Tuning: Multi-task fine-tuning adds auxiliary tasks to transfer cross-task knowledge, improve robustness, and reduce reliance on large amounts of labeled generation data.Examples include additional summarization datasets and auxiliary tasks for coherence or style control.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning addresses the time and resource costs of updating all PLM parameters.
  • Parameter-Efficient Fine-Tuning: Adapters add small bottleneck modules while freezing the original PLM, limiting per-layer additions to 2md + d + m parameters.The bottleneck dimension m is set much smaller than the original feature dimension d.
  • Parameter-Efficient Fine-Tuning: Other parameter-efficient approaches freeze most parameters or distill large teacher PLMs into smaller student models.Freezing methods may prioritize cross-attention layers, while distillation transfers generation knowledge into compact models such as LSTMs.

5.2 Prompt-Tuning for Text Generation

Prompt-tuning reformulates text generation tasks as language-modeling problems by converting inputs into prompts. The survey covers discrete and continuous prompt designs, including templates, slots, and trainable embeddings.

  • Generative PLMs are typically pre-trained with language-modeling objectives and then fine-tuned with task-specific objectives, creating a pre-training–fine-tuning discrepancy.
  • Prompt learning reformulates downstream text generation tasks into the language-modeling task used during pre-training.
  • A prompt function converts input text x into a prompt x′ through a two-step process.The supplied passage introduces the conversion but does not provide further procedural detail.
  • 5.2.1 Background.: A textual template uses an input slot [X] for x and an answer slot [Z] for an intermediate answer z later mapped into y.The input slot is filled with the input text after the template is applied.
  • 5.2.1 Background.: Cloze prompts place the empty answer slot within or at the end of a template, whereas prefix prompts place the input entirely before it.Prefix prompts are widely used for generation because they fit left-to-right language modeling; prompt tokens may also be virtual words represented by continuous embeddings.
  • 5.2.2 Discrete Prompts.: Manually designed discrete prompts define task-specific semantic mappings from input data to output text and allow one PLM to perform multiple generation tasks.GPT-2 is cited as an early example using prompts such as “translate to french, [input], [output]”.
  • 5.2.2 Discrete Prompts.: Continuous prompts, or soft prompts, use embedding vectors instead of natural-language templates and have parameters optimized on task training data.They relax the requirement that templates consist of natural-language words and need not be parameterized by PLM parameters.

5.3 Property-Tuning for Text Generation

Property-tuning addresses language properties that generation systems must satisfy, including relevance, faithfulness, and order-preservation. The survey links these properties to PLM mechanisms and alignment-based methods.

  • Text generation methods must consider task-specific language properties, with relevance, faithfulness, and order-preservation identified as major desired properties.
  • Relevance: Relevance means that generated text’s topical semantics are highly related to the input and its conditions.In dialogue, relevant responses should reflect historical utterances, speaker persona, and discourse topic.
  • Relevance: PLMs’ multi-layer cross-attention can model input–output semantic associations and enhance relevance in generated text.DialoGPT is presented as an autoregressive GPT-2 model trained on large-scale dialogue pairs and sessions.
  • Faithfulness: Faithfulness requires generated content to adhere to input semantics and, in some uses, to accord with world facts.Text summarization is given as an example requiring faithful communication of salient input information.
  • Faithfulness: PLMs’ language understanding and encoded world knowledge can support faithful generation by capturing core input semantics and supplying background knowledge.
  • Order-Preservation: Order-preservation requires semantic units to appear in consistent order across input and output, as needed in paraphrasing and machine translation.
  • Order-Preservation: Code-Switching Pre-training extracts source–target word alignments and continually pre-trains PLMs to predict source fragments from aligned target fragments.The method is designed to enhance order-preservation during translation.

6 CHALLENGES AND SOLUTIONS

The survey organizes challenges around data, architecture, and optimization, covering scarcity, bias, resource costs, and difficult generation properties. It summarizes transfer, augmentation, multitask learning, compression, scaling, knowledge enrichment, and planning solutions.

  • The survey maps major challenges and solutions to three aspects of PLM-based text generation: data, architecture, and optimization.Table 1 summarizes the challenges and existing PLM-based solutions.
  • Data Challenges: Data scarcity in text generation can be addressed through transfer learning, data augmentation, and multi-task learning.
  • Data Challenges: Transfer learning fine-tunes PLMs on large labeled external corpora before adapting them to target generation tasks with limited labeled data.Examples cover dialog and summarization in new domains.
  • Data Challenges: Data augmentation increases available training data using retrieved real examples, modified existing examples, or synthetic data.Retrieval models can obtain augmentation data from external corpora.
  • Data Challenges: Multi-task learning uses data-rich auxiliary generation tasks, while unified decoders can preserve alignments and patterns across tasks.Independent decoders may break semantic connections between high- and low-resource tasks.
  • Data Bias: PLMs inherit biases and stereotypes from real-world training data, creating downstream challenges for text generation.
  • Data Bias: No general unified approach currently reduces data bias in PLMs for text generation, and some debiasing methods capture oversimplified bias dimensions.The passage states that more holistic evaluation is needed for proper debiasing.
  • Architecture Challenges: Transformer backbones are bulky and resource-hungry, causing high memory consumption, computational overhead, and energy cost.The survey discusses quantization, pruning, and knowledge distillation as compression approaches.

7 EVALUATION AND RESOURCES

The survey reviews automatic metrics, open-source libraries, benchmarks, and task resources for evaluating PLM-based text generation. It groups metrics by word overlap, diversity, semantic similarity, and probabilistic modeling.

  • Automatic evaluation is cheaper, quicker, and repeatable compared with human evaluation, so the survey concentrates on automatic metrics.
  • The survey groups metrics into n-gram overlap, diversity, semantic similarity, and logit-based categories.The metrics used for each generation task are listed in Table 2.
  • N-gram Overlap Metrics: N-gram overlap metrics measure word-level matching between machine-generated and ground-truth texts.
  • N-gram Overlap Metrics: BLEU- n measures n-gram precision and applies a length penalty to shorter generated text.SacreBLEU is recommended for machine translation, and smoothing methods are used for short sentences.
  • N-gram Overlap Metrics: ROUGE-n measures the F1 score of overlapping n-grams between generated and ground-truth texts for multi-sentence summarization.
  • N-gram Overlap Metrics: METEOR uses the harmonic mean of unigram precision and recall and matches words using WordNet.
  • N-gram Overlap Metrics: ChrF++ evaluates character-level n-gram matching to capture morpheme overlap rather than word-level co-occurrence.
  • Diversity Metrics: Lexical diversity is desirable in dialogue and story generation, where generated texts require diversity evaluation.

8 APPLICATION

The survey reviews how PLMs are adapted across machine translation, summarization, and dialogue systems, including supervised, unsupervised, and task-specific approaches.

  • Machine Translation: Machine translation is categorized as supervised or unsupervised according to whether parallel corpora are available for fine-tuning PLMs.
  • Machine Translation: Unsupervised machine translation uses multilingual monolingual pre-training followed by iterative back-translation, reducing reliance on large annotated corpora.
  • Machine Translation: Supervised translation methods either fine-tune existing self-supervised PLMs on bilingual pairs or design PLMs with objectives tailored to parallel corpora.
  • Machine Translation: Pre-training translation models on parallel data can improve task alignment, but requires massive labor and financial resources to create bilingual pairs.
  • Text Summarization: PLM-based summarization includes extractive methods that select source sentences and abstractive methods that generate summaries from abstract representations.
  • Text Summarization: Summarization systems can directly fine-tune PLMs or provide generation guidance through extracted keywords, sentences, relations, and topic semantics.
  • Dialogue System: Dialogue systems generate responses from conversation history and are commonly divided into open-domain and task-oriented systems.
  • Dialogue System: Dialogue-focused PLMs use continual pre-training on conversational or informal online text to adapt general-purpose models to dialogue.

9 CONCLUSION AND FUTURE DIRECTIONS

The survey synthesizes PLM-based text generation research, resources, applications, and challenges. It identifies controllability, optimization, multilinguality, and ethics as important directions while noting that current performance remains below expectations.

  • Conclusion: The survey organizes representative PLM-based text generation research around input representation learning, model architecture design, and parameter optimization.
  • Conclusion: It also reviews evaluation metrics, open-source libraries, and common applications to help practitioners evaluate, choose, and employ PLMs.
  • Future Directions: Controllable generation remains at an early stage because universal-corpus pre-training makes multi-grained attributes such as sentiment, topic, and coherence difficult to control.
  • Future Directions: Future optimization research can explore approaches beyond fine-tuning and prompt-based learning that combine the advantages of current methods.
  • Future Directions: Most PLMs for text generation are mainly English-oriented, motivating research on language-agnostic models that capture universal features across languages.
  • Future Directions: Web-scale pre-training without fine-grained filtering can expose privacy and bias risks, requiring interventions to prevent misuse and prejudiced generation.
  • Conclusion: Although PLMs have advanced the state of the art, current results across text generation tasks remain far from expectations and require further adaptation.
Loading 2201.05273v4…