Source-linked AI summary
Pretrained Language Models for Text Generation: A Survey
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen
TL;DR
Text-generation research faces limited supervised data and the challenge of adapting models to varied inputs and desired output properties. This survey synthesizes PLMs for text generation, covering task definitions, architectures, input adaptation, output control, and fine-tuning strategies. It concludes with future directions involving pretraining-task alignment, steerable control, and ethical safeguards.
Problem
Supervised datasets for many text-generation tasks are small, while neural models can overfit; PLMs also require adaptation to varied inputs and output properties.
Method
The paper surveys PLM task definitions, architectures, methods for modeling different input data and output properties, and fine-tuning strategies.
Results
The survey provides an overview of recent advances in PLMs for text generation and synthesizes approaches for different inputs, special output properties, and fine-tuning.
Takeaways & Limitations
PLM-based text generation research can be organized around input adaptation, output requirements, and fine-tuning for task or domain characteristics.
Takeaways & Limitations
Web-scale pretraining without fine-grained filtering can create privacy and bias risks, and general methods for preventing these issues remain preliminary.
Abstract
from arXiv · showhide
Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present an overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition and briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing PLMs to model different input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation. Finally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research.
1 Introduction
Text generation is an important but challenging NLP task, and neural models have advanced it through end-to-end semantic mapping and dense representations. The survey focuses on pretrained language models (PLMs), which address dataset limitations by reusing knowledge learned from large corpora and organizes advances by input adaptation, output properties, and fine-tuning.
- Text generation produces plausible, readable human-language text from input data and supports applications including translation and summarization.
- Neural models enable end-to-end learning of semantic mappings and learn dense vectors that help represent linguistic features and alleviate data sparsity.
- Small supervised datasets create a performance bottleneck because large neural networks can overfit and generalize poorly.
- PLMs are pretrained on large-scale corpora, encode linguistic knowledge, and can be fine-tuned for downstream tasks instead of training new models from scratch.
- The survey reviews task definitions, mainstream PLM architectures, input-data adaptation, output properties, fine-tuning strategies, and future directions.
2 Task and Typical Applications
Text generation models a conditional token sequence P(Y|X), with applications determined by whether the input is absent, attribute-based, structured, multimedia, or textual. The section presents formulations for these major task types.
- Text generation produces a token sequence Y conditioned on input X, formally represented as P(Y|X).
- Without input or with random noise, generation becomes language modeling or unconditional generation without constraints.
- Discrete attributes such as topic words or sentiment labels guide generation and control the modes of the output text.
- Structured inputs such as knowledge graphs and tables define data-to-text generation, which produces descriptive text about the data.
- Multimedia inputs support image captioning and speech recognition, while text inputs support translation, summarization, and dialogue systems.
- Table 1 presents formulations for the major text-generation tasks.
3 Standard Architectures for Text Generation
PLMs for text generation primarily use Transformer backbones in encoder-decoder or decoder-only configurations. Decoder-only models can condition on input through sequence concatenation and specialized masking, while explicit encoder-decoder attention is reported as beneficial.
- PLMs encode linguistic and world knowledge in their parameters, enhancing language understanding and generation quality for new tasks.
- Most PLMs employ Transformer backbones, using either standard encoder-decoder architectures or decoder-only architectures.
- Encoder-decoder Transformer: Encoder-decoder Transformers encode an input sequence and generate an output sequence through encoder-decoder self-attention; MASS, T5, and BART follow this design.
- Decoder-only Transformer: Decoder-only models such as GPT and CTRL use unidirectional masking, allowing each token to attend only to preceding tokens.
- Decoder-only Transformer: Decoder-only PLMs can perform conditional generation by concatenating input and output sequences and applying seq2seq masking.
- Decoder-only Transformer: The survey reports that adding explicit encoder-decoder attention is beneficial compared with the alternative masking approach.
- Text-generation research must model both varied input data and properties required of the generated output.
4 Modeling Different Data Types from Input
PLM-based generation adapts to unstructured, structured, and multimedia inputs, each presenting distinct encoding or supervision challenges. Approaches include hierarchical and cross-lingual encoders, structured-data serialization or graph encoding, and unsupervised or semi-supervised multimedia pretraining.
- The survey organizes input modeling into unstructured, structured, and multimedia data types.
- Unstructured Input: PLM text encoders condense sentences, paragraphs, and documents into low-dimensional vectors while preserving their meaning.
- Unstructured Input: Long documents challenge PLMs trained on shorter text because accurately modeling long-range dependencies is difficult.
- Unstructured Input: Cross-lingual PLMs address multilingual generation involving multiple languages and low-resource settings where monolingual PLMs are limited.
- Structured Input: Few-shot PLM approaches are explored for structured-data generation because large labeled datasets with ground-truth text are difficult to collect.
- Structured Input: Structured inputs are adapted through graph or template-based serialization, linearized triples, or additional graph encoders.
- Structured Input: Pointer generators copy words from input knowledge data to improve fidelity, while external knowledge can support generalization.
- Multimedia Input: Multimedia PLM methods include video-caption pretraining and weakly supervised speech-recognition approaches using unpaired speech and transcripts.
5 Satisfying Special Properties for Output Text
PLM-based text generation addresses three output properties: relevance to inputs and conditions, faithfulness to source facts, and preservation of semantic-unit order. The survey describes task-specific strategies for improving each property, including context modeling, source alignment, and multilingual pretraining.
- Text generation outputs should satisfy relevance, faithfulness, and order-preservation across different task requirements.
- Relevance: PLMs can improve dialogue relevance and context consistency compared with traditional RNN-based models.TransferTransfo and DialoGPT are cited as examples.
- Relevance: Condition blocks, distinct input representations, self-attention masks, and attention routing incorporate external conditions into dialogue generation.
- Faithfulness: Faithfulness requires generated content to avoid contradicting input facts, while PLM initialization can make models more confident in predicting document tokens.
- Faithfulness: Theme modeling loss makes summaries semantically close to their source articles, supporting faithful generation in target domains.
- Order-preservation: Order-preservation keeps semantic-unit order consistent between input and output; alignment information and multilingual pretraining are described as supporting approaches.CSP uses word-pair alignments, while mRASP pretrains a universal multilingual translation model.
6 Fine-tuning Strategies for Text Generation
The survey organizes fine-tuning strategies around adapting PLMs to new domains and data scarcity while addressing coherence, fidelity, and overfitting. It reviews few-shot learning, continued domain pretraining, task-specific objectives, and model- or learning-process-based guidance.
- Fine-tuning strategies are reviewed from multiple views to adapt PLMs for text generation.
- Data View: Effective fine-tuning in new domains must account for domain characteristics and distribution differences from pretraining data.
- Data View: Few-shot, one-shot, and zero-shot learning fine-tune pretrained modules with few, one, or no task examples to address annotation scarcity.
- Data View: Cross-lingual PLMs transfer knowledge from high-resource languages to support low-resource multilingual generation.XLM is cited as a cross-lingual language-modeling approach.
- Data View: Domain transfer continues PLM training on specific data with pretraining objectives before downstream fine-tuning.Masking variants include TF-IDF-based masking for condition-related tokens and document masking for long-document summarization.
- Output Properties: Fine-tuning can target coherence through contrastive learning, next sentence prediction, sentence reordering, and denoising autoencoding.
- Output Properties: Fidelity can be improved by reconstructing table embeddings, copying knowledge-graph entities and relations, or using task-specific semantic objectives.
- Model and Learning Views: Model-guidance strategies include teacher supervision, separate optimizers, reinforcement learning, curriculum learning, and maximum mutual information scoring.
7 Conclusion and Future Outlooks
The survey identifies extensions of PLMs for input data, output properties, and fine-tuning, then outlines future challenges involving pretraining, controllability, compression, multilinguality, and ethics.
- The survey summarizes PLM extensions for modeling different input data, satisfying special output properties, and applying fine-tuning strategies.
- Model Extension: Pretraining should better align with downstream generation, including replacing mismatched objectives such as the unused “[MASK]” token and incorporating external knowledge.
- Controllable Generation: Controllable generation remains early-stage because universal-corpus pretraining makes multi-grained control of sentiment, topic, and coherence difficult.
- Model Compression: PLM compression is needed for resource-constrained deployment, but existing methods have focused mostly on BERT-based models rather than generation.
- Fine-tuning Exploration: Future fine-tuning research can transfer PLM knowledge through alternatives such as distillation, including using BERT as teacher and an RNN generator as student.
- Further directions include language-agnostic PLMs for non-English generation and interventions addressing privacy risks and biases in web-trained models.