PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

Albert Gatt, Emiel Krahmer

arXiv:1703.09902v4cs.CLcs.AIcs.NE

TL;DR

Earlier NLG surveys did not cover major developments in statistical methods, new applications, creative and affective generation, or evaluation methodology. This survey addresses those omissions by synthesizing core tasks, emerging interdisciplinary topics, and evaluation challenges, finding a field-wide shift toward data-driven and more integrated approaches alongside trade-offs in output quality, efficiency, and robustness.

  • Problem

    Earlier NLG coverage omitted newer statistical methods, applications beyond factual text generation, creative and affective generation, and evaluation methodology.

  • Method

    The survey synthesizes NLG core tasks and architectures, discusses newer interdisciplinary research topics, and examines problems and prospects in evaluating NLG applications.

  • Results

    The survey identifies a shift from rule-based to statistical, data-driven methods and from modular toward more integrated architectures, with output quality traded against efficiency and robustness.

  • Takeaways & Limitations

    NLG is expanding into applications such as journalism, weather, finance, image-to-text generation, social media, content creation, Twitter, and chatbots.

  • Takeaways & Limitations

    Controlling stylistic, affective, and personality-based variation remains fledgling, with open theoretical and computational questions about modeling and data.

Abstract

from arXiv · show

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past decade or so, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of relatively recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of Natural Language Processing, with an emphasis on different evaluation methods and the relationships between them.

1 Introduction

NLG generates text from non-linguistic data or existing text and now spans many applications, methods, audiences, and research concerns. This survey updates earlier coverage by synthesizing core tasks and architectures, highlighting newer AI-related topics, and examining evaluation challenges.

  • Applications: NLG includes data-to-text systems that convert non-linguistic data into reports across journalism, sports, weather, finance, healthcare, museums, and behavior change.The survey also discusses text-to-text applications such as translation, summarization, simplification, correction, paraphrasing, and question generation.
  • Applications: Data-to-text generation can tailor reports to specific audiences, varying technical detail and explanatory language for readers such as doctors, nurses, and parents.Computer-generated reports can also be personalized for different perspectives on the same event.
  • Scope: The survey defines its primary NLG focus by the input: systems generate text from data that is not, or is not exclusively, linguistic.Data-to-text and text-to-text generation can nevertheless share techniques, especially when systems learn to express information in new sentences.
  • Motivation: The survey is timely because NLG has expanded beyond earlier coverage to tailored reports, text-to-text and vision-to-text applications, affective and creative generation, and systematic evaluation.These developments are associated with increased use of statistical methods and new shared-task-driven attention to evaluation.
  • Survey goals: The paper provides an up-to-date synthesis of core NLG tasks and architectures, highlights synergies with computer vision, stylistics, and computational creativity, and reviews evaluation methods.Its stated goals cover foundational research, newer AI-related issues, and relationships among evaluation methods.

2 NLG Tasks

Traditional NLG decomposes data-to-text generation into six tasks, moving from deciding what to communicate toward expressing it as well-formed sentences. The survey presents these tasks while noting that data-driven architectures increasingly question a fixed modular order.

  • Task decomposition: The six commonly identified NLG tasks are content determination, text structuring, sentence aggregation, lexicalisation, referring expression generation, and linguistic realisation.They respectively select information, order it, group it into sentences, choose wording, identify objects, and form grammatical sentences.
  • Late tasks: Lexicalisation, referring expression generation, and linguistic realisation convert planned content into words, object-identifying phrases, and well-formed sentences.The figure illustrates these later stages after content selection and ordering.
  • Early tasks: Content determination selects which information to express from richer or more detailed data, with choices shaped by audience and communicative intention.Sensor streams in neonatal care, for example, must be filtered and abstracted into preverbal messages.

2.2 Text Structuring

Text structuring determines how selected information is ordered, while aggregation combines messages to make outputs more concise and readable. Both areas have moved from domain-specific rules toward learned or globally optimized approaches, with aggregation remaining difficult to define precisely.

  • Text Structuring: Text structuring orders information for presentation using temporal sequence, importance, grouping, and domain-specific discourse preferences.Soccer reports may begin with general game information before describing goals, while neonatal events may be ordered temporally.
  • Text Structuring: Early text-structuring approaches relied on hand-crafted domain rules or Rhetorical Structure Theory, while later work used machine learning and general information-ordering methods.Some approaches jointly model document structuring and content selection.
  • Aggregation: Aggregation combines multiple messages into individual sentences, potentially making generated text more fluid and readable, though it may be inappropriate in some settings.The three neonatal events in Figure 1 are merged into a single pre-linguistic representation.
  • Aggregation: Semantic aggregation combines messages, whereas syntactic aggregation removes redundancy by combining linguistic structures such as parallel verb phrases.The soccer example compresses three scoring events into a shorter description.
  • Aggregation: Syntactic aggregation generally produces smaller reductions than message-level aggregation and presupposes that sentence planning, including lexicalisation, is complete.This makes the traditional placement of aggregation before syntactic realisation depend on which kind of aggregation is performed.

2.4 Lexicalisation

Lexicalisation converts planned content into natural language by selecting words and phrases under semantic, contextual, stylistic, and affective constraints. Its difficulty increases with the number of plausible alternatives and the need for appropriate variation.

  • Lexicalisation: Lexicalisation converts preverbal messages into lexical items, often alongside aggregation, before referring expressions and full syntactic realisation are completed.In the neonatal example, grouped events are mapped to a representation containing a verb and arguments.
  • Lexicalisation: The number of alternatives, contextual requirements, desired variation, and stylistic constraints all affect lexicalisation complexity.A system may randomly choose among alternatives for variation, but expressions must remain appropriate to the event and application.
  • Lexicalisation: Lexical choice may involve selecting among semantically similar, near-synonymous, or taxonomically related words rather than directly naming a domain concept.A straightforward direct mapping is feasible mainly in well-defined domains.
  • Lexicalisation: Lexicalisation concerns overlap with psycholinguistic research on lexical access, including how speakers select words and make errors in a densely connected mental lexicon.The survey connects computational lexical choice with semantic and phonological relationships among lexical items.

2.5 Referring Expression Generation

Referring Expression Generation (REG) selects words or phrases that identify domain entities by providing enough information to distinguish a target from alternatives. Research covers referential form, content selection, user-oriented descriptions, and increasingly complex visual and situated contexts.

  • Task definition: REG selects words or phrases that identify domain entities while providing enough information to distinguish the target from other entities.The task is a discrimination problem rather than merely a lexicalisation choice.
  • Referential form: Referential form determines whether an entity is expressed with a pronoun, proper name, or definite or indefinite description, partly according to its discourse salience.This choice is distinct from determining which properties a description should include.
  • Content determination: Content determination searches known properties for a combination that identifies the target without being insufficient or unnecessarily informative.The appropriate combination depends on the underlying theory and communicative context.
  • Algorithms: Full Brevity exhaustively searches possible descriptions for the smallest identifying property set, whereas Greedy Heuristic incrementally chooses properties that eliminate the most distractors.In the example domain, both approaches select size first.
  • Algorithms and extensions: The Incremental Algorithm selects properties incrementally according to domain-specific preference or cognitive salience, predicting color before size in the example.Other REG work extends classical algorithms to plurals, relations, and more complex visual or situated environments.
  • Communicative goals: Dialogue research shows that descriptions may include redundant properties for goals such as confirmation, while user-tailored systems adapt descriptions to audience characteristics.Examples include different flight descriptions for frequent flyers and occasional flyers.

2.6 Linguistic Realisation

Linguistic realisation converts selected content into well-formed sentences by ordering constituents, generating morphology, and inserting function words and punctuation. Approaches range from controlled templates and hand-written grammars to statistical and fully data-driven systems.

  • Task definition: Linguistic realisation orders sentence constituents and generates morphology, function words, and punctuation that may not be explicit in the input representation.The task turns a selected linguistic message into a well-formed sentence.
  • Approaches: Realisation approaches include human-crafted templates, human-crafted grammar-based systems, and statistical approaches.These categories span increasingly data-driven ways of controlling generation.
  • Templates: Templates are effective when domains are small and variation is minimal because they provide full output control and avoid ungrammatical structures.Their disadvantage is that constructing them is labour-intensive.
  • Templates: A template fills variables such as player, team, and minute to generate a sentence such as “Ivan Rakitic scored for Barcelona in the 4th minute.”The same content can be expressed through alternative surface orderings.
  • Grammar-based systems: Grammar-based realisers make choices using manually written or acquired language grammars, including systems such as FUF/SURGE, MUMBLE, KPML, NIGEL, RealPro, and OpenCCG.OpenCCG is described as a broad-coverage English surface realiser based on Combinatory Categorial Grammar.
  • Statistical systems: Statistical realisation uses corpus-derived probabilities or classifiers to guide search, filter outputs, order constituents, or acquire grammatical rules.Some systems retain a hand-crafted base generator, while fully data-driven approaches acquire rules from treebanks.

2.7 Discussion

Across core NLG tasks, the field has moved from hand-crafted, rule-based methods toward stochastic approaches relying on corpus data and increasingly domain-independent learning.

  • Discussion: NLG research shows a steady shift from hand-crafted rule-based approaches toward stochastic methods that rely on corpus data.This trend appears across individual tasks such as referring expression generation and linguistic realisation.
  • Discussion: As statistical methods spread across NLG tasks, learning techniques have received increasing emphasis and domain specificity has increasingly come from training data.The survey connects this methodological trend to changing architectures for generating text from data.

3 NLG Architectures and Approaches

NLG architectures can be classified as modular, planning-based, or integrated/global, while architectural design and symbolic-versus-stochastic method choice remain separate dimensions. The classical pipeline is influential but contested, and systems vary in how they organise tasks.

  • Architectural typology: The survey distinguishes modular architectures, planning perspectives, and integrated or global approaches to organising NLG tasks.Modular systems divide subtasks, planning views generation as an integrated AI planning problem, and integrated approaches learn input-output correspondences across task divisions.
  • Classification dimensions: Architectural type is orthogonal to the choice between symbolic knowledge-based methods and stochastic data-driven methods.A modular architecture can incorporate stochastic methods in several or all subtasks.
  • Historical development: The modular architecture is historically oldest, while integrated non-modular approaches gained prominence alongside data-driven techniques and shared data repositories.Earlier challenges to the pipeline also included non-stochastic blackboard and revision-based architectures.

3.1 Modular Approaches

Modular NLG architectures organize generation into distinct strategic and tactical stages, most commonly through a pipeline. Although this decomposition offers clarity and simplicity, mismatches between stages motivate interactive, blackboard, planning-based, and integrated alternatives.

  • Pipeline architecture: The consensus pipeline separates strategic decisions about what to say from tactical decisions about how to say it.The Text Planner combines content selection and text structuring, while sentence planning and realization perform tactical generation.
  • Related architectures: The pipeline resembles architectures in text summarization and psycholinguistic models of speech production, but lacks the latter’s degree of feedback.Summarization separates source analysis and information selection, transformation, and synthesis; Levelt’s model distinguishes what to say from how to say it and permits limited self-monitoring.
  • Architectural variation: NLG systems vary considerably in how they organize core tasks, despite the pipeline’s status as a widely cited de facto standard.The strategic–tactical division is influential but is not universally accepted in practice.
  • Raw-data inputs: Raw-data NLG extends the pipeline with preprocessing, including data abstraction and interpretation, before content selection and planning.These systems address unstructured numeric input rather than logical forms or database entries, using techniques ranging from signal processing to fuzzy reasoning.
  • Pipeline challenges: The generation gap arises when early strategic decisions have unforeseen consequences downstream, such as ambiguity after realization or difficulty enforcing length constraints.Interactive and blackboard architectures allow later modules to provide feedback, but these alternatives can reduce efficiency.

3.2 Planning-Based Approaches

Planning-based NLG treats generation as action selection toward communicative goals, allowing strategic and tactical decisions to be unified. Rule-based planners offer reusable infrastructure but face efficiency and reasoning costs, while reinforcement-learning approaches support joint optimization under uncertainty.

  • Planning-based approaches: Planning-based NLG views text generation as executing actions that change discourse, situated context, and user beliefs to achieve a communicative goal.Because plans can contain many action types, planning approaches can combine what to say and how to say it.
  • Unified planning: Unified planning can connect interlocutors’ beliefs and mutual knowledge to sub-goals spanning property choice, referring-expression generation, and realization.In KAMP, reasoning about beliefs produces sub-goals whose predicted effect is to alter the hearer’s belief state.
  • Planning through grammar: Grammar-based planning represents linguistic structures as operators with semantic preconditions and pragmatic goals, often using multi-level formalisms such as LTAG.Lexical items can require support in the knowledge base and introduce noun-phrase structures that must subsequently be filled.
  • Planning efficiency: Planners can be used off the shelf once NLG is formulated in PDDL, but experiments found significant preprocessing time even when solutions were later found efficiently.This efficiency problem appeared across NLG tasks of differing complexity.
  • Stochastic planning: Uncertainty makes fixed action consequences unrealistic because user behavior, information trade-offs, and stochastic realization can alter the outcome of an utterance.These uncertainties motivate stochastic optimization and reinforcement-learning formulations of generation.
  • Joint optimization: Joint reinforcement-learning frameworks sharing knowledge across subtasks outperform isolated task models, producing less verbose utterances and shorter interactions.A hierarchical joint policy also adapts instruction granularity when users go off-track.
  • Conclusions: Planning research highlights unified representations of multi-level constraints, hierarchical subproblems, and joint optimization across generation tasks.These lessons arise from both AI planning formalisms and stochastic reinforcement-learning approaches.

3.3 Other Stochastic Approaches to NLG

Other stochastic NLG approaches acquire data through aligned or crowdsourced input–text pairs, then model generation with classification, optimisation, language models, or joint decision processes.

  • Acquiring Data: Data-to-text corpora commonly pair database records with free text, while statistical methods increasingly segment and align the two automatically.Alignment models identify regular co-occurrences between data and text segments.
  • Acquiring Data: Crowdsourcing can elicit realisations for semantic or pragmatic dialogue-act inputs when the semantics are transparent to non-specialists.Uncertainty sampling can reduce the amount of training data required for generation.
  • Acquiring Data: Alignment-based stochastic systems can be highly domain-specific, whereas newer neural and machine-learning methods can operate without precise input–output alignments.Examples include neural systems and joint classification approaches trained on unaligned meaning-representation and text pairs.
  • NLG as a Sequential, Stochastic Process: Markov-based models combine strategic choices, such as records and fields, with tactical choices, such as word sequences, in a single probabilistic model.Standard language models may be limited by their local-history assumption; discriminative models offer an alternative.
  • NLG as Classification and Optimisation: Alternative systems represent generation as sequences of classification or optimisation decisions, including approaches that jointly model content selection, planning, and realisation.The reviewed literature includes log-linear sequence decisions, tree-based global generation, and joint optimisation or reinforcement-learning approaches.

3.4 Discussion

Recent NLG research increasingly blurs traditional task boundaries through integrated and jointly optimised models. The field still faces trade-offs involving efficiency, robustness, data requirements, and generalisation.

  • Integrated Approaches: Stochastic data-to-text systems combine content-oriented and linguistic choices through alignments, challenging traditional separations between NLG subtasks.Neural work has also reported superior outputs from jointly learning content selection and realisation rather than keeping them separate.
  • Integrated Approaches: A central open question is which NLG subtasks should be jointly optimised and which knowledge sources should be shared among them.This issue arises in both stochastic and neural approaches.
  • Open Challenges: NLG architectures must balance adequate textual quality against efficient and robust generation.Earlier revision-based, blackboard, and some planning-based approaches illustrate this tension.
  • Open Challenges: Data-driven NLG has yet to demonstrate scalability to domains with large heterogeneous data and longer texts.Existing datasets often focus on recommendations, brief weather reports, or sports summaries.
  • Deep Learning: Deep learning reduces the need for explicit alignment in some settings, while raising questions about representation learning, depth, and transfer across domains.The survey highlights the prospect of reusable domain-invariant linguistic representations.

4 The Vision-Language Interface: Image Captioning and Beyond

Image captioning exemplifies data-to-text generation from visual input, linking computer vision and NLG. Systems range from detector-based templates and grammars to holistic or dense neural representations, while generalisation remains difficult.

  • Motivation and Task: Image captioning generates sequential multiword descriptions from static, non-sequential images and has become a focus of both NLG and computer vision.The task also offers practical applications and a setting for studying the vision–language interface.
  • Task Decomposition: Captioning systems separate image analysis and text generation conceptually, although the two subtasks need not be organised sequentially.The survey distinguishes these subtasks before reviewing architectures.
  • Image Analysis: Visual inputs are handled through detection, holistic scene analysis, or dense feature vectors from pretrained convolutional neural networks.Detector-based systems identify objects, attributes, relations, actions, or poses; dense-feature systems commonly use activation layers.
  • Text Generation: Detector-based generators map visual detections to sentence structures using templates, trees, grammars, or sequence-classification models.Midge represents images with object/stuff, action/pose, and spatial-relation triples before tree-substitution realisation.
  • Text Generation: Human evaluation found Midge superior to two earlier systems on criteria including humanlikeness and correctness.Its probabilistic model can insert likely words that are not directly grounded in image detections.
  • Current and Future Directions: Generalising beyond training data and describing novel images remain challenges, motivating work on vocabulary expansion, novel concepts, transfer, and zero-shot learning.These directions address data sparsity and limited coverage of previously unattested visual content.
  • Current and Future Directions: Recent work extends visual generation beyond concrete descriptions toward explanatory descriptions and visual question answering.These tasks produce explanations or answers to specific image-related questions rather than only scene captions.

5 Variation: Generating Text with Style, Personality and Affect

NLG increasingly varies textual delivery beyond factual content, including style, personality, affect and politeness. Data-driven approaches broaden this control, but modelling and evaluating these multidimensional effects remains difficult.

  • 5.1 Generating with Style: Textual Variation and Personality: Stylistic NLG varies textual delivery according to features relevant to goals such as formality, involvement, personality or author-specific preferences.Researchers often operationalise style for the problem at hand rather than adopting a single rigorous definition.
  • 5.1 Generating with Style: Textual Variation and Personality: Early systems used rules to control stylistic features, while contemporary approaches increasingly learn dimensions of variation from corpora.Examples include lexical choice, contractions, rhetorical structure and syntactic structure.
  • 5.1 Generating with Style: Textual Variation and Personality: Human evaluations show that readers vary significantly in the personality they perceive, while corpus-based character models produce more specific and easily perceived traits than personality-trait models.These findings indicate that the relationship between psychological features and linguistic effects is not straightforward.
  • 5.2 Generating with Feeling: Affect and Politeness: Affective NLG targets transient emotional states, whereas politeness strategies regulate the degree of threat to the listener and can therefore function as affective generation.Tutorial-feedback systems have used Bayesian networks to select language compatible with target politeness or affect values.
  • 5.2 Generating with Feeling: Affect and Politeness: Effects of affective tactical decisions on readers’ emotional states depend on factors including how directly readers are implicated in the text.The paper also identifies measuring emotion as an open question.
  • 5.4 Style and Affect: Concluding Remarks: Controlling stylistic, affective and personality-based variation remains fledgling, with open questions about multidimensional modelling, machine-learning strategies and evaluation data.The survey notes debate over ratings compared with ranking-based and physiological methods.

6 Generating Creative and Entertaining Text

Creative NLG extends text generation toward jokes, metaphors, similes and narratives, drawing on lexical resources, computational creativity and narratology. The survey finds useful advances, but evaluation and long-form, linguistically polished generation remain difficult.

  • 6 Generating Creative and Entertaining Text: NLG has increasingly addressed creative language because conventional applications often prioritise coherent information delivery over attention, surprise and creative expression.The survey reviews automatic generation of jokes, metaphors, similes and narratives.
  • 6.1 Generating Puns and Jokes: Jape generates punning riddles with templates and lexical resources such as pronunciation dictionaries and WordNet.Its riddles exploit spelling or word-sense ambiguities within a defined joke genre.
  • 6.1 Generating Puns and Jokes: Petrovic and Matthews proposed an unsupervised joke generator that uses measurable principles for selecting noun–noun–attribute triples, without labelled examples or hard-coded rules.The principles include shared describability, common and ambiguous attributes, and dissimilar nouns.
  • 6.1 Generating Puns and Jokes: Computational joke generation has mostly targeted relatively simple puns, while more advanced jokes may require solving broader AI problems involving sophisticated hypothetical reasoning.The survey presents this as a current boundary of the research.
  • 6.3 Generating Narratives: Narrative generation draws on narratology, distinguishing story worlds from the texts that narrate them and modelling event structure, character and point of view.Data-driven systems can extract entities and interactions from story corpora to construct partially ordered event graphs.
  • 6.3 Generating Narratives: Most story-generation work remains restricted to very short stories, although projects such as World Clock demonstrate book-length generation from compact programs.World Clock describes 1440 moments and was fully generated by 165 lines of Python code.
  • 6.4 Generating Creative Language: Concluding Remarks: Creative NLG evaluation is a bottleneck because quality must be assessed across planning, realisation, genre compatibility and other levels of generation.Narrative quality can suffer when coherent story plans are rendered with excessively simple realisation.

7 Evaluation

NLG evaluation is difficult to compare directly because methods address different aspects of quality and their agreement varies across tasks, domains, and datasets. The survey therefore emphasizes combining evaluation methods and examining their relationships.

  • Evaluation challenges: NLG evaluations are difficult to compare directly because they use diverse methods and lack a single agreed-upon input format.Common benchmarks help only when systems receive sufficiently similar inputs.
  • Subjective evaluation: Human judgements use scales whose design affects the nuance and comparability of collected evaluations.Ordinal scales can compress differences at the bottom or top, while continuous, comparative, and magnitude-estimation approaches offer alternatives.
  • Corpus-based evaluation: Corpus-based metrics assess humanlikeness through automatically computed overlap, distance, or content comparisons, but semantic variability complicates direct matching.Near-synonyms and word-order variations can make generated texts differ from references while remaining comparable in meaning.
  • Genre and style: Genre and stylistic evaluation can connect recognisability with extrinsic effectiveness, although readers’ perceptions of subtle style differences may be inconsistent.In joke-generation studies, recognisability as a joke was related to perceived funniness, while personality perceptions in another system were inconsistent.
  • Metrics versus human judgements: Automatic metrics and human judgements often disagree, with their correspondence varying across tasks, datasets, domains, and quality dimensions.Examples include differing rankings in image captioning, domain-dependent metric results, and inconsistent correlations for fluency and adequacy.
  • Concluding remarks: The survey concludes that NLG evaluation should use multiple methods and report their correlations because each method may illuminate different aspects of system quality.Weak correlations do not necessarily invalidate a measure; they may indicate that methods target different properties of outputs or systems.

8 Discussion and Future Directions

The survey identifies a shift toward data-driven, integrated NLG and highlights evaluation, theory, and real-world deployment as continuing challenges and future directions. It also points to expanding connections with other disciplines and new application areas.

  • Evaluation remains a major concern, motivating extensive discussion of the problems and prospects of evaluating NLG applications.
  • The field has shifted from rule-based toward statistical, data-driven approaches, alongside movement from modular to more integrated architectures.
  • Data-driven approaches may improve efficiency but can compromise output quality, making the quality–efficiency–robustness trade-off central.
  • NLG increasingly interacts with computer vision, computational creativity, theoretical models, and cognitive research.
  • The survey’s scope centers on data-to-text generation, while text-to-text generation remains comparatively homogeneous and is covered elsewhere.
  • Future directions include social-media language, situated language generation, broader knowledge-base alignment, and further theoretical and engineering integration.
  • Many data-driven techniques have not yet been tested in commercial or real-world applications, leaving control, data availability, and scalability as practical concerns.

9 Conclusion

NLG has expanded rapidly as data, computing power, and machine learning have enabled new applications and commercial services. These developments include systems for journalism, weather, finance, and image-to-text generation.

  • NLG interest and applications have increased markedly, supported by more data, computing power, and advances in machine learning.
  • Companies now offer NLG technology for journalism, weather, and finance, alongside newer applications such as image-to-text generation.
Loading 1703.09902v4…