Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

1 Introduction

• NLG spans text-to-text and data-to-text applications, including translation, summarization, correction, reporting, environmental information, clinical summaries, and audience-tailored generation.Data-to-text systems convert non-linguistic data into texts for journalism, sports, weather, finance, healthcare, museums, persuasion, and behavior change; tailoring can vary technical detail for doctors, nurses, or parents.Source: §1 · p. 5

1.1 What is Natural Language Generation?

• The survey uses NLG primarily for systems that generate text from non-linguistic data, while acknowledging that input types and application boundaries vary substantially.Inputs may include semantic representations, numerical data, structured knowledge bases, images, or video, and text-to-text, data-to-text, and dialogue-generation techniques can overlap.Source: §1.1 · pp. 6–7

1.2 Why a Survey on Natural Language Generation?

• An updated survey is timely because NLG has expanded since Reiter and Dale (2000) to tailored, text-to-text, vision-to-text, affective, creative, and systematically evaluated applications.Earlier overviews do not comprehensively cover these developments or evaluation methodology, while newer resources remain specialized or incomplete.Source: §1.2 · pp. 7–8

1.3 Goals of this Survey

• The survey synthesizes core tasks and architectures, highlights synergies with computer vision, stylistics, and computational creativity, and examines NLG evaluation methods and their relationships.Its main focus is data-to-text, while connections to text-to-text generation are discussed because their boundaries and data-driven frameworks increasingly overlap.Source: §1.3 · pp. 8–9

2 NLG Tasks

• The traditional NLG pipeline comprises content determination, text structuring, sentence aggregation, lexicalisation, referring expression generation, and linguistic realisation.These tasks respectively select information, order it, package it into sentences, choose expressions, identify domain objects, and form well-structured sentences.Source: §2 · p. 9
• The six tasks range from early data-oriented choices about what to say to late linguistic choices about which words to use and how to order them.This early–late distinction corresponds to strategy versus tactics and suggests a temporal order in modular pipeline systems, though data-driven architectures increasingly challenge that ordering.Source: §2 · pp. 9–10
• Early NLG tasks are crucial but often application-specific, whereas late tasks are more often studied independently and yield approaches shared across applications.The survey illustrates the six tasks with examples while highlighting recent developments in each.Source: §2 · p. 10

2.1 Content Determination

• Content determination first decides which information to include, because input data is often more detailed than the intended text and selection depends on audience and communicative intention.For example, a system may filter continuous physiological measurements or omit individual events from a soccer report.Source: §2.1 · pp. 10–11
• Content determination filters and abstracts data into preverbal messages represented through logical or database languages, attribute-value matrices, or graph structures.These representations can encode relations among domain entities, such as a player scoring a team’s first goal at a particular time.Source: §2.1 · p. 11
• Although content determination appears in most NLG systems, it is usually domain-specific; Guhe (2007) is a notable cognitively plausible, incremental exception focused on human language production.The account is based on studies of speakers describing dynamic events as they unfold.Source: §2.1 · p. 11
• Recent content-determination research uses data-driven methods, including Hidden Markov Models, clustering, preference scores, and weak supervision to model topics, selected events, dependencies, and data–text alignments.Examples include Barzilay and Lee (2004), Duboue and McKeown (2003), Barzilay and Lapata (2005), Liang et al. (2009), and Koncel-Kedziorski et al. (2014).Source: §2.1 · pp. 11–12

2.2 Text Structuring

• Text structuring produces an ordered discourse plan, arranging messages by temporal sequence, importance, relatedness, or discourse relations according to application-domain constraints.For example, soccer descriptions may begin with general game information before temporally ordered goals, while neonatal-care text may group related respiratory events.Source: §2.2 · p. 12
• Early text-structuring systems used hand-crafted, domain-dependent schemata, while Rhetorical Structure Theory supported domain-specific ordering rules for relations such as contrasts and elaborations.Williams and Reiter (2008) used rhetorical relations to identify orderings intended to maximize clarity for low-skilled readers.Source: §2.2 · p. 12
• Machine-learning approaches learn information orderings for text structuring and can also order summary-worthy sentences in multidocument summarisation.Some approaches jointly model document structuring and content selection, while general ordering methods seek an optimal arrangement of information-bearing items.Source: §2.2 · p. 12

2.3 Sentence Aggregation

• Sentence aggregation groups related messages into one sentence, potentially making generated text more fluid and readable, although some situations call for avoiding aggregation.The three selected bradycardia events can be merged into one pre-linguistic representation mapped to a single sentence.Source: §2.3 · p. 13
• Aggregation spans semantic grouping and syntactic combination, ranging from redundancy elimination to broader linguistic-structure combination.Reape and Mellish (1999) distinguish semantic aggregation from syntactic aggregation, such as combining repeated soccer-event descriptions.Source: §2.3 · p. 13
• Early aggregation relied on hand-crafted domain rules, whereas later systems learned aggregation decisions from corpora, including globally optimized entry linking with Integer Linear Programming.Barzilay and Lapata (2006) classify entry pairs by similarity, then select a globally optimal transitive set subject to document-level constraints.Source: §2.3 · pp. 13–14
• Syntactic aggregation can use more domain-independent redundancy-elimination rules, such as eliding repeated subjects and verbs, but typically yields smaller reductions than message-level aggregation.Its applicability also assumes sentence planning, including lexicalisation, is complete; therefore aggregation’s position before syntactic realisation depends on the aggregation type.Source: §2.3 · p. 14

• Lexicalisation maps finalized messages to lexical items, but a single event may have many natural-language expressions.For example, a soccer scoring event can be verbalized in several distinct ways.Source: §2.4 · p. 14
• The number of alternatives, contextual goals, stylistic constraints, affective stance, domain, and textual location determine whether lexical variation is desirable.Variation may suit soccer reports more than weather reports, while “to score a goal” is inappropriate for an own goal.Source: §2.4 · pp. 14–15
• Lexicalisation is difficult because choices involve near-synonyms, taxonomic relations, vagueness, gradable properties, and multiple expressions for numerical information.A timestamp such as 00:00 may be expressed as late evening, midnight, or evening, and human lexical choices vary considerably.Source: §2.4 · p. 15
• NLG increasingly treats lexicalisation as part of surface realisation, including unification-based approaches that combine conceptual representations with grammar rules encoding lexical and syntactic choices.This perspective differs from psycholinguistic lexical-access modelling, which studies how speakers retrieve words from a densely connected mental lexicon.Source: §2.4 · p. 15

• REG selects words or phrases that identify domain entities, chiefly by communicating enough information to distinguish one entity from others.Because it can be separated from particular application domains, REG has attracted substantial attention and supports standalone solutions.Source: §2.5 · p. 16
• Referential-form generation chooses among pronouns, proper names, and descriptions, often using discourse focus; shared-task systems commonly framed this choice as classification.Research has focused especially on pronouns, while proper-name generation remains comparatively understudied.Source: §2.5 · p. 16
• Content determination searches for properties that distinguish a target from distractors, balancing sufficient information against misleading or unnecessarily verbose descriptions.Full Brevity exhaustively selects the smallest identifying set; Greedy Heuristic incrementally chooses properties ruling out most distractors, while Incremental Algorithm follows domain-specific preference or cognitive salience.Source: §2.5 · pp. 16–17
• Dialogue research shows that referring expressions may include redundant properties for goals such as confirmation or user-tailored communication, beyond unambiguous identification.For example, descriptions may vary according to whether a listener is a frequent flyer or an occasional flyer.Source: §2.5 · pp. 17–18
• Recent REG work extends classical algorithms to plurals and relations, probabilistic and psycholinguistic modeling, joint content-form generation, and situated reference in realistic visual scenes.Applications now include virtual environments and photographs, linking REG with computer vision and broader vision-language research.Source: §2.5 · p. 18

2.6 Linguistic Realisation

• Linguistic realisation orders constituents, generates appropriate morphological forms, and inserts function words and punctuation to produce well-formed sentences.Because the output may contain linguistic components absent from the input, realisation projects between non-isomorphic structures.Source: §2.6 · p. 19

2.6.1 Templates

• Templates provide controlled, grammatical output for small domains with minimal variation but are labour-intensive and scale poorly to applications requiring substantial linguistic variation.Templates fill variables such as player, team, and scoring minute; modern variants may include syntactic information and complex filling rules.Source: §2.6.1 · p. 19

2.6.2 Hand-Coded Grammar-Based Systems

• Hand-coded grammar-based realisers support general-purpose, domain-independent generation but require detailed inputs and struggle to choose context-sensitive alternatives.Simpler engines expose syntax and morphology APIs while leaving choice-making to developers.Source: §2.6.2 · p. 20

2.6.3 Statistical Approaches

• Statistical realisation reduces manual labour and increases coverage by reranking grammar-generated alternatives or directly controlling generation decisions from corpus statistics.Fully data-driven systems also acquire grammatical rules from treebanks, use statistical language models for reranking, and support classifier cascades for syntax, linearisation, and morphology.Source: §2.6.3 · pp. 20–21
• As more decisions move into statistical generation, input representations become more abstract, enabling integrated end-to-end stochastic generation systems.This perspective can model the NLG process as a whole rather than linguistic realisation alone.Source: §2.6.3 · p. 21

2.7 Discussion

• NLG tasks have moved from hand-crafted rules toward stochastic methods that rely on corpus data and are more domain-independent.The domain-specific aspect increasingly comes from the training data rather than the approach itself.Source: §2.7 · p. 22

Architectural typology

• NLG architectures comprise three dominant approaches: modular designs, planning perspectives, and integrated or global approaches.Modular systems divide subtasks, planning offers a more integrated perspective, and integrated approaches cut across task divisions through learned input-output correspondences.Source: §3 · p. 22

Modular architectures

• Modular architectures maintain relatively crisp divisions among subtasks, although their specific designs vary substantially.The classical modular pipeline is the oldest design, while later blackboard and revision-based architectures challenged its status as the former consensus.Source: §3 · pp. 22–23

Planning perspectives

• Planning-based NLG links text generation to a longstanding AI planning tradition and treats subtasks in a less modular, more integrated way.Some planning-based systems use non-data-driven methods, demonstrating that non-modular organization does not require statistical modeling.Source: §3 · p. 22

Integrated or global approaches

• Integrated or global approaches, now dominant in NLG, cut across task divisions and typically rely heavily on statistical learning from non-linguistic inputs to outputs.Their broad adoption was supported by the uptake of data-driven techniques and the development of repositories for training and evaluation.Source: §3 · pp. 22–23

Classification dimensions

• Architectural organization and development methodology are separate classification dimensions, so modular systems can be stochastic and non-modular systems can be symbolic.The survey follows the architectural typology for exposition while repeatedly revisiting these qualifications.Source: §3 · pp. 22–23

Pipeline architecture

• The consensus pipeline comprises text planning for content selection and structuring, sentence planning for aggregation, lexicalisation and reference, and linguistic realisation for grammatical output.Text planning decides what to say, while sentence planning and realisation perform tactical generation by deciding how to say it and producing final sentences.Source: §3.1 · pp. 23–24

Architectural variation

• The pipeline’s strategic–tactical division is not universal: NLG systems vary considerably, may split tasks across modules, and can be described with Mellish et al.’s (2006) more flexible objects-and-arrows formalism.For referring expressions, content determination may occur during sentence planning while form decisions await realization-related choices.Source: §3.1 · p. 24

Raw-data input

• Pipeline systems receiving raw numeric data may require data abstraction and interpretation before text planning, so Reiter’s (2007) extension adds preprocessing stages before the Text Planner.Applications include weather reporting and summaries from patient data, using techniques ranging from signal processing extensions to fuzzy-set reasoning.Source: §3.1 · pp. 24–25

Pipeline limitations

• The main pipeline challenges are the generation gap, where early strategic or tactical choices have unforeseen downstream effects, and constraints that are difficult to express before linguistic realization.Length requirements illustrate the problem because pre-linguistic representations map unpredictably to numbers of words or characters.Source: §3.1 · p. 25

Alternative architectures

• Interactive, blackboard, and revision-based architectures introduce feedback or reactive coordination to address pipeline rigidity, but may reduce efficiency.Revision can also enrich drafts, as in Robin’s (1993) sports summaries, where historical information was added and positioned after the initial draft.Source: §3.1 · pp. 25–26

Recent influence

• Despite early criticisms, the strategic–tactical division continues to influence recent data-driven NLG approaches, while planning-based and integrated systems increasingly blur module boundaries.Dušek & Jurčíček (2015, 2016) are among the cited examples of data-driven approaches retaining this division.Source: §3.1 · p. 26

3.2 Planning-Based Approaches

• Planning-based NLG unifies strategic and tactical decisions by representing generation as actions whose effects update discourse, situated context, and user beliefs.This perspective connects NLG with language as action and allows plans to combine tasks normally separated in pipeline architectures, including content choice, referring-expression generation, and realisation.Source: §3.2 · p. 26

3.2.1 Planning through the Grammar

• Grammar-integrated planning treats linguistic structures as operators with semantic preconditions and pragmatic effects, thereby combining content determination, referring-expression generation, sentence planning, and realisation.In LTAG-based work, lexical items and elementary trees encode conditions and goals; adding predicates such as rabbit and white progressively excludes distractors until the referent is distinguished.Source: §3.2.1 · pp. 27–28
• Planning approaches can use off-the-shelf planners after formulation in languages such as PDDL, but planners may spend substantial time preprocessing even when solutions are found efficiently afterward.Koller and Petrick (2011) observed this efficiency pattern across NLG tasks of differing complexity.Source: §3.2.1 · p. 28

3.2.2 Stochastic Planning under Uncertainty using Reinforcement Learning

• Reinforcement learning models generation as stochastic optimisation in a Markov decision process, using rewards to learn policies that adapt to uncertain users and changing dialogue contexts.The approach optimises information presentation and referring-expression content, while joint optimisation shares knowledge across subtasks and supports hierarchical strategies.Source: §3.2.2 · pp. 29–30
• Joint reinforcement-learning policies outperform isolated task models, producing less verbose utterances and shorter interactions while adapting instruction levels when users go off-track.Dethlefs and Cuay´ahuitl (2015) use hierarchical agents spanning content selection through realisation; the joint policy gives high-level directions but switches to low-level instructions when needed.Source: §3.2.2 · p. 30

3.3.1 Acquiring Data

• Statistical NLG acquires training data through aligned data-text corpora, generative alignment models, linguistic multi-resolution alignment, crowdsourcing, and uncertainty sampling.Liang et al. (2009) model record, field, and string-generation likelihoods with latent correspondences learned using Expectation Maximization; newer neural and machine-learning systems can use unaligned pairs.Source: §3.3.1 · pp. 31–32
• Alignment-based end-to-end systems require precise input-output correspondences and are consequently highly domain-specific, whereas newer methods increasingly avoid explicit alignment.Duˇsek and Jurˇc´ıˇcek (2015) use an A* sentence planner with a rule-based realiser, while Lampouras and Vlachos (2016) jointly optimise content determination and realisation through imitation learning.Source: §3.3.1 · p. 32

3.3.2 NLG as a Sequential, Stochastic Process

• Sequential stochastic NLG combines content planning and realization through probabilistic models, with log-linear approaches supporting arbitrarily long histories and thus long-range coherence dependencies.Oh and Rudnicky (2002) use utterance-class and word-class n-gram models plus templates, while Angeli et al. (2010) condition staged decisions on previous decisions and select corpus-derived templates.Source: §3.3.2 · pp. 33–34

3.3.3 NLG as Classification and Optimisation

• Classification cascades model NLG as incremental choice-making, but their pipeline structure propagates early errors; optimisation instead seeks a globally consistent combination of decisions.The optimisation formulation assigns costs to labels and dependencies, then minimises total cost subject to constraints linking correlated labels.Source: §3.3.3 · p. 35, (14) Junge · p. 36
• Instance-based KStar classification outperformed a majority baseline on all eight route-description decisions, while revision-based generation achieved the best performance for syntax mapping and referring-expression insertion.Zarrieß and Kuhn (2013) found that task order affected performance, parallel execution yielded only marginal improvement, and revision improved over the alternatives.Source: §3.3.3 · p. 35, (14) Junge · pp. 35–36

(14) Junge

• Joint optimisation reduces error propagation: Integer Linear Programming produced more compact ontology verbalizations than a pipeline, while imitation learning achieved competitive results on sf and bagel.Joint policies in reinforcement learning also produced better dialogue interactions than isolated task policies, and Lampouras and Vlachos (2016) approached Wen et al. (2015) on sf and Duˇsek and Jurˇc´ıˇcek (2015) on bagel.Source: (14) Junge · pp. 36–37

3.3.4 NLG as ‘Parsing’

• Parsing-inspired NLG uses probabilistic CFG or synchronous CFG rules to map meaning representations to text; WASPER-GEN outperformed WASP on corpus metrics and approached human text in fluency and semantic correctness.Konstas and Lapata (2012, 2013) instead model unified input-to-text generation as hypergraph decoding with adapted CYK, language-model, and dependency-model components.Source: §3.3.4 · pp. 37–38

3.3.5 Deep Learning Methods

• Deep neural NLG benefits from dense distributed representations and recurrent models that handle variable-length sequences while reducing sparsity through shared low-dimensional history representations.Encoder-decoder architectures map variable-length inputs to outputs, and attention learns loose input-output correspondences without requiring direct alignment; these models are applied to data-to-text and dialogue generation.Source: §3.3.5 · p. 39, §3.3.6 · pp. 39–40

3.3.6 Encoder-Decoder Architectures

• On BAGEL, Duˇsek and Jurˇc´ıˇcek (2016) found jointly optimizing content selection and realization superior to generating deep syntax trees for subsequent realization.Their attention-based seq2seq model also reranks beam outputs, penalizing omitted relevant information and included irrelevant information.Source: §3.3.6 · p. 40
• Mei et al. (2016) use an attention-based aligner and refinement step to select and verbalize important weather records while modeling long-range dependencies with LSTMs.The model combines alignment likelihoods with record priors, improving the likelihood that important records are verbalized; comparable approaches also generate poetry.Source: §3.3.6 · p. 40

3.3.7 Conditioned Language Models

• Conditioned language models generate text from input features: Lebret et al. (2016) jointly model biography content selection and realization, while Lipton et al. (2016) and Tang et al. (2016) condition product-review generation on semantic, sentiment, contextual, or continuous information.Related models address stylistic and affective generation (Li et al., 2016; Asghar et al., 2017; Hu et al., 2017; Ficler & Goldberg, 2017).Source: §3.3.7 · pp. 40–41

3.4 Discussion

• NLG systems increasingly combine content selection and linguistic realization, raising questions about jointly optimizing subtasks and sharing knowledge sources.Stochastic data-to-text systems unify content-oriented and linguistic choices, while neural joint learning has been claimed to outperform separate models (Dušek & Jurčíček, 2016).Source: §3.4 · p. 41
• Unified and non-pipeline architectures can improve coherence but often sacrifice computational efficiency and robustness.Revision-based, blackboard, and planning-based approaches illustrate this trade-off; planning-based systems also incur considerable computational cost.Source: §3.4 · p. 41
• Data-driven NLG still requires suitable data to scale from narrow domains to heterogeneous inputs and longer texts.Crowdsourcing may help address the scarcity of such data.Source: §3.4 · p. 41
• Deep learning makes aligned training data less essential, because attention-based models can learn from looser input-output couplings.These techniques may become more prominent across NLG tasks and end-to-end systems.Source: §3.4 · p. 41
• Deep architectures may learn reusable, task-specific and domain-invariant representations, but reliable transfer across tasks remains elusive.This possibility is motivated by representation learning and seq2seq transfer-learning work, while domain-invariant linguistic representations remain an open frontier.Source: §3.4 · p. 42
• The paper next examines image-to-text generation, stylistic variation, and computational creativity as intersections between NLG, AI, and NLP.These topics shift attention away from standard tasks and their organization.Source: §3.4 · p. 42

4 The Vision-Language Interface: Image Captioning and Beyond

• Image captioning connects vision and language by generating multiword text from static images, making it a major shared focus of NLG and computer vision.The task also reflects longstanding scientific interest in grounding language in perceptual data.Source: §4 · pp. 42–43

4.1 Data

• Captioning datasets pair images, mostly photographs, with one or more human-authored captions and include general-purpose, naturally occurring, and domain-specific collections.Widely used resources include Flickr8k, Flickr30k, MS-COCO, SBU1M, and CUB.Source: §4.1 · p. 44

4.2 The Core Tasks

• Captioning systems comprise logically distinct image-analysis and text-generation subtasks, although architectures need not organize them separately or sequentially.Image analysis may identify scene content, while generation or retrieval converts visual information into captions.Source: §4.2 · p. 44

4.2.1 Image Analysis

• Visual processing approaches use object and relation detection, holistic scene descriptors, or dense features extracted from pretrained convolutional neural networks.Detection-based systems often map visual outputs to linguistic structures, whereas holistic methods frequently support image or multimodal retrieval.Source: §4.2.1 · pp. 44–45

4.2.2 Text Generation or Retrieval

• Midge outperformed the systems of Kulkarni et al. (2011) and Yang et al. (2011) on several human-evaluation criteria, including humanlikeness and correctness.Captioning approaches include templates and trees, language models, whole-caption retrieval, and fragment recombination; retrieval improves fluency but can produce generic captions for dissimilar images.Source: §4.2.2 · pp. 45–47

4.3 How is Language Grounded in Visual Data?

• Language grounding ranges from detector-mediated mappings between visual concepts and linguistic structures to direct similarity in unimodal or multimodal spaces.Neural architectures combine CNN image features with recurrent language models at different stages, including initial inputs, every time step, or a later multimodal layer.Source: §4.3 · pp. 47–48

4.4 Vision and Language: Current and Future Directions for NLG

• Current directions address generalization to novel images, localization and attention, richer explanatory or question-answering outputs, and video captioning with temporal dependencies and redundancy.Generalizing beyond training data remains difficult, while newer datasets couple images with both concrete conceptual and narrative texts.Source: §4.4 · p. 49

5.1 Generating with Style: Textual Variation and Personality

• Stylistic NLG treats variation in lexical, grammatical, and semantic features as a way to convey the same information through linguistically distinct choices.Approaches range from rule-based sentence planning and realisation to corpus-driven models that predict global stylistic dimensions from local generation decisions.Source: §5.1 · pp. 50–51
• Corpus-based character models produced more specific and easily perceived traits than personality-trait models, while film-derived models matched the styles of their source characters.Personage conditions generation on Big Five personality scores, but human judgements reveal that the link between personality and linguistic form is indirect.Source: §5.1 · pp. 52–53

5.2 Generating with Feeling: Affect and Politeness

• Affective NLG varies language to induce or reflect transient emotional states, but emotional impact depends on contextual factors such as readers’ personal involvement.Politeness can be treated as affective NLG because alternative face strategies impose different degrees of threat, with perception also varying by speech act and culture.Source: §5.2 · pp. 53–55

5.3 Stylistic Control as a Challenge for Neural nlg

• Neural NLG controls style and affect by conditioning response generators or language models on speaker profiles, personality traits, emotional categories, sentiment, content, and other attributes.Human judges identified high-trait responses as more expressive in Herzig et al. (2017), although that personality-based model did not outperform Li et al.’s model.Source: §5.3 · pp. 55–56

5.4 Style and Affect: Concluding Remarks

• Controlling style, personality, and affect remains fledgling because models must represent complex variation, identify effective linguistic choices, and address uncertain evaluation methods.Open questions include linear versus global models, human ratings versus ranking or physiological measures, and adaptive generation for interactive systems.Source: §5.4 · pp. 56–57

6 Generating Creative and Entertaining Text

• NLG output can be boring and repetitive because it often lacks narrative attention-maintaining techniques, surprise, and creative language such as jokes or metaphors.The section argues that NLG researchers and computational creativity researchers could benefit from greater interaction.Source: §6 · p. 57

6.1 Generating Puns and Jokes

• JAPE generates punning riddles with templates, pronunciation and semantic resources, and novelty checks, while later systems select humorous triples using ambiguity, shared attributes, and dissimilarity.Computational joke generation has mainly targeted relatively simple puns; more sophisticated jokes may require advances in broader AI reasoning.Source: §6.1 · pp. 58–59

6.2 Generating Metaphors and Similes

• Metaphor and simile generators use web and n-gram resources to retrieve candidate comparisons, filter inappropriate mappings, rank expressions, and add affective polarity.These techniques support applications ranging from clarifying complex information to generating affective similes and poetry.Source: §6.2 · pp. 59–60

6.3 Generating Narratives

• Narrative generation commonly plans fictional story worlds using schemas, reasoning, and causally linked events, while linguistic realization, temporal ordering, and longer narratives remain challenging.McIntyre and Lapata (2009) coupled story generation with an interest model predicting which stories readers would rate as interesting; World Clock described 1440 events occurring around the world, one randomly selected minute at a time.Source: §6.3 · pp. 61–63

6.4 Generating Creative Language: Concluding Remarks

• Creative NLG remains understudied, and evaluation is a bottleneck because quality must often be assessed across planning, realization, genre compatibility, and human recognition.Computational creativity offers resources for improving readability and variability, while narratology informs text planning, event ordering, character, and focalisation.Source: §6.4 · p. 64

7 Evaluation

• Shared tasks have made NLG evaluation more systematic across referring expressions, surface realisation, instructions, content determination, question generation, narratives, structured web data, and image captioning.They also supported large-scale datasets and evaluation servers such as MS-COCO.Source: §7 · pp. 64–65
• NLG evaluation is difficult to compare because systems vary in input assumptions and because multiple outputs may be appropriate for the same input.Dataset biases can favor particular content, while open-ended output variation makes independent human assessment important for determining which outputs are preferable.Source: §7 · pp. 65–66

7.1.1 Subjective (Human) Judgements

• Subjective evaluation commonly asks human judges to rate fluency and content accuracy, but scale design, inter-rater variation, platform reliability, and ethics constrain the results.Preference-based comparisons were more sensitive to system differences and less affected by subject variance than standard rating scales, while judge training and discussion can reduce disagreement at higher cost.Source: §7.1.1 · pp. 67–68

7.1.2 Objective Humanlikeness Measures Using Corpora

• Corpus-based intrinsic metrics cheaply measure humanlikeness, but surface overlap is only a proxy for content fidelity unless semantic or content-level representations are available.Metrics increasingly use semantic resources, embeddings, and direct content overlap; in image-captioning comparisons, WMD outperformed other metrics.Source: Table 1 · p. 69, §7.1.2 · pp. 70–71

7.1.3 Evaluating Genre Compatibility and Stylistic Effectiveness

• Genre and style evaluations test recognisability and effectiveness, but stylistic perceptions can be inconsistent and may not establish whether variation achieves an external goal.JAPE’s generated riddles were recognised as jokes and judged funnier than non-jokes, whereas human jokes were generally funnier; Petrovic and Matthews’ best model was rated funny in 16% of cases versus 33% for human jokes.Source: §7.1.3 · pp. 71–72

7.2 Extrinsic Evaluation Methods

• Extrinsic evaluation measures whether generated text achieves an application goal, but realistic studies require suitable users, settings, and controls for confounding variables.Examples assess persuasion, purchasing, ecological engagement, medical decision support, communication, learning, navigation, referent identification, and literacy self-assessment; realism and control can conflict, as in BabyTalk evaluations.Source: §7.2 · pp. 72–73

7.3 Black Box Versus Glass Box Evaluation

• Black-box evaluation assesses complete systems, whereas glass-box evaluation isolates component contributions, although exhaustive ablations can create too many configurations and reduce statistical power.Alternative designs compare human, machine, and hybrid outputs, combine corpus and subjective measures, or analyze user comments to identify problematic components.Source: §7.3 · p. 74

7.4 On the Relationship Between Evaluation Methods

• Evaluation methods seldom agree on system rankings, with corpus-based, judgement-based, and task-based measures frequently producing different verdicts.In image captioning, human readers preferred Kulkarni et al.’s system despite lower BLEU scores, while human and BLEU/ROUGE rankings agreed only when retrieved captions matched references.Source: §7.4 · p. 74, §7.4.1 · p. 75

7.4.1 Metrics Versus Human Judgements

• Automatic metrics correlate inconsistently with human judgements across NLG tasks and domains, often predicting linguistic quality better than semantic correctness.Reported correlations range from 0.3 to 0.87, while findings vary with metric design, text length, n-gram overlap, semantic variability, and reference availability.Source: §7.4.1 · pp. 75–76

7.4.2 Using Controlled Experiments

• NLG evaluations therefore increasingly combine multiple methods because metrics, human judgements, and behavioural measures capture different aspects of output quality.Controlled experiments found sentence-recall results largely consistent with judgements, generally poor correlations between reading times and automatic metrics, and reading times more useful than offline judgements for distinguishing coherent texts.Source: §7.4.1 · p. 77, §7.4.2 · p. 77

7.5 Evaluation: Concluding Remarks

• Meta-evaluation finds that correlations among human, behavioural, and automatic measures vary by task and application domain, leaving their relationships an ongoing research topic.The dimensions being judged, and the distinction between producer-oriented output measures and receiver-oriented processing measures, remain under-explored.Source: §7.5 · pp. 77–78
• NLG practitioners should favor diverse evaluation methods and report both their results and correlations, since weak correlations may reflect differing evaluation targets rather than invalid measures.Multiple methods can illuminate fluency, clarity, adequacy, semantic content, and communicative effectiveness, but their results are not always consistent.Source: §7.5 · pp. 77–78

8 Discussion and Future Directions

• The survey’s central theme is NLG’s shift from rule-based, domain-dependent and modular approaches toward statistical, data-driven, more general and integrated methods, with quality–efficiency–robustness trade-offs.Data-driven methods may improve efficiency but can compromise output quality, while NLG increasingly interacts with computer vision and computational creativity.Source: §8 · pp. 78–79

8.1 Why (and How) Should NLG be Used?

• NLG should be used when information is voluminous and difficult to present directly, while system design should jointly consider text, document layout, graphics, and decision-critical information delivery.The survey emphasizes choosing how NLG interacts with visual modalities, particularly in domains where accurate information supports fault-critical decisions.Source: §8.1 · p. 79

8.2 Does NLG Include Text-to-Text?

• Text-to-text generation can converge with data-to-text generation by treating text as one input type among heterogeneous data, enabling coherent generation from combinations of text, numbers, and images.Although text-to-text tasks often use more homogeneous inputs and established datasets, heterogeneous-input generation is identified as a promising direction.Source: §8.2 · pp. 79–80

8.3 Theories and Models in Search of Applications?

• NLG should more explicitly connect theoretical and cognitive models with implementation: theory can improve systems, implementations can expose underspecified model parameters, and theory-driven systems contribute to theoretical work.The survey argues that engineering and theoretical perspectives are not inherently opposed, despite emphasizing different goals.Source: §8.3 · pp. 80–81

8.4 Where do We Go from Here?

• Future NLG research should expand into social media, situated environments, structured knowledge bases, and commercial applications, while addressing interpretability, data acquisition, alignment, and scalability.Situated generation is especially underrepresented in gaming, and many data-driven systems remain insufficiently tested in commercial or real-world settings.Source: §8.4 · pp. 81–83

9 Conclusion

• NLG technology now supports applications in journalism, weather, finance, image-to-text generation, automatic content creation, Twitter, and chatbots.The authors connect this expansion to increased data and computing power and rapid machine-learning developments.Source: §9 · p. 83
• The field’s steady development and growing industrial use suggest a promising future, while motivating further multidisciplinary collaboration.The survey is intended to highlight potential avenues for such collaboration.Source: §9 · p. 83