PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-to-Text Generation with Content Selection and Planning

Ratish Puduppully, Li Dong, Mirella Lapata

arXiv:1809.00582v2cs.CL

TL;DR

Neural data-to-text systems often omit explicit decisions about which facts to mention and how to order them, despite difficulties with structure, redundancy, and factuality. This paper adds learned content selection and planning to an end-to-end architecture, then generates text from the resulting plan; evaluations on RotoWire show improved generation quality over competitive baselines, while the content plan is produced by a non-expert IE system rather than an expert.

  • Problem

    Neural data-to-text systems struggle with content selection, long-term structure, fact ordering, redundancy, and faithfulness because they do not explicitly model these stages.

  • Method

    The model uses two stages: content selection and planning produce an ordered plan of records to verbalize, and text generation produces the document conditioned on that plan.

  • Results

    Content planning improves generation across metrics and systems, with NCP+CC performing best overall and improving relation generation by approximately 12% over Wiseman et al. (2017).

  • Takeaways & Limitations

    Content planning improves the relevance and ordering of generated facts, with additional gains in grammaticality and conciseness.

  • Takeaways & Limitations

    The content plan is not expert gold because it is produced by a fairly accurate information-extraction system.

Abstract

from arXiv · show

Recent advances in data-to-text generation have led to the use of large-scale datasets and neural network models which are trained end-to-end, without explicitly modeling what to say and in what order. In this work, we present a neural network architecture which incorporates content selection and planning without sacrificing end-to-end training. We decompose the generation task into two stages. Given a corpus of data records (paired with descriptive documents), we first generate a content plan highlighting which information should be mentioned and in which order and then generate the document while taking the content plan into account. Automatic and human-based evaluation experiments show that our model outperforms strong baselines improving the state-of-the-art on the recently released RotoWire dataset.

1 Introduction

Data-to-text systems must select and order information while producing fluent, faithful documents. This work addresses neural systems’ structural and factual shortcomings by explicitly learning content plans and conditioning generation on them, improving evaluation results.

  • Motivation: Data-to-text generation produces text automatically from non-linguistic inputs such as databases, spreadsheets, and simulations.The paper illustrates the task with NBA game records paired with a game summary.
  • Motivation: Traditional systems separate content planning, sentence planning, and surface realization, whereas recent neural systems train end-to-end without explicitly modeling these stages.Content planning selects input content and determines output structure; sentence planning determines sentence structure and lexical content.
  • Motivation: Neural systems struggle with long-term structure, content selection, inter-sentential coherence, fact ordering, redundancy, and faithfulness to database inputs.Comparisons with template-based methods also report weaker content-selection recall and factual output generation, including unsupported hallucinated statements.
  • Approach: The proposed architecture learns a content plan from input records and conditions document generation on that plan while retaining end-to-end neural training.The plan specifies which information to mention and in what order before text generation.
  • Approach: Explicit planning organizes document structure, makes generation more interpretable through an intermediate representation, and reduces redundancy by limiting repeated information in the plan.The decoder can focus on sentence planning and surface realization after receiving the high-level organization.

2 Related Work

Earlier data-to-text research used hand-built or learned content planners, while more recent end-to-end systems often omit planning or make simplifying assumptions. The RotoWire dataset provides a document-scale benchmark for testing these capabilities.

  • Content selection: Content-selection components have been hand-built or learned from data across multiple generation domains, including sports-game summarization.Sports summarization has been studied since the early generation systems.
  • Content planning: Earlier content-planning research used generic planners, rhetorical structures, schemas, ordering constraints, and learned planners from aligned input-output corpora.These approaches often relied on analyzing target texts and devising hand-crafted rules.
  • End-to-end systems: Recent end-to-end models commonly generate without content selection or planning, or perform content selection without planning.Konstas and Lapata (2013) are identified as an exception using document-level grammar-rule content plans.
  • Neural generation: Neural generators have covered Wikipedia infoboxes, weather forecasts, soccer commentaries, and NBA game summaries, often targeting single sentences or short texts.The paper positions its work closest to neural models learned from data and accompanying text resources.
  • Benchmark: RotoWire pairs NBA game statistics with human-written summaries and supports document-scale evaluation, with average summaries of 330 words and 628 input records.The dataset was designed to be sufficiently large for neural training and challenging for document-scale generation.

3 Problem Formulation

The model transforms an unordered table of records into a document through explicit content selection, planning, and text generation stages. Record dependencies, ordered content plans, attention, and copying are integrated in an end-to-end neural architecture.

  • 3 Problem Formulation: The input is an unordered table of records, each containing type, entity, value, and home/away features, while the output is a variable-length document.Each record is embedded and transformed into a vector representation before later processing.
  • 3 Problem Formulation: The architecture decomposes generation into content selection and planning, which chooses and orders records, followed by text generation conditioned on that plan.The generator attends over vector representations of records selected for the content plan.
  • Content Selection Gate: The content selection gate uses contextual attention to control how much information from each record representation flows forward.The gate addresses dependencies among related records, such as a player's points, field goals, three-pointers, and rebounds.
  • Content Planning: The model learns explicit plans for long, canonically structured game summaries, including winning teams, player and team statistics, and upcoming games.Plans are extracted by aligning summary text with entities, values, and record types in the input table.
  • Content Planning: Content plans are sequences of pointers to input records, with their order matching the order in which entities appear in the summary.An LSTM decoder with attention generates record positions, using gated record vectors as inputs.
  • Text Generation: The text generator encodes the content plan with a bidirectional LSTM and decodes words with an attention-based recurrent network.The encoder directly consumes the corresponding gated record vectors, and the decoder receives the previous predicted word.
  • Text Generation: A copy mechanism lets the decoder copy words directly from record values in the content plan.The model experiments with joint and conditional copy variants, including a switch indicating whether each token is copied.
  • Training and Inference: Training maximizes the joint log-likelihood of gold content plans and gold texts, while inference uses beam search to approximate the best plan and output.The two modeled terms are p(z|r) for planning and p(y|r, z) for text generation.

4 Experimental Setup

The experiments use ROTOWIRE basketball summaries and extracted content plans, with an IE system supplying relation annotations and the model trained under fixed configurations.

  • Data: ROTOWIRE pairs professionally written basketball game summaries with box- and line-score tables and averages 337 words per summary.The dataset contains 39 record types, 628 records on average, an 11.3K-word vocabulary, and 1.6M tokens.
  • Data: The experiments use 3,398 training summaries, 728 test summaries, and 727 validation summaries following prior data partitions.
  • Content Plan Extraction: Content plans are extracted by identifying entity-value pairs in summaries and assigning matching database record types or the label “none.”The IE system considers entity-number pairs within each sentence and matches them against records in the database.
  • Content Plan Extraction: The relation-prediction system ensembles three convolutional and three bidirectional LSTM models trained on the ROTOWIRE training portion.
  • Content Plan Extraction: 94% held-out relation-classification accuracy and approximately 80% recall were achieved for relations licensed by the records.The resulting plans consist of entity, value, record type, and home/away tuples ordered by appearance in the summary.
  • Training Configuration: The generation model used one-layer pointer networks for planning, two-layer LSTMs for generation, 600-dimensional embeddings and hidden layers, and 0.3 dropout.Hyperparameters were validated on the development set, and models were trained for 25 epochs with Adagrad.

5 Results

Automatic and human evaluations show that explicit content selection and planning improve generated game summaries across quality dimensions and competitive baselines. NCP+CC performs best overall among automatic systems, although automatic summaries remain significantly behind human-authored summaries.

  • Automatic evaluation: 84.5% of NCP+CC’s extracted records are non-duplicates versus 72.9% for Wiseman et al. (2017), indicating less repetitive summaries.
  • Ablation and planning: Content selection and planning each improve performance over ED+CC, with further gains when both components are combined.Generated summaries follow the planner closely, with CS precision above 85%, CS recall above 93%, and CO above 84%.
  • Test-set evaluation: On the ROTOWIRE test set, NCP exceeds Wiseman et al. (2017) in relation generation, content selection, content ordering, and BLEU.The test-set results follow a pattern similar to the development set.
  • Human evaluation: NCP+CC has more supporting facts than WS-2017 and Gold, while human judgments find it more coherent than TEMPL and WS-2017 and more grammatical than WS-2017.Gold summaries remain significantly better than automatic systems across grammaticality, coherence, and conciseness; NCP+CC is significantly worse than TEMPL on conciseness.
  • Overall result: Across metrics and systems, the authors conclude that content planning improves data-to-text generation, with NCP+CC performing best overall but remaining below human-authored summaries.

6 Conclusions

The model enhances data-to-text generation with content selection and planning, improving relevant-fact coverage and ordering while also benefiting grammaticality and conciseness.

  • The model adds content selection and planning modules to neural data-to-text generation.
  • Automatic metrics and judgment studies show improvements in the number and ordering of relevant facts.
  • Content planning also improves the grammaticality and conciseness of generated text.
  • Future work targets detail-oriented plans involving multiple facts and entities, and evaluation across domains and languages.

Comparison with the Results in Wiseman et al.’s (2017) Webpage

The comparison uses corrected versions of the dataset and automatic-metric code, with scores recomputed to remain comparable to Wiseman et al.’s webpage results.

  • The experiments use the corrected ROTOWIRE dataset identified by Wiseman et al. (2017).
  • The paper reports scores computed with corrected automatic-metric code after discovering a bug in the original implementation.
  • The authors recompute their scores to make them comparable with numbers published on Wiseman et al.’s webpage.

Qualitative Examples

The qualitative examples compare template, WS-2017, and neural content-planning outputs, while the accompanying evaluations cover factuality, content selection, ordering, and BLEU.

  • Qualitative Examples: The template outputs are gold standard for relation-generation accuracy, whereas WS-2017 examples contain contradictions and duplicate text.
  • Qualitative Examples: NCP+CC produces more factual text with fewer contradictions and less duplicate information than the WS-2017 examples.
  • Automatic Evaluation: Tables 8 and 9 evaluate systems using relation-generation, content-selection, content-ordering, and BLEU metrics.
  • Qualitative Examples: Table 10 compares template-based, WS-2017, and NCP+CC documents using markings for accurate, erroneous, and duplicated text.
Loading 1809.00582v2…