PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-to-text Generation with Entity Modeling

Ratish Puduppully, Li Dong, Mirella Lapata

arXiv:1906.03221v1cs.CL

TL;DR

Neural data-to-text systems rely on learned representations to select, structure, and verbalize content, while treating entities as ordinary tokens. This paper dynamically models entities and generates text with hierarchical attention over input data and entity memories. On ROTOWIRE and a new, five-times-larger MLB dataset, the model outperforms competitive baselines and produces output humans find coherent, concise, and factually correct.

  • Problem

    Neural data-to-text models rely on representation learning for content selection, structure, and wording, without explicitly modeling entities in discourse context.

    Source: §1 · p. 1

  • Method

    The model creates dynamically updated entity-specific representations and generates text using hierarchical attention over the input and entity memories.

    Source: §1 · pp. 1–2, §8 · p. 9

  • Results

    On ROTOWIRE and the new MLB benchmark, the model outperforms competitive baselines, and humans find its output coherent, concise, and factually correct.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors report that explicit entity modeling yields more coherent, concise, and grammatical output across both datasets.

    Source: §1 · p. 2

    The MLB information-extraction system does not capture inning and team scores in play-by-play, and its held-out precision and recall are 83.4% and 66.7%, respectively.

    Source: §6 · p. 6

Abstract

from arXiv · show

Recent approaches to data-to-text generation have shown great promise thanks to the use of large-scale datasets and the application of neural network architectures which are trained end-to-end. These models rely on representation learning to select content appropriately, structure it coherently, and verbalize it grammatically, treating entities as nothing more than vocabulary tokens. In this work we propose an entity-centric neural architecture for data-to-text generation. Our model creates entity-specific representations which are dynamically updated. Text is generated conditioned on the data input and entity memory representations using hierarchical attention at each time step. We present experiments on the RotoWire benchmark and a (five times larger) new dataset on the baseball domain which we create. Our results show that the proposed model outperforms competitive baselines in automatic and human evaluation.

1 Introduction

Neural data-to-text systems rely on learned representations for content selection, organization, and wording, while treating entities as ordinary tokens. This work models entities explicitly, evaluates the approach on basketball and baseball data, and reports more coherent, concise, and grammatical output across both datasets.

1 Introduction

  • Neural data-to-text models use representation learning to select content, structure it, and verbalize it, without special-purpose mechanisms for these generation components.Data-to-text generation produces textual output from non-linguistic input, which can include tables, simulations, or spreadsheets.Source: §1 · p. 1
  • Descriptive summaries are entity coherent: they introduce and discuss a small set of important players and teams, making entities central to discourse coherence.The introduction connects this role of entities to linguistic and cognitive science work on how entities are realized and distributed in texts.Source: §1 · p. 1
  • The model dynamically updates entity-specific representations and uses a decoder with entity processors and hierarchical attention over entity memories and their corresponding records.The entity processors are gated recurrent neural networks with shared parameters.Source: §1 · pp. 1–2
  • The new MLB dataset is five times larger than ROTOWIRE and has summaries approximately 50% longer, with richer input records that include play-by-play.The experiments compare the entity model with neural architectures including conditional-copy and content-planning variants.Source: §1 · p. 2
  • Explicit entity modeling produces output that is more coherent, concise, and grammatical across both datasets.The authors also describe the model as resource lean, requiring no preprocessing such as extracting document plans.Source: §1 · p. 2

2 Related Work

Prior work models entities in language modeling, recipe generation, machine reading, and other tasks, but differs in whether representations are dynamic, fixed, or limited to one entity at a time. The authors position their approach as explicitly handling multiple entities in data-to-text generation and applying hierarchical attention over records and entity memories.

2 Related Work

  • Unlike approaches that condition on one entity at a time, this model dynamically represents and conditions on multiple entities in parallel.Ji et al. (2017) use dynamic entity representations for language modeling, while Clark et al. (2018) add entity context to the decoder.Source: §2 · p. 3
  • Recipe-generation approaches use fixed entity representations, whereas this work dynamically updates them during generation.Kiddon et al. (2016) use fixed representations for coverage and coherence; Bosselut et al. (2018) model entity actions and effects while also keeping representations fixed.Source: §2 · p. 3
  • The approach adapts entity representations used in machine reading by adding hierarchical attention over records and then entity memories.Henaff et al. (2017) score entity representations against a query or combine them as a weighted vocabulary prediction; hierarchical attention was introduced by Yang et al. (2016).Source: §2 · p. 3
  • Prior data-to-text work added content selection or planning to encoder-decoder systems, but the authors know of no earlier approach that explicitly handles entities in discourse context.Examples include content selection (Perez-Beltrachini and Lapata, 2018) and planning (Puduppully et al., 2019).Source: §2 · p. 3

3 Background: Encoder-Decoder with Conditional Copy

The encoder represents table records from their feature embeddings, and an LSTM decoder generates words conditioned on prior output and the encoded input. The model also uses conditional copying to copy input values into the generated text.

3 Background: Encoder-Decoder with Conditional Copy

  • Each input record is represented from its feature embeddings using a multilayer perceptron.Features can include values, entities, record types, and home-or-away team indicators; MLB records also include inning and play index.Source: §3 · p. 3
  • An LSTM decoder computes each target-word probability conditioned on previously generated words and the encoder output, with attention producing a dynamic context vector.For MLB, the encoder uses a bidirectional LSTM over records ordered by box score and then chronologically by play; the decoder is initialized from the encoder's final hidden states.Source: §3 · p. 3
  • Conditional copying lets the decoder copy values from the input, using a binary switch gate to indicate whether an output word is copied.Copied values include team names and score numbers in the game summary.Source: §3 · p. 4

4 Entity Memory and Hierarchical Attention

The model adds entity-specific memory to the decoder and updates it during generation; hierarchical attention selects entities and then their records.

4.1 Entity Memory

  • Entity memories start from the average record representation for each entity and are updated during decoding to accommodate discourse shifts in entity salience.Gates control whether and how much each memory changes, interpolating its previous value with a candidate memory.Source: §4.1 · pp. 4–5
  • The model omits a normalization term over entity memories because the authors found empirically that normalization hurt performance.Previous work used this normalization (Henaff et al., 2017; Ji et al., 2017).Source: §4.1 · p. 5

4.2 Hierarchical Attention

  • Hierarchical attention focuses first on entities and then on their records, using the resulting encoder context to condition text generation.The authors expect entity-first attention to constrain which records are selected and improve text precision.Source: §4.2 · p. 5
  • On development data, feeding the weighted entity-memory context directly as input degraded performance, so the authors did not pursue that variant.The alternative followed Clark et al. (2018).Source: §4.2 · p. 5

5 Training and Inference

Training maximizes the likelihood of summaries given input record tables, and inference uses beam search to approximate the best output.

5 Training and Inference

  • The training objective maximizes the log likelihood of output text conditioned on an input table of records.The training set consists of paired record tables and game summaries.Source: §5 · p. 5
  • At inference, beam search approximately finds the best output among candidate summaries.The candidate outputs are denoted y′ in the passage.Source: §5 · p. 5

6 Experimental Setup

The experiments use ROTOWIRE and a newly created MLB dataset, compare the entity model with several generation systems, and evaluate output along multiple content and language metrics.

Data

  • MLB is about five times larger than ROTOWIRE, with richer vocabulary and longer summaries; the datasets use 22,821/1,739/1,744 and 3,398/727/728 train/development/test instances, respectively.ROTOWIRE uses official splits; the authors created the MLB split.Source: §6 · p. 5
  • MLB summaries were tokenized, with quoted sentences and incidental “Game notes” sections removed.The removals targeted opinions or non-factual statements and incidental information, respectively.Source: §6 · p. 6

Information Extraction

  • The MLB IE system achieved 83.4% precision and 66.7% recall, but misses play-by-play inning and team-score attributes that are difficult to match deterministically to text.The authors note that building a highly accurate MLB IE module is outside this paper’s scope; low recall may disadvantage content-plan systems that rely on accurate plans.Source: §6 · p. 6

System Comparison

  • The comparison systems include template-based TEMPL, encoder-decoder ED+CC, and content-planning NCP+CC.For MLB, TEMPL describes teams, pitcher statistics, and play-by-play; NCP+CC uses pointer-network content plans followed by a BiLSTM and LSTM decoder.Source: §6 · p. 6

Evaluation

  • Evaluation reports relation-generation count and precision, content-selection precision and recall, content-ordering normalized Damerau-Levenshtein distance, and BLEU on both test sets.The IE approach extracts relations from human and model summaries to assess their alignment or divergence.Source: §6 · p. 6, Table 2 · p. 7

7 Results

ENT performs better than comparison systems on both RotoWire and MLB, while remaining conceptually simpler and more portable than NCP+CC because it does not rely on extracted content plans.

7 Results

  • On the RotoWire test set, ENT is comparable to NCP+CC overall but better on RG precision, CS precision, and CO; its CS recall is lower than NCP+CC's.ENT's CS precision is substantially higher than WS-2017 and NCP+CC, without a planning component; NCP+CC's planning component is trained for good coverage.Source: §7 · p. 7
  • Across both datasets, hierarchical attention improves relation generation, content selection, and BLEU, while dynamic memory and the update gate further improve content selection and BLEU.Source: Table 3 · p. 7, §7 · p. 8
  • On MLB, ENT has the highest BLEU among all models and highest CS recall and RG count among neural models, but lower RG precision than ED+CC.TEMPL leads on RG precision, RG count, and CS recall on both datasets, while its MLB BLEU is considerably worse than its RotoWire score.Source: §7 · p. 7
  • In a RotoWire example, ENT uses nominal coreference for Dwight Howard and James Harden and shifts team focus, while NCP+CC repeats Nuggets.Source: Table 4 · p. 8, §7 · p. 8
  • Human ratings place ENT near Gold on RotoWire coherence and highest on MLB grammaticality and coherence, while TEMPL leads MLB conciseness.On MLB, ENT's supporting-fact count is comparable to Gold and NCP+CC, and its contradictions are significantly lower than NCP+CC's; on RotoWire, its supporting and contradicting fact counts are comparable to NCP+CC.Source: §7 · pp. 8–9, Table 5 · p. 9
  • ENT performs better than comparison systems on both RotoWire and MLB, and is simpler and more portable than NCP+CC because it avoids extracted content plans.Source: §7 · p. 9

8 Conclusions

The model uses dynamically updated entity-specific representations and hierarchical attention; evaluation on two benchmarks finds it outperforms competitive baselines, while the authors identify directions for further work.

8 Conclusions

  • The model creates dynamically updated entity-specific representations and generates text through hierarchical attention over the input table and entity memory.Source: §8 · p. 9
  • On ROTOWIRE and the newly created MLB benchmark, automatic and human evaluations show the model outperforms competitive baselines and produces plausible, coherent, concise, factually correct output.Source: §8 · p. 9
  • Future work includes integrating content planning with entity modeling, emphasizing play-by-play, and exploiting dependencies across input tables.Source: §8 · p. 9

A Appendix

The appendix identifies a hyperparameter table for the ENT model on ROTOWIRE and MLB.

A Appendix

  • Table 6 lists hyperparameters used for the ENT model on ROTOWIRE and MLB.Source: A Appendix · p. 11

Results on the Development Set

On development sets, ENT performs comparably to NCP+CC on ROTOWIRE, with advantages on several metrics, and leads on selected MLB metrics; qualitative examples show nominal coreference.

Results on the Development Set

Loading 1906.03221v1…