Source-linked AI summary
Challenges in Data-to-Document Generation
Sam Wiseman, Stuart M. Shieber, Alexander M. Rush
TL;DR
The paper studies data-to-document generation for longer, more difficult database-conditioned texts, where existing evidence from short descriptions is insufficient. It introduces a large corpus and extractive evaluations, then finds that neural systems are fluent but remain far from human-generated documents, although copying and reconstruction improve performance.
Problem
The paper asks how neural generation methods perform on a more difficult data-to-document task involving longer texts, more complex inputs, and challenges in fidelity, content selection, and structure.
Method
It introduces a large-scale corpus paired with descriptive documents and evaluates neural and templated generators using extractive metrics for content selection, factual generation, and ordering.
Results
Neural systems produce fluent text but remain far from human-level output; the best Conditional Copy system reaches 71% Relation Generation precision, versus 47% for the Copy model.
Takeaways & Limitations
Data-to-document generation exposes important neural-generation challenges that standard metrics can hide, while copying and reconstruction provide noticeable improvements.
Takeaways & Limitations
Generated systems make copying, semantic, and factual errors, including incorrect records and hallucinated statements unsupported by the input tables.
Abstract
from arXiv · showhide
Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human-generated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy- and reconstruction-based extensions lead to noticeable improvements.
1 Introduction
The paper frames multi-sentence data-to-document generation as a harder NLG task involving content selection, realization, and ordering. It introduces a dataset, extractive evaluations, and neural and templated baselines, finding that neural systems remain far from human-level output.
- Motivation: Multi-sentence summaries of database records require selecting what to say and determining how to say it.The task is intended to preserve database fidelity while generating fluent text.
- Contributions: The paper introduces a large-scale corpus of basketball-game descriptions paired with extensive statistical tables.The dataset is designed to support fully data-driven approaches.
- Contributions: It proposes extractive evaluation models for analyzing content selection, factual generation, and document structure.These evaluations exploit the relative ease of post-hoc information extraction compared with generation.
- Contributions: The study applies state-of-the-art neural methods and a templated system as baselines for the data-to-document task.The baselines are used to establish performance and study generated documents.
- Findings: Neural systems generate fluent text and score well on word-match metrics but perform poorly on content selection and long-term structure.Copy-based models and reconstruction terms improve BLEU and extractive evaluations, yet neural models remain significantly worse than templated systems on content selection and realization.
2 Data-to-Text Datasets
The paper defines data-to-text generation over structured records and introduces a more demanding dataset of basketball-game articles paired with box- and line-score tables. Compared with earlier benchmarks, the new data target longer texts, broader vocabulary, and more difficult content selection.
- Task formulation: A data-to-text example pairs records s with a human-generated document y1:T, while the system generates an adequate and fluent summary ˆy1:T.Each record contains an entity, value, and type, such as a player’s points statistic.
- Existing benchmarks: Earlier benchmarks include WEATHERGOV and ROBOCUP, where neural systems achieved BLEU scores in the 60s and 70s and almost 30, respectively.These results motivated examining more challenging dataset statistics.
- Dataset design: The introduced dataset is comparable to WEATHERGOV in token count but has significantly longer target texts, a larger vocabulary, and harder content selection.It is publicly available through the boxscore-data repository.
- Dataset sources: The corpus pairs NBA game articles from ROTOWIRE and SBNATION with corresponding box- and line-score tables.ROTOWIRE provides professionally written, medium-length summaries, whereas SBNATION is larger, more informal, and often tangential to the statistics.
- Dataset characteristics: A ROTOWIRE example contains 628 game records, while the gold document mentions only a selected subset in a potentially complicated form.Generation therefore requires selecting, expressing, and ordering relevant record content.
3 Evaluating Document Generation
The paper argues that standard word-match metrics inadequately assess document generation because they emphasize fluency over database fidelity and coherence. It therefore uses information extraction to evaluate what generated documents say and how they organize it.
- Evaluation motivation: BLEU is considered unsatisfactory for document generation because it primarily rewards fluency rather than important database information or coherent reporting.Human evaluation remains potentially necessary but is less convenient than automatic metrics.
- Extractive evaluation: The proposed approach uses high-precision information extraction because extracting information from documents is easier than generating documents that convey it accurately.The scheme is most appropriate for generations primarily intended to summarize information.
- Relation extraction: The extractor considers entity-value span pairs in each sentence and predicts their relation type, using ε for unrelated pairs.Training uses partial supervision from lexical matches between candidate records and database records.
- Extractor performance: Over 90% accuracy and approximately 60% recall are achieved for relation prediction on held-out ROTOWIRE data.These predictions support alignment between generated text and database records.
- Metric interpretation: Because the metrics are defined from extractor predictions, they identify which generated facts fail to match the database and remain interpretable.This distinguishes them from black-box adversarial evaluation approaches.
- Metrics: Content Selection measures overlap with gold relations, Relation Generation measures factual relations appearing in the database, and Content Ordering compares record sequences.CS targets what to say, CO targets how to say it, and RG targets both.
4 Neural Data-to-Document Models
The paper builds on an attention-based encoder-decoder and extends it with copying and reconstruction objectives to better model database-conditioned documents.
- Base Model: The base model embeds each record’s type, entity, and value, then uses an attention-based LSTM decoder to generate the document.Training minimizes the negative log-likelihood of the gold text conditioned on the source records.
- Copying: Copy-based models introduce a binary variable indicating whether each target word is copied from the source or generated.In this task, copied words are assumed to come from record values.
- Joint Copy Model: The Joint Copy model parameterizes the joint distribution over target words and copy decisions, allowing source records and generated words to compete directly.The model assigns zero probability to copying when the target word is not present in the source.
- Conditional Copy Model: The Conditional Copy model conditions on whether copying occurs, so candidate source records compete only with one another.With supervised copy labels, its objective acts as a word-level reconstruction loss that selects the record producing the target value.
- Reconstruction Losses: Reconstruction losses divide decoder hidden states into contiguous blocks and train them to predict database records, potentially predicting K records from each block.The loss independently models record components and can include a total-variation-distance penalty between predicted distributions.
5 Experimental Methods
The experiments apply several neural architectures and a problem-specific template generator to ROTOWIRE, using truncated training and extractive evaluation procedures.
- Neural Models: The ROTOWIRE models use 600-dimensional record representations and two-layer LSTM decoders with dot-product attention and input-feeding.Two identically structured attention layers separately support standard generation probabilities and copying.
- Training: Generation models are trained with SGD and truncated backpropagation through time using contiguous target blocks of length 100.Gradients are backpropagated through each current block and to the encoder parameters.
- Evaluation: The extractive evaluator ensembles three convolutional models and three bidirectional LSTM models trained with SGD.The convolutional models use kernel widths 2, 3, and 5 with 200 feature maps.
- Templatized Generator: A templatized generator first describes the teams, then emits six player-specific sentences and a typical end sentence.The player sentences instantiate a template containing player and scoring statistics.
- Templatized Generator: The template includes fixed team descriptions and game-specific placeholders for team records, player statistics, and upcoming games.Its final sentence describes each team’s next game using fixed opponent and venue wording.
- Implementation: Code for all models is publicly available, and the encoder-decoder implementations are based on OpenNMT.The implementation is provided through the paper’s linked repository.
6 Results
On ROTOWIRE, neural systems improve fluency and BLEU but remain weak on factual content selection and long-term ordering, with Conditional Copy and reconstruction providing targeted gains. Human evaluations broadly support the extractive metrics, while qualitative examples expose persistent logical and factual errors.
- Dataset results: SBNATION models performed poorly, so subsequent analysis focuses on ROTOWIRE.The best SBNATION model achieved validation perplexity 33.34 and BLEU 1.78.
- Automatic evaluation: The Template model scores poorly on BLEU but performs well on extractive metrics, whereas Conditional Copy with beam search achieves the best neural BLEU.Neural models have roughly the same perplexity despite differing BLEU performance.
- Extractive evaluation: 92% precision on gold documents contrasts with RG precision rising from 47% for Joint Copy to 71% for the best Conditional Copy system.The 71% Conditional Copy result remains below gold performance.
- Extractive evaluation: CS and CO show little relationship with BLEU; beam search can worsen CO, and both metrics remain worse than the templated model.The authors identify better copying for content selection and better long-term ordering models as open research needs.
- Test results: Test results confirm Conditional Copy as most effective for BLEU, RG, and CS, while reconstruction substantially helps the Joint Copy model.This pattern is consistent with development results.
- Human evaluation: Human raters found model differences in supported and contradicted facts broadly aligned with RG and CS, including significant contrasts involving gold generations and Copy+Rec+TVD.The first study evaluated 20 sentences per game across 20 games.
- Human evaluation: Gold summaries received higher naturalness ratings than generated summaries, although pairwise differences were not significant.The reconstruction term again improved the Joint Copy model’s ordering rating.
- Qualitative analysis: A Conditional Copy example combines colloquial fluency with factual and logical errors, including flipped records, self-contradictory rebounds, and unsupported home-crowd text.The extractive metrics can detect several of these errors through RG, CO, and CS failures.
7 Related Work
Related work spans longstanding research on natural-language generation and sports-game summarization, with emphasis on content selection and surface realization. Recent neural studies condition language models on tables and generate short biographies or record-based text, motivating more challenging tasks.
- Research foundations: Natural-language generation and sports-game summarization have both been studied for decades.The cited work includes early NLG research and longstanding sports-summary efforts.
- Research foundations: Earlier NLG research addressed content selection and surface realization using grammars, SMT-like approaches, PCFGs, and other formalisms.These strands correspond to deciding what to say and how to say it.
- Neural generation: Recent neural work conditions language models on tables and generates short biographies from Wikipedia tables.Neural encoder-decoder systems also achieved strong results on standard record-based generation datasets.
- Neural generation: Strong results on standard record-based datasets motivate the need for more challenging natural-language-generation problems.The cited neural work reports impressive results on those datasets.
8 Conclusion and Future Work
The paper introduces a dataset and metrics for evaluating neural data-to-document generation, finding that copying and reconstruction improve results but a substantial gap remains versus templated systems.
- The work introduces a dataset and metrics evaluating content selection, generation, and ordering in neural data-to-document generation.
- Copying and reconstruction improve performance on the task.
- A significant gap remains between neural models and templated systems.
- Future approaches may use more sophisticated source-record attention, semantic or reference constraints, and conditioning on less explicit facts.
A. Additional Dataset Details
The appendix describes the ROTOWIRE and SBNATION datasets, including their game-date ranges and random training, validation, and test splits, plus integer normalization for score values.
- ROTOWIRE covers NBA games from 1/1/2014 to 3/29/2017 and is split into 3398 training, 727 validation, and 728 test summaries.
- SBNATION covers NBA games from 11/3/2006 to 3/26/2017 and is split into 7633 training, 1635 validation, and 1635 test summaries.
- Box- and line-score numbers are converted to integers, with fractional percentages multiplied by 100 into [0, 100].
B. Generation Model Details
The generation models encode records, initialize decoder LSTMs from pooled entity representations, use dual attention, and add copy and reconstruction mechanisms during decoding and training.
- Each record is embedded with entity, type, value, and home-or-away features before a one-layer ReLU MLP maps it to R600.
- Decoder LSTMs are initialized from mean-pooled record representations grouped by entity.
- The decoder computes two attention distributions with different parameters at every decoding step.
- The Conditional Copy model predicts a copy probability from pooled records and the current topmost LSTM hidden state.
- Reconstruction loss processes decoder hidden-state blocks with convolutions and predicts relation entities, values, and types.
- Models use SGD with learning rate 1, batch size 16, and dropout rate 0.5.
C. Information Extraction Details
The information-extraction appendix constructs labeled entity-number relation examples from summaries and predicts relation types with convolutional and bidirectional LSTM ensembles.
- Entity and number spans in each sentence are matched to database records and labeled with the corresponding relation type or ǫ.
- Number-word conversion uses text2num, while misleading expressions such as “three-point” are ignored.
- The extraction system ensembles 3 convolutional and 3 bidirectional LSTM models using words, span distances, and 39 relation types plus ǫ.
- The convolutional model uses 600 filters across kernel widths 2, 3, and 5, followed by maxpooling and an MLP.
- Table 4 lists the possible record types used in the data.
- The bidirectional LSTM has one layer with 500 units in each direction, concatenated before max-pooling and classification.