Source-linked AI summary
Neural Text Generation from Structured Data with Application to the Biography Domain
Remi Lebret, David Grangier, Michael Auli
TL;DR
Concept-to-text generation must scale from small, limited datasets to rich domains with many records and words. This paper introduces a neural table-conditioned model with global and local conditioning plus copy actions for table-specific vocabulary, and it evaluates the approach on 728,321 Wikipedia biographies. The model outperforms an adapted Kneser-Ney baseline by nearly 15 BLEU, while remaining focused on first-sentence generation and leaving factual-accuracy evaluation unresolved.
Problem
Concept-to-text generation must scale beyond small datasets with limited vocabularies to large, diverse biography domains.
Method
A neural table-conditioned language model combines global and local conditioning with copy actions for words drawn from each input infobox.
Results
The model outperforms a Template Kneser-Ney baseline by nearly 15 BLEU on first-sentence biography generation.
Takeaways & Limitations
The approach generates fluent descriptions of arbitrary people from structured data while scaling to a corpus of 728,321 biographies.
Takeaways & Limitations
The study generates only first sentences, and its loss does not explicitly penalize incorrect facts or evaluate factual accuracy beyond BLEU, ROUGE, and NIST.
Abstract
from arXiv · showhide
This paper introduces a neural model for concept-to-text generation that scales to large, rich domains. We experiment with a new dataset of biographies from Wikipedia that is an order of magnitude larger than existing resources with over 700k samples. The dataset is also vastly more diverse with a 400k vocabulary, compared to a few hundred words for Weathergov or Robocup. Our model builds upon recent work on conditional neural language model for text generation. To deal with the large vocabulary, we extend these models to mix a fixed vocabulary with copy actions that transfer sample-specific words from the input database to the generated output sentence. Our neural model significantly out-performs a classical Kneser-Ney language model adapted to this task by nearly 15 BLEU.
1 Introduction
The paper extends concept-to-text generation from small, narrow datasets to diverse Wikipedia biographies. Its neural model combines global and local table conditioning without restricting relations between fields and generated text.
- Wikipedia biographies provide over 700k records and a vocabulary exceeding 400k words, far beyond prior WEATHERGOV and ROBOCUP datasets.Earlier datasets contain only a few tens of thousands of records and about 350 words each.
- The model targets first-sentence biography generation, where it must select suitable fields from many possible infobox attributes.Large diversity makes rare-event probability estimation difficult for classical count-based models.
- Neural embeddings for words and fields, together with a neural language model, address sparsity in the large biography domain.The factorization is designed to scale to the dataset's large vocabulary.
- The approach does not restrict relations between field contents and generated text, unlike alignment-tree, probabilistic-context-free-grammar, and tree-adjoining-grammar strategies.This provides a more flexible conditioning structure than the cited alternatives.
- Global conditioning summarizes the personality's available information, while local conditioning represents previously generated tokens through their relations to the infobox.The paper analyzes both forms and reports that they are complementary.
2 Related Work
Earlier concept-to-text systems used rules, modular data-driven components, hybrid methods, or neural models developed for smaller domains. This paper is most closely related to neural WEATHERGOV and ROBOCUP generation but emphasizes greater scalability.
- Traditional generation systems relied on hand-crafted rules and modular decisions for content planning, sentence planning, and surface realization.These decisions are highly interdependent despite being organized into separate modules.
- Data-driven approaches learn individual generation modules using record-sentence alignment, hierarchical hidden semi-Markov models, or supervised fact partitioning.These methods separately address content selection, fact ordering, or sentence planning.
- Hybrid systems combine human-contributed rules with corpus statistics at the intersection of rule-based and statistical generation.The cited work includes systems for leveraging both sources of information.
- Neural language models have recently been applied to image captioning, machine translation, and conversations or dialogues.The paper's approach is inspired by these neural generation applications.
- The closest prior system uses an LSTM encoder-decoder with attention for WEATHERGOV and ROBOCUP, whereas this paper uses a simpler design intended to scale better.The paper states that the prior architecture's attention mechanism reduces scalability relative to its design.
3 Language Modeling for Constrained Sentence generation
The model conditions language generation on structured table information at both local and global levels. It also augments the output domain with table tokens so sample-specific and out-of-vocabulary words can be copied into generated text.
- The model treats a table as field/value pairs and conditions sentence generation on those pairs.This extends conditional language modeling from ordinary word context to structured facts.
- Local conditioning describes previously generated words through their occurrences in the table, represented by field and position pairs.Most words have an empty occurrence descriptor because they do not appear in the table.
- Field and within-field position features encode regularities such as date-token sequences and make field-specific continuations more likely.The position of a number within a date field provides additional predictive information.
- Adding position counted from the field end captures field termination and transitions, such as moving from a name to a birth-date introduction.The original field/position scheme cannot directly express that a token ends a field.
- Global conditioning uses all available fields and table words, helping distinguish entities whose field names are similar but whose values differ.For example, actual team or league tokens can distinguish basketball from hockey players when field names alone cannot.
- Copy actions define the output domain as W∪Q, allowing table tokens outside the fixed vocabulary to be generated through field-specific placeholders.For example, an out-of-vocabulary name token can be copied from its position in the name field.
4 A Neural Language Model Approach
The paper develops a table-conditioned neural language model that represents words, field positions, and global table information as embeddings. It combines vocabulary and table-based scores so generation can use both fixed-vocabulary words and table-specific tokens.
- 4 A Neural Language Model Approach: Word and table elements are mapped into continuous embeddings that are combined into a latent context representation.Word embeddings encode the preceding context, while table embeddings encode field positions and global table information.
- 4 A Neural Language Model Approach: Words occurring multiple times in a table are represented by component-wise maxima over their start and end position embeddings.This aggregation selects the strongest positional features across all occurrences of a word.
- 4 A Neural Language Model Approach: The model conditions language generation on preceding words together with local and global information from a fact table.Local conditioning represents field and position information, while global conditioning summarizes fields and words in the table.
- 4 A Neural Language Model Approach: The model scores each candidate word with a vocabulary component and, for table words, a field-based component before applying softmax.The final score sums vocabulary and field scores, with the field score set to zero for words outside the table-copy set.
- 4 A Neural Language Model Approach: The neural language model is trained by minimizing sentence negative log-likelihood with stochastic gradient descent.The training objective uses the conditional log probability of each target word given its context and table-related representations.
5 Experiments
The experiments evaluate table-to-biography generation using the WIKIBIO dataset, classical Kneser-Ney baselines, neural models, and multiple language-generation metrics. The setup includes fixed vocabulary and table-copy variants, with hyperparameters selected through validation.
- 5 Experiments: WIKIBIO contains 728,321 English Wikipedia biography articles with infoboxes and first-sentence targets.The corpus uses 80% training, 10% validation, and 10% test data, and is publicly available.
- 5 Experiments: The baseline is an interpolated 5-gram Kneser-Ney language model, including a template variant that replaces table words with field descriptors.The neural models use 11-gram language models with a learning rate of 0.0025.
- 5 Experiments: Table 1 separates models without copy actions from models with copy actions and reports five-run means with standard deviations for neural systems.Perplexities marked with different symbols are not directly comparable because the output vocabularies differ slightly.
- 5 Experiments: The experiments include fields occurring at least 100 times, the 20,000 most frequent vocabulary words, and validation-based hyperparameter selection.Training uses early stopping when BLEU stops improving on a 1,000-sentence validation subset.
- 5 Experiments: Evaluation uses perplexity for language modeling and BLEU-4, ROUGE-4 F-measure, and NIST-43 for generation quality.These metrics assess both probability modeling and generated-text quality.
6 Results
The results show that richer conditioning and copy actions substantially improve biography generation, while attention supports field continuation and transitions. The best neural model also combines strong BLEU performance with faster decoding than the baseline.
- 6.1 The more, the better: Local conditioning improves generation by linking previous predictions to table fields, and adding more conditioning information consistently improves BLEU, ROUGE, and NIST.Without copy actions, local field-start and field-end conditioning further improves accuracy over plain neural and interpolated Kneser-Ney models.
- 6.1 The more, the better: Nearly 15 BLEU improvement over the Template Kneser-Ney baseline results from global field conditioning and adding words, with corresponding gains of ROUGE +15 and NIST +2.8.Global conditioning improves performance by over 7 BLEU, and adding words contributes another 1.3 BLEU.
- 6.2 Attention mechanism: Attention assigns high probability to continuing unfinished fields and transitioning between field types, such as moving from birthdate to occupation.The attention visualization displays probability distributions over field-position pairs conditioned on preceding generated words.
- 6.1 The more, the better: At beam size K = 5, the model achieves its best validation BLEU and generates a sentence in about 200 ms, several times faster than the baseline.The comparison varies beam size and measures BLEU against average generation time on 1,000 validation samples.
- 6.4 Qualitative analysis: The qualitative example exposes inconsistent training data because the birth month in the infobox differs from the first sentence of the corresponding Wikipedia article.The discrepancy may result from an editor changing one source without updating the other.
- 6.4 Qualitative analysis: Qualitative generations copy names and dates from the table, while global conditioning on fields and words helps infer the correct occupation, including “computer scientist.”The word-level global conditioning is needed to infer the more specific occupation in the example.
7 Conclusions
The model generates fluent descriptions from structured data, improving substantially with local and global conditioning while outperforming a Kneser-Ney language model by nearly 15 BLEU. Its scope remains limited to first-sentence generation, and its loss does not explicitly penalize incorrect facts.
- Nearly 15 BLEU improvement over a Kneser-Ney language model demonstrates the model’s stronger generation performance.The model also generates fluent descriptions of arbitrary people from structured data.
- The task scales beyond previous work to an order of magnitude more data and a vocabulary three orders of magnitude larger.
- The paper evaluates only first-sentence generation, leaving longer biographies for future work.
- The training loss does not explicitly distinguish incorrect facts, such as nationality or occupation, from ordinary word-choice errors.The authors identify factual-accuracy-aware training and evaluation beyond BLEU, ROUGE, or NIST as important future directions.