Source-linked AI summary
ToTTo: A Controlled Table-To-Text Generation Dataset
Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das
TL;DR
Data-to-text generation needs datasets that support inference without sacrificing naturalness or faithfulness. ToTTo introduces an open-domain controlled table-to-text task and revision-based annotation process, then shows that neural models still hallucinate unsupported phrases despite faithful references.
Problem
Existing data-to-text datasets can leave content selection ambiguous, remove inference by specifying meaning representations, or use targets that are noisy or stylistically narrow.
Method
ToTTo pairs Wikipedia tables with highlighted cells and has annotators iteratively revise existing candidate sentences into faithful one-sentence targets.
Results
State-of-the-art neural models often generate fluent phrases that are not supported by ToTTo’s faithful tables and targets.
Takeaways & Limitations
ToTTo provides a benchmark for controllable, high-precision data-to-text generation and faithfulness evaluation.
Takeaways & Limitations
Models face additional difficulty on rare or complex topics in the open-domain task.
Abstract
from arXiv · showhide
We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.
1 Introduction
ToTTo addresses challenges in data-to-text generation by pairing a controlled task with an annotation process designed to produce natural, faithful targets. Its broad-domain dataset also reveals that neural models still struggle with faithfulness.
- Data-to-text generation produces textual descriptions y from structured source content x such as tables.
- Existing tasks either leave content selection open-ended or verbalize fully specified meaning representations, creating evaluation difficulties or reducing inference demands.
- Natural text paired with tables can be noisy and unsupported by the source, while from-scratch annotation often lacks structural and stylistic variety.
- ToTTo asks models to generate one sentence from a Wikipedia table and highlighted cells, while annotators revise candidate Wikipedia sentences into faithful targets.
- ToTTo spans varied domains with completely faithful targets, yet state-of-the-art neural models often generate fluent but unsupported phrases.
2 Related Work
ToTTo differs from prior data-to-text datasets through open-domain tables, highlighted-cell content selection, and revision-based annotation that aims to combine natural language with faithful targets.
- Task Design: Many existing datasets restrict topics and schemas, whereas ToTTo uses tables spanning varied topical categories across Wikipedia.
- Task Design: ToTTo uses highlighted cells to make content selection part of generation rather than limiting the task to surface realization.
- Annotation Process: From-scratch annotation can produce faithful targets but often lacks variety in structure and style.
- Annotation Process: Naturally occurring table-text pairs are more diverse but may contain information that cannot be inferred from the source.
- Annotation Process: ToTTo revises existing Wikipedia sentences so targets retain varied natural language while containing information supported by the table.
3 Preliminaries
ToTTo represents Wikipedia tables as sets of structured cells and associates each annotation example with table metadata and a sentence.
- A Wikipedia table t is defined as a set of τ cells, accommodating non-regular grids.
- Each cell stores its value, header status, row and column position, and the number of rows and columns it spans.
- Table metadata m includes the page title, section title, and up to the first two sentences of section text.
- An annotation example d is a tuple of table, metadata, and sentence, and the dataset D contains N such examples.
4 Dataset Collection
ToTTo collects table-sentence candidates from Wikipedia with heuristics based on numbers, cell contents, and hyperlinks, then samples them for annotation.
- The collection process excludes infobox tables and uses three heuristics to find candidate table-sentence pairs.
- Candidate Extraction: Number matching finds same-page tables and sentences sharing a non-date number with at least three non-zero digits.
- Candidate Extraction: Cell matching selects sentences containing tokens from at least three distinct cell contents in the same table row.
- Candidate Extraction: The hyperlink heuristic links sentences to list pages whose tables link back, producing more diverse but noisier examples.
- Dataset Splits: The sampled splits contain 191,693 training, 11,406 development, and 11,406 test examples.
5 Data Annotation Process
ToTTo’s annotation process incrementally removes unsupported content, replaces context-dependent references, and corrects grammar to produce faithful, fluent sentences. The process also measures agreement at each step.
- The primary annotation workflow comprises table readability, cell highlighting, phrase deletion, and decontextualization, followed by secondary grammar correction.These stages guide annotators from assessing the table through producing the final sentence.
- 99.5% of tables are judged readable before annotation proceeds.Unreadable or poorly formatted tables are excluded from subsequent steps.
- 69.7% of examples are supported by the table after annotators highlight cells supporting the sentence.Support may be directly stated in cell contents or metadata, or logically inferred from them.
- 85.3% of examples differ from the original sentence after unsupported phrases are deleted, reducing average length from 26.6 to 15.9 tokens.Annotators are restricted to deleting phrases during this stage.
- Decontextualization permits at most one replacement of a pronoun or ambiguous phrase with a named entity from the table or metadata.In the example, “she” is replaced with “Gabriele Becker”; multiple replacements caused excessive clarification.
- 68.3% of decontextualized sentences differ from deletion-only sentences, and secondary grammar correction edits 27.0% of sentences.Average length increases from 15.9 to 17.2 tokens after decontextualization and from 17.2 to 17.4 after secondary correction.
6 Dataset Analysis
TOTTO combines broad Wikipedia coverage with controlled analyses of annotation agreement, linguistic challenges, and dataset splits. Its construction yields substantial agreement, while filtering creates balanced overlap and non-overlap evaluations.
- Annotator agreement: 99.38% agreement was reached for table readability; highlighted-cell selection achieved 73.74% complete agreement and Fleiss’ kappa of 0.856.The highlighted-cell agreement is classified as almost perfect under the cited scale.
- Annotator agreement: Annotator BLEU-4 agreement decreased across revision stages from 82.19 to 72.56 to 68.98, compared with 43.17 between original and final sentences.The analysis uses single-reference BLEU among annotator pairs for examples with nonempty highlighted-cell sets.
- Topics and linguistic phenomena: Sports and Countries comprise 53.4% of the dataset, while the remaining 46.6% spans broader topics including Performing Arts, Politics, and North America.TOTTO is limited to topics represented in Wikipedia.
- Training, development, and test splits: Each development and test example has three annotated final sentences, while evaluation uses one randomly selected highlighted-cell set as input.Only examples with nonempty highlighted-cell selections are retained for these splits.
- Training, development, and test splits: After filtering, TOTTO contains 120,761 training examples and four evaluation subsets ranging from 3,784 to 3,916 examples.The overlap and non-overlap subsets are constructed to increase generalization challenge and have similar sizes.
7 Machine Learning Task Construction
The machine-learning task maps a Wikipedia table, related metadata, and highlighted cells to a final sentence. Unlike annotator revision, models must generate a new sentence from these inputs.
- Machine learning task construction: The task learns f: x → y, where x includes the table, metadata, and highlighted cells, and y is the final sentence.Metadata consists of the page title, section title, and table section text.
- Machine learning task construction: Models must generate a new sentence rather than revise an existing starting sentence provided to annotators.This distinction makes the machine-learning task more challenging than the annotation procedure.
8 Experiments
Experiments compare several neural baselines, input representations, and human-evaluated outputs on TOTTO. BERT-to-BERT performs best, but models lag substantially behind faithful reference targets, especially on non-overlap examples.
- 8.1 Results: BERT-to-BERT models perform best, followed by the pointer-generator model, while the content-planning baseline performs poorly.The authors attribute the planner’s weakness to its engineering for the Rotowire data format and schema.
- 8.1 Results: Performance is significantly lower on the non-overlap set than on the overlap set for all models.This indicates that non-overlap examples pose substantial challenges for machine-learning models.
- 8.1 Results: The full-table representation performs poorly despite preserving more table knowledge than the subtable representation.Table 10 compares full-table and subtable inputs for the BERT-to-BERT model.
- 8.2 Human evaluation: Human evaluation measures fluency, faithfulness precision, highlighted-cell coverage, and coverage relative to references.Faithfulness requires every information piece to be supported by the table or a reference.
- 8.2 Human evaluation: The oracle consistently performs strongly, while models show a considerable gap in coverage and faithfulness.Both baselines are significantly less faithful than the reference, with p < 0.001 for the χ2 test.
9 Model Errors and Challenges
TOTTO exposes hallucination, rare-topic handling, structural and numerical reasoning, and metric limitations as distinct challenges for current models.
- Hallucination: Models sometimes generate plausible phrases such as “first” or “winning” that are unsupported by the table.This provides evidence that neural models struggle with faithfulness even when reference targets are faithful.
- Rare topics: TOTTO’s open-domain coverage creates challenges for rare or complex topics near the tail of the topic distribution.The paper cites microdrive capacities as an example.
- Diverse table structure and numerical reasoning: Some examples require counting rows and columns or comparing columns to infer phrases such as “last” and “career highs.”These cases test structural and numerical reasoning beyond direct cell verbalization.
- Diverse table structure and numerical reasoning: The model cannot make these inferences from the simplistic source representation used.
- Evaluation metrics: BLEU can miss major semantic differences when prediction and reference differ by only one word.The authors therefore caution that the leaderboard is not a definitive measure of performance.
10 Conclusion
The paper presents TOTTO as a controlled table-to-text dataset built through iterative sentence revision. It positions the dataset, baselines, and evaluations as resources for model and metric development.
- 10 Conclusion: TOTTO is a table-to-text dataset with a controlled generation task and iterative sentence-revision annotation process.
- 10 Conclusion: The paper provides several state-of-the-art baselines for TOTTO.
- 10 Conclusion: The authors demonstrate that TOTTO can serve as a research benchmark for model and metric development.
A Appendix
The appendix documents the modified PARENT metric, baseline configurations, and additional complex-table examples.
- A Appendix: The appendix includes information about the PARENT variant used for evaluation.
- A Appendix: The appendix provides hyperparameter details for BERT-to-BERT, Pointer Generator, and Content planner baselines.
- A Appendix: Additional examples of complex tables appear in Figures 2–5, covering temporal reasoning, rare topics, and interesting reference language.
- A.1 PARENT metric: PARENT combines precision and recall terms while incorporating both predictions, references, and table content.Its recall terms compare predictions with references and tables, with an additional reward for table phrases absent from the reference.
- A.1 PARENT metric: TOTTO modifies PARENT so its additional recall term uses highlighted cells rather than the entire table.The other recall and precision terms continue to use the full table.