Source-linked AI summary
DART: Open-Domain Structured Data Record to Text Generation
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, Nazneen Fatema Rajani
TL;DR
Existing Data-to-Text datasets often flatten table structure and cover limited domains, motivating a richer open-domain resource. DART constructs ontology-preserving triple-set–sentence pairs from heterogeneous sources and finds that current models face new challenges, while DART augmentation improves performance on other tasks. Its pretrained models and source resources may carry biases, and model outputs can hallucinate words or fail to capture hierarchical ontology structure.
Problem
Existing Data-to-Text datasets use flat table representations or limited domains, leaving richer semantic structures and broader predicate coverage insufficiently represented.
Method
DART builds an open-domain corpus by encoding table headers and titles as tree ontologies and merging annotated tables with converted questions and existing meaning representations.
Results
DART introduces new challenges to state-of-the-art Data-to-Text models, while data augmentation with DART improves performance across WebNLG 2017 test splits.
Takeaways & Limitations
DART provides an ontology-preserving, open-domain benchmark that supports evaluation of richer semantic structures and out-of-domain generalization.
Takeaways & Limitations
The pretrained models and source resources may contain biases, and models may hallucinate words or fail to capture the hierarchical ontology structure.
Abstract
from arXiv · showhide
We present DART, an open domain structured DAta Record to Text generation dataset with over 82k instances (DARTs). Data-to-Text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and contain nontrivial structures. To this end, we propose a procedure of extracting semantic triples from tables that encodes their structures by exploiting the semantic dependencies among table headers and the table title. Our dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and dialogue-act-based meaning representation tasks by utilizing techniques such as: tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing. We present systematic evaluation on DART as well as new state-of-the-art results on WebNLG 2017 to show that DART (1) poses new challenges to existing data-to-text datasets and (2) facilitates out-of-domain generalization. Our data and code can be found at https://github.com/Yale-LILY/dart.
1 Introduction
DART addresses limitations in existing Data-to-Text datasets by representing richer table semantics and covering diverse open-domain predicates. Its evaluations show that DART creates new challenges for current models while supporting improved generalization to WebNLG.
- Existing datasets often use flat schemas that inadequately encode rich semantic relationships in structured data, especially tables.
- Domain-specific datasets such as E2E and WebNLG provide limited predicate and ontology diversity.
- DART is a large, open-domain corpus designed to harvest diverse predicates from Wikipedia tables.
- Its tree ontology annotation converts flat table schemas into hierarchical semantic frames reflecting core and auxiliary relations.
- Current data-to-text models perform impressively on domain-specific datasets but suffer on DART because of its open-domain nature and richer semantic structures.
- DART data augmentation improves performance on WebNLG 2017 and introduces new generalization challenges for existing models.
2 DART Data Collection
DART combines hierarchical table annotation with converted question-answer data and existing meaning representations to create triple-set–sentence pairs. Its collection process preserves table structure, controls extracted component shape, and broadens predicate and topical coverage.
- Data sources: DART combines Wikipedia table annotations, WikiSQL question conversions, WebNLG, and Cleaned E2E into triple-set–sentence pairs.
- Tree ontology annotation: Internal annotators assign parent columns and construct ontology trees rooted at [TABLECONTEXT], with [TITLE] representing table-title structure when appropriate.
- Tree ontology annotation: Ontology quality control rejects disconnected or cyclic trees, verifies every column header appears, and unifies alternative annotations for similar tables.
- Connected component extraction: Highlighted table cells are selected as a connected subtree, adding nodes up to their lowest common ancestor when necessary to support coherent triples.
- Connected component extraction: Extracted components vary in size from 2 to 5 nodes and in width or depth through an expansion parameter sampled from 0.5 to 0.7.
- Question conversion: WikiSQL questions are converted into declarative sentences with rule-based QA2D, manual screening, and SQL queries restricted to non-aggregate operations.
- Predicate diversity: The Wikipedia partition contains more unique predicates than the WebNLG and Cleaned E2E partitions combined, despite fewer triple-set–sentence pairs.
3 Experimental Results
DART evaluates several data-to-text models on its open-domain, structurally richer input and tests whether DART augmentation transfers to WebNLG 2017. Pretrained models perform strongly on DART, while DART augmentation improves WebNLG results, especially for unseen categories.
- DART: DART benchmarks bidirectional LSTM, Transformer, BART, and T5 models on its test set.The Transformer receives an unordered triple set after linearization.
- DART: 50.66 BLEU is achieved by T5-large, the highest performance among the evaluated DART models.Pretrained models generally outperform other models, and larger model sizes tend to improve performance.
- DART: Pretrained models often hallucinate words and do not fully capture the hierarchical ontology encoded in DART’s linearized triplesets.The authors suggest that models exploiting ontology structure more effectively may perform better.
- WebNLG: DART augmentation consistently improves WebNLG 2017 performance across models and yields new state-of-the-art results on seen, unseen, and all test splits with T5-large.The gains are larger on the unseen split, consistent with DART’s open-domain nature.
- Ablation Study: Human-written DART sentences provide the largest augmentation gains, especially on WebNLG’s unseen split, whereas E2E instances help seen performance but hurt unseen performance.This pattern is consistent across the evaluated models.
4 Human Evaluation
Human evaluation compares DART references with BART-base and T5-base outputs for fluency and semantic faithfulness. Human-written references score highest, while automatically generated declarative sentences are weaker but remain useful for scalable data construction.
- Evaluation Setup: Five annotators evaluate 100 triplesets from each sentence partition for fluency and semantic faithfulness.Each tripleset is paired with one reference sentence and outputs from BART-base and T5-base.
- Results: Human-written references achieve the highest fluency and semantic faithfulness compared with BART-base and T5-base outputs.The evaluation also reveals a considerable faithfulness gap between DART references and pretrained-model outputs.
- Results: Auto-generated declarative sentences are less fluent and faithful than model outputs because they are produced by a rule-based system.The authors retain this partition because it provides an economical way to obtain many instances and contains diverse topics that benefit generalization.
5 Related Work
Existing data-to-text datasets use varied representations, but many table-based approaches rely on flat schemas that miss hierarchical semantic dependencies. DART instead represents table structure with tree ontologies and extracts logically consistent triples for text generation.
- Data Representation: Data-to-text datasets represent structured inputs as slot-value pairs, AMR, MRS, RDF triples, or logic forms.ToTTo additionally supplies highlighted cells with flat row and column headers.
- Existing Limitations: Using only highlighted cells with flat row and column headers produced higher performance than using entire tables in ToTTo.This finding motivates attention to how tabular structure and context are represented.
- DART Representation: DART annotates tree-structured table ontologies that encode semantic dependencies among headers and can incorporate contexts such as table titles.Connected components extracted from the tree form each instance’s input.
- DART Representation: The DART annotation framework encodes hierarchical relationships so extracted triples remain logically consistent and can be described without information loss.This contrasts with flat representations used by many existing datasets.
6 Conclusion
DART is an open-domain corpus whose ontology-preserving inputs create new challenges for data-to-text models, while data augmentation with DART improves performance on other data-to-text tasks.
- DART is an open-domain corpus for structured data record-to-text generation with ontology-preserving data-input representations.
- DART challenges state-of-the-art data-to-text models through its open-domain nature and semantic-triple ontology structure.
- Using DART for data augmentation improves performance on other data-to-text tasks.
- Future work targets controlled, high-fidelity generation that better incorporates data-ontology hierarchies.
7 Ethics Statement
DART combines public tabular and text resources with new structural and sentence annotations. Its experiments use MTurk and internal annotators, while acknowledging possible biases in data, annotations, and pretrained models.
- The dataset adds annotations of tabular structure and human-written sentences describing data records.
- DART combines Wikipedia tables, E2E restaurant meaning representations, and WebNLG entity-relation data from 15 DBpedia categories.
- The source resources and annotator-written sentences may contain biases.
- Internal authors and Amazon Mechanical Turk workers provided additional annotations, with MTurk tasks taking 125 hours at approximately $15 per hour.
- Experiments use BART and T5, whose large-scale pretraining data may introduce biases into the models.
- The study reports 43 experiments: 7 on DART and 36 in the WebNLG ablation study.
Appendix
The appendix documents DART’s evaluation, ontology statistics, annotation examples, and model outputs, including ablation results and cases of factual or structural errors.
- Appendix contents: The appendix includes WebNLG 2017 ablation results, table-ontology statistics, annotation examples, and model-output examples.
- Ablation study: Table 6 compares DART partitions containing auto-generated sentences, human annotations, ontology data, E2E-derived instances, and their full combination.
- Ontology statistics: Table 7 reports ontology properties for WikiTableQuestions and WikiSQL samples, including average branching factor across non-leaf nodes.
- DART examples: A DART instance represents table-derived information as semantic triples and can generate a sentence describing the encoded relations.
- Annotation examples: Data cleaning adds missing column names, links columns to year information, and disambiguates repeated column names.
- Annotation examples: WikiTableQuestions annotations may include the table title in the ontology and in sentences describing highlighted row information.
- Annotation examples: Human validation of generated WikiSQL sentences checks coherence and table-title usage, marking nonsensical outputs separately.
- Model outputs: Examples show BART and T5 outputs that introduce incorrect locations, numbers, dates, or entities relative to the input triples.