Source-linked AI summary
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, William W. Cohen
TL;DR
Reference texts in automatically constructed table-to-text datasets can diverge from their tables, reducing the reliability of reference-only evaluation metrics. The paper introduces PARENT, which aligns generated and reference n-grams with the table, and finds the highest human correlation across divergent WikiBio settings while remaining applicable to human-elicited WebNLG references.
Problem
Reference-only metrics such as BLEU and ROUGE correlate poorly with human judgments when table-to-text references contain information that diverges from the table.
Method
PARENT aligns n-grams from generated and reference texts with the table using entailment probabilities before computing precision and recall.
Results
PARENT shows the highest correlation with human judgments across divergent-reference WikiBio settings and is comparable to the best existing metrics on human-elicited WebNLG references.
Takeaways & Limitations
PARENT provides an easier-to-use alternative to information-extraction metrics while evaluating table-to-text outputs against both tables and references.
Takeaways & Limitations
The entailment models are not sensitive to paraphrases between tables and text, and embedding-based similarities are beyond the paper’s scope.
Abstract
from arXiv · showhide
Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.
1 Introduction
Table-to-text evaluation commonly relies on human judgments or reference-based metrics, but automatically constructed datasets can contain references that diverge from their tables. The paper proposes PARENT, which incorporates the table to improve evaluation and studies its correlation with human judgments.
- Human evaluation is the gold standard for text-generation quality but is too expensive to apply repeatedly.
- Automatically collected references may contain information unavailable in the associated table, making the gold-standard assumption unreliable.Such references are called divergent from the table.
- PARENT aligns generated and reference text with the table, using a union for precision and an intersection for recall through entailment modeling.This rewards correct table information absent from the reference and ignores incorrect reference-only information.
- The study evaluates 16 table-to-text models on 1,100 WikiBio examples, many containing divergent references.
- PARENT improves correlation with human judgments over existing metrics for both similar and substantially different table-to-text systems.
- Information-extraction metrics have comparable correlation to PARENT, while PARENT is easier to use out of the box.
2 Table-to-Text Generation
Table-to-text generation produces fluent descriptions of structured records, but reference texts can diverge from their tables. This divergence can mislead reference-only metrics, motivating evaluation against both the table and reference.
- Tables are represented as records containing entities, attributes, and values, which the system summarizes as fluent text.
- Evaluation assumes a table, reference description, and model-generated text for each example, with model scores intended to correlate with human judgments.
- Divergent References: In WikiBio, 62% of references mention extra information not present in their associated tables.The paper calls these references divergent, while also noting that they may be technically correct and fluent.
- BLEU and ROUGE penalize a correct candidate for omitting divergent reference information and including correct information from the table.
- PARENT compares generated text with both the table and reference and correctly ranks the three illustrative candidates.
3 PARENT
PARENT evaluates generated text against both the reference and the underlying table, using entailment probabilities to handle divergent references. It combines entailed precision and recall so outputs must be both natural and informative.
- PARENT evaluates each instance by computing generated-text precision and recall against both the table and reference.
- Entailment Probability: Entailment probability w(g) estimates whether an n-gram is correct given the associated table, using lexical-entailment models.The paper presents word-overlap and co-occurrence models; more sophisticated inference is beyond scope.
- Limitations: The entailment models are not sensitive to paraphrases between tables and texts, while embedding-based similarities are identified as a possible extension.
- Entailed Recall: Entailed recall combines reference recall and table recall using a geometric average, balancing natural phrasing with coverage of table information.The parameter λ controls the trade-off between matching the reference and covering table information; the geometric average penalizes low values in either component.
- Entailed Recall: Divergent reference n-grams receive low entailment weights and are downweighted in reference recall rather than treated as required content.Table recall uses longest-common-subsequence matching so entity names appear in the same order as in the table.
- PARENT combines entailed precision and recall into an instance-level F-score and averages these scores across the evaluation set.With multiple references, it computes a score for each reference and takes the maximum; smoothing prevents zero geometric averages.
4 Evaluation via Information Extraction
The paper adapts information-extraction evaluation by extracting structured records from text and comparing generated outputs with references and ground-truth tables. This extends the approach beyond the closed-domain assumptions of earlier work, although the extracted-table system remains imperfect.
- Wiseman et al.’s extraction-based evaluation was limited to closed-domain basketball tables with fixed attributes and string-matched entities.Those assumptions do not hold for the open-domain WikiBio dataset.
- The paper uses a pointer-generator network to linearize tables from text while learning which attributes and values to populate.The output is parsed into attribute–value tuples for evaluation.
- 35.1%: the text-to-table system’s F-score on WikiBio, comparable to other challenging open-domain settings.
- The resulting metrics compare extracted content selection with references and relation generation with the ground-truth table.The passage also introduces RG-F for the task’s emphasis on recalling table information.
5 Experiments & Results
Experiments evaluate automatic metrics against human judgments for WikiBio table-to-text systems, including divergent references, and test robustness across reference quality and datasets.
- Compared Metrics: Reference-only metrics have weak correlations with human judgments, often negative for hyperparameter comparisons; adding table n-grams improves BLEU as BLEU-T.Correlations are higher for systems than hyperparameter comparisons, where similar models make evaluation more difficult.
- Compared Metrics: RG-F and both PARENT variants achieve the highest correlations across both settings, while PARENT-W is significantly better than the other two for systems.RG-F requires an information-extraction pipeline; PARENT-C uses co-occurrence counts, and PARENT-W works out of the box on any dataset.
- Correlation Comparison: For WikiBio entailment, the word-overlap model correlates more highly than the co-occurrence model; WebNLG reverses this ordering because it involves more paraphrasing.The heuristic for selecting λ is sufficient for high PARENT correlations, but human annotations can tune it further.
- Effect of Divergence: Only 38% of WikiBio references were judged entailed by their tables, with entailment defined as mentioning only information inferable from the table.Each pair was judged by three independent workers, using majority vote; workers sometimes disagreed about reasonable entailments.
- Effect of Divergence: PARENT and RG-F remain stable with high correlation as the proportion of entailed examples varies, whereas BLEU and BLEU-T vary substantially.PARENT avoids penalizing systems that mention more table information than references by measuring recall against the table as well as the reference.
- Sentence Level Discrimination: Both PARENT variants outperform other metrics for pairwise sentence-level judgments, but the best binary-task accuracy is only 60%.Higher instance-level accuracy would require more sophisticated language-understanding models for evaluation.
- WebNLG Dataset: On WebNLG, PARENT is competitive or better on average across grammaticality, fluency, and semantics, and performs best for semantics rather than grammar or fluency.Including source tables orients evaluation toward content fidelity, while WebNLG’s human references often have high recall of table information.
6 Related Work
Prior work finds that no single automatic metric works across all generation tasks, while reference-only metrics can fail badly when table-to-text references diverge. PARENT instead evaluates generated text against both references and source tables.
- Prior evaluations found no single metric suitable across all text-generation tasks.
- BLEU and NIST have repeatedly been found unsuitable for judging content quality in natural-language generation.
- PARENT treats reference divergence as analogous to hallucination in the reference text and discounts generated n-grams unsupported by both reference and table.
- Unlike iBLEU, which penalizes source overlap, PARENT rewards overlap because table-to-text evaluation prioritizes factual accuracy over paraphrasing tabular content.
7 Conclusions
The paper studies evaluation of table-to-text systems with divergent references and proposes PARENT as a table-aware alternative. PARENT achieves the highest human correlation across divergent-reference settings in WikiBio, while remaining comparable to strong alternatives on human-elicited WebNLG references.
- PARENT shows the highest correlation with human judgments across divergent-reference settings in WikiBio.
- The study empirically evaluates information-extraction metrics and finds RG-F effective.
- PARENT is comparable to the best existing metrics when references are elicited by humans on WebNLG data.
A.1 Information Extraction System
The information-extraction evaluation system converts reference text into table records for comparison with ground-truth WikiBio tables. It uses a pointer-generator sequence-to-sequence model, despite divergent references causing incomplete extraction and hallucinated false information.
- The WikiBio extractor maps text to tables despite open-domain attributes with no fixed schema.
- Figure 6 uses <R> to separate attribute-value pairs and <C> to separate attributes from values.
- The extractor is a pointer-generator network with bidirectional LSTM encoder and decoder, attention, and source-token copying.
- Divergent references prevent complete table extraction and lead to some hallucinated false information after training.
- Table 5 reports precision, recall, and F-score for extracted pairs against ground-truth tables using exact string matching.
A.2 Hyperparameters
The models use shared training settings for table-to-text and inverse information-extraction PG-Nets. These include 200-dimensional bidirectional LSTM states, 50,000-word vocabularies, GloVe initialization, Adam optimization, and dropout.
- The table-to-text and inverse information-extraction PG-Nets use the same tuned hyperparameter set.
- The bidirectional LSTM hidden-state size is 200.
- Input and output vocabularies contain the 50,000 most common corpus words plus special table-attribute symbols.
- Token embeddings are initialized with GloVe, training uses learning rate 0.0003 with Adam, and biLSTM outputs receive dropout 0.2.
- Models are trained until development-set loss stops decreasing.
A.3 Sample Outputs
Table 6 presents sample references alongside predictions from PG-Net, the best-performing model for WikiBio.
- Table 6 compares sample references with corresponding predictions from PG-Net for WikiBio.