Source-linked AI summary
Logical Natural Language Generation from Open-Domain Tables
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, William Yang Wang
TL;DR
Existing neural NLG largely emphasizes fluent surface realization rather than logical inference from open-domain tables. The paper introduces logical NLG, the LOGICNLG testbed, and fidelity metrics, then evaluates generation methods under this setting. It finds broad benefits from pretraining, fluency–fidelity trade-offs for RL and adversarial training, and partial mitigation from coarse-to-fine generation.
Problem
Existing NLG studies emphasize surface realization, with limited attention to generating statements logically entailed by open-domain table facts.
Method
The paper introduces logical NLG and LOGICNLG, proposes automatic logical-fidelity metrics, and evaluates architectures and training strategies including coarse-to-fine generation.
Results
Pre-trained language models improve fluency and logical-fidelity metrics, while coarse-to-fine generation partially alleviates fidelity problems while maintaining high fluency.
Takeaways & Limitations
Logical NLG exposes limitations of monotonic generation and provides a benchmark for studying planning and inference in table-to-text generation.
Takeaways & Limitations
Automatic fidelity metrics remain inaccurate because their underlying models are imperfect, although their trends are consistent with human evaluation.
Abstract
from arXiv · showhide
Neural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with limited emphasis on logical inference, an important aspect of human thinking and language. In this paper, we suggest a new NLG task where a model is tasked with generating natural language statements that can be \emph{logically entailed} by the facts in an open-domain semi-structured table. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset \cite{chen2019tabfact} featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic metrics to evaluate the fidelity of generation models w.r.t.\ logical inference. The new task poses challenges to the existing monotonic generation frameworks due to the mismatch between sequence order and logical order. In our experiments, we comprehensively survey different generation architectures (LSTM, Transformer, Pre-Trained LM) trained with different algorithms (RL, Adversarial Training, Coarse-to-Fine) on the dataset and made following observations: 1) Pre-Trained LM can significantly boost both the fluency and logical fidelity metrics, 2) RL and Adversarial Training are trading fluency for fidelity, 3) Coarse-to-Fine generation can help partially alleviate the fidelity issue while maintaining high language fluency. The code and data are available at \url{https://github.com/wenhuchen/LogicNLG}.
1 Introduction
The paper introduces logical NLG, which generates statements logically entailed by table facts rather than merely restating surface information. It highlights the mismatch between sequence order and logical order and studies models, metrics, and generation strategies for this challenge.
- Existing NLG methods primarily address surface-level fidelity by copying or selecting input facts, leaving logical inference underexplored.
- Logical NLG asks models to generate natural-language statements logically entailed by table data while remaining linguistically and logically consistent.
- LOGICNLG extends table-to-text generation with mathematical, comparison, and counting inferences over open-domain tables.
- Monotonic left-to-right generation struggles because logical dependencies may require future tokens that precede them in sequence order.
- The paper proposes logical-fidelity metrics, examines the sequence–logic mismatch, and evaluates neural generation models using automatic and human evaluation.
2 Dataset and Problem Definition
LOGICNLG is an open-domain table-to-text dataset built from TabFact statements requiring logical inference. The task maps semi-structured tables and titles to fluent statements that are numerically and logically supported by the table.
- Existing NLG datasets mainly describe records at the surface level, motivating a broader testbed for logical inference.
- LOGICNLG uses TabFact positive statements requiring logical inference, rather than arbitrary surface descriptions.
- The dataset contains 28,450 training, 4,260 validation, and 4,305 test examples from 7,392 open-domain Wikipedia tables.
- Its examples cover diverse operations, while open schemas and rich numerical records challenge rule-based systems and model generalization.
- The formal task conditions a neural generator on a table T and its title to produce a statement Y supported by the table.
3 Automatic Evaluation
The paper evaluates logical fidelity with parsing-based and NLI-based metrics, complemented by adversarial examples that test robustness to logic-changing perturbations.
- Motivation: The evaluation targets logical fidelity because extractive IE can miss non-triple sentences and logically composed statements.These failures motivate approximate automatic metrics rather than direct triple matching.
- Parsing-based Evaluation: Parsing-based evaluation extracts entities and predicates, synthesizes candidate logical forms through breadth-first search, and executes them against the table.A scorer reranks logical forms and filters spurious candidates.
- NLI-based Evaluation: NLI-based evaluation uses Table-BERT to estimate whether the generated sentence is entailed by a linearized table.The model is trained on positive and negative TabFact examples, and the entailed ratio approximates fidelity.
- Adversarial Evaluation: Adversarial evaluation changes the minimum number of words needed to reverse sentence logic while preserving linguistic properties such as length and style.A successful defense occurs when the model scores the original sentence above its adversarial counterpart.
- Discussion: SP-Acc and NLI-Acc have complementary sensitivities, whereas adversarial evaluation more accurately reflects reasoning on tested samples but may not represent general reasoning.The paper reports both model-based metrics and finds consistency with human judgment.
4 Baselines
The baselines combine non-pretrained and pretrained generation architectures with multiple training algorithms, including likelihood, adversarial, and reinforcement-learning objectives.
- Baseline Models: The baseline suite compares LSTM and Transformer models with copy mechanisms against GPT-2 and BERT models using sub-word units.These models are trained under maximum likelihood, adversarial training, or reinforcement learning.
- Non-pretrained Models: Non-pretrained models encode tables with field-infusing or field-gating strategies before decoding text with LSTM or Transformer layers.Field-infusing concatenates field, row, and word information into cell embeddings; field-gating controls cell-boundary encoding.
- Pre-trained Models: Pretrained models linearize each table into a template-based paragraph that serves as input to language models.The table is flattened by scanning its cells horizontally into a document representation.
- Pre-trained Models: GPT-TabGen generates the output sentence directly from the linearized table paragraph and is fine-tuned by maximizing p(Y|P_T; β).The paragraph is fed directly to GPT-2, whose parameters are fine-tuned on LOGICNLG.
- Pre-trained Models: BERT-TabGen repeatedly masks future target words, re-encodes the partial sentence, and predicts the next word from attention-derived states.Because BERT is bidirectional, generation requires n passes for an n-word sentence.
- Training: Adversarial regularization replaces sentence entities or numbers with table values, while reinforcement learning uses semantic-parser rewards for long-term fidelity.Reinforcement learning samples tokens, rolls out completions, and assigns the full sentence a binary reward.
5 Coarse-to-Fine Generation
The paper introduces coarse-to-fine generation to address the mismatch between sequence order and logical order. It first generates a template capturing global structure, then realizes a grounded sentence with additional context.
- Coarse-to-Fine Generation: The surface realization model receives more context, helping it capture logical dependencies that standard monotonic generation cannot model from future tokens.The approach exposes later sentence context during realization, although some dependencies remain uncaptured.
- Coarse-to-Fine Generation: Coarse-to-fine generation first predicts a template that determines global logical structure, then generates the final grounded sentence conditioned on it.Entities and numbers are replaced with [ENT] placeholders during template construction.
- Coarse-to-Fine Generation: Unlike rigid slot filling, fine-grained generation can modify non-slot surface words, which helps preserve linguistic coherence.This flexibility distinguishes the method from template-based or delexicalized generation.
- Coarse-to-Fine Generation: The scheme decouples sentence structure generation from entity grounding, partially alleviating the mismatch between sequence order and logical order.The two-step process is presented as an initial step toward fully non-monotonic generation.
6 Experiments
The experiments compare architectures and training algorithms using automatic and human evaluation. Pre-trained models improve fluency and fidelity, while reinforcement and adversarial training trade fluency for fidelity; coarse-to-fine improves logical metrics while preserving fluency.
- Experimental Setup: The study comprehensively evaluates LSTM, Transformer, and pre-trained language models with multiple training algorithms on LOGICNLG.The experiments include automatic evaluation followed by human evaluation and error analysis.
- Automatic Evaluation: Pre-trained models outperform non-pretrained models on fluency and fidelity, with GPT-TabGen and BERT-TabGen achieving similar performance.Pre-trained models also substantially lower perplexity; GPT-TabGen is preferred because BERT-TabGen decodes more slowly.
- Automatic Evaluation: Reinforcement learning and adversarial regularization improve only their optimized fidelity metric while significantly reducing fluency scores.The result is consistent with limitations of the monotonic generation paradigm.
- Automatic Evaluation: Coarse-to-fine generation yields reasonable improvement on NLI-Acc and Adv-Acc, indicating better capture of logical dependency.Entity-name phrase substitution also preserves entity completeness.
- Human Evaluation: Human evaluation finds that pre-training decreases non-sense outputs, whereas reinforcement learning and adversarial regularization harm fluency and increase them.Coarse-to-fine maintains the non-sense proportion while increasing correct and partial-correct sentences.
- Human Evaluation: The best model produces slightly over 20% logically correct predictions in human evaluation, underscoring the task’s difficulty.Human experts rated sampled sentences as non-sense, wrong, partial-correct, or correct.
- Evaluation Metrics: Automatic fidelity metrics remain challenging because parsing-based and NLI-based evaluators reach only 60% and 65% accuracy, respectively.Their stable and human-consistent trends nevertheless make them useful for model development.
- Fine-grained Analysis: Models perform best on entity ordering and relations, reasonably on superlative and counting operations, and poorly on unique, only, and mathematical aggregation.String-based operations are easier than numeric-based operations, leaving numeric knowledge infusion open.
7 Related Work
Related work frames logical NLG against surface-focused NLG, non-monotonic generation, and factualness evaluation. LOGICNLG extends these directions by emphasizing diverse logical inference over open-domain tables.
- Natural Language Generation: Earlier NLG datasets mainly use surface-level annotations, whereas LOGICNLG incorporates richer logical inference.Traditional NLG generates text from records or data, but LOGICNLG targets inference beyond surface realization.
- Non-monotonic Generation: Non-monotonic generation methods learn generation order without external supervision and can capture more sophisticated dependencies than left-to-right generation.The paper identifies these methods as a plausible direction for LOGICNLG.
- Factualness Evaluation: Factualness evaluation has used information-extraction methods for surface-level matching and natural-language-inference methods for generated summarization.These approaches seek alternatives to costly human evaluation.
8 Conclusion
The paper positions logical NLG as a next-step problem for generation systems and uses LOGICNLG to study inference beyond monotonic generation. It also identifies better automatic metrics as an unresolved need.
- Conclusion: The paper concludes that existing NLG models are restricted by their monotonic nature, making logical NLG an important next-step problem.The authors support this conclusion through comprehensive experiments across architectures and training algorithms.
- Conclusion: Improving automatic metrics remains an unsolved problem because they do not yet reliably measure the exact proportion of logically entailed sentences.The paper promotes a LogicNLG challenge to benchmark future progress.
A Dataset Examples
LOGICNLG pairs each table with five examples designed to cover diverse inference skills, including all, superlative, and count operations.
- Each table in LOGICNLG is associated with five examples covering diversified inference skills.
- Figure 9 requires an all operation to identify multiple rows sharing a property value.
- Figure 10 requires superlative or count reasoning to identify the numerically highest number.
B Logical Operation Distribution
The dataset covers common logical operations, from uniqueness and superlatives to counting, comparison, aggregation, and shared properties; evaluation uses weakly supervised logical-form scoring.
- Logical operations: Superlative operations use max, min, or comparison to identify the lowest or highest value.
- Logical operations: Only identifies a single entity possessing a property that all other entries lack.
- Logical operations: Before/after compares temporal or spatial order, while comparison evaluates relationships between entities.
- Logical operations: Count enumerates entries meeting a criterion, whereas sum/diff performs numeric addition or subtraction.
- Logical operations: Both/neither summarizes shared properties, and average represents the mean number of people attending a game.
- Logical operations: The dataset defines unique as assembling distinct entities using an SQL-like uniq operation.
- Evaluation: The scorer assigns consistency between 0 and 1 to candidate logical forms, using forms returning True as pseudo-positive examples and False forms as pseudo-negative examples.
- Evaluation: Evaluation resolves entities and numbers, composes candidate logical forms through breadth-first search, and ranks them with a scorer.
D Qualitative Example
A qualitative example shows that generated text can be coherent and express complex symbolic logic, yet still contain disfluency, contradictions, and factual errors that are difficult to diagnose.
- Generated samples: Most generated text is coherent and reasonable, but some outputs exhibit repetition, contradiction, disfluency, and erroneous sentences.
- Generated samples: One logically correct sentence combines argmax, argmin, and after operations.
- Generated samples: Two sentences are factually incorrect because the team competes with Seattle only once and the three games are not consecutive.
- Interpretation: The examples show diversified errors that are difficult to debug by inspecting the deep generation model alone.
- Interpretation: The paper argues that future generation models should make the inference process more transparent.