Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference

1 Introduction

• Neural D2T systems can omit or hallucinate facts, making recognition of these errors important for system evaluation and further research.Assessing semantic accuracy requires language understanding; wording changes can alter meaning, while handcrafted heuristics struggle to cover edge cases and human evaluation is costly to scale.Source: §1 · p. 1
• The method uses a pretrained NLI model to test entailment in both directions: generated text against input facts for omissions, and input facts against generated text for hallucinations.The aim is to check whether the semantic information implied by the input and output is equal.Source: §1 · p. 1
• Structured input facts are converted to text with trivial templates so they can be used with the NLI model.The paper presents this as a way to address the mismatch between structured D2T input and the natural-language hypotheses expected by standard NLI.Source: §1 · p. 1
• The two-way check is intended for tasks without content selection; with explicit selection, the selected facts can instead be checked in both directions.For tasks where the generator selects facts without making that selection explicit, the paper proposes checking only for hallucinations.Source: §1 · p. 1
• The metric achieves >90% accuracy on E2E and >75% accuracy on WebNLG, while remaining competitive with domain-specific handcrafted scripts.It requires no human references or in-domain training and uses minimal handcrafting.Source: §1 · p. 2

Automatic Evaluation of NLG

• Reference-based BLEU, ROUGE, and METEOR measure n-gram overlap, while referenceless language-model and linguistic-feature metrics focus on fluency rather than semantic accuracy.Finetuned pretrained-model metrics can capture some semantic-accuracy aspects, but only implicitly.Source: §2 · p. 2

Semantic Accuracy

• The paper reports no generally accepted automatic metric that explicitly measures semantic accuracy of NLG outputs.Slot error rate is a commonly used nearby alternative, typically based on dataset-tailored pattern matching; Goodrich et al. (2019) introduced a neural metric using named-entity recognition and fact extraction.Source: §2 · p. 2

Faithful NLG

• Harkous et al. (2020) also use a pretrained neural classifier to detect inaccurate output, but finetune it on manually augmented domain-specific data.Source: §2 · p. 2
• Unlike those approaches, the paper uses a pretrained NLI-finetuned model without further training on domain-specific data.Source: §2 · p. 2

3.1 NLI Model

• An NLI check passes when entailment has the highest probability among contradiction, neutral, and entailment.The method uses the roberta-large-mnli checkpoint, finetuned on MultiNLI, without further training.Source: §3.1 · p. 2

3.2 Data Preparation

• The metric takes input facts and their verbalization, with facts represented as RDF-like subject-predicate-object triples.Each triple is converted into natural language with a template: either a custom template per predicate or a single universal backoff template.Source: §3.2 · p. 2

3.3 Evaluation Process

• For omissions, each fact is tested as a hypothesis against the generated text as premise; for hallucinations, the concatenated facts are premise and generated text is hypothesis.The omission checks are run separately by fact, which identifies which facts were omitted; the hallucination check cannot be split into simpler checks.Source: §3.3 · p. 2, Figure 1 · p. 3
• FINE distinguishes OK, omission, hallucination, and both errors, while ROUGH collapses all error types into not_OK.FINE can distinguish whether a system tends to omit or hallucinate; ROUGH is suited to reranking or filtering outputs, and confidence is the minimum entailment probability across checks.Source: §3.3 · p. 3

4 Experimental Setup

• The experiments use the English WebNLG and E2E datasets, both of which have shared-task system outputs and semantic-accuracy measures.Source: §4 · p. 3
• For WebNLG, the metric is compared with crowdsourced semantic-adequacy ratings, treating scores of 2.5 or higher on a three-point scale as correct.The authors also tried a 2.0 threshold and report slightly worse results.Source: §4 · p. 3
• For E2E, the handcrafted challenge script supplies the ground truth, and expert-annotated outputs are used for gold-standard and classifier comparisons.The authors note that the crowdsourced E2E semantic-accuracy results were unreliable and overestimated errors; unlike their metric, the handcrafted approach is viable only for limited domains such as E2E.Source: §4 · p. 3
• The Default setup uses one custom template per predicate, while Backoff uses a single universal template; templates are filled with triple values and concatenated for multitriple inputs.WebNLG Default templates come from delexicalized single-triple training references, whereas E2E uses eight handcrafted templates.Source: §4 · p. 3

5 Results Analysis

• The evaluation reports accuracy, precision, recall, and F1, treating not_OK samples as positive when measuring error detection.Manual error analysis sampled 100 disagreements per dataset; the E2E challenge’s crowdsourced accuracy evaluation was reported as unreliable and prone to overestimating errors.Source: §5 · p. 3
• Results were well above the random baselines of 0.5 for ROUGH and 0.25 for FINE, but differed between WebNLG and E2E.The section discusses WebNLG and E2E results separately.Source: §5 · p. 4

5.1 WebNLG Analysis

• WebNLG disagreements involved noisy human ratings and imprecise templates, while annotators also penalized accurate but ungrammatical or poorly organized outputs that the metric rated OK.The authors note that better template selection could mitigate the template issue.Source: §5.1 · p. 4
• Re-examination found 42 of 100 WebNLG disagreement cases correctly classified by the metric, suggesting reported performance may be understated by erroneous human annotations.The WebNLG results also showed a −0.05 Spearman correlation between input-triple count and metric errors in the Default setting.Source: §5.1 · p. 4
• WebNLG confidence scores correlated with average human scores at 0.63, compared with reported BLEU and METEOR correlations of 0.59 and 0.73.The approach does not require human-written references.Source: §5.1 · p. 4, Table 1 · p. 4

5.2 E2E Analysis

• On E2E, the metric showed over 90% agreement with the handcrafted script, with only a slight performance drop as input triples increased (Default-setting Spearman correlation −0.08).The E2E results are summarized in Table 2.Source: §5.2 · p. 4, Table 2 · p. 4
• In the E2E error analysis, the metric correctly classified 45 of 100 examples labeled erroneous by the handcrafted script, suggesting reported performance may be understated because of annotation errors.The analysis also considered output expressions that do not correspond to input facts to be hallucinations.Source: §5.2 · p. 4

5.3 E2E MR Classifier Comparison

• On E2E system outputs, the approach reached 97.8% vs. 99.5% accuracy, slightly less precise than both handcrafted scripts.Excluding eat-Type=restaurant yielded 76.5% accuracy and 97.6% recall on human-written data.Source: §5.3 · p. 5, Table 3 · p. 5

• The metric detects omissions or hallucinations without human references or training and with minimal handcrafting, achieving results competitive with crowdsourced ratings or domain-specific scripts.Source: §6 · p. 5
• The authors say the WebNLG and E2E datasets remain limited, leaving performance on long-text generation and tasks requiring content selection for future evaluation.Source: §6 · p. 5
• Future work will explore combining the metric with fluency or grammaticality checks and better separating grammaticality from semantic correctness in human evaluation.The authors report that human semantic-correctness ratings are influenced by grammaticality.Source: §6 · p. 5

• The WebNLG subset contains 223 sampled inputs, each paired with outputs from 10 NLG systems, for 2,230 instances.Source: Dataset details · p. 8
• The main E2E experiments cover 13,230 outputs: 21 system outputs for each of 630 test inputs.Source: Dataset details · p. 8
• E2E semantic-classifier comparisons use expert-annotated sets of 200 training-set instances and 400 outputs from TGen variants.Source: Dataset details · p. 8

• Among disagreements, crowdsourced annotation was correct in 51 cases, the NLI approach in 42, and seven remained incorrect or ambiguous.Source: WebNLG Error Analysis · p. 8
• Annotation problems and inaccurate templates each accounted for 22 counts; one human rating fell below threshold without an apparent reason, while an underspecified template prompted a false hallucination assessment.Analyzed examples may belong to multiple or none of the error classes.Source: WebNLG Error Analysis · p. 8
• An underspecified Aenir language template led the NLI to treat an acceptable English-language statement as a hallucination.Source: WebNLG Error Analysis · p. 8
• Disfluent or disorganized sentences occurred 14 times; in an example, poor organization likely lowered the human score but caused no NLI problem.Source: WebNLG Error Analysis · p. 9
• Value-format errors occurred seven times; an example omitted “comicsCharacter,” which the NLI treated as an omission.Source: WebNLG Error Analysis · p. 9

E2E Error Analysis

• In 100 randomly sampled E2E examples where the methods differed, 34 cases favored the slot error script, 45 favored NLI, and 18 were both incorrect or ambiguous.The analyzed examples may belong to multiple error classes or none.Source: E2E Error Analysis · p. 10
• 40 value-interpretation cases included “adult only” for family-friendly=no, which the slot script accepted but NLI flagged as a hallucination.The text is generally considered correct in E2E, but “adult only” does not necessarily follow from “is not family friendly.”Source: E2E Error Analysis · p. 10
• 33 slot-script pattern errors included missing family-friendly=yes for “for all the family,” although NLI handled the expression without issue.The script likely lacks the “for all the family” pattern.Source: E2E Error Analysis · p. 10
• 18 difficult edge cases included “moderate restaurant” for price_range=moderate, which NLI accepted while the slot script did not.The analysis describes this wording as hard to interpret as a correct verbalization of the price range.Source: E2E Error Analysis · p. 11
• Eight off-topic hallucination cases included “beautiful” and “reasonably liked,” which NLI flagged while the slot script missed them; both methods detected the missing area.The example also omitted a verbalization of the area.Source: E2E Error Analysis · p. 11