PaperlayerView exact revision on arXiv

Source-linked AI summary

Validating DBpedia Triple Sets for Natural Language Generation

Mark Andrade, Simon Mille, Anya Belz, Brian Davis

arXiv:2609.07589v1cs.CL

TL;DR

Data-to-text NLG needs correct facts, while DBpedia quality work has often assessed the graph as a whole rather than individual triples for NLG. This paper validates and recovers entity-specific triple sets using ontology-based checks and property-definition improvements. The resulting selection reaches up to 98% correct triples against manual annotations, while property fixes improve recall without harming precision.

  • Problem

    For data-to-text NLG, input facts should be correct, but existing DBpedia quality assessments consider the graph as a whole rather than whether individual triples are suitable for NLG.

    Source: §1 · p. 1

  • Method

    The approach validates individual triples against DBpedia ontology types and applies validation and recovery to compile high-precision entity-based triple sets.

    Source: §1 · p. 1, §2.3 · p. 4, §5 · p. 12

  • Results

    Up to 98% of selected triples are correct against manually annotated data, and improving some Property definitions increases recall without harming precision.

    Source: §3.2.2 · p. 9, §5 · p. 12

  • Takeaways & Limitations

    The validated triple sets are intended for multilingual NLG, with creating challenging NLG input data identified as the next step.

    Source: §5 · p. 12

    Possibly valid triples are treated as invalid to maximize triple-set quality.

    Source: §3.1 · p. 9

Abstract

from arXiv · show

We present a study of the quality of individual DBpedia triples from the perspective of Natural Language Generation, and propose and evaluate an approach for collecting entity-specific triple sets that filters out questionable triples while minimizing the loss of correct ones. We show in an evaluation against manually annotated data that with validation rules, it is possible to reach 98% precision in triple selection, and with improvements to a few Property definitions, it is possible to improve recall by 40% without harming precision.

1. Introduction

For data-to-text NLG, factual correctness matters more than exhaustive input, so this paper studies the quality of individual DBpedia triples and proposes filtering questionable ones while limiting the loss of correct triples.

1. Introduction

  • For NLG, meaningful and factually correct text takes priority over exhaustive input that risks including nonsensical or incorrect information.This motivation contrasts NLG with knowledge-graph uses where coverage is often more relevant.Source: §1 · p. 1
  • The study assesses individual triples for NLG suitability, not triple-set coverage, cross-triple consistency, ontology quality, or error correction by prediction.It focuses on semantic correctness against the DBpedia ontology and uses information from existing knowledge repositories.Source: §1 · p. 1
  • The assessment collects about 600,000 unique triples for about 11,000 entities and finds a small set of Properties responsible for much invalidity.It applies validation checks and diagnoses causes of invalidity.Source: §1 · p. 1
  • 0.98 precision in triple selection is achievable, with recall around 0.75.The evaluation uses manually annotated data.Source: §1 · p. 2

2. Quality assessment of DBpedia triples

The section assesses DBpedia triple validity by selecting entity sets, retrieving ontology triples, and checking expected against actual types. It finds widespread type uncertainty and recurring gaps in Property definitions.

2.1. Entity selection

  • Experiments used popular Top 1,000 and Top 10,000 lists and a randomly selected 1,000-entity set excluding colon-title pages; random entities were classified into Top 1,000 categories with Claude Sonnet 4.Classification ability was assessed on a separate set of 100 DBpedia entities; two human annotators had observed agreement of 0.96 and 0.95 with Sonnet 4 at Section level and 0.97 with each other (Cohen’s κ=0.65).Source: §2.1 · p. 2

2.2. Triple Selection

  • The collection retrieves dbo: triples with selected entities as Subject or Object, excludes NLG-incompatible Properties, and caps each Property at 50 triples.Ontology Properties were used because expected Subject and Object types are usually available for checking.Source: §2.2 · pp. 2–3

2.3. Validation strategy

  • Validation matches each Property’s expected domain and range against the Subject’s and Object’s actual types, including their superclasses, then assigns one of four labels.A missing entity type or an underspecified Property can yield a possibly-valid label rather than an invalid one.Source: §2.3 · p. 4

2.4. Triple assessment results on three sets of entities

  • 13%, 10%, and 43% of triples were valid on average for Top 1,000, Top 10,000, and Random 1,000 entities, respectively.Valid-or-possibly-valid triples averaged 86%, 80%, and 86% in those datasets, respectively.Source: §2.4 · pp. 4, 6, Table 3 · p. 6, Table 4 · p. 7

2.5. A preliminary diagnosis and recovery strategy

  • Most possibly valid triples stem from Properties lacking expected Subject or Object types, rather than entities lacking DBpedia types.Defining expected domains or ranges for frequently occurring Properties can validate many triples that otherwise would remain unvalidated.Source: §2.5 · pp. 6, 8

3. Towards high quality inputs for Natural Language Generation

The section develops a validation-and-recovery pipeline for NLG triple sets, refining its filters through a pilot assessment and evaluating quality and coverage against manual annotations.

3.1. Preliminary assessment of validation and evaluation guidelines

  • Excluding triple sets where the entity appeared only as Object raised A1 precision/recall to 0.995/0.901 and A2 to 0.937/0.900; final evaluation capped each Property at 10 instances per entity.The pilot found that allowing up to 100 instances of a Property could bias results toward frequently occurring Properties.Source: §3.1 · pp. 8–9

3.2.1. Impact of the validation on triple filtering

  • Validation left 501 of 2,004 triples (25%); the authors considered triples filtered at O1–O3 likely noisy and the remaining non-valid triples potentially recoverable.Source: §3.2.1 · p. 9

3.2.2. Recovering invalid and possibly valid triples

  • Authors revised 19 of 35 problematic Property definitions, generally broadening expected domains or ranges; entity-type problems and difficult-to-define Properties remained unresolved.The authors explicitly state that these changes are not general-purpose Property definitions.Source: §3.2.2 · pp. 9–10
  • Revised Property definitions increased validated triples by 4% on V1, 5% on V2, and up to 55% on V3, raising the count from 501 to 775.Source: §3.2.2 · p. 10

3.3. Evaluation

  • Evaluation manually annotated 1,243 Top 1,000 and 208 random-entity triples, then grouped Unsure with Invalid when scoring precision, recall, and F1.The annotation guidelines distinguished Valid, Unsure, and Invalid triples.Source: §3.3 · p. 11
  • Recovery increased recall by up to 39.5% (V3 0.487 to R3 0.727) and reduced false negatives from 517 to 275, with false positives changing from 10 to 12.R1 achieved the best recall and F1 overall when selecting more triples mattered more than validation priority.Source: §3.3 · p. 11
  • R3 reaches precision 0.984 and 0.969, while R1 has best recall (0.921 and 0.995) and F1 (0.913 and 0.978) on the two datasets.Figure 1 reports results for Top 1,000 entities and fully random entities; its random-entity scores warrant caution because few triples were available.Source: §3.3 · p. 11, Figure 1 · p. 12

4. Related work

Related work covers manual, semi-automatic, and automatic DBpedia error detection; most cited correction approaches use manual correction.

4. Related work

  • DBpedia error detection spans manual, semi-automatic, and automatic methods; most cited corrections are manual, and the closest work is Paulheim and Gangemi.Source: §4 · p. 12

5. Conclusions and Future work

The authors report high-precision triple sets for multilingual NLG, while noting that this comes at a recall cost and outlining further recovery and evaluation work.

5. Conclusions and Future work

  • The approach yields triple sets with up to 98% correct triples, but high precision comes at lower recall; planned work includes recovering more triples and testing other DBpedia editions.The authors also plan to test fixing entity types and using dbp: Properties alongside dbo: Properties.Source: §5 · p. 12

Declaration on Generative AI

The authors report no Generative AI use to create paper content, while noting AI support in developing some code.

Declaration on Generative AI

  • The authors say they used no Generative AI to create the paper’s content, but used Perplexity and Claude 4.8 Opus to support development of some code.Source: Declaration on Generative AI · p. 13

A. Properties not queried

Table 9 lists DBpedia Properties excluded from entity triple queries; the list was compiled manually by observing the data.

A. Properties not queried

B. Entities used in the pilot assessment study

Table 10 presents sample entities for categories used in the Top and Random 1,000 datasets.

B. Entities used in the pilot assessment study

C. Properties in invalid triples

Tables 11–13 list frequently occurring Properties in invalid triples detected by the validation rules, ranked by count across three datasets.

C. Properties in invalid triples

D. Properties in possibly valid triples

Tables 14–16 list frequently occurring Properties in possibly valid triples detected by the validation rules, ranked by count across three datasets.

D. Properties in possibly valid triples

E. Details of the evaluation

Evaluation details include inter-annotator agreement for both entity sets, stepwise precision/recall and true/false positive/negative counts, and tables of common invalid and partially valid triples.

E. Details of the evaluation

  • Table 10 gives examples of entities used in each Top and Random 1,000 category.Source: Table 10 · p. 15
  • Top 1,000 triples had observed agreement 0.9 and linear weighted Cohen’s κ 0.75; removing Unsure annotations raised κ to 0.83, while random-entity triples had agreement 0.97 and κ 0.70.Agreement for fully random entities is only indicative because the number of collected triples was very low.Source: E. Details of the evaluation · p. 15
  • Tables 11–16 list common Properties in invalid or partially valid triples across Top 1,000, Top 10,000, and Random 1,000 categories, marking redefined Properties in bold.Source: Table 11 · p. 16, Table 12 · p. 17, Table 13 · p. 17
  • Tables 17 and 18 report precision and recall statistics at each evaluation step for 50 randomly selected Top 1,000 entities and 50 fully randomly selected entities, with true/false positive/negative details.Source: E. Details of the evaluation · p. 14, Table 17 · p. 20, Table 18 · p. 20
Loading 2609.07589v1…