Source-linked AI summary
FactKG: Fact Verification via Reasoning on Knowledge Graphs
Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, Edward Choi
TL;DR
Fact verification has not fully used knowledge graphs as evidence, despite the demand for reliable claim checking. This paper introduces FACTKG, a dataset of claims for reasoning over graph evidence, and reports that graphical evidence improves performance over baselines without it.
Problem
Knowledge graphs have not been fully used as fact-verification evidence, despite the demand for reliable claim checking.
Source: §1 · p. 1Method
The paper introduces FACTKG, a dataset of 108k natural-language claims generated from graph-text pairs and organized into five reasoning types and varied linguistic styles.
Source: §1 · p. 2Results
Graphical evidence yielded superior performance to baselines that did not incorporate it, and GEAR outperformed other baselines in most reasoning types.
Source: §1 · p. 2, §4.4 · p. 8Takeaways & Limitations
FACTKG claims are difficult to solve without reasoning over the knowledge graph, whose structure can also serve as an explanation for verdicts.
Source: §5 · pp. 8–9FACTKG uses a 2015-10 version of DBpedia and claims based on single sentences; multi-sentence claims remain future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
In real world applications, knowledge graphs (KG) are widely used in various domains (e.g. medical applications and dialogue agents). However, for fact verification, KGs have not been adequately utilized as a knowledge source. KGs can be a valuable knowledge source in fact verification due to their reliability and broad applicability. A KG consists of nodes and edges which makes it clear how concepts are linked together, allowing machines to reason over chains of topics. However, there are many challenges in understanding how these machine-readable concepts map to information in text. To enable the community to better use KGs, we introduce a new dataset, FactKG: Fact Verification via Reasoning on Knowledge Graphs. It consists of 108k natural language claims with five types of reasoning: One-hop, Conjunction, Existence, Multi-hop, and Negation. Furthermore, FactKG contains various linguistic patterns, including colloquial style claims as well as written style claims to increase practicality. Lastly, we develop a baseline approach and analyze FactKG over these reasoning types. We believe FactKG can advance both reliability and practicality in KG-based fact verification.
1 Introduction
The paper introduces FACTKG to address the limited use of knowledge graphs as evidence in fact verification. It presents evidence that graph evidence improves performance over baselines without it and describes the dataset’s reasoning coverage and broader relevance.
1 Introduction
- Knowledge graphs are underused in fact verification, although their node-edge structure can make reasoning links clearer than text-evidence summaries.The paper also motivates KG-based verification through reliability and applicability beyond fact-checking.Source: §1 · p. 1
- KG-based verification can check consistency between text and graphs, including whether a dialogue system’s internal knowledge matches a user’s utterance.The introduction identifies dialogue systems as one application of internal knowledge graphs.Source: §1 · pp. 1–2
- FACTKG contains 108k natural-language claims labeled SUPPORTED or REFUTED, generated from graph-text pairs and spanning five reasoning types and colloquial and written styles.The five types are One-hop, Conjunction, Existence, Multi-hop, and Negation.Source: §1 · p. 2
- Graphical evidence produced superior performance to baselines that did not incorporate it.The experiments were conducted on FACTKG.Source: §1 · p. 2
2 Related Works
Prior fact-verification research has focused largely on text and tables, while earlier KG tasks used single-triple claims. FACTKG builds on WebNLG’s paired text and graph data.
2.1 Fact Verification and Structured Data
- Fact verification research has mainly used text evidence, with table-based datasets also requiring structured-data reasoning.Examples include FEVER for text, TabFact and SEM-TAB-FACTS for tables, and FEVEROUS for table and text sources.Source: §2.1 · p. 2
- Earlier KG datasets use single-triple claims requiring only One-hop reasoning, whereas FACTKG is described as the first KG-based dataset with natural-language claims requiring complex reasoning.FACTKG uses the full DBpedia graph, reported as 0.1B triples, compared with 592K for FB15K, 310K for FB15K-237, and 150K for WN18.Source: §2.1 · p. 2
2.2 WebNLG
- FACTKG creates REFUTED claims through entity substitutions requiring distant entities and bidirectional contradiction, or relation substitutions preserving head and tail entity types.Relation substitutions use templates specific to the selected relation.Source: Figure 2 · p. 3
- FACTKG uses WebNLG’s paired text and RDF triples from DBpedia as its construction basis.The 2020 WebNLG version used by the authors contains 45,040 text-triple pairs.Source: §2.2 · p. 3
3 Data Construction
FACTKG claims are generated from WebNLG text-graph pairs across five reasoning types, then diversified through colloquial style transfer and presupposition. The section describes how those claims are constructed and quality-checked.
3.1.1 One-hop
- One-hop claims use a single triple; REFUTED examples substitute an entity or relation, with entity replacements checked for graph distance and bidirectional NLI contradiction.Entity replacements must be more than four hops from entities in the original claim; relation replacements preserve compatible head and tail entity types.Source: §3.1.1 · p. 3
3.1.2 Conjunction
- Conjunction claims combine multiple triples and are SUPPORTED only when all corresponding triples exist; REFUTED claims use entity substitution.Source: §3.1.2 · p. 4
3.1.3 Existence
- Existence claims omit either the head or tail of a triple, use templates, and draw from 22 relations selected to avoid unrealistic claims.Source: §3.1.3 · p. 4
3.1.4 Multi-hop
- Multi-hop claims require a graph path through multiple relations; their underspecified entity is replaced with the type name that yields the lowest GPT-2 perplexity.Source: §3.1.4 · p. 4
3.1.5 Negation
- Negations are added across the four claim-generation methods; one-hop labels reverse, conjunction labels depend on negation placement, and multi-hop negatives require a matching path ending outside Papenburg.Source: §3.1.5 · pp. 4–5
3.2.1 Model based
- The colloquial-style model generates 500 candidates, filters for distance, preserved verbs and entities, and semantic consistency, then uses AFLITE to select the most colloquial survivor.On average, 41.2 of 500 candidates survived filtering; if none passed, the original written claim was retained.Source: §3.2.1 · p. 5
3.2.2 Presupposition
- Presupposition claims use factive, non-factive, and structural forms; structural-question claims are limited to one-hop and existence types.Non-factive claims receive the opposite label of the original; structural questions are treated as claims.Source: §3.2.2 · pp. 5–6
3.3 Quality Control
- Two graduate students identified 99.4% of 1,000 randomly sampled substituted claims as REFUTED; in colloquial-style test claims, at least two of three reviewers marked only 9.8% as losing important information.To ensure test-set quality, only claims selected as preserving all facts by two or more reviewers were included.Source: §3.3 · p. 6
4 Experiments
The experiments compare claim-only classifiers with a KG-evidence baseline on FACTKG, using graph evidence that is retrieved from claim entities. Results generally favor graph evidence, with differences across reasoning types and cross-style settings.
4.1 Dataset Statistics
- The train, development, and test splits use an 8:1:1 proportion, with disjoint triple sets across splits.Source: §4.1 · p. 6
4.2 Experimental Setting
- FACTKG is published with claims, graph evidence, and labels; the test set provides entities alone as graph evidence.Evidence includes entities and relation sequences; the example links AIDAstella and Papenburg through shipBuilder and location relations.Source: §4.2 · p. 6
4.3.1 Claim Only
- Claim-only baselines classify claims without graph input: BERT and BlueBERT are fine-tuned, while Flan-T5 is used zero-shot via True/False token probabilities.BlueBERT provides a comparison with a model not pretrained on Wikipedia.Source: §4.3.1 · pp. 6–7
4.3.2 With Graphical Evidence
- The KG-evidence baseline adapts GEAR with relation and hop classifiers to retrieve paths, then converts triples into text for claim verification.The classifiers predict relations and a maximum hop count; the retrieved paths may include supporting or counterfactual evidence.Source: §4.3.2 · p. 7
4.4 Results
- GEAR outperforms other baselines in most reasoning types, especially Existence and Negation, where graph evidence is particularly useful.Existence claims contain less information, while Negation claims require additional inference steps.Source: Table 3 · p. 8, §4.4 · p. 8
- GEAR’s accuracy is lower than BERT’s on Multi-hop claims, possibly because complex retrieval with under-specified intermediate nodes can return incorrect graphs.The authors suggest masked language modeling may help BERT handle unspecified entities in these claims.Source: §4.4 · p. 8
- In the Claim Only setting, all baselines exceed the 51.35% Majority Class baseline, and BERT performs best.The authors attribute BERT’s advantage over BlueBERT to its Wikipedia pretraining.Source: §4.4 · p. 8
- In C→W evaluation, BERT and BlueBERT improve over C→C, while GEAR’s score drops only slightly, suggesting varied colloquial forms may benefit generalization.Table 4 reports cross-style evaluation; Flan-T5 is excluded because it is used only zero-shot.Source: Table 4 · p. 8, §4.4 · p. 8
5 Conclusion
FACTKG is a dataset for fact verification using knowledge graphs, with claims corresponding to graph patterns and colloquial-style claims. Its graph structure can support explanation and the design of more complex multi-hop reasoning claims.
5 Conclusion
- FACTKG is a dataset for fact verification using knowledge graphs, with claims generated to correspond to graph patterns.The conclusion also says its analysis found the claims difficult to solve without reasoning over the knowledge graph.Source: §5 · p. 8
- The knowledge graph can serve as an explanation for a verdict, including in settings such as medical question answering where explainability matters.The passage presents justification prediction as one possible research direction because many text-justification approaches lack gold references.Source: §5 · pp. 8–9
- Using KG structure for claim generation allows more complex multi-hop reasoning to be built in without relying on annotator creativity.Source: §5 · p. 9
Limitations
FACTKG uses WebNLG derived from the 2015-10 DBpedia version and constructs claims from single sentences.
Limitations
- FACTKG does not reflect the latest knowledge because WebNLG derives from the 2015-10 version of DBpedia.Source: Limitations · p. 9
- Claims are constructed from single sentences, and generating claims from multiple sentences is left as future work.Source: Limitations · p. 9
A Qualitative analysis
This section describes qualitative graph evidence and points to tables of relation and claim-generation templates.
A Qualitative analysis
- The qualitative analysis reports claims and retrieved graphical evidence in Table A, with GEAR prediction correctness shown in the Result column.The authors describe using subgraph retrieval to visualize a graph path and say the retrieved evidence makes verdict reasoning interpretable.Source: A Qualitative analysis · p. 12
B Relation Substitution
- Table 6 lists four groups of compatible relations for relation substitution.Source: B Relation Substitution · p. 12
C.1 Existence
- Table 7 describes the templates used to generate existence claims.Source: §C.1 · p. 12
C.2 Factive and Non Factive Presupposition
- Table 8 contains factive and non-factive presupposition templates.Source: §C.2 · p. 12
C.3 Structural Presupposition
- Table 9 contains structural presupposition templates.Source: §C.3 · p. 12
D.1 Conjunction
For REFUTED claims, adding negation can change the label depending on its position relative to entity substitution.
D.1 Conjunction
- Adding negation to every part of a REFUTED claim with substituted entities makes it SUPPORTED.Source: §D.1 · p. 12
- Other negation placements preserve the REFUTED label when they are unrelated to entity substitution.Source: §D.1 · p. 12
D.2 Multi-hop
A claim’s truth depends on a distinctive path matching its intent. The example checks for a builder connection to AIDAstella and no location connection to New York.
D.2 Multi-hop
- The example verifies intent by requiring a builder link to AIDAstella and no location link to New York.The passage says the claim’s truth depends on the presence of a distinctive path that matches its intent.Source: §D.2 · p. 12
E Colloquial Style Claim Survey
A survey assessed information loss in colloquial-style claims, while the section also recalls GEAR’s claim-verification components and evidence-processing steps.
E Colloquial Style Claim Survey
- 9.8% of colloquial claims were judged to lose important information by at least two reviewers in a survey of 9 graduate students.Reviewers used five labels, ranging from all facts preserved to loss of important information.Source: E Colloquial Style Claim Survey · p. 12
F Details of GEAR
- GEAR encodes claims and evidence, builds a fully connected evidence graph, propagates information with ERNet, and aggregates evidence for its final prediction.This section recalls these details of GEAR (Zhou et al., 2019).Source: F Details of GEAR · p. 12
Sentence Encoder
- GEAR uses BERT sentence representations, encoding each evidence-claim pair and the claim alone.The claim representation is obtained by feeding the claim into BERT alone.Source: Sentence Encoder · p. 12
Evidence Reasoning Network
- ERNet computes attention between neighboring evidence nodes, normalizes the coefficients, and uses them to combine neighbor features into updated node representations.The initial hidden state of each evidence node is its evidence representation.Source: Evidence Reasoning Network · pp. 12–13
Evidence Aggregator
- GEAR’s mean aggregator combines evidence-node hidden states into a final state, which a one-layer MLP uses to produce the prediction.The mean is computed elementwise across hidden states.Source: Evidence Aggregator · p. 13