Source-linked AI summary
Fine-grained Fact Verification with Kernel Graph Attention Network
Zhenghao Liu, Chenyan Xiong, Maosong Sun, Zhiyuan Liu
TL;DR
Fact verification must identify claims that are linguistically plausible but unsupported amid noisy retrieved evidence. KGAT combines kernel-based evidence selection and propagation in a graph, achieving a 70.38% FEVER score and outperforming prior approaches on FEVER. Its analyses associate effectiveness with focused attention on relevant evidence and meaningful clues.
Problem
Fact verification requires fine-grained reasoning because retrieved evidence is noisy and false claims can be semantically correct but unsupported.
Method
KGAT builds a claim–evidence graph and combines node kernels for evidence selection with edge kernels for fine-grained evidence propagation.
Results
70.38% FEVER score: KGAT significantly outperforms previous BERT- and GNN-based fact verification approaches on FEVER.
Takeaways & Limitations
Kernel-based attentions focus on correct evidence pieces and gather necessary information along reasoning chains, producing more intuitive attention patterns than dot-product attentions.
Abstract
from arXiv · showhide
Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims. This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions. Given a claim and a set of potential evidence sentences that form an evidence graph, KGAT introduces node kernels, which better measure the importance of the evidence node, and edge kernels, which conduct fine-grained evidence propagation in the graph, into Graph Attention Networks for more accurate fact verification. KGAT achieves a 70.38% FEVER score and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification. Our analyses illustrate that, compared to dot-product attentions, the kernel-based attention concentrates more on relevant evidence sentences and meaningful clues in the evidence graph, which is the main source of KGAT's effectiveness.
1 Introduction
Fact verification must distinguish subtly unsupported claims from true statements despite noisy retrieved evidence. KGAT addresses this challenge with kernel-based graph reasoning and achieves strong FEVER performance.
- Fact verification systems retrieve evidence, jointly reason over sentences, and aggregate signals to verify claim integrity.
- Retrieved evidence contains noise, while fabricated claims can be semantically correct but unsupported, requiring fine-grained reasoning.
- KGAT constructs an evidence graph and uses edge kernels for clue propagation and node kernels for claim-relevant evidence selection.
- 70.38% FEVER score: KGAT significantly outperforms previous BERT- and GNN-based fact verification approaches.
- Kernel-based attentions concentrate on relevant evidence sentences and meaningful clues, identified as the main source of KGAT’s effectiveness.
2 Related Work
Related work frames FEVER as evidence-grounded claim verification and develops retrieval, NLI, graph reasoning, and neural matching approaches. KGAT builds on graph-based reasoning and interaction-based matching kernels.
- FEVER evaluates automatic verification of human-generated claims using evidence extracted from Wikipedia.
- Existing systems commonly use document retrieval, sentence retrieval, and claim verification, with many models emphasizing claim verification.
- Graph-based models establish interactions among claim and evidence nodes for joint reasoning over multiple evidence pieces.
- NLI methods classify premise–hypothesis relationships as entailment, contradiction, or neutral, while FEVER additionally requires finding evidence pieces.
- Interaction-based neural retrieval models use matching kernels to summarize word or phrase interactions in learned embedding spaces.
3 Kernel Graph Attention Network
KGAT represents claim–evidence pairs as a fully connected graph and combines kernel-based edge propagation with node-based evidence selection. The resulting graph model predicts claim labels from jointly reasoned and relevance-weighted node signals.
- Evidence Graph: KGAT constructs a fully connected evidence graph whose nodes are claim–evidence pairs and whose edges connect all node pairs.
- Reasoning with Evidence Graph: The model decomposes prediction into graph-conditioned node label probabilities and evidence selection probabilities, then combines them through readout.
- Initial Node Representations: Node representations are initialized from BERT hidden states for concatenated claim, Wikipedia title, and evidence-sentence sequences.
- Edge Kernel for Evidence Propagation: Edge kernels use token-level and sentence-level attentions to propagate fine-grained information between graph nodes and produce per-node claim predictions.
- Edge Kernel for Evidence Propagation: The token-level mechanism computes cosine-similarity translation matrices, applies Gaussian kernels for multi-level interactions, and aggregates token representations for propagation.
- Node Kernel for Evidence Aggregation: Node kernels match each evidence node with the claim to estimate evidence importance and weight node-level predictions during graph readout.
- Training: The model is trained end-to-end by minimizing cross-entropy loss using the ground-truth verification label.
4 Experimental Methodology
The experiments evaluate fact verification on FEVER using established baselines, retrieval variants, and implementation settings. They report claim accuracy, FEVER-based metrics, and evidence-retrieval quality.
- Dataset: The experiments use FEVER, containing 185,455 annotated claims and 5,416,537 Wikipedia documents, labeled SUPPORTS, REFUTES, or NOT ENOUGH INFO.
- Evaluation Metrics: Evaluation reports Label Accuracy and FEVER score, with FEVER score additionally requiring a complete set of golden evidence.
- Evaluation Metrics: Golden FEVER evaluates the easier setting with golden evidence supplied, while Precision, Recall, and F1 assess evidence-sentence retrieval.
- Baselines: Baselines include top FEVER 1.0 systems, BERT-pair, BERT-concat, GEAR, and models using BERT sentence retrieval.
- Implementation Details: Sentence retrieval compares ESIM-based and BERT-based models, with BERT using the [CLS] representation and a learning-to-rank layer.
- Implementation Details: Training uses batch size 4, gradient accumulation step 8, two epochs, maximum length 130, Adam with learning rate 5e-5, and kernel size 21.
5 Evaluation Result
KGAT outperforms the compared baselines across testing scenarios and shows particular advantages when verification requires combining multiple evidence pieces. Kernel-based attention improves evidence selection and propagation by concentrating on relevant nodes and tokens.
- Overall Performance: KGAT is best across all testing scenarios, outperforming graph attention baselines and maintaining gains with different sentence retrieval and encoder configurations.With BERT-based retrieval, KGAT exceeds BERT (Base) by almost 1% FEVER score; KGAT with RoBERTa performs best across previously published results.
- Overall Performance: BERT sentence retrieval significantly outperforms ESIM retrieval, but experiments use ESIM retrieval for fair comparison with GEAR.The evaluation therefore separates retrieval quality from reasoning-model comparisons.
- Performance on Different Scenarios: KGAT-Node improves over GAT by more than 0.3% on both single- and multiple-evidence scenarios, highlighting the role of evidence selection.Its benefit is limited on GFEVER, where golden evidence is provided and node selection is unnecessary.
- Performance on Different Scenarios: KGAT-Edge improves over GAT by more than 0.8% on multiple-evidence scenarios and 0.1% on single-evidence scenarios, mainly by combining multiple evidence pieces.The full scenario analysis associates edge kernels with evidence interaction and information propagation.
- Effectiveness of Kernel in KGAT: KGAT produces more concentrated edge and node attention than GAT, with lower entropy and attention focused on fewer tokens and evidence sentences.The analysis links edge kernels to focused token-level propagation and node kernels to concentrated evidence selection.
- Effectiveness of Kernel in KGAT: KGAT’s first-ranked evidence sentence covers nearly 80% of ground-truth evidence, demonstrating focused evidence selection.Figure 4 evaluates recall as the ranking depth increases, while Figure 5 examines the top 10% of token attention weights.
6 Case Study
The case study compares KGAT’s edge-kernel attention with GAT’s dot-product attention on multi-step evidence reasoning. KGAT focuses on the missing clue and relevant entity, whereas GAT’s attention is diffuse.
- KGAT’s edge kernels identify “rhythm guitarist” in the second evidence as the additional information needed to complete the reasoning chain.
- KGAT also assigns more attention to “Al Jardine,” helping check that the second evidence concerns the correct person.
- GAT’s dot-product attention is nearly uniform across tokens, making its joint reasoning difficult to explain.
7 Conclusion
KGAT uses kernel-based attentions in graph neural networks for evidence selection and fine-grained joint reasoning. Its node kernels focus on correct evidence pieces, while edge kernels gather information across nodes to complete reasoning chains.
- KGAT uses kernels in graph neural networks for more accurate evidence selection and fine-grained joint reasoning.
- Figure 6 visualizes edge attention weights on evidence tokens, with darker red indicating higher attention weights.
- Node kernels focus attention on the correct evidence pieces, while edge kernels gather necessary information between nodes to complete reasoning chains.
- Kernel-based attentions produce intuitive and effective patterns, unlike the scattered and difficult-to-explain dot-product attentions.