Source-linked AI summary
Reasoning Over Semantic-Level Graph for Fact Checking
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin
TL;DR
Fact checking requires reasoning over multiple evidence sentences, but common representations miss their rich semantic structure. This paper constructs semantic role labeling graphs and uses graph-based mechanisms with XLNet, achieving state-of-the-art FEVER performance while still encountering errors from misleading retrieved evidence.
Problem
Fact checking requires reasoning about multiple retrieved evidence sentences, while concatenation and isolated-feature fusion fail to capture their rich semantic-level structure.
Method
The approach constructs evidence graphs with semantic role labeling, redefines XLNet’s relative word distances using graph structure, and applies graph neural networks to aggregate neighboring-node information.
Results
76.85% label accuracy on the FEVER test set, and the model achieved state-of-the-art performance on both official evaluation metrics.
Takeaways & Limitations
Both graph-based mechanisms improve accuracy, supporting the usefulness of explicit semantic structure for fact checking.
Takeaways & Limitations
Misleading information in retrieved evidence can cause incorrect judgments, and comparisons with GEAR and KGAT are confounded by differences in other components.
Abstract
from arXiv · showhide
Fact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.
1 Introduction
The paper frames fact checking as assessing a claim’s truthfulness from textual evidence, while noting that existing approaches do not capture rich semantic structure across multiple evidence sentences. It proposes graph-based reasoning that uses semantic role labeling and graph-driven representations, achieving state-of-the-art FEVER performance.
- False news can influence public opinions, stock prices, and presidential elections, motivating automatic assessment of textual claims using textual evidence.
- Existing systems typically concatenate evidence sentences or fuse features from isolated sentences, but these approaches fail to capture rich semantic-level structure among multiple evidence sentences.
- The proposed approach applies semantic role labeling to construct evidence graphs and uses graph-driven representation learning mechanisms for fact checking.
- The system achieves state-of-the-art performance on FEVER in both claim classification accuracy and FEVER score.
2 Task Definition and Pipeline
FEVER fact checking retrieves evidence from Wikipedia and predicts whether a claim is supported, refuted, or unverifiable. The paper’s pipeline adds graph-based reasoning to the standard retrieval, evidence-selection, and claim-verification workflow.
- Task Definition: FEVER requires systems to retrieve Wikipedia evidence and classify claims as SUPPORTED, REFUTED, or NOT ENOUGH INFO.
- Pipeline: The pipeline retrieves related Wikipedia documents, selects top-k evidence sentences, and passes the claim with those sentences to a verification model.
- Graph-Based Reasoning: The paper’s main contribution is a graph-based reasoning approach for claim verification.
- Graph Construction: Semantic role labeling extracts tuples from evidence, with graph edges connecting arguments within tuples and across different tuples.
3 Graph-Based Reasoning Approach
The approach represents claims and retrieved evidence as semantic graphs, then uses graph-aware contextual representations, graph convolution, and graph attention for claim verification.
- Graph-Based Reasoning Approach: The system predicts claim truthfulness from retrieved evidence using a graph-based reasoning approach for three-way FEVER classification.The classes are SUPPORTED, REFUTED, and NOT ENOUGH INFO.
- Graph Construction: The graph construction module is presented as a practical approach that could be further developed.The paper notes that multiple graph-construction strategies are possible and adopts SRL as a flexible implementation.
- Graph Construction: Semantic role labeling parses evidence into tuples whose typed elements become graph nodes, with within-tuple and cross-tuple edges capturing structure.Node types include verbs, arguments, locations, and temporal elements; cross-tuple edges connect literally similar nodes.
- Contextual Word Representations with Graph Distance: Graph-based distances shorten relationships between semantically related words before XLNet produces contextual representations.The method reorders evidence according to graph-based relative positions and feeds the resulting sequence into XLNet.
- Graph Convolutional Network: Word-span node representations are averaged from contextual word representations, then updated by multi-layer graph convolution over neighboring nodes.The GCN uses normalized adjacency and stacked layers to incorporate multi-hop neighborhood information.
- Graph Attention Network: Graph attention aligns claim-graph and evidence-graph node representations before combining graph information with XLNet’s joint claim-evidence representation for prediction.The attention mechanism forms a claim-centric evidence representation and the final prediction uses it with h([CLS]).
4 Document Retrieval and Evidence Selection
The pipeline retrieves relevant Wikipedia documents, selects evidence sentences by semantic matching with pretrained representations, and passes the top five candidates onward.
- Document Retrieval: The document retrieval model takes a claim and Wikipedia documents as input and returns the most relevant documents.It filters candidates with keyword matching, handles disambiguation titles with NSMN, and selects the top 10 documents.
- Evidence Selection: The evidence selection model ranks candidate evidence sentences by semantic relevance to the claim using contextual representations from pretrained models.XLNet or RoBERTa representations are used to measure claim-evidence relevance.
- Evidence Selection: The selector extracts the [CLS] representation for each claim-evidence pair, scores candidates with an MLP and softmax, and ranks them by score.The model is trained with standard cross-entropy loss.
- Evidence Selection: The system selects the top 5 evidence sentences following the official FEVER setting.These selected sentences are used for subsequent claim verification.
5 Experiments
The experiments evaluate DREAM on FEVER against established systems and through ablations. DREAM achieves strong benchmark performance, while both graph-based mechanisms contribute to accuracy; error analysis identifies semantic-matching and misleading-evidence failures.
- Model Comparison: DREAM achieves 76.85% label accuracy on the FEVER blind test set and state-of-the-art performance on both official evaluation metrics.The comparison includes top-performing shared-task systems, GEAR, and KGAT.
- Model Comparison: DREAM uses explicit SRL-derived graph structure, whereas GEAR and KGAT represent evidence sentences as graph nodes and learn implicit structure.The paper cautions that differences in sentence selection and pretrained models prevent attributing superiority solely to explicit graph structure.
- Ablation Study: Combining both graph-based modules improves label accuracy by 3.76% over the XLNet baseline on the development set.The ablation removes graph-based distance and graph reasoning separately to assess their contributions.
- Ablation Study: Removing graph-based distance lowers label accuracy by 0.81%, while removing graph-based reasoning lowers it by 2.04%.The paper associates these drops with shortened distances between linked nodes and structural information for deeper reasoning, respectively.
- Error Analysis: Error analysis finds failures from unmatched semantic meanings and misleading retrieved evidence.Examples include treating a novel as unrelated to a book and choosing a misleading movie description over the ground-truth television-series evidence.
6 Related Work
Related work commonly treats fact checking as claim verification over retrieved evidence, using concatenation, pairwise aggregation, or sentence-level graph nodes. This paper instead focuses on semantic structure within evidence and differs from prior graph-based work in graph construction.
- Task Scope: The paper focuses on textual claims and evaluates claim verification with evidence from Wikipedia in the FEVER setting.FEVER labels claims as SUPPORTED, REFUTED, or NOT ENOUGH INFO and uses three-way accuracy and FEVER score.
- Existing Fact-Checking Pipelines: FEVER systems commonly use document selection, evidence sentence selection, and claim verification in a three-stage pipeline.Document selection often queries Wikipedia with named entities extracted from the claim.
- Evidence Aggregation: Prior systems aggregate evidence by concatenating sentences, combining separately classified claim-evidence pairs, or pooling pairwise representations.These approaches reflect several common strategies used by top FEVER systems.
- Graph-Based Related Work: GEAR uses BERT-based claim-specific sentence representations followed by a graph network whose nodes are evidence sentences.The present work distinguishes itself by constructing graphs through semantic understanding of evidence.
7 Conclusion
The paper presents a graph-based fact-checking approach that represents multiple evidence sentences through semantic-role-derived graphs. Two graph-based modules improve the system, which achieves state-of-the-art performance on the public leaderboard at submission time.
- The approach constructs an evidence graph automatically using semantic role labeling when assessing claims against multiple evidence sentences.
- A graph-based distance module computes contextual word embeddings in XLNet, while another learns graph-component representations for reasoning.
- Both graph-based modules improve performance in experiments.
- The final system was state-of-the-art on the public leaderboard when the paper was submitted.
- Evidence selection remains separate from joint learning with claim verification, which the authors leave for future work.
A Typology Sort Algorithm
The typology-sort procedure prepares graph nodes for distance calculation by operating on an acyclic graph and recursively traversing child nodes.
- Algorithm 1 takes a node sequence S and relation set R as inputs.
- The procedure recursively visits eligible child nodes with depth-first search before appending the current node to the sorted sequence.
- Before traversal, the node and relation structures are changed to an acyclic graph.
B FEVER
FEVER reports class split statistics and evaluates fact-checking systems with a ground-truth label and evidence-based correctness criterion.
- Table 3 reports the split sizes for SUPPORTED, REFUTED, and NOT ENOUGH INFO (NEI) classes in FEVER.
- FEVER score combines the predicted label with correctness of the predicted evidence relative to ground-truth evidence.
- The FEVER score equation uses y and ŷ for ground-truth and predicted labels, and E and Ê for ground-truth and predicted evidence sets.
C Evidence Selection Results
The evidence-selection module ranks sentences using the claim and each evidence sentence, and compares RoBERTa with XLNet. RoBERTa performs slightly better and is used for leaderboard submission.
- The evidence-selection model concatenates the claim with each evidence sentence and uses the last hidden vector to score evidence for ranking.
- The experiments compare RoBERTa and XLNet as backbones for evidence selection.
- RoBERTa performs slightly better than XLNet for evidence selection.
- The submitted system uses RoBERTa as its evidence-selection model.
D Training Details
The experiments use standard optimization components and separate training configurations for evidence selection and claim verification. The claim verification pipeline trains XLNet before freezing it to train the graph-based reasoning network.
- Training uses cross-entropy loss with AdamW optimization.
- Evidence selection: The evidence selection model uses a 1e-5 learning rate, batch size 8, and maximum sequence length 128.
- Claim verification: The claim verification model trains XLNet and the graph-based reasoning network separately, freezing XLNet before graph-based training.
- Claim verification: Claim verification uses a 2e-6 learning rate, batch size 6, maximum sequence length 256, and 100-dimensional node representations.