PaperlayerView exact revision on arXiv

Source-linked AI summary

GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification

Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, Maosong Sun

arXiv:1908.01843v1cs.CLcs.AIcs.LGstat.ML

TL;DR

Fact verification systems must retrieve and reason over multiple evidence pieces, while earlier approaches often combine evidence too simply. GEAR propagates information through a fully-connected evidence graph, aggregates multi-evidence information, and uses BERT for semantic encoding. On FEVER, it outperforms recent baselines and is reported to better leverage and reason over multi-evidence information.

  • Problem

    Fact verification requires retrieving relevant plain-text evidence, but many claims need joint reasoning over multiple pieces that simple evidence-combination methods do not capture.

  • Method

    GEAR uses BERT, a fully-connected evidence graph with an evidence reasoning network, and an evidence aggregator for claim verification.

  • Results

    GEAR outperforms recent state-of-the-art baseline systems on FEVER and achieves significant improvements over the BERT-Concat baseline.

  • Takeaways & Limitations

    GEAR could better leverage multi-evidence information and reason over evidence for fact verification.

  • Takeaways & Limitations

    The framework's main errors arise from upstream document retrieval and sentence selection failing to extract sufficient evidence for inference.

Abstract

from arXiv · show

Fact verification (FV) is a challenging task which requires to retrieve relevant evidence from plain text and use the evidence to verify given claims. Many claims require to simultaneously integrate and reason over several pieces of evidence for verification. However, previous work employs simple models to extract information from evidence without letting evidence communicate with each other, e.g., merely concatenate the evidence for processing. Therefore, these methods are unable to grasp sufficient relational and logical information among the evidence. To alleviate this issue, we propose a graph-based evidence aggregating and reasoning (GEAR) framework which enables information to transfer on a fully-connected evidence graph and then utilizes different aggregators to collect multi-evidence information. We further employ BERT, an effective pre-trained language representation model, to improve the performance. Experimental results on a large-scale benchmark dataset FEVER have demonstrated that GEAR could leverage multi-evidence information for FV and thus achieves the promising result with a test FEVER score of 67.10%. Our code is available at https://github.com/thunlp/GEAR.

1 Introduction

Fact verification requires retrieving evidence and, for many claims, jointly reasoning over multiple evidence pieces. GEAR addresses this need with graph-based evidence propagation, aggregation, and BERT-enhanced claim verification.

  • Fact verification labels claims as SUPPORTED, REFUTED, or NOT ENOUGH INFO using evidence retrieved from plain text.
  • Existing methods often concatenate evidence or process evidence-claim pairs separately, missing relational and logical information across evidence.
  • Many claims can be verified only by simultaneously integrating and reasoning over multiple evidence sentences.
  • GEAR builds a fully-connected evidence graph to propagate information, aggregates evidence, and classifies claims as supported, refuted, or insufficient.
  • BERT is used to better represent the semantics of both evidence and claims.
  • GEAR outperforms recent state-of-the-art baseline systems on FEVER, while case analysis indicates stronger use and reasoning of multi-evidence information.

2 Related Work

Related work frames FEVER as an NLI-based fact-verification problem and describes systems using concatenation, pairwise aggregation, enhanced inference models, and pretrained language models. GEAR's pipeline is situated within this broader FEVER and NLI research landscape.

  • The FEVER shared task evaluates systems that verify human-generated claims by extracting evidence from Wikipedia using a blind test set.
  • Earlier FV systems concatenate all evidence and apply an NLI model, or predict each claim-evidence pair individually before aggregation.
  • Subsequent systems use ESIM, GPT fine-tuning, NSMN, or jointly trained evidence identification and claim verification modules.
  • NLI models classify premise-hypothesis relationships as entailment, contradiction, or neutral, supported by datasets such as SNLI and Multi-NLI.
  • BERT uses bidirectional transformers and pretraining tasks, and the paper reports that fine-tuned BERT outperforms other NLI-based models on FEVER claim verification.
  • GEAR places graph-based claim verification after document retrieval and sentence selection in a three-step pipeline.

3 Method

The method uses a three-step pipeline for fact verification, with GEAR performing graph-based reasoning over retrieved evidence during claim verification. It encodes evidence and claims with BERT, propagates information across a fully connected evidence graph, aggregates node states, and predicts the claim label.

  • 3.1 Document Retrieval and Sentence Selection: The pipeline performs document retrieval, sentence selection, and claim verification, using GEAR in the final stage.A threshold filter removes noisy evidence after sentence selection.
  • 3.3 Evidence Reasoning Network: GEAR builds a fully connected evidence graph with self-loops and uses ERNet to propagate information among evidence nodes.Each node represents one evidence piece, and the initial hidden state is its evidence representation.
  • 3.2 Sentence Encoder: BERT encodes each evidence-claim pair and the claim alone to produce evidence and claim representations.The evidence representation includes claim information to guide message passing.
  • 3.3 Evidence Reasoning Network: ERNet computes attention coefficients between neighboring nodes, normalizes them with softmax, and combines neighbor features to update each node.Stacked layers allow evidence nodes to communicate before aggregation.
  • 3.4 Evidence Aggregator: The evidence aggregator collects final node states using attention, max, or mean aggregation before a one-layer MLP predicts the claim label.The attention aggregator uses the claim representation to attend to evidence states.

4 Experimental Settings

Experiments use the FEVER benchmark and compare GEAR with top shared-task systems and BERT baselines using different evidence-combination strategies. Evaluation includes retrieval, selection, verification, and implementation settings for both baselines and GEAR.

  • 4.1 Dataset: FEVER contains 185,455 annotated claims and 5,416,537 Wikipedia documents from the June 2017 dump.The experiments follow the FEVER Shared Task partition.
  • 4.2 Baselines: The baselines include the top three FEVER shared-task systems and two BERT systems using concatenated evidence or independent evidence-claim pairs.The BERT baselines share the proposed retrieval and sentence-selection components.
  • 4.2 Baselines: BERT-Concat encodes all evidence as one string, whereas BERT-Pair encodes each evidence-claim pair independently and aggregates the predictions.Training and test evidence-selection procedures differ between the two systems.
  • 4.5 Implementation Details: GEAR uses BERTBASE with learning rate 2e-5, while ERNet uses 768 features, weight dimension 64, Adam optimization, and 0-3 stacked layers.Dev-set GEAR scores are means over 10 random-seed runs.
  • 4.4 Evaluation Metrics: Evaluation reports label accuracy, F1, FEVER score, and OFEVER score.FEVER score conditions label accuracy on providing a complete evidence set, while OFEVER is an upper bound assuming perfect downstream systems.

5 Experimental Results and Analysis

The experiments evaluate retrieval and sentence selection before analyzing GEAR across thresholds and evidence-reasoning settings. Filtering creates a trade-off between retrieval-oriented metrics and precision-oriented metrics, while the study also examines multi-evidence reasoning on a difficult subset.

  • 5 Experimental Results and Analysis: The experiments cover retrieval, sentence selection, GEAR comparisons, threshold effects, sentence embeddings, and a difficult multi-evidence subset.The difficult subset targets claims requiring reasoning over multiple evidence pieces.
  • 5.1 Document Retrieval and Sentence Selection: The proposed retrieval model obtains a slightly lower OFEVER score than the reproduced Athene model, possibly because of random factors.This comparison concerns document retrieval on the development set.
  • 5.1 Document Retrieval and Sentence Selection: Increasing the sentence-selection threshold gradually lowers recall and OFEVER score while increasing precision and F1 score.Filtering noisy evidence improves precision and F1 but reduces the amount of retrieved evidence.

Model Evaluation

GEAR improves claim-verification accuracy over BERT-Concat and is evaluated with sentence-selection thresholds and evidence aggregators.

  • 1.17%: The final GEAR model outperforms the best BERT-Concat baseline.
  • Table 5 compares label accuracy on the difficult dev set across ERNet layers and evidence aggregators.
  • Label accuracy decreases from τ = 10^-3 to 10^-1 because informative evidence is filtered out.

Effect of Sentence Embedding

The evaluation examines BERT fine-tuning and multi-evidence reasoning on a difficult dev subset containing claims that require multiple evidence pieces.

  • Effect of Sentence Embedding: Fine-tuning BERT, rather than using sentence embeddings alone, is necessary for effective claim verification.Without fine-tuning, final dev label accuracy is close to random guessing.
  • Effect of Sentence Embedding: Table 7 evaluates the full pipeline using the model selected by dev FEVER score.
  • Effect of Sentence Embedding: 7870 samples: The difficult subset contains more than 39% of the dev set.It filters single-evidence SUPPORTED and REFUTED claims and includes all NEI claims.
  • Effect of Sentence Embedding: ERNet models outperform models without ERNet on the difficult subset, with a minimum improvement of 1.27%.Models with 2 ERNet layers achieve the best results, indicating multi-step evidence propagation is useful for these claims.

Error Analysis

Error analysis attributes the main performance losses to upstream retrieval and sentence selection, while oracle evidence raises the framework’s theoretical upper bound.

  • Error Analysis: Upstream document retrieval and sentence selection are the main sources of errors because they may fail to extract sufficient evidence.The claim-verification component cannot infer correctly when required evidence is missing.
  • Error Analysis: Table 8 presents a case requiring multiple evidence pieces, while Figure 2 visualizes attention weights across evidence nodes and the attention aggregator.
  • Error Analysis: Adding ground-truth evidence increases all Table 6 scores by more than 1.4% over the original dev-set label accuracy.The evidence-enhanced set includes ground-truth and retrieved evidence before threshold filtering.
  • Error Analysis: The evidence-enhanced results represent the theoretical upper-bound label accuracy under oracle upstream components.
  • Error Analysis: Better evidence retrieval remains future work because current retrieval models cannot solve the task’s evidence-retrieval challenges.The paper contrasts its improvement from adding ground-truth evidence with a 0.08% improvement from a prior two-hop method.

5.3 Full Pipeline

The full pipeline achieves a test FEVER score of 67.10% and significantly outperforms the BERT-Concat baseline. Its final configuration uses one ERNet layer, an attention aggregator, and a 10^-3 sentence-filter threshold.

  • 67.10% test FEVER score is achieved by the final full pipeline.The model uses one ERNet layer, the attention aggregator, and a sentence-filter threshold of 10^-3.
  • 1.46% improvement over the BERT-Concat baseline is obtained by the full pipeline.The improvement is reported as significant.
  • Nearly 1% separates BERT fine-tuning systems from other shared task models on test FEVER score.

5.4 Case study

The case study shows that verifying one claim requires integrating two complementary pieces of evidence. The model’s attention concentrates on those useful evidence nodes while assigning low weights elsewhere.

  • Two evidence pieces are needed to verify that Al Jardine is an American rhythm guitarist.One states that he is best known as the band’s rhythm guitarist, while another identifies him as an American musician.
  • The ground-truth evidence sentences rank in the top two of the retrieved evidence set.
  • All evidence nodes attend mainly to the first and second nodes, which provide the most useful information.Attention weights on the other nodes are pretty low, indicating selection of useful information from multiple evidence pieces.

6 Conclusion

GEAR combines BERT encoding, graph-based evidence propagation through ERNet, and evidence aggregation for claim verification on FEVER. The final pipeline achieves significant improvements, while future work targets multi-step extraction and external knowledge.

  • GEAR combines BERT, ERNet, and an evidence aggregator to encode, propagate, and aggregate multiple evidence pieces.
  • The final GEAR pipeline achieves significant improvements on the FEVER claim verification subtask.
  • Future work will design a multi-step evidence extractor and incorporate external knowledge.
Loading 1908.01843v1…