Source-linked AI summary
HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, Mohit Bansal
TL;DR
Existing fact-verification resources provide limited support for claims requiring multiple sources and reasoning hops. HoVer introduces a many-hop dataset and evaluation task built around evidence from up to four Wikipedia articles, and state-of-the-art systems degrade as hops increase. The resulting benchmark exposes the difficulty of jointly retrieving complete evidence and verifying claims.
Problem
FEVER is mostly single-hop, while existing multi-hop datasets are limited in reasoning steps and semantic overlap, motivating claims that require evidence from several Wikipedia articles.
Method
HoVer constructs human-written claims through validation, multi-hop extension, mutation, and evidence-based labeling, with claims requiring evidence from up to four Wikipedia articles.
Results
14.9% of dev examples received complete evidence retrieval and correct verification from the best pipeline, while existing state-of-the-art models degrade as reasoning hops increase.
Takeaways & Limitations
HoVer provides a challenging benchmark intended to encourage models capable of complex many-hop reasoning for information retrieval and verification.
Takeaways & Limitations
The distinction between REFUTED and NOTENOUGHINFO is subjective, so HoVer merges them into NOT-SUPPORTED.
Abstract
from arXiv · showhide
We introduce HoVer (HOppy VERification), a dataset for many-hop evidence extraction and fact verification. It challenges models to extract facts from several Wikipedia articles that are relevant to a claim and classify whether the claim is Supported or Not-Supported by the facts. In HoVer, the claims require evidence to be extracted from as many as four English Wikipedia articles and embody reasoning graphs of diverse shapes. Moreover, most of the 3/4-hop claims are written in multiple sentences, which adds to the complexity of understanding long-range dependency relations such as coreference. We show that the performance of an existing state-of-the-art semantic-matching model degrades significantly on our dataset as the number of reasoning hops increases, hence demonstrating the necessity of many-hop reasoning to achieve strong results. We hope that the introduction of this challenging dataset and the accompanying evaluation task will encourage research in many-hop fact retrieval and information verification. We make the HoVer dataset publicly available at https://hover-nlp.github.io
1 Introduction
HoVer addresses the limits of single-hop fact verification by introducing claims that require evidence from up to four Wikipedia articles and reduced semantic overlap with the claim. Experiments show that retrieval and verification performance degrades sharply as reasoning hops increase.
- FEVER is mostly single-hop, with more than 87% of claims requiring information from one Wikipedia article.
- HoVer claims require evidence from as many as four English Wikipedia articles and are designed to reduce shallow word-matching shortcuts.The dataset targets cases where some evidence cannot be retrieved precisely from the claim alone.
- The task predicts whether a claim is SUPPORTED or NOT-SUPPORTED, combining REFUTED and NOTENOUGHINFO because their distinction is ambiguous for many-hop claims.The evidence retrieval components are evaluated against document- and sentence-level ground-truth evidence.
- 80% of 2-hop, 39% of 3-hop, and 15% of 4-hop claims had all supporting documents recovered by top-100 bi-gram TF-IDF retrieval.The baseline combines TF-IDF document retrieval with BERT models for document retrieval, sentence selection, and claim verification.
- 14.9% of dev examples received complete evidence retrieval and correct verification from the best pipeline, versus 81% human performance.With all evidence supplied, the verification model reached 81.2% accuracy, compared with 73.7% when relying on retrieved evidence.
2 Data Collection
HoVer constructs human-written many-hop claims through claim creation, hop extension, mutation, and evidence-based labeling. The pipeline preserves evidence requirements while introducing diverse reasoning structures and controlled unsupported claims.
- Claim Creation: HoVer claims begin by having trained annotators rewrite HOTPOTQA question-answer pairs into claims requiring information from two Wikipedia articles.Annotators must preserve all information from the original question-answer pairs without adding new information.
- Claim Creation: Created claims are retained when at least two of three validators judge them valid and faithful to the original question-answer pair.Validated 2-hop claims are automatically labeled SUPPORTED.
- Hop Extension: Entity substitution extends 2-hop claims to 3 hops and then 4 hops by adding supporting documents, with Method 2 enabling attachment to a non-title entity.The two methods generate different reasoning graphs, including chains and star-shaped structures.
- Claim Mutation: Crowd-workers and automatic procedures mutate claims through specificity changes, word substitution, entity substitution, and negation.Supporting document titles and location entities are constrained in some mutations to preserve the intended evidence set and avoid external evidence.
- Claim Labeling: Annotators label original and mutated claims using supporting facts alone, and claims are retained when at least three of five workers agree.The original three labels are SUPPORTED, REFUTED, and NOTENOUGHINFO; REFUTED and NOTENOUGHINFO are merged into NOT-SUPPORTED.
- Claim Labeling: The REFUTED versus NOTENOUGHINFO distinction is subjective because evidence may contradict part of a claim without refuting the entire claim.The paper illustrates this boundary with a claim about Christian Bale, a 2010 film, and an American director.
3 Dataset Analysis
HoVer constructs increasingly complex claims requiring two to four supporting Wikipedia documents, with diverse reasoning graphs and longer, syntactically richer language. Its dataset scale is constrained by the sharply increasing annotation and quality-control demands of higher-hop claims.
- HoVer uses 12k examples from HOTPOTQA because annotation time, job complexity, and quality control increase drastically with the number of hops.
- 2-hop, 3-hop, and 4-hop claims average 19.0, 24.2, and 31.6 tokens, compared with 9.4 tokens in FEVER.
- Diverse Many-Hop Reasoning Graphs: Each reasoning-graph node is a unique evidence-containing document, while edges represent Wikipedia hyperlinks or comparisons between titles.
- Diverse Many-Hop Reasoning Graphs: HoVer includes three unique 4-hop reasoning graphs, created by appending a fourth node to an existing node in the 3-hop graph.
- Qualitative Analysis: Replacing bridge entities with relative clauses or phrases makes some 3/4-hop claims longer and structurally complex, although overly complicated claims are discarded during labeling.
4 Baseline System
The HoVer baseline combines lexical and neural retrieval with BERT-based sentence selection and claim verification. Its pipeline supplies progressively filtered documents and sentences to a final binary entailment classifier.
- The baseline follows a FEVER system with TF-IDF document retrieval and three BERT models for document retrieval, sentence selection, and claim verification.
- Rule-based Document Retrieval: The rule-based retriever returns k closest Wikipedia documents using cosine similarity between binned unigram and bigram TF-IDF vectors.
- Neural-based Document Retrieval: A BERT document retriever scores each TF-IDF candidate against the claim and selects documents above a relatedness threshold.
- Neural-based Sentence Selection: A second BERT model scores sentences from selected documents and retains the top sentences whose scores exceed a sentence threshold.
- Claim Verification Model: The verification model predicts binary entailment from the claim and retrieved evidence, separated by a [SEP] token, using the [CLS] representation.
5 Experiments and Results
Experiments show that retrieval and verification become substantially harder as the number of reasoning hops increases. Even with complete evidence, verification outperforms the full pipeline, while humans substantially exceed the best BERT systems.
- Document Retrieval Results: Neural document and sentence retrieval F1 scores also decrease significantly as the number of hops increases.Oracle settings provide upper bounds when the complete evidence is supplied.
- Claim Verification Results: 81.2% accuracy was achieved for claim verification with complete evidence, compared with 63.7% in the claim-only setting.The claim-only model exploits limited claim bias but remains 17.5% below the complete-evidence model.
- Full Pipeline Results: The best full pipeline retrieved complete evidence and predicted the correct label for 14.9% of dev examples and 15.32% of test examples.The HOVER Score requires at least one supporting fact from every supporting document and the correct label.
- Human Performance: Human performance exceeded the best BERT model in retrieval, oracle verification, and the full pipeline.On the dev set, humans achieved 88% accuracy and an 81% HOVER score, versus 67.6% accuracy and 14.9% HOVER for the best BERT model.
6 Related Work
HoVer builds on fact-verification, multi-hop reasoning, and synthetic reasoning datasets while addressing their limits in document count, shallow matching, and naturalness. It requires verification across multiple documents and expands language and reasoning diversity.
- Fact-verification datasets such as FEVER use single-sentence claims and evidence from at most two documents.
- HOVER requires claims to be verified from multiple documents, with accurate retrieval of relevant documents and their internal context.
- Multi-hop QA datasets generally challenge reasoning across only a limited number of documents or sentences.
- HOVER is larger and richer in language and reasoning paradigms than prior multi-hop fact-checking efforts.
- HOVER uses human-created examples governed by rules that enforce multi-hop structure, making them more natural than synthetic datasets such as RuleTaker.
7 Conclusion
HoVer is a fact extraction and verification dataset requiring evidence from up to four Wikipedia articles with diverse reasoning graphs. Existing state-of-the-art models degrade significantly as reasoning hops increase, highlighting the need for robust many-hop reasoning.
- HoVer requires evidence retrieval from as many as four Wikipedia articles forming reasoning graphs of diverse shapes.
- Existing state-of-the-art models degrade significantly as the number of reasoning hops increases on HoVer.
- The results demonstrate the necessity of robust many-hop reasoning for strong performance in information retrieval and verification.
A Experimental Setup
The experimental setup uses BERT-based models across document retrieval, sentence selection, and claim verification, alongside human evaluation of all three tasks. The pipeline and claims vary in length and hop count.
- The system fine-tunes a 110M-parameter BERT-base uncased model for document retrieval, sentence selection, and claim verification.
- Training uses batch size 16, learning rate 5e-5, and three epochs, with separate GPU configurations for retrieval, selection, and verification.
- The baseline pipeline has a four-stage architecture visualized in Figure 2.
- Claims in the dataset have different average token lengths for the 2-, 3-, and 4-hop settings.
- Human performance is measured on 100 sampled claims across document retrieval, sentence-level evidence selection, and oracle claim verification.
- Human evaluation allows web search for document retrieval, provides sentence-tokenized retrieved documents for evidence selection, and gives golden documents for oracle verification.
B Annotation Guidelines
The annotation guidelines construct clear, faithful, grammatical claims from question-answer pairs while preserving their information and avoiding unsupported or speculative additions. Claims may be true or mutated.
- A claim is a single- or multi-sentence statement containing true or mutated information about one or more entities.
- Annotators generate single-sentence claims from HOTPOTQA question-answer pairs.
- Claims should preserve as much information as possible from the original question and answer without adding or omitting content.
- Claims must be grammatically correct, formal, and use correct capitalization and spelling for entities.
- Claims must avoid speculative language such as “probably,” “might be,” and “maybe.”
- Some annotated claims may not be true.
B.1.2 Claim Validation
Claim validation uses multiple annotator judgments to check whether generated claims meet task criteria and are clear. The process also specifies constraints for entity substitution and coherent multi-sentence claims.
- Annotators assess whether each claim meets the criteria and rate question-answer clarity from 1 to 5.
- Three judgments are collected per claim, and claims are retained when at least two annotators validate them.
- Entity substitution replaces a given entity with information from one selected Wikipedia paragraph while preserving the rest of the claim.
- Rewritten claims must include the selected paragraph title when applicable, avoid the replaced entity, and use concise, fluent wording.
- The procedure prohibits fact-checking or using external knowledge during entity substitution.
- Claims may be split into multiple sentences, with pronouns and coreference used to connect entities across sentences.
B.2.1 Automatic Word Substitution using BERT
The paper generates claim mutations by masking non-entity words and using BERT to predict replacements, then creates implication variants and labels claims against supporting evidence. The labeling protocol distinguishes refuted claims from claims requiring additional information.
- Automatic Word Substitution using BERT: BERT-based mutation masks a sampled non-entity, non-stopword and retains replacements with cosine similarity between 0.7 and 0.8 and no common lemma.
- Negated claims must preserve the original claim’s key words, capitalization, and spelling without adding extra information.
- Specifically implied claims add information about existing target entities so that the mutated claim implies the original claim without reducing the number of hops.
- Generally implied claims delete or generalize target-entity information so that the original claim implies the mutated claim.
- Annotators classify claims as SUPPORTED, REFUTED, or NOTENOUGHINFO using supporting documents, while allowing linguistic knowledge and commonsense but excluding external encyclopedia knowledge.
- REFUTED applies when claim and evidence cannot both be true, whereas NOTENOUGHINFO applies when they are compatible or additional information is needed.