Source-linked AI summary
Data-Efficient Autoregressive Document Retrieval for Fact Verification
James Thorne
TL;DR
Document retrieval provides textual evidence for knowledge-intensive tasks, but autoregressive retrievers have required labeled examples linking queries to Wikipedia pages. This paper trains them with distant supervision from Wikipedia, then evaluates zero-shot use and task-specific fine-tuning. On fact-verification tasks, fine-tuning with fewer annotations can match or exceed fully supervised retrieval, though question answering remains challenging under domain shift.
Problem
Autoregressive retrieval predicts relevant Wikipedia titles but has required human-labeled examples linking inputs to documents, limiting its reliance on unlabeled data.
Source: §1 · p. 1Method
Results
With 16K FEVER annotations, DEARDR exceeds the fully supervised baseline on R-Precision and Recall@10; zero-shot it is 6.36% lower in R-Precision and 0.91% lower in Recall@10, with similar findings on HoVer.
Source: §1 · p. 2, §5.2 · p. 4Takeaways & Limitations
For the evaluated fact-verification tasks, distant supervision and pre-training support high-precision autoregressive retrieval with fewer annotated training data.
Source: §6 · p. 5Question-answering results were unsatisfactory in the reported applications, with domain shift identified as a limitation for adapting DEARDR to wider task ranges and multi-hop reasoning.
Source: §6 · p. 5, §5.2 · p. 5
Abstract
from arXiv · showhide
Document retrieval is a core component of many knowledge-intensive natural language processing task formulations such as fact verification and question answering. Sources of textual knowledge, such as Wikipedia articles, condition the generation of answers from the models. Recent advances in retrieval use sequence-to-sequence models to incrementally predict the title of the appropriate Wikipedia page given a query. However, this method requires supervision in the form of human annotation to label which Wikipedia pages contain appropriate context. This paper introduces a distant-supervision method that does not require any annotation to train autoregressive retrievers that attain competitive R-Precision and Recall in a zero-shot setting. Furthermore we show that with task-specific supervised fine-tuning, autoregressive retrieval performance for two Wikipedia-based fact verification tasks can approach or even exceed full supervision using less than $1/4$ of the annotated data indicating possible directions for data-efficient autoregressive retrieval.
1 Introduction
The paper introduces self-supervised Wikipedia training for autoregressive retrieval, avoiding human-annotated training data. On FEVER, task-specific fine-tuning with 16K annotations can exceed a fully supervised baseline, while zero-shot retrieval is close to GENRE.
1 Introduction
- Retrieval supplies passages from sources such as Wikipedia to condition answer generation in question answering and fact-verification tasks.The introduction describes this as a common component of knowledge-intensive NLP tasks.Source: §1 · p. 1
- Earlier autoregressive retrieval predicts Wikipedia document titles but requires labeled training examples linking inputs to titles.GENRE is described as improving retrieval-oriented NLP tasks in the KILT benchmark while using labeled data.Source: §1 · p. 1
- DEARDR trains an autoregressive Wikipedia retriever without human-annotated data and can be trained in less than 6 hours on a single GPU.Its recall is reported to exceed conventional methods such as TF-IDF and BM25.Source: §1 · p. 1
- With 16K FEVER annotations, DEARDR exceeds the fully supervised baseline on R-Precision and Recall@10; zero-shot it is 6.36% lower in R-Precision and 0.91% lower in Recall@10.The full FEVER dataset has 109K annotated instances; similar findings are reported for HoVer.Source: §1 · p. 2
2 Background
Retrieval approaches differ in how they represent and match queries with documents. Autoregressive retrieval instead generates a relevant document title, with constrained decoding keeping predictions within the indexed corpus.
2 Background
- Sparse retrieval scores query-document similarity using token frequencies without supervision, while dense neural retrieval learns query and passage encodings from supervised data.Sparse methods can have low recall when phrasing varies; negative-instance sampling affects dense retrieval suitability for downstream tasks.Source: §2 · p. 2
2.1 Autoregressive Document Retrieval
- Autoregressive retrieval uses a sequence-to-sequence model to predict a relevant document title, decoding tokens left to right.Constrained decoding excludes token sequences that do not occur in the index, and Wikipedia titles commonly serve as canonical entity or concept descriptors.Source: §2.1 · p. 2
- Related autoregressive work uses hyperlink information or data augmentation, while substring prediction avoids requiring unique document identifiers.The cited passage also notes that mGENRE’s multilingual entity-linking extension had not been evaluated for document retrieval.Source: §2.1 · p. 2
3 Data Efficient Document Retrieval
DEARDR uses Wikipedia structure as distant supervision to train autoregressive document retrieval without task-labeled instances. The approach predicts page titles, sentence hyperlinks, or both, and can be applied zero-shot or fine-tuned on task-specific data.
3 Data Efficient Document Retrieval
- DEARDR’s distant supervision uses Wikipedia page titles and sentence hyperlinks as targets for sampled sentences, separately or in combination.The training regimens are denoted PT, HL, and PTHL, respectively.Source: §3 · p. 2
- The authors hypothesize that predicting titles or hyperlinks approximates a component shared by retrieval-oriented tasks, supporting zero-shot retrieval without human annotations.The stated rationale is that this may support applying knowledge-intensive NLP tasks to new domains or languages.Source: §3 · pp. 2–3
- At test time, beam-search results are aggregated with constrained decoding, and DEARDR predicts sequences of all hyperlinked page names.This differs from GENRE, which predicts one entity per beam.Source: §3 · p. 3
- After Wikipedia pre-training, DEARDR can be fine-tuned with a small amount of task-specific data to address differences between pre-training and the target task.The authors hypothesize this reduces the number of instances needed to approach fully supervised performance.Source: §3 · p. 3
4 Experimental Setup
The experiments evaluate zero-shot and data-efficient retrieval on FEVER and HoVer using a June 2017 Wikipedia snapshot. Retrieval quality is measured with R-Precision and Recall@k, and comparisons include supervised, sparse, dense, and autoregressive systems.
4 Experimental Setup
- Experiments use June 2017 Wikipedia data to create 16.8M distant-supervision instances, evaluating document retrieval on FEVER and HoVer.The setup includes zero-shot pre-training and fine-tuning with a low number of labeled target-task instances.Source: §4 · p. 3
- Comparisons include a fully supervised target-task baseline, GENRE, BM25, TF-IDF, and DPR.Contrastive retrieval is not evaluated because it did not offer significant advantages over BM25 for FEVER.Source: §4 · p. 3
4.1 Evaluation
- The evaluation uses R-Precision and Recall@k; R-Precision measures precision among the top R results, and Recall@k measures the share of gold documents among the top k.For an instance with one valid document, R-Precision equals Precision@1.Source: §4.1 · p. 3
4.2 Implementation
- The implementation uses T5-base and selects hyperparameters by maximizing dev-set R-Precision; Table 1 compares page-title and hyperlink retrieval across PT, HL, PTHL, and supervised training.The decoding index is built from Wikipedia article-title subtokens.Source: §4.2 · p. 3, Table 1 · p. 4
5 Results and Discussion
DEARDR’s zero-shot retrieval is competitive on FEVER, and task-specific fine-tuning can match or exceed supervised retrieval with fewer labeled instances. Results also show hyperlink pre-training helps on multi-hop retrieval, while question answering and complex multi-hop reasoning remain less established.
5.1 Pre-training Intrinsic Evaluation
- Without hyperlink pre-training, recall is low, indicating that hyperlink pre-training may benefit multi-entity retrieval for some FEVER instances.The authors selected the model with the highest R-Precision on the FEVER shared task.Source: §5.1 · p. 4
5.2 Downstream Extrinsic Evaluation
- In FEVER zero-shot retrieval, R-Precision is 6.36% lower than GENRE’s and Recall@10 is less than 1% lower, while retrieval far exceeds TF-IDF and BM25.GENRE was trained on 11 tasks, including more than 100K fact-verification and 500K question-answering instances.Source: §5.2 · p. 4
- With 16,384 FEVER instances, DEARDR exceeds the 81.36% full-supervision R-Precision baseline; on HoVer, fine-tuning with one-quarter of the data gets within 1% of a fully supervised model.The HoVer comparison concerns R-Precision; the paper also reports higher R-Precision than supervised baselines with fewer training data.Source: Table 2 · p. 4, §5.2 · pp. 4–5, Table 3 · p. 5
- Figure 2’s learning curve shows greater R-Precision with fewer fine-tuning instances for DEARDR pre-training than for conventional supervised training.With 2,048 FEVER instances, DEARDR attains at least 80% R-Precision, while models without pre-training remain below 30% at the same data sizes.Source: Figure 2 · p. 4, §5.2 · p. 4
- On HoVer’s multi-hop challenge, HL and PTHL pre-training yield higher R-Precision than PT, whereas the zero-shot model’s R-Precision is less than 1% of GENRE’s.HoVer presents more complex reasoning challenges than FEVER; GENRE was trained on HotpotQA, from which HoVer is derived.Source: §5.2 · pp. 4–5
- The authors identify complex multi-hop reasoning as a remaining challenge requiring further data augmentation for autoregressive retrieval.They note that DEARDR is beneficial for FEVER with fewer data, but do not claim that this resolves the more complex multi-hop challenges.Source: §5.2 · p. 5
- For question answering, pre-training offers a benefit, but similar retrieval gains are not attained with fewer fine-tuning data; broader application requires further study.The discussion names TriviaQA, HotpotQA, and NaturalQuestions as settings for further study.Source: §5.2 · p. 5
6 Conclusions and Future Work
The paper uses distant supervision and pre-training to reduce annotated-data needs for sparse autoregressive retrieval. Its fact-verification results do not yet transfer satisfactorily to question answering or broader tasks.
6 Conclusions and Future Work
- Distant supervision and pre-training support high-precision autoregressive retrieval with fewer annotated examples, extending pre-training research to sparse autoregressive retrieval.The conclusion identifies fact verification as an application where fewer labeled examples were required.Source: §6 · p. 5
- Question answering did not yield satisfactory results, which the authors attribute to domain shift between question answering and fact verification.Adapting DEARDR to a wider range of tasks and multi-hop reasoning requires better understanding this limitation.Source: §6 · p. 5
A.1 Hardware Requirements
Experiments ran on a single workstation with one NVIDIA GTX 1080 Ti GPU.
A.1 Hardware Requirements
- Experiments used a single workstation equipped with one NVIDIA GTX 1080 Ti GPU.Source: §A.1 · p. 8
A.2.1 Pre-training
The appendix reports which hyperparameters were explored and identifies settings described as more beneficial for pre-training and fine-tuning.
A.2.1 Pre-training
- For HL pre-training, a learning rate of 5e-6 was more beneficial among the tested learning rates and schedulers.The tested learning rates ranged from 1e-4 to 1e-6, and scheduler options were constant with warmup, constant, or linear.Source: §A.2.1 · p. 8
A.2.2 Fine-tuning + Supervised
- For fine-tuning, dropout of 0.2 was more beneficial among the tested learning rates, schedulers, and dropout values.Dropout candidates were 0.1, 0.2, and 0.3.Source: §A.2.2 · p. 8