Source-linked AI summary
Multi-task Retrieval for Knowledge-Intensive Tasks
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oğuz, Veselin Stoyanov, Gargi Ghosh
TL;DR
Neural retrievers can lose performance outside their training domain, while task-specific systems complicate low-data use and deployment across tasks. The paper jointly trains one general-purpose retriever and finds it often matches task-specific models, improves downstream results, and quickly overtakes BM25 with minimal in-domain training.
Problem
Task-specific neural retrievers often perform poorly on other domains and tasks, raising whether one model can work across varied problems without requiring task-specific fine-tuning.
Source: §1 · p. 1Method
The paper proposes one general-purpose retriever trained jointly across datasets, rather than separate task-specific models.
Source: §1 · p. 2, §3.1 · p. 3, §6 · p. 9Results
A universal retriever trained jointly on 8 datasets often performs comparably to task-specific models, improves downstream performance with new top results on several KILT tasks, and quickly overtakes BM25 with minimal in-domain training.
Source: §6 · p. 9Takeaways & Limitations
The reported results support using a shared retriever across multiple KILT tasks while improving downstream systems.
Source: §6 · p. 9The zero- and few-shot evaluation omits T-REx because retrieving relevant documents for all training sets was infeasible given its size.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be universal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.
1 Introduction
The paper asks whether one neural retriever can work across knowledge-intensive tasks despite the out-of-domain weakness and deployment burden of task-specific models. It proposes joint training and reports benefits for retrieval and downstream performance.
1 Introduction
- Knowledge-intensive tasks require substantial world knowledge, and practical systems retrieve a limited set of relevant information for downstream models to use.Examples include open-domain question answering and fact checking.Source: §1 · p. 1
- Neural retrievers can perform poorly out of domain, limiting their suitability for few- and zero-shot use and complicating deployments that need multiple task-specific models.Separate models may also require separate copies of the supporting knowledge source.Source: §1 · p. 1
- The study tests whether joint training on a broad selection of retrieval tasks can produce a robust universal neural retriever.The target is a model that works across problems without task-specific fine-tuning and can be further fine-tuned when labelled in-domain data is available.Source: §1 · p. 1
- A single general-purpose retriever performs comparably to or better than specialised approaches in zero-shot and few-shot retrieval.The paper also investigates model variants to identify architectural factors affecting performance.Source: §1 · p. 2
- The proposed retriever’s gains in retrieval translate into performance gains for a variety of downstream knowledge-intensive tasks.The paper presents the model as a strong out-of-the-box retrieval system that can also be further trained in domain.Source: §1 · p. 2
2 Background
The background introduces sparse and dense retrieval, the DPR architecture and training setup, and the varied knowledge-intensive tasks that share a need for retrieval.
2 Background
- The background surveys sparse and dense retrieval before discussing knowledge-intensive NLP tasks where retrieval plays a crucial role.Source: §2 · p. 2
2.1 Retrieval methods
- Sparse methods such as tf-idf and BM25 match keywords efficiently, while dense methods encode text as lower-dimensional latent vectors and can represent paraphrases.Recent dense methods can outperform sparse counterparts when fine-tuned on a small set of in-domain labelled data (Karpukhin et al., 2020; Lewis et al., 2020b; Xiong et al., 2020).Source: §2.1 · p. 2
- DPR uses separate query and passage encoders, compares their embeddings by inner product, and retrieves passages through fast approximate nearest-neighbour search.The approach builds on DPR (Karpukhin et al., 2020) and uses FAISS for search.Source: §2.1 · p. 2, Figure 1 · p. 3
- DPR training uses a contrastive loss with relevant passages for other queries in the batch and a hard confounder that has high lexical overlap but is not relevant.Karpukhin et al. (2020) report that including hard confounders improves training results.Source: §2.1 · pp. 2–3
2.2 Knowledge-intensive Tasks
- KILT spans five classes of knowledge-intensive tasks, with varied query forms grounded in a shared snapshot of Wikipedia (Petroni et al., 2020).Across these tasks, retrieval supplies relevant passages to support the downstream task.Source: §2.2 · p. 3
3 Methods
The methods contrast per-dataset specialist retrievers with a multi-task DPR setup that shares encoders, and explore task-specific query encoders and task markers. They also iteratively train with model-selected confounders, testing this approach on Natural Questions and TriviaQA.
3.1 Universal retrieval
- The task-specific baseline uses separate models per dataset, which also entails separate copies of the Wikipedia index.Source: §3.1 · p. 3, Figure 2 · p. 3
- The base multi-task DPR shares passage and query encoders, allowing one encoded-passage index to serve all retrieval tasks.Source: Figure 3 · p. 3, §3.1 · p. 3
- Task-specific query encoders let each task family use a different encoder while retaining the shared-passage setup.Source: §3.1 · p. 4
- Task markers add a specialized token for each of KILT’s five task classes to help the model distinguish query types.Source: §3.1 · p. 4
3.2 Adversarial confounder selection
- Self-adversarial training retrieves high-ranking non-relevant passages with an initial model, then trains a second model using those added confounders.Source: §3.2 · p. 4
- The adversarial-confounder experiments use Natural Questions and TriviaQA, where answer presence enables automatic exclusion of relevant passages.Source: §3.2 · p. 4
- Treating passages without the answer span as irrelevant is not formally correct, although the authors report that the check correlates well with overall model quality.Source: §3.2 · p. 4
4 Experiments
The experiments evaluate a shared retriever across KILT tasks, downstream systems, low-data settings, model variants, and adversarial confounder selection. The shared model performs strongly across retrieval tasks and improves average downstream performance over the DPR + BART baseline.
4.1 Experimental settings
- The shared model is trained on eight KILT datasets spanning five task classes, with Wikipedia split into disjoint 100-token retrieval passages.The shared knowledge source contains 36 million passages, and training combines the datasets after downsampling two much larger sets.Source: §4.1 · p. 5
4.2 Universal retrieval
- The multi-task model achieves the best retrieval performance or comes within a few percentage points of the top score on all but trivial Zero-shot RE, where the task-specific model is near-perfect.Table 3 reports page- and passage-level R-precision on KILT validation data; AIDA-YAGO 2 has page-level retrieval only.Source: Table 3 · p. 6, §4.2 · p. 6
4.3 Downstream performance
- Multi-task training yields better average downstream performance than DPR + BART, with significantly better scores on AY2, zsRE, and HoPo and gaps below two points on other tasks.The model also compares favourably to RAG, whose query encoder is fine-tuned on the end task.Source: §4.3 · pp. 6–7, Table 4 · p. 7
4.4 Zero- and few-shot performance
- 10 points higher at the page level and 5 points lower at the passage level on average versus BM25 in leave-one-out zero-shot retrieval.The multi-task neural retriever is thus competitive with BM25 without training on the evaluation domain.Source: §4.4 · p. 7
- For zsRE and WoW, DPR scores lower than the multi-task model at page level but higher at passage level, so retrieval rankings differ by granularity.The few-shot evaluation excludes T-REx because retrieving relevant documents for all training sets was impractical for its very large size.Source: §4.4 · p. 7
- Average few-shot performance is markedly better than alternatives, and multi-task initialization more than doubles vanilla DPR on the smaller training set.Fine-tuning helps AY2 most; T-REx shows no improvement for either DPR or the multi-task model, though the latter achieves the top performance there.Source: §4.4 · pp. 7–8
4.5 Model variants
- Model-variant results show no clear winner, and the base architecture may be preferable for its simplicity and generally good performance.The comparison covers only three datasets because task-specific encoders require a full query encoder per task family in addition to the passage encoder.Source: Table 6 · p. 8, §4.5 · p. 8
4.6 Adversarial confounder selection
- Training with additional model-selected confounders improves performance across multiple tasks, and the approach is described as orthogonal to the multi-task retriever.The experiments add confounders for TriviaQA and Natural Questions, then train a new multi-task model on the augmented data.Source: §4.6 · p. 8
5 Related work
The section situates the approach among retrieval architectures, orthogonal pretraining and dense-retrieval enhancements, and task-specific entity linkers.
5 Related work
- RAG augments DPR with a generative reader and trains the query encoder end-to-end, unlike traditional retriever-reader architectures (Lewis et al., 2020b).DPR is the basis for the paper's retrieval systems (Karpukhin et al., 2020).Source: §5 · p. 8
- Inverse Cloze Task and its extensions are self-supervised pretraining methods for open-domain question answering, while whether this pretraining transfers beyond QA remains open (Lee et al., 2019; Chang et al., 2020).CERT is introduced as an alternative pretraining approach inspired by computer vision (Fang et al., 2020); sentence-level semantic pretraining is also suggested as potentially promising relative to token-level masked language modeling.Source: §5 · pp. 8–9
- ColBERT and ME-BERT embed passages into multiple fixed-size vectors; ColBERT-QA also uses data augmentation closely related to this paper's approach (Khattab and Zaharia, 2020; Luan et al., 2020; Khattab et al., 2020).Source: §5 · p. 9
- GENRE and BLINK are entity linkers expected to outperform retrieval-based approaches on entity linking, but are not generally applicable to information retrieval (Cao et al., 2020; Wu et al., 2020).Source: §5 · p. 9
6 Conclusions
A single retriever trained across eight datasets often matched task-specific models, and its use in downstream systems improved performance across KILT tasks. The study also reports zero-shot and few-shot retrieval findings, mixed results for specialized variants, and a practical case for multi-task deployment.
6 Conclusions
- A single universal retriever jointly trained on 8 datasets often performed comparably to task-specific models.Source: §6 · p. 9
- Overall downstream performance improved, setting new top results for a number of tasks in the KILT benchmark.Source: §6 · p. 9
- In zero-shot retrieval, the approach performed comparably to BM25 and quickly overtook it with minimal in-domain training.Source: §6 · p. 9
- More complex task-specialized variants showed no clear performance improvements, while simple iterative data augmentation could improve performance.Source: §6 · p. 9
- Multi-task retrieval may suit industry settings that need several retrieval operations on the same content when separate task-specific models are impractical for space or computation.Source: §6 · p. 9