PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-aware Retrieval with Instructions

Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih

arXiv:2211.09260v2cs.CL

TL;DR

Queries may not specify users’ intended relevance, and task-specific retrievers need annotated examples and may transfer poorly to distant tasks. The paper introduces instruction-conditioned retrieval, BERRI, and multi-task TART; TART advances zero-shot benchmarks and outperforms state-of-the-art methods in X2-Retrieval.

  • Problem

    Task-specific retrievers may require many annotated examples and transfer poorly to tasks unlike those they were trained for.

    Source: §1 · p. 1

  • Method

    The paper introduces instruction-conditioned retrieval, builds BERRI, and trains TART as a multi-task system that follows instructions without task-specific parameter updates.

    Source: §1 · pp. 1–2

  • Results

    TART advances state of the art on BEIR and LOTTE-pooled and outperforms state-of-the-art methods in X2-Retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TART demonstrates that explicit textual intents can guide document retrieval in a large-scale open-domain corpus with diverse query intents.

    Source: §1 · p. 2

    TART-dual drops from its non-instruction-following counterpart on a few datasets; the authors hypothesize model size and limited query-document interaction are bottlenecks.

    Source: §9 · p. 11

Abstract

from arXiv · show

We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries. We aim to develop a general-purpose task-aware retrieval system using multi-task instruction tuning, which can follow human-written instructions to find the best documents for a given query. We introduce the first large-scale collection of approximately 40 retrieval datasets with instructions, BERRI, and present TART, a multi-task retrieval system trained on BERRI with instructions. TART shows strong capabilities to adapt to a new retrieval task via instructions and advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger. We further introduce a new evaluation setup, X^2-Retrieval to better reflect real-world scenarios, where diverse domains and tasks are pooled and a system needs to find documents aligning users' intents. In this setup, TART significantly outperforms competitive baselines, further demonstrating the effectiveness of guiding retrieval with instructions.

1 Introduction

Queries alone may underspecify user intent, while task-specific retrievers require annotated examples and may transfer poorly to distant tasks. The paper introduces explicit instruction-conditioned retrieval, builds BERRI, trains TART, and evaluates it on zero-shot and cross-task settings.

1 Introduction

  • Existing intent-specific retrievers may require many annotated examples and transfer poorly to tasks unlike their training task.The paper notes that related-task data could otherwise be used more broadly.Source: §1 · p. 1
  • Retrieval with instructions makes the search task explicit, requiring documents to be relevant to the query and well-suited to the user’s instruction.The instruction describes the search task and the intended relevance.Source: §1 · p. 1
  • BERRI collects approximately 40 retrieval datasets across 10 domains, with an average of 3.5 expert-annotated instructions per task.The instructions use a unified format and a novel schema for retrieval tasks.Source: §1 · p. 2
  • TART is a single multi-task system trained on BERRI that follows instructions without task-specific parameter updates, using dual-encoder and cross-encoder architectures.Training includes instruction-unfollowing negative samples.Source: §1 · p. 2
  • TART advances state of the art on BEIR and LOTTE-pooled, outperforming systems up to three times larger, and beats state-of-the-art methods on X2-Retrieval.The X2-Retrieval setup pools diverse domains and tasks to test retrieval aligned with users’ intents.Source: §1 · p. 2

2 Background and Related Work

Prior work explores zero-shot retriever training, instruction tuning, and richer query descriptions, but retrieval instruction tuning lacks dedicated annotated resources. Existing approaches also face adaptation costs or limited gains from descriptions.

2 Background and Related Work

  • Zero-shot retrieval work includes unsupervised pretraining, transfer from supervised datasets, and task-specific retrievers trained with automatically generated data.Generating data and training separate retrievers can require massive language models and make adaptation slow and costly.Source: §2 · p. 2
  • Large-scale instruction-annotated datasets do not include retrieval tasks, limiting systematic exploration of instruction tuning for retrieval.Successful instruction-following language models are often difficult to adapt when retrieval requires encoding millions of documents.Source: §2 · pp. 2–3
  • Table 1 illustrates instructions for six datasets, with each instruction specifying intent, domain, and unit.Source: Table 1 · p. 3
  • Earlier retrieval with descriptions or narratives found that concatenating them only marginally improved over title-only baselines, while later work suggested powerful encoders could use their linguistic context better.Source: §2 · p. 3

3 Task Formulation

The task formulation adds a natural-language instruction to the query so retrieval can reflect the user’s intended relevance. It also motivates shared training across tasks and zero-shot transfer through instructions.

3 Task Formulation

  • Retrieval with instructions takes a document collection, query, and task instruction, and seeks a document relevant to the query according to that instruction.The instruction explicitly defines relevance, allowing the system to adapt its relevance measure by attending to it.Source: §3 · p. 3
  • The formulation requires retrievers to modify their search behavior according to instructions.Source: §3 · p. 3
  • Grouping datasets for multi-task training can use cross-task interdependence, while natural-language instructions allow zero-shot transfer and avoid hosting multiple task-specific retrievers.Source: §3 · p. 3
  • The authors introduce an expert-annotated large-scale retrieval benchmark and multi-task instruction-following retrievers to address the lack of resources and dedicated models.Source: §3 · p. 3

4 BERRI: Collections of Instruction-annotated Retrieval Tasks

BERRI unifies retrieval tasks with explicit natural-language instructions, drawing on curated retrieval datasets and NLP datasets recast as retrieval. Its task schema and negative-sample strategies are designed to represent user intent and distinguish relevant documents from plausible but unsuitable ones.

4 BERRI: Collections of Instruction-annotated Retrieval Tasks

  • BERRI comprises instructions and instances from diverse domains and tasks, built by curating retrieval datasets and casting NLP datasets as retrieval tasks.The collection is called the Bank of Explicit RetRieval Instructions.Source: §4 · p. 3

4.1 Unified Task and Instructions Scheme

  • Each BERRI task pairs a corpus and queries with an explicit intent instruction, and each instance includes a query, relevant documents, and irrelevant documents.An intent can be paraphrased in multiple ways.Source: §4.1 · p. 3
  • Retrieval instructions specify intent, the expected document domain, and the text unit to retrieve.Examples include whether a text answers or paraphrases a query, whether it comes from Wikipedia or PubMed, and whether it is a sentence or paragraph.Source: §4.1 · p. 4

4.2 Dataset Collection

  • The collection draws on widely used retrieval, retrieval-centric, and non-retrieval datasets, with manually annotated instructions and post-processing of query, gold, and negative documents.Figure 2 gives examples of datasets included in BERRI.Source: §4.2 · p. 4, Figure 2 · p. 4
  • BERRI contains 37 datasets and more than 5 million instances, selected from over 60 candidates after checking whether queries were self-contained.For datasets much larger than others, up to 300,000 instances were sampled, except for MS MARCO.Source: §4.2 · p. 4
  • Hard negatives are retrieved from each target corpus with Contriever, then filtered using a cross-encoder whose normalized scores are below 0.1.The filtering is intended to remove false negatives.Source: §4.2 · p. 5
  • Instruction-unfollowing negatives are documents retrieved from another task’s target corpus, which may be topically plausible but fail the specified intent.Figure 3 contrasts these with gold documents and hard negatives.Source: §4.2 · p. 5, Figure 3 · p. 5

5 TART: Multi-task Instructed Retriever

TART is a unified multi-task retriever trained on BERRI through instruction tuning. It includes dual-encoder and cross-encoder architectures and uses positive and negative examples, with a distillation procedure to improve the smaller dual encoder’s training data.

5 TART: Multi-task Instructed Retriever

  • TART is a unified multi-task retriever trained on BERRI through multi-task instruction tuning.The system is presented as task-aware and instruction-following.Source: §5 · p. 5

5.1 Model Architecture

  • TART-dual encodes the instruction and query together, encodes documents independently, and retrieves by maximum inner product search.Document embeddings can be computed offline, trading storage space for inference efficiency.Source: §5.1 · p. 5
  • TART-full jointly encodes a query and candidate document with cross-attention to score their relevance given the instruction.A lightweight retriever first selects top documents because cross-encoders are costly to scale across millions of documents.Source: §5.1 · pp. 5–6

5.2 Training TART

  • Both TART models train on BERRI’s positive documents and three types of negative documents paired with instructions.TART-dual uses annotated documents and in-batch negatives; TART-full uses cross-entropy loss.Source: §5.2 · p. 6
  • The authors use TART-full predictions to update hard negatives and positive documents before distilling the stronger model’s knowledge into TART-dual.This procedure addresses less reliable mined negatives in some domains by using instruction-aware predictions to refine training examples.Source: §5.2 · p. 6

6 Experiments

The experiments evaluate zero-shot transfer on BEIR and LOTTE-pooled, and introduce X2-Retrieval to test retrieval across pooled tasks and domains. They compare TART with unsupervised, broadly trained, and task-specialized baselines.

6 Experiments

  • The experiments assess zero-shot retrieval and the new X2-Retrieval cross-task, cross-domain setup against state-of-the-art methods.The evaluation is organized into zero-shot and X2-Retrieval experiments.Source: §6 · p. 6

6.1 Zero-shot Retrieval Evaluations

  • BEIR and LOTTE-pooled test zero-shot transfer without overlap between evaluation datasets or instructions and BERRI.Many evaluation tasks also differ substantially from training tasks; BEIR excludes several datasets included in encoder pretraining or BERRI.Source: §6.1 · p. 6

6.2 X2-Retrieval: Cross-task Cross-domain Retrieval Evaluation

  • X2-Retrieval pools tasks with different intents and documents from Wikipedia, Science, and Technical domains, requiring zero-shot adaptation to natural-language intent.Its primary pooled corpus contains approximately 3.7 million documents; the closed setup restricts retrieval to the original task corpus.Source: §6.2 · p. 7
  • X2-Retrieval reports NDCG@10 for pooled and closed setups and uses their performance gap as a robustness measure.A smaller gap means less distraction from documents in undesirable corpora.Source: §6.2 · p. 7

6.3 Baselines

  • Baselines include unsupervised retrievers, models transferred from MS MARCO or other broad training, and task-specific retrievers trained on automatically generated data.The listed methods include Contriever and BM25, MonoT5 and GTR, and Promptagator and GPL.Source: §6.3 · p. 7

6.4 Experimental Settings

  • TART-full is trained up to 10k steps with the best checkpoint selected by development-split performance; TART-dual starts from a Contriever-MS MARCO checkpoint and trains up to 30k steps.For TART-full, Contriever-MS MARCO retrieves candidates and the top 100 are reranked. For each TART-dual positive, five negatives are sampled: 90% randomly and 10% from dHD and dUF, in addition to in-batch negatives.Source: §6.4 · pp. 7–8

7 Results

TART-full advances zero-shot retrieval on BEIR and LOTTE, while TART models also perform strongly when tasks and domains are pooled in X2-Retrieval. Results distinguish the stronger TART-full from TART-dual, whose pooled robustness contrasts with weaker performance on some individual benchmarks.

7.1 Zero-shot Evaluation Results

  • TART-full advances the state of the art on BEIR and LOTTE, outperforming larger systems and retrievers trained on millions of synthetic in-domain examples.The reported zero-shot evaluations use datasets and instructions not present in BERRI.Source: §7.1 · p. 8
  • TART-dual improves on six of nine BEIR tasks but deteriorates on the other three, reducing its average performance; TART-full significantly outperforms prior state of the art on LOTTE-pooled.The authors hypothesize that TART-dual’s BERT-base capacity and limited query–document interactions are bottlenecks.Source: §7.1 · pp. 8–9

7.2 X2-Retrieval Evaluation Results

  • Contriever and Contriever+CE are competitive in closed X2-Retrieval but struggle when tasks are pooled; Contriever+CE’s AmbigQA-pooled drop reflects a bias toward retrieving documents rather than queries.The paper attributes that bias to fine-tuning on the QA dataset MS MARCO alone.Source: §7.2 · p. 9
  • TART-full achieves the best closed and pooled performance in X2-Retrieval, indicating strong zero-shot adaptation and cross-task abilities.The setup pools diverse tasks and domains, requiring systems to find relevant documents across the pooled corpus.Source: §7.2 · p. 9
  • TART-dual performs below Contriever in the closed X2-Retrieval setting but has the best pooled robustness among the compared systems.The authors note that limited model capacity and query–document interactions may restrict its transfer to new tasks.Source: §7.2 · p. 9

8 Analysis

The analyses examine instruction use, data and model scale, and negative sampling in TART-full. They report benefits from instructions, diverse training datasets, larger models, and challenging negatives, with instruction-unfollowing negatives particularly relevant to pooled retrieval.

8.1 Effects of Instructions

  • Removing instructions at training or test time causes notable performance drops across benchmarks; training with instructions still helps somewhat when test-time instructions are omitted.The ablations compare models without instructions in training, without them at test, or without them at either stage.Source: §8.1 · p. 9, Figure 4 · p. 9
  • Informative, correct instructions often let TART-full outperform competitive baselines, while inaccurate instructions cause larger performance deterioration.Performance varies only slightly across different instructions overall.Source: §8.1 · pp. 9–10, Figure 5 · p. 9

8.2 Dataset Scale

  • Increasing TART-full’s training set from 5 to 10 to 20 randomly sampled datasets improves average performance on four BEIR datasets.The analysis varies the number of training datasets and reports the four-dataset average.Source: §8.2 · p. 10, Figure 6 · p. 10
  • Task and domain diversity improve zero-shot transfer: QA-only training struggles on Arguana, while Wikipedia-only training struggles on Touche-2020 and SciFact.The cited target documents come from argument websites and scientific papers, respectively.Source: §8.2 · p. 10, Figure 7 · p. 10

8.3 Model Scale

  • Increasing TART-full’s model size produces clear average-performance improvements.The comparison trains T5 LM-Adapt base, large, and XL models and uses them to rerank the top 100 Contriever results.Source: §8.3 · p. 10, Figure 6 · p. 10

8.4 Negative Samples

  • Adding challenging negative documents, including hard and instruction-unfollowing negatives, largely improves BEIR performance.Without instruction-unfollowing negatives, X2-Retrieval performance is lower even though BEIR performance remains on par with the original TART-full.Source: §8.4 · pp. 10–11, Figure 8 · p. 10
  • Instruction-unfollowing negatives contribute to distinguishing instructions and are described as crucial for a robust task-aware retrieval system.The ablation without these samples has lower X2-Retrieval performance.Source: §8.4 · pp. 10–11

9 Discussions and Conclusion

The paper concludes that multi-task instruction tuning with BERRI supports task-aware retrieval, with TART advancing results across BEIR, LOTTE, and X2-Retrieval. It also identifies efficiency and further dataset scaling as open challenges, and the appendices describe BERRI’s dataset listing, task unification, and instructions.

9 Discussions and Conclusion

  • TART advances state-of-the-art results on BEIR, LOTTE, and X2-Retrieval; the analyses identify informative instructions, data diversity, model scale, and designed negatives as key factors.The paper presents TART as a multi-task instruction-following system trained on BERRI using two architectures.Source: §9 · p. 11
  • TART-dual’s 110-million-parameter size and limited query–document interactions may make instruction-following retrieval more challenging than with the cross-encoder.The authors propose studying larger bi-encoders and architectures that allow richer interaction while remaining more efficient than cross-encoders.Source: §9 · p. 11
  • Whether expanding beyond approximately 40 retrieval tasks improves zero-shot and cross-task retrieval remains an open question.The authors call for collecting additional retrieval tasks and human-written instructions.Source: §9 · p. 11

A.1 Dataset List

  • Tables 5 and 6 provide the BERRI dataset list and dataset references.Some BERRI datasets were not originally retrieval datasets.Source: §A.1 · p. 17, §A.2 · p. 17

A.2 Details of Dataset Unification

  • BERRI unifies non-retrieval tasks by treating source text or questions as queries and target summaries, simplified text, or evidence as gold documents.For code search, source comments are queries and corresponding implications are gold documents; the Python subset is excluded because it is used for X2-Retrieval.Source: §A.2 · p. 17

A.3 Instructions for BERRI

  • The appendix presents one instruction per BERRI dataset and says the full instruction list will be released in the repository.Source: §A.3 · p. 17

A.4 BERRI Statistics

BERRI spans varied retrieval intents and domains: its tasks include many categories beyond the most common ones, and its datasets range from general to expert domains.

A.4 BERRI Statistics

  • Around 50% of BERRI tasks fall outside its three most common categories, including code search and caption generation.The source task categories are used as a proxy for intents; sentence simplification and dialogue are also repurposed as retrieval tasks.Source: §A.4 · p. 17, Figure 9 · p. 17
  • QA is the most representative source task category, while summarization and question-duplication detection are also common.The distribution is shown in Figure 9.Source: §A.4 · p. 17, Figure 9 · p. 17
  • Wikipedia, web, Community QA, and news dominate BERRI, alongside smaller expert-domain datasets such as medical, legal, and technical data.The authors report that adding high-quality expert-domain datasets helps the system adapt to those domains or unseen expert domains with similar writing styles, such as scientific papers.Source: §A.4 · p. 17, Figure 10 · p. 17

B Further Detail about the X2-Retrieval evaluation

The appendix details X2-Retrieval construction and TART training choices, then analyzes instruction use and model and dataset scale. Across benchmarks, instruction ablations and scale analyses report performance patterns, while the appendix also describes qualitative comparisons on pooled retrieval.

B Further Detail about the X2-Retrieval evaluation

  • The X2-Retrieval appendix specifies query and corpus construction and reports a false-negative ratio below 10%.For example, CodeSearchNet-Python uses code comments as queries and corresponding Python code as positive documents; the false-negative check sampled top-10 documents for 20 questions per task.Source: B Further Detail about the… · pp. 20–21

C.1 Hyperparameters of TART

  • TART-dual uses a batch size of 1024 and seven negatives per instance; TART-full uses a batch size of 1 with eight gradient-accumulation steps.The listed maximum sequence lengths are 256 for TART-dual document chunks and 512 for TART-full.Source: §C.1 · p. 21

C.2 Instructions for Evaluations

  • The appendix points to a full list of instructions used for evaluation in Table 9.The instructions are supplied for the BEIR and X2-Retrieval evaluations.Source: Table 9 · p. 20, §C.2 · p. 21

C.3 Negative Sampling

  • TART-full training combines random and hard negatives, samples four negatives per positive, and limits instruction-unfollowing negatives to less than 20% of negatives.The authors note that hard negatives alone can cause large out-of-domain performance drops, while combining random and carefully designed negatives supports competitiveness in both in-domain and out-of-domain retrieval (Ni et al., 2021).Source: §C.3 · p. 21

D.1 Qualitative Results on X2-Retrieval

  • The appendix compares TART-full and Contriever-MS MARCO predictions on X2-Retrieval; the table marks whether predictions satisfy the instructions.The examples use the same queries with different instructions.Source: §D.1 · p. 21, Table 11 · p. 22

D.2 Analysis of Instruction Effectiveness

  • Removing instructions at training or test time largely hurts performance across the benchmarks; informative, related instructions often perform strongly, while irrelevant ones degrade performance.The ablation tables cover BEIR, LOTTE, and X2-Retrieval; prompt comparisons include three BEIR datasets.Source: §D.2 · p. 21

D.3 Analysis on Model and Dataset Scale

  • Larger TART-full models generally perform better, and adding training datasets often improves BEIR performance.The reported patterns are in the model-scale and training-dataset-scale comparisons.Source: §D.3 · pp. 21–22

D.4 Analysis on Different Pre-trained Models

The analysis compares TART-full initialized with T0-3B, FLAN-T5-XL, and Tk-Instruct-3B on BEIR. FLAN-T5 has the best average performance, while Tk-Instruct shows drops on some datasets.

D.4 Analysis on Different Pre-trained Models

  • The comparison initializes TART-full with T0-3B, FLAN-T5-XL, and Tk-Instruct-3B.The paper notes that FLAN-T5-XL and Tk-Instruct were trained on an order of magnitude more datasets.Source: §D.4 · p. 22
  • Tk-Instruct-3B's TREC COVID drop contributes to slightly lower performance than the original TART-full.This is a dataset-specific qualification to the average-BEIR comparison.Source: §D.4 · p. 22
  • FLAN-T5 shows the best average BEIR performance, outperforming TART-full initialized with T0-3B by 0.7 NDCG@10.Tk-Instruct-3B has notable drops on some datasets, including TREC COVID, and slightly lower performance than the original TART-full.Source: §D.4 · p. 22, Table 19 · p. 25
Loading 2211.09260v2…