Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models

1 Introduction

• Modern retrieval’s single semantic-similarity score can make search opaque, requiring users to find particular phrasings, apply filters, and iterate.Source: §1 · p. 1
• Promptriever is a retrieval model controlled through natural-language prompts, allowing relevance to adapt to a query’s specific constraints.For example, a user can specify that movie results must be pre-2022 and not codirected rather than applying repeated searches or hard filters.Source: §1 · p. 1
• Promptriever is trained on roughly 500K query-passage pairs augmented with diverse per-query relevance instructions and instruction negatives.Instruction negatives are relevant to the query alone but become less relevant when the instruction is added, encouraging instruction-conditioned relevance.Source: §1 · p. 2
• +14.3 p-MRR and +3.1 nDCG/MAP on FollowIR are reported for Promptriever’s state-of-the-art bi-encoder instruction-following performance.The paper also reports improved robustness to query length and phrasing, including +12.9 on InstructIR Robustness@10.Source: §1 · p. 2
• The results show that modern bi-encoders can follow free-form natural-language instructions when trained with suitable data.The authors hope this alignment between LM prompting and IR will support further dense-retrieval improvements.Source: §1 · p. 2

2 Promptriever Data Generation

• The augmented MS MARCO training set contains roughly 491K queries, each with a positive passage and 30 hard negatives, and adds instructions through two generation stages.The stages are instruction generation and instruction-negative mining.Source: §2 · pp. 2–3

2.1 Instruction Generation

• Llama 3 generates query-specific instructions that add requirements, exclude passage types, or disambiguate the original query.For the volcano example, the generated instruction asks for both the volcano type and information about its formation.Source: §2.1 · p. 3
• Instructions vary in length and style, including short or paragraph-length formats and persona, negation, or background styles.Llama 3 generally followed the requested length description across the dataset’s length-format and feature subsets.Source: §2.1 · p. 3, Table 1 · p. 3
• To preserve original MS MARCO positives, instruction generation conditions on the query and positive passage and asks that the passage remain relevant.FollowIR-7B judged about 15% of generated instructions to make the original positive no longer relevant; those cases use a generated positive instead.Source: §2.1 · p. 3

2.2 Instruction Negative Mining

• Adding instructions alone leaves relevance relations unchanged, so models could ignore the instructions and retain standard retrieval behavior.An ablation reports that models indeed ignore instructions in this setting.Source: §2.2 · p. 3
• Instruction negatives are query-positive passages whose relevance decreases when the instruction is added, forcing training to condition on both query and instruction.The approach is described as similar in intuition to Asai et al. (2022), but gathered at the instance rather than dataset level.Source: §2.2 · p. 3
• Because MS MARCO lacked enough nuanced query-positive but instruction-negative passages, the authors generate passages with an LM and filter candidates post hoc.GPT-4o generates one instruction-positive and three instruction-negative candidates per pair; initial testing found only about two-thirds of generated passages correctly instruction-negative.Source: §2.2 · p. 4
• Human-model agreement for relevance filtering averaged 84% (N=64), compared with 75% human-human agreement (N=32).The authors conclude that FollowIR-7B acts as a sufficiently high-quality filter.Source: §2.2 · p. 4

3 Experimental Settings

• The experimental goal is to test instruction training against standard training, primarily using RepLLaMA and its data and recipe for an apples-to-apples comparison.Source: §3 · p. 4

3.1 Training

• Promptriever training uses RepLLaMA’s MS MARCO data and recipe alongside instruction data generated by Llama 3 and GPT-4o.The authors retain RepLLaMA’s learning rate and other hyperparameters; training uses all valid instruction negatives while keeping the same number of hard negatives per query.Source: §3.1 · p. 4

3.2 Evaluation Datasets

• Evaluation covers MS MARCO, BEIR, and per-query instruction-following datasets InstructIR and FollowIR, alongside other retrieval baselines.Metrics include nDCG@10 for BEIR and TREC DL19/DL20, MRR for MS MARCO Dev, and instruction-following metrics including p-MRR and Robustness@10.Source: §3.2 · p. 4

4.1 Instruction Following

• +14.3 p-MRR (-3.1 →+11.2) and +3.1 on the standard retrieval score (nDCG/MAP average) over RepLLaMA; Promptriever leads dense retrievers and scores comparably to FollowIR-7B.The comparison uses the same Llama 2 backbone; Promptriever is a bi-encoder, while FollowIR-7B is a cross-encoder.Source: §4.1 · pp. 4–5, Table 2 · p. 5

4.2 Standard Retrieval

• Promptriever performs comparably to RepLLaMA on MS MARCO splits and averages 55.0 vs. 54.9 on BEIR without instructions.The BEIR comparison is for the None-prompt setting.Source: §4.2 · p. 5, Table 3 · p. 5, §4.3 · p. 5

4.3 Retrieval with Prompts

• +1.4 nDCG@10 on BEIR is the average gain for Promptriever with the best prompt; it gains on 12/13 datasets and ties on the last, while RepLLaMA and BM25 do not improve.RepLLaMA and BM25 have -0.1 and -5.0 nDCG deltas, respectively.Source: Table 4 · p. 6, §4.3 · p. 6
• Selecting a prompt using a small dev set picks the best of ten prompts in 6/7 cases; the remaining SciFact choice still beats no prompt by almost one point.Not all BEIR datasets have train/dev sets for prompt selection.Source: §4.3 · pp. 5–6
• Promptriever’s variance across prompts is 44% lower than RepLLaMA’s and 77% lower than BM25’s; RepLLaMA drops in six of seven cases and BM25 drops across the board.The variance comparison concerns prompt sensitivity; some RepLLaMA scores have lower variance while clustering at lower scores.Source: §4.3 · pp. 6–7, Table 5 · p. 6

5 Analysis

• Length-only baselines improve standard retrieval by +0.6 nDCG/MAP for Repeat Query and +1.4 nDCG/MAP for Generic Instruction, but query repetition adds little p-MRR.This comparison tests whether query/instruction length alone accounts for Promptriever’s gains.Source: §5 · p. 7
• Swapping real instructions across queries yields +1.6 in score and +2.1 p-MRR, suggesting the model learns their distribution but not how to use them effectively.Source: §5 · p. 7
• Training with instructions adds +6.6 p-MRR and +1 standard retrieval over Swap; adding instruction negatives brings another +3.1 p-MRR and +0.6 nDCG/MAP.The instruction-negative gain is described as extra data for instruction sensitivity, not necessarily for standard retrieval metrics.Source: §5 · p. 7
• +2.4 p-MRR comes from adding MS MARCO jointly to the instruction data, doubling the training data; each step in the final recipe provides independent value.The recipe’s gains are not attributed to query length or surface lexical features of instructions.Source: §5 · pp. 7–8

5.1 Does this process work for other models?

• Other backbone models provide comparable performance with the same recipe and hyperparameters, though further tuning would likely improve their results.The comparison keeps RepLLaMA’s training hyperparameters and Promptriever’s recipe unchanged.Source: §5.1 · p. 8, Table 7 · p. 8

6.1 Instructions in Retrieval

• Recent instruction retrievers reuse the E5-Mistral instruction set, whereas this work evaluates adaptability per query rather than with a dataset-wide prefix.The passage identifies E5-Mistral (Wang et al., 2023) among newer models that reuse that set.Source: §6.1 · p. 8
• Unlike prior phrasing tests that change one word, this work evaluates broader instruction length and style modifications; existing bi-encoders were found to fail at instruction following on FollowIR and InstructIR.The passage describes Promptriever as the first bi-encoder with significantly above-random performance on those benchmarks.Source: §6.1 · p. 8

6.2 Prompting LMs

• Promptability in retrieval has received little study; GritLM attempted in-context learning but reported worse results than zero-shot.The passage also mentions BGE-ICL, but says no associated paper described its procedure or training details as of publication.Source: §6.2 · pp. 8–9

6.3 Synthetic Instruction Generation

• This work builds on LM-generated retrieval instructions, using them to generate instruction-negative passages.The cited line of work includes Wang et al. (2022b), Li et al. (2023a), and Chung et al. (2024).Source: §6.3 · p. 9

• Promptriever is presented as the first zero-shot promptable retriever trained from a new instruction-based retrieval dataset based on MS MARCO.Source: §7 · p. 9
• Experiments show that Promptriever performs well on standard retrieval and follows instructions more effectively than prior work, adapting relevance per query.Source: §7 · p. 9
• The authors conclude that prompting techniques from the LM community can extend to dense retrievers and hope to inspire joint research between the LM and IR communities.Source: §7 · p. 9

8 Limitations

• In-context learning with explicit examples versus imperative requests remains unexplored for retrieval prompting.Source: §8 · p. 9
• Why some IR prompts perform better than others is often unclear, and the authors hope retrieval models become more robust to different prompts.Source: §8 · p. 9
• LM-generated passages and instructions may contain errors, pernicious social biases, or incorrect information, and unintended characteristics may persist despite filtering and exploration.The authors say further audits of their corpus and retrieval training sets more broadly would be appropriate.Source: §8 · p. 9

A Retrieval Prompts

• The appendix reports retrieval prompts and their scores per dataset in Table 12.Source: A Retrieval Prompts · p. 11

B Hyperparameter Details

• Training follows the RepLLaMA authors’ Tevatron settings, using LLaMA-2-7B, LoRA rank 32, one epoch, and an effective batch size of 128.The query maximum length is 304 to accommodate longer instructions, rather than RepLLaMA’s 32; the full run takes approximately four days on 8x40GB A100s.Source: B Hyperparameter Details · p. 11

C Generation Prompts

• System, instruction-generation, and instruction-negative prompts are Prompts 5, 3, and 4, respectively.Source: C Generation Prompts · p. 11

D Generic Instructions

• GPT-4o and Claude-3.5-Sonnet generated generic retrieval descriptions used as instructions for the models.Source: D Generic Instructions · p. 11

E Error Analysis Attempts

• The error analysis binned per-query nDCG@10 changes and trained BERT-base and bag-of-words classifiers on 80% of examples, holding out 20%.Source: E Error Analysis Attempts · p. 11
• Instruction generation starts from queries and relevant and non-relevant documents; negative prompts request passages that answer the query but fail its specific requirements.Source: E Error Analysis Attempts · pp. 12–13, Figure 3 · p. 12, Figure 4 · p. 13
• Accuracy stayed below the majority baseline, typically around 66%, and the best AUC across datasets was 54%; tested variations did not change the results.The authors hypothesize that documents or the query-document connection are critical to understanding Promptriever’s stronger performance or prompt helpfulness.Source: E Error Analysis Attempts · p. 14
• Improved and non-improved query groups were identical in length, IDF, and other basic text statistics.Source: E Error Analysis Attempts · p. 14

F Comparison to state-of-the-art models on BEIR

• Promptriever was not trained on BEIR datasets other than MS MARCO, while most compared state-of-the-art models use BEIR training/dev sets.Source: F Comparison to state-of-the-art… · p. 14, Table 13 · p. 19
• The BEIR comparison places Promptriever among some of the leaderboard’s best models.The comparison is reported in Table 13.Source: F Comparison to state-of-the-art… · p. 14, Table 13 · p. 19
• On true OOD BEIR datasets, Promptriever performs comparably to many leaderboard models and beats GritLM, LLM2Vec, and Google Gecko.The comparison covers datasets without train/dev sets.Source: F Comparison to state-of-the-art… · p. 14, Table 13 · p. 19