Effective Dense Retrieval using Only In-Context Examples

1 Introduction

• LLM-based dense retrieval typically relies on supervised or unsupervised retriever training, leaving training-free strong embeddings an open challenge.Out-of-the-box LLM hidden states do not automatically provide effective dense-retrieval representations.Source: §1 · p. 1
• PromptReps generates query and document representations independently, without shared context connecting the encoding tasks.RICE builds on directly prompting LLMs for representations while adding in-context retrieval cues.Source: §1 · p. 1
• RICE uses a few query–document examples to provide shared context for query and document encoding, aiming to improve dense-retrieval representations without retrieval-specific training.For each example pair, the LLM generates a representative word that is used in prompting.Source: §1 · p. 1
• RICE improves prompt-based LLM representations, achieves the highest average accuracy among evaluated out-of-the-box approaches, and is competitive with LLM2Vec-Gen.It remains less effective than dense retrievers trained on large collections of high-quality data.Source: §1 · p. 2

2.1 Preliminaries: PromptReps-Dense

• PromptReps-Dense represents an input by extracting the LLM’s final hidden state before the language-modeling head after prompting it to generate a representative word.Document vectors are precomputed and indexed, while query vectors are computed at query time for search.Source: §2.1 · p. 2
• PromptReps-Dense may lack task-specific relevance examples and shared query–document context, potentially limiting how well its representations capture relevance.The paper hypothesizes that these gaps hinder generation of words aligned with a retrieval objective and relevance-reflecting inner products.Source: §2.1 · p. 2

2.2 RICE

• RICE conditions query and document representations on the same demonstrations, with the aim of making their inner products better reflect relevance.The examples are hypothesized to expose the LLM to task- and domain-specific language.Source: §2.2 · p. 2
• RICE uses a few human-annotated query–document pairs and generates a PromptReps representative word for each example document.Each pair becomes a demonstration with the query–document pair as input and its representative word as output; the target query or document is then encoded in that context.Source: §2.2 · p. 2

3 Experimental Setup

• The experiments assess whether LLMs can produce effective dense-retrieval representations using in-context examples without additional training.RICE is implemented using Qwen3-8B and Qwen3.5-9B with thinking disabled, and uses ten examples.Source: §3 · p. 3
• Evaluation covers ten BEIR retrieval datasets across news, financial QA, biomedical IR, fact checking, citation prediction, tweets, arguments, and question answering, using Recall@100.For datasets whose example pairs come from the test set, the encoded query or document is excluded from the examples at inference time.Source: §3 · p. 3
• PromptReps-Dense is the primary zero-shot comparison, and a RICE (Zero-Shot) baseline controls for prompt wording by removing in-context examples.RICE (Zero-Shot) differs from PromptReps only in the prompt text.Source: §3 · p. 3
• The comparison includes trained retrievers LLM2Vec-Gen and Promptodile, as well as zero-shot methods HyDE, PRF-UMBRELA, and CSQE using Contriever with Rocchio feedback.LLM2Vec-Gen is self-supervised; Promptodile uses synthetic LLM-generated training data.Source: §3 · p. 3
• BGE-base-en-v1.5 and Qwen3-Embedding-8B serve as trained upper bounds, while BM25 is the unsupervised sparse-retrieval baseline.The upper-bound systems were fine-tuned using dense-retriever training pipelines on large amounts of data.Source: §3 · p. 3

• RICE improves on PromptReps-Dense by 3.6 points with Qwen3-8B and 2.4 points with Qwen3.5-9B in Recall@100.The 3.6-point gain over RICE Zero-Shot with Qwen3.5-9B attributes the improvement to including in-context examples rather than prompt differences.Source: §4 · p. 3
• RICE achieves the highest average Recall@100 across BEIR tasks among evaluated methods that require no training.The main-results table reports Recall@100 across BEIR datasets.Source: §4 · p. 3, Table 1 · p. 4
• RICE is more effective than HyDE, PRF-UMBRELA, and CSQE, which use LLMs to improve query representations for existing unsupervised encoders.The authors suggest those methods may be constrained by the separate encoder’s representational capacity.Source: §4 · p. 4
• With Qwen3-8B, RICE performs on par on average with LLM2Vec-Gen; with Qwen3.5-9B, it improves average Recall@100 by 1.6 points.The authors say the latter improvement requires another indexing round for RICE, versus another round of self-supervised training for LLM2Vec-Gen.Source: §4 · p. 4
• RICE lags behind BGE-base-en-v1.5 and Qwen3-Embedding-8B, dense retrievers trained on large amounts of high-quality data.The authors describe RICE as a middle ground between approaches requiring no retriever training and those relying on dedicated retriever training.Source: §4 · p. 4

How many in-context examples are needed for query encoding?

• RICE accuracy generally improves as the number of in-context examples increases, but can begin to decrease when utilizing 20 examples.Source: §5 · p. 4

Dynamic exemplar selection for RICE

• Dynamic example selection improves RICE across FiQA, NFCorpus, and SciFact, with primary gains from query-side selection; document-side selection alone can hurt.BM25 retrieves the top 10 similar queries or documents from the training exemplar pool for dynamic selection.Source: §5 · p. 5, Table 2 · p. 5

Ablation on query-document exemplars

• Same-corpus example-pair variations outperform zero-shot RICE, with no clear advantage for relevant pairs over alternatives; in-domain negative examples outperform relevant cross-corpus examples.The authors suggest task- and domain-specific language may matter more than example accuracy or query–document relevance alone.Source: §5 · p. 5, Table 3 · p. 5

Ablation on representative words

• Shuffling representative-word labels or blanking them has little impact, while assigning distinct unrelated words substantially reduces RICE effectiveness.A fixed random word has a less negative effect, though its effect depends on the word and dataset; the authors suggest the model can disregard it as noise.Source: §5 · pp. 5–6, Table 4 · p. 5

• RICE derives dense retrieval representations from LLMs using only in-context examples.Source: §6 · p. 6
• RICE improves prompt-based LLM dense retrievers and outperforms other strong training-free dense retrieval baselines.Source: §6 · p. 6
• Fully supervised dense retrievers trained on large collections of high-quality relevance data remain more effective, though RICE narrows the gap.The authors describe RICE as a direction for dense retrieval systems that do not require retriever training and can easily adapt to new tasks and domains.Source: §6 · p. 6