Source-linked AI summary
Precise Zero-Shot Dense Retrieval without Relevance Labels
Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan
TL;DR
Fully zero-shot dense retrieval is difficult without relevance labels or test-time training corpora. HyDE addresses this by generating hypothetical documents and encoding them with an unsupervised contrastive retriever, outperforming Contriever-only zero-shot retrieval across diverse tasks and languages.
Problem
Fully zero-shot dense retrieval requires learning query and document representations without relevance supervision, while large judged datasets cannot always be assumed.
Method
HyDE prompts an instruction-following language model to generate a hypothetical document, then uses an unsupervised contrastive encoder to retrieve similar real documents.
Results
HyDE significantly outperforms the previous state-of-the-art Contriever-only zero-shot system across 11 query sets covering multiple tasks and languages.
Takeaways & Limitations
HyDE delegates relevance modeling and instruction understanding to a language model, removing the need for relevance labels within the evaluated zero-shot setting.
Takeaways & Limitations
The setup assumes no access to query sets, document sets, or relevance judgments for defining the retrieval functions, and HyDE leaves ambiguous-query diversity for future work.
Abstract
from arXiv · showhide
While dense retrieval has been shown effective and efficient across tasks and languages, it remains difficult to create effective fully zero-shot dense retrieval systems when no relevance label is available. In this paper, we recognize the difficulty of zero-shot learning and encoding relevance. Instead, we propose to pivot through Hypothetical Document Embeddings~(HyDE). Given a query, HyDE first zero-shot instructs an instruction-following language model (e.g. InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is unreal and may contain false details. Then, an unsupervised contrastively learned encoder~(e.g. Contriever) encodes the document into an embedding vector. This vector identifies a neighborhood in the corpus embedding space, where similar real documents are retrieved based on vector similarity. This second step ground the generated document to the actual corpus, with the encoder's dense bottleneck filtering out the incorrect details. Our experiments show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers, across various tasks (e.g. web search, QA, fact verification) and languages~(e.g. sw, ko, ja).
1 Introduction
HyDE targets fully zero-shot dense retrieval without relevance labels by combining instruction-following generation with unsupervised document encoding. It substantially outperforms Contriever-only zero-shot retrieval across tasks and languages.
- Motivation: Fully zero-shot dense retrieval remains difficult because relevance supervision and large judged datasets such as MS-MARCO are not always available.The paper seeks systems that work out of the box and generalize across tasks without relevance supervision.
- Approach: HyDE decomposes retrieval into generation by an instruction-following language model and document-document similarity modeling by a contrastive encoder.The query prompts a hypothetical document, which is then encoded for retrieval.
- Approach: Generated hypothetical documents capture relevance patterns despite being unreal and potentially factually incorrect.The subsequent dense representation is intended to filter incorrect details.
- Training setting: HyDE leaves the generative model and contrastive encoder intact, with supervision used only to teach the backbone language model instruction following.No model is trained or fine-tuned specifically for HyDE.
- Results: HyDE significantly outperforms the previous state-of-the-art Contriever-only zero-shot system on 11 query sets spanning web search, question answering, fact verification, and Swahili, Korean, and Japanese.The comparison covers multiple tasks and languages rather than a single benchmark setting.
2 Related Works
The paper situates HyDE against supervised dense retrieval, instruction-aware encoders, and generative retrieval. Its setup avoids relevance labels, training data, and access to test-time corpora while using standard dense retrieval infrastructure.
- Instruction-following language models: Instruction-following language models can zero-shot generalize to new tasks from instructions, providing the generative component used by HyDE.The related work describes instruction execution learned through supervised sequence-to-sequence learning or reinforcement learning.
- Instruction-following language models: Unlike task-aware retrieval, HyDE uses an unsupervised encoder and handles task instructions through an instruction-following generative language model.The comparison concerns where task-specific instruction handling occurs.
- Zero-shot dense retrieval: Prior zero-shot retrieval work commonly transfers dense retrievers trained on richly supervised MS-MARCO data.The paper notes that such a large collection cannot always be assumed.
- Zero-shot dense retrieval: HyDE studies dense retrieval without relevance labels and without assuming access to test-time corpora for training.The authors describe this as a more realistic setup that prevents over-engineering on test corpora.
- Generative retrieval: Unlike generative search systems, HyDE uses a standard MIPS index, requires no training or training data, and generates an intermediate hypothetical document rather than document identifiers.The hypothetical document is encoded before retrieving real corpus documents.
3 Methodology
HyDE avoids learning query-document relevance directly by generating hypothetical documents and searching in an unsupervised document embedding space. Contrastive encoding grounds those hypotheses to real corpus documents.
- 3.1 Preliminaries: Standard dense retrieval maps queries and documents into a shared vector space whose inner product measures similarity.The zero-shot challenge is learning both embedding functions without relevance judgments.
- 3.2 HyDE: HyDE circumvents this challenge by searching document-only embedding space, where document-document similarity can be learned with unsupervised contrastive learning.The document encoder is set directly to a contrastive encoder.
- 3.2 HyDE: An instruction-following language model takes a query and textual instruction to perform the specified task.HyDE uses this model to construct the query-side hypothesis.
- 3.2 HyDE: HyDE samples hypothetical documents using the instruction “write a paragraph that answers the question,” requiring relevance patterns rather than factual grounding.The generated documents may be factually ungrounded, while relevance modeling is delegated to the language model.
- 3.2 HyDE: The method assumes a unimodal distribution of query representations and leaves ambiguous queries and diversity for future work.The representation is estimated by sampling N hypothetical documents.
- 3.2 HyDE: The contrastive encoder maps the hypothetical document into a dense vector, filters extra details through a lossy bottleneck, and retrieves the most similar real documents.Inner products are computed against corpus document vectors.
4 Experiments
HyDE is evaluated across web search, low-resource BEIR tasks, and multilingual retrieval, using standard and fine-tuned systems as comparisons. It consistently improves unsupervised Contriever and remains competitive with supervised retrievers, though multilingual gains leave some gap to fine-tuned mContriever.
- Experimental Setup: HyDE uses InstructGPT with Contriever or mContriever, evaluated on web search, six low-resource BEIR datasets, and Swahili, Korean, Japanese, and Bengali retrieval.Different dataset-specific instructions control the form of generated hypothetical documents.
- Experimental Setup: HyDE is compared with unsupervised Contriever models, BM25, and retrievers fine-tuned or transferred from MS-MARCO.The HyDE and Contriever systems share embedding spaces; their query-vector construction differs.
- Web Search: HyDE brings sizable improvements to Contriever across TREC DL19/20 precision-oriented and recall metrics, and outperforms BM25 by large margins.Unsupervised Contriever can underperform BM25 on these web-search tasks.
- Web Search: On TREC DL19, HyDE has comparable map and ndcg@10 to ContrieverFT and the best recall@1k; on DL20, it is around 10% lower on map and ndcg@10 with similar recall@1k.ANCE has better ndcg@10 but lower recall than HyDE.
- Low-Resource Retrieval: Across low-resource BEIR tasks, HyDE improves Contriever on ndcg and recall and is outperformed by BM25 only on TREC-Covid, by a 0.2 margin.On TREC-Covid, underlying Contriever underperforms BM25 by more than 50%.
- Low-Resource Retrieval: HyDE generally outperforms ANCE and DPR on low-resource tasks, while ContrieverFT performs better on FiQA and DBPedia, possibly because instructions are under-specified.The paper suggests more elaborative instructions may help.
- Multilingual Retrieval: On Mr.Tydi, HyDE improves mContriever and can outperform non-Contriever models fine-tuned and transferred from MS-MARCO, but remains behind fine-tuned mContrieverFT.The authors hypothesize that non-English languages are under-trained in pre-training and instruction learning.
5 Analysis
The analysis varies HyDE’s instruction-following language model and encoder. Larger instruction models improve unsupervised Contriever more, while weaker instruction models can slightly hurt fine-tuned retrievers and InstructGPT can improve them.
- Analysis Setup: The analysis studies smaller instruction-following language models and fine-tuned encoders on TREC DL19/20.These experiments examine how HyDE changes when its generative and encoding backbones are replaced.
- Effect of Different Generative Models: Cohere’s 52-billion and FLAN’s 11-billion instruction models both improve unsupervised Contriever, with larger models bringing larger improvements.The paper only tentatively attributes performance differences partly to training techniques because Cohere was experimental.
- HyDE with Fine-tuned Encoder: Less powerful instruction language models slightly reduce fine-tuned retriever performance, whereas InstructGPT further improves it, especially on DL19.The degradation remains small, and the result suggests the generative model captures factors absent from the fine-tuned encoder.
- HyDE with Fine-tuned Encoder: HyDE with a fine-tuned encoder is not the intended usage because HyDE is described as more powerful and irreplaceable when few relevance labels are present.The analysis nevertheless tests how HyDE embeddings affect fine-tuned encoders.
6 Conclusion
The paper presents HyDE as a new interaction paradigm in which language generation captures relevance and dense retrieval grounds generated content in corpus representations. It argues that this removes the need for relevance labels and can provide practical zero-shot performance before supervised search logs accumulate.
- HyDE can be as effective as dense retrievers that learn numerical relevance scores in many cases.
- HyDE introduces a new interaction paradigm between large language models and dense encoders or retrievers.
- HyDE delegates part of relevance modeling and instruction understanding to a more powerful and flexible language model.
- The paper argues that this delegation removes the need for relevance labels.
- The paper presents multi-hop retrieval or question answering and conversational search as directions for further generalization.
- At a search system’s beginning, HyDE offers performance comparable to a fine-tuned model, after which supervised retrieval can be gradually deployed as logs grow.
A.1 Instructions
The appendix defines task-specific language-model instructions for generating passages in web search, scientific claim verification, counterargument generation, and financial question answering.
- A.1.1 Web Search: In A.1.1 Web Search, the model is instructed to write a passage answering a question.
- A.1.1 Web Search: The web-search template marks the input with a Question field and requests a Passage output.
- The supplied appendix passages specify task prompts rather than retrieval outcomes or benchmark results.
- Both templates ask the language model to generate a passage conditioned on a structured input.
- A.1.2 SciFact: In A.1.2 SciFact, the model is instructed to write a scientific paper passage supporting or refuting a claim.
- A.1.2 SciFact: The SciFact template provides a Claim field and requests a Passage response.
A.1.3 Arguana
The supplied Arguana appendix passages define prompts for generating counterarguments and scientific passages answering questions.
- The Arguana appendix includes a prompt requesting a counterargument to a supplied passage.
- The counterargument template labels the input as Passage and the requested output as Counter Argument.
- A separate template requests a scientific paper passage answering a supplied question.
- The question-answering template labels the input as Question and the output as Passage.
A.1.5 FiQA
The supplied FiQA appendix passages define prompts for generating financial articles and passages answering questions.
- The FiQA appendix includes a prompt requesting a financial article passage answering a supplied question.
- The financial-article template labels the input as Question and the requested output as Passage.
- The two FiQA templates differ in output framing: one requests a financial article, while the other requests a passage.
- A separate template requests a passage answering a supplied question without specifying a financial-article style.
A.1.7 TREC-NEWS
The supplied passages provide generation instructions rather than TREC-NEWS results: one requests a news passage, while the other requests a detailed response in several languages.
- One instruction requests writing a news passage about a specified topic.
- Neither passage reports retrieval metrics, datasets, comparisons, or empirical findings for TREC-NEWS.
- A second instruction requests a detailed answer to a specified question in Swahili, Korean, Japanese, or Bengali.