Source-linked AI summary
Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
HongChien Yu, Chenyan Xiong, Jamie Callan
TL;DR
Short, ambiguous queries make it difficult for dense retrievers to encode a user’s information need. ANCE-PRF refines a query embedding using the query and top ANCE-retrieved documents, while reusing the document index. It outperforms ANCE across the reported benchmarks, with a 9.3% NDCG@10 improvement on DL-HARD.
Problem
Concise, ambiguous queries make it challenging for dense retrievers to encode the underlying information need accurately.
Source: §1 · p. 1Method
Results
ANCE-PRF outperforms ANCE across the reported datasets, including a 9.3% NDCG@10 improvement on DL-HARD.
Source: §1 · p. 1, §5.1 · p. 3Takeaways & Limitations
The encoder attends more to relevant PRF content, especially terms complementary to the query, while distinguishing relevant from irrelevant documents.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Dense retrieval systems conduct first-stage retrieval using embedded representations and simple similarity metrics to match a query to documents. Its effectiveness depends on encoded embeddings to capture the semantics of queries and documents, a challenging task due to the shortness and ambiguity of search queries. This paper proposes ANCE-PRF, a new query encoder that uses pseudo relevance feedback (PRF) to improve query representations for dense retrieval. ANCE-PRF uses a BERT encoder that consumes the query and the top retrieved documents from a dense retrieval model, ANCE, and it learns to produce better query embeddings directly from relevance labels. It also keeps the document index unchanged to reduce overhead. ANCE-PRF significantly outperforms ANCE and other recent dense retrieval systems on several datasets. Analysis shows that the PRF encoder effectively captures the relevant and complementary information from PRF documents, while ignoring the noise with its learned attention mechanism.
1 INTRODUCTION
Short, often ambiguous queries make it difficult for dense retrievers to encode a user’s information need. ANCE-PRF uses top-ranked documents as pseudo relevance feedback (PRF) to refine query embeddings, and experiments report stronger retrieval accuracy than ANCE and several recent dense systems.
1 INTRODUCTION
- Dense retrieval depends on embeddings capturing the information need, but concise, ambiguous queries make that intent challenging to infer.The passage says even powerful pre-trained language models cannot be expected to perfectly embed an information need from a few query terms.Source: §1 · p. 1
- PRF enriches a query with top retrieved documents, and its efficacy has been observed across search scenarios in sparse retrieval.Prior approaches use query expansion or query-term reweighting.Source: §1 · p. 1
- ANCE-PRF uses a BERT encoder to combine a query with top-ranked documents from ANCE, learns query embeddings from relevance labels, and reuses ANCE’s document index.The index reuse avoids duplicating index storage.Source: §1 · p. 1
- ANCE-PRF is consistently more accurate than ANCE and several recent dense retrieval systems on MS MARCO and TREC Deep Learning passage ranking benchmarks.The paper also reports large improvements on DL-HARD and places ANCE-PRF among the best-performing first-stage systems on the MARCO passage ranking leaderboard.Source: §1 · p. 1
- The authors attribute ANCE-PRF’s advantages to using relevant PRF information while ignoring irrelevant-document noise; its attention favors relevant and query-complementary terms.A case study reports that the encoder focuses on PRF terms complementary to query terms when representing search intents.Source: §1 · p. 1
2 RELATED WORK
Prior dense retrieval work emphasizes training strategies and efficient single-vector document representations, while neural PRF work explores several ways to incorporate feedback. A parallel approach expands multi-vector query representations with clustered feedback embeddings.
2 RELATED WORK
- Dense retrieval research has focused particularly on training strategies such as negative sampling, while this work focuses on one-vector-per-document models for retrieval efficiency.Other approaches use one vector per document token.Source: §2 · p. 2
- Neural PRF approaches have combined feedback relevance scores, modified query-document interaction using encoded feedback documents, or learned contextualized query-document interactions.The passage cites these approaches as [21], [1] [3], and [35], respectively.Source: §2 · p. 2
- A parallel approach expands multi-vector query representations using feedback embeddings extracted with a clustering technique [31].Source: §2 · p. 2
3 METHOD
ANCE-PRF enriches the query embedding with top documents retrieved by ANCE, then retrieves again using the refined query. Its training uses relevance information and attention to learn from feedback while retaining the original document embeddings and index.
3 METHOD
- A typical dense retriever scores a query-document pair by taking the similarity of BERT-style query and document embeddings.The final-layer [CLS] embeddings serve as the query and document vectors, and the base model is fine-tuned with ranking losses and negative sampling.Source: §3 · p. 2
- ANCE-PRF feeds the query and the top k ANCE documents to a new encoder to produce a refined query embedding, then performs another retrieval.Source: §3 · p. 2
- The PRF query encoder is trained with standard negative log-likelihood using embeddings of relevant and irrelevant documents.Only the PRF query embedding is newly learned; document embeddings from the initial dense retrieval model are reused.Source: §3 · p. 2
- The encoder uses Transformer attention to identify relevant PRF information, with the [CLS] representation attending to input tokens.The prose identifies the attention query and token key vectors; the method distinguishes meaningful PRF tokens from irrelevant ones.Source: §3 · p. 2
- With document embeddings and the index unchanged, ANCE-PRF adds one query-encoder pass and one nearest-neighbor search.The paper describes these as a minor addition to dense retrieval.Source: §3 · p. 2
4 EXPERIMENTAL SETUP
The experiments use MS MARCO training and evaluation data, plus TREC Deep Learning and DL-HARD benchmarks, with sparse and dense retrieval baselines. Training inherits ANCE settings, and inference retrieves ANCE feedback documents before searching again with the refined query.
4 EXPERIMENTAL SETUP
- MS MARCO passage training uses 530K queries, with evaluation on a 7K-query dev set and leaderboard testing; its official metric is MRR@10.Source: §4 · p. 2
- TREC DL uses 2019 and 2020 labeled queries and reports NDCG@10, Recall@1K, and HOLE@10; DL-HARD contains 50 curated queries and uses NDCG@10.HOLE@10 reflects coverage of pooled labels on dense retrieval systems.Source: §4 · p. 2
- Baselines include BM25 and RM3, alongside dense retrievers DPR, DE-BERT, ME-BERT, LTRe, and ANCE.The passage distinguishes their training or encoding strategies, including hard-negative mining, in-batch negatives, and multi-vector encoding.Source: §4 · p. 2
- Training initializes the query encoder from ANCE FirstP and keeps ANCE document embeddings and its negative index; inference retrieves ANCE top-k documents, encodes the refined query, and searches again.The results tables cover overall ranking, accuracy at varying PRF depth with Avg_Rel, and DL-HARD results.Source: §4 · p. 2, Table 1 · p. 3, Table 2 · p. 3
5 EXPERIMENTAL RESULTS
ANCE-PRF improves retrieval over ANCE across datasets, with gains on difficult queries and under deeper feedback. Analyses show selective use of relevant, complementary PRF information, while the case study also illustrates how repeated feedback text can mislead it.
5.1 Overall Results
- ANCE-PRF outperforms ANCE across MS MARCO and TREC DL, improves DL-HARD NDCG@10 by 9.3%, and raises R@1K about 5% on both TREC DL sets.The authors report the DL-HARD gain as an indication of advantage on queries challenging for neural systems.Source: §5.1 · p. 3
- ANCE-PRF rankings are more precise than sparse baselines; RM3 has best R@1K on both TREC DL sets, with less improvement on DL-HARD.Source: §5.1 · p. 3
- ANCE-PRF outperforms strong dense baselines and is most accurate on almost all datasets, with results comparable to ME-BERT without its multi-vector overhead [24].Source: §5.1 · p. 3
5.2 Ablation on PRF Depths
- Models with k>0 beat ANCE (k=*) and k=0; gains persist for k=1 to 5 despite noisy deeper PRF documents.The authors identify noisy relevance information in PRF documents at k>1 as a known challenge for traditional PRF.Source: §5.2 · p. 3
5.3 Analyses of Embedding Space & Attention
- On MARCO dev, ANCE-PRF query embeddings are closest to the original query, then relevant documents, and farther from irrelevant documents.About one out of three PRF documents is relevant in this study.Source: §5.3 · p. 3
- As training proceeds, attention shifts toward relevant over irrelevant feedback documents, while the original query remains most attended.Source: §5.3 · p. 4
5.4 Case Study
- The case study shows complementary terms can clarify query intent, but repeated misleading information across feedback documents can attract excess attention [19].In the losing example, repeated “pattern their shotguns” text drew too much attention for ANCE-PRF to make a better choice.Source: §5.4 · p. 4
6 CONCLUSION
ANCE-PRF refines query representations with top-ranked documents, learning to distinguish useful feedback from irrelevant material. The authors present it as a plug-in approach for embedding-based retrieval and a direction for further work on PRF.
6 CONCLUSION
- ANCE-PRF refines query representations using the original query and top-ranked documents from ANCE.Source: §6 · p. 4
- Its encoder distinguishes relevant from irrelevant feedback, emphasizing relevant portions and terms complementary to the query while showing robustness to imperfect feedback.Source: §6 · p. 4
- ANCE-PRF is a plug-in for embedding-based retrieval; the authors conclude that classic PRF can improve neural retrieval and merits further research.Source: §6 · p. 4