Source-linked AI summary
Document Expansion by Query Prediction
Rodrigo Nogueira, Wei Yang, Jimmy Lin, Kyunghyun Cho
TL;DR
Vocabulary mismatch makes exact retrieval difficult when query terms differ from relevant document terms. Doc2query predicts relevant queries for documents and appends them before indexing; combined with reranking, it reaches leading retrieval results, while direct retrieval is substantially faster than neural reranking.
Problem
Vocabulary mismatch occurs when users’ query terms differ from those used in relevant documents, challenging exact-term retrieval.
Method
Doc2query trains a sequence-to-sequence transformer on query–relevant-document pairs to predict queries for each document and append them before indexing.
Results
The method improves retrieval effectiveness by approximately 15% on both datasets; with BERT reranking it achieves the best-known TREC-CAR results and near-state-of-the-art MS MARCO results.
Takeaways & Limitations
Without reranking, Doc2query improves over a bag-of-words baseline while shifting neural inference costs from retrieval to indexing.
Takeaways & Limitations
The top leaderboard entries lack system descriptions, preventing direct comparison with them.
Abstract
from arXiv · showhide
One technique to improve the retrieval effectiveness of a search engine is to expand documents with terms that are related or representative of the documents' content.From the perspective of a question answering system, this might comprise questions the document can potentially answer. Following this observation, we propose a simple method that predicts which queries will be issued for a given document and then expands it with those predictions with a vanilla sequence-to-sequence model, trained using datasets consisting of pairs of query and relevant documents. By combining our method with a highly-effective re-ranking component, we achieve the state of the art in two retrieval tasks. In a latency-critical regime, retrieval results alone (without re-ranking) approach the effectiveness of more computationally expensive neural re-rankers but are much faster.
1 Introduction
The paper addresses vocabulary mismatch by enriching document representations before indexing rather than expanding queries. For question answering, Doc2query predicts questions a document might answer and appends them to the document.
- Vocabulary mismatch arises when users’ query terms differ from those in relevant documents.
- Most scalable search engines still use exact term matching for initial retrieval despite neural models’ semantic ranking potential.
- Doc2query enriches document representations before indexing instead of enriching query representations while documents remain static.
- For question answering, a sequence-to-sequence model generates possible questions that each document might answer.
- The method was presented as the first successful neural-network application of document expansion and was competitive on MS MARCO and best-known on TREC CAR.
- Because enrichment occurs before indexing, expanded retrieval results can be returned directly without expensive neural inference at retrieval time.
2 Related Work
Earlier approaches addressed vocabulary mismatch through feedback, query expansion, and term-relationship modeling, while neural rankers offered semantic alternatives. The paper instead uses neural networks to augment documents before conventional term-based retrieval.
- Keyword matching dominated earlier retrieval systems because they relied on discrete one-hot representations.
- Relevance feedback, query expansion, and statistical translation modeled vocabulary mismatch within keyword-based retrieval constraints.
- Document expansion enriches document representations and has worked particularly well for speech and multilingual retrieval with noisy terms.
- Document expansion is less amenable to rapid experimentation because changing the technique typically requires re-indexing the corpus.
- Neural ranking models are generally used to rerank candidates because applying inference over every corpus document is impractical.
- This work uses neural networks before indexing while leaving subsequent term-based retrieval unchanged.
3 Method: Doc2query
Doc2query predicts queries for which each document will be relevant using a sequence-to-sequence transformer trained on query–document pairs. Ten sampled queries are appended to every document before BM25 retrieval, with optional BERT reranking.
- Doc2query predicts a set of queries for which each document will be relevant.
- A sequence-to-sequence transformer maps document terms to queries using query–relevant-document training pairs.
- Documents are truncated to 400 tokens and queries to 100 tokens to avoid excessive memory usage.
- The trained model generates 10 queries with top-k random sampling, which are appended to each document without special markup.
- Expanded documents are indexed and retrieved with BM25, with optional BERT reranking of the retrieved documents.
4 Experimental Setup
The experiments evaluate BM25, Doc2query document expansion, query expansion, and BERT reranking on MS MARCO and TREC-CAR. MS MARCO uses MRR@10, while TREC-CAR uses MAP.
- The study trains and evaluates models on the MS MARCO and TREC-CAR datasets.
- MS MARCO contains 8.8M passages, approximately 500k training query–document pairs, and development and test sets of about 6,900 queries each.
- TREC-CAR queries concatenate a Wikipedia article title with a section title, and relevant documents are paragraphs within that section.
- TREC-CAR uses approximately 3M training queries, 700k validation queries, and a 2,250-query test set.
- The evaluated methods include BM25, BM25 with Doc2query, BM25 with BERT, and combinations with Doc2query, BERT, or RM3.
- MS MARCO is evaluated with MRR@10, whereas TREC-CAR is evaluated with MAP.
5 Results
Doc2query improves retrieval effectiveness, supports strong re-ranking results, and offers a faster retrieval-only alternative. Its gains reflect complementary term copying and related-word expansion, improved candidate recall, and advantages over query expansion.
- BM25 + Doc2query improves retrieval effectiveness by ∼15% for both datasets.With BERT re-ranking, it achieves the best-known results on TREC CAR and approaches the state of the art on MS MARCO.
- The top leaderboard entries cannot be compared directly because they do not include system descriptions.
- 31% of predicted query words are new and 69% are copied from the original document.Using both types yields MRR@10 21.5, compared with 18.8 using only new words and 19.7 using only copied words.
- Recall@1000 rises from 85.3 with BM25 to 89.3 with BM25 + Doc2query on the MS MARCO development set.The improved candidate pool gives BERT more relevant documents to identify and rank highly.
- RM3 query expansion hurts retrieval on both datasets, whereas document expansion is more effective in these precision-oriented tasks.The authors suggest longer documents provide more signals to exploit.
- BM25 + Doc2query takes 90 ms versus 50 ms for baseline BM25 and is approximately seven times faster than a neural re-ranker.The re-ranker has a three points higher MRR@10, making the retrieval-only speed-quality trade-off potentially useful in some operating scenarios.
6 Conclusion
The paper presents neural document expansion as a successful approach that shifts neural inference from retrieval to indexing. Its simple open-source implementation supports reproducibility and further development.
- The paper presents the first successful use of document expansion based on neural networks.
- Neural document expansion can shift neural-network inference costs from retrieval to indexing.
- The implementation integrates OpenNMT, Anserini, and TensorFlow BERT to aid reproducibility and future improvements.
Appendix A Architecture and Training Details
The Doc2query transformer uses the base Transformer architecture and is trained with specified optimization, regularization, and schedule settings.
- The transformer has 6 encoder layers, 6 decoder layers, 512 hidden units, 8 attention heads, and 2048 feedforward hidden units.
- Training uses batches of 4096 tokens for up to 30 epochs with Adam and a learning rate of 10^-3.The supplied passage also specifies β1 = 0.9, β2 = 0.998, L2 weight decay of 0.01, and 8,000-step warmup.
Appendix B Evaluating Various Decoding Schemes
The decoding comparison evaluates beam search and top-k random sampling across beam sizes. Sampling is slightly better across tested beam sizes, with performance peaking at an intermediate number of generated queries.
- Top-k random sampling is slightly better than beam search across all tested beam sizes.
- Retrieval effectiveness peaks at an intermediate number of generated queries.The authors conjecture that too few queries reduce semantic-match diversity, while too many add noise and dilute the original text.