Source-linked AI summary
A Dual Embedding Space Model for Document Ranking
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, Rich Caruana
TL;DR
Web search must match query content to relevant documents, including for new or tail queries where click evidence may be absent. The paper proposes DESM, which uses both input and output word embeddings to compare query words with document words. DESM works well on telescoped candidates, while combining it with BM25 improves ranking across all-document candidates.
Problem
Web search needs to match query content to relevant documents, especially for new or tail queries where memorized connections or click evidence may be absent.
Source: §1 · p. 1Method
DESM retains jointly learned input and output embeddings, maps query and document words into separate spaces, and aggregates their cross-space similarities.
Source: §1 · p. 2Results
On telescoped sets, DESMIN−OUT significantly outperforms BM25, LSA, and DESMIN−IN; on all-document candidates, the DESM–BM25 mixture gives the best NDCG and significantly improves over BM25.
Source: §4 · p. 6Takeaways & Limitations
DESM is suited to late-stage ranking, where precomputed document centroids keep its added computational overhead low.
Source: §6 · p. 8Standalone DESM performance depends strongly on candidate-set relevance; it needs combination with BM25 to perform well on raw, unfiltered collections.
Source: §6 · p. 8
Abstract
from arXiv · showhide
A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs. We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features.
1. INTRODUCTION
The introduction motivates content-based ranking when query–document relationships cannot be reliably inferred from clicks, and proposes using both word2vec embedding spaces to assess document aboutness. DESM compares query words with document words, including related terms, and is presented as useful both for reranking and alongside word-counting features.
1. INTRODUCTION
- Content matching matters for new or tail queries, where memorized query–document connections and click evidence may be absent or outdated.The introduction also notes that smaller services may lack enough behavior data to learn click-based embeddings or translation models.Source: §1 · p. 1
- Common BM25 counts query terms, with term frequency intended to distinguish documents about a term from those that merely mention it.The introduction contrasts this repetition-based evidence with DESM’s use of relationships between query terms and all document terms.Source: §1 · pp. 1–2
- IN–IN and OUT–OUT similarities capture functional or type similarity, whereas IN–OUT similarities capture topical co-occurrence; Figure 1 shows denser relevant documents near queries for IN–OUT.The figure uses blue for queries, yellow for relevant documents, and gray for irrelevant documents.Source: Figure 1 · p. 1, Table 1 · p. 2, §1 · p. 2
- Related-term occurrences, such as “faculty” and “alumni” for Yale, can distinguish documents about an entity from documents that merely mention it.The approach treats sufficient occurrences of related terms as evidence of aboutness.Source: §1 · p. 2
- DESM compares every query word with document words, equivalently scoring each query word against a document-embedding centroid, using separate jointly learned embeddings for queries and documents.The model is learned from an unlabelled text corpus.Source: §1 · p. 2
- On telescoped rankings DESM works alone, while general retrieval combines it with word-counting features to address potential false positives.The introduction describes reranking top documents from a commercial Web search engine and combination with word counting for general retrieval.Source: §1 · p. 2
2. DISTRIBUTIONAL SEMANTICS FOR IR
The paper uses both word2vec embedding spaces to model document aboutness, comparing query words with document-word evidence beyond exact term counts. It also identifies where this signal is useful and where false positives require a term-based ranking feature or a restricted candidate set.
2.1 Continuous Bag-of-Words
- CBOW predicts a center word from its surrounding context, learning separate IN and OUT embedding spaces through its two weight matrices.The experiments use a negative-sampling objective to make training scalable.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
2.2 Dual Embedding Space Model
- DESM compares query words with a centroid of normalized document-word vectors, using IN-OUT similarities to capture topical co-occurrence evidence.The document centroid can be precomputed, leaving query-term score contributions to be summed at ranking time.Source: §2.2 · p. 3
- DESM_IN−OUT is reported as a better indication of aboutness than BM25 and DESM_IN−IN, because it uses word-distributional properties.The authors attribute its advantage over DESM_IN−IN to topical similarity being a better indicator of aboutness than typical similarity.Source: §2.2 · p. 4
- DESM ignores out-of-vocabulary terms, unlike TF-IDF, and uses linear query- and document-length normalization while leaving more sophisticated normalization for future work.The paper notes that ignoring OOV terms may advantage TF-IDF.Source: §2.2 · pp. 4–5
- In a word-substitution example, DESM scores giraffe text low despite an inserted “Cambridge,” but also scores Oxford text highly, exposing false-positive matches.The Oxford passage contains words likely to co-occur with “Cambridge” in training data.Source: §2.2 · p. 4, Table 2 · p. 5
2.3 The Mixture Model
- Because DESM is weak on larger candidate sets, the paper proposes mixing it with a term-based feature such as BM25 for non-telescoping ranking.The mixture weight α is selected by a parameter sweep on an implicit-feedback training set.Source: §2.3 · p. 5
3. EXPERIMENTS
The experiments compare DESM with BM25 and LSA using explicit and implicit relevance judgments, under both re-ranking and broad-candidate evaluation settings. They use Bing-derived datasets and NDCG to evaluate ranking performance.
3. EXPERIMENTS
- The evaluation compares DESM with BM25 and LSA on explicit and implicit relevance judgments, using both a telescoped subset and a large document collection.Performance is measured by NDCG at different rank positions.Source: §3 · p. 5, §3.2 · p. 6
3.1 Datasets
- The explicit test set contains 7,741 queries and 171,302 unique documents judged by humans; the implicit test set contains 7,477 queries and 42,573 documents labeled by clicks.A separate implicit-feedback training set of 7,429 queries and 42,253 documents is used to tune BM25 and the mixture model.Source: §3.1 · p. 6
3.2 Experiment Setup
- Telescoped evaluation re-ranks Bing-retrieved candidates to isolate top-rank performance, while non-telescoped evaluation makes every distinct test-set document a candidate for every query.The first setup uses candidates considered at least marginally relevant by Bing; the second resembles retrieval from a large collection.Source: §3.2 · p. 6
3.3 Baseline models
- The baselines are BM25, with k1=1.7 and b=0.95, and LSA trained on 366,470 Bing-index documents with a vocabulary of 480,608 words.BM25 parameters were selected by a sweep on the implicit-feedback training set; LSA trains on a word-document matrix.Source: §3.3 · p. 6
4. RESULTS
DESM_IN−OUT performs well when re-ranking Bing results, while standalone DESMs struggle with broad candidate sets. Combining DESM_IN−OUT with BM25 gives the best reported non-telescoping NDCG, and results vary across baselines and training data.
Results
- DESM_IN−OUT is the most discriminating feature for relevant versus irrelevant Bing-retrieved documents, while BM25 better separates random irrelevant documents.The models make different errors, and the mixture combines their favorable properties.Source: §4 · p. 6
- The DESM_IN−OUT and BM25 mixture achieves the best NDCG under non-telescoping evaluation and significantly improves over BM25, while standalone DESMs perform poorly on broad candidate sets.In telescoping evaluation, DESM_IN−OUT significantly outperforms BM25, LSA, and DESM_IN on both test sets.Source: §4 · p. 6, Table 4 · p. 7
- The authors do not report telescoping mixture results because tuning α on the training set selects standalone DESM as the best-performing option.They conclude DESM is primarily suited to top-position ranking or use with other ranking features.Source: §4 · p. 7
- LSA improves slightly over BM25 on the implicit-feedback test set but loses on the explicitly judged test set.Source: §4 · p. 7
- DESM models trained on query-corpus embeddings perform significantly better across configurations than models trained on document body text.The authors hypothesize that queries filter out less important document terms, making important terms more likely to appear within the training context window.Source: §4 · p. 7
5. RELATED WORK
Related work contrasts term-counting retrieval with dense representations and describes embeddings as task-dependent. The paper positions its approach as using related terms to model aboutness alongside term repetition.
Term based IR
- Classical vector-space retrieval represents queries and documents as sparse vocabulary-sized vectors; LSA, PLSA, and LDA instead learn lower-dimensional dense representations [8] [17] [5, 47].Prior work suggests these dense models can perform poorly alone unless combined with TF-IDF-like features.Source: §5 · p. 7
- BM25 uses term frequency as evidence that a document is about a term, while this paper’s model treats occurrences of related terms as additional aboutness evidence.For example, a document about Eminem may contain related terms such as “rap” and “tracklist.”Source: §5 · p. 7
Neural embeddings for IR
- Neural embeddings have been studied across NLP tasks and connected to explicit vector spaces, matrix factorization, and density-based representations [28] [29] [42] [45].Source: §5 · p. 7
- Prior findings suggest no single embedding approach is best for every task [16]; the paper likewise distinguishes IN-IN and IN-OUT relationships and their task performance.Earlier IR work also incorporated IN-IN similarities for term expansion and reweighting.Source: §5 · p. 7
6. DISCUSSION AND CONCLUSION
DESM’s performance depends on candidate-set relevance: it ranks telescoped sets effectively on its own but needs BM25 combined with it for raw collections. The paper also argues that IN-OUT proximity better supports retrieval than IN-IN proximity.
6. DISCUSSION AND CONCLUSION
- IN-OUT proximity outperforms IN-IN for retrieval, and the authors caution against treating IN-space cosine similarity as a default measure of semantic relatedness.The input and output spaces can yield drastically different nearest neighbors.Source: §6 · p. 8
- DESM uses both CBOW embedding spaces to compare query terms with document terms and score document aboutness.The model retains the often-discarded output embeddings and aggregates query-document word-pair similarities.Source: §6 · p. 8
- On telescoped datasets, standalone DESM bests BM25 and LSA, but raw, unfiltered collections require combining DESM with BM25 for good performance.The authors note that telescoping is common in industrial IR pipelines and that DESM has little computational overhead for late-stage ranking.Source: §6 · p. 8
- The authors identify further questions about using embedding distances for feedback or query expansion, composing document representations, and filtering noisy comparisons.The noisy comparisons are associated with degraded performance on non-telescoped datasets.Source: §6 · p. 8
- The retrieval problem includes vocabulary mismatch and query-term mentions that do not establish relevance; DESM lets all document words contribute to ranking for semantic subtleties.The paper presents embeddings as an effective and efficient way to make all document words contribute.Source: §6 · pp. 8–9
- BM25 can score irrelevant documents that mention query terms and miss relevant documents that do not mention them, while embedding features also vary on random documents.The Figure 5 discussion links embedding scores on unjudged random documents to poor non-telescoping evaluation performance.Source: Figure 5 · p. 9