Source-linked AI summary
Learning to Match Using Local and Distributed Representations of Text for Web Search
Bhaskar Mitra, Fernando Diaz, Nick Craswell
TL;DR
Ad-hoc retrieval needs to handle both exact matches and semantic relationships, yet neural embedding models have often lagged traditional term-based methods. The paper jointly trains local and distributed matching networks in a duet architecture. This combination outperforms the individual models and reported neural and traditional baselines on document ranking.
Problem
Neural retrieval models have shown limited success on ad-hoc retrieval, while traditional exact-match methods and distributed representations capture different relevance signals.
Method
The duet jointly trains separate deep neural networks that match query-document pairs using local and distributed text representations.
Results
The duet significantly outperforms its individual local and distributed models, traditional IR baselines, and other neural ranking models on document ranking.
Takeaways & Limitations
Combining local exact-match signals with distributed representations provides a stronger document-ranking model than either representation alone.
Abstract
from arXiv · showhide
Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent semantic space. In traditional information retrieval models, on the other hand, terms have discrete or local representations, and the relevance of a document is determined by the exact matches of query terms in the body text. We hypothesize that matching with distributed representations complements matching with traditional local representations, and that a combination of the two is favorable. We propose a novel document ranking model composed of two separate deep neural networks, one that matches the query and the document using a local representation, and another that matches the query and the document using learned distributed representations. The two networks are jointly trained as part of a single neural network. We show that this combination or `duet' performs significantly better than either neural network individually on a Web page ranking task, and also significantly outperforms traditional baselines and other recently proposed models based on neural networks.
1. INTRODUCTION
Neural retrieval models use distributed representations, while traditional IR relies on exact term matches; the paper proposes jointly combining both through a duet architecture. The duet outperforms individual models, traditional baselines, and other neural models on document ranking.
- Motivation: Neural embedding models have had limited success on ad-hoc retrieval and sometimes underperform traditional term-based approaches such as BM25.Long document bodies and limited training data make effective neural document ranking challenging.
- Motivation: Traditional models use local representations and exact query-term matches, whereas latent semantic models compare distributed representations in a continuous space.These representations capture different aspects of query-document relevance.
- Approach: The duet architecture jointly trains separate local and distributed deep neural networks within one neural network.The local model captures exact-match position and proximity, while the distributed model detects synonyms, related terms, and content well-formedness.
- Results: The duet outperforms both individual subnetworks and produces large improvements over traditional baselines and other DNN-based ranking models.The paper reports this outcome on the document ranking task.
- Results: Training with judged-irrelevant documents as negatives is more effective than randomly sampling negatives from the corpus.
2. DESIDERATA OF DOCUMENT RANKING
Effective document ranking should combine exact matching, match position, and inexact matching to address both rare-term reliability and vocabulary mismatch. The paper positions joint learning as a way to combine these signals.
- Exact matching: Exact query-document term matches are fundamental and remain especially important for new or rare queries.Traditional models such as BM25 can use exact matches with little or no training data.
- Match positions: Match positions indicate both where relevant content occurs and how clustered query-term matches are within the document.Figure 2 contrasts match patterns in human-judged relevant and irrelevant documents.
- Inexact matching: Inexact matching addresses vocabulary mismatch by using related non-matching terms as additional relevance evidence.Terms such as “Sydney” and “koala” can support a query about “Australia.”
- Combining signals: Effective retrieval systems often combine exact matching with proximity or inexact matching, while the proposed method jointly learns parameters across these techniques.
3. THE DUET ARCHITECTURE
The duet architecture jointly trains local and distributed subnetworks that score query–document relevance through complementary matching representations. The local model captures exact matches and positions, while the distributed model learns embedding-based inexact matches before combining their scores.
- Architecture: The duet architecture jointly trains separate local and distributed DNNs to model query–document relevance.The two subnetworks cooperate within a single neural network.
- Combining Scores: The final duet score is the sum of the local and distributed network scores.The two matching signals are combined after each subnetwork produces its score.
- Local Model: The local model builds an interaction matrix that records exact query-term matches at document positions.It uses one-hot term representations and compares every query term with every document term.
- Local Model: The local interaction matrix is processed by convolutional and fully connected layers to produce one relevance score.The convolutional layer uses 300 filters, followed by two fully connected layers, dropout, and a final fully connected layer.
- Distributed Model: The distributed model learns dense, lower-dimensional query and document representations and computes positional similarity in the learned embedding space.It uses character n-graph frequency vectors rather than one-hot term encodings.
- Distributed Model: The distributed pipeline uses convolution and window-based max pooling to distinguish matches across different document regions.The document representation contains 899 span-level embeddings, while the query produces one embedding.
4. MATERIALS AND METHODS
The experiments compare the duet model with individual subnetworks and established neural and non-neural baselines using Bing query-log datasets. Evaluation uses human-judged documents and NDCG at positions one and ten, with both frequent-query and uniformly sampled test sets.
- Experimental Design: The experiments test duet performance against separate local and distributed models, existing ranking baselines, and random versus judged negative documents.The study defines three experiments covering model effectiveness, baseline comparisons, and negative-document sampling.
- Data: The training set contains 199,753 query–document instances sampled from Bing search logs between January 2012 and September 2014.Human judges rated documents on a five-point relevance scale, and body text came from Bing’s Web document index.
- Data: The test data include frequency-weighted queries and uniformly sampled unique queries, representing repeated-query and less-popular-query settings.The weighted set was sampled from October through December 2014, while the unweighted set reduces bias toward popular queries.
- Baselines: The baselines include BM25 and query likelihood for exact matching, the dependence model for proximity, and neural models for inexact matching.The neural baselines include DESM, DSSM, CDSSM, and DRMM variants.
- Evaluation: Evaluation averages NDCG at positions one and ten over queries, with paired t-tests and Bonferroni correction for significance testing.The protocol evaluates ranking quality at two cutoff positions and corrects for multiple comparisons.
5. RESULTS
The duet model outperforms its individually trained local and distributed components and improves significantly over baseline systems. Judged irrelevant documents are more effective negative examples than randomly sampled documents.
- The duet model performs significantly better than the individual local and distributed models on both test datasets under NDCG evaluation.The local and distributed NDCG values come from individually trained networks, whereas the duet trains both jointly.
- Judged irrelevant documents produce significantly better duet performance than randomly sampled corpus documents as negative examples on both test sets.The distributed model also gains significantly on the weighted set, but its gain on the unweighted set is not statistically significant.
- The duet model significantly improves over the strongest traditional and neural baselines on both test sets at NDCG@1 and NDCG@10.CDSSM and DESM are strongest on different weighted-test metrics, while DRMM is the best baseline on the unweighted test set.
- The independent local and distributed models are competitive with existing baselines, while their joint combination provides a significant boost.
6. DISCUSSION
The discussion links model behavior to query characteristics, retrieval properties, training data, and deployment cost. It reports benefits from judged negatives, larger datasets, and caching, while finding that retrieval properties matter more than neural versus non-neural implementation.
- Judged nonrelevant documents outperform random negatives, likely because topically similar confusable documents better support staged ranking.The explanation is presented as a speculation about discrimination after an earlier retrieval stage.
- Figure 5 groups performance by query length and by the training-data frequency of each query’s rarest term, including unseen terms.
- Distributed-representation models outperform term-match-only models on the weighted test set, while rare navigational terms favor exact matching.The discussion contrasts queries requiring related terms such as “ESPN” with rare-term queries such as “pekarovic land company.”
- The PCA compares models by per-query retrieval performance using markers for exact-term, proximity, and inexact matching; it does not separate models by neural architecture.The analysis represents each model by NDCG values across 2,000 sampled weighted-test queries.
- Larger training datasets particularly improve the duet and distributed models, suggesting that still larger datasets may improve performance further.BM25 requires little training, whereas the deep models benefit from additional data.
- Caching document representations can mitigate much of the runtime cost of deploying the distributed subnetwork at Web-search scale.The distributed document matrix can be precomputed and stored, leaving only later operations at runtime.
7. RELATED WORK
Prior retrieval methods use either local exact-match representations or distributed semantic representations, while related neural models often focus on short text. This paper learns query-document representations end-to-end from ranking labels and combines both matching types for Web-page ranking.
- Local representations encode entities with distinct units, whereas distributed representations encode concepts through patterns across multiple active units.Local representations favor precise storage, while distributed representations provide greater robustness to noise and generalization.
- Traditional exact-match systems rank documents using query-term matches, proximity, weighting, and normalization without requiring large training sets.
- Distributed term representations address vocabulary mismatch by recognizing related terms that do not exactly match the query.Examples include treating “Sydney” and “koala” as evidence for a query about “Australia.”
- Earlier neural retrieval work often combines local and distributed matching matrices for short-text similarity, whereas this paper targets document body text.
- This paper learns text representations end-to-end from query-document ranking labels rather than primarily from surrounding corpus context.
8. CONCLUSION
The paper proposes a duet model that combines local exact matching with distributed semantic matching in two jointly trained deep neural networks. It outperforms solo models and diverse baselines, with larger datasets identified as a potential source of further gains.
- The duet architecture combines separate local- and distributed-representation DNNs for document ranking.
- The duet achieves higher performance than either solo model and significant improvements over traditional IR and other shallow or deep neural baselines.
- The analysis indicates that much larger datasets may produce further improvements in model performance.