Source-linked AI summary
Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach
Saar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, Marc Najork
TL;DR
Retrieval-stage neural methods had received limited study, despite their potential to complement lexical retrieval. This paper combines semantic and lexical retrieval and reports that their merged results improve recall on a TREC collection, with the two approaches retrieving complementary relevant documents.
Problem
Neural IR research largely emphasized re-ranking, while semantic models for retrieval had been rarely studied and lexical matching can miss relevant documents without query terms.
Source: §1 · p. 1Method
Results
On a public TREC collection, semantic retrieval found relevant documents missed by lexical retrieval, and merging the lists significantly improved recall.
Source: §1 · p. 2Takeaways & Limitations
The analysis supports the complementary nature of semantic and lexical retrieval for improving retrieval-stage recall.
Source: §1 · p. 2, §5 · p. 8The study does not fully optimize the semantic model and leaves evaluation of other semantic models and retrieval tasks for future work.
Source: §3.1 · p. 3, §5 · p. 8
Abstract
from arXiv · showhide
Search engines often follow a two-phase paradigm where in the first stage (the retrieval stage) an initial set of documents is retrieved and in the second stage (the re-ranking stage) the documents are re-ranked to obtain the final result list. While deep neural networks were shown to improve the performance of the re-ranking stage in previous works, there is little literature about using deep neural networks to improve the retrieval stage. In this paper, we study the merits of combining deep neural network models and lexical models for the retrieval stage. A hybrid approach, which leverages both semantic (deep neural network-based) and lexical (keyword matching-based) retrieval models, is proposed. We perform an empirical study, using a publicly available TREC collection, which demonstrates the effectiveness of our approach and sheds light on the different characteristics of the semantic approach, the lexical approach, and their combination.
1 INTRODUCTION
This work studies semantic retrieval as a complement to lexical retrieval, targeting recall at the retrieval stage rather than precision at re-ranking. Its TREC analysis reports that merging the two result lists improves recall and that the approaches retrieve complementary relevant documents.
1 INTRODUCTION
- The study targets retrieval-stage recall, where systems must retrieve an initial document set efficiently, rather than re-ranking precision at high ranks.Lexical models such as BM25 are common at retrieval because term matching supports recall and inverted indexes are efficient.Source: §1 · p. 1
- Lexical matching can miss relevant documents with no query terms and may not reflect whether results address users’ information needs.One example contrasts a BM25 result containing “fatality” with a semantic result using the related term “kill.”Source: §1 · p. 1
- Semantic retrieval had received limited study at the retrieval stage, partly because soft matching may lower recall and neural scoring across collections was costly.Fast approximate KNN search [18] made retrieval more practical.Source: §1 · p. 1
- The proposed hybrid runs semantic and lexical retrieval in parallel, then merges their result lists into the initial list for re-ranking.The authors describe the approach as efficient because the retrievals can run in parallel.Source: §1 · pp. 1–2
- The semantic model uses weakly supervised tasks to learn corpus-specific semantics without large-scale query logs, supporting settings where logs cannot be collected.The approach adds a second retrieval source and uses approximate KNN search; the authors say it is not expected to hurt lexical performance or system latency.Source: §1 · p. 2
- The TREC analysis found that semantic retrieval can surface relevant documents missed by lexical retrieval, and merging the lists significantly improves recall.The analysis also characterizes the retrieved documents as complementary.Source: §1 · p. 2
2 RELATED WORK
Prior neural IR work emphasized re-ranking, while this paper examines hybrid lexical-semantic retrieval for recall. It situates its approach among other retrieval methods and distinguishes the retrieval-stage combination from earlier hybrid re-ranking.
2 RELATED WORK
- This work studies lexical-semantic hybrid retrieval to improve retrieval-stage recall, unlike much neural IR work focused on re-ranking precision.A prior lexical-semantic hybrid approach [26] addressed re-ranking; this paper studies the hybrid at retrieval.Source: §2 · p. 2
- Earlier semantic retrieval without neural networks used LSI or statistical-translation KNN [6], with LSI showing limited retrieval-stage effectiveness.The present work focuses on neural-network-based approaches.Source: §2 · p. 2
- Earlier neural retrieval work includes sparse representations and embedding-based KNN [19], whose document-specific representations were described as unsuitable for large collections.The paper instead studies general integration and says its approach can be applied atop semantic models.Source: §2 · p. 2
- BERT-based IR work treated query and document as consecutive sentences for re-ranking [31] [33], whereas this paper uses BERT to generate retrieval embeddings.The representation-based approach is described as more applicable to retrieval.Source: §2 · pp. 2–3
- Neural retrieval has also been studied in QA, conversational agents, and product search [2] [23] [30] [42], while document expansion [34] and query expansion offer other recall-oriented approaches.The paper notes that document expansion had limited effectiveness for document retrieval, and pseudo-relevance feedback requires sequential retrieval steps.Source: §2 · p. 3
3 A HYBRID RETRIEVAL APPROACH
The approach runs lexical and semantic retrieval in parallel and merges their candidate documents into the initial list. Its semantic component uses learned embeddings and weak supervision, while RM3 combines the pooled candidates.
3 A HYBRID RETRIEVAL APPROACH
- Parallel lexical and semantic searches each return c documents, which are pooled and merged to select c candidates for the initial list.The system uses an inverted lexical index and a semantic embeddings matrix; BM25 supplies lexical retrieval and approximate KNN searches semantic embeddings.Source: §3 · p. 3
- The hybrid design aims to avoid extra retrieval overhead by running the independent searches in parallel and using approximate KNN for semantic retrieval.The authors expect approximate KNN to be as efficient as inverted-index search [6].Source: §3 · p. 3
3.1 Semantic Retrieval
- The semantic model is not fully optimized; the study prioritizes testing the benefits of combining result lists, leaving comparisons among semantic models for future work..Source: §3.1 · p. 3
3.1.1 Neural Model Architecture.
- A shared BERT [14] module and separate query and document dense layers produce embeddings whose dot product predicts relevance.The architecture uses query-independent document representations for indexing, and the loss combines cross-entropy, mean squared error, and masked-language-model objectives.Source: §3.1.1 · pp. 3–4
3.1.2 Training data.
- The end-to-end weak-supervision pipeline generates retrieval training data without auxiliary resources, addressing the difficulty of obtaining large labeled datasets.The authors identify generalizing to semantic models learned from labeled data as future work.Source: §3.1.2 · p. 4
3.1.3 Retrieval.
- After training, passage embeddings are indexed offline; at retrieval time, passage scores are summed by document to produce document scores.This converts passage-level semantic retrieval results to the document level.Source: §3.1.3 · p. 4
3.2 Hybrid Merging
- The merger is intended to preserve relevant documents from both sources because semantic scores may be less discriminative than lexical scores.The authors note semantic retrieval can find similar text, while lexical scoring can better distinguish exact matches.Source: §3.2 · p. 4
- RM3 scores documents pooled from both retrieval lists, then selects the top c as the initial result list.RM3 is induced from lexical results because semantic scores are less discriminative; scoring only pooled candidates is not expected to greatly increase query-processing time.Source: §3.2 · pp. 4–5
4.1 Experimental Setup
The evaluation uses a TREC news collection and restricts queries to those supported by the semantic model’s vocabulary. It compares lexical and hybrid retrieval using recall-oriented measures and query-level robustness.
4.1.1 Data set.
- The evaluation uses 441,676 TREC news-wire documents from disks 1&2 and titles of topics 51–200 as queries.The collection was selected for systematic textual analysis with minimal noise and reliable relevance judgments; evaluation on larger labeled collections was left for future work.Source: §4.1.1 · p. 5
4.1.3 Semantic model implementation.
- The semantic model was trained from about 1B passage-query examples and used approximate in-memory KNN search for retrieval.The implementation used a six-layer BERT architecture with hidden size 256, four attention heads, and a 7,500-word vocabulary.Source: §4.1.1 · p. 5, §4.1.3 · p. 5
4.1.2 Lexical model implementation.
- The lexical baseline is BM25 implemented with Anserini, using default parameters, Krovetz stemming, and stopword removal.Only 121 queries with all terms in the semantic model’s vocabulary were evaluated; other queries were left for future work.Source: §4.1.2 · p. 5
4.1.4 Evaluation measures.
- The evaluation reports recall, MAP, and total relevant documents retrieved (#rel) at result-list sizes c∈{500, 1000, 1500, 2000}.Robustness is measured with Reliability of Improvement, and paired two-tailed t-tests use pval<0.05 for significance.Source: §4.1.4 · p. 5
4.2 Experimental Results
The hybrid retrieval approach combines lexical and semantic results: lexical retrieval has higher recall alone, but semantic results add relevant documents and improve recall when merged. Analyses show robust gains across queries and complementary document and query characteristics.
4.2.2 Hybrid approach performance.
- The hybrid approach improves recall over lexical retrieval by about 3% at all result-list sizes, with statistically significant differences between methods in each block.The reported recall improvements are also highly robust across queries.Source: Table 3 · p. 6, §4.2.2 · p. 6
4.2.1 The potential benefits of the hybrid approach.
- Lexical retrieval has higher recall than semantic retrieval alone, but adding semantic results yields large, significant recall improvements across result-list sizes.The analysis treats this replacement experiment as an upper bound for hybrid performance.Source: §4.2.1 · pp. 5–6
4.2.2 Hybrid approach performance.
- About 700 unique relevant documents are added by semantic retrieval across the tested lexical-list lengths, including longer lists clipped to their original length.This indicates that longer lexical lists did not eliminate the semantic model’s added relevant documents in this analysis.Source: §4.2.2 · p. 6, Figure 3 · p. 6
4.2.3 Robustness analysis.
- For a 1,000-document list, hybrid retrieval improves recall for 50% of queries, leaves it unchanged for 40%, and degrades it for 20%.Average gains for improved queries are around 18%, while decreases for degraded queries are about 4%.Source: §4.2.3 · p. 6, Figure 4 · p. 7
- Hybrid recall gains are largest for the poorest lexical-performing quarter, averaging 14% for Q1, and only slight for the highest-performing quarter, Q4.The query set is split into four equally sized groups based on lexical-model performance.Source: §4.2.3 · p. 6, Table 4 · p. 6
- Better-performing hybrid query groups have higher mean and maximum idf values and longer queries; the authors offer possible explanations involving lexical term weighting and neural use of multiple words.The query-property comparisons are reported by group.Source: §4.2.3 · p. 7, Table 5 · p. 7
4.2.4 An analysis of relevant documents.
- Semantic results can cover distinct or narrower topics than lexical results, and semantic-retrieved relevant documents are often longer, sometimes by a large margin.The authors suggest semantic approaches may better retrieve long relevant documents; the length comparison pools five documents per query, 250 overall.Source: §4.2.4 · pp. 7–8, Figure 6 · p. 8
- Semantic and lexical relevant-document vocabularies have low overlap, consistent across term-list lengths and stable across queries.The quantitative analysis uses 50 queries with at least five relevant documents from each model, considering the first five relevant documents per query.Source: Table 7 · p. 7, §4.2.4 · p. 8
5 CONCLUSIONS
The paper proposes a general hybrid approach combining semantic and lexical models for ad hoc retrieval and empirically studies their complementary nature. Future work includes more sophisticated merging, document-structure-aware representations, and evaluation on other retrieval tasks.
5 CONCLUSIONS
- The paper’s contribution is a systematic study of combining semantic and lexical models for ad hoc document retrieval and a general hybrid approach leveraging both.Source: §5 · p. 8
- The empirical analysis demonstrates hybrid effectiveness and sheds light on the complementary nature of semantic and lexical models.Source: §5 · p. 8
- Future work includes more sophisticated result-list merging, document-structure-aware representations instead of breaking long documents into short passages, and evaluation on question answering, recommendation, and conversational tasks.Source: §5 · p. 8