Source-linked AI summary
Learning Passage Impacts for Inverted Indexes
Antonio Mallia, Omar Khattab, Nicola Tonellotto, Torsten Suel
TL;DR
Contextualized language models improve neural retrieval effectiveness but make query processing expensive, motivating efficient first-stage retrieval that also addresses vocabulary mismatch. DeepImpact expands documents with DocT5Query and jointly learns contextualized term impacts for storage in a standard inverted index. It outperforms inverted-index baselines and, when paired with ColBERT re-ranking, improves effectiveness over ColBERT end-to-end while achieving a 4.4×−5.1× speedup.
Problem
Contextualized re-ranking models improve effectiveness but have high query-processing costs, while retrieval must also address vocabulary mismatch between queries and documents.
Method
DeepImpact expands documents with DocT5Query and uses a contextualized language model to jointly learn a single semantic impact score for each document term in an inverted index.
Results
DeepImpact outperforms inverted-index baselines, and DeepImpact plus ColBERT achieves higher effectiveness than ColBERT end-to-end with a 4.4×−5.1× speedup.
Takeaways & Limitations
The method provides an efficient first-stage retrieval option and supports a re-ranking pipeline with strong effectiveness and substantially reduced query-processing latency.
Takeaways & Limitations
DeepImpact's mean response time exceeds other methods because its learned score distribution is not efficiently exploited by MaxScore; query-processing optimization remains open.
Abstract
from arXiv · showhide
Neural information retrieval systems typically use a cascading pipeline, in which a first-stage model retrieves a candidate set of documents and one or more subsequent stages re-rank this set using contextualized language models such as BERT. In this paper, we propose DeepImpact, a new document term-weighting scheme suitable for efficient retrieval using a standard inverted index. Compared to existing methods, DeepImpact improves impact-score modeling and tackles the vocabulary-mismatch problem. In particular, DeepImpact leverages DocT5Query to enrich the document collection and, using a contextualized language model, directly estimates the semantic importance of tokens in a document, producing a single-value representation for each token in each document. Our experiments show that DeepImpact significantly outperforms prior first-stage retrieval approaches by up to 17% on effectiveness metrics w.r.t. DocT5Query, and, when deployed in a re-ranking scenario, can reach the same effectiveness of state-of-the-art approaches with up to 5.1x speedup in efficiency.
1 INTRODUCTION
DeepImpact addresses the high computational cost of contextualized neural re-ranking while improving inverted-index retrieval through learned term impacts and document expansion. It jointly models impacts over expanded documents to address vocabulary mismatch and reports stronger effectiveness than prior first-stage approaches.
- Motivation: Contextualized models such as BERT improve search effectiveness but impose expensive query-processing costs that challenge production deployment.Prior work demonstrates strong passage- and document-re-ranking effectiveness, alongside high computational cost.
- Prior approaches: Prior approaches either pre-compute document representations for late query-document interaction or learn contextual term weights for inverted-index retrieval.EPIC, PreTTR, ColBERT, DeepCT, and HDCT reduce query-time computation through different representation or weighting strategies.
- Prior approaches: DocT5Query expands documents with terms predicted by a sequence-to-sequence model to improve retrieval for relevant queries.The expansion strategy adds query-like terms to document representations, addressing vocabulary mismatch through document enrichment.
- Contribution: DeepImpact jointly learns final term impacts across query terms in expanded documents, rather than learning isolated term scores or retaining unchanged BM25 scoring.The method uses contextualized representations to model richer interactions among term impacts and targets vocabulary mismatch with DocT5Query expansion.
- Contributions: The paper evaluates DeepImpact on MS MARCO passage ranking and reports improved first-stage effectiveness, competitiveness with ANN-based neural retrieval, and lower computational cost.Its re-ranking pipeline is reported to match or outperform strong baselines while remaining highly efficient.
2 DEEP IMPACT FRAMEWORK
DeepImpact expands documents and uses a contextualized encoder to assign a single impact score to each unique term, storing the resulting weights in an inverted index. Its expansion and scoring design combines vocabulary coverage with learned term importance for efficient query processing.
- Document Expansion: DocT5Query expansion combines Rewrite, which changes existing-term frequencies, with Inject, which adds new terms to address term mismatch.Rewrite alone provides stronger MRR@10, Inject provides higher recall, and using both significantly outperforms either component on both measures.
- Document Expansion: DeepImpact first injects new terms with DocT5Query, then directly learns impact scores for both original and injected terms.This separates vocabulary enrichment from learning the appropriate term weighting.
- Neural Network Architecture: The contextualized encoder processes original and injected terms separately, and a two-layer ReLU MLP assigns one impact score to each unique document term.For a query, document scores are computed by summing impacts for terms shared by the query and document.
- Impact Scores Computation: After training, DeepImpact predicts query-independent term-score pairs that are converted into an inverted index for ordinary efficient query processing.The learned semantic importance of document tokens can therefore be used without queries during score computation.
- Quantization and Query Processing: Real-valued impact scores are quantized into [1, 2^b−1] and summed for matching query terms, with b=8 producing no observed precision loss.Quantization avoids storing floating-point values for every posting while preserving the scoring procedure.
3 EXPERIMENTAL RESULTS
The evaluation compares DeepImpact with inverted-index baselines and neural retrieval pipelines across first-stage, re-ranking, recall, effectiveness, and latency settings. DeepImpact achieves strong effectiveness and efficient re-ranking, although its mean response time is higher than other first-stage methods.
- Experimental setup: The experiments use MSMARCO Dev Queries, TREC 2019 queries, and TREC 2020 queries to measure effectiveness and efficiency.Reported measures include MRR@10, NDCG@10, MAP, recall, and mean response time.
- Overall comparison: DeepImpact significantly outperforms all compared methods on every effectiveness metric for MSMARCO Dev Queries.On TREC 2019 and TREC 2020, it is consistently better, with significant gains on NDCG@10 and MAP in some cases.
- Efficiency: DeepImpact has higher mean response time than other first-stage methods because its learned score distribution is not efficiently exploited by MaxScore.Additional unoptimized disjunctive-query experiments place its speed in line with other approaches, while optimization remains future work.
- Re-ranking evaluation: 4.4×–5.1× speedup is achieved by DeepImpact followed by ColBERT compared with ColBERT end-to-end.The DeepImpact-plus-ColBERT pipeline also obtains higher effectiveness values across all query sets.
- First-stage cutoff evaluation: DeepImpact achieves statistically significantly higher first-stage recall than all compared methods except at cutoff 1000.The recall gap is larger at smaller cutoffs, and DeepImpact outperforms all other methods at cutoff 10 in re-ranking.
4 CONCLUSIONS AND FUTURE WORK
The paper concludes that DeepImpact combines contextualized language models with inverted indexes to improve retrieval effectiveness while retaining efficient retrieval. It also identifies exact matching, term expansion, and score-distribution effects on query processing as directions for future work.
- Conclusions: DeepImpact combines traditional inverted indexes with contextualized language models to produce one impact score for each document token.It outperforms inverted-index baselines and can match more complex neural retrieval approaches such as ColBERT.
- Conclusions: Replacing ANN candidate generation with DeepImpact before ColBERT reduces query-processing latency and modestly improves whole-pipeline effectiveness.The conclusion describes this as a dramatic latency reduction with a more modest effectiveness improvement.
- Future work: Future work includes relaxed query-document matching, more sophisticated term expansion, and adapting query processing to learned impact-score distributions.The authors specifically mention MaxScore as an algorithm affected by the score distribution.