Source-linked AI summary
Expansion via Prediction of Importance with Contextualization
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, Ophir Frieder
TL;DR
Short passages lack the term-frequency signals available to longer documents, making relevance identification difficult. EPIC models term importance and expands lexicon-grounded representations using a contextualized language model. On MS-MARCO passage ranking, it improves on prior state-of-the-art approaches and narrows the effectiveness gap with costly BERT re-ranking.
Problem
Short passages lack the term-frequency signals available to longer documents, making it difficult to estimate term importance for passage relevance.
Source: §1 · p. 1Method
Results
EPIC improves on prior state-of-the-art passage retrieval on MS-MARCO and narrows the effectiveness gap with more expensive BERT re-ranking.
Source: §1 · p. 1Takeaways & Limitations
Abstract
from arXiv · showhide
The identification of relevance with little textual context is a primary challenge in passage retrieval. We address this problem with a representation-based ranking approach that: (1) explicitly models the importance of each term using a contextualized language model; (2) performs passage expansion by propagating the importance to similar terms; and (3) grounds the representations in the lexicon, making them interpretable. Passage representations can be pre-computed at index time to reduce query-time latency. We call our approach EPIC (Expansion via Prediction of Importance with Contextualization). We show that EPIC significantly outperforms prior importance-modeling and document expansion approaches. We also observe that the performance is additive with the current leading first-stage retrieval methods, further narrowing the gap between inexpensive and cost-prohibitive passage ranking approaches. Specifically, EPIC achieves a MRR@10 of 0.304 on the MS-MARCO passage ranking dataset with 78ms average query latency on commodity hardware. We also find that the latency is further reduced to 68ms by pruning document representations, with virtually no difference in effectiveness.
1 INTRODUCTION
Short passages lack the term-frequency signals available to longer documents, making relevance identification difficult. EPIC models term importance and expands contextualized, lexicon-grounded representations for passage ranking.
1 INTRODUCTION
- Short passages lack term-frequency signals that document retrieval can use to estimate term importance, making passage relevance identification difficult.Traditional retrieval approaches often perform poorly on passages.Source: §1 · p. 1
- EPIC improves on prior state-of-the-art passage retrieval on MS-MARCO and narrows the effectiveness gap with more expensive BERT re-ranking.The approach is representation-based, with document representations precomputed at index time.Source: §1 · p. 1
- EPIC models query and document term importance and expands representations over a contextualized language model, grounding dimensions in the lexicon.The lexicon-grounded dimensions make the representations interpretable.Source: §1 · pp. 1–2
- Precomputing document representations at index time reduces query-time cost, and EPIC’s added latency can be as low as 5ms per query.The reported total latency in that setting is 68ms.Source: §1 · pp. 1–2
- Unlike document-expansion methods that predict added terms and importance methods that replace term frequencies, EPIC models query and document importance while expanding documents.The introduction contrasts EPIC with doc2query, docTTTTTquery, and DeepCT-Index.Source: §1 · pp. 1–2
2 MODEL
EPIC builds independent query and document representations from contextualized term embeddings, modeling term importance and expanding documents into vocabulary space. It compares the representations by similarity, with document representations computable at index time.
2 MODEL
- EPIC learns independent query and document representations in a fixed-length vector space and ranks them by similarity.Source: §2 · p. 2
- Contextualized language-model embeddings support query vectors that weight query terms by their contextual importance.The query representation is sparse, with non-query terms set to zero.Source: §2 · p. 2
- Document expansion projects each contextualized term embedding into vocabulary space and weights predicted terms by document-term importance.Source: §2 · p. 2
- The document-quality factor uses the contextual model’s [CLS] embedding and helps assign lower values to poor-quality documents.Source: §2 · p. 2
- Each document-vocabulary coordinate takes the maximum score generated by any document term, so representations can be computed at index time.Source: §2 · p. 2
3 EXPERIMENTAL EVALUATION
On MS-MARCO, combining EPIC with docTTTTTquery achieves an MRR@10 of 0.304. Pruning retains ranking effectiveness while reducing representation costs, and lexicon-grounded dimensions make the representations inspectable.
Experimental setup
- Experiments use MS-MARCO's roughly 1 million questions and 8.8 million passages, training on official triples with BERT-base and evaluating MRR@10.The dataset averages 0.7 relevance judgments per query and is shallowly annotated.Source: §3 · pp. 2–3
Ranking effectiveness
- MRR@10 reaches 0.304 when EPIC is combined with docTTTTTquery, significantly improving on either approach alone and narrowing the gap to BERT Large.EPIC alone performs comparably to docTTTTTquery (0.273 versus 0.277, with no statistically significant difference).Source: §3 · p. 3, Table 1 · p. 3
- On TREC CAR 2017, BM25 with EPIC raises MRR from 0.235 to 0.353, above DeepCT's reported 0.332 [2].Source: §3 · p. 3
Pruning and efficiency
- Approximately 74% of document-representation values are 0.1 or below, motivating pruning to reduce storage with little impact on performance.Source: §3 · p. 3, Figure 2 · p. 3
- At r = 1000, pruning reduces vocabulary by 96.7% without a statistically significant effectiveness difference; EPIC adds 5ms per query to docTTTTTquery and stores the collection in 34 GiB.By contrast, effectiveness drops considerably at r = 100, with MRR@10 values of 0.241 for BM25 and 0.285 for docTTTTTquery.Source: §3 · p. 4
Interpretability of representations
- Lexicon-grounded dimensions make representations inspectable, and EPIC's expansion of “cost” was critical for scoring a passage relevant to “cost of endless pools/swim spa.”The document visualization shows in-passage terms and top expansion terms alongside docTTTTTquery terms.Source: §3 · p. 4, Figure 4 · p. 4
4 CONCLUSION
EPIC is an inexpensive passage reranker built from lexicon-grounded representations generated by contextualized language models. The authors report competitive reranking performance and identify document-retrieval generalization as future work.
4 CONCLUSION
- EPIC provides an effective, inexpensive passage-reranking technique based on lexicon-grounded representations from contextualized language models.Source: §4 · p. 4
- The approach further approaches fully BERT-based reranking performance while offering low query-time latency and interpretable representations.Source: §4 · p. 4
- Pruning can reduce query latency and precomputed passage-representation size without sacrificing effectiveness; generalization to document retrieval remains future work.Source: §4 · p. 4