PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval

Zhuyun Dai, Jamie Callan

arXiv:1910.10687v2cs.IR

TL;DR

Frequency-based term weighting is unreliable when text has flat term distributions, motivating better estimates of which terms are central in passages and long queries. DeepCT maps BERT contextualized representations to context-aware term weights, with DeepCT-Index supporting indexed passage retrieval and DeepCT-Query producing weighted queries. Experiments report substantial first-stage retrieval gains, including up to 50% improvement for DeepCT-Index and reduced candidate requirements for a BERT re-ranker.

  • Problem

    Frequency signals do not reliably identify central terms in passages and long queries, especially when their distributions are flat.

  • Method

    DeepCT maps BERT-based contextualized representations to context-aware term weights for passage indexing and query weighting.

  • Results

    Up to 50% improvement in accuracy was reported for two first-stage retrieval algorithms with DeepCT-Index.

  • Takeaways & Limitations

    DeepCT weights are compatible with ordinary inverted indexes and popular first-stage retrieval models, while DeepCT-Query improves longer-query accuracy.

  • Takeaways & Limitations

    BERT limits DeepCT inputs to 512 text tokens.

Abstract

from arXiv · show

Term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. This paper proposes a Deep Contextualized Term Weighting framework that learns to map BERT's contextualized text representations to context-aware term weights for sentences and passages. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for passage retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms. This is novel because most deep neural network based ranking models have higher computational costs, and thus are restricted to later-stage rankers. Experiments on four datasets demonstrate that DeepCT's deep contextualized text understanding greatly improves the accuracy of first-stage retrieval algorithms.

1 INTRODUCTION

First-stage retrieval commonly relies on frequency-based term weights, but flat frequency distributions make those signals poor indicators of contextual importance. DeepCT uses BERT representations to estimate context-aware term weights for passages and queries, improving first-stage retrieval.

  • 1 INTRODUCTION: First-stage ranking fetches candidate documents efficiently, typically using Boolean, probabilistic, or vector-space bag-of-words models over an inverted index.These pipelines may apply more expensive re-ranking afterward.
  • 1 INTRODUCTION: Frequency-based weights can misidentify importance when terms occur with similar frequencies, especially in sentences and short passages.Two passages mentioning “stomach” twice can receive similar scores even though only one is about stomach.
  • 1 INTRODUCTION: Deep contextualized representations encode a token as a function of the entire text, capturing semantic and syntactic variation across contexts.BERT provides the contextualized representations used by DeepCT.
  • 1 INTRODUCTION: DeepCT learns a supervised mapping from BERT-based contextualized representations to term weights that vary according to each term’s context.The framework supports text inputs of up to 512 tokens, limited by BERT’s input length.
  • 1 INTRODUCTION: DeepCT-Index predicts passage terms likely to appear in relevant queries, then stores scaled integer weights in an ordinary inverted index for efficient retrieval.The prediction is query-independent and can be performed offline during indexing.
  • 1 INTRODUCTION: DeepCT representations improve first-stage retrieval accuracy by estimating term importance from context rather than frequency alone.The framework also generates weighted queries for long-query retrieval through DeepCT-Query.

2 RELATED WORK

Prior work improved term weighting with frequency refinements, graphs, embeddings, and neural ranking, but contextualized term importance remained insufficiently addressed for efficient early retrieval. DeepCT builds on contextual representations while retaining inverted-index compatibility.

  • 2 RELATED WORK: Bag-of-words models such as BM25 and query likelihood remain foundational because they combine efficiency with effective retrieval.Their term weighting generally uses frequency-based signals such as tf, ctf, and df.
  • 2 RELATED WORK: Graph-based document weighting ranks terms using co-occurrence graphs and algorithms such as PageRank, then indexes the resulting scores.These methods are prominent alternatives to frequency-based document weighting.
  • 2 RELATED WORK: Feature-based query weighting can improve accuracy but may require collection-dependent features and pseudo-relevant feedback, increasing computational cost.DeepTR instead predicts query term weights from query content using word embeddings.
  • 2 RELATED WORK: Embedding-based methods encode some word meaning, but classic word embeddings assign the same representation regardless of textual context.This limits their ability to model context-dependent term importance.
  • 2 RELATED WORK: DeepCT leverages BERT to explicitly model query and document term importance for efficient first-stage retrieval using discrete word representations.This targets a gap left by neural approaches concentrated on re-ranking.
  • 2 RELATED WORK: Many neural ranking models are impractical for first-stage retrieval because continuous representations allow queries to match every document to some degree.Dense approximate-nearest-neighbor and sparse latent-word approaches address efficiency but introduce accuracy or vocabulary tradeoffs.

3 DEEP CONTEXTUALIZED DOCUMENT AND QUERY TERM WEIGHTING

DeepCT learns context-aware term weights from BERT representations, then applies them to passage indexing and long-query weighting for first-stage retrieval. Its framework supports offline passage weighting and online weighted queries while retaining compatibility with standard retrieval models.

  • 3.1 DeepCT Framework: DeepCT combines BERT contextualized word embeddings with linear regression to predict context-dependent term importance scores.The same term can receive different estimated importance depending on its surrounding context.
  • 3.1 DeepCT Framework: DeepCT is trained per token by minimizing mean square error between predicted and ground-truth term weights.Predicted weights are generally in [0, 1], while negative predictions are discarded during query or document weighting.
  • Motivation and Scope: DeepCT targets flat frequency distributions because frequency alone may fail to distinguish central terms from frequent but off-topic mentions.The framework is presented as a general approach whose learned notion of importance depends on the ground-truth weights used for training.
  • 3.2 Index Passages with DeepCT: DeepCT-Index predicts passage-term importance from relevant query-passage pairs and computes the weights offline during indexing.At inference, only the passage is required, so the resulting weights can be stored before online retrieval.
  • 3.2 Index Passages with DeepCT: TFDeepCT scales predicted passage weights to integers and replaces TF values in an ordinary inverted index searchable by BM25 or query likelihood.Terms with predicted weight zero can also be pruned as a side effect of the indexing representation.
  • 3.3 Query Term Weighting with DeepCT: DeepCT-Query predicts the importance of terms in long queries and uses those weights to generate weighted bag-of-words and sequential dependency queries.Training uses relevant query-document pairs, whereas online inference requires only the query text.

4 EXPERIMENTAL METHODOLOGY FOR DeepCT-Index

The DeepCT-Index experiments evaluate passage retrieval across MS MARCO and TREC-CAR using standard first-stage rankers, multiple indexing baselines, embedding variants, and later-stage re-rankers. The methodology compares contextual BERT-based weighting with frequency-based, graph-based, expansion-based, and alternative embedding approaches.

  • Datasets: Experiments use MS MARCO and TREC-CAR, collections selected because passage-length texts have flat term-frequency distributions and fit BERT’s 512-token limit.MS MARCO contains 8.8M passages averaging around 55 words, while TREC-CAR contains 29.7M passages averaging 61 words.
  • Baselines: Baselines include a standard TF index, TextRank term weighting, and Doc2Query neural document expansion.Doc2Query was the best-performing first-stage ranking method on MS MARCO when the paper was written.
  • Experimental Indexing Methods: The experimental indexing methods include DeepCT-Index and variants that replace BERT with context-independent word embeddings or ELMo.The ELMo variant changes both the contextual representation model and its pre-training architecture and task.
  • Indexing, Retrieval, and Evaluation: First-stage retrieval uses BM25 and query likelihood with Jelinek-Mercer smoothing, implemented with Anserini and tuned on 500 training queries.Conv-KNRM and a BERT re-ranker are used for later-stage re-ranking.
  • Indexing, Retrieval, and Evaluation: Ranking and re-ranking are evaluated primarily with MRR@10, with MAP at depth 1,000 additionally reported for TREC-CAR.MRR@10 is the official MS MARCO evaluation metric.
  • DeepCT Settings: BERT initialization and training settings are dataset-specific, including an overlap-filtered pre-trained model for TREC-CAR.The DeepCT models are trained for 3 epochs after initialization with pre-trained BERT parameters.

5 DeepCT-Index RESULTS

DeepCT-Index substantially improves first-stage retrieval by assigning context-aware weights that emphasize central passage terms rather than relying on frequency alone. These gains extend across retrieval models and improve the accuracy and efficiency of downstream re-ranking.

  • Retrieval accuracy: 27% on MS MARCO and 46% on TREC-CAR: DeepCT-Index improved BM25 over baseline indexes, with similar gains for QL.It improved 25–35% of queries and hurt 8–16%.
  • Retrieval accuracy: DeepCT-Index outperformed the neural Doc2Query baseline, demonstrating the value of explicit term weighting over implicit passage expansion.Both methods were trained using the same data.
  • Sources of effectiveness: BERT contextualization produced the most effective term weights, while non-contextual word2vec weights were less effective than tf and ELMo was intermediate.The comparison supports the importance of contextual information in DeepCT weighting.
  • System comparisons: DeepCT-Index BM25 matched or exceeded several ranking and re-ranking systems while avoiding a separate re-ranking stage.It was more accurate than feature-based learning-to-rank and K-NRM, and had similar accuracy to Duet V2 and Conv-KNRM with greater efficiency.
  • Downstream re-ranking: DeepCT-Index improved downstream re-ranking: Conv-KNRM MRR@10 rose from 0.256 to 0.278 while the re-ranking depth fell from 100 to 50.For BERT Re-Ranker, top-100–200 DeepCT-Index passages produced similar MRR@10 to the top 1,000 tf-index passages.
  • Downstream re-ranking: DeepCT-Index reduces re-ranking depth by 5× to 10×, addressing the latency and resource costs of deep neural re-rankers.The paper links this reduction to making neural re-ranking practical in latency- and resource-sensitive systems.
  • Sources of effectiveness: DeepCT-Index emphasizes central terms and suppresses off-topic terms, lowering BM25 scores for passages that mention query concepts but are not relevant.In one case, less than 10% of weight went to “susan boyle” in an off-topic passage, while “troll” received the central emphasis.

6 EXPERIMENTAL METHODOLOGY AND RESULTS FOR DeepCT-Query

DeepCT-Query is evaluated on Robust04 and Gov2 across short, sentence-long, and passage-long queries using established query structures and term-weighting baselines. Longer queries benefit most from contextual term weighting, while short title queries show little benefit.

  • Datasets: Experiments use Robust04 and Gov2, whose topics contain short title, sentence-long description, and passage-long narrative queries.Robust04 contains 0.5M documents and 249 topics; Gov2 contains 25M web pages and 150 topics.
  • Baselines: The study compares BOW and SDM query structures with tf, DeepTR, and Oracle term weights.Oracle weights represent an upper limit based on ground-truth term recall weights.
  • Evaluation: Retrieval uses Indri with stemming and stop-word filtering, five-fold cross-validation, query likelihood, and NDCG@20 and MAP@1000.DeepCT-Query and DeepTR are trained and evaluated with five-fold cross-validation.
  • Results: Short title queries do not benefit from term weighting because their few keywords are typically all essential and provide little context.The passage suggests external query information may be needed to understand short queries.
  • Results: Weighted queries outperform unweighted queries for description and narrative queries, with DeepCT-Query usually more accurate than DeepTR.Improvements are larger for narratives than descriptions; BERT-based modeling helps more complex queries.

7 CONCLUSION

DeepCT transfers BERT-based contextual understanding into efficient term weights for first-stage retrieval, supporting both indexed passages and weighted queries. The resulting representations improve retrieval accuracy and can make later-stage neural reranking more efficient.

  • CONCLUSION: DeepCT addresses the limits of frequency-based weighting by estimating term importance from each word’s specific context.This targets flat frequency distributions in sentence-long queries and passage-long documents.
  • CONCLUSION: BERT supplies contextual representations, and supervised per-token regression maps them to term weights using relevance-based query-document signals.The training signals align predicted weights with the retrieval task.
  • CONCLUSION: DeepCT-Index stores integer document-term weights in ordinary inverted indexes, while DeepCT-Query weights query terms for first-stage retrieval.The weighted representations are compatible with BM25 and query likelihood.
  • CONCLUSION: Up to 50%: DeepCT-Index improves the accuracy of two popular first-stage retrieval algorithms.BM25 on DeepCT-Index can match several previous state-of-the-art multi-stage systems.
  • CONCLUSION: A BERT-based reranker achieves similar accuracy with 5× fewer candidate documents after DeepCT-Index improves the initial ranking.This improves the accuracy/efficiency tradeoff for computation-intensive rerankers.
  • CONCLUSION: DeepCT transfers deep neural text understanding into simple signals that early-stage ranking systems can efficiently consume.Its analysis emphasizes central words even when mentioned once and suppresses frequent non-central words.
Loading 1910.10687v2…