Source-linked AI summary
CEDR: Contextualized Embeddings for Document Ranking
Sean MacAvaney, Andrew Yates, Arman Cohan, Nazli Goharian
TL;DR
Neural ranking has emphasized scoring architectures more than the term representations supplied to them, motivating investigation of contextualized language models for ad-hoc ranking. The paper integrates ELMo and BERT representations into existing rankers, proposes the CEDR joint model with BERT’s classification vector, and reports improved benchmark ranking performance while addressing runtime costs.
Problem
The paper addresses limited attention to term representations in neural ranking and investigates whether pretrained contextualized language models can improve ad-hoc document ranking.
Method
The paper incorporates contextualized language-model layers as multiple similarity channels in existing neural rankers and combines BERT token representations with its classification vector in a joint approach.
Results
Contextualized representations improve existing neural ranking architectures, while CEDR further enhances performance and achieves state-of-the-art results on Robust 2004 and WebTrack 2012–2014.
Takeaways & Limitations
Contextualized term representations can be effectively combined with existing neural ranking architectures, and limiting processed BERT layers can retain comparable performance while improving runtime.
Takeaways & Limitations
Contextualized language models impose considerable inference-time costs, although processing only through BERT layer 5 is reported to preserve comparable PACRR performance while running more than twice as fast.
Abstract
from arXiv · showhide
Although considerable attention has been given to neural ranking architectures recently, far less attention has been paid to the term representations that are used as input to these models. In this work, we investigate how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking. Through experiments on TREC benchmarks, we find that several existing neural ranking architectures can benefit from the additional context provided by contextualized language models. Furthermore, we propose a joint approach that incorporates BERT's classification vector into existing neural models and show that it outperforms state-of-the-art ad-hoc ranking baselines. We call this joint approach CEDR (Contextualized Embeddings for Document Ranking). We also address practical challenges in using these models for ranking, including the maximum input length imposed by BERT and runtime performance impacts of contextualized language models.
1 INTRODUCTION
CEDR addresses the limited attention to term representations in neural ranking by incorporating pretrained contextualized language models into existing architectures. The approach uses contextual information and BERT’s classification vector to improve ad-hoc document ranking, while also considering computation costs.
- Motivation: Pretrained contextualized term representations are hypothesized to improve ad-hoc ranking because relevance supervision can be difficult to obtain at massive scale.ELMo and BERT generate context-dependent representations, unlike conventional static word vectors.
- Contributions: CEDR incorporates contextualized language models into existing neural ranking architectures through multiple similarity matrices, one for each language-model layer.The approach targets PACRR, KNRM, and DRMM.
- Contributions: CEDR combines BERT’s classification vector with BERT token vectors and existing neural ranking architectures to use both semantic information and contextualized term matches.The joint model is designed to obtain benefits from both BERT’s classification mechanism and existing rankers.
- Contributions: The authors report state-of-the-art performance on the Robust 2004 and WebTrack 2012–2014 datasets when incorporating contextualized language models.The reported improvement comes at the expense of computation costs.
- Practical considerations: The work also proposes partially computing contextualized representations to address their runtime impact and releases code for replication and future work.The runtime strategy limits computation rather than processing every representation fully.
2 METHODOLOGY
The methodology represents query–document matching with context-sensitive similarities across all layers of pretrained language models. It then integrates these representations, and optionally BERT’s [CLS] vector, into existing neural ranking models.
- Ranking inputs: Neural ranking architectures typically receive a query–document similarity matrix whose cells contain pairwise term-similarity scores.The conventional similarity is usually cosine similarity between query and document word vectors.
- Contextualized representations: Contextualized models generate token representations conditioned on sentence context, so the same word can receive different representations in different contexts.The paper contrasts contextualized representations with one-vector-per-word static embeddings.
- Contextualized similarity tensors: The model expands the similarity matrix into a three-dimensional tensor with one channel for each language-model layer.The tensor has dimensions L × |Q| × |D|, where L is the number of layers.
- Contextualized similarity tensors: All contextualized layers are included so neural rankers can learn which levels of contextual information matter most.The paper notes that deeper layers generally incorporate more context.
- Joint BERT approach: The joint BERT approach adds the [CLS] representation to existing rankers, combining deep semantic information with individual contextualized token matches.For dense-combination models it is concatenated with model signals; for DRMM it enters each term-score calculation.
- Joint BERT approach: The joint approach is motivated by complementary strengths: BERT uses deep semantic information, while traditional rankers exploit query-term repetition.The paper presents both signals as likely relevant to relevance ranking.
3 EXPERIMENT
The experiments evaluate contextualized representations and BERT-based ranking against neural and traditional baselines, showing ranking gains alongside substantial inference costs. Fine-tuning and the joint CEDR approach improve results, while limiting BERT processing to layer 5 preserves performance with higher throughput.
- Experimental setup: The study evaluates PACRR, KNRM, and DRMM with ELMo and BERT representations on Robust04 and WebTrack 2012–14.It also compares against BM25, SDM, ConvKNRM, Vanilla BERT, TREC-best runs, and GloVe-based models.
- Ranking results: Vanilla BERT significantly outperforms tuned BM25 and ConvKNRM, while untuned contextualized representations provide modest or comparable gains over GloVe.KNRM improves modestly, whereas PACRR and DRMM are comparable to GloVe before fine-tuning.
- Ranking results: Fine-tuning BERT significantly improves all three neural models, with PACRR and KNRM reaching comparable or higher performance than Vanilla BERT.The results indicate that fine-tuning contextualized language models is important for ranking performance.
- Ranking results: CEDR models always outperform Vanilla BERT and nearly always significantly outperform their non-CEDR counterparts across Robust04 and WebTrack 2012–14.The authors suggest that term-counting methods are complementary to BERT’s classification mechanism.
- Runtime analysis: Static GloVe processing is orders of magnitude faster than contextualized representations, while BERT is faster than ELMo.BERT’s advantage is attributed to its Transformer architecture rather than an RNN.
- Runtime analysis: Processing BERT only through layer 5 achieves performance comparable to the full model while running more than twice as fast.The approach limits sequential, query-dependent layer processing to reduce inference time.
4 CONCLUSION
The conclusion emphasizes that contextualized word embeddings can be integrated effectively into existing neural ranking architectures. It also proposes limiting processed language-model layers to improve runtime performance.
- Contextualized word embeddings can be effectively incorporated into existing neural ranking architectures.
- Limiting the number of processed language-model layers is suggested as an approach for improving runtime performance.