PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Behaviors of BERT in Ranking

Yifan Qiao, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu

arXiv:1904.07531v4cs.IRcs.CL

TL;DR

The paper asks how BERT’s surrounding-context pre-training transfers to ranking tasks with different demands. It evaluates several BERT rankers on MS MARCO and TREC, then analyzes their attentions and matching behavior. BERT is highly effective for QA-focused passage ranking through cross-sequence interactions, but click-based pre-training better fits TREC-style ad hoc document ranking.

  • Problem

    The paper examines whether BERT’s surrounding-context pre-training suits both QA-focused passage ranking and ad hoc document ranking.

  • Method

    The authors evaluate representation-based and interaction-based BERT rankers on MS MARCO and TREC, analyzing learned attentions and term matches.

  • Results

    BERT significantly outperforms prior Neu-IR models on MS MARCO through cross question-passage interactions, but TREC rankers do not significantly outperform LeToR and trail a Bing click-pre-trained Neu-IR model.

  • Takeaways & Limitations

    BERT’s surrounding-context pre-training fits QA-focused seq2seq matching, whereas user clicks are better pre-training signals for TREC-style ad hoc document ranking.

  • Takeaways & Limitations

    TREC labels alone were insufficient to fine-tune BERT or train other neural rankers to outperform SDM, so neural methods were first pre-trained on MS MARCO.

Abstract

from arXiv · show

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.

1 INTRODUCTION

The paper examines how BERT can be used for ranking and finds sharply different behavior across question-answering passage ranking and ad hoc document ranking. Its analyses attribute these differences to BERT’s interaction patterns, attentions, and pre-training signals.

  • Research scope: The study explores representation-based, interaction-based, and combined BERT rankers on MS MARCO passage ranking and TREC Web Track ad hoc ranking.MS MARCO ranks answer passages for questions, whereas TREC ranks ClueWeb documents for keyword queries.
  • Main findings: BERT fine-tuning significantly outperforms prior Neu-IR models on MS MARCO, with effectiveness mostly coming from cross question-passage interactions.This contrasts with earlier views that surrounding-context-trained models were less effective for search relevance modeling.
  • Main findings: On TREC ad hoc ranking, BERT rankers perform worse than feature-based learning to rank and a click-pre-trained Neu-IR model, even after MS MARCO pre-training.The result indicates that BERT’s surrounding-context pre-training does not match this ranking setting as well as click-based signals.
  • Behavioral analysis: BERT propagates information globally through Transformer attention, unlike interaction-based rankers that operate more individually on term pairs.Its matching behavior resembles surrounding-context-based seq2seq models but differs from relevance matches learned from user clicks.
  • Behavioral analysis: BERT focuses more on document terms that directly match query terms, while click-trained neural rankers learn different relevance-match patterns.The comparison concerns semantic matching behavior rather than only benchmark effectiveness.

2 BERT BASED RANKERS

The paper implements several BERT ranking variants using either separate query-document representations or concatenated sequences with cross-token interactions. These designs test how BERT’s layers, contextual embeddings, and matching features contribute to ranking.

  • Model family: The four BERT rankers use pre-trained BERT representations of the query, document, or concatenated query-document sequence marked by [SEP].The variants are BERT-Rep, BERT-Last-Int, BERT-Mult-Int, and BERT-Term-Trans.
  • BERT ranking models: BERT-Rep separately represents the query and document with final-layer [CLS] embeddings, then scores them using cosine similarity.This makes BERT-Rep a representation-based ranker.
  • BERT ranking models: BERT-Last-Int concatenates query and document sequences, uses the final-layer [CLS] embedding as matching features, and combines them linearly.Its Transformer cross-match attentions include query-document term-pair interactions, making it interaction-based.
  • BERT ranking models: BERT-Term-Trans constructs query-document translation matrices from cosine similarities between projected contextual embeddings, then combines matrices across layers.This adds a neural ranking network over BERT-derived matching features.
  • Training: The models are fine-tuned from Google’s BERT-Large using classification loss to predict whether each query-document pair is relevant.Pairwise ranking loss produced no observed difference in the experiments.

3 EXPERIMENTAL METHODOLOGIES

The experiments evaluate BERT-based rankers on MS MARCO passage reranking and ClueWeb ad hoc ranking, using standardized candidates, metrics, baselines, and training procedures. Neural methods for ClueWeb are generally pre-trained on MS MARCO, with a click-trained Conv-KNRM variant included for comparison.

  • Datasets: The study uses MS MARCO passage reranking and TREC Web Track ClueWeb ad hoc ranking tasks.MS MARCO contains question-like queries and answer-passage relevance labels; ClueWeb uses 200 TREC queries with relevance judgments.
  • Datasets: ClueWeb experiments use 10-fold cross-validation and re-rank the top 100 Galago SDM candidates.The setup follows prior research, including shared preprocessing and candidate documents.
  • Training Setup: Because TREC labels alone were insufficient for neural methods to outperform SDM, all neural methods were first pre-trained on MS MARCO before ClueWeb fine-tuning.This establishes the cross-task pre-training protocol used for the ad hoc ranking experiments.
  • Evaluation Metrics: MS MARCO is evaluated with MRR@10, while ClueWeb is evaluated with NDCG@20 and ERR@20.MS MARCO development results re-rank BM25 candidates; ClueWeb uses TREC Web Track’s official metrics.
  • Compared Methods: Baselines include BM25 or Galago-SDM candidate retrieval, feature-based LeToR, K-NRM, and Conv-KNRM.Conv-KNRM (Bing) uses the same model pre-trained on Bing user clicks, whereas other ClueWeb neural methods use MS MARCO pre-training.
  • Implementation Details: BERT rankers are fine-tuned with classification loss, while pairwise ranking loss produced no observed difference.Fine-tuning uses BERT-Large; convergence typically takes about one day, compared with shorter times for K-NRM and Conv-KNRM.
  • Results Reporting: Table 1 reports relative performance percentages against LeToR and marks significance against Base, LeToR, K-NRM, and Conv-KNRM.The table distinguishes MS MARCO and ClueWeb results and identifies the pre-training source for ClueWeb neural methods.

4 EVALUATIONS AND ANALYSES

BERT performs strongly on MS MARCO through cross query-document interactions but does not transfer as effectively to ClueWeb ad hoc ranking. Analyses show deep attention contextualization alongside concentrated, semantically aligned term matches.

  • 4.1 Overall Performances: All interaction-based BERT rankers improve Conv-KNRM by 30%-50% on MS MARCO, while BERT (Rep) performs close to random without cross-sequence interactions.These results support using BERT primarily as an interaction-based matching model rather than a representation model.
  • 4.1 Overall Performances: More complex Multi-Int and Term-Trans architectures perform worse than the simpler BERT (Last-Int) despite extensive MARCO fine-tuning.The authors suggest that substantially modifying pre-trained BERT may require end-to-end training and more accessible training methods.
  • 4.1 Overall Performances: None of the BERT models significantly outperforms LeToR on ClueWeb, whereas Conv-KNRM pretrained on Bing user clicks performs best.The paper attributes this contrast to different signal requirements across question-answering passage ranking and keyword-query ad hoc ranking.
  • 4.2 Learned Attentions: Removing markers decreases MRR by 15%, while removing stopwords has no effect despite stopwords receiving attention comparable to non-stopwords.Markers help BERT distinguish the query and document sequences, whereas stopword attention is treated as redundant.
  • 4.2 Learned Attentions: Deeper Transformer layers spread attention across more tokens and contextualize embeddings, but this does not necessarily produce more global matching decisions.The attention analysis uses BERT (Last-Int) on 100 randomly sampled MS MARCO Dev queries and groups passage terms into markers, stopwords, and regular words.
  • 4.3 Learned Term Matches: BERT assigns more extreme scores and concentrates ranking decisions on a few influential terms, whereas Conv-KNRM distributes term contributions more evenly.Removing influential terms can move BERT scores from 1 to near 0; most other removals have little effect.
  • 4.3 Learned Term Matches: BERT’s influential terms often exactly match or closely paraphrase query terms, while Conv-KNRM favors more loosely related terms.The contrast is consistent with surrounding-context pretraining favoring semantically closer sequence pairs.

5 CONCLUSIONS AND FUTURE DIRECTION

BERT is highly effective for QA-focused MS MARCO passage ranking but less effective for TREC ad hoc document ranking, where user-click signals are better suited. Its globally distributed attention and semantic matching behavior help explain this task-dependent performance.

  • BERT performs well on QA-focused MS MARCO passage ranking but not as well on TREC ad hoc document ranking.The results associate BERT’s surrounding-context pre-training with seq2seq matching tasks, while TREC benefits more from user-click signals.
  • BERT uses globally distributed attentions over entire contexts and assigns extreme matching scores to most query-document term pairs.Most pairs receive either one or zero ranking scores.
  • BERT prefers semantically close text pairs, a behavior that helps explain its weaker effectiveness on TREC-style ad hoc ranking.TREC-style ranking is described as benefiting more from user-click pre-training than surrounding-context pre-training.
  • The results suggest training deeper networks on user-click signals and comparing BERT-sized models with shallower neural rankers trained on relevance labels.The paper identifies this comparison as a future research direction.
Loading 1904.07531v4…