PaperlayerView exact revision on arXiv

Source-linked AI summary

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych

arXiv:2104.08663v4cs.IRcs.AIcs.CL

TL;DR

Existing retrieval research offers limited evidence about zero-shot generalization across diverse tasks and domains. BEIR addresses this gap with 18 heterogeneous datasets and evaluations of ten retrieval methods across five architectures. The results identify strong cross-task performance from BM25, re-ranking, and late-interaction approaches, while exposing efficiency trade-offs and annotation-selection bias.

  • Problem

    Existing neural retrieval evaluations are often narrow and leave generalization across other text domains, tasks, and out-of-distribution data unclear.

  • Method

    BEIR constructs a heterogeneous zero-shot benchmark from 18 datasets across nine tasks and diverse domains, then evaluates ten methods spanning five retrieval architectures.

  • Results

    BM25 remains a strong zero-shot baseline, while re-ranking and late-interaction methods perform best overall at higher computational cost; dense and sparse methods can underperform.

  • Takeaways & Limitations

    BEIR provides a unified, model-agnostic framework for studying retrieval generalization across tasks and domains and supporting more robust systems research.

  • Takeaways & Limitations

    Annotation selection bias can treat documents retrieved by new systems as irrelevant, disadvantaging systems that differ from the annotation-pooling methods.

Abstract

from arXiv · show

Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.

1 Introduction

BEIR addresses limited evidence about how retrieval systems generalize across tasks, domains, and out-of-distribution data. It introduces a heterogeneous benchmark and finds substantial differences in zero-shot performance, efficiency, and generalization across retrieval architectures.

  • Prior neural retrieval studies often use narrow, homogeneous training and evaluation settings, leaving out-of-distribution generalization unclear.
  • BEIR contains 18 retrieval datasets spanning nine tasks, diverse domains, text types, dataset sizes, and query and document lengths.
  • Ten retrieval methods across lexical, sparse, dense, late-interaction, and re-ranking architectures are evaluated in a zero-shot setting.
  • No single approach consistently outperforms all others, and in-domain performance does not reliably predict zero-shot generalization.
  • Re-ranking and late-interaction models perform best but cost more computationally, while efficient dense and sparse methods can underperform BM25.
  • Dataset lexical bias can disadvantage non-lexical systems, motivating better unbiased datasets and diverse relevance pooling.

2 Related Work and Background

Earlier retrieval benchmarks provide limited evidence about generalization because they concentrate on particular tasks, domains, or corpus scales. Neural retrieval methods address lexical limitations through sparse, dense, and re-ranking strategies, each with distinct trade-offs.

  • MultiReQA evaluates sentence-level answer retrieval across eight QA datasets, mostly from Wikipedia and generally using small corpora.
  • KILT includes eleven datasets across five knowledge-intensive tasks, but retrieval is not its primary task and documents come only from Wikipedia.
  • Lexical retrieval methods such as TF-IDF and BM25 are limited by lexical gaps and keyword matching.
  • Sparse methods expand documents or adjust term weights while retaining BM25 for remaining retrieval operations.
  • Neural re-rankers use first-stage retrieval results, often from BM25, but cross-attention improves comparisons at high computational overhead.

3 The BEIR Benchmark

BEIR is designed as a broad zero-shot benchmark by selecting difficult datasets across heterogeneous tasks and domains. Its standardized software and data format support comparable evaluation and extension to new datasets and models.

  • BEIR selects datasets using diverse tasks, diverse domains, and sufficient task difficulty as its main construction criteria.
  • The benchmark includes 18 English zero-shot datasets from nine retrieval tasks, while MS MARCO is reported separately from zero-shot comparisons.
  • BEIR balances broad and specialized domains, including Wikipedia, scientific publications, Twitter, news, and online communities.
  • Low weighted Jaccard overlap across domains indicates that BEIR evaluates generalization across diverse out-of-distribution text.
  • BEIR provides a Python framework with model wrappers, retrieval metrics, and support for evaluating existing or newly added models and datasets.
  • The benchmark uses a standardized corpus, queries, and qrels format to simplify evaluation across datasets.
  • A single rank-aware metric is emphasized because precision and recall are not suitable for comparable ranking evaluation across tasks.

4 Experimental Setup

The experimental setup compares diverse retrieval architectures using publicly available pretrained checkpoints. Models are grouped into lexical, sparse, dense, late-interaction, and re-ranking categories, with neural documents truncated to 512 word pieces.

  • Experiments use publicly available pretrained checkpoints and retain only the first 512 word pieces of each document for neural architectures.
  • The evaluated systems are grouped into lexical, sparse, dense, late-interaction, and re-ranking architectures.
  • Lexical Retrieval: BM25 performs lexical retrieval through token matching between sparse vectors weighted by TF-IDF.
  • DeepCT and SPARTA represent sparse retrieval methods, while DPR and ANCE represent dense bi-encoder approaches.
  • Late-Interaction: ColBERT performs late interaction by aggregating token-level query and passage similarities after approximate nearest-neighbor candidate retrieval.
  • Re-ranking: BM25 + CE reranks the top 100 BM25 hits using a cross-attentional MiniLM model.

5 Results and Analysis

BEIR shows that retrieval systems generalize unevenly across diverse datasets: BM25 remains a strong baseline, while re-ranking and late-interaction models perform well but incur higher computational costs. Dense models vary substantially across tasks, and domain adaptation helps specialized domains but can hurt broader ones.

  • BM25 remains a strong baseline for out-of-distribution generalization despite substantially lower in-domain performance than neural approaches on MS MARCO.The benchmark therefore exposes differences between in-domain performance and broader generalization.
  • Document expansion with docT5query outperforms BM25 on 11/18 datasets, whereas DeepCT and SPARTA underperform BM25 on nearly all datasets.The contrast is attributed to adding relevant keywords rather than only learning term weights.
  • Dense retrievers perform strongly on some datasets but can fall substantially below BM25 under domain or task shifts, with DPR showing the weakest overall generalization.Examples include BioASQ and Touché-2020, which differ from the models’ training settings.
  • BM25+CE outperforms BM25 on 16/18 datasets, while ColBERT does so on 9/18 datasets, indicating strong zero-shot performance for re-ranking and late-interaction models.BM25+CE fails on ArguAna and Touché-2020, whereas ColBERT uses token-level late interaction with MaxSim scoring.
  • GenQ improves TAS-B on specialized scientific, financial, and StackExchange domains but performs worse on broader generic domains such as Wikipedia.This indicates that synthetic-query domain adaptation has domain-dependent effects.
  • Re-ranking and late-interaction models achieve the best out-of-distribution performance at latency above 350 ms, while dense models stay below 20 ms and sparse models reach 20–25 ms on CPU.The results show a performance–retrieval-latency trade-off across architectures.

6 Impact of Annotation Selection Bias

BEIR’s evaluation datasets can favor lexical retrieval because annotation pools are built from lexically retrieved candidates, leaving non-lexical hits unjudged. On TREC-COVID, adding missing relevance judgments substantially improves dense retrieval scores but barely changes lexical scores.

  • Table 4 reports Hole@10 for each system, measuring top-10 hits unseen by annotators.Lower values indicate that more retrieved hits were already represented in the annotation pool.
  • Lexical candidate selection can disadvantage non-lexical retrieval because relevant hits without lexical overlap may be assumed irrelevant.
  • 31.8% Hole@10 for TAS-B and 14.4% for ANCE contrast with 6.4% for BM25 and 2.8% for docT5query.Dense systems therefore had more top-ranked documents that annotators had not seen.
  • After missing judgments were added, ANCE’s nDCG@10 rose from 0.654 to 0.735, while docT5query increased only from 0.713 to 0.714.The ANCE result reached 6.7 points above BM25, whereas lexical performance changed only slightly.
  • The TREC-COVID analysis indicates that its annotation pool remained biased toward lexical approaches despite pooling results from multiple systems.

7 Conclusions and Future Work

BEIR concludes that broad zero-shot evaluation reveals how retrieval models generalize across diverse tasks and datasets. Its results show that in-domain performance is not predictive of zero-shot generalization, while annotation bias remains a fairness concern.

  • BEIR spans nine retrieval tasks and 18 diverse datasets, broadening evaluation from narrow task- or domain-specific benchmarks.
  • BEIR’s standardized format and adaptable code support unified zero-shot evaluation and integration of diverse retrieval strategies.
  • In-domain performance cannot predict how an approach will generalize in a zero-shot setup.Many approaches that outperform BM25 in-domain perform poorly on BEIR.
  • Cross-attentional re-ranking, ColBERT, and docT5query performed overall well across the evaluated tasks.
  • Diverse pooling strategies are needed because annotation selection bias can negatively affect non-lexical retrieval approaches.

Checklist

The checklist records reproducibility and reporting practices alongside limitations concerning compute disclosure, personally identifiable or offensive content, and supplementary materials. It also points readers to appendices covering benchmark limitations, task details, licenses, and additional metrics.

  • The experiments are reproducible through the authors’ repository, but the checklist reports that error bars were not provided.
  • The paper reports GPU and CPU resource types but not the total amount of compute used.
  • The paper identifies no supplemental material attached to the submission, directing readers to the repository for further material.
  • The authors reused existing datasets and did not check all more than 50 million documents for offensive content.They state that comprehensive checking would be difficult and could alter the underlying dataset.
  • Appendices provide benchmark limitations, training and in-domain evaluation details, dataset descriptions, licenses, and metric overviews.

B Limitations of the BEIR Benchmark

BEIR spans diverse retrieval settings but remains bounded by English-only data, short documents, pure textual search, limited fields, and its emphasis on broadly generalizing models.

  • Multilingual Tasks: All BEIR datasets are currently English because multilingual retrieval datasets are limited.The authors propose adding multi- and cross-lingual tasks and models.
  • Long Document Retrieval: Most tasks contain documents averaging a few hundred words, leaving long-document retrieval underrepresented.Transformer approaches often have a 512-word-piece limit, requiring a fundamentally different setup for longer documents.
  • Multi-factor Search: BEIR focuses on pure textual search and does not integrate signals such as PageRank, recency, authority, or click-through rates.The authors identify integrating these signals as an open research direction.
  • Multi-field Retrieval: The benchmark primarily evaluates datasets with one or two fields, although retrieval can use titles, abstracts, bodies, authors, and journal names.
  • Task-specific Models: BEIR emphasizes models that generalize broadly, whereas task-specific models can outperform generic models on individual tasks or domains.
  • Training Data: BEIR uses MS MARCO’s 8.8M passages and 532,761 query-passage training pairs to fine-tune most retrievers.The dataset is selected for broad topic coverage and its large number of training pairs.

D Zero-shot Evaluation Tasks

BEIR’s zero-shot evaluation covers 18 datasets across heterogeneous retrieval tasks, including biomedical, question-answering, news, argument, entity, citation, and fact-checking retrieval.

  • Benchmark Scope: BEIR includes 18 evaluation datasets spanning nine heterogeneous retrieval tasks.
  • Biomedical Retrieval: Biomedical retrieval searches scientific documents or blogs for scientific queries, including TREC-COVID, NFCorpus, and BioASQ.TREC-COVID uses CORD-19, NFCorpus uses PubMed documents, and BioASQ uses roughly 15M PubMed articles.
  • Question Answering: Open-domain question answering retrieves passages containing answers from an entire knowledge source.Natural Questions includes complex development queries, HotpotQA requires reasoning over multiple paragraphs, and FiQA-2018 uses financial StackExchange data.
  • News Retrieval: News retrieval maps headlines or articles to relevant tweets or news articles providing background context.The benchmark includes Signal-1M Related Tweets, TREC-NEWS, and Robust04.
  • Argument Retrieval: Argument retrieval ranks argumentative texts, including counterarguments, by relevance to a textual query.ArguAna retrieves counterarguments, while Touché-2020 evaluates conversational argument retrieval.
  • Other Retrieval Tasks: Other tasks retrieve duplicate questions, entity pages, cited papers, or evidence passages for claims.These include CQADupStack and Quora, DBPedia-Entity-v2, SCIDOCS, and FEVER.

E Dataset Licenses

The benchmark documents licenses for most datasets, while four datasets lack reported license information; it also defines evaluation measures used in BEIR.

  • License Coverage: Four of the 19 listed datasets—NFCorpus, FiQA-2018, Quora, and Climate-Fever—lack reported dataset licenses.
  • License Types: Reported licenses range from MIT and Creative Commons licenses to Apache, GPL, copyright-restricted, and non-commercial research terms.Examples include CC BY-SA 3.0 for FEVER and NQ, Apache 2.0 for CQADupStack, and GPL for SCIDOCS.
  • Dataset Similarity: Weighted Jaccard similarity compares normalized word-frequency distributions between source dataset S and target dataset T.The sum covers all unique words occurring in both datasets.
  • Recall Metrics: Recall@k measures the fraction of relevant documents retrieved among the top k scored documents.For query i, A_i is the model’s scored document list and A⋆_i is the relevant-document set.
  • Recall Metrics: Capped recall replaces the relevant-document denominator with min(k, |A⋆_i|) when a query has more than k relevant documents.This prevents datasets with many relevant documents from receiving unintuitively low recall scores.

H Document Length Preference for Dense Retrieval System

Dense retrieval systems can prefer documents of different lengths, and that preference can affect measured performance because relevance judgments vary with document length.

  • Observed Length Preferences: TAS-B prefers shorter retrieved documents, whereas ANCE retrieves longer documents.On TREC-COVID, TAS-B often retrieves title-only documents while ANCE retrieves documents with non-empty abstracts.
  • Controlled Comparison: Identifying the source of TAS-B and ANCE’s contrasting length preferences is difficult because their models, losses, and hard-negative strategies differ.The authors therefore harmonize training and vary only the similarity function.
  • Similarity Functions: Cosine-similarity and dot-product dense models use their similarity function during training with InfoNCE and in embedding-space retrieval.The setup uses in-batch negatives and a scaling factor τ, with d+ denoting the relevant document.
  • Performance Effects: 15.3 points is the biggest TREC-COVID improvement for the dot-product model, while it performs worse than cosine similarity on most other datasets.
  • Observed Length Preferences: Nearly identical models retrieve documents of vastly different lengths; cosine similarity prefers shorter documents across all datasets.On TREC-COVID, about 42k of 171k scientific papers contain only publication titles without abstracts, which the cosine model favors.
  • Relevance and Length: Relevance scores are not uniformly distributed over document lengths, with longer or shorter documents receiving higher relevance depending on the dataset.This may reflect annotation procedures or the retrieval task’s information needs.
  • Evaluation View: Table 10 combines top-10 document-length violin plots with nDCG@10 scores for cosine-similarity and dot-product models.
Loading 2104.08663v4…