PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustness of IR Models to Collection Growth

Emmanouil Georgios Lionis, Debasis Ganguly, Sean MacAvaney

arXiv:2608.23419v1cs.IRcs.CL

TL;DR

The paper asks whether retrieval effectiveness remains stable when non-relevant documents are added to evolving collections. It formalises this property and tests MDA and MDD models by merging collections with negligible topical overlap. All tested models degrade somewhat; MDA retrieval models outperform MDD retrieval models, while their rerankers perform equally effectively.

  • Problem

    IR collections evolve through additions, but it is unclear whether retrieval effectiveness remains stable when non-relevant documents are added.

  • Method

    The study formalises the Collection Growth Axiom, classifies models as MDA or MDD, and evaluates them on merged collections with controlled non-relevant additions.

  • Results

    MDA retrievers are more robust during first-stage retrieval, while MDA and MDD rerankers are equally robust under non-relevant additions.

  • Takeaways & Limitations

    Current IR architectures show systematic limitations as collections grow, motivating models designed for collection growth and pipeline stage.

  • Takeaways & Limitations

    The study uses one MS MARCO–TREC-COVID corpus pairing, with the injected subcollection comprising 1.9% of the merged corpus and shared MS MARCO pre-training.

Abstract

from arXiv · show

Information Retrieval (IR) systems seek to identify relevant documents within a collection. In practical applications, collections are dynamic, with documents frequently added. We argue that ideally, a retriever's effectiveness should not decrease when non-relevant documents are added to a collection. This study formalises this concept and empirically evaluates it by merging two collections with negligible topic overlap. We hypothesise that the way an IR model conditions its ranking on other documents in a collection (e.g., the IDF component in BM25 or contextual documents in listwise rerankers) plays an important role in its robustness to the addition of non-relevant documents. We broadly classify models as those that do not depend on other documents (Multi-Document-Agnostic, MDA) and those that do (Multi-Document-Dependent, MDD). Our results show that neither MDD nor MDA models are fully robust to the addition of non-relevant documents, as all models exhibit some performance degradation. Interestingly, among the models we test, MDA is more effective than MDD for retrieval, whereas MDD and MDA rerankers are equally effective.

1 Introduction

The paper formalises robustness to non-relevant additions and evaluates whether IR models preserve retrieval effectiveness as collections grow. It classifies models by inter-document dependency and tests them using merged collections with negligible topical overlap.

  • The study defines robustness as preserving retrieval effectiveness when non-relevant documents are added to a collection.This property is formalised as the collection growth axiom.
  • Models are classified as Multi-Document-Agnostic or Multi-Document-Dependent according to whether scoring depends on other collection documents.MDA models score independently, while MDD models account for inter-document dependencies.
  • The evaluation merges two standard benchmark collections whose relevance judgements remain applicable and whose topics have negligible overlap.This creates a controlled setting where one collection supplies non-relevant additions for queries from the other.
  • The study evaluates diverse IR models under collection growth and finds varying degrees of robustness to added non-relevant documents.The observations indicate a systematic limitation of current architectures in evolving collections.
  • The work contributes the CG Axiom, an inter-document-dependency taxonomy, and a controlled examination of robustness across model architectures.

2 Collection Growth

The CG Axiom requires retrieval performance to remain nearly invariant when non-relevant documents expand a collection. The paper complements this performance criterion with Collection Precision to track whether top-ranked results come from the original collection.

  • Collection growth is represented by expanding an existing collection C with new documents D to form C+ = C ∪ D.
  • The axiom evaluates a model ϕ using a metric M over the top-k list L_k(C,q) returned for query q.
  • The CG Axiom formalises robustness by requiring nearly unchanged retrieval performance after adding documents irrelevant to the query.The allowed change is bounded by a small positive ε.
  • Collection Precision measures the proportion of top-k results drawn from the original collection C.It addresses the limitation that the performance metric M does not identify whether retrieved documents originated in C or the added set D.
  • The taxonomy distinguishes retrieval and ranking architectures by how they model inter-document dependency in relevance estimation.

3 Inter-Document Dependency

The paper classifies IR models by whether relevance scoring depends on other documents or collection-level context. MDA models score query–document pairs independently, while MDD models incorporate candidate sets, feedback, clusters, or corpus statistics.

  • Multi-Document-Agnostic (MDA) Models: MDA models estimate relevance independently for each document without inter-document or collection-level context.Bi-encoders and pointwise cross-encoders belong to this category.
  • Multi-Document-Agnostic (MDA) Models: Bi-encoders separately encode queries and documents, then compute relevance from their representations without conditioning on other documents.The similarity function may be a dot product between embedded vectors.
  • Multi-Document-Agnostic (MDA) Models: Pointwise cross-encoders jointly encode a query and document, but their relevance computation remains independent of other documents and collection statistics.Thus, both bi-encoders and pointwise cross-encoders are multi-document agnostic.
  • Multi-Document-Dependent (MDD) Models: MDD models estimate relevance using information beyond an isolated query–document pair, including retrieved documents, candidate sets, feedback signals, or corpus structure.These dependencies create shared context in relevance estimation.
  • Multi-Document-Dependent (MDD) Models: Listwise cross-encoders condition document scores on the candidate set, so collection growth can alter the local reranking context.Their candidate window may contain multiple documents, such as n=10 for RankZephyr.
  • Multi-Document-Dependent (MDD) Models: Contextual Document Encoding derives document representations from collection-based neighbourhoods, allowing newly added distractors to change relevance scores.Its dependency comes from a contextual structure derived during indexing.
  • Multi-Document-Dependent (MDD) Models: Lexical retrievers and statistical PRF can change scores as collections grow because they depend on corpus statistics and query-expansion evidence.Statistical PRF is exposed through both new terms and global collection statistics.

4 Experiments and Discussion

The experiments merge MS MARCO and TREC-COVID into a heterogeneous collection to test robustness to non-relevant additions. Results show asymmetric collection effects: MDA retrievers generally degrade less, PRF is biased toward the dominant collection, and MDA and MDD rerankers perform comparably.

  • Experimental setup: The study merges MS MARCO and TREC-COVID into a 9.1M-document heterogeneous collection, with MS MARCO comprising 98.1% and TREC-COVID 1.9%.DL-2019 queries are used to minimise topical overlap, while DL-2020 queries are excluded to avoid health- or coronavirus-related bias.
  • Experimental setup: The evaluation compares MDD retrievers BM25 and CDE, MDA retrievers RetroMAE and SPLADE, PRF variants, and MDA/MDD rerankers.RM3 and VectorPRF use the top-10 feedback documents; rerankers process the top 100 candidates.
  • Collection effects: TREC-COVID performance drops by ΔnDCG@10 ≈ [−0.244, −0.019], whereas DL-2019 remains stable at ≈ [−0.009, +0.002].No model satisfies the CG Axiom for TREC-COVID, while DL-2019 satisfies it under the reported stability and CP@10 conditions.
  • RQ1: Retriever robustness: MDA retrievers achieve the highest nDCGHet@10 on both TREC-COVID and DL-2019, while MDD retrievers show larger TREC-COVID decreases.RetroMAE and SPLADE reach 0.735 and 0.700 on TREC-COVID, and 0.680 and 0.728 on DL-2019; BM25 and CDE decrease by −0.220 and −0.107 on TREC-COVID.
  • RQ2: PRF effects: PRF generally worsens performance for the smaller collection and biases retrieval toward the dominant collection and top-k feedback documents.For example, RetroMAE-N/A scores 0.735 versus 0.728 for RetroMAE-VectorPRF, while CDE CP@10 falls from 0.924 to 0.906.
  • RQ3: Reranker robustness: MDA and MDD rerankers perform comparably, with nearly identical TREC-COVID ΔnDCG@10 values and only a 0.014-point DL-2019 nDCG@10 difference.The reported TREC-COVID reranker values are 0.779 and 0.775; for DL-2019, SPLADE-MonoELECTRA and SPLADE-Set-Encoder score 0.767 and 0.781.

5 Conclusion

The conclusion finds that robustness to irrelevant-document additions varies across retrieval pipelines. MDA retrievers are more robust at first-stage retrieval, while MDA and MDD rerankers are equally robust.

  • MDA retrievers are more robust than MDD retrievers during first-stage retrieval under heterogeneous, non-relevant document additions.PRF modules exhibit bias toward larger collections, and current architectures show systematic limitations as collections grow.
  • MDA and MDD rerankers are equally robust at the reranking stage.The conclusion states that future retrieval models should account for pipeline stage and evolving collection characteristics from the outset.

Limitations

The study’s empirical scope is limited to one corpus pairing in which the injected subcollection represents only 1.9% of the merged corpus. This imbalance and shared MS MARCO pre-training may influence the results.

  • The experiments use a single MS MARCO–TREC-COVID pairing, with the injected subcollection forming only 1.9% of the merged corpus.The CG Axiom and MDA/MDD taxonomy are described as corpus-agnostic, but other domains and injection ratios remain for future work.

GenAI Usage Disclosure

Generative AI was used only for coding, grammar, and phrasing assistance; the authors state that the scientific content, methodology, and conclusions are their own.

  • Generative AI assisted with coding, grammar, and phrasing, while the authors retained responsibility for the scientific content, methodology, and conclusions.
Loading 2608.23419v1…