Source-linked AI summary
Variations in Relevance Judgments and the Shelf Life of Test Collections
Andrew Parry, Maik Fröbe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Faggioli, Saber Zerhoudi, Sean MacAvaney, Eugene Yang
TL;DR
Prior work found stable rankings despite disagreement, but it is unclear whether that holds for modern, frequently reused collections with less defined query intent. The paper re-annotates TREC DL’19 and reproduces the finding that disagreement does not alter rankings, while observing degradation for some systems and effectiveness near human judges.
Problem
It is unclear whether assessor disagreement remains negligible for system comparisons in modern collections, where neural retrieval, graded judgments, and less formalized query intent may affect evaluation.
Source: §1 · pp. 1–2, §3.2 · p. 3Method
The authors re-annotate TREC DL’19 with two secondary annotators per topic and compare systems under alternative combinations of graded judgments.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
System rankings remain stable despite low agreement, while some LLM-based and distilled systems degrade under new judgments; a reasonable human upper bound is around nDCG@10 of 0.81.
Source: §1 · p. 2Takeaways & Limitations
The authors advise caution when maximizing metrics beyond human relevance-estimation bounds and publish code and judgments for reproducibility and future work.
Source: §1 · p. 2The study does not consider large ensemble systems, which the authors exclude because of their risks.
Source: §1 · p. 2
Abstract
from arXiv · showhide
The fundamental property of Cranfield-style evaluations, that system rankings are stable even when assessors disagree on individual relevance decisions, was validated on traditional test collections. However, the paradigm shift towards neural retrieval models affected the characteristics of modern test collections, e.g., documents are short, judged with four grades of relevance, and information needs have no descriptions or narratives. Under these changes, it is unclear whether assessor disagreement remains negligible for system comparisons. We investigate this aspect under the additional condition that the few modern test collections are heavily re-used. Given more possible query interpretations due to less formalized information needs, an ``expiration date'' for test collections might be needed if top-effectiveness requires overfitting to a single interpretation of relevance. We run a reproducibility study and re-annotate the relevance judgments of the 2019~TREC Deep Learning track. We can reproduce prior work in the neural retrieval setting, showing that assessor disagreement does not affect system rankings. However, we observe that some models substantially degrade with our new relevance judgments, and some have already reached the effectiveness of humans as rankers, providing evidence that test collections can expire.
1 Introduction
Classic evaluations found that system rankings remain stable despite disagreement among relevance assessors. This study tests whether that robustness holds for modern, frequently reused collections, where query intent may be less defined and model iteration may fit systems to existing judgments.
1 Introduction
- Modern collections often have short documents, four relevance grades, and no query narratives, leaving more room for assessors to interpret intent.The study asks whether these conditions change the robustness seen in traditional collections.Source: §1 · p. 1
- Repeated model improvement on static benchmarks may overfit to their relevance judgments and introduce research-design bias.TREC DL’19 remains a commonly used benchmark, and the authors note warnings against iterative neural-ranking design.Source: §1 · p. 2
- Despite low assessor agreement, system rankings remain stable; human judges reach an approximate nDCG@10 upper bound of 0.81, and some LLM-derived systems degrade under new judgments.The authors suggest results above the bound may reflect overfitting to the original relevance labels.Source: §1 · p. 2
- The study re-annotates TREC DL’19 with two secondary annotators per topic and tests whether alternative judgments alter system effectiveness and ordering.The evaluation follows Voorhees’s re-annotation setting while adding the hypothesis that model iteration may also affect rankings.Source: §1 · p. 2
- The authors caution against maximizing metrics beyond human relevance-estimation bounds and release code and judgments for reproducibility and future study.They warn that exceeding human effectiveness may not improve information access and may stagnate progress.Source: §1 · p. 2
2 Related Work
Related work frames offline test-collection reliability around relevance subjectivity, incomplete judgments, and the stability of system comparisons. It also connects collection retirement concerns to repeated benchmark use and model-training pipelines.
Offline Evaluation
- Offline evaluations rely on relevance judgments for query–document pairs, while exhaustive annotation is infeasible at modern corpus scale.TREC-style collections pool system runs to select documents for annotation, commonly assigning one assessor per topic.Source: §2 · p. 2
Notions of Relevance
- Prior re-annotation studies found that system ordering can remain stable despite assessor disagreement, including under depth-pooled TREC collections.Reliability concerns also include annotation settings, grading, and pooling.Source: §2 · p. 2
Collection Retirement
- Earlier collection-retirement concerns focused on pooling bias and incomplete recall judgments; this work instead examines popularity and possible fitting to labels over time.The authors focus on precision in keeping with neural systems and the Deep Learning track.Source: §2 · p. 3
- Existing strong systems may shape later models through semi-supervised training, and continued comparison against the same dataset may contribute to stagnation.The cited discussion warns against experimental iteration on Deep Learning collections [9].Source: §2 · p. 3
3 Evaluating Reliability through Re-Annotation
The study adapts re-annotation reliability tests to modern neural retrieval and graded judgments. It re-annotates TREC DL’19 and investigates judgment aggregation, query narratives, and human annotators as effectiveness references.
3 Evaluating Reliability through Re-Annotation
- The study applies Voorhees’s collection-reliability approach to test whether models trained after a collection’s release remain effective under new judgments.The evaluation includes the probability that two systems swap effectiveness order across annotation samples.Source: §3 · p. 3, §3.1 · p. 3
3.1 Reproduction Source
- For each system pair, sampled combinations of primary and secondary judgments yield a swap probability from the counts of samples favoring each system.The procedure adapts the original binary-judgment approach to four-grade relevance and neural systems on TREC DL’19 [8].Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Re-Annotation of a Modern Collection
- Two retrieval academics from an eight-person pool re-annotate each TREC DL’19 topic; the secondary pool includes documents judged at least related and sampled non-relevant documents.The study reduces the number of sampled non-relevant judgments, reasoning that agreement is higher for non-relevance; annotators use guidelines and a BM25 search endpoint over ClueWeb22. The annotators also contributed to analysis and write-up, which the authors say had minimal effect.Source: §3.2 · pp. 3–4
3.3 Further Investigation
- Further analyses aggregate judgments by maximum, minimum, or mean and ask annotators to record ambiguity and write narratives to examine query intent.For selected topics, annotators are assigned one of two narratives to separate narrative interpretation from ambiguity in the query title.Source: §3.3 · p. 4
- Annotators are also evaluated as oracle-like rankers against the original judgments, with in-grade shuffling producing minimal effectiveness variation (SD < 0.0001).The approach compares annotator performance with modern systems using relevance grades aggregated and sorted for ranking.Source: §3.3 · p. 4
4 Evaluation
The evaluation re-annotates TREC DL’19 to test agreement, collection reliability, and how system comparisons change under alternative judgments. Despite low agreement, system rankings remain highly correlated, while model-level changes and human-level effectiveness raise questions about benchmark robustness.
4.1 Experiment Design
- The study re-annotates 43 depth-pooled DL’19 topics and compares judgment combinations using agreement measures and system-ranking correlations.The pool includes top-ranked texts from 75 systems plus additional HiCal texts; effectiveness is assessed primarily with nDCG@10.Source: §4.1 · pp. 4–5
4.2 Pilot Study Findings
- In the 10-topic pilot, secondary annotations had inflated grade-1 and grade-3 prevalence, amid ambiguity about relevance grades and query context.Annotators could use a controlled search engine to familiarize themselves with topics; nationality-related bias and query difficulty were left for future work.Source: §4.2 · p. 5, Table 3 · p. 5
4.3 Core Assessment of Agreement
- Four-grade judgments show low agreement, with Fleiss’ κ near random and generally around 10 points higher than binary judgments; secondary annotators often lower relevance grades.Among 800 texts initially labeled non-relevant, 13% changed grade on re-annotation, and 77% of those changes were to grade 1.Source: §4.3 · pp. 5–6, Table 4 · p. 5, Figure 1 · p. 6
4.4 Narratives, Intent, and Relevance
- The examples show that valid but different query intents can produce sharply different judgments, while partially relevant criteria also diverge for a general airport query.Some initially non-relevant texts were judged perfectly relevant under another intent when context was inferred, despite the information not being self-contained; the authors suggest query-specific grade specifications as a diagnostic aid.Source: §4.4 · pp. 6–7
4.5 Narrative and Intent Ambiguity
- Fixed narratives improve agreement from difficult to median queries, but agreement changes inconsistently from median to easy queries across groups.The authors report a 79% global agreement increase from difficult to median and a further 1% from median to easy, while noting that natural-language query ambiguity may be the largest factor.Source: §4.5 · p. 7, Figure 2 · p. 8
4.6 Core Assessment of System Order
- Despite low assessor agreement, system ordering remains highly correlated with original DL’19 rankings under alternative judgments.Comparisons with large nDCG@10 differences can still swap system order; LLM–neural comparisons form a distinct Pareto frontier.Source: §4.6 · p. 7, Table 5 · p. 7, Figure 3 · p. 8
- Some modern systems shift more than lexical systems: smaller supervised and semi-supervised models often improve rank, while list-wise encoders and smaller distilled models can degrade.GPT-4o reranking is less stable over BM25 than over a ColBERT first stage; the authors link this pattern to positional bias as a hypothesis.Source: §4.6 · p. 7, Figure 4 · p. 8, Figure 5 · p. 9
4.7 Systems versus Humans
- Systems can match or exceed secondary annotators on nDCG@10 and MRR@10, but recall remains lower in all reported settings.The authors caution that parity should be interpreted carefully; they also describe reduced recall as undesirable for downstream tasks.Source: §4.7 · pp. 8–9, Table 6 · p. 9
5 Conclusion
Re-annotating TREC DL’19 reproduced stable system ordering under disagreement but found variation in rankings for large language models and their distilled systems. The authors argue that the collection may be nearing a realistic precision bound and that further gains may not indicate meaningful system differences.
5 Conclusion
- Re-annotation validated stable system ordering under four-grade relevance and ambiguous query intent, while rankings varied for large language models and systems distilled from them.The authors also suggest that repeated tuning on static judgments may reduce a collection’s reliability over time.Source: §5 · p. 9
- Current state-of-the-art systems can outperform combinations of human annotators on original relevance judgments, suggesting a realistic precision bound for TREC DL’19.Source: §5 · p. 9
- Further improvement on this collection may not indicate that one system is meaningfully better than another.Source: §5 · p. 9
- Re-annotation is costly, and future work will investigate ways to assess collection expiration without extensive human effort.Source: §5 · p. 9