Source-linked AI summary
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M. Voorhees, Ian Soboroff
TL;DR
TREC DL collection details are scattered, and reusing test sets can invite selection bias. This paper consolidates dataset descriptions, offers reuse best practices, and analyzes the collections; its case study finds that the dev set predicts test performance, while judgments are sufficiently complete to evaluate a new ranker without more relevance judgments.
Problem
Method
The paper consolidates TREC DL dataset descriptions, presents best practices for reuse, studies dev-set decisions, and analyzes test-set reusability.
Source: §1 · p. 1Results
The dev set predicts test-set performance without overfitting to test data, and judgments are sufficiently complete to evaluate a new ranker without additional relevance judgments.
Source: §1 · p. 1Takeaways & Limitations
Testing a ranking approach across several years of TREC data can increase confidence that a positive result is a real improvement rather than a result from one test set.
Source: §4 · p. 4Deep models trained on large text corpora can encode societal biases or incur ecological costs from computationally intensive training.
Source: §6 · p. 6
Abstract
from arXiv · showhide
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data may be deep neural networks. This paper supports the reuse of the TREC DL test collections in three ways. First we describe the data sets in detail, documenting clearly and in one place some details that are otherwise scattered in track guidelines, overview papers and in our associated MS MARCO leaderboard pages. We intend this description to make it easy for newcomers to use the TREC DL data. Second, because there is some risk of iteration and selection bias when reusing a data set, we describe the best practices for writing a paper using TREC DL data, without overfitting. We provide some illustrative analysis. Finally we address a number of issues around the TREC DL data, including an analysis of reusability.
1 INTRODUCTION
TREC DL evaluates ranking for ad hoc search with large human-labeled training data, across passage and document retrieval. The paper consolidates the collections’ descriptions, recommends practices to avoid test-set overuse, and analyzes their reusability.
1 INTRODUCTION
- TREC is an information-retrieval evaluation effort organized into tracks, and TREC DL’s tasks model passage-answer retrieval and document discovery.Each year, the tasks produce new reusable test collections for comparing rankers.Source: §1 · p. 1
- TREC DL studies ranking for new queries against an existing text corpus, using large human-labeled training data that has benefited deep-learning models.Its two tasks are passage retrieval and document retrieval.Source: §1 · p. 1
- The paper brings scattered descriptions of the TREC DL collections together and presents best practices, a dev-set case study, and a reusability analysis.The case study examines whether a differently labeled dev set predicts test performance without overfitting; the analysis finds judgments sufficiently complete for evaluating new rankers without additional judgments.Source: §1 · p. 1
2 TASK DESCRIPTION
The track evaluates document and passage retrieval in full-retrieval and reranking settings. Reranking supplies a shared candidate set, letting methods focus on ordering the same candidates.
2 TASK DESCRIPTION
- Document retrieval has full retrieval and top-100 reranking; passage retrieval has full retrieval and top-1000 reranking.Full retrieval searches the full collection, while reranking starts from candidates supplied to all methods.Source: §2 · pp. 1–2
- Document retrieval ranks documents by query relevance, while passage retrieval ranks passages by their likelihood of containing an answer.The passage task models question answering; document retrieval models finding more content about a query.Source: §2 · pp. 1–2
- Shared candidate sets make reranking approaches more comparable and let participants focus on relevance estimation rather than end-to-end retrieval.The document reranking set has 100 candidates; the passage reranking set has 1,000 candidates.Source: §2 · p. 2
3 TEST COLLECTION DATA
The TREC DL collections combine large MS MARCO-derived training sets with yearly test sets for document and passage retrieval. The section describes their construction and use, reports run comparisons, and recommends selecting models on dev data before test evaluation.
3.1 Training Data Sets
- The large human-assessed training sets derive from MS MARCO; passage positives indicate an answer, while document labels transfer passage positives to source documents.The transfer assumes a document containing a relevant passage is relevant, though the document snapshot and passage data were collected at different times.Source: §3.1 · p. 2
3.2 Test Sets
- Test collections used shallow pooling across submitted runs; HiCAL/NIST judgment selection excluded candidates with median MARCO sparse-judgment MRR above 0.5 or equal to 0, yielding 43 queries per task in 2019 and 45 document/54 passage queries in 2020.Source: §3.2 · pp. 2–3
- Four-point judgments distinguish relevance levels; for binary measures, only highly and perfectly relevant passages count as relevant.A passage labeled Related is on-topic but does not answer the question; relevant-but-unjudged items can complicate reuse.Source: §3.2 · p. 3
3.3 ORCAS
- The 2020 track released ORCAS click data for document retrieval, usable as additional training data or as another document field.ORCAS was constructed from a major search engine’s logs and is almost 50 times larger than the main training set.Source: §3.3 · p. 3
3.4 TREC runs
- Run surveys classify submissions as large-pretrained-model “nnlm,” other neural “nn,” or traditional “trad”; blind submissions avoid access to test judgments.TREC runs can be reused as baselines.Source: §3.4 · p. 3
- Best “nnlm” runs outperformed other run types in both tasks in both years, with the difference more pronounced for passage retrieval.The paper attributes the stronger passage effect to greater query-result vocabulary mismatch for shorter texts.Source: §3.4 · p. 3
3.5 Reusing the Test Collections
- Use train data for learning, dev data for hyperparameter choices and early stopping, then evaluate the selected model on the 2019 and 2020 test sets.The paper warns that repeated test-set evaluation can overfit query sets and produce false positives; Figure 2 illustrates selecting a checkpoint by dev performance.Source: §3.5 · p. 3, Figure 2 · p. 4
3.6 Code
- The paper provides an open-source PyTorch Conformer-Kernel implementation that automates document-task data downloads and supports rerank and fullrank training and evaluation.Use of the codebase is optional.Source: §3.6 · p. 4
4 VALIDITY OF RESULTS
Reusable TREC test sets allow hypotheses to be checked across multiple years, but repeated testing and selection can undermine result validity. The paper recommends making decisions on dev data rather than iterating on test sets.
4 VALIDITY OF RESULTS
- Testing a ranking approach across several TREC years can increase confidence that a positive result is not a single-test fluke.The benefit of reuse is faster evaluation without waiting years for new TREC iterations.Source: §4 · p. 4
- Repeated test-set trials and configuration changes increase the chance of lucky results and reduce the chance that a published result is correct.Selection also biases publication toward positive findings when papers with negative results are rejected.Source: §4 · p. 4
- The authors recommend making model decisions on the dev set instead of iterating on reusable test sets.They describe this as the preferred way to avoid test-set-driven selection.Source: §4 · p. 4
A B C
The case study shows how dev-set checkpoint selection can avoid test-set selection bias, while still producing near-optimal test performance. Training variability also motivates multiple trials and care when comparing rankers.
A B C
- The 32 checkpoints differ only in training stopping point, and selecting the best checkpoint on test data risks a meaningless positive result amid random variation.The metrics are largely correlated, but checkpoint-level results also vary randomly.Source: A B C · p. 5
- Selecting ranker A by dev performance makes the TREC 2019 and TREC 2020 NDCG@10 results valid to report; selecting on either TREC set invalidates that set.Ranker A is the 31st checkpoint; rankers B and C are the 27th and 28th.Source: Figure 3 · p. 4, A B C · p. 5
- A test-set difference between checkpoints B and C was statistically significant (p=0.00837) yet meaningless, illustrating how checkpoint cherry-picking can mislead.The rankers came from the same training setup and differed only in stopping point.Source: A B C · p. 5
- Splitting the dev set could separate checkpoint choice from architecture comparison, with a third unused split enabling dev results to be reported.The authors plan to publish standard dev-set splits.Source: A B C · p. 5
- Multiple training trials with different random seeds better characterize a learning approach and reduce the chance that results are dominated by a lucky or unlucky seed.The paper recommends comparing overall machine-learning approaches rather than particular rankers.Source: A B C · p. 5
5 REUSABILITY ANALYSIS
The reusability analysis adapts leave-out-uniques testing to the DL Track’s HiCAL-based judgment process. Across simulated leave-team-out collections, run rankings were generally stable, with modest rank changes.
5 REUSABILITY ANALYSIS
- Standard leave-out-uniques testing does not fit HiCAL-built collections because document selection is not based on runs, so uniquely retrieved relevant documents do not apply.HiCAL selects likely relevant unjudged documents using the current judgments.Source: §5 · p. 5
- Because HiCAL selection is nondeterministic and depends on documents encountered so far, the analysis simulates the process multiple times for a fair test.The simulations use ten seeded trials and construct qrels from trace prefixes matched to the original judgment counts.Source: §5 · p. 5
- The adapted leave-out-uniques test removes one team’s uniquely retrieved relevant documents, reruns HiCAL, and compares induced run rankings with official-qrels rankings.The same random seed is used for the corresponding full-pool and omitted-team trials.Source: §5 · p. 6
- Most runs retained a dominant rank across simulations, and rank changes were modest; passage-task results showed even less variability.The figure covers 120 trials for the 2019 document task, using MAP and Precision@10.Source: §5 · p. 6, Figure 4 · p. 6
6 LIMITATIONS
The paper notes that the benchmark’s scope and development context impose limitations. Its queries are English and answerable by a short passage, and large-scale neural development can carry societal and environmental costs.
6 LIMITATIONS
- Developing large-scale neural retrieval models may involve specialized hardware and can encode societal biases or incur ecological costs.The paper urges serious consideration of these negative externalities and strict ethical guidelines.Source: §6 · p. 6
- The benchmark uses English queries answerable by a short passage, restricting work on non-English models and other intents such as transactional search.The queries come from Bing search logs.Source: §6 · p. 6
- The authors still consider TREC DL a critical IR resource despite these limitations.They also note that the training data may support methods not requiring large-scale training or training models for other tasks.Source: §6 · p. 6
7 CONCLUSION
The conclusion presents TREC DL resources and usage practices, reports stable results from dev-based decisions and leave-team-out analysis, and identifies scope boundaries for reuse.
7 CONCLUSION
- The TREC DL resources cover document and passage retrieval, including training data, ORCAS click data, test queries, and sample code.The conclusion describes the resources as intended for future use in the field.Source: §7 · p. 6
- A dev-set decision using a different labeling scheme gave near-optimal performance on both test sets without making invalidating test-set decisions.The authors suggest splitting the dev set for separate decisions and running multiple trials because results can vary by random seed.Source: §7 · p. 6
- Leave-team-out analysis found results quite stable, suggesting reasonable evaluation of ranking approaches absent from the judging process.The conclusion qualifies the benchmark as one dataset, in one language, for a particular task.Source: §7 · p. 6