Source-linked AI summary
Overview of the TREC 2022 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff
TL;DR
The track asks how retrieval methods can be evaluated with complete, reusable collections at large scale. It constructs and assesses a harder, passage-focused TREC 2022 test collection using MS MARCO v2, then analyzes submitted passage and document ranking runs. Large-scale-pretrained neural methods continued to outperform traditional retrieval, while some top results did not rely on dense retrieval and single-stage dense retrieval was less competitive than the previous year.
Problem
The track addresses incomplete and insufficiently reusable retrieval test collections, especially the need to distinguish methods reliably without excessive additional judging.
Method
The track focuses judging on passages, propagates passage labels to documents, and uses parallel CAL processes with near-duplicate handling to construct a more complete collection.
Results
Large-scale-pretrained neural ranking methods outperformed traditional retrieval across passage and document tasks, while single-stage dense retrieval was less competitive than the previous year.
Takeaways & Limitations
The 2022 passage collection is presented as reusable for evaluating different retrieval methods, with stronger discrimination between neural and traditional approaches.
Takeaways & Limitations
The v1 data had query-generated corpus and passage-document mapping problems, while deduped-run reuse would require additional run deduplication and could introduce errors.
Abstract
from arXiv · showhide
This is the fourth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we also leverage both the refreshed passage and document collections that were released last year leading to a nearly $16$ times increase in the size of the passage collection and nearly four times increase in the document collection size. Unlike previous years, in 2022 we mainly focused on constructing a more complete test collection for the passage retrieval task, which has been the primary focus of the track. The document ranking task was kept as a secondary task, where document-level labels were inferred from the passage-level labels. Our analysis shows that similar to previous years, deep neural ranking models that employ large scale pretraining continued to outperform traditional retrieval methods. Due to the focusing our judging resources on passage judging, we are more confident in the quality of this year's queries and judgments, with respect to our ability to distinguish between runs and reuse the dataset in future. We also see some surprises in overall outcomes. Some top-performing runs did not do dense retrieval. Runs that did single-stage dense retrieval were not as competitive this year as they were last year.
1 Introduction
TREC 2022 continued benchmarking passage and document retrieval with MS MARCO data, while emphasizing a more complete and reusable passage test collection. The results showed neural ranking advantages, but dense retrieval outcomes were less uniformly dominant.
- The track used MS MARCO datasets with hundreds of thousands of human-annotated training labels for passage and document ranking.
- The new query and judging procedure avoided using Bing top-10 passages during corpus construction and did not require positive qrels to appear in the corpus.
- NIST manually judged passage results and propagated those labels to source documents for the document ranking task.
- 2022 emphasized passage retrieval and collection quality, while document relevance labels were inferred from passage-level judgments.
- Neural ranking methods continued to outperform traditional retrieval, but a top run did not use dense retrieval and single-stage dense retrieval was less competitive than the previous year.
2 Task description
The track defined passage and document ranking tasks around reusable evaluation, with passage ranking receiving most assessment effort. Query selection, near-duplicate handling, and CAL-based judging were designed to produce more complete and discriminative judgments.
- The 2022 track prioritized a sufficiently complete and reusable passage test collection, retaining document ranking as a secondary task.
- The new query-sampling method aimed to make queries more difficult and improve evaluation discrimination, with 250 backup queries sampled using the prior method.
- Passage ranking task: Passage ranking included full ranking and top-100 reranking subtasks, with reranking providing every participant the same candidate set.
- Only passages were judged, their labels were propagated to documents, and one canonical passage represented each near-duplicate cluster.
- Participants submitted 100 passage-ranking runs, of which 82 contributed to the initial judgment pools.
- 76 topics were accepted after parallel CAL threads iteratively added ranked passages while applying relevance-density and judgment-count criteria.
3 Datasets
The track moved to MS MARCO v2, a larger document-native dataset that linked passages and documents and addressed structural problems in v1. The original data arose from crowd-generated answers and sparse passage relevance labels.
- MS MARCO v2 was used for both tasks and was introduced to increase collection scale and document variety.
- MS MARCO origins: The original MS MARCO crowd task asked workers to judge passage answerability, write answers, and identify the passages used.
- Ranking v1: The v1 passage dataset contained 8.8 million distinct passages and highly incomplete relevance judgments derived from selected crowd annotations.
- Ranking v1: The v1 corpus was query-generated and limited passage-document mapping, while document data also had character-set and whitespace problems.
- Ranking v2: The v2 collection contained 11.9 million documents, combining surviving v1 URLs with 9.2 million additional documents selected for useful passages.
- Ranking v2: The new passage-document mapping supports cross-level ranking features, while the larger corpus constrains completely unscalable ranking approaches.
4 Results and analysis
The 2022 track received more runs despite fewer participating groups, with most submissions using pretrained neural models and full-corpus ranking. Neural and full-ranking approaches substantially outperformed traditional and reranking methods, while single-stage dense retrieval was less competitive.
- Participation: 142 runs came from 14 groups, including 59 baseline and 83 official submissions.The track received more runs than in previous years despite fewer participating groups.
- Run types: 85% of submissions were “nnlm” runs using pretrained models, while “trad” runs accounted for 15% and “nn” runs disappeared.The authors suggest this may reflect convergence toward large language models, while noting that the health of this homogenization remains uncertain.
- Run settings: 90% of runs used full ranking, including 89% of passage runs and 93% of document runs.The authors associate this growth partly with neural methods applied in full ranking and with the smaller official reranking candidate set.
- Neural vs. traditional methods: The best “nnlm” run improved NDCG@10 over the best “trad” run by 125% for passages and 76% for documents.The authors caution that percentage comparisons across years or tasks are not directly meaningful because the underlying data distributions differ.
- Neural vs. traditional methods: The best “nnlm” run won on 74 of 76 passage queries and 71 of 76 document queries against the best “trad” run.These correspond to 97% and 93% of queries, respectively, and exceeded the comparable proportions in recent prior years.
- Full ranking vs. reranking: The best “fullrank” run exceeded the best “rerank” run by 36% in passage NDCG@10 and 125% in document NDCG@10.The authors identify possible contributions from full-ranking progress, harder test queries, and under-optimized reranking runs.
5 Reusability of test collection
The 2022 track addressed reusability and metric saturation by expanding passage judging, deduplicating near-duplicate passages, and inferring document labels from passage judgments. These changes produced more complete passage judgments and greater discrimination among runs, although eliminating document judging was crucial to meeting the target.
- 2021’s excess of relevant results harmed reusability by leaving too many relevant passages unjudged for evaluating new systems.
- A relevance density of 0.4 or lower was used as a rule of thumb for sufficiently complete judgments and confident treatment of unjudged results as irrelevant.
- Metric saturation made queries less useful when most systems reached Precision@10 of 1.0, obscuring small differences among top-performing systems.
- The track concentrated judging resources on passages, inferred document labels from passage labels, and deduplicated passages to reduce redundant assessment.
- 2022 had no passage queries above 0.4 relevance density, whereas 17 topics in 2021 failed to reach the stopping condition.
- 24,004 passage judgments contained around 2% duplicates versus 15% of 10,828 judgments in 2021; eliminating document judging was crucial for reaching sufficient topics.
- The 2022 document task had fewer saturated queries and more queries where top runs differed in Precision@10, but its judging scheme differed from 2021.
- Passage evaluation improved: no 2022 query had median Precision@10 of 1.0, while some 2021 passage queries did.
6 Conclusion
The 2022 TREC Deep Learning track aimed to create a reusable passage-ranking collection through harder topics, focused judgment, and passage deduplication. Large-scale-pretrained deep learning models still outperformed traditional retrieval methods, while single-stage dense retrieval was less competitive than in the previous year.
- The 2022 track sought a complete, reliable, and reusable collection for evaluating passage-ranking methods.
- Harder topics, focused judgment, and passage deduplication supported construction of the reusable 2022 passage collection.
- Large-scale-pretrained deep learning models continued to outperform traditional retrieval methods.
- Single-stage dense retrieval was less competitive in 2022 than in the previous year.
- Participation remained healthy, although the number of participating groups decreased slightly because test-query release was delayed.
A Results Including Baselines
Baseline runs were included to diversify evaluation pools and support additional analysis, while the primary results tables summarized passage- and document-ranking runs separately.
- Baseline runs enriched the pools and increased the diversity of approaches used in the evaluation.
- Baseline runs were excluded from the main results tables but included in other analyses and appendix tables.
- Table 5 summarizes passage-ranking results including baseline runs for the first portion of the results.
- Table 6 summarizes the second portion of passage-ranking results including baseline runs.
- Table 7 summarizes document-ranking results including baseline runs.