Source-linked AI summary
Overview of the TREC 2021 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin
TL;DR
The report examines TREC 2021 retrieval benchmarking after refreshing the MS MARCO passage and document collections. It analyzes neural and traditional methods, retrieval pipeline stages, query effects, and dataset-label issues, finding continued neural advantages while single-stage systems remained below multistage pipelines.
Problem
The track needed to assess retrieval methods on substantially refreshed, larger collections while considering whether judgments and training labels remained complete and reliable.
Method
The authors benchmarked submitted document and passage retrieval runs, comparing neural with traditional methods, full retrieval with reranking, single-stage with multistage pipelines, and query-length effects.
Results
Large-scale-pretrained neural rankers outperformed traditional methods across both tasks, while single-stage retrieval was 6% worse for documents and 10% worse for passages on NDCG@10 than the top runs.
Takeaways & Limitations
Single-stage deep retrieval performed well and gained ground, but multistage retrieval still led on both document and passage tasks.
Takeaways & Limitations
The v1 datasets were constrained by query-dependent corpus construction, sparse qrels, changing web content, and extraction-related text problems.
Abstract
from arXiv · showhide
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.
1 Introduction
The TREC 2021 Deep Learning Track benchmarked retrieval methods using refreshed MS MARCO passage and document collections in a large-data regime. It found continued advantages for pretrained deep neural rankers, while single-stage retrieval remained below multistage pipelines.
- 1 Introduction: The track continued benchmarking ad hoc retrieval methods using MS MARCO's large set of human-annotated training labels for passage and document ranking.The 2021 track was the third year of this evaluation effort.
- 1 Introduction: The refreshed collections grew nearly four times for documents and nearly 16 times for passages.The collection refresh also raised questions about NIST judgment completeness and mapped training-label quality.
- 1 Introduction: Participants annotated runs by dense-retrieval use and whether ranking was single-stage, enabling analysis against previous state-of-the-art methods.The track also aimed to encourage neural methods in first-stage retrieval.
- 1 Introduction: Deep neural ranking models using large-scale pretraining continued to outperform traditional retrieval methods.Single-stage retrieval achieved good performance on both tasks but remained below multistage retrieval pipelines.
2 Task description
The track evaluated document and passage retrieval through full-retrieval and shared-candidate reranking subtasks. Participants used common queries, while stratified sampling and budget-constrained NIST judging produced separate judged test sets.
- 2 Task description: The 2021 track comprised document retrieval and passage retrieval tasks, each offering full retrieval and top-100 reranking subtasks.Participants could submit up to three runs per task and report external data, pretrained models, resources, and model style.
- 2 Task description: 477 queries were shared across tasks, with equal stratified sampling from short and long queries for pooling and judging.Queries with at least 10 words formed the long-query stratum.
- 2 Task description: 57 document queries and 53 passage queries formed the judged test sets after NIST selected queries under budget constraints.The document set contained 28 short and 29 long queries; the passage set contained 25 short and 28 long queries.
- 2.1 Document retrieval task: Document full retrieval ranked items from the complete collection, while reranking began from a common Pyserini-retrieved list of 100 documents.The shared candidate set made reranking runs more comparable and focused evaluation on relevance estimation.
- 2 Task description: Document judgments mapped levels 3, 2, and 1 to relevant, whereas passage judgments mapped only levels 3 and 2 to relevant for binary metrics.Both tasks collected judgments on a four-point scale.
- 2.2 Passage retrieval task: Passage full retrieval ranked up to 100 passages from the complete collection, while reranking reordered 100 Pyserini-generated candidates.Both subtasks ranked passages by estimated likelihood of containing an answer to the query.
3 Datasets
MS MARCO v2 refreshed the ranking datasets by identifying documents first, expanding the corpus, and adding passage-document mappings and text-cleanup improvements. The report also documents limitations inherited from v1, including sparse labels, changing web content, and corpus construction tied to queries.
- 3 Datasets: MS MARCO v2 was introduced for both tasks and was used for the first time in TREC 2021.The report compares the new ranking data with the original generation data and v1 ranking datasets.
- 3 Datasets: The original MS MARCO generation data processed one million queries, with crowd workers judging answerability and writing non-extractive answers from related passages.Workers also annotated which passages supported each answer.
- 3 Datasets: V1 passage ranking formed an 8.8-million-passage union of top-ten lists and used selected crowd-annotated passages as positive qrels, producing highly incomplete labels.Training and evaluation with sparse qrels were reported as correlated with more comprehensive NIST judgments.
- 3 Datasets: V1 document data was affected by web-page changes, missing URLs, differing extraction pipelines, and character-set processing issues.Only 3.2 million of the 3.5 million collected URLs still existed when documents were extracted.
- 3 Datasets: V1 corpus construction depended on the original queries and often supplied only one passage per document, preventing use of passage-document mappings and making the setup unrealistic.The document dataset also had character-set and whitespace problems.
- 3 Datasets: V2 added passage-document mappings, expanded the corpus, reduced encoding and whitespace issues, and supported future document-processing tasks.The mapping enables using document information in passage ranking and passage information in document ranking.
4 Results and analysis
TREC 2021 found that neural methods remained stronger than traditional retrieval across both tasks, while full retrieval and multistage pipelines retained advantages. The refreshed collections also exposed evaluation and label-quality concerns, especially for sparse labels and transferred passage judgments.
- Participation and run composition: 19 groups submitted more runs than in prior years, with “nnlm” runs reaching 76% of submissions and “fullrank” runs 79%.The track had 19 participating groups, compared with 25 in 2020 and 15 in 2019; baseline runs contributed to the larger run count.
- Neural versus traditional methods: 15% and 36% were the NDCG@10 improvements of the best “nnlm” over “trad” runs for document and passage ranking, respectively.The corresponding gaps were consistently larger for passage ranking than document ranking, although cross-year and cross-task percentage comparisons are cautioned against because data distributions differ.
- Neural versus traditional methods: 41 out of 57 (72%) document queries and 47 out of 53 (89%) passage queries favored the best “nnlm” run over the best “trad” run.The document-query win rate dropped from previous years, while the passage-query win rate was marginally higher than in 2019 and 2020.
- End-to-end retrieval versus reranking: 4% and 6% were the NDCG@10 advantages of the best “fullrank” over “rerank” runs for document and passage ranking, respectively.The authors report stronger correlation between NDCG@10 and NCG@100 and caution that significant overall gains from neural methods under full retrieval remain unsettled.
- Single-stage versus multistage retrieval: 6% and 10% were the NDCG@10 deficits of single-stage retrieval versus the top run for document and passage ranking, respectively.Single-stage methods performed surprisingly well but remained behind multistage rank-and-prune approaches on both tasks.
- Query length: Evaluation based solely on long queries correlated more with all-query rankings, while long-versus-short system rankings were relatively weakly correlated.This pattern was stronger for “nnlm” systems than for “trad” systems; longer-query pools also contained more relevant items than shorter-query pools.
- Label agreement and collection refresh: 0.43 and 0.51 were the 2021 Kendall’s Tau agreements between sparse MS MARCO RR and full-label NDCG@10 for document and passage ranking.Agreement decreased over the track’s years, and passage labels transferred by text matching raised potential for increased label noise.
- Label agreement and collection refresh: 0.72 oldness@10 runs could top MRR while not producing the best NDCG@10, indicating sparse-label evaluation favored old documents.The expected oldness@10 for a random ranker was 0.225, based on 2.7 million old URLs among 12 million v2 URLs.
5 Conclusion
The 2021 track refreshed both collections and continued to find that pretrained deep models outperform traditional retrieval methods. Single-stage deep retrieval gained ground but remained below multistage pipelines.
- Collection refreshes significantly increased the sizes of both the document and passage collections.
- Deep learning models with large-scale pretraining continued to outperform traditional retrieval methods.
- Single-stage retrieval with deep models gained more ground in 2021.
- The report summarizes submitted-run analyses and the impact of the v2 collections.