Source-linked AI summary
Overview of the TREC 2020 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos
TL;DR
The paper studies which ranking methods work best for document and passage retrieval when large training data is available, using rigorous single-shot TREC evaluation. It compares neural, language-model-based, and traditional runs, finding that neural runs outperform non-neural runs on both tasks, while reranking and full retrieval show no huge difference.
Problem
Ad hoc ranking had not initially shown dramatic deep-learning improvements, raising questions about whether larger training data changes retrieval performance.
Method
The track provides large training datasets and reusable test collections, then compares retrieval methods through blind single-shot evaluation in full-ranking and reranking settings.
Results
Neural network runs outperformed non-neural runs on both tasks, while ORCAS improved within-group NDCG@10 comparisons but was not necessary for the highest NDCG@10.
Takeaways & Limitations
The results provide further evidence that neural methods, particularly those using language-model pretraining, perform strongly in the large-data regime.
Takeaways & Limitations
The track has not yet shown a strong advantage of fullrank over rerank, and transferred document labels may mismatch because the document snapshot was generated at a different time.
Abstract
from arXiv · showhide
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.
1 Introduction
The track addresses whether deep learning can improve ad hoc ranking when large human-labeled training sets are available. It establishes reusable datasets and rigorous evaluation to compare retrieval methods in this regime.
- Motivation: Deep learning produced dramatic gains in speech recognition and computer vision, but early ad hoc ranking results did not show similarly large improvements.This raised questions about whether deep learning helped information retrieval and whether training sets were too small.
- Track goal: The TREC Deep Learning Track studies which retrieval methods work best when large training data is available.Its associated MS MARCO leaderboards introduced human-labeled training sets that were previously unavailable.
- Evaluation setting: Document retrieval and passage retrieval each provide hundreds of thousands of human-labeled training queries, but their labels are sparse.Evaluation instead uses more comprehensive relevance judgments on newly selected TREC test queries.
- Track design: The 2020 track provides large training datasets, reusable test collections, blind single-shot evaluation, and both end-to-end and reranking setups.These goals support comparisons of deep learning and traditional ranking methods in two retrieval tasks.
2 Task description
The track evaluates document retrieval through full retrieval and top-100 reranking. Participants submit model information and runs for two tasks under a shared test-query setup.
- Track structure: The track contains document and passage retrieval tasks, with participants allowed up to three runs per task using the same 200 test queries.The submission limit was not strictly enforced.
- Evaluation: The judging process selected 45 document queries and 54 passage queries under budget constraints to create reusable test collections.The document and passage judged queries were not subsets of one another.
- Run reporting: Participants reported external data, pretrained models, other resources, and model style for their submitted runs.The paper then describes the document and passage tasks and their datasets.
- Document retrieval: Document retrieval has full retrieval and top-100 reranking subtasks.Full retrieval models the end-to-end scenario, while reranking starts from an initial ranking and focuses on relevance estimation.
- Document retrieval: Reranking uses the same initial 100-document candidate list for every participant, improving comparability and avoiding the need to implement end-to-end retrieval.The initial rankings were retrieved with Indri using Krovetz stemming and stopwords eliminated.
2.2 Passage retrieval task
Passage retrieval tests both end-to-end ranking and reranking from a shared top-1000 candidate set. Relevance is judged on a four-point scale centered on whether passages answer the query.
- Task structure: Passage retrieval includes full retrieval and top-1000 reranking subtasks.Both subtasks evaluate ranking methods for passages rather than whole documents.
- Full retrieval: Full retrieval ranks passages from the complete collection by their estimated likelihood of containing an answer, with up to 1000 passages submitted per query.This is the passage task’s end-to-end retrieval setting.
- Reranking: Top-1000 reranking gives all participants the same 1000 BM25 candidates and asks them to rerank those passages by estimated answer likelihood.A shared candidate set enables direct comparison of reranking methods.
- Relevance judgments: Passage judgments use four levels: perfectly relevant, highly relevant, related, and irrelevant.Perfectly relevant passages contain the exact answer, while related passages do not answer the query.
- Relevance judgments: For binarized metrics, passage judgments 3 and 2 are relevant, while document judgments 1 and 0 are irrelevant.This mapping converts the four-point scales for metrics that require binary relevance.
3 Datasets
The track combines sparse MS MARCO-derived training labels with the ORCAS click dataset and compares its TREC collections with sparse leaderboard evaluation. The supplied figure indicates that BERT-like language-model runs performed best on both tasks.
- Training data: Both tasks use large, sparse MS MARCO-derived training sets with no negative labels and often one positive label per query.The labels resemble some real-world training data such as click logs.
- Label construction: Document labels transfer passage-level relevance to source documents under the assumption that a document containing a relevant passage is relevant.The paper notes possible mismatch because the document snapshot was generated at a different time, although models trained with these labels appear useful under NIST judgments.
- ORCAS: The ORCAS dataset supplies large-scale search-engine click data that can augment training or serve as an additional document field.It is described as almost 50 times larger than the main training set.
- Evaluation datasets: The MS MARCO leaderboards use the same corpus and sparse training data but evaluate with sparse labels, whereas TREC uses NIST test sets.The paper analyzes agreement between these evaluation types.
- Reported comparison: Figure 1 reports NDCG@10 by run type, with nnlm runs using language models such as BERT performing best on both tasks.The figure contrasts nnlm, nn, and trad runs.
4 Results and analysis
The track compares neural, pretrained neural-language-model, and traditional ranking methods across document and passage retrieval, using multiple evaluation metrics and run categories. Neural-language-model runs generally outperform traditional runs, while full retrieval shows little consistent advantage over reranking and evaluation agreement varies by task and run type.
- Run categories: 70 (57%) runs used large-scale pretrained neural language models, compared with 13 (10%) neural-network and 40 (33%) traditional runs.Runs were manually categorized as nnlm, nn, or trad.
- Neural vs. traditional methods: The best nnlm and nn document-retrieval models outperformed the best traditional model by 23% and 11%, respectively.The best nnlm model also held an advantage over the best nn model.
- Neural vs. traditional methods: The best nnlm and nn passage-retrieval runs exceeded the best traditional run by 42% and 17%, respectively.The paper offers vocabulary mismatch and greater experience with the public passage leaderboard as possible explanations.
- Neural vs. traditional methods: The best nnlm model beat the best traditional run on 38 of 45 document queries, or 84%, with neither task showing a large class of nnlm losses.The passage task showed a similar query-level pattern.
- End-to-end retrieval vs. reranking: Fullrank runs did not achieve much better NDCG@10 than rerank runs; the best fullrank run was 5% better for documents but 0.3% worse for passages.Higher NCG@k was possible for fullrank runs but was not necessary for good NDCG@10.
- Additional training data: Using ORCAS always improved within-group NDCG@10 comparisons, with a maximum increase of around 0.0513, although it was not required for the highest score.The authors suggest ORCAS provides information unavailable in the track’s training data and that deep models may still need more data.
- Evaluation agreement: Agreement between MS MARCO and NIST evaluations was lower for document retrieval than passage retrieval, with Kendall correlations of 0.46 and 0.69.Agreement on the best neural-network runs was high, whereas correlation for document traditional runs was close to zero.
5 Conclusion
The 2020 track released reusable retrieval datasets and test collections, compared neural and non-neural methods, and examined full retrieval versus reranking. Neural runs outperformed non-neural runs, while ORCAS helped but was not required for the highest NDCG@10, and evaluation-label agreement differed across tasks.
- Datasets and evaluation: The track provided document and passage training datasets, reusable ad hoc test collections, and the ORCAS click dataset from Bing logs.The 2020 training datasets matched those from 2019, while ORCAS added large-scale click data.
- Results: Neural network runs outperformed non-neural runs on both retrieval tasks in the large-training-data setting.This pattern held for both document retrieval and passage retrieval.
- Results: Full retrieval and reranking showed no huge performance difference, with some runs performing well in both regimes.The comparison concerns end-to-end retrieval versus reranking approaches.
- Evaluation and reuse: More submissions, especially non-neural runs, may make this year’s test collections fairer and more reusable for evaluating unseen non-neural methods.The authors also plan to encourage methodological diversity and diversify judging pools.
- Evaluation and reuse: Passage-task results correlated strongly across evaluation-label types, whereas document-task correlation was lower and may have multiple causes beyond ORCAS.The official labels were more comprehensive than the typically sparse MS MARCO labels.