Source-linked AI summary
Overview of the TREC 2019 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M. Voorhees
TL;DR
The paper addresses limited evidence about deep-learning ad hoc ranking under large human-labeled training regimes. It introduces two retrieval tasks with reusable test collections and blind evaluation, finding that neural runs outperformed non-neural runs, while noting important collection and labeling boundaries.
Problem
Prior ad hoc ranking studies largely used small, proprietary, or synthetic data, leaving baseline and human-labeled data-volume questions open.
Method
The track introduces document and passage retrieval tasks with large human-labeled training data, end-to-end and reranking evaluations, blind testing, and reusable collections.
Results
Neural network runs outperformed non-neural network runs in both tasks, while reranking and end-to-end approaches showed no huge difference this year.
Takeaways & Limitations
The results support further study of transfer learning, data mixtures, and multistage training for retrieval ranking.
Takeaways & Limitations
The document labels transfer passage relevance under an assumption and may mismatch the document snapshot because the snapshots were generated at different times.
Abstract
from arXiv · showhide
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods. Deep learning runs significantly outperformed traditional IR runs. Possible explanations for this result are that we introduced large training data and we included deep models trained on such data in our judging pools, whereas some past studies did not have such training data or pooling.
1 Introduction
The track addresses open questions about deep-learning ad hoc ranking by introducing large human-labeled datasets and rigorous comparisons of retrieval methods. It supports both end-to-end retrieval and reranking to study large-data ranking across practical deployment settings.
- Motivation: Work on deep learning for ad hoc ranking had largely used small, proprietary, or synthetic data, leaving comparisons with traditional IR baselines unresolved.A prior proprietary study used 200,000 training queries but could not be independently followed up without released data.
- Motivation: Large human-labeled datasets and blind one-shot evaluation let participants study data volume while reducing overfitting concerns.The track welcomes strong baselines and enables subsampling to investigate the role of training-set size.
- Track scope: It offers document and passage retrieval tasks within a common ad hoc search setting involving fixed collections and new queries.Ad hoc retrieval is described as a common scenario in real-world search applications and TREC.
- Goals: The track compares traditional IR, feature-based LTR, deep architectures, supervision strategies, and training-label subsets in a large-data regime.These comparisons target how ranking methods use large-scale supervision.
- Participation styles: Participants can implement full end-to-end retrieval or only rerank a fixed top-k candidate set, reflecting alternative practical system designs.Reranking lowers participation barriers, while full retrieval allows phase 1 to enrich the phase 2 candidate set.
- Participation styles: The track explicitly encourages studying deep ranking models together with traditional IR indexing structures beyond the predominantly reranking setting.Full-retrieval exploration is an explicit goal of the track.
2 Task description
The document retrieval task separates unrestricted full retrieval from top-100 reranking. Reranking standardizes the initial candidates so participants can focus on relevance estimation and comparisons are more controlled.
- Subtasks: The document retrieval task contains full retrieval and top-100 reranking subtasks.The two subtasks represent end-to-end retrieval and a fixed-candidate ranking setting.
- Full retrieval: In full retrieval, systems rank documents drawn from the complete document collection to model end-to-end retrieval.Most full-retrieval runs returned 1,000 results per query before truncation for the main task table.
- Top-100 reranking: In reranking, every participant receives the same initial ranking of 100 documents and reorders those candidates by estimated relevance.The candidates were retrieved with Indri using Krovetz stemming and stopwords eliminated.
- Top-100 reranking: The reranking setup supports focused relevance-estimator comparisons without requiring participants to build an end-to-end retrieval system.Using identical candidate sets also makes reranking runs more comparable.
- Evaluation: Judging pooled both subtasks and added documents through classifiers and additional judgments for queries with many relevant documents.These procedures aimed to create a sufficiently comprehensive reusable test collection.
2.2 Passage retrieval task
The passage retrieval task mirrors the document task with full retrieval and top-1000 reranking. The reranking subtask fixes a BM25-generated candidate set for comparing passage-ranking methods.
- Subtasks: The passage retrieval task includes full retrieval and top-1000 reranking subtasks.Both subtasks evaluate ranking passages for their likelihood of containing an answer to the query.
- Full retrieval: Full retrieval requires systems to retrieve ranked passages from the entire collection, with up to 1,000 passages submitted per query.Ranking is based on the estimated likelihood that a passage contains an answer.
- Top-1000 reranking: In top-1000 reranking, all participants receive the same 1,000 BM25-retrieved passages and reorder them by estimated answer relevance.BM25 retrieval used no stemming on the full collection.
- Top-1000 reranking: The fixed candidate sets enable comparisons among reranking methods under a common initial retrieval condition.This isolates differences in how systems rerank the provided passages.
- Evaluation: Judging pooled both subtasks and added classifier-selected passages plus extra judgments for queries with many relevant passages.The goal was a comprehensive test collection suitable for future reuse.
3 Datasets
The track uses large, human-assessed datasets for document and passage retrieval, with sparse labels derived from MS MARCO. Document labels transfer passage relevance to source documents under an explicit alignment assumption.
- Training data: Both tasks use large training sets based on human relevance assessments derived from MS MARCO.The labels are sparse, contain no negative labels, and often provide only one positive label per query.
- Labels: Passage positives indicate that a passage contains an answer, while document positives are transferred to the source document containing that passage.The transfer treats a document containing a relevant passage as relevant.
- Labels: The document-label transfer assumes that a document with a relevant passage is relevant, but the document and passage snapshots were generated at different times.The differing snapshot times can create a mismatch between the transferred labels and the document collection.
- Labels: Machine-learning models appeared to benefit from the transferred labels when evaluated with NIST’s non-sparse, non-transferred labels.The authors interpret this as suggesting that the transferred labels are meaningful for the TREC task.
- Dataset documentation: The datasets and run statistics are summarized in Tables 1 and 2.The surrounding text directs readers to the track website for additional dataset details.
- Corpora: The passage corpus matches the MS MARCO passage-retrieval leaderboard, whereas the document corpus was newly released for TREC.Each document contains a URL, title, and body-text field.
4 Results and analysis
Across both retrieval tasks, neural language-model runs outperformed neural and traditional runs, while full retrieval offered only modest NDCG@10 gains over reranking. The analysis also reports strong candidate-recall improvements from selected traditional and expansion-based systems, but cautions that broader validation is still needed.
- 15 groups submitted 75 runs across the document and passage retrieval tasks.
- 29.4% and 14.8%: the best “nnlm” and “nn” document-retrieval runs outperformed the best “trad” run on NDCG@10.
- 37.4% and 23.7%: the best “nnlm” and “nn” passage-retrieval runs outperformed the best “trad” run on NDCG@10.
- 36 of 43 document-retrieval test queries favored the best “nnlm” model over the best “trad” run, or 83.7%.Neither task showed a large class of queries where “nnlm” performed worse, but the authors call for more rigorous blind evaluations and stronger baselines.
- 0.9% and 3.6%: the best “fullrank” runs exceeded the best “rerank” runs by these NDCG@10 margins for document and passage retrieval, respectively.Higher NCG@k was possible for fullrank runs, but was not necessary for good NDCG@10.
- 22.9% and 20.7%: the strongest candidate-recall results improved NCG@100 for documents and NCG@1000 for passages relative to provided reranking candidates.The document result used tuned BM25 and RM3 over doc2query-expanded documents.
5 Reusability of test collections
The track built separate document and passage test collections from shared topics, with NIST judging designed to support reliable reuse. Agreement analyses compare rankings produced by sparse MS MARCO and NIST-based metrics.
- Collection construction: Two separate test collections were constructed for document and passage ranking from a common set of topics judged by the same NIST assessor.Document and passage assessments occurred at different times.
- Collection construction: NIST selected 52 topics from 200 submitted topics for judging based on document-run behavior under sparse MS MARCO judgments.Topics with median MRR scores greater than 0.0 and no more than 0.5 were candidates.
- Metric agreement: Metrics-agreement analyses compare system orderings using Kendall correlation between RR from sparse MS MARCO labels and RR or NDCG@10 from NIST labels.The analyses are broken down by group and model type for both retrieval tasks.
5.1 Collection Robustness
Collection robustness was assessed through simulated HiCAL judgments, run-removal experiments, and alternative stopping criteria. System rankings were generally stable, although reuse cannot be guaranteed because simulations depend on submitted runs.
- Robustness tests: Leave-Out-Uniques tests assess collection reusability by removing team-specific relevant documents and comparing resulting system rankings with the originals.Kendall’s τ measures ranking agreement, while rank drops capture the largest change experienced by any run.
- Robustness tests: Standard Leave-Out-Uniques testing is unsuitable because HiCAL does not depend on runs for document selection, making unique relevant documents undefined.The authors instead simulate HiCAL repeatedly after removing a team’s depth-10 documents.
- Robustness tests: Ten-trial simulations used the original depth-10 pools and tracked the first 2,500 documents encountered per topic, with separate trials for document and passage collections.Each trial used a different random seed.
- Robustness results: System rankings were generally stable across simulations, supporting the reusability of the collections; Figure 9 summarizes run positions across 120 document and 130 passage trials.The heat map places ranks on the x-axis and runs on the y-axis.
- Stopping criteria: The simulations constructed qrels with the same number of judgments per topic as the official qrels, while traces enabled testing alternative stopping conditions.Unjudged documents encountered during simulation were treated as not relevant.
- Stopping criteria: Equal-budget and heuristic stopping criteria were compared, with heuristic allocation capping each topic at 1,000 judgments and potentially changing judged-document totals.The equal-budget method assigns totalBudget/X judgments to each topic and requires at least three relevant documents for inclusion.
- Stopping results: HiCAL remained stable across tested stopping conditions, with differences in mean scores driven mainly by including or excluding topics rather than changing individual topic scores.Prec(10) ranking differences arise from changed mean scores when topics are omitted because the top-10 pool is always judged.
- Scope: The evidence supports high-quality reusable collections, but reuse cannot be established with certainty because LOU tests and simulations depend on submitted track results.This dependence limits certainty for partially judged collections.
6 Conclusion
The track introduced large reusable datasets and evaluated neural and traditional ranking methods for document and passage retrieval. Neural runs outperformed non-neural runs, while reranking and end-to-end retrieval showed little difference this year.
- Contributions: Two large training datasets produced reusable ad hoc test collections for document retrieval and passage retrieval.The collections were created for the track’s two tasks.
- Results: Neural network runs outperformed non-neural network runs on both document and passage retrieval tasks in the presence of large training data.The strongest neural runs tended to use transfer learning with pretrained language models such as BERT.
- Results: Reranking and end-to-end retrieval showed no huge performance difference in this year’s track, with some runs performing well in both regimes.The authors expect end-to-end retrieval could improve if it recalls documents unavailable to reranking.
- Future work: Future evaluation should include more runs of all types and guard against overfitting when reused test sets are repeatedly used during method development.The authors identify new, unseen test collections as the most convincing evidence for a new approach.