Source-linked AI summary
T2Ranking: A large-scale Chinese Benchmark for Passage Ranking
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, Jin Ma
TL;DR
Existing passage-ranking benchmarks are concentrated in English, while Chinese datasets have limitations in scale, fine-grained judgments, and false negatives. T2Ranking provides a large-scale Chinese benchmark with four-level annotations and diverse passages; experiments find the tasks challenging, with room for improvement.
Problem
Common passage-ranking datasets focus on English, while Chinese datasets have limitations in scale, fine-grained relevance annotations, and false negatives.
Source: §1 · p. 2Method
T2Ranking constructs a Chinese passage-ranking benchmark using segmentation, clustering, active-learning-based sampling, and four-level relevance judgments.
Source: §4.1 · p. 4, §4.2 · p. 4, §4.4 · p. 6Results
Experiments across passage retrieval and re-ranking find T2Ranking challenging, with room for further performance improvement.
Source: §1 · p. 2Takeaways & Limitations
The authors suggest that more sophisticated models may bring further improvement on T2Ranking.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Passage ranking involves two stages: passage retrieval and passage re-ranking, which are important and challenging topics for both academics and industries in the area of Information Retrieval (IR). However, the commonly-used datasets for passage ranking usually focus on the English language. For non-English scenarios, such as Chinese, the existing datasets are limited in terms of data scale, fine-grained relevance annotation and false negative issues. To address this problem, we introduce T2Ranking, a large-scale Chinese benchmark for passage ranking. T2Ranking comprises more than 300K queries and over 2M unique passages from real-world search engines. Expert annotators are recruited to provide 4-level graded relevance scores (fine-grained) for query-passage pairs instead of binary relevance judgments (coarse-grained). To ease the false negative issues, more passages with higher diversities are considered when performing relevance annotations, especially in the test set, to ensure a more accurate evaluation. Apart from the textual query and passage data, other auxiliary resources are also provided, such as query types and XML files of documents which passages are generated from, to facilitate further studies. To evaluate the dataset, commonly used ranking models are implemented and tested on T2Ranking as baselines. The experimental results show that T2Ranking is challenging and there is still scope for improvement. The full data and all codes are available at https://github.com/THUIR/T2Ranking/
1 INTRODUCTION
T2Ranking is introduced as a large-scale Chinese benchmark for passage retrieval and re-ranking. It targets limitations in existing datasets, including limited human-generated fine-grained judgments and false negatives, and experiments show room for improvement.
1 INTRODUCTION
- Passage ranking comprises retrieval and re-ranking stages, and its results support applications including question answering and reading comprehension.Retrieval recalls relevant passages, while re-ranking orders retrieved passages by relevance.Source: §1 · pp. 1–2
- Existing datasets, especially in Chinese, have limitations in scale and human-generated judgments, fine-grained annotation coverage, and false negatives.False negatives occur when relevant results are labeled irrelevant and can harm evaluation accuracy.Source: §1 · p. 2
- Experiments evaluate passage retrieval and re-ranking models and find T2Ranking challenging, with potential for further performance improvement.Table 1 summarizes statistics of commonly used datasets, including query and passage counts and first- and second-stage task coverage.Source: §1 · p. 2, Table 1 · p. 3
- T2Ranking contains more than 300K queries and over 2M unique passages, with fine-grained relevance annotations and auxiliary resources.The dataset supports both passage retrieval and re-ranking.Source: §1 · p. 2
- The benchmark uses passage segmentation and clustering to improve passage integrity and diversity, and active learning-based sampling to improve annotation efficiency and quality.Queries are sampled from Sogou logs, and expert annotators provide four-level judgments for query-passage pairs.Source: §1 · p. 2
2 RELATED WORK
Related benchmarks cover English and Chinese passage ranking, with differences in their query and passage sources, relevance labels, and supported tasks. The cited Chinese datasets vary in scale, annotation type, and passage form.
2 RELATED WORK
- Some benchmarks support both first-stage retrieval and second-stage re-ranking, while others provide annotations only for re-ranking candidate passages.The paper uses FR for passage retrieval and SR for passage re-ranking.Source: §2 · p. 3
- CAR, TriviaQA, and MS-MARCO are commonly used English benchmarks, with passage relevance in TriviaQA and MS-MARCO judged in binary fashion.MS-MARCO uses question-based queries from Bing and passages from returned web documents.Source: §2 · p. 3
- Chinese datasets include Sogou-SRR, Sogou-QCL, TianGong-PDR, and machine-translated mMarco-Chinese, with varied scales and relevance-label sources.Sogou-SRR provides four-grade relevance scores; Sogou-QCL uses click labels, while TianGong-PDR provides four-grade human assessments.Source: §2 · p. 3
- DuReaderretrieval uses question-based queries and human-generated answers, while Multi-CPR covers three vertical domains with human binary judgments on search-result titles.Multi-CPR passages are titles rather than passages extracted from web documents.Source: §2 · p. 3
3 TASK DEFINITION
T2Ranking defines passage ranking as a retrieval-then-re-ranking pipeline. Retrieval returns candidate passages for a query, and re-ranking orders that candidate set by relevance.
3 TASK DEFINITION
- T2Ranking focuses on passage retrieval and re-ranking, matching a retrieval-then-re-ranking pipeline.Source: §3 · p. 3
- Given a query, a retrieval model retrieves a candidate set of passages from the corpus.Source: §3 · p. 3
- Re-ranking derives an ordering over the retrieved candidates, placing more relevant passages higher and requiring fine-grained relevance modeling.Retrieval instead emphasizes efficiently finding relevant passages from a large corpus.Source: §3 · p. 4
4 DATASET CONSTRUCTION
T2Ranking is constructed through query sampling, document retrieval, passage extraction, and expert relevance annotation. The pipeline combines model-based segmentation, training-set de-duplication, and active learning to shape its annotated data.
4.1 Overall Pipeline
- Queries are sampled and filtered from Sogou, documents are retrieved from multiple search engines, and at least three expert annotators provide four-level judgments.A fourth annotator is consulted for three-way disagreement; pairs remain excluded if all four scores differ, and the final label uses majority voting.Source: §4.1 · p. 4
- For retrieval evaluation, Level-2 and Level-3 passages count as relevant; Levels 0 and 1 count as irrelevant.The four levels distinguish mismatch, relevance without satisfying the information need, partial satisfaction, and a precise answer.Source: §4.1 · p. 4
4.2 Model-based Passage Segmentation
- Model-based segmentation addresses natural paragraphs that can combine topics and fixed sliding windows that can omit complete semantics.The model is trained on well-written Baike and Wikipedia documents and predicts word-level boundaries; a lower threshold produces more passages.Source: §4.2 · pp. 4–5
4.3 Clustering-based Passage De-duplication
- Ward hierarchical clustering groups nearly duplicated training passages, and only one passage per cluster is annotated to save cost while retaining diversity.Test queries instead receive annotations for all segmented passages to mitigate false negatives.Source: §4.3 · pp. 5–6
4.4 Active Learning-based Data Sampling
- Active learning filters out high-confidence pairs and low-confidence noise, sending the remaining query-passage pairs for expert annotation and model updates.The process iteratively adds newly annotated pairs to training data and updates the cross-encoder.Source: §4.4 · p. 6
5 DATA STATISTICS
T2Ranking reports query-domain, query-diversity, passage, and relevance-annotation statistics. Its training and test queries have consistent domain distributions, while its queries are more diverse than those in compared datasets.
5 DATA STATISTICS
- Training and test queries have nearly identical maximum and mean lengths, and their domain distributions are consistent across a broad range of domains.Figure 3 presents the domain statistics for the two sets.Source: §5 · p. 6, Figure 3 · p. 7
- T2Ranking queries have lower ILS scores than those in compared datasets, indicating higher diversity even than Multi-CPR queries.Table 4 reports ILS scores; lower scores indicate higher query diversity.Source: §5 · p. 6, Table 4 · p. 7
- T2Ranking contains 2,303,643 passages segmented from 1,752,482 web documents, averaging 1.31 passages per document and 632.6 in mean passage length.Source: §5 · p. 6
- Each query has an average of 6.25 annotated passages in training and 15.75 in test, with the relevance annotations distributed across four levels.Figure 4 shows the annotation distribution.Source: §5 · p. 6, Figure 4 · p. 7
6 EXPERIMENTS AND RESULTS
The experiments evaluate retrieval followed by re-ranking on T2Ranking using commonly used sparse and dense models. Dual-encoder retrieval beats BM25, while re-ranking dual-encoder candidates beats re-ranking BM25 candidates; both tasks remain challenging under the benchmark’s annotations.
6 EXPERIMENTS AND RESULTS
- The experiments follow a retrieval-then-re-ranking pipeline and examine commonly used retrievers and re-rankers on T2Ranking.Retrieval baselines include sparse and dense models; the re-ranking experiment uses a cross-encoder trained on hard negatives mined from the corpus.Source: §6 · p. 7, §6.1 · p. 7, §6.2 · p. 8
6.1 Retrieval Performance
- Dual-encoder models significantly outperform BM25 on retrieval; BM25 Recall@50 is 0.4918 on T2Ranking, versus 0.601 on MS-MARCO-DEV Passage and 0.700 on DuReaderretrieval.The test set averages 4.74 positive paragraphs per query, which the paper says makes retrieval more difficult and eases false negatives to some extent.Source: Table 5 · p. 8, §6.1 · p. 8
6.2 Re-ranking Performance
- Re-ranking dual-encoder candidates significantly outperforms re-ranking BM25 candidates, while T2Ranking re-ranking performance is lower than on other benchmarks.The authors attribute the candidate-source difference to the dual encoder’s higher recall, and the lower benchmark performance to more fine-grained relevant passages and more diverse queries.Source: §6.2 · p. 8, Table 6 · p. 9
7 CONCLUSION
T2Ranking is a large-scale Chinese benchmark for passage retrieval and re-ranking, built with strategies for passage processing and annotation. Its expert-assessed four-level labels and scale are intended to support nuanced modeling and more precise evaluation.
7 CONCLUSION
- T2Ranking contains over 300K queries, more than 2M unique passages, and around 2.4 million expert-annotated query-passage pairs; the authors describe it as the largest Chinese fine-grained passage-ranking benchmark to their knowledge.It covers both retrieval and re-ranking tasks.Source: §7 · p. 9
- Construction combines model-based passage segmentation, clustering-based de-duplication, and active-learning-based sampling.Segmentation aims to preserve complete passage semantics; de-duplication removes highly similar passages retrieved for a query, and active learning selects informative samples while minimizing annotation needs.Source: §7 · p. 9
- Expert annotators apply a four-level fine-grained relevance scheme to both training and test sets.The paper says this supports more nuanced modeling during training and more precise evaluation during testing.Source: §7 · p. 9