Source-linked AI summary
DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, Haifeng Wang
TL;DR
Chinese passage-retrieval datasets and existing benchmark labels have limitations, including false negatives and semantically similar training and test queries. DuReader_retrieval builds a large-scale benchmark with pooled human review and query filtering; experiments show that retrieval remains challenging, with salient-phrase and syntactic mismatches.
Problem
Chinese passage-retrieval options were limited, while existing benchmarks can contain unlabeled relevant passages and semantically overlapping training and test queries.
Source: §1 · pp. 1–2Method
DuReader_retrieval builds a benchmark with more than 90K queries and over 8M passages, using pooled human review and filtering semantically similar training queries.
Source: §1 · pp. 1–2, §2.3 · p. 4Results
The benchmark remains challenging, with salient-phrase and syntactic mismatches between queries and paragraphs.
Source: §1 · p. 2Takeaways & Limitations
Passage retrieval still has substantial room for improvement on this benchmark.
Source: §1 · p. 2About 14.8% of the best re-ranking model’s wrong predictions are attributed to false negatives, reflecting the difficulty of capturing all positives in training labels.
Source: §3.5 · p. 7, §6 · p. 9
Abstract
from arXiv · showhide
In this paper, we present DuReader_retrieval, a large-scale Chinese dataset for passage retrieval. DuReader_retrieval contains more than 90K queries and over 8M unique passages from a commercial search engine. To alleviate the shortcomings of other datasets and ensure the quality of our benchmark, we (1) reduce the false negatives in development and test sets by manually annotating results pooled from multiple retrievers, and (2) remove the training queries that are semantically similar to the development and testing queries. Additionally, we provide two out-of-domain testing sets for cross-domain evaluation, as well as a set of human translated queries for for cross-lingual retrieval evaluation. The experiments demonstrate that DuReader_retrieval is challenging and a number of problems remain unsolved, such as the salient phrase mismatch and the syntactic mismatch between queries and paragraphs. These experiments also show that dense retrievers do not generalize well across domains, and cross-lingual retrieval is essentially challenging. DuReader_retrieval is publicly available at https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval.
1 Introduction
DuReader_retrieval is a large-scale Chinese passage-retrieval benchmark designed to improve data quality and evaluate retrieval across domains and languages. Experiments find persistent retrieval challenges, including query–paragraph mismatches and weak cross-domain generalization by dense retrievers.
1 Introduction
- Existing Chinese retrieval datasets are described as small, machine-generated, or lacking human annotation, motivating a large-scale manually annotated benchmark.The paper contrasts DuReader_retrieval with prior non-English datasets on size and construction.Source: §1 · p. 1
- Experiments find salient-phrase and syntactic mismatches, weak dense-retriever cross-domain generalization, and challenging cross-lingual retrieval.The authors also characterize the benchmark as challenging and passage retrieval as having room for improvement.Source: §1 · p. 2
- The benchmark contains more than 90K queries and over 8M unique passages from Baidu Search.Queries come from real Baidu Search requests, and passages come from search results.Source: §1 · pp. 1–2
- Pooled retrieval results are manually checked to reduce false negatives, and semantically similar training queries are removed to limit test-information leakage.The paper identifies false negatives and near-duplicate train–test queries as shortcomings that can affect evaluation or leak testing information.Source: §1 · pp. 1–2
- Two out-of-domain test sets and a cross-lingual set support evaluation of domain generalization and cross-lingual retrieval.Source: §1 · p. 2
2 DuReaderretrieval
DuReaderretrieval builds a Chinese passage-retrieval benchmark from web-search data, using DuReader as its source and adding quality-control procedures. It includes in-domain, out-of-domain, and cross-lingual evaluation sets.
2.1 Task Definition
- The task is to return the top-K relevant passages from a large collection for a given query, ideally ranking all relevant passages as highly as possible.K is a manually defined number.Source: §2.1 · p. 3
2.2.1 An Introduction of DuReader Dataset
- DuReader instances pair user queries and query types with top-five retrieved documents and human-written answers; the retrieval benchmark uses web-search data.Training labels are distant-supervised, while development and test labels are human-annotated.Source: §2.2.1 · p. 3, Table 2 · p. 3
2.2.2 Constructing DuReaderretrieval from DuReader
- DuReader positives require answer-span F1 above 0.5; short paragraphs are concatenated, and document titles are removed to limit query-title overlap.Documents under 256 Chinese characters are concatenated whole; in longer documents, short paragraphs are joined until the new passage exceeds 256 characters.Source: §2.2.2 · p. 4
2.3 Quality Improvement
- 71.53% of queries had at least one false negative relabeled, and average positive paragraphs per query rose from 2.43 to 4.91.Annotators checked top-ranked passages pooled from five retrieval systems.Source: §2.3 · p. 4
- The query-matching model removed 566 training queries similar to 387 development or test queries, using a similarity threshold above 0.5.The removed training queries accounted for approximately 6.45% of development and test queries.Source: §2.3 · p. 5
2.4 Out-of-domain Evaluation
- The out-of-domain test sets sample 949 cCOVID-News cases and 3,999 cMedQA cases for cross-domain evaluation.Source: §2.4 · p. 5
2.5 Cross-lingual Evaluation
- The cross-lingual set pairs English queries with Chinese positive passages and provides 9.5K/4K/2K train/dev/test queries, with machine-translated English manually checked.Source: §2.5 · p. 5
3 Experiments and Results
The experiments compare sparse and neural retrieval baselines, assess dataset-quality interventions, and examine mismatch errors and transfer across domains and languages.
3.1 Baselines
- The baselines use a retrieve-then-rerank framework, evaluating BM25 and neural dual-encoder retrieval alongside cross-encoder rerankers.The neural rerankers are trained using hard negatives from BM25 or the dense retriever.Source: §3.1 · p. 5
3.3 Baseline Performance
- For reranking, neural-retriever hard negatives outperform BM25 negatives on MRR@10 and Recall@1; the paper expects DE w/ BM25 Neg to outperform BM25 across first-stage metrics.The first-stage comparison is reported in Table 4, while reranking comparisons use Table 5.Source: Table 5 · p. 6, §3.3 · p. 6, Table 4 · p. 6
3.2 Evaluation Metrics
- Recall@50 assesses first-stage retrievers, while MRR@10 and Recall@1 are used to assess second-stage rerankers.The experiments report MRR@10, Recall@1, and Recall@50.Source: §3.2 · p. 6
3.4 Effects of Quality Improvements
- All metrics for BM25, the dense retriever, and the reranker improve after false-negative reduction in development data.On duplicated queries, training with semantically similar queries yields higher MRR@10 and Recall@1, which the authors suggest may reflect memorization.Source: §3.4 · pp. 6–7, Figure 2 · p. 7
3.5 The Challenges and Limitations
- Salient-phrase mismatch accounts for 53.4% of incorrect predictions, while syntactic mismatch appears in around 1%.Other challenges account for 22.6% of incorrect predictions, and about 14.8% false negatives remain.Source: §3.5 · p. 7
3.6 Out-of-Domain Evaluation
- On both out-of-domain test sets, the dense retriever loses its clear in-domain advantage over BM25 in zero-shot evaluation, but target-domain fine-tuning restores a large advantage.The fine-tuned performance is treated as an estimated upper bound for the dense retriever on these tests.Source: §3.6 · pp. 7–8, Table 7 · p. 7, Table 8 · p. 7
3.7 Cross-lingual Evaluation
- The zero-shot model is less effective than the supervised model cross-lingually and performs worse cross-lingually than monolingually; the transferred model outperforms the other baselines.The authors conclude that cross-lingual retrieval requires capturing semantic relevance rather than relying simply on shared terms.Source: §3.7 · p. 8, Table 9 · p. 8
4 Related Works
Prior passage-retrieval benchmarks are widely used, but large-scale, high-quality benchmarks for non-English retrieval remain limited. Related work also describes dense retrieval models and the common retrieve-then-rerank architecture.
Passage Retrieval Benchmarks
- English benchmarks include MS-MARCO and Natural Questions, while Chinese datasets TianGong-PDR and Sougou-QCL are described as small or lacking human annotation.The passage also notes false negatives in MS-MARCO and training-query overlap in Natural Questions.Source: §4 · p. 8
Dense Retrieval Model
- Dense retrievers encode queries and documents as contextualized representations and score relevance by similarity; retrieval systems commonly retrieve candidates before reranking them.The passage distinguishes unsupervised pretraining from supervised contrastive training with positives and designed negatives.Source: §4 · p. 9
5 Conclusion
The paper presents DuReaderretrieval as a large-scale Chinese passage-retrieval benchmark, with data-quality strategies and out-of-domain and cross-lingual test sets. It evaluates retrieval baselines and discusses remaining challenges and limitations.
5 Conclusion
- DuReaderretrieval is a large-scale Chinese passage-retrieval dataset with false-negative reduction, removal of semantically similar training queries, and out-of-domain and cross-lingual test sets.The paper evaluates sparse and neural baselines and presents challenges and limitations of the dataset.Source: §5 · p. 9
6 Limitations
The authors report that false negatives remain among model errors despite their quality-improvement strategy. Their out-of-domain test sets are limited to medical data, which may constrain evaluation in other domains.
6 Limitations
- Approximately 14.8% of the best re-ranking model’s wrong predictions are attributed to false negatives despite the quality-improvement strategy.The authors identify difficulty capturing all positive passages in training-data annotation as the primary reason.Source: §6 · p. 9
- Incomplete annotation of positives is the primary stated reason false negatives persist.Source: §6 · p. 9
- The two out-of-domain test sets cover medical data, potentially limiting evaluation in domains such as finance or law.Source: §6 · p. 9
7 Ethical Consideration
DuReader_retrieval is developed for research purposes using open-source data subject to its licenses or publicly available benchmarks. The authors do not guarantee copyright ownership and may remove resources if needed.
7 Ethical Consideration
- DuReader_retrieval is developed only for research purposes.Source: §7 · p. 9
- The dataset draws on open-source projects subject to their license restrictions and publicly available benchmarks.Source: §7 · p. 9
- The authors do not guarantee copyright ownership of the data and may discard resources without copyright if necessary.Source: §7 · p. 9
A.1 Details for Re-ranker Used in Reducing False Negatives
The false-negative reduction re-ranker uses four pretrained model initializations and ensembles their predictions. Its training negatives are sampled from pooled passages.
A.1 Details for Re-ranker Used in Reducing False Negatives
- Four cross-encoder re-rankers use ERNIE, BERT, RoBERTa, and MacBERT initializations.Source: §A.1 · p. 12
- The re-rankers are trained with negatives sampled from the pooled passages.Source: §A.1 · p. 12
- The four re-ranker prediction scores are averaged for each query-passage pair.Source: §A.1 · p. 12
A.2 Baseline Implementation Details
The baseline implementation uses ERNIE-based dual- and cross-encoders, with a second re-ranker initialized from the first. The appendix also specifies negative sampling, training settings, hardware, and dense-index construction.
A.2 Baseline Implementation Details
- Experiments use PaddlePaddle on up to eight NVIDIA Tesla A100 GPUs with 40G RAM.Source: §A.2 · p. 12
- ERNIE 1.0 base initializes the dual-encoder and first cross-encoder, while the second re-ranker starts from the first cross-encoder’s parameters.Source: §A.2 · p. 12
- Training uses cross-batch negatives and samples four negatives per positive from the top 50 retrieved items.Source: §A.2 · p. 12
- Dual-encoders use batch size 256 and train for 10 epochs; cross-encoders use batch size 64 and train for 3 epochs.Source: §A.2 · p. 12
- The dual-encoder uses ADAM at 3e-5 with 0.1 linear warm-up, while cross-encoders use ADAM at 1e-5 without warm-up.Source: §A.2 · p. 12
- Questions and passages are capped at lengths 32 and 384, respectively, and FAISS indexes dense passage representations for inference.Source: §A.2 · p. 12
A.3 Details for Human Annotations
Human annotation uses a package-based review process with an accuracy threshold. The error analysis highlights salient-phrase and syntax mismatches, alongside errors from co-occurring terms and noisy text.
A.3 Details for Human Annotations
- Reviewers independently check 100 random query–paragraph pairs per package and require average accuracy above 93%, with revisions below that threshold.Annotators label pooled top-5 paragraphs for all models in 1K-sample packages.Source: §A.3 · p. 12
A.4 Cases for Challenges in Error Analysis
- The 500 query–passage pairs from the strongest re-ranker illustrate salient-phrase mismatch, syntax mismatch, and other challenges.Salient-phrase cases include entity inconsistency; syntax cases involve confusing subject and object roles.Source: §A.4 · p. 12, Table 10 · p. 13
- In the analysis, about 21% of errors arose when retrieval relied on co-occurring low-frequency query and paragraph terms with unrelated meanings, while about 1.6% arose from noise in either.Source: §A.4 · p. 12