Source-linked AI summary
Simple Entity-Centric Questions Challenge Dense Retrievers
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, Danqi Chen
TL;DR
Dense retrievers can excel on popular QA benchmarks yet remain poorly generalized to simple entity-centric questions. The paper introduces EntityQuestions, analyzes entity and question-pattern effects, and tests augmentation and specialized question encoders. It finds that rare entities and unseen patterns expose major weaknesses, while augmentation does not consistently transfer across domains and robust passage encoders support memory-efficient question adaptation.
Problem
Dense retrievers are not yet robust enough to replace sparse methods, particularly for questions involving unseen entities and patterns.
Method
The paper constructs EntityQuestions from Wikidata facts, disentangles entity and question-pattern effects, and investigates data augmentation plus specialized question encoders with a fixed passage index.
Results
Dense retrievers drastically underperform BM25 on EntityQuestions, degrade on rare entities, and generalize to unseen entities only when the question pattern was observed during training.
Takeaways & Limitations
Robust passage encoders facilitate memory-efficient transfer through specialized question encoders, whereas data augmentation does not consistently improve unseen-domain performance.
Takeaways & Limitations
The Wikidata-based training setup may amplify harmful or toxic biases learned from BERT and Wikidata triples, especially for stereotyped and marginalized groups.
Abstract
from arXiv · showhide
Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples. However, in this paper, we demonstrate current dense models are not yet the holy grail of retrieval. We first construct EntityQuestions, a set of simple, entity-rich questions based on facts from Wikidata (e.g., "Where was Arve Furset born?"), and observe that dense retrievers drastically underperform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions towards addressing this critical problem. First, we demonstrate that data augmentation is unable to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using specialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.
1 Introduction
The paper argues that dense retrievers remain insufficiently robust despite strong benchmark performance. EntityQuestions exposes poor generalization, especially when entities are rare or question patterns are unseen, while the proposed investigations examine augmentation and specialized encoders.
- Dense retrievers are not yet robust enough to replace sparse methods across input distributions.
- 49.7% versus 72.0% average top-20 retrieval accuracy shows DPR substantially underperforms BM25 on EntityQuestions.The gap reaches 60% absolute on some question patterns.
- EntityQuestions evaluates simple entity-centric questions to test whether dense retrievers generalize beyond familiar benchmark distributions.
- Dense models handle common entities but degrade on rarer entities and generalize to unseen entities only when the question pattern appeared during training.
- The paper investigates data augmentation and specialized question encoders as practical approaches to improving transfer across question distributions.The specialized-encoder approach uses a fixed passage index, while augmentation is evaluated across single- and multi-task fine-tuning.
- Data augmentation can close single-domain gaps but does not consistently improve performance on unseen domains.
2 Background and Related Work
The background contrasts sparse lexical matching with supervised dense representations and frames unseen-question generalization as an under-explored weakness. Prior work reports memorization, rare-entity difficulties, and broad cases where dense retrieval underperforms BM25.
- Sparse retrieval: Sparse retrievers use weighted term matching without training on a particular data distribution, favoring lexical matches but missing synonyms and paraphrases.
- Dense retrieval: Dense retrievers learn question and passage representations from supervised QA data, typically using pretrained language models such as BERT.DPR uses separate question and passage encoders with a contrastive training objective.
- Generalization problem: Dense retriever generalization to unseen questions remains under-explored despite strong in-domain performance.
- Generalization problem: Prior studies find models memorize overlapping benchmark questions, perform worse on rare entities, and often underperform BM25 in zero-shot evaluation.
3 EntityQuestions
EntityQuestions is a Wikidata- and Wikipedia-grounded benchmark for simple relation questions, evaluated with macro-averaged top-20 retrieval accuracy. DPR substantially trails BM25, and REALM also remains below BM25, although the comparison uses different passage-index units.
- Dataset collection: EntityQuestions converts Wikidata facts into natural-language questions using manually defined templates across 24 common relations.Triples come from T-REx so each fact is aligned with an evidentiary Wikipedia sentence.
- Results: Figure 1 groups top-20 retrieval accuracy for place-of-birth and creator questions into eight buckets by entity frequency in Wikipedia.The figure distinguishes randomly selected Wikidata entities from entities appearing in the NQ training set.
- Dataset collection: The benchmark includes person and non-person relations, samples up to 1,000 facts per relation, and reports macro-averaged accuracy across relations.
- Results: 49.9% absolute top-20 accuracy separates BM25 from DPR on “Where was [E] born?”, while multi-dataset DPR improves average performance from 49.7% to 56.7%.The multi-dataset model still remains below BM25, with especially large gaps for person entities.
- Results: REALM scores 19.6% on average and remains much lower than BM25 across relations, indicating that salient-span pretraining does not solve this benchmark’s generalization problem.
- Results: REALM and DPR retrieval accuracies are not directly comparable because their indexes use 288 BPE-token blocks and 100-word passages, respectively.
4 Dissecting the Problem: Entities vs. Question Patterns
The analysis separates entity frequency from question-pattern exposure, finding that DPR struggles primarily with rare entities unless the question pattern appears during training. Fine-tuning the passage encoder is substantially more effective than fine-tuning only the question encoder.
- DPR is evaluated on place-of-birth, headquarter, and creator templates to distinguish novel-entity effects from unseen-question-pattern effects.
- 4.1 Dense retrievers exhibit popularity bias: DPR performs well on common entities but quickly degrades on rarer entities, whereas BM25 is less sensitive to entity frequency.Evaluation uses eight frequency buckets based on Wikipedia hyperlink counts and compares randomly sampled entities with entities seen during NQ training.
- 4.1 Dense retrievers exhibit popularity bias: DPR generally performs better on entities observed during NQ training than on randomly selected entities.This indicates stronger representations for common and training-observed entities.
- 4.2 Observing questions helps generalization: Observing a question pattern during training allows DPR to generalize well to unseen entities across all three relations.Training on a syntactically different but semantically equivalent pattern achieves comparable performance to training on the exact evaluation pattern.
- 4.2 Observing questions helps generalization: Fine-tuning spreads positive passage representations more sparsely, making relevant passages easier to discriminate using an inner product.Before fine-tuning, positive passages for place-of-birth questions form a cluster; after fine-tuning, they are more sparsely distributed.
5 Towards Robust Dense Retrieval
The paper tests data augmentation and specialized question encoders as remedies for poor generalization. Single-relation fine-tuning trades off performance across distributions, while a robust PAQ-trained passage encoder nearly closes the gap with BM25 using one fixed index.
- Data augmentation: Fine-tuning on one EntityQuestions relation improves that relation but degrades NQ performance and remains substantially behind BM25 on average.
- Data augmentation: Multi-task fine-tuning on relation questions and NQ retains most NQ performance, but produces much smaller gains on EntityQuestions.The result exposes a trade-off between accuracy on the original distribution and improvement on new questions.
- Data augmentation: Fine-tuning one type of entity-centric question does not necessarily solve generalization to other relations.
- Specialized question encoders: A fixed passage index requires a robust passage encoder because the index cannot be adapted through fine-tuning for different question patterns.The proposed alternative adapts specialized question encoders while keeping the passage index fixed.
- Specialized question encoders: Fine-tuning question encoders over a PAQ-trained passage encoder improves performance over an NQ-trained passage encoder and nearly closes the gap with BM25.PAQ samples 10M passages and identifies plausible answer spans with automatic tools; the passage encoder remains fixed across question patterns.
6 Conclusion
The study finds that DPR significantly underperforms BM25 on EntityQuestions and struggles with rare entities unless trained on the question pattern. It suggests entity memory and entity-aware representations as future directions.
- DPR significantly underperforms BM25 on EntityQuestions, which contains simple questions based on facts mined from Wikidata.
- DPR remembers robust representations for common entities but struggles to differentiate rarer entities without training on the question pattern.
- Future work could incorporate entity memory or entity-aware embedding models to help dense retrievers differentiate rare entities.
Ethical Considerations
The dataset and models may amplify harmful biases, particularly because Wikidata triples are used to further train BERT-based models. The authors caution against real-world deployment without studying these implications.
- Training BERT-based models on Wikidata triples has the potential to amplify harmful and toxic biases.
- Deployed open-domain QA systems using biased pre-trained models may be less accurate or biased for questions about stereotyped and marginalized groups.
- The authors caution builders to consider and study these implications before deploying systems in the real world.
A Full Results on EntityQuestions
Across EntityQuestions evaluations, BM25 significantly outperforms DPR and REALM. DPR also retrieves irrelevant passages by confusing similarly named entities or missing entity mentions, while Table 6 reports Top-20 accuracy using passage text alone.
- BM25 significantly outperforms DPR models trained on either NQ alone or a combination of common QA datasets.
- REALM significantly underperforms BM25 on EntityQuestions even with extra pre-training tasks.
- DPR retrieves irrelevant passages by confusing similarly named entities or missing the presence of an entity.
- Table 6 reports Top-20 retrieval accuracy for REALM and BM25 across EntityQuestions subsets, checking answers in passage text while excluding titles.
B Experimental Details
The experiments use specified DPR and BM25 implementations and training settings, while PAQ supplies sampled passages and hard negatives. Table 7 presents representative DPR retrieval examples.
- DPR experiments use released or retrained models, with fine-tuning for 10 epochs at learning rate 2 × 10^-5 and batch size 24.Retraining from scratch uses 20 epochs with batch size 24.
- BM25 experiments use Pyserini’s unigram implementation with default parameters and the Wikipedia passage splits from the official DPR release.
- PAQ sampling selects 100K passages with at least three generated questions, splits them 70K/15K/15K, and mines hard negatives with BM25.The dataset is not evaluated in these experiments.
- Table 7 provides examples of DPR retrieval results on three representative simple entity-centric questions.
C Per-relation Accuracy with Different Passage Encoders
The passage encoder’s pretraining source affects per-relation robustness: a PAQ-based encoder outperforms an NQ-based encoder and can surpass BM25 for many non-person relations.
- C Per-relation Accuracy with Different Passage Encoders: Fine-tuning DPR with the passage encoder fixed compares question encoders built on NQ- and PAQ-trained passage encoders.The evaluation reports per-relation accuracy for both fixed-encoder settings.
- C Per-relation Accuracy with Different Passage Encoders: A PAQ-based passage encoder is much more robust than an NQ-based passage encoder across per-relation EntityQuestions accuracy.
- C Per-relation Accuracy with Different Passage Encoders: For many non-person relations, DPR with a PAQ-based passage encoder outperforms BM25.