Source-linked AI summary
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, Sebastian Riedel
TL;DR
QA-pair models are faster and easier to update than retrieve-and-read systems but lose accuracy because their QA-pairs cover less knowledge. This paper introduces 65M-question PAQ and the RePAQ retriever, which match recent retrieve-and-read accuracy, support efficient configurations, and enable selective back-off. PAQ also improves CBQA training, though CBQA remains behind RePAQ.
Problem
QA-pair models lack retrieve-and-read accuracy because their available QA-pairs cover substantially less knowledge than corpora such as Wikipedia.
Method
The paper constructs PAQ, a 65M-pair natural-language knowledge base, and develops RePAQ to retrieve and optionally rerank similar questions from it.
Results
RePAQ reaches the accuracy of recent large-scale retrieve-and-read systems while supporting configurations under 500MB or above 1,000 questions per second; PAQ-trained CBQA improves by 5% but trails RePAQ by over 15%.
Takeaways & Limitations
RePAQ provides a flexible QA system combining high accuracy with speed, memory efficiency, and selective back-off to more expensive models.
Takeaways & Limitations
PAQ generation is computationally intensive, and its coverage needs further expansion; CBQA models trained on PAQ retain a large accuracy gap relative to RePAQ.
Abstract
from arXiv · showhide
Open-domain Question Answering models which directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared to conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models lack the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically-generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) whilst retaining high accuracy. Lastly, we demonstrate RePAQ's strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to ``back-off" to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.
1 Introduction
QA-pair models offer speed, interpretability, and easy updates but lag retrieve-and-read systems because their QA-pairs cover less knowledge. PAQ and RePAQ address this gap through large-scale QA-pair generation and retrieval, improving accuracy and enabling selective answering.
- QA-pair retrievers provide fast inference, interpretable outputs, and test-time knowledge updates by retrieving training QA-pairs instead of memorizing them.
- Existing CBQA and QA-pair retrievers trail retrieve-and-read systems because their training QA-pairs cover substantially less knowledge than Wikipedia-like corpora.
- PAQ is a 65M-question natural-language knowledge base designed for models to memorize or retrieve from, with globally filtered generated questions.
- RePAQ uses dense question retrieval and optional reranking, reaching retrieve-and-read-level accuracy while supporting configurations for memory, speed, or accuracy.It can exceed 1,000 questions per second, and memory-optimized versions use 336MB and 29MB.
- 5%: BART models trained on PAQ outperform standard-data baselines, but lag RePAQ by 15%, showing the value of explicit retrieval.
- RePAQ’s confidence correlates with accuracy, enabling abstention and back-off to a slower, more accurate QA system.The combined system is reported as more accurate and 2x faster than the state-of-the-art model alone.
2 Open-Domain Question Answering
Open-domain question answering seeks to answer factoid questions across an open set of domains by learning an answer function over questions and answers. PAQ models the underlying QA distribution with a large generated sample that can overlap test questions and shift computation earlier.
- Open-domain question answering answers natural-language factoid questions drawn from an open set of domains.
- The task defines an answer function from possible questions to answers and evaluates it using empirical QA-pair samples and answer-string loss.
- PAQ implicitly models the QA distribution to generate a large sample for training or retrieval, potentially preempting questions asked at test time.
- Generating and caching likely test questions shifts computation from test time to train time relative to retrieve-and-read methods.
3 Generating Question-Answer Pairs
PAQ generation sequentially selects likely question-worthy passages, extracts candidate answers, generates questions, and globally filters them for consistency. The pipeline combines learned and named-entity extraction alternatives with BART-based question generation.
- PAQ generation begins with a passage-selection model that identifies corpus passages likely to contain information humans will ask about.
- An answer-extraction model identifies spans likely to answer questions, using either named-entity recognition or a trained BERT span extractor.
- A BART-base question generator produces likely questions from each passage and selected answer, with training data from Natural Questions, TriviaQA, and SQuAD.
- The generation pipeline applies passage selection, answer extraction, question generation, and filtering sequentially before adding consistent QA-pairs to PAQ.
- The trained extractor directly models joint answer-span probabilities for spans up to length 30, rather than assuming independent start and end probabilities.
- Global filtering supplies generated questions without their source passage to an open-domain QA model, removing questions whose predicted answers do not match the source answers.This addresses ambiguity that local passage-based filtering cannot remove and is described as vital for strong downstream results.
4 Question Answering using PAQ
PAQ is used both to expand training data for closed-book QA and as a knowledge base for explicit QA-pair retrieval. RePAQ learns to retrieve relevant questions, optionally reranks them, and returns stored answers with interpretable matching.
- Using PAQ: PAQ supports two uses: supplying training QA-pairs for CBQA models and serving as a knowledge base for direct retrieval.These uses are related because CBQA models can implicitly retrieve memorized training data at test time.
- RePAQ: RePAQ retrieves the most relevant QA-pair from a knowledge base and returns its answer to the test question.The model consists of a neural dense retriever and may include a neural reranker.
- RePAQ Retriever: RePAQ embeds questions and indexed QA-pairs in a shared representation space, maximizing inner product for relevant items.Using question embeddings for both sides creates a symmetric retriever and aids training stability.
- RePAQ Retriever: Training retrieves top-K QA-pairs and uses a sequence-to-sequence generator to produce answers before marginalizing over retrieved pairs.Token-level generation can credit helpful pairs even when their stored answers differ in granularity from the target.
- Inference and reranking: At test time, RePAQ embeds and indexes a QA knowledge base, retrieves the most similar stored question, and returns its answer for display.FAISS-based MIPS enables answering hundreds to thousands of questions per second; the reranker improves accuracy but is distinct from the slower generator-based alternative.
5 Results
PAQ provides broad, filtered QA coverage, while RePAQ achieves strong accuracy with favorable size and speed trade-offs. Its confidence scores also support selective answering and faster back-off systems.
- PAQ resource: 65M filtered QA-pairs were produced from 165M answer spans and 279M unique questions using global filtering.The PAQ pipeline retains 24.4% of QA-pairs for PAQL, compared with 18% for PAQNE.
- PAQ resource: Over 90% answer coverage was achieved for both TriviaQA and NaturalQuestions, with higher coverage when question-generation beams contributed more questions.The union of PAQL and PAQNE produced the highest coverage overall.
- Question answering accuracy: 47.7% Exact Match on NaturalQuestions let RePAQ outperform RAG by over 3%, while PAQ-trained BART improved over an NQ-only model by 5%.The PAQ-trained BART result remained 3% behind T5-11B despite T5-11B having 27x more parameters.
- PAQ quality: +0.9% accuracy came from generating four rather than one question per answer span, and combining PAQNE with PAQL added 1.2%.Accuracy did not decrease when the number of globally filtered QA-pairs increased, even with millions already present.
- Efficiency: RePAQ retrievers answered over 1000 questions per second, while the slowest RePAQ remained 3x faster than FiD-base and 12x faster than FiD-large.The slowest RePAQ was only 0.8% less accurate than FiD-base.
- Selective question answering: At 75% answer coverage, RePAQ reached 59% accuracy on answered NaturalQuestions, outperforming FiD by over 10%.A RePAQ-FiD back-off system was 2.1x faster than FiD-large and 1% more accurate, while an alternative threshold achieved 4.6x speedup at FiD’s accuracy.
6 Related Work
The paper situates PAQ among approaches that store or retrieve knowledge for open-domain question answering. It generates globally relevant natural-language QA pairs rather than extracting all facts or relying on contextual passage retrieval.
- KBQA: Structured knowledge bases can offer high precision but are limited by schema constraints and mismatches with user requirements.
- Open Information Extraction: Open information extraction typically mines free text into structured triples, whereas this work generates natural-language QA pairs focused on likely questions.
- Real-time ODQA: Real-time QA systems prioritize fast runtimes, while PAQ indexes globally answerable questions for fast answering.
- Question Generation for QA: Prior question-generation work includes data augmentation, retrieval improvement, contextual QA, and generative modeling.
- Question Generation for QA: Unlike OceanQA, this work builds a large knowledge base of non-contextual, globally consistent ODQA questions and studies the QA systems it enables.
7 Conclusion
The paper concludes that PAQ is a large reusable resource for improving open-domain QA, particularly through RePAQ retrieval. Its scale brings computational costs and leaves coverage and CBQA accuracy as areas for improvement.
- 65M QA-pairs in PAQ support improvements to open-domain QA models.
- RePAQ demonstrates benefits in accuracy, speed, space efficiency, and selective QA by retrieving from PAQ.
- PAQ generation is computationally intensive, motivating future work to improve generation efficiency and expand coverage.
- CBQA models trained with PAQ remain separated from RePAQ by a large accuracy gap.
- Future work could use PAQ for data augmentation or to improve retrieve-and-read and other systems, although these applications are outside this paper’s scope.
A.1 Dataset splits
The experiments use standard open-domain splits for Natural Questions and open-domain train-test splits for TriviaQA.
- Natural Questions uses 79,168 training, 8,757 development, and 3,610 test question-answer pairs.
- TriviaQA uses open-domain splits corresponding to the unfiltered-train and unfiltered-dev reading-comprehension splits.
A.2 Futher details on Passage selection
Passage selection uses a RoBERTaBASE classifier to score every Wikipedia passage and construct a candidate pool for answer generation.
- A RoBERTaBASE model with an MLP over the [CLS] representation predicts whether each passage is positive or negative.
- Inference scores every passage in Wikipedia, and the top N passages form the candidate pool for answer generation.
- The passage-selection model is optimized for higher recall.
A.3 Further Details on Question Quality
Retrieved PAQ questions usually align with evaluation questions or preserve their answers, but a minority contain semantic or question-type errors. Human evaluation found that most generated questions capture relevant answer context.
- 82% of sampled questions accurately captured the answer’s passage context and included relevant details for locating it.
- 16% of sampled questions failed to understand certain answer semantics, often by confusing similar entities or generalising rare phrases.
- 2% of questions mismatched Wh-type words with answers, mainly when answers were rarely asked words.
A.4 Further details on System Size vs Accuracy
The supplementary experiments characterize RePAQ’s storage assumptions and selective-answering behavior against FiD. RePAQ remains better calibrated and outperforms FiD below 50% answer coverage on TriviaQA, while its calibration scores can also improve FiD-large.
- The system-size experiment assumes fp16 models, LZMA-compressed text, and 768-dimensional product-quantized dense indexes.
- Inference benchmarking used mixed precision, parallel question answering, three repeated measurements, and an HNSW index over all 65M PAQ pairs.
- RePAQ remains better calibrated than FiD and outperforms it for answer coverages below 50% on TriviaQA.
- Using RePAQ’s answer confidence scores to calibrate FiD produces the best FiD results among the compared calibration approaches.
- RePAQ training ran for up to 3 days on eight 32GB V100 GPUs, while PAQ BART CBQA training ran for 6 days on eight V100s.