Source-linked AI summary
Distilling Knowledge from Reader to Retriever for Question Answering
Gautier Izacard, Edouard Grave
TL;DR
Retriever training for downstream question answering lacks annotated query–document pairs, and heuristic supervision is limited. The paper distills reader cross-attention into synthetic retriever targets, yielding state-of-the-art results on competitive question-answering benchmarks. It also shows that attention-based relevance depends on the reader's assumptions about passage importance.
Problem
Training retrievers requires query–document annotations, but such labels are time-consuming to create, absent from many datasets, and heuristics are limited for some tasks.
Method
The method uses reader cross-attention scores as synthetic targets and iteratively trains a dense retriever to reproduce reader-derived passage relevance.
Results
The method obtains state-of-the-art performance on competitive question-answering benchmarks and improves EM by +4.5 on TriviaQA and +2.3 on NaturalQuestions when initialized from DPR passages.
Takeaways & Limitations
Reader-derived attention can provide useful supervision for training retrievers without annotated query–document pairs.
Takeaways & Limitations
The relevance interpretation assumes that greater reader attention to passage tokens indicates greater passage relevance.
Abstract
from arXiv · showhide
The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.
1 INTRODUCTION
The paper addresses the challenge of training retrievers without annotated query–document pairs by distilling relevance information from a reader. It proposes using reader attention as synthetic supervision and evaluates the approach on question answering.
- Training retrievers requires query–document pairs, but hand-labeling them is time-consuming and many datasets lack these annotations.Heuristic alternatives can produce false positives and may not apply directly to fact checking or long-form question answering.
- The proposed procedure uses a retriever to select documents and a reader to solve the task, then uses reader attention activations as synthetic retriever labels.The approach is inspired by knowledge distillation and assumes attention activations proxy document relevance.
- The method trains the retriever to reproduce the document ranking induced by the reader-derived relevance metric.
- The paper identifies attention-based relevance measurement, iterative retriever training, alternative loss functions, and evaluation on three question-answering benchmarks as contributions.The reported benchmark evaluation obtains state-of-the-art results.
2 RELATED WORK
Prior work progresses from sparse vector-space retrieval to neural ranking and end-to-end dense retrieval, while recent unsupervised methods reduce reliance on annotations. The paper differs from concurrent distillation work in the synthetic labels used.
- Traditional retrieval represents queries and documents with sparse term vectors using weighting schemes such as inverse document frequency and BM25.Sparse matching requires query terms to match terms in returned documents.
- Neural retrieval uses learned representations, including bi-encoders and cross-encoders, but independently encoded queries and documents miss fine-grained interactions.
- DPR applies BERT-based bi-encoder embeddings and nearest-neighbor search to open-domain question answering.
- Unsupervised approaches pre-train or augment retrievers using objectives such as inverse cloze prediction, masked language modeling, or sequence-to-sequence generation.
- Concurrent distillation work transfers signals from a DPR cross-encoder reader, whereas this paper uses sequence-to-sequence reader attention as synthetic labels.
3 METHODOLOGY
The method combines a retriever and reader in a student–teacher pipeline: reader cross-attention scores provide passage-relevance targets, and a dense bi-encoder learns to retrieve accordingly. Iterative retraining repeatedly improves the retriever using newly retrieved passages.
- 3.1 CROSS-ATTENTION MECHANISM: The standard pipeline first retrieves support passages and then processes them with a reader to generate an answer.The reader uses a Fusion-in-Decoder sequence-to-sequence architecture.
- 3.1 CROSS-ATTENTION MECHANISM: Cross-attention computes queries, keys, and values from decoder and encoder representations, then combines values using attention probabilities.These operations are performed across multiple attention heads with normalization and skip connections.
- 3.2 CROSS-ATTENTION SCORE AS A RELEVANCE MEASURE FOR PASSAGE RETRIEVAL: The method treats decoder attention over passage tokens as a proxy for passage relevance, with greater attention indicating greater relevance to answering the question.Passage scores are aggregated across tokens, layers, and decoder heads.
- 3.2 CROSS-ATTENTION SCORE AS A RELEVANCE MEASURE FOR PASSAGE RETRIEVAL: Selecting the top 10 passages by reader attention reduced EM from 48.2 to 46.8, compared with 42.9 using the top 10 DPR passages.The experiment uses 100 passages initially retrieved with DPR.
- 3.3 DENSE BI-ENCODER FOR PASSAGE RETRIEVAL: A dense bi-encoder makes passage indexing practical by representing questions and passages as vectors and retrieving passages through efficient similarity search.The implementation uses BERT [CLS] representations with shared question and passage encoding parameters.
- 3.4 DISTILLING THE CROSS-ATTENTION SCORE TO A BI-ENCODER: Retriever training minimizes KL divergence between normalized retriever scores and reader-derived relevance scores, with regression and max-margin objectives also considered.The max-margin objective penalizes ranking inversions by enforcing a margin γ.
- 3.5 ITERATIVE TRAINING: Iterative training alternates reader training, attention-based target construction, retriever training, and retrieval with the new retriever.Repeated iterations use the current retriever to sample negative examples; initial document quality affects final performance.
4 EXPERIMENTS
The experiments evaluate iterative reader-guided retriever training across three question-answering benchmarks, comparing initialization choices, retrieval metrics, and end-to-end performance. The method improves over baselines, but results depend strongly on the quality of the initial document sets.
- Datasets: The experiments cover TriviaQA, NaturalQuestions, and NarrativeQA, using Wikipedia or full-story passages as support-document collections.TriviaQA lacks gold support documents, NaturalQuestions includes Wikipedia support documents, and NarrativeQA uses long books and movie scripts without their summaries.
- Evaluation: R@k measures whether at least one top-k passage contains the gold answer, while end-to-end evaluation measures the combined retriever-reader system.The study reports exact match for TriviaQA and NaturalQuestions, and ROUGE-L, BLEU-1, BLEU-4, and METEOR for NarrativeQA.
- Initialization: DPR initialization outperforms BM25 and untuned BERT initialization, with training using 100 passages and the reader using 20 passages for NarrativeQA.The reader uses 100 passages for NaturalQuestions and TriviaQA; all retrievers are trained with 100 passages.
- Iterative training: Accuracy generally increases after a few self-training iterations, while higher-quality initial document sets produce stronger final systems.BM25 starts with better-quality documents than BERT, and therefore yields stronger results despite the iterative procedure.
- Results: +4.5 EM on TriviaQA and +2.3 EM on NaturalQuestions result from starting with DPR passages and evaluating with a large reader.The method also improves over the state of the art when initialized with BM25 documents, supporting document training without document-level supervision.
- Results: Attention-guided retriever training improves NarrativeQA performance compared with the BM25 baseline in the full-document setting.NarrativeQA uses the whole document as the knowledge source and does not use the summary.
5 ABLATIONS
The ablations examine training objectives and cross-attention aggregation on NaturalQuestions after one iteration. KL-divergence performs best among the tested objectives, while aggregation choices produce relatively stable results.
- Training objectives: KL-divergence between aggregated reader scores and retriever scores outperforms the other training objectives.Table 5 reports this comparison on NaturalQuestions after one iteration using validation metrics.
- Cross-attention aggregation: The aggregation scheme combines cross-attention scores across output tokens, input tokens, layers, and heads to produce one passage score.The resulting scalar Gq,p is used to train the retriever.
- Cross-attention aggregation: Performance is relatively stable across the tested aggregation schemes.The alternatives change whether scores are averaged or maximized over input tokens, output tokens, layers, and heads.
- Cross-attention aggregation: Averaging is best across most aggregation dimensions, except that using only the first output token performs best for output-token aggregation.The reported best configuration averages over input tokens, layers, and heads while selecting the first output token.
6 CONCLUSION
The paper introduces a knowledge-distillation method that trains a retriever without query-document annotations by using reader cross-attention scores as synthetic targets. Iterative reader-retriever training improves performance and reaches state-of-the-art results on competitive question-answering benchmarks.
- 6 CONCLUSION: The method trains an information retrieval module for downstream tasks without annotated query-document pairs.The retriever is treated as the student and the reader as the teacher.
- 6 CONCLUSION: Reader cross-attention scores provide synthetic targets for training the retriever.The paper compares alternative score aggregations and training objectives.
- 6 CONCLUSION: Iteratively training the reader and retriever leads to better performance and state-of-the-art results on competitive question-answering benchmarks.The conclusion reports this outcome without restricting it to a single benchmark.
A.1 SETTING
The NaturalQuestions and TriviaQA settings use standard open-domain question answering procedures, while answer text is normalized for dataset-specific capitalization differences.
- A.1 SETTING: For NaturalQuestions and TriviaQA, the original development set is used for testing and 10% of the training set for development.NaturalQuestions questions with answers longer than five tokens are discarded.
- A.1 SETTING: TriviaQA trains the reader with its unique human-generated answer, after normalizing uppercase answers using title case.The normalization capitalizes the first letter of each word and lowercases the remaining characters.
- A.1 SETTING: For NarrativeQA, questions and answers written in uppercase are converted to lowercase.
A.2 TRAINING
Training fine-tunes both reader and retriever models with dropout and dataset-specific optimization schedules, while ranking similarity monitors retriever training. Inputs are capped at 250 tokens and formatted with question, title, and context fields where applicable.
- A.2 TRAINING: Both reader and retriever models use a 10% dropout rate during fine-tuning.Most models use ADAM with learning rates of 10^-4 for the base reader and 5·10^-5 for the retriever; the large reader uses ADAMW.
- A.2 TRAINING: The base reader trains for 10k gradient steps with batch size 64, while the retriever trains until performance saturates.Model selection uses validation performance, and ranking similarity monitors retriever training.
- A.2 TRAINING: Each encoder input is restricted to at most 250 tokens.For NaturalQuestions and TriviaQA, inputs concatenate the question, Wikipedia title, and support passage with special field tokens.
A.3 INFERENCE
Inference uses different decoding strategies across the evaluated question-answering datasets. Table 8 examines iterative training from DPR-retrieved documents using validation-set metrics without reinitializing the reader between iterations.
- Greedy decoding is used for TriviaQA and NaturalQuestions.
- NarrativeQA uses beam search with 3 beams at test time.
- Table 8 evaluates iterative training starting from documents retrieved with DPR.
- Iteration 0 measures the reader trained on the initial support documents.
- Table 8 reports validation-set metrics without reinitializing the reader between iterations.