Re3val: Reinforced and Reranked Generative Retrieval

1 Introduction

• Re3val is trained with 500k pre-training examples and 48k task-specific examples despite using reduced data for distant supervision.The paper presents the framework as a response to generative retrieval’s lack of contextual information and the non-differentiability of page-title decoding.Source: §1 · p. 1
• Contextual generative reranking outperforms other generative retrieval models, including GENRE, CorpusBrain, and SEAL, by an average 1.9% in R-Precision across five tasks.The reranker uses DPR contexts to reduce uncertainty in initially retrieved page titles.Source: §1 · pp. 1–2
• With question generation, REINFORCE improves average R-Precision by 8% over CorpusBrain zero-shot retrieval across five tasks.REINFORCE integrates information during generative retrieval decoding.Source: §1 · p. 2
• Re3val’s retrieve-and-read pipeline achieves the highest KILT scores among generative retrieval models, with an average increase of 2.1%.The pipeline retrieves contexts for reranked page titles, applies a context reranker, and grounds answers in the reranked contexts.Source: §1 · p. 2

2.1 Document Retrieval

• DPR uses a bi-encoder to create dense embeddings of questions and related corpus passages, compares them by dot product, and at inference retrieves top-k relevant contexts using nearest-neighbor or maximum inner-product search on FAISS.FiD combines independently encoded queries and retrieved passages to decode an answer, but does not rerank retrieved documents.Source: §2.1 · p. 2

2.2 Generative Retrieval

• Generative retrieval methods encode corpus knowledge or generate page titles, but the paper says they leave contextual information and knowledge during decoding underused.Examples include GENRE (Cao et al., 2021), CorpusBrain (Chen et al., 2022), DEARDR, SEAL (Bevilacqua et al., 2022), DSI (Tay et al., 2022), and NCI (Wang et al., 2022).Source: §2.2 · p. 2

2.3 Question Generation

• Prior question-generation work includes GAR, which uses BM-25, and a method that reranks contexts by likelihood over generated questions; the paper identifies epistemic uncertainty and pre-training–fine-tuning domain gap as further concerns.The cited examples are GAR (Mao et al., 2021) and Sachan et al. (2022).Source: §2.3 · pp. 2–3

2.4 Reranking Models

• Existing reranking approaches rerank documents or candidate lists, but the paper says they do not rerank page titles using a rerank query.Examples include Atlas (Izacard et al., 2022b), Re2G (Glass et al., 2022), and FiD-Light (Hofstatter et al., 2022).Source: §2.4 · p. 3

2.5 Reinforcement Learning

• The paper frames text generation as reinforcement learning with rewards that can include non-differentiable evaluation metrics, and uses REINFORCE.The related-work passage also names A2C (Mnih et al., 2016) and PPO (Schulman et al., 2017).Source: §2.5 · p. 3

3 Methodology

• Re3val’s methodology combines contextual generative page-title reranking, REINFORCE during distant supervision, question generation for pre-training, and reading contexts retrieved from generatively obtained titles.The training pipeline includes question generation, few-shot training, reranker training, KILT context retrieval, and FiD fine-tuning.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Page Title Retrieval (Stage 1-4)

• Generated questions are filtered and added to pre-training to mitigate domain shift, while REINFORCE optimizes zero- and few-shot retrieval using page-title R-Precision as its reward.Few-shot training adds labeled data to narrow target candidates.Source: §3.1 · p. 3

3.2 Page Title Reranker (Stage 5-7)

• The page-title reranker uses DPR contexts as a reranking query and trains with mixed zero- and few-shot titles whose top-half ordering is shuffled to introduce noise.The method uses an auxiliary index because KILT datasets do not provide a specific reranking query; the authors say input noise helps filter inconsistencies and misleading patterns.Source: §3.2 · p. 4

3.3 Context Retrieval (Stage 10-11)

• After reranking, Re3val retrieves contexts for the top five page titles from KILT and uses BM-25 to impute titles unavailable in that database.Contexts are divided into 100-word chunks and certain title-only or patterned sentences are filtered.Source: §3.3 · pp. 4–5

3.4 Context Reranker (Stage 8-11)

• A cross-encoder reranks contexts from the five titles using query-context relevance, trained with gold passages and two types of BM25-retrieved hard negatives.The negatives are top-128 unlabeled chunks mapped to labeled titles and top-128 chunks mapped to unlabeled titles retrieved by the page-title reranker.Source: §3.4 · p. 5

3.5 Reader (Stage 12-14)

• The FiD reader is pre-trained on gold passages, with DPR contexts imputed when fewer than five gold contexts are available, then fine-tuned on reranked contexts.The described fine-tuning uses the top five or ten contexts retrieved by the context reranker.Source: §3.5 · p. 5

4.1 Datasets

• The experiments use Natural Questions, TriviaQA, and HotpotQA for question answering, FEVER for fact checking, and WoW for dialogue.These datasets are from KILT (Petroni et al., 2021).Source: §4.1 · p. 5

4.2 Evaluation

• KILT evaluates page-level retrieval with R-Precision and Recall@k, while its KILT score uses post-processed task scores only when a query’s R-Precision is 1.Downstream task metrics include EM and F1 for QA, ROUGE-L and F1 for dialogue, and Accuracy for fact verification.Source: §4.2 · p. 5

4.3 Page Title Retrieval

• Retrieval training uses Wikipedia pre-training data, generated questions, and 48k task-specific examples; evaluation selects the top five generated page titles per query.The passage also reports multi-beam search evaluation on development and test sets.Source: §4.3 · pp. 5–6

4.4 Page Title Reranker

• The page-title reranker is initialized either from plain T5 models or reinforced few-shot checkpoints, with query and combined input lengths constrained.Queries are limited to 250 words, and the input of query, ten titles, and five contexts to 512 tokens.Source: §4.4 · p. 6

4.5 Context Reranker

• Context-reranker inputs use the first 150 query words for QA and fact verification, but the last 300 words for dialogue.The input sequence length is also bounded by task-specific maximums.Source: §4.5 · p. 6

4.6 Reader

• FiD pre-training uses gold passages alone or gold passages plus top-ranked DPR contexts, with different query truncation for WoW and other datasets.The cited checkpoints have 770 million parameters; WoW retains the last 385 query words, while other datasets use the first 125.Source: §4.6 · p. 6

5.1 Page Title Retrieval

• CorpusBrain has 8% lower average R-Precision than Re3val in zero-shot retrieval despite using more than 500 times more data.The authors hypothesize that question generation mitigates epistemic uncertainty and minimizes the domain shift between pre-training and task-specific fine-tuning data.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Page Title Reranker

• The page-title reranker fine-tuned from reinforced few-shot retrieval outperforms a T5-pretrained reranker at small sizes, but the trend reverses as model size grows.The authors attribute the small-model advantage to ranking knowledge compensating for limited capacity, and the larger-model reversal to prior ranking knowledge interfering with reranking.Source: §5.2 · p. 7

5.3 Context Reranker

• The Context Reranker has higher precision than recall, prioritizing relevant documents at the top while potentially missing some; its precision–recall trade-off for downstream reading needs further study.Its performance was evaluated using gold and hard-negative passages.Source: §5.3 · pp. 7–8

5.4 Reader

• With five passages, Re3val achieves superior results to SEAL, which uses 100 contexts for grounding with FiD.The final KILT scores are reported in Table 2.Source: Table 2 · p. 8, §5.4 · p. 8

• Question generation addresses epistemic uncertainty and domain shift, while REINFORCE is applied to constrained beam-search outputs; the data-efficient approaches reduce training time and labeling costs.Source: §6 · p. 8
• With a context reranker before grounding, Re3val achieves top-1 KILT scores among generative retrieval models, with an average 2.1% improvement across five datasets.Source: §6 · p. 8
• Re3val improves average R-Precision by 8% over the CorpusBrain zero-shot baseline and by 1.9% over other generative models across five tasks.The 8% comparison uses reduced pre-training data; the 1.9% gain is attributed to page-title reranking with limited task-specific data.Source: §6 · p. 8

Limitations

• A comprehensive comparison of REINFORCE with PPO and TRPO was infeasible given the project’s time and resource limitations.The authors note that those algorithms require more memory for their reference models.Source: Limitations · p. 8
• Disparities between development and test performance for both retrieval and reader components require further investigation.Source: Limitations · p. 9
• Some labeled FEVER page titles are absent from the KILT database, creating a dataset discrepancy to account for.Source: Limitations · p. 9

Ethics Statement

• The study uses NQ, TriviaQA, HotpotQA, FEVER, and WoW datasets derived from KILT’s August 2019 Wikipedia dump, with June 2017 Wikipedia data also used for pre-training.Source: Ethics Statement · p. 9
• Dataset errors and language-model training or preprocessing may contribute to biased, toxic, or fabricated outputs, so the authors urge caution and suitable real-world policies.Source: Ethics Statement · p. 9

A.1 Hyperparameters

• Limited academic hardware resources led the authors to use different GPUs, and FiD models trained with ten passages used half the specified batch size.Source: §A.1 · p. 11

A.2 Data

• Re3val pre-training combines Wikipedia dumps and DPR-retrieved samples, with generated questions added for several tasks; HotpotQA uses a distinct context-generation mix and Hotpot contexts for title reranking.The prefix tree is built and searched using the KILT knowledge source.Source: §A.2 · p. 11, §A.3 · p. 11

A.3 Prefix Tree

• The KILT knowledge source provides the basis for building and searching the prefix tree used across tasks.Source: §A.3 · p. 11

A.4 Constrained Decoding

• Unlike GENRE’s one-entity-per-beam decoding, Re3val decodes a list of page titles per beam to capture variability among related entities.Source: §A.4 · p. 11

A.5 REINFORCE

• The appendix presents a formal proof of the optimization achieved by applying REINFORCE to the retrieval system.Source: §A.5 · p. 12

A.5.1 Notation

• The notation defines T as sequence length, R(τ) as cumulative trajectory reward, and π with parameter θ as the policy.A trajectory τ is a sequence of actions and states.Source: §A.5.1 · p. 12

A.5.2 Proof

• The REINFORCE objective uses constrained-decoding prefix-tree R-Precision as a scalar reward from the nondifferentiable retrieval component to guide the policy gradient.Source: §A.5.2 · p. 12
• The policy assigns action probabilities using a softmax applied to cross-entropy loss, with more updates directed toward higher-reward samples when the compared rewards are positive.Source: §A.5.2 · p. 12

A.6.1 Missing Page Imputation

• BM25 imputes the best-matching KILT page title for missing retrieved titles, which the authors attribute to labeler mistakes and characterize as MNAR; imputation improves test reranker performance.Source: §A.6.1 · p. 12

A.6.2 Missing Context Imputation

• Relevant contexts can remain unlabeled due to labeler bias, and Table 2 shows a notable performance improvement when imputation addresses sparse DPR contexts.Source: §A.6.2 · p. 12

A.7 KILT Leaderboard

• Reported values prioritize original-paper results in Tables 1 and 2, using KILT leaderboard values when originals are unavailable; model-version differences may cause slight variation.Source: §A.7 · p. 13

A.8.1 Page Title Retrieval

• R-Precision is the proportion of retrieved documents judged relevant among all retrieved documents.The definition frames retrieval quality at the page level.Source: §A.8.1 · p. 13

R. Similarly, Recall@k is calculated as w

• Recall@k measures the proportion of actual relevant documents retrieved within the top k results.The calculation considers successful retrievals among k retrieved documents relative to all actual documents.Source: R. Similarly, Recall@k is calculated… · p. 13

A.8.2 Context Reranker

• The Context Reranker section defines Precision, Recall, F1, and Accuracy using classification outcomes.F1 balances Precision and Recall; Accuracy counts true positives and true negatives among all classifications.Source: §A.8.2 · p. 13

A.8.3 Reader

• KILT scores count reader metrics only when a query has R-Precision of 1.For blind test sets, KILT post-processes outputs by lowercasing, removing articles and punctuation, and normalizing duplicate whitespace; decoding conditions each page on its predecessor.Source: §A.8.3 · p. 13, Figure 4 · p. 13

A.9 Recall Curve of the Page Title Reranker

• Recall curves show the effects of parameter counts and numbers of retrieved documents on page-title retrieval recall.The passage directs readers to Section 5.1 for analysis.Source: §A.9 · p. 13

A.9.1 NQ

• The plots show recall performance at varying levels of documents retrieved for models labeled 60m, 220m, and 770m, plus corresponding two-model variants labeled 2x60m, 2x220m, and 2x770m.Source: §A.9 · p. 13, §A.9.1 · p. 13

A.9.2 TriviaQA

• The plots show how parameter count affects recall as the number of retrieved documents varies, with listed model sizes of 60m, 220m, 770m and variants 2x60m, 2x220m, 2x770m.Source: §A.9 · p. 13, §A.9.2 · p. 13

A.9.5 WoW

• The appendix tables cover best dev-set FiD scores, test-set reader scores, and development performance across training stages.The test reader scores are not final KILT scores, which award reader scores only when R-Precision is 1.Source: Table 10 · p. 16, Table 11 · p. 17, Table 12 · p. 17