Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

1 Introduction

• Retrieval-based methods expose models to related knowledge, but indexing large corpora and retrieving from them can impose substantial computational costs.The introduction notes kNN-MT generation is two orders of magnitude slower than traditional machine translation.Source: §1 · p. 1
• REINA indexes labeled training input-output pairs, retrieves those most similar to each input during training and testing, and concatenates them with that input.During training, it excludes the current instance from retrieval to avoid data leakage.Source: §1 · p. 1
• Unlike few-shot prompt learning and non-parametric instance-based methods, REINA fine-tunes model parameters on supervised examples.The authors describe this as a critical distinction from few-shot and non-parametric approaches.Source: §1 · p. 1
• REINA improves performance across 11 datasets in summarization, language modeling, machine translation, and question answering, with state-of-the-art results on XSum, BigPatent, and CommonsenseQA.The introduction also reports that REINA with BART-base rivals BART-large on three summarization tasks.Source: §1 · p. 2
• The authors argue that retrieved training examples can remind models of patterns they may not memorize, allowing decent-size models to approach or match megasized-model performance at lower computational cost.This implication is drawn from the summarization results and is framed as a potential alternative to continually increasing model size.Source: §1 · p. 2

2 Related Work

• Retrieval research includes sparse bag-of-words and neural dense representations, with dense retrieval reported to improve recall while bringing document-length and parallel-data constraints.Sparse approaches are described as readily adaptable to general large-scale search.Source: §2 · p. 2
• Sparse bag-of-words retrieval commonly ranks with rules such as BM25 and has been used in open-domain question answering and machine translation.The paper contrasts this general large-scale search capability with dense representations.Source: §2 · p. 2
• Dense retrieval uses task-trained neural encoders and can achieve better recall than sparse retrieval, but DPR typically handles documents under 128 tokens and requires task-specific parallel training data.The paper cites applications including open-domain QA, knowledge-grounded generation, and machine translation.Source: §2 · p. 2
• REINA uses BM25 to retrieve from training data, requiring no parallel data for task-specific retriever training and applying across summarization, QA, machine translation, and language modeling.The authors distinguish it from non-parametric systems guided by search engines and from methods relying on additional data augmentation.Source: §2 · p. 2
• Prompt learning selects labeled examples related to an input and prepends them for few-shot inference, whereas REINA includes similar labeled training data across training, validation, and test in a supervised setting.The paper mentions continuous prompt optimization as another prompt-engineering approach.Source: §2 · p. 2

3.1 Retrieval-based Methods

• BM25 retrieves the top K key-value pairs most similar to an input, and a combination function joins retrieved results with the input before the NLP model generates an output.The paper chooses BM25 over dense representations mainly for faster speed; concatenation is one example of a combination function.Source: §3.1 · p. 3

3.2 Retrieval from Training Data ( REINA )

• REINA indexes labeled training data as input-label pairs and retrieves the top K similar instances to combine with the input during both training and inference.During training, the input itself is filtered from retrieval results to avoid data leakage.Source: §3.2 · pp. 3–4
• Task-specific formats use document-summary pairs for summarization, preceding next-chunk text for language modeling, and target-language translations for machine translation.For summarization, only retrieved summaries are concatenated with the input; for language modeling, retrieved next chunks precede it; for translation, retrieved target text is concatenated.Source: §3.2 · p. 4
• For multiple-choice question answering, retrieval uses each question-choice pair as a query and combines retrieved question-answer examples with that pair.A separate version adds ConceptNet relations and Wiktionary definitions to the retrieval query and index; the model scores each choice.Source: §3.2 · p. 4

3.3 Model Training and Inference

• Training and inference use Seq2Seq models for generation and encoder-only models for question answering; training filters retrieved exact gold labels, while inference does not.Table 1 reports training, development, and test-set sizes; language-model counts are chunked sequences of 64 words for WikiText2 and 128 words for WikiText103.Source: §3.3 · pp. 4–5, Table 1 · p. 5

4.1 Dataset

• The evaluation spans summarization, language modeling, machine translation, and question answering, including five summarization datasets and both low- and rich-resource translation settings.The datasets include single- and multi-document summaries, Wikipedia text, and commonsense or physical-reasoning questions.Source: §4.1 · p. 5

4.2 REINA Details

• REINA retrieves task-relevant training examples: it selects documents for summarization, retrieves chunks for language modeling, and indexes question-answer pairs with or without external knowledge.For language modeling, retrieval occurs once per chunk; for multi-choice QA, retrieved information is encoded as a prompt or additional knowledge.Source: §4.2 · pp. 5–6

4.4 Experiment Results

• REINA significantly boosts summarization baselines across five datasets, reaches state-of-the-art on XSum and BigPatent, and lets BART-base beat larger models on BigPatent and WikiHow.The comparisons include PEGASUS, BART-base, and BART-large initializations.Source: Table 2 · p. 6, §4.4 · p. 7
• On XSum, adding an auxiliary source to REINA's retrieval index improves Rouge scores by 1%, whereas joint training on merged data yields no gains.REINA is trained on XSum pairs while its index can include the additional source.Source: Table 3 · p. 6, §4.4 · pp. 7–8

4.3 Optimization Details

• Task configurations vary: summarization appends five retrieved summaries, language modeling prepends retrieved text, translation uses mBART, and QA uses 1.5B-parameter DeBERTa.For commonsense-reasoning datasets, REINA also uses ConceptNet and Wiktionary.Source: §4.3 · pp. 6–7

4.4 Experiment Results

• REINA significantly improves DeBERTa baselines on all three question-answering datasets, reaching state-of-the-art and leaderboard No.1 on CommonsenseQA.Retrieval without external knowledge works best on CommonsenseQA, while adding knowledge helps on aNLI and PIQA.Source: Table 4 · p. 7, §4.4 · p. 8
• REINA boosts mBART in English-to-Turkish, Turkish-to-English, and English-to-German translation; language modeling improves on WikiText103 but not WikiText2.The paper suggests WikiText2's smaller size may make related text harder to retrieve and notes that Seq2Seq is a strong baseline.Source: Table 6 · p. 7, Table 5 · p. 7, §4.4 · p. 8

4.5 Further Analysis

• The case study presents retrieved training examples for XSum and CommonsenseQA, with the authors reporting strong correlation between retrieved data and golden labels.The examples include a summary sharing the security-concerns point and a retrieved QA pair related to the warm-room answer.Source: Table 7 · p. 8, §4.5 · p. 8

• REINA improves baseline performance on 11 datasets covering summarization, language modeling, machine translation, and question answering.Source: §5 · p. 9
• The method is general and can be integrated into different models across different tasks.Source: §5 · p. 9
• REINA is presented as a simple method for making fuller use of the training dataset.Source: §5 · p. 9