PaperlayerView exact revision on arXiv

Source-linked AI summary

RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation

Fangyuan Xu, Weijia Shi, Eunsol Choi

arXiv:2310.04408v1cs.CL

TL;DR

Prepending retrieved documents improves retrieval-augmented LMs but increases inference costs and can leave models struggling to use long contexts. RECOMP compresses retrieved documents into summaries, selectively omitting unhelpful ones, and improves frozen-LM performance across language modeling and three QA datasets while prepending fewer tokens.

  • Problem

    Prepending retrieved documents increases inference costs, and LMs may struggle to use all information in the longer context.

    Source: §1 · p. 1

  • Method

    RECOMP uses extractive and abstractive compressors to create concise query-focused summaries of retrieved documents, with empty summaries enabling selective augmentation.

    Source: §1 · pp. 1–2

  • Results

    RECOMP improves frozen-LM performance across language modeling and three QA datasets with fewer tokens; both trained compressors achieve a 25% compression ratio with minimal drop on language modeling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Selective augmentation avoids prepending summaries when retrieved documents are irrelevant or unhelpful to the target task.

    Source: §1 · p. 1

    Selective compression was not tested with the extractive compressor, which used a fixed number of prepended sentences across each dataset.

    Source: §6 · p. 8

Abstract

from arXiv · show

Retrieving documents and prepending them in-context at inference time improves performance of language model (LMs) on a wide range of tasks. However, these documents, often spanning hundreds of words, make inference substantially more expensive. We propose compressing the retrieved documents into textual summaries prior to in-context integration. This not only reduces the computational costs but also relieves the burden of LMs to identify relevant information in long retrieved documents. We present two compressors -- an extractive compressor which selects useful sentences from retrieved documents and an abstractive compressor which generates summaries by synthesizing information from multiple documents. Both compressors are trained to improve LMs' performance on end tasks when the generated summaries are prepended to the LMs' input, while keeping the summary concise.If the retrieved documents are irrelevant to the input or offer no additional information to LM, our compressor can return an empty string, implementing selective augmentation.We evaluate our approach on language modeling task and open domain question answering task. We achieve a compression rate of as low as 6% with minimal loss in performance for both tasks, significantly outperforming the off-the-shelf summarization models. We show that our compressors trained for one LM can transfer to other LMs on the language modeling task and provide summaries largely faithful to the retrieved documents.

1 INTRODUCTION

RECOMP adds a compression step before in-context augmentation, producing concise summaries of retrieved documents—or an empty summary when the documents are irrelevant or unhelpful.

1 INTRODUCTION

  • Prepending retrieved documents raises encoding costs, can make LMs miss information in long contexts, and can introduce irrelevant information that degrades performance.Source: §1 · p. 1
  • RECOMP compresses retrieved documents into a concise, faithful summary that guides the LM when prepended to its input, or returns an empty summary when retrieval is unhelpful.The approach is illustrated in Figure 1.Source: §1 · p. 1, Figure 1 · p. 2
  • The extractive compressor selects query-relevant sentences and uses contrastive learning; the abstractive compressor synthesizes multiple documents and is distilled from an extreme-scale LM.Both are trained to improve end-task performance when summaries are prepended to the input.Source: §1 · p. 2
  • Oracle compressors reach a 6% compression rate and outperform full-document prepending; trained compressors use 25% for language modeling and 5–10% for QA with less than 10% relative performance drop.RECOMP improves frozen-LM performance on language modeling and Natural Questions, TriviaQA, and HotpotQA while prepending significantly fewer tokens than RALM without compression.Source: §1 · p. 2

2 PROBLEM FORMULATION: RECOMP

RECOMP trains a lightweight compressor to turn retrieved documents into a query-relevant summary for a black-box language model. It defines three summary objectives and describes extractive and abstractive compressor designs.

Problem formulation

  • The compressor maps input x and retrieved documents D to a shorter summary s for a black-box LM M, with retriever improvement outside the work’s focus.The compressor is designed to be substantially smaller than M to reduce the cost of encoding retrieved documents; the formulation assumes a set of retrieved documents is provided.Source: §2 · p. 2

Summary objectives

  • A summary should be concise, effective when prepended to x, and faithful to the retrieved documents; it may be empty when augmentation is unnecessary.The stated training focus is conciseness and effectiveness; faithfulness is defined as the summary being entailed by the input document set.Source: §2 · p. 2

Extractive Compressor

  • The extractive compressor ranks input sentences by their inner product with x-conditioned embeddings and concatenates the top-ranked sentences into s.The dual encoder’s inner product represents each sentence’s estimated helpfulness for generating target y; the approach assumes extractive summaries mostly satisfy faithfulness.Source: §2 · p. 3

Abstractive Compressor

  • The abstractive compressor uses a lightweight encoder-decoder distilled from extreme-scale LMs to summarize x and concatenated retrieved documents D.It is not specifically trained for faithfulness; the paper later evaluates faithfulness manually.Source: §2 · p. 3

3 LEARNING THE COMPRESSORS

The compressors are trained to produce concise text that improves language-model performance on downstream tasks, rather than summaries optimized for human readers. The section describes extractive sentence selection and abstractive summarization, including a mechanism for omitting unhelpful retrievals.

3.1 EXTRACTIVE COMPRESSION

  • The extractive compressor uses a 110M-parameter dual encoder to rank sentences by their usefulness to the language model on the downstream task.It separately encodes the input context and each candidate sentence, then scores them by embedding inner product.Source: §3.1 · p. 3
  • Extractive methods can occasionally fail to preserve faithfulness, but such cases are described as rarer than hallucinations from abstractive approaches [Zhang et al., 2022].For extractive training data, the method uses the top 20 sentences from the top five BM25-retrieved documents for language modeling and top five Contriever-MS MARCO documents for QA, prepending page titles for decontextualization.Source: §3.1 · pp. 3–4
  • Extractive training uses contrastive learning: sentences that improve target likelihood are positive examples, while lower-scoring retrieved sentences provide negatives.For QA, candidate sentences are scored by whether the language model generates the correct answer with the summary prepended.Source: §3.1 · pp. 3–4

3.2.1 CREATING TRAINING DATASET FOR DISTILLATION

  • For language modeling, GPT-3.5 generates a summary of the retrieved document set with respect to the input, using four manually constructed prompts.The encoder-decoder model has 775M parameters and is initialized from T5-large. Training examples use 2% of Wikitext-103 training data; QA examples use all NQ and TriviaQA training data, and HotpotQA examples are limited to cases where the gold answer is in retrieved documents (56% of training data).Source: §3.2.1 · pp. 4–5
  • Abstractive training selects summaries by end-task performance and replaces a summary with an empty string when it hurts performance, enabling selective augmentation.For each example, the best-performing candidate is compared with using the input alone; summaries that worsen performance are omitted to mitigate irrelevant-document risk.Source: §3.2.1 · p. 4, §3.2 · p. 4
  • For QA abstractive training, each dataset uses one prompt, and examples are retained only when prepending the summary improves performance.For HotpotQA, summaries are generated only when the retrieved documents contain the gold answer, covering 56% of its training data.Source: §3.2.1 · p. 5

4 EXPERIMENTAL SETTINGS

The experiments evaluate language modeling and open-domain QA, measuring task performance alongside the number of context tokens. They compare compression methods with heuristic, model-based, and oracle baselines.

4 EXPERIMENTAL SETTINGS

  • Experiments measure task performance for effectiveness and the number of context tokens for efficiency across language modeling and open-domain QA.The evaluations follow prior work (Shi et al., 2023b; Ram et al., 2023).Source: §4 · p. 5

4.1 LANGUAGE MODELING

  • Language modeling uses WikiText-103 perplexity with GPT2, GPT2-XL, and GPT-J, training compressors on GPT2 to test transfer to larger models.BM25 retrieves from a December 20, 2018 Wikipedia corpus of non-overlapping 100-word documents; documents containing the input are excluded, and retrieval occurs every 32 tokens.Source: §4.1 · p. 5

4.2 OPEN-DOMAIN QA

  • Open-domain QA evaluates NQ development, TriviaQA test, and 500 sampled HotpotQA development examples using EM and token-level F1.Experiments use Flan-UL2 (20B) with Contriever retrieval over December 20, 2018 Wikipedia, segmented into non-overlapping 100-word documents; prompts include five sampled training examples before retrieved documents and the question.Source: §4.2 · p. 5

4.3 BASELINES AND ORACLES

  • Baselines include BoW and named-entity lists, sentence-ranking and random extractive methods, and off-the-shelf T5 and GPT-3.5 abstractive summaries.Extractive language-modeling baselines use BM25 and Contriever; QA also uses MS MARCO-finetuned Contriever and NQ-finetuned DPR, while GPT-3.5 uses its best single prompt for language modeling.Source: §4.3 · pp. 5–6
  • Oracle compressors select the extractive sentence or abstractive prompt summary—or empty summary—that yields the best end-task performance for each example.Because extractive oracles depend on the model, the study also tests transfer using sentences selected with GPT-2 as the reference LM.Source: §4.3 · p. 6

5 RESULTS

Across language modeling and open-domain QA, retrieval augmentation improves performance over using no retrieved evidence. Trained compressors reduce the evidence provided, with the best approach and performance trade-off varying by task.

Language modeling

  • All retrieval methods improve perplexity over no retrieval across three LMs, but BoW and NE perform worse than prepending uncompressed documents.The paper suggests disfluency in the prepended text may explain the heuristic methods’ poorer results.Source: §5 · p. 6
  • In language modeling, oracle retrieval gains over prepending the entire document set with only 6–13% of its tokens, and prepending the top one document beats the top five.The results suggest that adding more documents can hurt when retrieved evidence is irrelevant.Source: §5 · p. 6
  • The trained abstractive compressor achieves the lowest perplexity and highest compression across language-modeling results, prepending summaries to only 33% of examples.The trained extractive compressor also outperforms Contriever and BM25 across all three LMs while using slightly fewer tokens; versus prepending one document, it reaches a 25% compression ratio with minimal performance drop.Source: §5 · p. 7

Open-domain QA

  • With open-domain QA, all retrieval methods improve performance over no retrieval across three datasets, and prepending five documents significantly outperforms one.This contrast with language modeling motivates compression as a way to incorporate more documents.Source: §5 · p. 7
  • On QA, the extractive oracle outperforms the abstractive oracle on every dataset, and both improve over prepending all information.The extractive oracle selects the best of N candidate sentences; the abstractive oracle chooses between a GPT-3.5 summary and no added evidence.Source: §5 · p. 7
  • On NQ, top-one DPR-ranked sentences from five documents outperform one full document with 39 versus 132 tokens, but DPR degrades out of domain; T5 gains 4–6 EM points with 7–10 tokens.DPR performs best among extractive baselines, while T5 provides the highest compression; the paper notes that abstractive compression for multihop HotpotQA needs further study because multi-document synthesis is difficult and may hallucinate more often (Shaib et al. (2023)).Source: §5 · pp. 7–8
  • On NQ and TQA, abstractive compression uses 5% of the tokens for losses of 2 and 3.7 EM points; on HotpotQA, extractive compression uses 11% for a 2.4-point loss.Each loss is compared with prepending the full document set.Source: §5 · pp. 7–8

6 ANALYSIS AND DISCUSSIONS

RECOMP’s analyses show that its summaries can transfer across language models with caveats and vary in length to support selective augmentation. They also show that RECOMP reduces incorrect copying from distracting evidence, while answer presence alone does not ensure correct use.

Transferring Across Different LMs

  • Compressors transferred well from GPT2 to GPT2-XL and GPT-J, while QA transfer to LLaMA-13B was weaker; on NQ/TQA, 5% compression brought less than 5 EM drop versus full documents.QA transfer sometimes failed to outperform other compression baselines and remained considerably below oracle compression for LLaMA-13B.Source: §6 · p. 8

How do the length of the summaries vary?

  • Abstractive summary lengths varied despite relatively stable input-document lengths, and 4–24% of training examples had empty summaries, indicating selective augmentation.Extractive compression used a fixed number of prepended sentences per dataset; adaptive extractive augmentation was not tested and was proposed as future work.Source: §6 · p. 8, Figure 4 · p. 8

How does model leverage the in-context documents?

  • Incorrect copying was 81% with top-5 documents versus 51% with top-1 and 85% with GPT-3 compression, while our compressor reduced it to 39%.The authors note that query-focused compression may produce sentences that seem to contain the answer.Source: Table 3 · p. 8, §6 · pp. 8–9
  • Full documents most often contained the gold answer, followed by NE and GPT-3, but answer presence did not ensure better performance: gold answers were in evidence 84% for NE versus 98% for T5 (ours).The analysis notes that models do not always identify the correct answer from evidence.Source: Table 3 · p. 8, §6 · p. 8

Is generated summary faithful and comprehensive?

  • A manual evaluation compared GPT-3.5 and the abstractive compressor on 30 sampled non-empty summaries from NQ, TQA, and HotpotQA.The authors assessed faithfulness to retrieved documents and comprehensiveness for answering the question, using Yes, Partially, or No labels and reporting summaries that were both faithful and comprehensive as useful.Source: §6 · p. 9, Table 4 · p. 9

7 RELATED WORK

The section positions RECOMP against efficiency-oriented retrieval changes, prompt and context compression, and goal-oriented summarization, emphasizing textual compression as its approach.

Efficient RALM

  • Prior RALM work improves efficiency through compressed data stores, dimensionality reduction, selective retrieval, or larger strides; RECOMP instead summarizes retrieved documents or returns an empty sequence.Efficient RALM is described by He et al. (2021); selective retrieval by He et al. (2021) and Mallen et al. (2022); larger strides by Martins et al. (2022).Source: §7 · p. 9

Prompt Compression

Distillation / Goal Oriented Summarization

  • Related distillation and summarization work targets high-quality summaries, while RECOMP optimizes for downstream-LM effectiveness; Hsu & Tan (2021) optimize sentiment-prediction accuracy.Symbolic knowledge distillation transfers teacher knowledge through generated training data, with critic criteria filtering undesirable examples (West et al., 2022); this technique has also been applied to summarization (Jung et al., 2023).Source: §7 · pp. 9–10

8 CONCLUSION

RECOMP compresses retrieved documents into summaries before prepending them to retrieval-augmented language models. Its extractive and abstractive compressors use end-task signals from a black-box LM to generate useful summaries and support selective augmentation, improving efficiency with minimal performance drop.

8 CONCLUSION

  • RECOMP’s compressors significantly improve retrieval-augmented LM efficiency with minimal performance drop.The conclusion reports this overall experimental finding for the proposed compressors.Source: §8 · p. 10
  • RECOMP presents two compression models: an extractive compressor and an abstractive compressor.Both compress retrieved documents into textual summaries before they are prepended to the LM input.Source: §8 · p. 10
  • A training scheme uses end-task signals from a black-box LM to generate useful summaries and enable selective augmentation.Selective augmentation allows the compression models to return no augmentation when appropriate.Source: §8 · p. 10

ETHICS STATEMENT

The authors note that compressor training data generated with a commercial language model may contain factual errors, and report conducting human evaluation and analysis.

ETHICS STATEMENT

  • Commercial language models generate training data for the compressors; the authors say they carefully evaluate the data and present an analysis.The generated data might include factual errors.Source: ETHICS STATEMENT · p. 10

A APPENDIX · A.1 COMPRESSOR TRAINING DATA GENERATION · A.2 COMPRESSOR TRAINING DETAILS

The appendix describes how extractive and abstractive compressor training data are generated and gives model-training details, alongside tables and figures documenting datasets, examples, prompts, and analyses.

A.1 COMPRESSOR TRAINING DATA GENERATION

  • Training-data statistics are reported in Table 5, and SpaCy is used to extract named entities.Source: §A.1 · p. 15, Table 5 · p. 17
  • Extractive training examples come from Wikitext, NQ, TQA, and HotpotQA; sentences are split with NLTK, and examples without negatives are removed.Source: §A.1 · p. 15
  • For language modeling, four prompts produce candidate summaries; the lowest-perplexity summary is selected, or an empty summary if none lowers perplexity.The authors generate four summaries for a randomly sampled 2% of training data (48,013 examples), using the OpenAI API with temperature 0.7 and top p = 1.Source: §A.1 · p. 15, Table 8 · p. 19

A.2 COMPRESSOR TRAINING DETAILS

  • The extractive compressor uses an unsupervised Contriever checkpoint for language modeling and an MSMARCO-finetuned checkpoint for QA, with Adam training for three epochs.Training uses batch size 64, learning rate 2e-5, and 1,000 warmup steps; the selected model has best validation reranked perplexity for language modeling or reranked accuracy for QA.Source: §A.2 · p. 15
  • Figures 6 and 7 show extractive and abstractive QA compressor learning, with decoded-answer exact match as the score; Figure 8 shows test-summary length distributions.Source: Figure 6 · p. 16, Figure 7 · p. 16, Figure 8 · p. 17
  • The abstractive compressor is trained with Adam for three epochs, using batch size 16, learning rate 1e-5, and 1,000 warmup steps.Source: §A.2 · p. 17
  • Appendix tables provide compression examples, an NQ input example, an NQ case study, LLaMA-13B oracle QA results, and manually labeled example summaries.Tables 11 and 12 contain the manual-label examples; Table 12 continues Table 11.Source: Table 6 · p. 18, Table 7 · p. 18, Table 9 · p. 20

Related papers

Loading 2310.04408v1…