PaperlayerView exact revision on arXiv

Source-linked AI summary

RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation

Fangyuan Xu, Weijia Shi, Eunsol Choi

arXiv:2310.04408v1cs.CL

TL;DR

Long retrieved documents improve retrieval-augmented LMs but increase inference cost and can make relevant information difficult to use. RECOMP trains extractive and abstractive compressors to prepend concise, faithful summaries—or nothing when retrieval is unhelpful—to frozen LMs. It achieves compression as low as 6% with minimal performance loss, while trained compressors transfer across language models and reduce erroneous copying.

  • Problem

    Retrieved documents improve retrieval-augmented LM performance but substantially increase encoded tokens, while LMs can miss information in long contexts.

  • Method

    RECOMP trains extractive sentence selectors and abstractive multi-document summarizers using downstream LM signals, with empty summaries enabling selective augmentation.

  • Results

    RECOMP improves frozen-LM performance across language modeling and three QA datasets while reducing context length; oracle compression reaches 6%, and trained compressors retain performance with 5–25% of original tokens.

  • Takeaways & Limitations

    Textual compression makes retrieval augmentation more efficient, transfers across LMs on language modeling, and reduces incorrect copying from retrieved evidence to 39%.

  • Takeaways & Limitations

    Selective compression is demonstrated for abstractive compressors, while extractive compression uses a fixed number of prepended sentences across each dataset.

Abstract

from arXiv · show

Retrieving documents and prepending them in-context at inference time improves performance of language model (LMs) on a wide range of tasks. However, these documents, often spanning hundreds of words, make inference substantially more expensive. We propose compressing the retrieved documents into textual summaries prior to in-context integration. This not only reduces the computational costs but also relieves the burden of LMs to identify relevant information in long retrieved documents. We present two compressors -- an extractive compressor which selects useful sentences from retrieved documents and an abstractive compressor which generates summaries by synthesizing information from multiple documents. Both compressors are trained to improve LMs' performance on end tasks when the generated summaries are prepended to the LMs' input, while keeping the summary concise.If the retrieved documents are irrelevant to the input or offer no additional information to LM, our compressor can return an empty string, implementing selective augmentation.We evaluate our approach on language modeling task and open domain question answering task. We achieve a compression rate of as low as 6% with minimal loss in performance for both tasks, significantly outperforming the off-the-shelf summarization models. We show that our compressors trained for one LM can transfer to other LMs on the language modeling task and provide summaries largely faithful to the retrieved documents.

1 INTRODUCTION

RECOMP inserts a compression step between retrieval and in-context augmentation, producing concise summaries that guide frozen LMs while reducing encoded tokens. Its extractive and abstractive compressors are trained for end-task performance and can selectively omit unhelpful retrievals.

  • 1 INTRODUCTION: RECOMP compresses retrieved documents into textual summaries before prepending them to the LM input.The summary is intended to guide generation while reducing the computation required to encode retrieved documents.
  • 1 INTRODUCTION: Two compressors are proposed: extractive selection of relevant sentences and abstractive synthesis across multiple retrieved documents.Both use query-focused summarization trained to improve downstream task performance.
  • 1 INTRODUCTION: RECOMP improves frozen-LM performance on language modeling and three open-domain QA datasets while prepending significantly fewer tokens than uncompressed retrieval augmentation.The evaluated QA datasets are Natural Questions, TriviaQA, and HotpotQA.
  • 1 INTRODUCTION: 6% compression rates are achieved by oracle methods while significantly outperforming prepending full documents.Trained compressors reach 25% compression for language modeling and 5–10% of original tokens on QA with at most less than 10% relative performance drop.

2 PROBLEM FORMULATION: RECOMP

RECOMP maps a query and retrieved documents to a concise summary that is prepended to a black-box LM. The summary should be efficient, useful for generating the target output, and faithful to the retrieved evidence, including the option of being empty.

  • 2 PROBLEM FORMULATION: RECOMP: RECOMP compresses N retrieved documents D with respect to input x into a shorter summary s containing relevant core information.The compressor is designed to be substantially smaller than the LM.
  • 2 PROBLEM FORMULATION: RECOMP: The summary may be empty when retrieved documents are irrelevant or retrieval augmentation is unnecessary.This supports selective augmentation rather than forcing retrieval context into every input.
  • 2 PROBLEM FORMULATION: RECOMP: The summary must be effective: prepending s to x should enable the LM to generate target output y.Effectiveness is defined through the downstream LM task.
  • 2 PROBLEM FORMULATION: RECOMP: The summary must be faithful and interpretable, meaning it is entailed by the retrieved document set.Conciseness, effectiveness, and faithfulness are the stated output criteria.
  • 2 PROBLEM FORMULATION: RECOMP: The framework assumes retrieved documents are already provided; improving the retriever is outside the work’s focus.The compressor operates after retrieval.

3 LEARNING THE COMPRESSORS

RECOMP learns extractive and abstractive compressors using downstream LM signals rather than human summary annotations. The extractive model ranks sentences, while the abstractive model distills and filters teacher-generated summaries, including empty outputs when augmentation hurts.

  • 3.1 EXTRACTIVE COMPRESSION: The extractive compressor ranks candidate sentences by their usefulness to the downstream LM and concatenates the top-ranked sentences.A dual encoder embeds the input and each sentence; their inner product determines ranking.
  • 3.1 EXTRACTIVE COMPRESSION: Extractive training identifies positive and negative sentences from retrieved documents using contrastive learning signals from the base LM.The positive sentence gives the highest target-output log likelihood when prepended to the input.
  • 3.1 EXTRACTIVE COMPRESSION: Extractive compression generally preserves faithfulness, although it does not always do so.The paper notes that faithfulness violations are rarer than for abstractive approaches, which can hallucinate more easily.
  • 3.2.1 CREATING TRAINING DATASET FOR DISTILLATION: The abstractive compressor is trained by distilling teacher-generated query-focused summaries and filtering them with end-task performance.The model is an encoder-decoder compressor trained from filtered examples.
  • 3.2.1 CREATING TRAINING DATASET FOR DISTILLATION: When a generated summary worsens performance relative to no retrieval, the training target is replaced with an empty string.This filtering procedure enables selective augmentation and mitigates irrelevant-document risks.
  • 3.2.1 CREATING TRAINING DATASET FOR DISTILLATION: For QA, training examples use dataset-specific prompts and discard cases where prepending the summary fails to improve performance.HotpotQA summary generation is restricted to examples whose gold answer appears in retrieved documents.

4 EXPERIMENTAL SETTINGS

The experiments measure both task effectiveness and context-token efficiency on language modeling and open-domain QA. They compare RECOMP with heuristic, retrieval-based, random, and oracle compression baselines across multiple LMs and datasets.

  • 4 EXPERIMENTAL SETTINGS: Effectiveness is measured by task performance, while efficiency is measured by the number of context tokens provided.Both measures are reported for language modeling and open-domain QA.
  • 4 EXPERIMENTAL SETTINGS: Language modeling uses WikiText-103 perplexity across GPT2, GPT2-XL, and GPT-J, with compressors trained using GPT2.BM25 retrieves from a Wikipedia corpus dated December 20, 2018.
  • 4 EXPERIMENTAL SETTINGS: Open-domain QA evaluates Natural Questions, TriviaQA, and HotpotQA using Exact Match and token-level F1.The base LM is Flan-UL2 and retrieval uses a Contriever model on Wikipedia.
  • 4.3 BASELINES AND ORACLES: Baselines include BoW, Named Entities, BM25, Contriever, DPR, and Random sentence selection.The extractive baselines vary by task and dataset.
  • 4.3 BASELINES AND ORACLES: Oracle compression selects sentences or summaries yielding the best end-task performance, including a no-augmentation option for abstractive compression.Model-independent oracle results use GPT-2 as a reference LM for language-modeling transfer tests.

5 RESULTS

Across language modeling and open-domain QA, compression preserves or improves retrieval-augmented performance while substantially reducing the number of prepended tokens. Results also show that selective removal of irrelevant information and task-specific compressor behavior are important.

  • Language modeling: All retrieval augmentation methods improve language-modeling perplexity over no retrieval, while heuristic BoW and NE compression underperform prepending uncompressed documents.The authors attribute the heuristic methods' weakness potentially to disfluent prepended text.
  • Language modeling: 6-13% of tokens is sufficient for both oracle settings to substantially outperform prepending the entire document set.Prepending the top 1 document also outperforms the top 5, indicating that additional retrieved documents can be harmful when irrelevant.
  • Language modeling: 25% compression achieves minimal performance drop for the trained compressors in language modeling, while the trained abstractive compressor attains the lowest perplexity and highest compression ratio.The trained extractive compressor significantly outperforms Contriever and BM25 across all three language models while prepending slightly fewer tokens.
  • Open-domain QA: Across three QA datasets, retrieval augmentation improves performance over no retrieval, and prepending five documents significantly outperforms prepending one.This motivates compression methods that can incorporate information from multiple retrieved documents without retaining all their tokens.
  • Open-domain QA: Extractive oracles outperform abstractive oracles across all QA datasets, and both improve over prepending all information by removing irrelevant content.The extractive oracle selects the best sentence from N candidates, whereas the abstractive oracle chooses between a GPT-3.5 summary and no augmentation.
  • Open-domain QA: On NQ and TQA, the trained abstractive compressor reaches 5% token compression while losing 2 and 3.7 EM points, respectively, versus prepending full documents.On HotpotQA, which requires multihop understanding, the extractive approach is more helpful at 11% compression with a 2.4 EM-point loss.

6 ANALYSIS AND DISCUSSIONS

The analysis examines transfer, summary-length variability, evidence use, and faithfulness/comprehensiveness. RECOMP’s summaries support selective augmentation and reduce incorrect copying from irrelevant contexts.

  • Transfer Across Different LMs: Trained compressors transfer well from GPT2 to GPT2-XL and GPT-J on language modeling.
  • Summary Length and Selective Augmentation: Abstractive summary lengths vary substantially despite similar input-document lengths, supporting selective retrieval augmentation.For abstractive compressors, 4-24% of training examples contain empty summaries because retrieved documents can hurt performance.
  • Evidence Use: More gold answers in evidence do not guarantee better performance because models may fail to identify the correct answer.The reported copying-related comparison is 84% for NE versus 98% for T5(ours).
  • Evidence Use: Incorrect copying rises from 51% with the top 1 document to 81% with the top 5 documents and 85% with GPT-3 compression.RECOMP reduces this erroneous behavior to 39%.
  • Summary Quality: Abstractive summaries are manually evaluated for faithfulness and comprehensiveness using Yes, Partially, and No labels.Faithfulness asks whether summaries are entailed by retrieved documents, while comprehensiveness asks whether they contain enough information to answer the question.

7 RELATED WORK

Related work improves retrieval-augmented LM efficiency through retrieval optimization, selective retrieval, context compression, distillation, and goal-oriented summarization. RECOMP differs by producing concise textual summaries for in-context use.

  • Efficient RALM: Efficient RALM methods improve retrieval components through datastore compression, dimensionality reduction, reduced retrieval frequency, or larger retrieval strides.
  • Prompt Compression: Prompt-compression methods represent long contexts as summary vectors or soft prompts to reduce inference costs.
  • Distillation / Goal Oriented Summarization: Context distillation transfers information into models through distillation-based approaches.The passage introduces context distillation as a related line of work alongside soft-prompt compression.
  • Distillation / Goal Oriented Summarization: Prior goal-oriented summarization trains an extractive model to optimize a downstream sentiment-prediction model’s accuracy from the summary.

8 CONCLUSION

The conclusion presents RECOMP as a textual compression method for in-context retrieval augmentation, using extractive and abstractive compressors trained with end-task signals. Experiments report substantially improved efficiency with minimal performance loss.

  • RECOMP compresses retrieved documents into textual summaries before prepending them to retrieval-augmented language models.
  • RECOMP includes extractive and abstractive compression models.
  • End-task signals from a black-box LM train compressors to generate useful summaries and perform selective augmentation.
  • Experiments show significantly improved retrieval-augmented LM efficiency with minimal performance drop.

ETHICS STATEMENT

The paper uses a commercial language model to generate compressor training data. The authors note that this data may contain factual errors and report conducting human evaluation.

  • Commercial language-model outputs are used to generate training data for the compressors.
  • The generated training data might include factual errors.
  • The authors conduct careful human evaluation of the generated data and present an analysis.

A.1 COMPRESSOR TRAINING DATA GENERATION

The paper describes data-generation procedures for training extractive and abstractive compressors, including empty summaries when augmentation does not help.

  • Extractive Data Generation: Extractive compressor training data uses the four tested datasets: Wikitext, NQ, TQA, and HotpotQA.Sentence splitting is performed with NLTK, and examples without negatives are removed.
  • Abstractive Data Generation: Abstractive training summaries are generated with GPT-3.5-turbo using prompts reported in Table 8.The prompts use temperature 0.7 and top p = 1.
  • Abstractive Data Generation: For language modeling, four generated summaries are ensembled and the summary yielding the lowest perplexity is selected as the target.When no summary decreases perplexity, an empty summary is used instead.
  • Compressor Implementation: The compressors use Contriever checkpoints, with unsupervised training for language modeling and MSMARCO fine-tuning for question answering.The implementation uses Transformers and sentence-transformer libraries.
  • Training Inputs: The training data records inputs, retrieved sentences or documents, and target answers for compressor training.The extractive setup uses {x_i, S_i, y_i}, while the abstractive setup uses {x_i, D_i, y_i}.
Loading 2310.04408v1…