PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressing Context to Enhance Inference Efficiency of Large Language Models

Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin

arXiv:2310.06201v1cs.CL

TL;DR

Long documents and conversations make LLM inference expensive and can exceed fixed context windows. The paper introduces Selective Context, which prunes redundant input using self-information, and reports lower inference costs with comparable performance and minor metric decreases. Its scope is constrained by lexical-unit boundary detection and task-dependent pruning thresholds.

  • Problem

    Long-context processing increases LLM memory and computation costs and can cause context truncation under fixed context windows.

  • Method

    Selective Context computes self-information for tokens or grouped lexical units and retains higher-information content while pruning redundancy.

  • Results

    Selective Context significantly reduces GPU memory cost and accelerates generation with minor performance decreases across arXiv, BBC News, and conversation experiments.

  • Takeaways & Limitations

    Selective Context can potentially help LLMs handle long documents and extended conversations without the risk of context truncation.

  • Takeaways & Limitations

    Performance depends on phrase-boundary detection and task- and context-specific pruning percentiles.

Abstract

from arXiv · show

Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

1 Introduction

Long contexts are costly for LLMs because attention computation and memory grow quadratically, causing accessibility and context-window problems. Selective Context addresses input redundancy by pruning less informative content while preserving comparable performance.

  • Motivation: Quadratic growth in Transformer attention makes long-context processing computationally expensive and can cause context truncation.Fixed context windows further constrain LLMs’ ability to process long conversations and documents.
  • Motivation: Input redundancy arises from natural-language repetition and overlap between context and LLM pre-training material.Such content can be unnecessary for generating correct answers, despite helping human comprehension.
  • Method: Selective Context ranks tokens, phrases, or sentences by self-information and retains higher-information content to compact the input.The method is model-agnostic and targets redundancy in the input context rather than changing model architecture or distillation.
  • Evaluation: Experiments on arXiv papers, BBC News, and ShareGPT conversations across four NLP tasks maintained comparable performance to full-context processing.The evaluated tasks included summarisation, question answering, original context reconstruction, and conversation.

2 Self-Information

Self-information measures how much information a token or lexical unit conveys, with rarer events receiving higher values. Selective Context uses this measure because additive token information can score larger units for redundancy filtering.

  • Definition: Self-information quantifies the information conveyed by an event under a probability distribution.In language modelling, the event is a generation step corresponding to a token.
  • Definition: A token’s self-information is its negative log likelihood under the language model’s output distribution.The supplied notation defines I(x) as token self-information and P(x) as its output probability.
  • Use in filtering: Lower-self-information lexical units are less informative and more likely to be inferred from surrounding context.The paper therefore treats them as potential redundant input content.
  • Related measures: Entropy is the average self-information of a sentence’s words, while perplexity is calculated from sentence entropy.These related quantities connect token-level information to sentence-level language-model measures.
  • Additivity: Self-information is additive, allowing a lexical unit’s value to be computed by summing the self-information of its tokens.Lexical units can therefore be evaluated at token, phrase, or sentence scale.

3 Method

Selective Context computes self-information for context content, groups tokens into lexical units when needed, and filters units by informativeness. Percentile-based retention adapts the selected content to each context’s information distribution.

  • 3.1 Computing Self-Information: Selective Context computes token self-information with a causal language model, optionally merges tokens into phrases or sentences, and removes least-necessary content.The pipeline supports causal models such as GPT, OPT, and LLaMA.
  • 3.2 Merging into Lexical Units: Lexical units can be tokens, phrases, or sentences, enabling filtering beyond individual tokens.Unit-level filtering avoids the disjoint context that token-only filtering might produce.
  • 3.2 Merging into Lexical Units: Phrase and sentence self-information is obtained by summing the self-information values of their constituent tokens.The implementation uses sentence tokenization and noun-phrase merging, but does not merge verb phrases.
  • 3.3 Selective Retention of Informative Context: Percentile-based filtering ranks lexical units by self-information and retains units at or above the selected percentile.This adapts retention to the distribution of self-information rather than using a fixed threshold or fixed top-k count.
  • 3.3 Selective Retention of Informative Context: At p=50, half of phrases are filtered out, retaining 57.2% of tokens and saving 42.7% of context length.The example is presented as phrase-level selective context.

4 Experiments

The experiments evaluate Selective Context across long documents and conversations, multiple tasks, datasets, models, and compression settings. They compare selective-context outputs with full-context references using standard generation-quality metrics.

  • Datasets: The evaluation covers BBC News, arXiv articles, and ShareGPT conversations, with arXiv experiments restricted to the first two sections.ShareGPT examples span multiple languages and scenarios, while arXiv papers cover several scientific disciplines.
  • Data controls: The datasets were created after March 2023, and inputs exceeding 2048 tokens were excluded to reduce contamination and accommodate model limits.Models were evaluated using versions released before 30 March 2023.
  • Models: Models include GPT-3.5, GPT-4, LLaMA models from 7B to 30B parameters, and Vicuna models at 7B and 13B.The selection spans proprietary and open-source model families and tests scaling within the LLaMA family.
  • Tasks: Selective Context is evaluated on summarisation, question answering, original context reconstruction, and conversation tasks.The evaluation uses compressed contexts as inputs and compares outputs with references derived from original or full contexts.
  • Metrics and references: Evaluation uses BLEU, METEOR, ROUGE, and BERTScore, with full-context generations serving as reference answers.For non-deterministic decoding, four full-context reference answers are used.
  • Experimental settings: Experiments vary compression ratios from 0.2 to 0.8 and adjust the lexical unit used for content reduction.These settings probe the trade-off between efficiency and performance as retained information changes.

5 Results

Selective Context generally preserves task performance while reducing context cost, with phrase-level filtering identified as optimal. Its benefits are strongest for summarisation and question answering, though aggressive compression can shorten, omit, or alter generated content.

  • 5.1 Overview: Selective Context causes only marginal performance drops at reduction ratios of 0.2 and 0.35, despite reducing context cost.At ratio 0.2, BLEU drops by 0.05 and ROUGE-1 by 0.03.
  • 5.1 Overview: At ratio 0.5, average BLEU drops 0.17 and average ROUGE-1 drops 0.12; higher ratios produce larger declines, including a 0.32 BERTScore-F1 drop at 0.8.A reduction ratio of 0.65 also produces a 0.18 ROUGE-1 drop.
  • 5.1 Overview: BERTScore-F1 remains above 0.9 at compression ratios of 0.5 or lower, while ROUGE-1 stays above 0.64 and ROUGE-L above 0.5.These results compare Selective Context with the random deletion baseline under greedy decoding.
  • 5.2 Faithfulness: At compression ratio 0.65, gpt-3.5 refuses 19 questions and produces answers 35% shorter, while non-entailed tuples rise from 3.8% at 0.5 to 5.1%.The faithfulness evaluation covered 1,000 question-answering pairs.
  • 5.3 Tasks: Selective Context preserves most key points in reconstruction below ratio 0.5, while summarisation and question answering decline more gradually and remain stronger than the other evaluated tasks.The authors identify summarisation and answer generation as especially suitable applications.
  • 5.6 Case Study: At ratio 0.5, Selective Context uses 61,885 MB of CUDA memory, takes 76.3 ms/token, and requires 46.1 ms to construct for the example.The construction cost is one-time and includes self-information computation and lexical-unit tokenisation.
  • 5.5 Lexical Units: Phrase-level filtering consistently outperforms token- and sentence-level filtering, making lexical-unit granularity an important design choice.Sentence-level redundancy removal is described as unstable compared with token- and phrase-level filtering.
  • 5.6 Case Study: Selective Context reduces inference memory by about 36% and makes generation 1.32 times faster per token at compression ratio 0.5.The case study also reports omitted background information in one selective-context summary, without factual errors in that summary.

6 Conclusion

Selective Context improves LLM inference efficiency by removing redundant content measured through self-information. Experiments across long documents and conversations show lower GPU memory cost and faster generation with minor performance decreases.

  • Selective Context deletes redundant content measured by self-information to improve context efficiency during LLM inference.
  • Experiments on arXiv papers, BBC news articles, and conversation transcripts showed reduced GPU memory cost and accelerated generation.
  • Selective Context produced minor performance decreases while potentially helping LLMs handle long documents and extended conversations without context truncation.

7 Limitations

The method has limitations related to lexical-unit boundary detection and compression-threshold selection. These choices can affect compression quality and vary across tasks and contexts.

  • Selective Context depends partly on phrase-boundary detection using spaCy noun-phrase tokenisation, while verb phrases are not considered.
  • Dependency-tree-based filtering may improve lexical-unit boundary identification and compression performance.
  • The optimal compression percentile varies across specific tasks and contexts, motivating tools that identify suitable thresholds.

A Dataset statistics

Table 4 reports averaged dataset statistics for sentences, phrases, and tokens per document across the three datasets.

  • Table 4 lists average numbers of sentences, phrases, and tokens per document for three datasets.

B Example of selective context on long context

The appendix presents a long-context example from an arXiv paper, showing the original paragraphs, the resulting selective context, and a reference summary. The example concerns autonomous-driving research.

  • The example uses original paragraphs from arXiv paper 2303.07352 as the long input context.
  • The resulting context is shown after applying Selective Context to the original paragraphs.
  • A reference summary generated from the full context is provided for comparison.
  • The source text discusses autonomous-driving research, including collision avoidance and methods for complex traffic conditions.
Loading 2310.06201v1…