PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Prompt Compression Methods for Long Context Inference

Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, Amir Gholami

arXiv:2407.08892v1cs.CLcs.LG

TL;DR

Long-context inference brings compute, memory, cost, and reasoning challenges, while prior prompt-compression evaluations vary enough to make comparisons difficult. This paper characterizes and evaluates extractive, abstractive, and token-pruning methods across tasks. It finds extractive compression often outperforms the alternatives, with up to 10× compression and minimal accuracy degradation.

  • Problem

    Long-context inference raises system and reasoning challenges, while inconsistent benchmarks, compression rates, and prompt lengths leave the performance of compression methods difficult to compare.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    The paper characterizes and evaluates extractive, abstractive, and token-pruning methods across single-document QA, multi-document QA, and summarization datasets.

    Source: §1 · pp. 1–2

  • Results

    Extractive compression often outperforms other approaches, achieving up to 10× compression with minimal accuracy degradation; token pruning often lags and summarization gains are marginal.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Extractive compression is a strong-performing option among the evaluated approaches, while results vary by method and task.

    Source: §6 · p. 10

    The study focuses on English datasets and knowledge-intensive long-context inference, so its findings do not cover other languages or prompt settings such as many-shot prompting and verbose system prompts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long context inference presents challenges at the system level with increased compute and memory requirements, as well as from an accuracy perspective in being able to reason over long contexts. Recently, several methods have been proposed to compress the prompt to reduce the context length. However, there has been little work on comparing the different proposed methods across different tasks through a standardized analysis. This has led to conflicting results. To address this, here we perform a comprehensive characterization and evaluation of different prompt compression methods. In particular, we analyze extractive compression, summarization-based abstractive compression, and token pruning methods. Surprisingly, we find that extractive compression often outperforms all the other approaches, and enables up to 10x compression with minimal accuracy degradation. Interestingly, we also find that despite several recent claims, token pruning methods often lag behind extractive compression. We only found marginal improvements on summarization tasks.

1. Introduction

Longer contexts support applications such as legal and financial analysis, but increase system costs and can challenge model reasoning. The paper compares prompt-compression paradigms and finds extractive compression often performs best.

1. Introduction

  • Long-context applications face increased computational demands, memory requirements, and costs, alongside potential declines in reasoning over extended sequences.The paper cites legal and financial document analysis, codebase copilots, summarization, and ongoing dialogue systems as examples.Source: §1 · p. 1
  • The paper characterizes extractive, abstractive, and token-pruning methods, distinguishing query-aware from query-agnostic approaches.It also studies chunk size and query-aware abstractive summarization.Source: §1 · pp. 1–2
  • Extractive compression often outperforms other approaches, reaching up to 10× compression with minimal accuracy degradation; token pruning often lags, and summarization gains are marginal.The evaluation spans single-document QA, multi-document QA, and summarization datasets.Source: §1 · p. 2

2. Related Work

Longer contexts raise compute, memory, cost, and reasoning challenges, while retrieval and compression aim to provide enough relevant information without unnecessarily long prompts. The related work organizes text-based compression by how it alters content and whether it uses the query.

2.1. Long Context LLMs

  • Attention compute and memory scale quadratically with sequence length, while long prompts may be costly and models can underuse relevant information in the middle of a context.The passage describes a lost-in-the-middle effect in which relevant context is not properly utilized.Source: §2.1 · p. 2

2.2. Retrieval-Augmented Generation

  • RAG retrieves relevant chunks, but retrieving too few risks missing key information and retrieving too many lengthens prompts; the stated objective is the minimum information needed to answer.The passage notes that limited context windows can make many retrieved chunks impossible to provide.Source: §2.2 · p. 2

2.3. Prompt Compression

  • Text-based prompt compression distills a long prompt to retain critical information while reducing length; the paper focuses on direct text manipulation over embedding-based compression.Embedding-based approaches require extensive fine-tuning and inference-pipeline changes, restricting use with API-based LLM services.Source: §2.3 · p. 2

2.3.1. TOKEN PRUNING BASED COMPRESSION

  • Token pruning removes tokens judged irrelevant using signals such as self-information, perplexity, or attention, with some methods first removing whole documents or examples.LongLLMLingua conditions perplexity on the question and supporting documents; PCRL instead trains a policy to keep or remove tokens.Source: §2.3.1 · pp. 2–3

2.3.2. ABSTRACTIVE COMPRESSION

  • Abstractive compression uses summarization to shorten context, with methods ranging from prompted or fine-tuned summarizers to knowledge-graph-based selection.RECOMP can condition summaries on the question, while PRCA uses supervised training followed by policy optimization.Source: §2.3.2 · p. 3, Figure 2 · p. 3

2.3.3. EXTRACTIVE COMPRESSION

  • Extractive compression keeps relevant documents, sentences, or phrases from the original context, including via relevance-scoring encoders or document rerankers.Unlike abstractive compression, the selected text is taken from the original context.Source: Figure 2 · p. 3, §2.3.3 · p. 4

2.3.4. QUERY-AWARE VS QUERY-AGNOSTIC COMPRESSION

  • Query-aware methods tailor compression to the question or task, whereas query-agnostic methods do not use it and can be run offline once.The methods are also grouped into token pruning, abstractive, and extractive classes.Source: §2.3.4 · p. 4, Table 1 · p. 4

3. Evaluation Methodology for Prompt Compression Methods

Existing evaluations vary in benchmarks, compression ratios, and original prompt lengths, making method comparisons and application choices difficult. This study standardizes evaluation across three model families, three task types, and multiple compression approaches.

3.1. Motivation

  • Existing studies differ in benchmarks, compression ratios, and initial prompt lengths, making performance difficult to characterize and method choice unclear.Examples include LongLLMLingua at roughly 10,000-token prompts and near-5× compression, RECOMP at 500 tokens and 20×, and Prompt-SAW on two datasets.Source: §3.1 · pp. 4–5
  • The evaluation asks about design challenges, approach trade-offs, and whether application settings favor particular methods.The study aims to address the lack of standardization with a rigorous comparison of token pruning, extractive, and abstractive compression.Source: §3.1 · p. 5

3.2. Setup

  • Experiments use GPT-3.5-Turbo, Mixtral 8x7B, and DBRX Instruct with temperature zero and greedy decoding.Because GPT-3.5-Turbo is nondeterministic at these settings, its results are averaged over three trials; the listed context windows are 16k tokens for GPT-3.5-Turbo and 32k for the other two.Source: §3.2 · p. 5
  • LongBench supplies three datasets each for single-document QA, multi-document QA, and summarization, evaluated with F1 for QA and ROUGE for summarization.The datasets include NarrativeQA, Qasper, MultiFieldQA-en; HotpotQA, 2WikiMultihopQA, MuSiQue; and GovReport, QMSum, MultiNews.Source: §3.2 · p. 5
  • Chunks are approximately 128 tokens with sentence boundaries preserved, allowing chunk-based methods to assess segments independently for query relevance.The study uses “chunk” and “document” interchangeably.Source: §3.2 · p. 5

3.3. Evaluated Methods

  • The uncompressed Original method sends the whole prompt, truncating only if needed, and serves as the reference for other methods’ compression rates.The methods evaluated include LongLLMLingua, reranking, their combinations with token pruning, and query-agnostic abstractive compression.Source: §3.3 · p. 5
  • The reranker scores each question–chunk pair from 0 to 1, and the most relevant chunks are retained; selected-chunk count sets the compression rate.The reranker is mxbai-rerank-large-v1, a fine-tuned DeBERTa model.Source: §3.3 · p. 5
  • The custom reranker token-pruning method removes the lowest-attention document tokens relative to query tokens, pruning a fixed percentage at each layer.It prunes 2% of tokens in each of the last 10 layers, while reranker-selected chunk count varies the compression rate; LongLLMLingua and its reranker variant combine coarse chunk pruning with token pruning.Source: §3.3 · pp. 5–6

4. Experiments

The experiments compare extractive, abstractive, and token-pruning methods, then examine retrieval, pruning intensity, query-aware summarization, chunk size, and Text-to-SQL. Main results use GPT-3.5-Turbo, with similar trends reported for Mixtral 8x7B and DBRX Instruct.

4.1.2. ABSTRACTIVE COMPRESSION

  • Query-agnostic abstractive compression trails extractive compression by 3–5 points on summarization and is typically 10–15 accuracy points below reranking on MultifieldQA at matched compression.The authors identify possible information loss or hallucinations from smaller summarizers as a challenge.Source: §4.1.2 · p. 6

4.1.3. TOKEN PRUNING

  • LongLLMLingua variants typically perform worst, while reranker + token pruning generally trails reranking but can compete on summarization at high compression.Reranker + token pruning outperforms reranking on GovReport and QMSum with Mixtral 8x7B at higher compression; the authors hypothesize unstructured pruning disrupts grammar and sentence comprehension.Source: §4.1.3 · p. 6

4.1.1. EXTRACTIVE COMPRESSION

  • On 2WikiMultihopQA with GPT-3.5-Turbo, reranker compression reaches 7.75× while accuracy rises 7.89 points.The reranker performs strongly across models and datasets; the authors attribute an extractive-compression advantage partly to preserving grammatical constructs.Source: §4.1.1 · p. 6, Figure 4 · p. 7

4.2.1. RETRIEVER VS RERANKER

  • The reranker outperforms embedding retrieval, while retrieval requires fewer inference resources because document embeddings are computed offline.The study uses text-embedding-3-small; retrieval may precede reranking to reduce the number of documents reranked.Source: §4.2.1 · p. 7, Figure 5 · p. 8

4.2.2. AGGRESSIVE TOKEN PRUNING

  • Applying 50% rather than 20% token pruning with the reranker degrades accuracy, which the authors hypothesize reflects disruption of grammatical structure.In this study the reranker selected twice as many chunks for the aggressive-pruning setup.Source: §4.2.2 · p. 7, Figure 6 · p. 8

4.2.3. QUERY-AWARE ABSTRACTIVE COMPRESSION

  • Query-aware abstractive compression performs 3–6 points better than query-agnostic compression on NarrativeQA, MultiFieldQA, and HotpotQA across both summarizer models.The method reranks chunks, then summarizes selected chunks with Mistral 7B or Llama 8B; the authors note it requires on-the-fly summarization.Source: §4.2.3 · pp. 7–8, Table 2 · p. 9

4.2.4. IMPACT OF CHUNK SIZE

  • At high compression ratios, large chunks perform worse than smaller chunks; chunk size has less impact at lower compression ratios.The authors hypothesize that larger chunks provide fewer regions of the original context, and recommend choosing chunk size for the data source and target compression ratio.Source: §4.2.4 · p. 8, Figure 7 · p. 9

4.3. Case Study: Text-to-SQL

  • On Text-to-SQL, reranking significantly outperforms reranking plus token pruning, and join-query accuracy falls more sharply than overall accuracy as compression increases.From 1.62× to 4.29× compression, join accuracy drops from 0.63 to 0.37, while overall accuracy drops from 0.67 to 0.56; contexts are chunked by CREATE TABLE statement.Source: §4.3 · p. 9, Figure 8 · p. 9

5. Future Directions

The authors identify combining compression methods and tailoring token pruning to specific applications as future directions. They also leave other prompt settings and languages for future study.

5. Future Directions

  • Combining methods such as LLMLingua and LongLLMLingua’s coarse-to-fine scheme is proposed as a future direction.The scheme combines extractive compression and token pruning.Source: §5 · pp. 9–10
  • Application-specific token pruning could exploit task structure, such as SQL grammar in Text-to-SQL.The authors report that their general token-pruning method led to significant performance loss in Text-to-SQL.Source: §5 · p. 10
  • The study focuses on English and knowledge-intensive long-context inference, leaving other languages and many-shot or verbose-system prompts for future work.Source: §5 · p. 10

6. Conclusions

The study evaluates extractive, summarization-based abstractive, and token-pruning methods for long-context inference. Extractive compression often outperforms the alternatives, while summarization-task improvements are marginal.

6. Conclusions

  • The study characterizes and evaluates a broad spectrum of prompt compression methods for long-context inference systems.Source: §6 · p. 10
  • Extractive compression often outperforms other approaches and achieves up to 10× compression with minimal accuracy degradation.Source: §6 · p. 10
  • Token-pruning methods often lag behind extractive compression, despite recent claims.Source: §6 · p. 10
  • Summarization tasks show only marginal improvements from prompt compression methods.Source: §6 · p. 10

A. LongBench Dataset Details

The appendix describes LongBench datasets and reports a LongLLMLingua hyper-parameter sweep. The tested configurations exhibit similar trends, and the authors’ chosen settings perform well.

A. LongBench Dataset Details

  • LongBench includes single-document and multi-document question-answering datasets, with average token counts ranging from 4,923 for Qasper to 29,780 for NarrativeQA.MultiFieldQA covers varied document types and averages 6,938 tokens.Source: A. LongBench Dataset Details · p. 13
  • Other LongBench datasets cover 2-hop through 5-hop QA and summarization of government reports, meeting notes, and news articles.HotpotQA averages 12,793 tokens; 2WikiMultihopQA 7,116; MuSiQue 15,577; GovReport 10,242; QMSum 13,855; and MultiNews 2,609.Source: A. LongBench Dataset Details · p. 13

B.1. LongLLMLingua Hyper-Parameter Sweep

  • The chosen LongLLMLingua settings perform well, and all tested configurations exhibit similar trends across the sweep.The sweep toggles sentence-level filtering and varies the dynamic context compression ratio between 0, 0.2, 0.3, and 0.4; Figures B.1 and B.2 show Mixtral 8x7B and GPT-3.5-Turbo results, respectively.Source: Figure B.1 · p. 13, §B.1 · p. 13, Figure B.2 · p. 14

B.2. Full Retriever vs Reranker Results

The appendix presents the full retriever-versus-reranker results from Figure 5 across nine LongBench datasets. The figure concerns extractive compression and uses GPT-3.5-Turbo.

B.2. Full Retriever vs Reranker Results

  • Figure B.3 provides the Figure 5 results across all nine LongBench datasets.Source: Figure B.3 · p. 14, §B.2 · p. 14
  • The comparison is between standard retrieval over embedding space and reranking for extractive compression.Source: Figure B.3 · p. 14
  • The retrieval embeddings are produced with text-embedding-3-small, and GPT-3.5-Turbo is the LLM used.Source: Figure B.3 · p. 14

B.3. Full Aggressive Token Pruning Results

Figure B.4 compares aggressive token pruning at 20% and 50% pruning on nine LongBench datasets, using GPT-3.5-Turbo.

B.3. Full Aggressive Token Pruning Results

  • The Figure B.4 results cover all nine LongBench datasets.The passage identifies these as results from Figure 6 in the main text.Source: §B.3 · p. 14
  • Figure B.4 compares the original 20% token-pruning method with a more aggressive 50% version.GPT-3.5-Turbo is the LLM used.Source: Figure B.4 · p. 15
  • GPT-3.5-Turbo is used to evaluate the two pruning rates.Source: Figure B.4 · p. 15

B.4. Full Query-Aware Abstractive Compression Results

The appendix reports query-aware abstractive-compression results and prompts, compares reranker sizes, and examines chunk size. Summary length is difficult to control, making compression rates hard to predict.

B.4. Full Query-Aware Abstractive Compression Results

  • Summary length was difficult to control, making query-aware abstractive compression rates hard to predict.The results cover seven LongBench datasets with GPT-3.5-Turbo and Mixtral 8x7B; Mistral 7B Instruct and LLaMA-3 8B Instruct generate summaries from reranker-selected chunks.Source: §B.4 · p. 15, Table B.1 · p. 15, Table B.2 · p. 16

B.5. Abstractive Compression Prompts

B.6. Impact of Weaker Reranker Model

  • The 435M large reranker generally outperforms the 184M base model across three datasets, though the base wins at some points and may suit resource-constrained settings.The base-model study modifies token pruning to prune by 4% starting at layer 8; Figures B.5 and B.6 compare the rerankers with GPT-3.5-Turbo and Mixtral 8x7B, respectively.Source: §B.6 · p. 16, Figure B.5 · p. 16, Figure B.6 · p. 17

B.7. Impact of Chunk Size on Token Pruning Reranker

  • Figure B.7 varies token-pruning-reranker chunk size across 64, 128, 256, and 512 tokens while respecting sentence boundaries.Source: Figure B.7 · p. 17, §B.7 · p. 17

B.8. Mixtral 8x7B and DBRX Instruct Results

The appendix presents results for various compression methods with DBRX Instruct and Mixtral 8x7B. The figures plot accuracy against compression rate for each dataset.

B.8. Mixtral 8x7B and DBRX Instruct Results

Related papers

Loading 2407.08892v1…