Source-linked AI summary
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu
TL;DR
Long-context LLMs face higher costs, performance reduction from prompt noise, and position bias. LongLLMLingua uses question-aware prompt compression with document reordering and related mechanisms to improve key-information perception. Across long-context tasks, compressed prompts achieve higher performance while reducing API costs and end-to-end latency, with limitations involving repeated recompression and more complex context-question relationships.
Problem
Long-context LLMs face higher computational costs, performance reduction from irrelevant information, and position bias affecting the use of relevant information.
Method
LongLLMLingua combines efficient prompt compression with question-aware coarse-to-fine compression, document reordering, dynamic compression ratios, and subsequence recovery.
Results
Across multi-document QA, multi-hop QA, and long-context benchmarks, LongLLMLingua’s compressed prompts achieve higher performance while reducing API costs and end-to-end latency.
Takeaways & Limitations
LongLLMLingua improves LLM perception of key information while reducing the cost and latency of long-context inference.
Takeaways & Limitations
LongLLMLingua requires recompression for different questions and increases computation to twice that of LLMLingua, while coarse question-aware compression may be affected by complex context-question relationships.
Abstract
from arXiv · showhide
In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias. Research indicates that LLM performance hinges on the density and position of key information in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges. Our extensive evaluation across various long context scenarios demonstrates that LongLLMLingua not only enhances performance but also significantly reduces costs and latency. For instance, in the NaturalQuestions benchmark, LongLLMLingua boosts performance by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo, leading to substantial cost savings. It achieves a 94.0% cost reduction in the LooGLE benchmark. Moreover, when compressing prompts of about 10k tokens at ratios of 2x-6x, LongLLMLingua can accelerate end-to-end latency by 1.4x-2.6x. Our code is available at https://aka.ms/LongLLMLingua.
1 Introduction
Long-context prompts create higher costs, performance degradation from irrelevant information, and position bias. LongLLMLingua addresses these challenges through question-aware compression and document reordering, and evaluations report gains in performance, cost efficiency, and latency.
- Long-context LLM use increases financial and latency costs, while irrelevant information can weaken downstream performance.
- Position bias makes LLM performance depend on where relevant information appears, including degradation when it is located in the middle.
- LongLLMLingua uses question-aware coarse-to-fine compression to improve key-information density and reduce noise retained during compression.
- LongLLMLingua introduces document reordering to minimize position bias and improve LLM perception of relevant information.
- Evaluations across five long-context benchmarks report that compressed prompts outperform original prompts in performance, cost efficiency, and system latency.
2 Problem Formulation
The problem formulation represents prompts as instructions, documents, and a question, then optimizes a compressed prompt to preserve LLM outputs while controlling compression. It also considers document permutations for joint optimization.
- A prompt is represented as an instruction, K documents, and a question, with the definition adjustable for specific scenarios.
- The compressed prompt is a token-level subsequence of the original prompt, and the objective compares their LLM-generated results.
- The distance function can measure output differences, such as KL divergence, while λ balances the compression ratio.
- The formulation explores permutations of the K documents for joint optimization.
3 Preliminary: LLMLingua
LLMLingua compresses prompts by using a small language model to score token perplexity and removing tokens with lower perplexity. Its pipeline includes budget control, iterative token compression, and distribution alignment.
- LLMLingua uses a small language model to evaluate each prompt token’s perplexity and removes tokens with lower perplexities.
- The method assumes lower-perplexity tokens have negligible effects on overall language-model entropy gain.
- LLMLingua includes a budget controller, iterative token-level prompt compression, and distribution alignment.
4 LongLLMLingua
LongLLMLingua extends prompt compression for long contexts by increasing question-relevant information density, reducing position bias, adapting compression budgets, and recovering subsequences in generated responses.
- Overview: LongLLMLingua builds on LLMLingua to improve LLM recognition of question-relevant key information in long prompts.It combines multiple compression perspectives with subsequence recovery.
- Question-aware compression: Question-aware coarse compression ranks documents by their association with the question, while fine-grained compression assesses tokens across instructions, questions, and retained documents.The coarse stage retains documents with higher importance scores before token-level compression.
- Question-aware compression: Contrastive perplexity measures the distribution shift caused by conditioning on the question to identify question-associated tokens.It is equivalent to conditional pointwise mutual information, and high-scoring tokens tend to cluster near the ground-truth document.
- Document reordering: Documents are reordered using question-association scores to better exploit LLMs’ differing information perception across positions.The strategy targets performance degradation when relevant information appears in the middle of long contexts.
- Adaptive granular control: Dynamic compression assigns more budget, meaning a lower compression ratio, to documents that are more relevant to the question.A linear scheduler allocates document and token budgets according to coarse-compression importance rankings.
- Subsequence recovery: Subsequence recovery replaces response substrings found in the compressed prompt with corresponding subsequences from the original prompt.The procedure matches response tokens to the compressed prompt, locates corresponding original spans, and restores them.
5 Experiments
The experiments evaluate LongLLMLingua across multiple long-context benchmarks, baselines, compression settings, and ablations. Results report higher performance alongside reduced inference cost and latency.
- LongLLMLingua is evaluated on NaturalQuestions, LongBench, ZeroSCROLLS, MuSiQue, and LooGLE, covering multi-document, general long-context, multi-hop, and long-dependency QA.
- LongLLMLingua outperforms original prompts across performance, cost efficiency, and system latency in the evaluated long-context scenarios.
- 21.4% performance improvement on NaturalQuestions is achieved with approximately 4x fewer GPT-3.5-Turbo input tokens when the ground-truth document is at the 10th position.
- Compression-based baselines perform poorly on many tasks with abundant irrelevant information, while retrieval methods decline as compression increases.
- Ablation study: Removing any proposed component causes a performance drop regardless of ground-truth answer position, supporting the question-aware compression, dynamic ratio, and subsequence recovery components.
- Latency evaluation: 2.6x acceleration is reached as compression increases, whereas OpenAI embedding and Selective-Context incur longer latency from repeated API calls or sequential entropy calculation.
6 Related Works
Related work addresses long-context modeling, prompt information distribution, retrieval, and prompt compression through complementary strategies. These approaches modify context capacity, relevance selection, or prompt representation.
- Long-context research expands usable context through staged pre-training, position-embedding modification or interpolation, sparse attention, and external memory.
- Information distribution in prompt: LLM performance decreases with less effective prompt information, and relevant information placed in the middle is harder to comprehend than information at the edges.
- Retrieval methods: Retrieval methods use sparse n-gram relevance or dense latent-space relevance computed with embedding and reranker models.
- Prompt compression methods: Prompt compression includes token pruning or merging, soft prompt tuning, and other methods with differing requirements for fine-tuning or inference-time information.
7 Conclusion
LongLLMLingua combines efficient prompt compression with mechanisms targeting key-information perception and position bias. Across long-context evaluations, its compressed prompts achieve higher performance while reducing cost and latency.
- LongLLMLingua addresses higher computational cost, performance reduction, and position bias through efficient prompt compression.
- Its four components are question-aware coarse-to-fine compression, document reordering, dynamic compression ratios, and subsequence recovery.
- Experiments on multi-document QA, multi-hop QA, and long-context benchmarks show compressed prompts outperform original prompts while reducing API inference costs and end-to-end latency.
Limitation
LongLLMLingua’s effectiveness may be constrained by question-specific recompression and by complex, subtle relationships between context and prompts.
- Limitation: Different questions require re-compression even for the same context, preventing context caching.The authors note that extending the approach from question-aware to task-aware compression could enable reuse and caching.
- Limitation: 2x greater computation than LLMLingua can create additional overhead in real-world applications.The authors identify this overhead as a limitation of the current question-aware approach.
- Limitation: Effectiveness may decline when context–prompt relationships are more complex and subtle.The authors attribute this scope boundary to the method’s coarse-level question-aware compression, despite testing across many tasks.
A Derivation Of Question-Aware Fine-Grained Compression
The derivation connects question-conditioned token probabilities with contrastive perplexity, yielding a token-level sensitivity distribution for question-aware compression.
- A Derivation Of Question-Aware Fine-Grained Compression: Bayes’ theorem expands the conditional likelihood of the question given a token and its preceding context.This expansion provides the derivation used for the question-aware sensitivity measure.
- A Derivation Of Question-Aware Fine-Grained Compression: The question distribution p(xque) and ground-truth distribution q(xi) are constants, so si represents the derived expression.The constant terms allow the sensitivity score to stand in for the expanded formulation.
- A Derivation Of Question-Aware Fine-Grained Compression: A single inference computes the token-level sensitive distribution p(xque|xi, x<i) for the question xque.This distribution represents the conditional likelihood of generating the question given token xi and its preceding context.
- A Derivation Of Question-Aware Fine-Grained Compression: Unrelated tokens can retain high original information while exhibiting relatively low contrastive perplexity.The passage identifies contrastive perplexity as equivalent in form to conditional pointwise mutual information.
B.1 Dataset Details
The evaluation covers several long-context benchmarks spanning question answering, summarization, classification, reordering, synthetic tasks, and code completion, using specified datasets and experimental settings.
- B.1 Dataset Details: Five benchmarks cover diverse long-context scenarios: NaturalQuestions, LongBench, ZeroSCROLLS, MuSicQue, and LooGLE.The benchmark set includes multi-document QA, multi-hop QA, general long-context tasks, and long-dependency QA.
- B.1 Dataset Details: LongBench contains 3,750 English problems across 16 tasks, with an average prompt length of 10,289 tokens.Its categories include single- and multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion.
- B.1 Dataset Details: ZeroSCROLLS contains 4,378 problems across 10 tasks, with an average prompt length of 9,788 tokens.Its task categories include summarization, question answering, aggregated sentiment classification, and information reordering.
- B.1 Dataset Details: MuSiQue requires multiple inferences across several documents and has an average prompt length of 2,477 tokens.The dataset includes 39,876 training, 4,834 validation, and 4,918 testing problems; experiments used the validation set.
- B.1 Dataset Details: LooGLE has an average prompt length of 24,005 tokens, and experiments use its 1,101-problem long-dependency QA subset.The subset covers information retrieval, timeline reordering, computation, and comprehension.
- B.1 Dataset Details: Experiments used GPT-3.5-Turbo for token counting on a Tesla V100, with fixed compression and segmentation settings.The setup specifies k=2, τins=0.85, τque=0.9, segment size 200, and δτ=0.3.
C.2 Ablation in LongBench
LongBench ablations show that each LongLLMLingua component contributes to performance, with question-aware compression especially important for document QA and synthetic tasks. The compressed prompt also generalizes beyond GPT-3.5-Turbo and preserves or improves performance across several long-context benchmarks.
- Ablation findings: Removing any proposed component causes a performance drop, while question-aware coarse-grained compression has the largest impact on document-based QA and synthetic tasks.Its maximum ablation drop reaches 35.8 points; its effect is smaller on summarization and code tasks.
- Ablation findings: Document reordering improves all task types and alleviates the “lost in the middle” issue without disrupting context understanding, including timeline-related tasks.Subsequence recovery helps reference-based tasks more than summarization, code, or synthetic tasks.
- Transfer across models: 15.5 points is the maximum improvement reported with compressed prompts on LongChat-13b-16k, with 3–5 point gains on most other tasks.Few-shot learning shows a slight drop, while retrieval-based and compressed-based methods follow trends similar to GPT-3.5-Turbo.
- Other benchmark results: 5.4 points improvement is achieved on the MuSiQue multi-hop QA task, while LooGLE improvements reach 15.9 points on long-dependency tasks.The results indicate gains for tasks requiring global information, retrieval, timeline reordering, and computation.
- Efficiency: 94.0% cost reduction is achieved on LooGLE, alongside savings on Multi-document QA, LongBench, ZeroSCROLLS, and MuSiQue.The estimates are per 1,000 samples using GPT-3.5-Turbo pricing and include input prompts and generated output text.
E Ablation Analysis
The paper analyzes how compression preserves question-relevant information and reports benchmark performance and inference-cost comparisons. Examples illustrate prompt transformations across question answering, code completion, and few-shot learning.
- Compression analysis: Contrastive perplexity preserves question-relevant information better than token-level compression without question awareness, especially at a 30x compression ratio.The comparison uses Multi-document QA prompts and GPT-3.5-Turbo.
- Case studies: LongLLMLingua is evaluated through before-and-after compression examples spanning NaturalQuestions, LCC code completion, and TREC few-shot learning.Figures 7–9 compare compressed prompts with original prompts and show the corresponding LLM responses.
- Benchmark and cost comparisons: Table 8 reports performance comparisons on the LooGLE long-dependency QA benchmark, while Table 9 reports inference costs per 1,000 GPT-3.5-Turbo samples.The supplied passages identify the benchmark and cost-analysis scopes but do not provide the individual table values.