Source-linked AI summary
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu
TL;DR
Long prompts make LLM inference costly, especially when chain-of-thought, in-context learning, and retrieved content are used. LLMLingua compresses prompts through budget control, iterative token selection, and distribution alignment, achieving state-of-the-art performance across four datasets with up to 20x compression and a 1.5-point performance drop.
Problem
Long prompts used in modern LLM applications increase the computational demands of inference, motivating compression that preserves essential information for black-box LLMs.
Method
LLMLingua combines a budget controller, iterative token-level compression, and instruction-tuning-based distribution alignment in a coarse-to-fine prompt-compression method.
Results
LLMLingua achieves state-of-the-art performance across four datasets, with up to 20x compression and only a 1.5 point performance drop.
Takeaways & Limitations
Prompt compression reduces computational costs and offers a potential solution for accommodating longer contexts.
Takeaways & Limitations
Performance can drop notably at excessively high compression ratios such as 25x-30x on GSM8K, and the upper limit varies by prompt length, task type, and sentence count.
Abstract
from arXiv · showhide
Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.
1 Introduction
Long prompts improve LLM applications but increase computational demands, motivating black-box prompt compression. LLMLingua combines coarse-to-fine compression and distribution alignment, achieving strong results across four application domains.
- Longer prompts from chain-of-thought, in-context learning, and retrieval increase the computational demands of LLM applications.
- Prompt compression reduces prompt length without gradient flow through the LLM, supporting applications using a broader range of accessible models.
- LLMLingua dynamically allocates compression ratios across instructions, demonstrations, and questions, then performs demonstration-level and token-level compression.
- Its instruction-tuning method aligns the small compressor language model’s distribution with that of the target LLM.
- LLMLingua achieves state-of-the-art performance across GSM8K, BBH, ShareGPT, and Arxiv-March23.
2 Related Work
Related work reduces LLM inference costs through model modification, prompt processing, demonstration selection, and token operations. LLMLingua builds on Selective-Context while targeting its independence and model-distribution limitations.
- Efficient-LLM methods modify model parameters through quantization, compression, instruction tuning, or delta tuning to reduce inference and fine-tuning costs.
- Prompt-oriented methods copy prompt tokens, learn special tokens, summarize inputs, or select demonstrations to reduce inference cost or prompt length.
- Several existing prompt-compression and token-processing methods require model fine-tuning or intermediate inference results, limiting their application scenarios.
- Selective-Context compresses prompts by dropping less informative lexical units identified through self-information from a small language model.
- This paper distinguishes its use of perplexity for prompt compression from out-of-distribution detection, where high perplexity indicates unreliable predictions.
3 Problem Formulation
The formulation represents prompt compression as producing a shorter prompt whose generated outputs remain similar to those from the original prompt. Lower compression rates correspond to lower inference costs.
- The original prompt consists of instruction, demonstration, and question components, while the compressed prompt contains their compressed representation.
- The compression rate τ is the compressed-token length divided by the original-token length, and the compression ratio is 1/τ.
- A smaller τ implies lower inference cost and is therefore preferable.
- The generated-token distribution from the compressed prompt is expected to be as similar as possible to that from the original prompt.
4 Methodology
LLMLingua uses a coarse-to-fine pipeline: a budget controller selects and budgets prompt components, iterative token compression preserves information under dependencies, and alignment reduces model-distribution discrepancy.
- Budget Controller: The budget controller assigns different compression ratios to instructions, demonstrations, and questions at sentence or demonstration level.
- Budget Controller: Instructions and questions receive more budget than demonstrations because they directly influence generated results, while demonstrations may contain redundant information.
- Budget Controller: It ranks demonstrations by small-language-model perplexity, selects them until the demonstration budget is reached, and allocates the remaining budget to instructions and questions.
- Budget Controller: At high compression ratios, sentence- or demonstration-level dropout preserves linguistic integrity better than token-level dropout alone.
- Iterative Token-level Prompt Compression: ITPC mitigates errors from conditional-independence assumptions by conditioning each segment’s compression on previously compressed content.
- Iterative Token-level Prompt Compression: The algorithm computes segment-level perplexity distributions, dynamically sets thresholds, and retains tokens whose perplexity exceeds each segment’s threshold.
- Distribution Alignment: Instruction tuning uses LLM-generated texts to align a pre-trained small language model with the target LLM’s distribution.
5 Experiments
Experiments across reasoning, in-context learning, conversation, and summarization evaluate compressed prompts under varied settings. LLMLingua generally outperforms prior methods, preserves performance at high compression, and reduces inference latency.
- 5.1 Settings: Experiments evaluate compressed prompts on GSM8K, BBH, ShareGPT, and Arxiv-March23 using Exact Match, BLEU, ROUGE, and BERTScore.GSM8K and BBH measure reasoning and ICL; ShareGPT measures conversation; Arxiv-March23 measures summarization.
- 5.2 Main Results: LLMLingua consistently outperforms prior methods across almost all experiments.The comparison includes GPT4-Generation and Random Selection baselines.
- 5.2 Main Results: 5x and 3x compression on GSM8K and BBH respectively achieves slightly higher results than full-shot prompting under the 1-shot constraint.These results support retention of reasoning information in compressed prompts.
- 5.2 Main Results: 9x and 3.3x acceleration ratios on ShareGPT and Arxiv-March23 accompany high BERTScore F1.The results indicate that semantic information from the initial prompts is retained.
- 5.2 Main Results: 51.55 and 24.53 performance improvements over zero-shot results remain at the largest compression ratios, while GSM8K 20x compression exceeds 8-shot 3-step CoT by 2.43.These findings concern preservation of in-context learning capacity and reasoning information.
- 5.4 Ablation: Removing iterative token-level compression causes a significant Exact Match decline, while budget control reflects differing sensitivity across prompt components.Instructions and questions require lower compression ratios, and conditional dependencies help preserve essential information.
- 5.5 Discussion: With Claude-v1.3, LLMLingua improves over the simple prompt by 0.8 and 1.7 EM points at compression ratios of 5x and 14x.The evaluation considers one-shot and half-shot constraints because of API cost limitations.
- 5.5 Discussion: Using Alpaca-finetuned GPT2-small instead of Alpaca-7B produces performance drops of 2.06, 0.99, and 1.06 EM points at different compression ratios.The authors attribute this to distribution discrepancy between the small LM and target LLM, while still reporting satisfactory difficult-task performance.
6 Conclusion
LLMLingua is a coarse-to-fine prompt compression method based on a small language model’s perplexity, combining three modules for black-box LLMs. Across four domains, it achieves state-of-the-art performance, up to 20x compression, and additional efficiency benefits.
- LLMLingua combines a Budget Controller, Iterative Token-level Compression, and Alignment for black-box LLM prompt compression.The method is based on the small LM’s perplexity.
- Across GSM8K, BBH, ShareGPT, and Arxiv-March23, LLMLingua achieves state-of-the-art performance.
- 20x compression is achieved with only a 1.5 point performance drop.
- LLMs can effectively restore compressed prompts, while compression also reduces generated text length.
- Prompt compression reduces computational costs, may accommodate longer LLM contexts, and can improve inference efficiency through KV-cache compression.
Limitations
The method has a practical compression boundary: performance drops substantially at extreme ratios, and the maximum useful ratio varies by prompt characteristics. Tokenizer differences can also underestimate prompt length.
- 25x-30x compression can cause a notable performance drop on GSM8K.
- All methods experience substantial performance degradation around 25x-30x, although LLMLingua’s drop is shifted to higher ratios.
- The upper compression limit varies with prompt length, task type, and the number of sentences.
- Differences between the small LM and black-box LLM tokenizers may underestimate prompt token length.
A.1 Dataset Details
The evaluation covers four datasets spanning reasoning, in-context learning, conversation, and summarization. The datasets include GSM8K, BBH, ShareGPT, and Arxiv-March23, with task-specific prompts and test constructions.
- GSM8K contains 8,000 math problems, including a 1,300-problem test set for arithmetic reasoning and language-based mathematical steps.The experiments use a complex multi-step chain-of-thought prompt.
- BBH comprises 6,500 problems across 23 language and symbolic reasoning subsets designed to evaluate chain-of-thought prompting.The experiments use a 3-shot chain-of-thought prompt.
- ShareGPT provides 575 multilingual conversations spanning coding, chitchat, and writing-assistant scenarios.All dialogue rounds except the final one form the prompt, with GPT-3.5-Turbo generating the reference.
- Arxiv-March23 contains 500 recent arXiv papers whose first five sections are truncated and concatenated for summarization evaluation.Each section is truncated to 10,000 characters, and GPT-3.5-Turbo supplies the reference summary.
A.2 Other Implementation Details
Experiments ran on a Tesla V100, using GPT2-Alpaca trained on Alpaca and GPT-3.5-Turbo with tiktoken for token counting.
- All experiments used a Tesla V100 with 32GB of memory.
- GPT2-Alpaca was trained on Alpaca for eight epochs with a 1e-4 learning rate and AdamW.Training took approximately 150 minutes.
- tiktoken and GPT-3.5-Turbo were used to count all tokens.
B Economic Cost
The section reports inference-cost savings from the approach across four datasets and describes black-box GPT-4 reconstruction of compressed prompts using guiding instructions.
- Economic Cost: $4.7, $8.0, $0.4, and $0.8 cost reductions were observed for GSM8K, BBH, ShareGPT, and Arxiv, respectively.The estimates use GPT-3.5-Turbo pricing.
- Economic Cost: Three compressed prompts with varying compression ratios, produced by distinct small language models on different datasets, were given to GPT-4 with guiding instructions.The examples illustrate black-box LLM reconstruction of compressed prompts.
E Cases Study
The case studies examine compressed prompts across reasoning, conversation, and scientific-text datasets, comparing compressed outputs with original-prompt and selective-context results. Examples also show GPT-4 reconstructing compressed prompts at multiple compression ratios.
- E Cases Study: A GSM8K example compresses a nine-step chain-of-thought prompt into a fragmented prompt that still includes a profit calculation and answer reasoning.The compressed example retains references to the box purchase, repackaging, sales, and total profit calculation.
- E Cases Study: 17x and 19x compression examples from GSM8K and 7x compression from BBH are reconstructed from compressed prompts using GPT-4.Alpaca-7B and GPT2-Alpaca serve as small language models in the GSM8K examples, while Alpaca-7B is used for BBH.
- E Cases Study: The case studies span GSM8K, BBH, ShareGPT, and Arxiv-March23 under half-shot, quarter-shot, 2x, and 200-token constraints.The section presents examples from multiple datasets and scenarios.
- E Cases Study: Additional examples cover BBH boolean reasoning, TOPAS radiation-beam spectra, and redshift drift in a universe with structure.These examples extend the case studies beyond GSM8K arithmetic and include scientific prompts.