Prompt Compression for Large Language Models: A Survey

• Longer task prompts increase memory costs, reduce inference speed, and can negatively affect user experience.Prompt-centric methods lower input complexity, typically with minimal or no changes to LLM parameters.Source: §1 · p. 1
• Hard methods filter or paraphrase natural-language prompts, while soft methods compress them into special tokens with different attention patterns.In the illustration, compressed soft tokens attend to prior input, while subsequent outputs cannot attend to tokens before the compressed token.Source: Figure 1 · p. 1
• The survey reviews hard and soft methods, their mechanisms and downstream adaptations, and challenges and potential solutions.It focuses specifically on prompt compression rather than general efficient prompting or reasoning prompts.Source: §1 · p. 2
• The survey interprets hard methods through filtering or paraphrasing and soft methods through attention modification, PEFT, modality integration, or synthetic language.It identifies overfitting and performance degradation, high compression latency, and the need to compare against attention optimization; proposed directions include encoder optimization, combining hard and soft methods, and multimodal insights.Source: §1 · p. 2

• Prompts typically combine an instruction or question with task input or context, and compression methods may target some or all components.Instruction-input-output structures appear in instruction fine-tuning datasets, while context-question-answer structures are common in QA and reading comprehension.Source: §2 · p. 2
• Hard prompts use natural-language tokens and are interpretable, but ambiguity, authoring effort, and wording variation can affect utility and performance.Prompts may be written by humans or generated by models.Source: §2 · p. 2
• Soft prompts are trainable continuous vectors that can express meanings beyond discrete vocabulary tokens, but are less human-readable and require more computation as datasets grow.The passage says they are expected to help across tasks when tuned on diverse datasets.Source: §2 · p. 2
• Prompt compression reduces prompt length by removing low-information content or encoding prompt information as continuous representations.Hard methods retain natural-language tokens and may generalize across embedding configurations; soft methods produce latent vectors that people cannot understand directly.Source: §2 · pp. 2–3

• SelectiveContext deletes redundant or less informative prompt content using self-information, grouping tokens into noun phrases to preserve coherence.It uses SpaCy dependency parsing and does not rely on external models or additional parameters, so it can apply across model architectures.Source: §3 · p. 3
• SelectiveContext depends on accurate SpaCy phrase-boundary detection and currently lacks a method for merging verb phrases.Re-encoding filtered prompts in the LLM also limits potential efficiency gains.Source: §3 · p. 3
• LLMLingua scores content with a smaller language model, selects demonstrations by perplexity, then filters tokens while preserving key numbers and units.Its token-level filtering can split words into subword units and avoids noun-phrase merging; reported compression ratios reach 20x.Source: §3 · pp. 3–4
• LLMLingua's external language model adds memory requirements and may use a tokenizer different from the target LLM.The passage also cautions that prompt compression should be distinguished from in-context example selection when demonstrations are not abundant.Source: §3 · p. 4
• Nano-Capsulator paraphrases prompts into fluent, concise text using semantic-preservation loss and a reward function for downstream utility.Its fine-tuned Vicuna-7B compressor operates independently of the LLM, but adds memory and inference costs.Source: §3 · p. 4
• Later hard-prompt methods extend compression through longer windows, query-adaptive document selection, distillation, reinforcement learning, and embedding-based relevance or summarization.Examples include LongLLMLingua, AdaComp, LLMLingua-2, PCRL, TACO-RL, CPC, and TCRA-LLM.Source: §3 · p. 4

4.1 Architectures

• CC trains a shorter soft prompt to approximate a natural-language prompt’s output distribution, but each prompt requires its own training.It minimizes KL divergence across token sequences; contrastive contexts can train prompts for attributes such as sentiment.Source: §4.1 · p. 4
• GIST restricts generated tokens to attending to compressed tokens and can compress unseen prompts without additional fine-tuning, but its use is limited by training prompt length and model compatibility.Its compressed tokens cannot be used with the original untuned LLM; AutoCompressor recursively compresses sub-prompts and handles contexts up to 30,720 tokens, but is time-consuming to train and also requires a tuned LLM.Source: §4.1 · pp. 4–5
• ICAE compresses up to 512 context tokens into 32, 64, or 128 tokens at 4x–16x, and concatenated groups support contexts up to 5,120 tokens.Its frozen decoder allows compressed tokens to be used with the original LLM, while Pile overlap raises potential data-leakage concerns.Source: §4.1 · p. 5
• 1–16 tokens compress 96–480 tokens at 6x–480x, retaining 62.26%–72.89% of uncompressed-prompt capabilities with 500xCompressor.Its ArxivQA test set uses abstracts from January to April 2024 to evaluate on strictly unseen data.Source: §4.1 · p. 5
• xRAG compresses information into a single token for QA, but its selected embedding model requires loading two LLMs and a projector.ICAE and 500xCompressor instead require one LLM and LoRA parameters.Source: §4.1 · p. 5
• UniICL compresses demonstrations while leaving the query unchanged; its frozen shared encoder and decoder use a trainable projector, reducing gradient computation and LLM-loading memory.Its compressed tokens can serve as embeddings for in-context examples, avoiding an additional embedding process during example selection.Source: §4.1 · p. 5

4.2 Insights

• Soft prompt compression limits generation-time attention to compressed tokens; ICAE-like embeddings resemble prompt tuning, while 500xCompressor’s key-value inputs resemble prefix tuning and retain more detail at high compression ratios.The survey notes that the extra parameters in key-value values may increase overfitting risk; embeddings and key-value values are both generated by the encoder.Source: §4.2 · p. 6
• Compressed text can be viewed as a rich-information modality and as a more efficient language for LLMs, with tokens that can be saved, transferred, and interpreted in response to changing inputs.Because text is more information-dense than images, prompt compression requires greater precision and carries greater potential information loss than image encoding.Source: §4.2 · pp. 6–7

5 Downstream Adaptions

• Prompt compression is used in general QA, RAG, ICL, role playing, agent-based systems, and interdisciplinary tasks.In general QA, methods compress instructions or contexts for instruction-following tasks.Source: §5 · p. 7
• In RAG, xRAG answers questions using embeddings encoded from retrieved documents rather than processing the full text.The passage also identifies general QA as an application domain.Source: §5 · p. 7
• UniICL compresses in-context examples to fewer embedding tokens to help select relevant examples, while agent systems can compress API documentation for more efficient tool use.Source: §5 · p. 7

6.1 Current Challenges

• Fine-tuning can cause overfitting and degraded generalization; soft-prompt encoders also need large, diverse training data and retraining when their decoder LLM is updated.Filtered hard prompts may also disrupt grammatical correctness and create an unfamiliar input distribution.Source: §6.1 · p. 7
• Compression can consume substantial time and memory, so efficiency gains may be limited to generation and may be small for short-output tasks.Hard prompts may require extra token-selection models and re-encoding; large soft-prompt encoders also increase compression time.Source: §6.1 · p. 7
• Prompt compression lacks comparison with attention optimization, despite attention methods avoiding encoder memory costs and applying the same attention mechanism to input and generated tokens.The survey calls for comparing compression ratios under equivalent computational requirements.Source: §6.1 · pp. 7–8

6.2 Future Directions

• Smaller encoders such as BERT may encode semantics effectively while accelerating compression; the survey also suggests trying additional PEFT methods for encoder fine-tuning.Current soft-prompt encoders can approach decoder size, making compression time comparable to processing the original input.Source: §6.2 · p. 8
• Hard methods filter unnecessary tokens while soft methods represent text with a few special tokens; combining their orthogonal mechanisms may raise compression ratios but can reduce encoding efficiency.Their compression times cannot be parallelized.Source: §6.2 · p. 8
• The survey proposes adapting multimodal ideas to prompt compression, including cross-attention and image-text representation alignment, which remain unexplored or adaptable approaches.Current soft-prompt methods use self-attention to transfer information from natural-language prompts to compressed tokens.Source: §6.2 · p. 8

• The survey reviews prompt compression methods from the perspectives of hard and soft prompts.Source: §7 · p. 8
• It discusses technical approaches, ways to understand compression, current challenges, and potential future directions.Source: §7 · p. 8
• The conclusion highlights comparison with attention optimization and insights from multimodal LLMs as relevant to prompt compression.Source: §7 · p. 8

• The paper focuses specifically on prompt compression, not the full range of efficient prompting methods or LLM efficiency techniques.Source: Limitations · p. 9
• It offers detailed explanations and insights within that narrower topic.Source: Limitations · p. 9
• The paper says prompt-compression methods should be used according to their guidelines and copyright restrictions.Source: Limitations · p. 9

• The research involved no studies with human participants or animals.Source: Ethics Statement · p. 9
• The authors state that ethical approval was therefore not required.Source: Ethics Statement · p. 9
• The authors report that data and materials were collected consistently with ethical guidelines and that no ethical concerns were present.Source: Ethics Statement · p. 9

A Appendix

• Table 1 summarizes prompt-compression methods and applications and supports locating reference papers.The survey says Figure 2 was translated into Table 1 to provide an interactive way to locate references.Source: A Appendix · p. 12, Table 1 · p. 13
• Table 2 presents QA-item examples in the {Instruction, Input, Output} and {Context, Question, Answer} formats.Source: Table 2 · p. 13
• Figures 3 and 4 compare architectures for prompt-compression models using hard and soft prompt methods, respectively.Source: A Appendix · p. 12, Figure 3 · p. 14, Figure 4 · p. 14