PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Compress Prompt in Natural Language Formats

Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, Xia Hu

arXiv:2402.18700v2cs.CLcs.AIcs.LG

TL;DR

Long contexts challenge LLM understanding and efficiency, and soft-prompt compression has limited transferability across models. Nano-Capsulator compresses prompts into natural-language Capsule Prompts using reward-based semantic preservation and length constraints. It reports shorter prompts, lower latency and budget overheads, and transferability across LLMs and similar unseen datasets.

  • Problem

    Long contexts impair LLMs’ understanding of lengthy documents and inference efficiency, while natural-language prompts are difficult to optimize with gradients and strict length constraints.

    Source: §1 · pp. 1–2

  • Method

    Nano-Capsulator summarizes prompts into natural-language Capsule Prompts, using reward scores with semantic-preservation objectives and length constraints.

    Source: §1 · p. 2, §3.1.1 · p. 3

  • Results

    81.4% shorter prompts, up to 4.5× lower inference latency, and 80.1% lower budget overheads are reported alongside strong transferability across LLMs and similar unseen datasets.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Capsule Prompts can be applied to unseen datasets without retraining the compressor when downstream tasks have similar domains.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.

1 Introduction

Long contexts impair LLMs’ document understanding and inference efficiency, while soft-prompt compression has limited transferability across models. Nano-Capsulator compresses prompts into natural-language Capsule Prompts, aiming to preserve utility and transferability while reducing length, latency, and cost.

1 Introduction

  • Long contexts impair LLMs’ ability to understand lengthy documents and reduce inference efficiency as demand for million-token processing grows.The introduction frames efficient long-context processing as increasingly important for cost-conscious deployment.Source: §1 · p. 1
  • Natural-language prompt compression is difficult because discrete text is incompatible with back-propagation and strict length limits can degrade performance.Unlike soft prompts, NL prompts are difficult to optimize with fixed lengths.Source: §1 · p. 2
  • Nano-Capsulator uses reward scores and an explicit semantic-preservation objective to produce shorter NL Capsule Prompts while retaining utility and transferability.The framework is also described as applicable without further training to unseen datasets with similar downstream-task domains.Source: §1 · p. 2
  • Experiments cover few-shot demonstration CoT and reading-comprehension passage prompts, with transferability across LLMs and similar unseen datasets.The reported adaptation to unseen datasets does not require retraining the prompt compressor.Source: §1 · p. 2
  • 81.4% of original prompt length, up to 4.5× lower inference latency, and 80.1% lower budget overheads are reported for Capsule Prompts.The framework is reported to retain prompt utility and transferability across diverse LLMs.Source: §1 · p. 2

2 Related Work

Prior compression work includes trainable soft prompts and context filtering. The former is constrained in cross-model transfer, while the latter is described as lacking downstream-task information.

2.1 Soft Prompt Compression

  • Soft-prompt methods compress hard prompts into trainable vectors, using distillation, LLM summarization, or a learned prefix predictor [Wingate et al., 2022; Chevalier et al., 2023; Mu et al., 2023].The summarization approach segments prompts and sequentially compresses them into soft-prompt fragments.Source: §2.1 · p. 2
  • Soft prompts are tailored to a specified LLM and require retraining when the target model changes, limiting transferability, especially for API-based LLMs.The passage names Claude2 and PaLM as examples.Source: §2.1 · pp. 2–3

2.2 Context Distillation for Compression

  • Context-distillation methods shorten prompts by filtering words with lower self-information or perplexity scores [Li et al., 2023; Jiang et al., 2023].These methods do not use downstream-task information, and the passage reports inferior performance when applied to downstream tasks or similar unseen datasets.Source: §2.2 · p. 3

3 Long Prompt Encapsulation

Nano-Capsulator compresses long prompts into natural-language Capsule Prompts while combining semantic preservation, downstream-task utility rewards, and length constraints. Its training aligns these objectives, and inference produces a compressed prompt from the long input.

3.1 Prompt Encapsulation

  • Nano-Capsulator jointly optimizes natural-language prompt compression and utility preservation to produce Capsule Prompts that meet a length constraint while retaining the original prompt’s utility.The framework combines semantics-preserving summarization with reward scores.Source: §3.1 · p. 3, §3 · p. 3
  • Training repeatedly generates Capsule Prompts, samples downstream questions, obtains rewards from a frozen LLM, and updates the compressor until convergence.The algorithm takes the original long prompt and compression instructions as inputs and returns natural-language Capsule Prompts.Source: §3.1 · p. 3

3.1.1 NL-formatted Prompt Compression

  • Semantic loss compares hidden-state embeddings of the original prompt and its compressed summary; the study uses mean square error to measure their similarity.The preserved semantics include few-shot CoT logical thinking and beneficial content in context passages.Source: §3.1.1 · p. 3

3.1.2 Prompt Utility Preservation

  • The utility reward compares downstream-task scores using the original prompt and a length-truncated Capsule Prompt, with sampled training questions informing the reward.The study uses hidden-state embedding mean square error and notes that accuracy or GPT4Eval Scores can also serve as reward metrics.Source: §3.1.2 · p. 4, Figure 2 · p. 4

3.1.3 Compression with Reward

  • Low reward scores increase the semantic-loss penalty, discouraging compressed prompts that have inferior performance.The reward is synchronized with semantic loss to maintain utility during optimization.Source: §3.1.3 · p. 4

3.2 Algorithm of Nano-Capsulator

  • At inference, Nano-Capsulator only needs to generate the compressed Capsule Prompt from the provided long input prompt.Source: §3.2 · p. 4

4 Experiments

The experiments assess Nano-Capsulator’s efficacy and transferability, utility preservation, and inference latency across reasoning and reading-comprehension tasks. They compare transfer across LLMs and unseen datasets, examine prompt length and text-dropping alternatives, and measure latency.

4.1 Dataset

  • The evaluation covered few-shot CoT reasoning on CSQA and GSM8K and reading comprehension on MultiRC and TriviaQA-Long.TriviaQA-Long paragraphs exceeding 2,000 tokens were excluded, leaving an average length of 900 tokens.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Experiment Settings

  • Model-transfer tests trained the compressor on one downstream LLM and evaluated it on different LLMs; data-transfer tests used unseen datasets with the same downstream task.Same-dataset model-transfer evaluation used a 70/30 training-and-validation versus test split.Source: §4.2 · p. 5

4.3 Main Results (RQ1)

  • Capsule Prompt achieved up to 81.4% compression rate and 80.1% Claude2 API-cost savings, mostly retained original performance, and nearly matched uncompressed MultiRC performance.The primary experiments imposed dataset-specific maximum lengths: 150 tokens for CSQA and MultiRC, 350 for GSM8K, and 500 for TriviaQA-Long.Source: Table 1 · p. 6, §4.3 · p. 6
  • Nano-Capsulator trained on MultiRC and tested on BoolQ without further training achieved competitive performance, with a slight accuracy drop versus its training-version Capsule Prompt under Vicuna-13B and Claude2.BoolQ is an unseen dataset with the same downstream reading-comprehension task.Source: Figure 3 · p. 6, §4.3 · p. 6

4.4 Contributions of Utility Preservation (RQ2)

  • Capsule Prompt outperformed Vicuna-7B zero-shot summarization across evaluated scenarios and GPT-3.5-Turbo summarization on CSQA and GSM8K; removing the reward degraded performance, while GSM8K cases preserved complete logical structures.The comparisons included Vicuna-13B and Claude2; the reward ablation used TriviaQA-Long.Source: §4.4 · p. 7

4.5 Exploration of Impact Factors (RQ3)

  • Capsule Prompt outperformed random demonstration elimination on CSQA and Selective Context on CSQA and MultiRC, particularly at similar prompt lengths.Selective Context removes words according to self-information values.Source: Figure 7 · p. 7, §4.5 · p. 8
  • Capsule Prompt length affected utility: 150 tokens performed best under Claude2 and 200 tokens under Vicuna-13B, while longer prompts could add noise or misinformation.Nano-Capsulator and zero-shot summarization showed similar performance trends.Source: §4.5 · pp. 7–8

4.6 Latency of Nano-Capsulator (RQ3)

  • Capsule Prompt reduced inference latency by 2.1×–4.5× while mostly retaining original performance; OPT-2.7B could also accommodate batch size 16 when the original prompt ran out of memory.Maximum speedups were 4.5× for OPT-2.7B and 4.1× for Vicuna-13B as batch size increased.Source: Figure 9 · p. 8, Figure 8 · p. 8, §4.6 · p. 8

5 Conclusion

Nano-Capsulator compresses long prompts while preserving essential information. The reported results include substantial reductions in prompt length, inference latency, and budget overhead, with almost identical accuracy and relevance.

5 Conclusion

  • Prompt length falls by 81.4%, inference latency decreases by up to 4.5×, and budget overheads fall by 80.1%, with almost identical accuracy and relevance.Source: §5 · p. 8
  • Nano-Capsulator compresses long prompts for language models while preserving essential information.Source: §5 · p. 8
  • Future work will refine Nano-Capsulator for broader domain applications and explore its use in data-intensive fields.Source: §5 · p. 8

A Computation Infrastructure

The experiments use physical computing infrastructure, reported for fair comparison of testing algorithmic throughput.

A Computation Infrastructure

B Additional Experiments of Comparison to Soft Prompt Baselines

The comparison with AutoCompressors evaluates prompt compression on GSM8K using the same pretrained model. The paper reports a substantial performance drop for AutoCompressor, while Capsule Prompt's PaLM cost comparison reports savings of up to 80.1%.

B Additional Experiments of Comparison to Soft Prompt Baselines

C API Cost of PaLM

Capsule Prompt reduces PaLM evaluation costs by up to 80.1% compared with the original prompt.

C API Cost of PaLM

D Training Costs of Nano-Capsulator

Nano-Capsulator training costs vary with the volume and types of training data. After training, compressed hard prompts can be derived with a single forward pass.

D Training Costs of Nano-Capsulator

E The Case Studies of Capsule Prompt

Case studies on GSM8K and MultiRC report that Nano-Capsulator better obtains semantic meanings of math logic from the original CoT prompt. Without Nano-Capsulator, Zero-shot Summarization struggles to compress long prompts effectively.

E The Case Studies of Capsule Prompt

F Instruction Usage in Inference LLMs

E The Case Studies of Capsule Prompt

  • Figure 10 presents a GSM8K case study comparing Capsule Prompt with an in-context summarization prompt generated by Vicuna-7B.The figure follows the RQ2 settings.Source: Figure 10 · p. 11

F Instruction Usage in Inference LLMs

E The Case Studies of Capsule Prompt

  • Figure 11 presents a MultiRC case study comparing Capsule Prompt with an in-context summarization prompt generated by Vicuna-7B.The figure follows the RQ2 settings.Source: Figure 11 · p. 12
Loading 2402.18700v2…