Learning to Compress Prompts with Gist Tokens

Jesse Mu; Xiang Lisa Li; Noah Goodman

Source-linked AI summary

Learning to Compress Prompts with Gist Tokens

Jesse Mu, Xiang Lisa Li, Noah Goodman

arXiv:2304.08467v3Apr 17, 2023cs.CL

TL;DR

Prompting preserves multitask ability but repeatedly encoding prompts is inefficient, while task-specific adaptation requires retraining for each prompt. Gisting learns to compress prompts into reusable gist-token activations through modified attention masks, achieving up to 26x compression with up to 40% FLOPs reduction and 4.2% wall-clock speedups while retaining similar output quality. The main trade-off is some loss of instruction nuance, requiring evaluation before deployment.

Problem
Prompting is computationally inefficient for repeated or long prompts, while finetuning and distillation require retraining the model for each task.
Method
Gisting trains an LM to predict compact gist-token activations from prompts, using modified Transformer attention masks during instruction finetuning.
Results
Up to 26x prompt compression, up to 40% FLOPs reduction, and 4.2% wall-clock speedups were achieved while maintaining output quality similar to original models.
Takeaways & Limitations
Gisting enables prompt activations to be compressed, cached, and reused, including caching substantially more prompts and supporting encoder-decoder prompt caching.
Takeaways & Limitations
Compression can lose nuance from the original instruction, so practitioners should evaluate the compute-accuracy trade-off for their use case before deployment.

Abstract

from arXiv · show

Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

1 Introduction

Prompting preserves multitask capabilities but repeatedly encoding long prompts is costly, while task-specific finetuning is efficient yet requires retraining for each prompt. Gisting addresses this trade-off by compressing prompts into reusable activations, achieving substantial compression and efficiency gains with similar output quality.

Motivation: Prompt encoding is computationally inefficient because self-attention scales quadratically with prompt length, while repeated prompts also consume memory and storage.Caching activations prevents some recomputation but still incurs growing memory and storage costs.
Motivation: Finetuning or distillation can remove prompt-conditioning costs but requires retraining the model for each new prompt.This limits reuse across changing prompts or tasks.
Gisting: Gisting compresses arbitrary prompts into smaller Transformer activations on virtual gist tokens that can be cached and reused.Unlike task-specific prefix learning, it predicts gist prefixes zero-shot from prompts and can generalize to unseen instructions.
Learning gisting: Gist models can be learned through instruction tuning with modified attention masks and no additional training cost.The masks encourage prompt compression while the model learns instruction following.
Results: 26x prompt compression was achieved on LLaMA-7B and FLAN-T5-XXL while maintaining output quality similar to the original models in human evaluations.The reported efficiency effects include up to 40% FLOPs reduction and 4.2% latency speedups.

2 Gisting

Gisting amortizes prompt inference and task-specialization costs by predicting compact gist activations for new prompts. It can be viewed as meta-context distillation, where a predictor produces compressed representations without per-task optimization.

Instruction finetuning: Instruction-following models condition on a natural-language task prompt t, optional input x, and desired output y.At inference, the model receives a novel task and input and decodes its prediction.
Motivation: Long prompts are costly because Transformer context windows are limited and attention complexity grows quadratically with input length.These costs are especially relevant when prompts are repeatedly reused.
Alternatives: Task-specific finetuning is faster without prompt conditioning but requires storing task-specific weights and collecting input-output data for each task.Parameter-efficient methods reduce adaptation cost but do not remove these task-specific requirements.
Gisting: Gisting learns a function G(t) that produces a shorter gist prefix whose activations induce similar behavior and can be cached for reuse.The predictor is intended to generalize to unseen tasks without additional training.
Meta-context distillation: The meta-distillation objective trains gist predictions across a distribution of tasks rather than distilling a single task separately.This replaces per-task gradient-based distillation with prediction of gist representations from each prompt.

3 Learning Gisting by Masking

The paper learns gisting by inserting virtual gist tokens between prompts and inputs, then modifying attention masks so later tokens must rely on the gist prefix rather than the original prompt. This preserves standard instruction-tuning cost while enforcing compression.

Training setup: Gisting uses the LM itself as the gist predictor, allowing prompt compression to be learned through standard instruction finetuning with modified attention masks.The method incurs no additional training cost over standard instruction finetuning.
Gist tokens: The input sequence places k copies of a learned gist token between the task prompt t and input x.The gist token is shared, while its activations differ across positions.
Compression mechanism: Masking prevents tokens after the gist prefix from attending to prompt tokens before it, forcing prompt information into gist activations.Input and output tokens can still attend to the gist tokens.
Architecture-specific masks: Decoder-only models mask the lower-left portion of the causal attention triangle to block post-gist access to the prompt.Encoder-decoder models additionally block input access to the prompt and decoder cross-attention to prompt representations.
Implementation: The masking changes require roughly 10 source lines and can serve as a drop-in replacement for attention masking in common deep-learning libraries.The implementation is described as simple and compatible with libraries such as Hugging Face Transformers.

4 Experiments

Experiments evaluate gisting across architectures, task splits, compression baselines, and automated and human-assisted metrics. The setup emphasizes generalization across diverse instructions and comparison with positive, negative, and discrete-compression controls.

Data: Alpaca+ contains 130,321 examples and 104,664 unique tasks, combining Self-Instruct and Stanford Alpaca instruction-tuning data.Its broad task variety is intended to support generalization of gist models.
Data: Approximately 59% of Alpaca+ tasks have no inputs, but these examples still provide training signal for prompt compression.The dataset is noisy and imperfect, while prior work reports comparable performance to the source models.
Evaluation splits: Validation uses Seen, Unseen, and Human splits, with the Human split providing a stronger OOD challenge because its prompts average about 26 tokens versus about 20 in training.The Unseen and Human splits test generalization to unseen instructions.
Baselines: TF-IDF compression replaces each instruction with a keyword-derived subword token, producing compression rates comparable to a single gist token.The approach is evaluated as a discrete-compression baseline after standard instruction finetuning.
Metrics: Evaluation combines ROUGE-L, ChatGPT pairwise judgments, and human ratings of selected models on a subset of Human-split examples.ROUGE-L uses lexical overlap, while ChatGPT and human evaluation compare response quality against the positive control.

5 Results

Gist models generally preserve output quality across seen, unseen, and out-of-distribution prompts, with performance varying by model and evaluation setting. Human evaluations corroborate the broader results, while failure cases include copying-specific instructions and runaway generations.

Gist-token count: Models are generally insensitive to gist-token count, while ten gist tokens can hurt LLaMA-7B performance, so the experiments use single-token models thereafter.Figure 3 reports ROUGE-L and ChatGPT scores for LLaMA-7B and FLAN-T5-XXL across different gist-token counts.
Model performance: Single-token gist models remain competitive across Seen, Unseen, and OOD Human prompts, though performance drops most on the challenging Human split.Win rates against positive controls are 48.6% and 50.8% on Seen prompts, 49.7% and 46.2% on Unseen prompts, and 45.8% and 42.5% on Human prompts for LLaMA-7B and FLAN-T5-XXL, respectively.
Compression baselines: Gist compression substantially outperforms TF-IDF compression, whose models only marginally improve over negative controls across the evaluated settings.The comparison uses single-token gist, TF-IDF, and positive/negative control models.
Human evaluation: 52.3% and 40.6% average human win rates, versus 48.0% and 42.0% for positive controls, broadly align with ChatGPT-based comparisons for LLaMA-7B and FLAN-T5-XXL.Human annotators show pairwise Cohen’s κ of .24 and .33, while ChatGPT’s average κ with the three annotators is .29 for both models.
Output matching: Gist outputs exactly match positive-control outputs nearly 50% of the time on Seen tasks, declining to approximately 20–25% on Unseen tasks and 10% on OOD Human tasks.The matching rates measure exact output identity rather than broader semantic agreement.
Failure cases: Failure cases include instructions requiring verbatim copying and runaway generations, with the latter appearing more often in gist models than in positive controls.Examples include an incorrect category label for a news title and repetitive LaTeX generation extending for hundreds of tokens.

6 Compute, Memory, and Storage Efficiency

Gist caching reduces inference computation and storage by replacing full prompts with compressed gist prefixes, with benefits depending on the caching baseline and model. It also expands effective prompt-cache capacity for heavily reused prompts.

The evaluated caching strategies are no caching, instruction caching of uncompressed prompt activations, and gist caching of the compressed prefix G(t).Instruction caching is applicable only to decoder-only models because bidirectional encoder representations depend on the input.
40% FLOPs savings and 4-7% lower wall time are reported for gist caching versus unoptimized models across both evaluated models.These profiles average a single decoding step across 252 Human instructions.
Compared with full-instruction caching on LLaMA-7B, gist caching yields a 0.11% FLOPs decrease and a 1% wall-time speedup.The smaller compute improvement reflects that forward-pass FLOPs are dominated by processing new input tokens rather than KV-cache self-attention.
Gist caching compresses 26 tokens into 1, increasing the number of prompts cacheable within the same storage by up to 26x.For LLaMA-7B, each KV-cache token requires 1.05 MB, although its contribution was negligible at the tested prompt lengths.

7 Additional Related Work

The paper positions gisting as a simple zero-shot approach to adapting language models and as a prompt-compression variant related to memory and sparse-attention methods.

Adapting LMs without Backprop.: Gisting predicts task prefixes from language instructions alone, making it a zero-shot analogue of few-shot prefix prediction in HyperTuning.Unlike HyperTuning, the language model itself acts as the HyperNetwork, requiring only a small attention-mask change.
Compression and memory in transformers.: Gisting relates to Compressive Transformer work by storing reduced representations, while differing in how prompt compression is performed.The supplied passage introduces this connection but truncates before specifying all three differences.
Sparse attention mechanisms.: Gisting draws on sparse-attention ideas but is specifically oriented toward caching arbitrary input portions with varying prompt lengths.The supplied passage contrasts this requirement with more general sliding-window attention methods.

8 Discussion and Limitations

The paper concludes that gisting provides efficient prompt compression with little reported quality loss, while recognizing that compression can remove instruction nuance and requires careful deployment evaluation.

Up to 26x compression of unseen OOD prompts and up to 40% FLOPs reduction are reported, alongside 4.2% wall-clock speedups over unoptimized models.The conclusion also reports that gisting enables caching an order of magnitude more prompts than full instructions.
Gisting’s attention-mask implementation adds no cost over standard instruction finetuning and supports prompt caching in encoder-decoder models.The paper presents gisting as modified instruction finetuning or meta-context distillation.
Compression necessarily causes some loss of nuance, including a concrete failure case, so practitioners should evaluate the compute–accuracy trade-off before deployment.The paper emphasizes safety and robustness evaluation because language-model behavior on edge cases is poorly understood.
Future work includes retrofitting frozen language models with smaller prompt-compressor models and targeting very long prompts such as large-k-shot prompts.The masking method can also be integrated into existing instruction-finetuning workflows.

B Data, Training, Evaluation, and Compute Details

The appendix documents released code, sequence-length limits, training configurations, decoding, hardware, and the hyperparameter table used for the experiments.

Implementation and resources: Code, data, and model checkpoints are available in the project repository.
Data: LLaMA-7B uses maximum sequence lengths of 512 tokens, or 768 for Human validation, while FLAN-T5-XXL uses input/output limits of 128/256, or 384/384 for Human.Examples exceeding the LLaMA limits are truncated from the end.
Training: Training hyperparameters follow published Alpaca and Tk-INSTRUCT configurations, with a 5e-5 learning rate used for FLAN-T5-XXL.
Training: LLaMA-7B training runs use 3000 steps and FLAN-T5-XXL runs use 16000 steps, corresponding to approximately three and two epochs, respectively.The reported training setup saw no substantial gains beyond one epoch for FLAN-T5-XXL.
Evaluation: Evaluation uses greedy decoding, with limited gains from beam search at beam size B = 4.
Compute: Experiments use 4x A100-SXM4-80GB GPUs, 480GB RAM, 16 CPUs, PyTorch 2.0, Hugging Face Transformers, and DeepSpeed.Benchmarking uses one A100 GPU; training takes approximately 7 hours for LLaMA-7B and 25 hours for FLAN-T5-XXL.
Hyperparameters: Table A.1 reports the hyperparameters for the training runs.
Implementation: The sample implementation defines PyTorch functions for cumulative sums and masks that select tokens around gist-token positions.

C Details of ChatGPT Evaluation

The ChatGPT evaluation compares two assistant responses for instruction-following and factual accuracy, using structured judgments with randomized presentation. The appendix documents its prompts, examples, model-order controls, agreement measures, and collection procedure.

ChatGPT selected the more helpful response by judging instruction compliance and factual accuracy, with ties allowed when outputs were equal or equally poor.
The evaluation prompt included examples favoring factual accuracy, instruction-following despite imperfections, and ties when both responses were wrong.
Weighted Cohen’s κ with linear weights was used because judgments formed an ordinal response scale.
Model presentation order was randomized between Assistant A and Assistant B for each query to reduce order effects.
Approximately 22.5k ChatGPT judgments were collected at an estimated cost of $29.28.

D.1 Experimental Details

The human evaluation sampled 100 Human validation examples and recruited three English-fluent annotators per example through Prolific. Participants received task-comprehension checks, and agreement with ChatGPT was assessed using weighted Cohen’s κ.

100 randomly selected Human validation examples were each rated by three US- or UK-based, English-fluent Prolific annotators.
Participants completed two example items correctly before proceeding to the main task, and annotation costs were estimated at USD $141.64.
Weighted Cohen’s κ with linear weights measured agreement because the response categories were ordinal.

E Exact Match Results

The appendix describes exact-match and FLOPs analyses for compressed prompts. It also shows that KV-cache-dependent operations account for a small share of layer computation, limiting compute gains from cache reduction despite memory and storage benefits.

Figure A.3 measures the rate at which one-token gist models exactly match positive-control outputs, with 95% exact binomial confidence intervals.
The FLOPs analysis modifies Transformer equations to account for self-attention to a KV cache by defining seq_len_with_past = seq_len + kv_cache_len.
Only three attention-layer operations depend on KV-cache size, and they contribute at most about 10% of total attention-layer FLOPs in the illustrated setting.
A 26x KV-cache compression rate predicts a 0.12% relative improvement, close to the 0.11% improvement observed in Table 3.

G Additional Examples of Model Outputs

The examples compare positive-control, gist, TF-IDF, and negative outputs across tasks and model architectures, alongside human and ChatGPT judgments. They illustrate both cases where gist matches or exceeds the control and cases where it loses detail or correctness.

In the Twitter-bio example, LLaMA-7B human judgments included ties and gist wins, while ChatGPT preferred the positive control because the gist output went off-topic.
For the France-cities task, ChatGPT preferred the gist output when it listed more cities and attractions, whereas human judgments included positive-control wins, ties, and gist wins.
In the aphorism task, LLaMA-7B gist completed the quote correctly and won human and ChatGPT comparisons against the positive control.
For FLAN-T5-XXL’s aphorism example, neither output was correct, so ChatGPT judged the comparison a tie.
In the HTML task, the positive-control output supplied valid image syntax while the gist model produced no output, leading ChatGPT to prefer the positive control.