PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Former: Lightning-fast Compressing Context for Large Language Model

Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, Enhong Chen

arXiv:2406.13618v2cs.CL

TL;DR

Long-context compression can reduce LLM inference costs, but existing compressors based on self-attention have quadratic time complexity. IC-Former instead uses cross-attention and learnable digest tokens; at 4x compression it uses 1/32 of the baseline’s floating-point operations and compresses 68 to 112 times faster while maintaining over 90% of baseline performance on evaluation metrics.

  • Problem

    Existing LLM-based compressors use self-attention whose quadratic time complexity as context lengthens limits efficient compression of long contexts.

    Source: §1 · pp. 1–2

  • Method

    IC-Former uses cross-attention and learnable digest tokens to condense contextual embeddings into digest vectors that replace the original context.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    At 4x compression, IC-Former uses 1/32 of the baseline’s floating-point operations and compresses 68 to 112 times faster while maintaining over 90% of baseline performance on evaluation metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that IC-Former reduces time and space complexity while preserving contextual semantics, supporting broader applications requiring extensive context.

    Source: §5 · p. 9

    IC-Former has not surpassed the baseline’s performance in downstream tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach is to compress the long input contexts. Existing methods typically leverage the self-attention mechanism of the LLM itself for context compression. While these methods have achieved notable results, the compression process still involves quadratic time complexity, which limits their applicability. To mitigate this limitation, we propose the In-Context Former (IC-Former). Unlike previous methods, IC-Former does not depend on the target LLMs. Instead, it leverages the cross-attention mechanism and a small number of learnable digest tokens to directly condense information from the contextual word embeddings. This approach significantly reduces inference time, which achieves linear growth in time complexity within the compression range. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times while achieving over 90% of the baseline performance on evaluation metrics. Overall, our model effectively reduces compression costs and makes real-time compression scenarios feasible.

1 Introduction

IC-Former compresses long contexts into compact digest vectors using cross-attention rather than costly contextual self-attention. It reduces compression cost while retaining more than 90% of baseline performance on evaluation metrics.

1 Introduction

  • Existing compressors often use LLM self-attention, whose quadratic time complexity as context lengthens raises compression costs and restricts real-time applications.Source: §1 · p. 2
  • IC-Former uses cross-attention and learnable digest tokens to extract information from context embeddings, producing compact vectors that replace the original context as LLM input.Source: §1 · p. 2
  • Training combines context-reconstruction pretraining with instruction-data fine-tuning so digest vectors can reconstruct context and respond to context-related prompts.Source: §1 · p. 2
  • At 4x compression, IC-Former uses 1/32 of the baseline’s floating-point operations and compresses 68 to 112 times faster while maintaining over 90% of baseline performance on evaluation metrics.Source: §1 · p. 2
  • The model compresses context to a quarter of its original length as a soft prompt while preserving most of its contextual information.Source: §1 · p. 2
  • IC-Former is lightweight, with a parameter size equal to 9% of the target LLM, and the paper analyzes interactions between the compressor and context to improve interpretability.Source: §1 · p. 2

2 Related Work

Prior work explores soft prompts, context distillation, and long-context compression, but approaches differ in context length, model updates, and compression cost.

Soft prompt compression

  • Wingate et al. (2022) learn compact soft prompts aligned to original-prompt predictions, but require retraining a prompt for each context; IC-Former predicts one from the input context.Source: §2 · pp. 2–3

Context distillation

  • GIST compresses short prompts into gist tokens but does not address long contexts and requires language-model parameter updates, whereas IC-Former keeps the language model fixed.Source: §2 · p. 3

Context compression

  • AutoCompressors and ICAE compress long contexts using self-attention, giving quadratic complexity; IC-Former avoids contextual interactions to reduce time and space complexity.Source: §2 · p. 3

3 Method

IC-Former compresses context embeddings into digest vectors that can replace the original context in an LLM prompt. Its cross-attention design, positional embeddings, and staged training support compression and response to prompts.

3.1 Task Formulation

  • A compressor maps soft-token and context embeddings into k compact context vectors, with k much smaller than the original context length, for use with prompts in an LLM.The resulting vectors substitute for the original context while the LLM output remains faithful to its content.Source: §3.1 · p. 3

3.2 In-Context Former

  • Digest tokens query context and preceding digest embeddings through causal cross-attention, condensing context into compact digest vectors with O(kn+k^2) complexity.RoPE supplies relative positional information so digest tokens can perceive context and other digest tokens in sequence.Source: §3.2 · pp. 3–4

3.3 Training process

  • Pretraining uses digest vectors and an LLM with an [AE] token to reconstruct the original context; instruction fine-tuning trains prompt-conditioned responses.When contexts exceed the compression limit, the model compresses uniformly split chunks separately and concatenates their local vectors for training and inference.Source: §3.3 · p. 5, Figure 4 · p. 5

4 Experiments

Experiments compare IC-Former with a baseline on compression efficiency, context reconstruction, and downstream performance, then analyze how digest tokens aggregate context. The evaluations use Llama2-7b-chat and include tests across context lengths and compression strategies.

4.1 Experimental setting

  • The experiments use Llama2-7b-chat as the target LLM and ICAE as the baseline, with approximately 2.29 million Pile entries for pretraining and PwC for fine-tuning.The default digest-token count k is 128.Source: §4.1 · pp. 5–6

4.2.1 Compression & Inference Efficiency

  • At 4x compression, IC-Former requires 1/32 of the baseline FLOPs and runs 68–112 times faster while achieving over 88% memory savings relative to the baseline.With compression followed by inference, processing is approximately four times faster than direct inference on the original context.Source: §4.2.1 · p. 6, Table 2 · p. 6, Table 1 · p. 6

4.2.2 Pretraining: Context Reconstruction

  • For contexts shorter than 400, reconstruction scores 0.99 BLEU-4 and around 0.05 cross-entropy; at length 500, BLEU-4 is 0.96 and cross-entropy is approximately 0.1.These results are reported for a 4x compression ratio.Source: Table 3 · p. 6, §4.2.2 · p. 7
  • Reconstruction quality deteriorates as digest-token length k decreases, and progressively declines as text randomness increases.The randomness pattern may suggest compression relies on semantic understanding rather than rote memorization.Source: §4.2.2 · p. 7, Figure 5 · p. 7, Table 5 · p. 7

4.2.3 Performance on Downstream Task

  • On the PwC test set, IC-Former achieves over 92% of baseline performance while significantly surpassing it in compression speed.The reported divide-and-conquer strategy has a slight performance degradation compared with direct compression, and removing pretraining performs poorly at capturing context and is more prone to hallucinations.Source: Table 4 · p. 7, §4.2.3 · p. 8

4.3 Analysis

  • In attention analysis, each digest token attends to 3–5 consecutive context tokens in sequence, while successive layers broaden attention from grammatical function words toward more varied content words.The authors suggest the layer-wise pattern may indicate reliance on semantic structures.Source: §4.3 · p. 8, Figure 6 · p. 8, Table 6 · p. 8

5 Conclusion

The conclusion presents IC-Former as a linear-complexity context compressor that uses cross-attention to condense contextual information into digest vectors. The authors report reduced time and space complexity while preserving contextual semantics.

5 Conclusion

  • IC-Former condenses contextual information into digest vectors with linear complexity by removing irrelevant interaction processing.The conclusion describes the resulting vectors as an efficient representation of context.Source: §5 · p. 9
  • The model uses cross-attention to enhance the extraction ability of digest tokens.Source: §5 · p. 9
  • Experiments show reduced time and space complexity while preserving contextual semantics, supporting broader applications requiring extensive context.Source: §5 · p. 9

Limitations

The authors identify limits in evaluation coverage and downstream performance: experiments used one target model, did not test longer contexts, and did not surpass the baseline on downstream tasks.

Limitations

  • Experiments used only Llama2-7b-chat; evaluation on larger-scale models remains future work.Source: Limitations · p. 9
  • Although implementation can handle longer texts, resource constraints prevented compression experiments on longer contexts.Source: Limitations · p. 9
  • IC-Former significantly outperforms the baseline in efficiency but has not surpassed its downstream-task performance.Source: Limitations · p. 9

A Experiment Details

The appendix points to a table containing IC-Former’s detailed configuration.

A Experiment Details

A.2 Training Configuration

The appendix points to separate pretraining and fine-tuning configurations, gives the evaluation prompt template, and reports the profiling hardware and software environment.

A.2 Training Configuration

  • Training configurations are provided separately for pretraining and fine-tuning in Tables 8 and 9.The passage points to the tables but does not give their configuration values.Source: §A.2 · p. 12, Table 8 · p. 12, Table 9 · p. 12

A.3 Prompt Template on Evaluation

  • Evaluation uses the template “Response the Prompt based on the below text: {context} Prompt:{prompt}”.The template places the context before the prompt.Source: §A.3 · p. 12

B Profiling Setup

  • Profiling uses one Nvidia RTX A6000 GPU (48GB) and an Intel Xeon Gold 6326 CPU with 16 cores and 1007GB RAM.The software configuration is Python 3.8.18, PyTorch 1.13.1, CUDA 11.7, and cuDNN 8.5.Source: B Profiling Setup · p. 12

C.1 Complexity Analysis

Figure 7 contrasts the attention masks used in IC-Former and ICAE, identifying their digest tokens and encoder memory tokens.

C.1 Complexity Analysis

  • Figure 7 contrasts IC-Former’s attention mask, with digest tokens (d_i), against ICAE’s mask, with encoder memory tokens (m_i).The caption identifies the token types but does not state a performance comparison.Source: Figure 7 · p. 12

C.2 Floating Point Operations Calculation

The appendix describes a theoretical FLOP calculation for compressing context embeddings, then illustrates how pretraining relates to digest quality and attention patterns.

C.2 Floating Point Operations Calculation

  • FLOPs are calculated for compressing context embeddings shaped [b, s, h] into vectors of length k, using attention and FFN matrix multiplications.Normalization and softmax overheads are omitted; Tables 10 and 11 give per-layer theoretical complexity for ICAE and IC-Former.Source: §C.2 · p. 13, Table 10 · p. 13, Table 11 · p. 13

D Case Study

  • In the Llama2-7b-chat examples, digests from non-pretrained IC-Former capture context poorly and are more prone to hallucination.Table 12 presents outputs based on 128 digest vectors from pretrained and non-pretrained models.Source: D Case Study · p. 13, Table 12 · p. 14

E Attention Maps in IC-Former

  • Comparing attention maps, the authors observe that non-pretrained IC-Former does not exhibit neighborhood information aggregation; captured words show no distinct grammatical patterns across layers, which they say underscores pretraining’s necessity.Source: E Attention Maps in IC-Former · p. 13

Related papers

Loading 2406.13618v2…