Source-linked AI summary
Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission
Jiangnan Ye, Hanqi Yan, Zhenyi Shen, Heng Chang, Ye Mao, Yulan He
TL;DR
Long contexts impose growing token, memory, and latency costs, while existing LLM-as-a-compressor methods inadequately preserve contextual information. ComprExIT treats a frozen LLM as a feature extractor, adaptively aggregates layerwise features, and globally transmits them to compression slots. Across 12 datasets, it consistently outperforms prior compression methods while adding approximately 1% parameters and compressing more than 2× faster than the fastest baselines.
Problem
Long-context LLM use incurs substantial token, memory, and latency costs, while existing LLM-as-a-compressor methods remain below the uncompressed baseline because they insufficiently retain contextual information.
Method
ComprExIT uses frozen LLM hidden states, adaptively aggregates features across layers, and globally allocates anchor information to compression slots through an explicit transmission plan.
Results
ComprExIT consistently outperforms prior context compression methods across 12 datasets, improving average F1 by 18.5%.
Takeaways & Limitations
ComprExIT provides a lightweight and faster soft-compression paradigm that expands context-compression design beyond standard attention-based formulations.
Takeaways & Limitations
The identified allocation and dilution limitations constrain standard compressors’ contextual information retention and motivate ComprExIT’s structural redesign.
Abstract
from arXiv · showhide
Long-context LLM agents often struggle with growing token, memory, and latency costs, making efficient context compression essential for practical deployment. Existing LLM-as-a-compressor methods remain noticeably inferior to using the full context. We find that this gap partly stems from their inability to preserve contextual information effectively. In this work, we revisit context compression from a structural perspective and identify two key bottlenecks in standard LLM-based compressors: limited coordination among compression tokens during information aggregation, and layerwise dilution that weakens useful signals from intermediate hidden states. To address these limitations, we propose ComprExIT, a new context compression framework based on explicit information transmission. ComprExIT adaptively selects features across frozen LLM layers, then allocates information from anchors to compression slots through a globally coordinated transport plan. Experiments on 12 datasets show that ComprExIT consistently outperforms strong soft-compression baselines, improving average F1 by up to 18.5%, while adding only ~1% trainable parameters and achieving more than 2x faster compression than the fastest baselines. The code will be released upon acceptance.
1 Introduction
Long contexts impose substantial token, memory, and latency costs, while existing LLM-as-a-compressor methods lose contextual information. ComprExIT addresses structural bottlenecks through explicit information transmission and improves compression performance with low overhead.
- Long contexts consume tokens, saturate effective context windows, and enlarge KV caches, creating substantial latency and memory overhead.
- Existing LLM-as-a-compressor methods remain below the uncompressed baseline because their compressed representations insufficiently retain contextual information.Higher reconstruction NLL is associated with lower downstream F1.
- Standard compressors suffer from limited allocation among gist tokens and information dilution across layers.These bottlenecks can leave context regions under-covered and weaken early-layer features.
- ComprExIT treats a frozen LLM as a feature extractor and combines coordinated widthwise transmission with depthwise layer aggregation.The widthwise stage globally allocates information across compression slots, while the depthwise stage aggregates features from different layers.
- 18.5% average F1 improvement across 12 datasets accompanies approximately 1% additional parameters and more than 2× faster compression than the fastest baselines.
2 Related Work
Context compression methods operate either by discrete token pruning or continuous soft compression. Soft compression offers greater flexibility at high compression ratios, while LLM-as-a-compressor methods modify internal LLM computation to encode context into gist tokens.
- Context compression divides into token pruning and soft context compression, operating in discrete and continuous spaces respectively.
- Token pruning: Token-pruning methods estimate token or span importance and retain only the most informative parts of the context.Examples include SelectiveContext and LLMLingua-style methods.
- Soft context compression: Existing LLM-as-a-compressor methods feed context tokens and gist tokens into an LLM trained to encode contextual information into the gist tokens.
- Token pruning: Hard token-removal methods remain discrete and lossy, with bounded compression ratios and less expressiveness than continuous soft compression.
- Soft context compression: Soft context compression provides greater flexibility and expressiveness than discrete removal at high compression ratios.ComprExIT belongs to this category.
3 Preliminary
Context compression maps a long context to a shorter representation that a decoder uses to generate outputs. In the LLM-as-a-compressor paradigm, compression tokens are encoded through the same self-attention and layer-propagation mechanisms as ordinary LLM computation.
- A compressor maps context x of length N to a compact representation Z of K vectors, where K ≪ N, and a decoder conditions on Z to generate outputs.
- LLM-as-a-compressor methods append K learnable gist tokens to the embedded context before processing the sequence with a Transformer.
- The compressed representation is taken from compression-token positions in the Transformer’s final-layer output.
- This paradigm relies on self-attention for widthwise aggregation and residual layer propagation for depthwise information transfer.
4 Limitations and Observations
The analysis identifies two structural bottlenecks in LLM-based context compression: uncoordinated information allocation across compression tokens and dilution of useful features across layers. Experiments show that gist tokens often capture redundant content, while useful compression features vary non-monotonically by layer and task.
- Widthwise limitation: lack of allocation: Independent self-attention normalizes each gist token separately, allowing multiple tokens to focus on the same context region while leaving other regions weakly covered.This is identified as a widthwise lack-of-allocation bottleneck.
- Widthwise limitation: lack of allocation: Highly correlated gist-token attention distributions and steep singular-value decay indicate that gist tokens often capture redundant rather than complementary content.Figure 3 uses attention-distribution correlations and the aggregation matrix spectrum to assess aggregation diversity.
- Depthwise limitation: layerwise dilution: Intermediate-layer features can be attenuated before reaching the final compression state, creating a layerwise dilution bottleneck.The proposed diagnostic treats a drop in usable information after an intermediate peak as evidence that earlier information is not fully retained.
- Depthwise limitation: layerwise dilution: Final-layer compression performs worst in probing experiments, while compression quality varies non-monotonically across layers.The probing compressor uses single-layer hidden states, 4× mean pooling, and a 2-layer MLP on SQuAD and HotpotQA, with decoder F1 as the metric.
- Depthwise limitation: layerwise dilution: Layer contributions are task-dependent: early and late layers help reasoning-intensive QA, whereas middle layers are more effective for extractive QA.These findings motivate preserving and adaptively exploiting features from different layers.
5 Method
ComprExIT treats a frozen LLM as a feature extractor and compresses context in two coordinated stages. It adaptively aggregates information across layers into token anchors, then uses a globally coupled transmission plan to allocate anchor information across compression slots.
- 5.1 Depthwise Transmission to Token Anchors: ComprExIT addresses layerwise dilution by directly reading hidden states across depths and constructing a token anchor for each token position.The method is presented in computational order from depthwise to widthwise computation.
- 5.1 Depthwise Transmission to Token Anchors: Learned layer priors and token-wise gating attention adaptively mix layer representations into each token anchor.The gating uses learnable query, key, and value projections, layer embeddings, and a temperature.
- 5.1 Depthwise Transmission to Token Anchors: The resulting anchor is a gated multi-layer readout from the frozen LLM, keeping useful intermediate features directly accessible to the compression interface.Token-conditioned gating can preserve different information granularities for different tokens and tasks.
- 5.2 Widthwise Transmission to Compression Slots: ComprExIT addresses lack of allocation by transmitting information from N token anchors to K compression slots through a shared transmission plan.The shared plan replaces independent slot-wise retrieval and coordinates sender-slot assignments globally.
- 5.2 Widthwise Transmission to Compression Slots: The utility matrix scores sender-slot paths, with higher utility indicating a greater likelihood of preserving useful compressed content.Receivers are formed from local fields of token anchors to retain alignment with the original token order.
- 5.2 Widthwise Transmission to Compression Slots: Learned sender capacities let less useful anchors transmit less mass, while salient anchors retain more influence.Sender capacities are predicted with a linear layer followed by a softmax over token anchors.
- 5.2 Widthwise Transmission to Compression Slots: Uniform receiver capacities give every compression slot the same total budget while allowing important anchors to connect to non-local slots.The design is intended to better preserve long-range dependencies.
- 5.2 Widthwise Transmission to Compression Slots: The transmission plan is an entropy-regularized optimal transport solution whose shared marginal constraints couple all sender-slot assignments.The resulting strictly convex objective can be optimized efficiently with the Sinkhorn algorithm.
6 Experiments and Analysis
Experiments evaluate ComprExIT across QA benchmarks, ablations, latency, allocation behavior, and scalability. It consistently improves compression performance while preserving efficiency and adapting information allocation across layers and slots.
- Overall performance: ComprExIT improves average F1 over the strongest compression baseline by 18.50% on Llama-3.2-1B and 10.56% on Llama-3.2-3B across six in-domain QA benchmarks.Performance remains within 0.25% and 0.64% of the uncompressed prompt-tuning baseline, respectively.
- Out-of-domain performance: ComprExIT improves average F1 over the strongest compression baseline by 30.97% on Llama-3.2-1B and 14.27% on Llama-3.2-3B across six out-of-domain MRQA datasets.It also outperforms the uncompressed baseline on XSum and FEVER while remaining competitive on SST-2.
- Ablation studies: Removing global allocation or layer-wise aggregation consistently degrades performance across datasets and single-layer settings.The ablation results support coordinated allocation and selective aggregation across layers as important components of ComprExIT.
- Efficiency: ComprExIT is more than 2× faster than ICAE in compression-stage latency across tested context lengths.Compressed methods are also substantially faster end-to-end than decoding the full context.
- Higher compression ratios: ComprExIT improves over the baseline by 17.2% Avg. F1 at 8× compression and 7.6% at 16× compression.At 8× compression, it reaches 58.66 versus ICAE’s 56.16 at 4× compression.
- Scalability: ComprExIT continues to outperform ICAE at larger model scale, slightly exceeding the uncompressed prompt-tuning baseline in F1 with 76.48 versus 75.92.On long-context QA with contexts up to 8k tokens, it improves over the uncompressed baseline by 18.0% in F1.
- Allocation analysis: ComprExIT produces less-correlated, higher-rank allocation patterns, indicating that compression slots absorb more complementary and less redundant information.Figure 5 analyzes Pearson correlation and singular-value spectra of allocation patterns.
- Layer preference: Gating concentrates on early and middle layers, while token types receive different preferred layers during compression.Entity tokens, nouns, adjectives, and numbers favor middle layers, whereas other tokens more often use initial layers.
7 Conclusion
The paper presents ComprExIT as explicit information transmission over frozen LLM hidden states, addressing allocation and layerwise dilution. It reports consistent improvements over prior compression methods while identifying important evaluation boundaries.
- Conclusion: ComprExIT formulates soft context compression as explicit information transmission over frozen LLM hidden states to mitigate lack of allocation and layerwise dilution.The framework is presented as a paradigm beyond standard attention-based formulations.
- Conclusion: The experiments show that ComprExIT consistently outperforms prior context compression methods.
A.1 Experiment Specification
The supplied experiment-specification passage states that more experiment details are presented, without specifying the experimental procedures or settings.
- A.1 Experiment Specification: The paper presents additional details of its experiments.
A.1.1 Measuring Information Loss during Compression
The study measures how much contextual information compressed gist representations retain by using them to reconstruct the original context with a frozen decoder.
- Compression quality is evaluated by feeding layer-specific gist states to a frozen decoder as a prefix before the original context.The evaluation uses a fixed reproduction prompt and computes causal language-model loss on context tokens.
- The reported reconstruction loss is aggregated across examples using the number of non-padded target tokens as weights.
- The evaluation contexts are drawn from SQuAD and HotpotQA, using compressors trained with Llama-3.2-1B.
A.1.2 Latency Analysis
Latency is benchmarked under fixed synthetic workloads while separately measuring compression and decoding costs.
- Latency experiments use context lengths of 512 and 2048, batch size 8, generation length 128, and compression ratio 4.
- Compression latency is measured separately from decoding latency on an NVIDIA L40S ADA GPU.The compression stage is the encoder forward pass producing the gist, while decoding is autoregressive generation.
- Each sequence uses 50 untimed warm-up iterations followed by 200 timed iterations, with mean and standard deviation reported.
A.1.3 Gist Token Allocation Patterns
The section examines whether compressed slots represent distinct context regions and connects allocation structure to broader empirical, theoretical, and deployment considerations. Across additional tasks and benchmarks, ComprExIT shows strong transfer while retaining risks and scope limitations.
- A.1.3 Gist Token Allocation Patterns: ComprExIT’s allocation analysis represents per-slot attention or transport scores as a [K, N] matrix for 1000 random SQuAD samples.
- A.1.3 Gist Token Allocation Patterns: High effective rank and near-diagonal similarity indicate that compressed slots cover input regions non-redundantly.
- A.4 Evaluation on More Tasks: ComprExIT achieves ROUGE-1/ROUGE-L scores of 0.333/0.262 on XSum, outperforming ICAE and the uncompressed prompt-tuning baseline.
- A.4 Evaluation on More Tasks: On FEVER, ComprExIT scores 64.39, surpassing prompt tuning by 3.91% and ICAE by 8.69%; on SST-2 ICL, it scores 69.17.
- A.5 Out-of-Domain Evaluation: ComprExIT remains the strongest compression method across six out-of-domain MRQA benchmarks for both Llama-3.2-1B-Base and Llama-3.2-3B-Base.ICAE is competitive on RelationExtraction, where contexts are very short on average.
- A.6 The Allocation Matrix: Its transport plan softly anchors each compression slot to a contiguous local region while retaining connections to distant tokens.The pattern preserves semantic order and supports long-range dependencies.
- Theory and Optimization: Layerwise drift can accumulate with depth and increase M, making the compression objective harder to optimize.
- Theory and Optimization: ComprExIT starts with the lowest NTP loss and converges faster to the best plateau, consistent with decoder-friendly compressed states.