Source-linked AI summary
Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
Yijiong Yu, Shuai Yuan, Jie Zheng, Huazheng Wang, Ji Pei
TL;DR
Existing soft context compression uses uniform ratios despite large variation in information density, while continuous input-dependent structural hyperparameters can trigger model failure. The paper introduces a Discrete Ratio Selector that quantizes density-based targets, jointly trained with a mean-pooling compressor through synthetic-data SFT. Evaluations report consistent gains over static baselines and a Pareto frontier for density-aware compression.
Problem
Existing frameworks apply uniform compression ratios despite extreme variation in natural-language information density, while fully dynamic continuous structural hyperparameters can cause performance collapse.
Source: §1 · pp. 1–2Method
Semi-Dynamic Context Compression predicts a density-based target, quantizes it to predefined discrete ratios, and jointly trains ratio prediction and compression using synthetic-data SFT.
Source: §1 · p. 2Results
Evaluations with Qwen3 0.6B and 4B confirm that the density-aware framework, using mean-pooling, consistently outperforms static fixed-ratio baselines.
Source: §1 · p. 2Takeaways & Limitations
Discrete ratio selection adapts compression to text diversity while avoiding continuously varying structural execution and allowing smooth global compression control.
Source: §1 · p. 2, §5 · p. 10
Abstract
from arXiv · showhide
Soft context compression reduces the computational workload of processing long contexts in LLMs by encoding long context into a smaller number of latent tokens. However, existing frameworks apply uniform compression ratios, failing to account for the extreme variance in natural language information density. While adopting a density-aware dynamic compression ratio seems intuitive, empirical investigations reveal that models struggle intrinsically with operations parameterized by input dependent, continuous structural hyperparameters. To resolve this pitfall, we introduce Semi-Dynamic Context Compression framework. Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density and quantizes it to a predefined set of discrete compression ratios. It is efficiently jointly trained with the compressor on synthetic data, with the summary lengths as a proxy to create labels for compression ratio prediction. Extensive evaluations confirm that our density-aware framework, utilizing mean pooling as the backbone, consistently outperforms static baselines, establishing a robust Pareto frontier for context compression techniques. Our code, data and model weights are available at https://github.com/yuyijiong/semi-dynamic-context-compress
1 Introduction
Soft context compression reduces long-context costs but fixed ratios ignore variation in information density, while fully dynamic continuous ratios cause performance collapse. Semi-Dynamic Context Compression uses discrete ratio selection, efficient synthetic-data training, and mean-pooling to adapt compression while outperforming static baselines.
Motivation
- Fixed compression ratios ignore natural-language information-density variance, forcing heuristic trade-offs between efficiency and quality.Dense technical reports and redundant conversations require different compression budgets.Source: §1 · p. 1
- Fully dynamic continuous compression can cause profound performance collapse because LLMs struggle with input-dependent structural hyperparameters.The authors attribute this to finite model capacity and insufficient training coverage of continuously varying sequence reductions.Source: §1 · pp. 1–2
Method
- The Discrete Ratio Selector predicts density-based compression targets and quantizes them to predefined ratios such as 2×, 4×, and 8×.A user-controlled scale parameter separately enables smooth global adjustment of compression aggressiveness.Source: §1 · p. 2
- A single-stage pure-SFT pipeline jointly trains ratio prediction and context encoding using synthetic tasks, with teacher-generated summary lengths as density-proxy labels.Synthetic summarization and single- and multi-document QA supervise compression and decoding.Source: §1 · p. 2
Experiments
- Without heavy pre-training, mean-pooling significantly outperforms trainable compression tokens, motivating its use as the experimental backbone.The comparison concerns feature extraction from encoder output hidden states.Source: §1 · p. 2
Results
- Evaluations with Qwen3 0.6B and 4B models show consistent improvements over static fixed-ratio baselines.The magnitude of improvement positively correlates with the variance of dynamically selected ratios.Source: §1 · p. 2
2 Related Work
Hard compression prunes discrete tokens but struggles at extreme ratios without severe information loss. Soft methods use continuous latent representations, while adaptive approaches add text-dependent compression but may rely on heuristics or lack global scaling control.
Hard Prompt Compression
- Hard prompt compression avoids extensive model training but is limited by discrete vocabulary representations.Its discrete nature makes extreme compression difficult without severe information loss.Source: §2 · pp. 2–3
Soft Context Compression
- Soft compression maps discrete token sequences into shorter continuous latent representations.Early one-token document compression methods caused massive information loss for lengthy documents.Source: §2 · p. 3
- Compression-token methods popularized a soft-compression paradigm but typically require massive text-reconstruction pre-training and often freeze the decoder.Source: §2 · p. 3
Dynamic and Adaptive Compression
- Dynamic Large Concept Models chunk text by adjacent-token similarity and mean-pool each chunk, but their strategy is heuristic and lacks user-controlled global scaling.Source: §2 · p. 3
3 Methodology
The methodology motivates semi-dynamic compression by showing why continuous input-dependent structural parameters are difficult to learn, then uses discrete ratio selection to adapt compression to information density. It combines this selector with mean-pooling, single-pass execution, and synthetic-data SFT.
Feature extraction methods
- Benchmarking selects mean-pooling as the experimental backbone because trainable compression tokens are significantly worse without heavy pre-training.The paper’s stated method context includes mean-pooling as the chosen structural backbone for its semi-dynamic framework.Source: Figure 2 · p. 4
- Existing extraction paradigms use target token count M or pooling size S as structural hyperparameters, creating dependencies on context length when compression must adapt.The paradigms include last tokens, compression tokens, and mean-pooling; mean-pooling uses non-overlapping windows.Source: §3.1 · p. 4
Continuous structural hyperparameters
- Continuous dynamic structural parameters create an effectively infinite range of computational variations, causing unstable optimization and severe accuracy degradation.Models retain near-optimal accuracy when trained on a small discrete set of operations such as S ∈{2, 4, 8, 16}.Source: §3.2 · p. 4
Semi-dynamic framework
- The selector predicts compression in logarithmic space, allows inference-time scale adjustment, exponentiates the prediction, and then branches into ratio- or length-based quantization.For mean-pooling, the selected ratio deterministically yields S = int(1/rtarget); length-based backbones instead select from discrete token counts.Source: §3.3 · p. 5
Training and execution
- A single encoding pass performs density prediction and compression, while a single-stage pure-SFT pipeline uses teacher-generated ultra-concise summary lengths as density-proxy labels.The architecture extracts the final-token hidden state for prediction before executing structural compression; synthetic QA and summaries provide the training data.Source: §3.3 · p. 5, §3.4 · p. 6
4 Experiments
Experiments evaluate compression methods across benchmarks, model scales, attention patterns, and training designs. Mean-pooling and semi-dynamic compression consistently perform strongly, while discrete structural choices avoid the severe optimization problems associated with broad dynamic variation.
Experimental Setup
- Evaluation uses a synthetic 10-million-sample training dataset and a mixed reading-comprehension benchmark sampled from four standard datasets.The evaluation reports substring answer accuracy and average compression ratio.Source: §4.1 · pp. 6–7
Backbone Comparisons
- Mean-pooling consistently outperforms last tokens and compression tokens at equivalent average compression rates.Compression-token extraction is significantly worse than mean-pooling and is also outperformed by last-token extraction without heavy pretraining.Source: §4.3.1 · p. 8
Hyperparameter Pitfall
- Over 200 distinct structural operations in fixed-ratio token-based settings cause massive quality drops, whereas five discrete ratios produce minimal accuracy loss.Fixed-length mean-pooling exposes 28 stride operations and causes a smaller accuracy drop than token-based fixed-ratio variation.Source: §4.3.1 · pp. 8–9
Semi-Dynamic Results
- At identical average compression ratios, semi-dynamic mean-pooling maintains higher accuracy than static ratios across the evaluated spectrum except at the lowest ratio.Fully dynamic compression performs worse than static ratios, while semi-dynamic gains are largest at moderate scales where selected-ratio variance peaks.Source: Figure 4 · p. 7, §4.3.2 · p. 9
Scaling
- The static-to-semi-dynamic performance gap persists with Qwen3-4B-Instruct, indicating that the framework scales across model capacity.The larger model has higher overall accuracy at every setting.Source: Figure 6 · p. 8, §4.4 · p. 9
Training Design
- Single-stage joint compression and ratio prediction performs very close to a two-stage pipeline while achieving higher efficiency.The comparison tests whether jointly training both functions in one encoder harms performance.Source: Figure 7 · p. 8, §4.4 · p. 9
Attention
- Bidirectional attention is slightly better than causal attention for semi-dynamic compression and provides a distinct advantage at fixed ratios of 16× or higher.Differences are negligible at fixed ratios of 2× and 4×.Source: Figure 8 · p. 8, §4.4 · p. 9
5 Conclusion
The Semi-Dynamic Context Compression framework addresses fixed-ratio inefficiencies by using a Discrete Ratio Selector to adapt compression to text information density. Its efficient training design and empirical results support a density-aware Pareto frontier for context compression.
5 Conclusion
- The Discrete Ratio Selector quantizes continuous density predictions into discrete, learnable compression ratios.This enables density-aware adaptation while avoiding continuous structural hyperparameters.Source: §5 · pp. 9–10
- A single-stage architecture performs ratio prediction and context encoding efficiently in one encoding pass.The framework uses summary length as a density proxy in a pure-SFT training pipeline, avoiding reinforcement learning and text-reconstruction pre-training.Source: §5 · p. 10
- The framework dynamically adapts compression to varying text densities while providing user control over global compression aggressiveness through a scaling parameter.Source: §5 · p. 10
- Extensive evaluations show that the density-aware framework outperforms static baselines and establishes a robust new Pareto frontier for context compression.Source: §5 · p. 10