Source-linked AI summary
Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
Yijiong Yu, Shuai Yuan, Jie Zheng, Huazheng Wang, Ji Pei
TL;DR
Existing soft context compression uses uniform ratios despite large variation in information density, while continuous input-dependent structural hyperparameters can trigger model failure. The paper introduces a Discrete Ratio Selector that quantizes density-based targets, jointly trained with a mean-pooling compressor through synthetic-data SFT. Evaluations report consistent gains over static baselines and a Pareto frontier for density-aware compression.
Problem
Existing frameworks apply uniform compression ratios despite extreme variation in natural-language information density, while fully dynamic continuous structural hyperparameters can cause performance collapse.
Method
Semi-Dynamic Context Compression predicts a density-based target, quantizes it to predefined discrete ratios, and jointly trains ratio prediction and compression using synthetic-data SFT.
Results
Evaluations with Qwen3 0.6B and 4B confirm that the density-aware framework, using mean-pooling, consistently outperforms static fixed-ratio baselines.
Takeaways & Limitations
Discrete ratio selection adapts compression to text diversity while avoiding continuously varying structural execution and allowing smooth global compression control.
Takeaways & Limitations
The approach is motivated by an intrinsic difficulty adapting to continuously varying structural hyperparameters, and hard compression remains vulnerable to information loss at extreme ratios.
Abstract
from arXiv · showhide
Soft context compression reduces the computational workload of processing long contexts in LLMs by encoding long context into a smaller number of latent tokens. However, existing frameworks apply uniform compression ratios, failing to account for the extreme variance in natural language information density. While adopting a density-aware dynamic compression ratio seems intuitive, empirical investigations reveal that models struggle intrinsically with operations parameterized by input dependent, continuous structural hyperparameters. To resolve this pitfall, we introduce Semi-Dynamic Context Compression framework. Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density and quantizes it to a predefined set of discrete compression ratios. It is efficiently jointly trained with the compressor on synthetic data, with the summary lengths as a proxy to create labels for compression ratio prediction. Extensive evaluations confirm that our density-aware framework, utilizing mean pooling as the backbone, consistently outperforms static baselines, establishing a robust Pareto frontier for context compression techniques. Our code, data and model weights are available at https://github.com/yuyijiong/semi-dynamic-context-compress
1 Introduction
Soft context compression reduces long-context costs but fixed ratios ignore variation in information density, while fully dynamic continuous ratios cause performance collapse. Semi-Dynamic Context Compression uses discrete ratio selection, efficient synthetic-data training, and mean-pooling to adapt compression while outperforming static baselines.
- Motivation: Fixed compression ratios ignore natural-language information-density variance, forcing heuristic trade-offs between efficiency and quality.Dense technical reports and redundant conversations require different compression budgets.
- Motivation: Fully dynamic continuous compression can cause profound performance collapse because LLMs struggle with input-dependent structural hyperparameters.The authors attribute this to finite model capacity and insufficient training coverage of continuously varying sequence reductions.
- Method: The Discrete Ratio Selector predicts density-based compression targets and quantizes them to predefined ratios such as 2×, 4×, and 8×.A user-controlled scale parameter separately enables smooth global adjustment of compression aggressiveness.
- Method: A single-stage pure-SFT pipeline jointly trains ratio prediction and context encoding using synthetic tasks, with teacher-generated summary lengths as density-proxy labels.Synthetic summarization and single- and multi-document QA supervise compression and decoding.
- Experiments: Without heavy pre-training, mean-pooling significantly outperforms trainable compression tokens, motivating its use as the experimental backbone.The comparison concerns feature extraction from encoder output hidden states.
- Results: Evaluations with Qwen3 0.6B and 4B models show consistent improvements over static fixed-ratio baselines.The magnitude of improvement positively correlates with the variance of dynamically selected ratios.
2 Related Work
Hard compression prunes discrete tokens but struggles at extreme ratios without severe information loss. Soft methods use continuous latent representations, while adaptive approaches add text-dependent compression but may rely on heuristics or lack global scaling control.
- Hard Prompt Compression: Hard prompt compression avoids extensive model training but is limited by discrete vocabulary representations.Its discrete nature makes extreme compression difficult without severe information loss.
- Soft Context Compression: Soft compression maps discrete token sequences into shorter continuous latent representations.Early one-token document compression methods caused massive information loss for lengthy documents.
- Soft Context Compression: Compression-token methods popularized a soft-compression paradigm but typically require massive text-reconstruction pre-training and often freeze the decoder.
- Dynamic and Adaptive Compression: Dynamic Large Concept Models chunk text by adjacent-token similarity and mean-pool each chunk, but their strategy is heuristic and lacks user-controlled global scaling.
3 Methodology
The methodology motivates semi-dynamic compression by showing why continuous input-dependent structural parameters are difficult to learn, then uses discrete ratio selection to adapt compression to information density. It combines this selector with mean-pooling, single-pass execution, and synthetic-data SFT.
- Feature extraction methods: Benchmarking selects mean-pooling as the experimental backbone because trainable compression tokens are significantly worse without heavy pre-training.The paper’s stated method context includes mean-pooling as the chosen structural backbone for its semi-dynamic framework.
- Feature extraction methods: Existing extraction paradigms use target token count M or pooling size S as structural hyperparameters, creating dependencies on context length when compression must adapt.The paradigms include last tokens, compression tokens, and mean-pooling; mean-pooling uses non-overlapping windows.
- Continuous structural hyperparameters: Continuous dynamic structural parameters create an effectively infinite range of computational variations, causing unstable optimization and severe accuracy degradation.Models retain near-optimal accuracy when trained on a small discrete set of operations such as S ∈{2, 4, 8, 16}.
- Semi-dynamic framework: The selector predicts compression in logarithmic space, allows inference-time scale adjustment, exponentiates the prediction, and then branches into ratio- or length-based quantization.For mean-pooling, the selected ratio deterministically yields S = int(1/rtarget); length-based backbones instead select from discrete token counts.
- Training and execution: A single encoding pass performs density prediction and compression, while a single-stage pure-SFT pipeline uses teacher-generated ultra-concise summary lengths as density-proxy labels.The architecture extracts the final-token hidden state for prediction before executing structural compression; synthetic QA and summaries provide the training data.
4 Experiments
Experiments evaluate compression methods across benchmarks, model scales, attention patterns, and training designs. Mean-pooling and semi-dynamic compression consistently perform strongly, while discrete structural choices avoid the severe optimization problems associated with broad dynamic variation.
- Experimental Setup: Evaluation uses a synthetic 10-million-sample training dataset and a mixed reading-comprehension benchmark sampled from four standard datasets.The evaluation reports substring answer accuracy and average compression ratio.
- Backbone Comparisons: Mean-pooling consistently outperforms last tokens and compression tokens at equivalent average compression rates.Compression-token extraction is significantly worse than mean-pooling and is also outperformed by last-token extraction without heavy pretraining.
- Hyperparameter Pitfall: Over 200 distinct structural operations in fixed-ratio token-based settings cause massive quality drops, whereas five discrete ratios produce minimal accuracy loss.Fixed-length mean-pooling exposes 28 stride operations and causes a smaller accuracy drop than token-based fixed-ratio variation.
- Semi-Dynamic Results: At identical average compression ratios, semi-dynamic mean-pooling maintains higher accuracy than static ratios across the evaluated spectrum except at the lowest ratio.Fully dynamic compression performs worse than static ratios, while semi-dynamic gains are largest at moderate scales where selected-ratio variance peaks.
- Scaling: The static-to-semi-dynamic performance gap persists with Qwen3-4B-Instruct, indicating that the framework scales across model capacity.The larger model has higher overall accuracy at every setting.
- Training Design: Single-stage joint compression and ratio prediction performs very close to a two-stage pipeline while achieving higher efficiency.The comparison tests whether jointly training both functions in one encoder harms performance.
- Attention: Bidirectional attention is slightly better than causal attention for semi-dynamic compression and provides a distinct advantage at fixed ratios of 16× or higher.Differences are negligible at fixed ratios of 2× and 4×.
5 Conclusion
The Semi-Dynamic Context Compression framework addresses fixed-ratio inefficiencies by using a Discrete Ratio Selector to adapt compression to text information density. Its efficient training design and empirical results support a density-aware Pareto frontier for context compression.
- 5 Conclusion: The Discrete Ratio Selector quantizes continuous density predictions into discrete, learnable compression ratios.This enables density-aware adaptation while avoiding continuous structural hyperparameters.
- 5 Conclusion: A single-stage architecture performs ratio prediction and context encoding efficiently in one encoding pass.The framework uses summary length as a density proxy in a pure-SFT training pipeline, avoiding reinforcement learning and text-reconstruction pre-training.
- 5 Conclusion: The framework dynamically adapts compression to varying text densities while providing user control over global compression aggressiveness through a scaling parameter.
- 5 Conclusion: Extensive evaluations show that the density-aware framework outperforms static baselines and establishes a robust new Pareto frontier for context compression.