PaperlayerView exact revision on arXiv

Source-linked AI summary

CompLLM: Compression for Long Context Q&A

Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran, Mubarak Shah

arXiv:2509.19228v2cs.CL

TL;DR

Long-context Q&A can be unfeasibly expensive because transformer processing has quadratic complexity, while soft compression remains scarce in real-world applications. CompLLM splits context into segments and compresses each independently, producing shorter representations that the original unmodified LLM can consume. CompLLM speeds up Time To First Token (TTFT) by up to 4x for long contexts, reduces KV cache size by 2x, and is competitive with a standard uncompressed LLM pipeline on short contexts but better on long contexts.

  • Problem

    Long-context Q&A can be unfeasibly expensive because transformer processing has quadratic complexity, while soft compression remains scarce in real-world applications.

    Source: §1 · pp. 1–2

  • Method

    CompLLM splits context into segments and compresses each independently, producing shorter representations that the original unmodified LLM can consume.

    Source: §1 · p. 2

  • Results

    CompLLM speeds up Time To First Token (TTFT) by up to 4x for long contexts, reduces KV cache size by 2x, and is competitive with a standard uncompressed LLM pipeline on short contexts but better on long contexts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Compressed segments can be reused across queries that share context, including repeated-document and mostly unchanged-codebase settings.

    Source: §1 · p. 2

    CompLLM is designed to encode semantic content rather than text structure, so it would not work well for tasks such as counting a letter or finding typos.

    Source: §4.7 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) face significant computational challenges when processing long contexts due to the quadratic complexity of self-attention. While soft context compression methods, which map input text to smaller latent representations, have shown promise, their real-world adoption is limited. Existing techniques typically compress the context as a single unit, which leads to quadratic compression complexity and an inability to reuse computations across queries with overlapping contexts. In this work, we introduce CompLLM, a soft compression technique designed for practical deployment. Instead of processing the context holistically, CompLLM divides it into segments and compresses each one independently. This simple design choice yields three critical properties: efficiency, as the compression step scales linearly with the context length; scalability, enabling models trained on short sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and reusability, allowing compressed segments to be cached and reused across different queries. Our experiments show that with a 2x compression rate, at high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance comparable to that obtained with the uncompressed context, and even surpasses it on very long sequences, demonstrating its effectiveness and practical utility.

1 INTRODUCTION

Long-context Q&A can be computationally expensive because transformer processing has quadratic complexity. CompLLM compresses short context segments independently, aiming to improve efficiency, scalability, and reuse; at a 2x compression rate, it reports faster TTFT and a smaller KV cache while maintaining or improving results depending on context length.

1 INTRODUCTION

  • Long-context Q&A is costly because transformer processing has quadratic complexity, motivating reductions in computational cost as contexts grow.The paper cites coding assistants, web agents, document queries, and RAG systems as applications.Source: §1 · p. 1
  • Soft compression maps context to latent representations and has achieved higher compression rates with little accuracy drop, but remains scarce in real-world applications.CompLLM aims to compress text into fewer tokens consumable by the original, unmodified LLM.Source: §1 · p. 2
  • CompLLM independently compresses context segments rather than compressing the entire context as one unit.In whole-context methods, every input token affects the entire compressed representation.Source: §1 · p. 2
  • Segment-wise compression makes the compression step linear in context length, supports training on short sequences for much longer test contexts, and allows compressed segments to be reused across queries.The paper reports training on sequences no longer than 2K tokens and compression of contexts hundreds of thousands of tokens long.Source: §1 · p. 2
  • CompLLM speeds up TTFT by up to 4x for long contexts, reduces KV cache size by 2x, and matches a standard LLM pipeline on short contexts while performing better at long contexts.The authors hypothesize that fewer tokens reduce attention dilution.Source: §1 · p. 2

2 RELATED WORK

Prior context-compression work includes hard methods that produce shorter natural-language prompts and soft methods that use latent representations. The paper positions CompLLM against soft methods that compress contexts holistically, limiting efficiency, scalability, and reusability.

2.1 HARD COMPRESSION

  • Hard compression prunes tokens or sentences or paraphrases prompts; it is interpretable and usable with closed LLM APIs, but usually has lower compression rates and larger accuracy drops than latent-space methods.Some hard-compression methods are question-aware, while others are question-agnostic.Source: §2.1 · p. 3

2.2 SOFT COMPRESSION

  • Soft compression represents prompts as latent embeddings of length N1 or as KV caches with dimensions N2 × L × D × 2.The KV-cache dimensions correspond to generated sequence length, LLM layers, embedding dimension, and key/value embeddings.Source: §2.2 · p. 3
  • KV-cache embeddings depend on preceding tokens, making these representations holistic with O(N^2) time complexity and preventing independent sentence representations; sparse attention does not reduce KV-cache memory.The passage connects the holistic representation to non-reusability and nonlinear scalability.Source: §2.2 · p. 3
  • Related latent-embedding methods include task-specific embeddings, prompt compression, summary vectors, and memory slots, but the cited approaches compress context as a block.The paper identifies block compression as incompatible with its targeted efficiency, scalability, and reusability properties.Source: §2.2 · p. 3

3 METHOD

CompLLM reduces the number of embeddings passed to an LLM by estimating Concept Embeddings (CEs) from Token Embeddings (TEs). It compresses short segments independently, preserving similar outputs with fewer embeddings while keeping compression computation linear in total context length for fixed segment size.

3.1 COMPLLM

  • Concept Embeddings lie in the same latent feature space as Token Embeddings and can be fed to the LLM without tuning it, representing similar information with fewer embeddings and similar outputs.The figure illustrates the sentence “golden dogs are called” using four TEs or two CEs.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Given N input tokens, CompLLM splits the text into segments of at most S tokens and compresses each segment independently, yielding overall O(NS) complexity.Attention is quadratic within each segment, while the total complexity is linear in the number of segments for fixed S.Source: §3.1 · p. 4
  • With S = 20 and compression rate C = 2, each segment of 20 TEs is compressed into 10 CEs.The architecture must accept S or fewer embeddings and output S/C embeddings.Source: §3.1 · p. 4
  • The CompLLM architecture attaches a LoRA to the generation LLM and appends a single linear layer.The passage says this design takes inspiration from Ge et al. (2024).Source: §3.1 · p. 4

S C embeddings corresponding to EOS tokens, whose corresponding outputs are used as the S

CompLLM compresses context while leaving questions uncompressed, and trains by aligning answer-token representations. Its segment-based design reduces inference costs at long contexts, though compression can slow generation at very small contexts.

S C embeddings corresponding to EOS tokens, whose corresponding outputs are used as the S

  • CompLLM reuses the LLM’s parameters, leaves its weights untouched by LoRA, and can be unplugged to use the standard LLM pipeline.Benchmarking alternative compressor architectures is outside the manuscript’s scope.Source: S C embeddings corresponding to EOS… · p. 4

3.2 TRAINING

  • Training compresses context but leaves the question uncompressed, then distills answer-token hidden states rather than matching output distributions.The loss uses answer-embedding activations; no ground-truth labels are required because the LLM supplies the answer during training.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 COMPUTATIONAL COMPLEXITY ANALYSIS

  • At C=2, the TTFT latency ratio asymptotically approaches 4x, while the ratio for generating 10k tokens approaches 2x; very small contexts may be slower.The small-context slowdown occurs when compression time is non-negligible.Source: Figure 4 · p. 6, §3.3 · p. 6
  • Compression time scales linearly with context length, while KV-cache prefill scales quadratically; next-token prediction cost falls linearly with compression rate C.At large context lengths, the linear compression overhead becomes negligible relative to quadratic prefill cost.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate CompLLM across long-context Q&A datasets, context lengths, and compression baselines, alongside ablations and stated limitations. Results compare performance and inference costs with uncompressed and alternative compression pipelines.

4.1 IMPLEMENTATION DETAILS

  • CompLLM uses Gemma3-4B and Qwen3-4B with rank-32 LoRA, and splits text into segments capped at 20 tokens.The implementation uses Adam at learning rate 0.0001 and trains until convergence with batch size 4.Source: §4.1 · pp. 6–7

4.2 DATASETS

  • Training uses NarrativeQA and RACE, while testing covers those datasets plus SQuAD and QuAIL; evaluation uses LLM-as-a-judge for open-ended QA and regex matching for multiple choice.The experiments also evaluate LOFT datasets based on HotpotQA, Musique, NQ, Qampari, and Quest.Source: §4.2 · p. 7

4.3 MAIN RESULTS

  • At context lengths over 50k tokens, CompLLM achieves better results while reducing cache-prefill latency 4x, next-token latency 2x, and KV cache size 2x.Figure 5 compares compressed and uncompressed results across context lengths for four datasets and two base models.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4 RESULTS ON LOFT

  • On LOFT, CompLLM always matches or improves on the baseline across tasks designed to challenge smaller models.LOFT combines context-based Q&A with distractor contexts to reach 128k tokens.Source: §4.4 · p. 8

4.5 COMPARISON WITH EXISTING TECHNIQUES

  • For short-to-medium contexts, CompLLM alone matches the uncompressed LLM; LongLLMLingua beats LLMLingua-2, while AutoCompressor does better near its training distribution than on RACE and QuAIL.CompLLM is the only soft compression method in Table 3 that preserves LLM weights and has linear compression complexity; LLMLingua-2 has reusable sentence-level outputs, whereas LongLLMLingua is question-dependent.Source: §4.5 · pp. 8–9, Figure 6 · p. 9, Table 3 · p. 9

4.6 ABLATIONS

  • At C=4, results drop 4% versus the uncompressed baseline for one context; at C=8, they drop 15% for 64 documents while asymptotic speedup is 64x.Results show no noticeable change for S=10, 20, or 40; the authors also report empirical support for reduced attention dilution as CompLLM reduces the number of keys and values.Source: §4.6 · pp. 9–10

4.7 LIMITATIONS

  • CompLLM encodes semantic content rather than text structure, limiting its fit for exact-form tasks such as counting letters or detecting typos.The authors also state that an English-trained compressor cannot compress Chinese characters or code; compression can be skipped for out-of-distribution cases while retaining the original LLM.Source: §4.7 · p. 10

4.8 FUTURE WORK

  • Future work includes dynamic compression rates, testing higher rates and alternative architectures, and training from plain text rather than context-question pairs.The authors suggest larger models may accommodate higher compression rates, but frame this as a possibility to investigate.Source: §4.8 · p. 10

5 CONCLUSION

CompLLM speeds up long-context answers while reducing KV-cache memory and producing outputs of similar or better quality. It compresses text segments independently into embeddings that can be fed to the original LLM without fine-tuning.

5 CONCLUSION

  • CompLLM generates answers up to 4 times faster, reduces KV-cache memory by 2x, and produces outputs of similar or better quality for long-context Q&A.The paper reports that it works across multiple LLMs and datasets.Source: §5 · p. 10
  • CompLLM extracts new embeddings that can be fed directly to the LLM without fine-tuning it.Source: §5 · p. 10
  • Compressing text segments independently gives linear computational complexity and allows documents to be compressed offline for reuse across contexts and questions.The paper also states that CompLLM is orthogonal to techniques including chain of thought, RAG, beam search, and paged attention.Source: §5 · p. 10

A APPENDIX

The appendix reports ablations on compression rate and segment length, plus attention-score results. Higher compression rates degrade accuracy, while small changes in segment length have little effect on results.

A APPENDIX

  • Higher compression rates degrade results, but the drop is not steep even at C = 8.The ablation compares C = 2, 4, and 8.Source: A APPENDIX · p. 18
  • With C = 2, changing maximum segment length among S = 10, 20, and 40 has little effect, while training memory rises with larger S.Gemma3-4B training takes 57GB with S = 20; with S = 80, it does not fit into 80GB.Source: A APPENDIX · p. 18
  • CompLLM reduces attention dilution, although most generated-token attention is paid to earlier generated tokens and the question.Attention dilution is described as difficulty attending to relevant context tokens as context grows.Source: A APPENDIX · p. 18

A.1 EXAMPLES OF GENERATED ANSWERS

The appendix presents qualitative generations on four test sets, covering open-ended and multiple-choice Q&A. The examples use different prompting setups for those two task types.

A.1 EXAMPLES OF GENERATED ANSWERS

  • The qualitative examples cover NarrativeQA, SQuAD, RACE, and QuAIL test sets.Source: §A.1 · p. 18
  • Multiple-choice questions receive a prompt to think step by step and end with the answer letter, whereas open-ended questions receive no extra prompt.Source: §A.1 · p. 18
  • The examples compare generation with and without CompLLM across open-ended and multiple-choice Q&A datasets.Source: §A.1 · p. 18

A.1.1 EXAMPLE FROM NARRATIVEQA

The NarrativeQA example asks how Charlie communicates with the Angels, using a context synopsis that mentions speakers in his offices. Both generated answers give that core answer, while the CompLLM version adds details.

A.1.1 EXAMPLE FROM NARRATIVEQA

  • The context synopsis says Charlie communicates with the Angels through a speaker in his offices.Source: §A.1.1 · p. 18
  • The question asks how Charlie communicates with the Angels, and the ground-truth answer is that he uses speakers in his offices.Source: §A.1.1 · p. 20
  • Both generations identify a speaker in Charlie's office; the CompLLM answer adds remote and audio-only details.Source: §A.1.1 · p. 20

A.1.2 EXAMPLE FROM SQUAD

Harvard was named the second most commonly chosen “dream college” by students and parents in the 2013 Princeton Review poll.

A.1.2 EXAMPLE FROM SQUAD

  • Harvard consistently topped the Academic Ranking of World Universities from 2003 and THE World Reputation Rankings from 2011.Source: §A.1.2 · p. 20
  • Harvard ranked first every year in the THE-QS World University Rankings partnership from 2004 through 2009.Source: §A.1.2 · p. 20
  • In 2013, Harvard was the second most commonly named “dream college” by students and parents in the Princeton Review poll.The generated answer identifies the response as second place.Source: §A.1.2 · pp. 20–21

A.1.3 EXAMPLE FROM RACE

The example asks why Tencent and Alibaba launched lotteries, and the generated answer selects attracting more app users as the main business purpose.

A.1.3 EXAMPLE FROM RACE

  • The passage describes Tencent and Alibaba competing in the red-envelope market and launching lotteries to attract public attention.Source: §A.1.3 · p. 21
  • The answer without CompLLM likewise identifies attracting more users as the most plausible purpose of the lotteries.It frames the lotteries as a way to increase platform engagement and user numbers.Source: §A.1.3 · p. 21
  • The generated answer with CompLLM selects gaining more new app users as the lotteries’ main business purpose.It also says the companies aim to gain a larger share of the market.Source: §A.1.3 · p. 22

A.1.4 EXAMPLE FROM QUAIL

The example asks when the narrator imagined nurses giving pain relief to a boy, and both generated answers place it after the conversation with his parents.

A.1.4 EXAMPLE FROM QUAIL

  • The boy was banging his head because of earache, and his current episode had lasted three hours.His father’s account distinguished the current episode from an earache five years earlier.Source: §A.1.4 · p. 22
  • The narrator imagines the nurses giving the boy pain relief in the hospital, with his parents realizing they could have given him painkillers.Source: §A.1.4 · p. 22
  • Both generated answers say the imagined hospital scene occurs after the narrator has talked with the parents.The ground-truth answer is option A, “after talking to the parents.”Source: §A.1.4 · pp. 22–23
Loading 2509.19228v2…