PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentPress: Context Compression Beyond Text and Vision

Zhengze Zhou, Hejian Sang

arXiv:2609.01507v2cs.LGcs.AI

TL;DR

Long contexts are commonly compressed into text or reconstructed images, although language models may not require human-readable context. LatentPress writes histories and documents into continuous memory tokens for direct reading by a frozen decoder, achieving preserved or improved accuracy and faster context handling across LongMemEval and LongBench-QA with a small trainable footprint.

  • Problem

    Long-context systems typically keep the machine-facing representation as discrete text or reconstruct text from another modality, motivating a direct continuous interface.

  • Method

    LatentPress trains a small reader-matched writer to compress histories and documents into continuous memory tokens consumed directly by a frozen decoder.

  • Results

    Across LongMemEval and LongBench-QA, compressed readers preserve or improve accuracy at moderate compression, while writing and reading are substantially faster than text-based alternatives or raw context.

  • Takeaways & Limitations

    Direct soft tokens provide a practical machine-facing context interface beyond text and vision while keeping the downstream decoder frozen.

  • Takeaways & Limitations

    Compression rates are hand-specified rather than learned, and LongMemEval isolates compression and reading using oracle evidence sessions.

Abstract

from arXiv · show

Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference. A small reader-matched writer compresses $4$-$16\times$ while training only an adapter (4.2M-26.2M parameters, $\sim\!0.1\%$ of the decoder). On LongMemEval, LatentPress reaches $0.504$ accuracy at $7.70\times$ compression versus $0.490$ for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at $4$-$8\times$ compression, while $16\times$ trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is $5$-$9\times$ faster than raw context or cached OCR. We validate the interface under two transfer settings, zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived QA to unseen LongBench document domains, establishing direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation of the experiments could be found at: https://github.com/HJSang/LatentPress .

1 Introduction

LatentPress asks whether long conversational or document context can be represented as compact continuous tokens that a frozen language model reads directly, without reconstructing text. It presents a reader-matched WRITE/READ interface and evaluates accuracy, efficiency, transfer, and trainable footprint across memory and document tasks.

  • Motivation: LatentPress targets the gap between long-context demands and text-based machine interfaces by writing context into compact continuous representations read without text recovery.The motivation covers conversational histories and long documents, where systems otherwise retrieve, summarize, prune, or reconstruct text.
  • Interface: The interface separates WRITE, which maps text to a compact state, from READ, which supplies that state to a frozen decoder for downstream question answering.The abstraction applies to conversational histories and long documents rather than replacing retrieval or other complete memory-system components.
  • Contribution: LatentPress maps text segments into continuous vectors that enter a frozen decoder through its input-embedding interface, followed by the question.A small reader-matched adapter is trained while the downstream decoder remains frozen.
  • Design scope: The compression schedule is hand-specified: conversational turns use structure-based rates, whereas unstructured documents use a uniform rate.Learning the allocation automatically is left for future work.
  • Positioning: LatentPress differs from prior methods by combining a frozen reader, direct soft-token consumption, variable-length compression, and coverage of multi-turn histories and whole documents.Unlike reconstruction-based interfaces, it consumes vectors directly at the decoder input-embedding layer.
  • Experiments: 0.48–0.50 accuracy at 4.6–7.7× compression is reported for LongMemEval across three frozen readers, while LongBench-QA in-domain readers match or exceed raw-context baselines at mild compression.Both document transfer settings degrade at the most aggressive compression rate.

2 LatentPress

LatentPress uses a small writer to pool context into reader-space soft tokens, which a frozen decoder consumes directly before the question. Its fixed compression rules preserve structure for dialogue, use uniform pooling for documents, and train the writer through reconstruction or distillation-based supervision.

  • Positioning: Table 1 positions LatentPress against continuous-vector compression methods by contrasting full fine-tuning, autoencoding text reconstruction, reader freezing, and direct vector consumption.The table caption defines FT as full fine-tuning and autoenc. as decoding memory vectors back to text before answering.
  • 2.1 Direct-read soft context: LatentPress defines a direct-read interface in which a small writer maps context to continuous vectors and a frozen decoder reads them through its input-embedding interface.The compressed vectors are placed before the embedded question.
  • 2.1 Direct-read soft context: The writer fuses literal input embeddings with context-aware abstractions, pools the resulting vectors into shorter reader-space soft tokens, and injects them without changing decoder weights.LatentPress does not decode the vectors back to text at inference; writing is a single forward pass.
  • 2.2 Choosing compression rates: The rule π specifies how many neighboring positions are pooled into each soft token, with uniform pooling for documents and role-based pooling for conversational memory.User turns retain raw embeddings with k_user = 1, while assistant turns use k_assistant ∈ {8, 16, 32}.
  • 2.3 Small writer and bottleneck supervision: Only the writer head is trained while borrowed reader layers and the decoder remain frozen, yielding backbone-dependent trainable sizes from 4.196M to 26.220M parameters.The reported sizes include 12.849M for Qwen2.5-7B, 16.781M for Qwen3-8B, 4.196M for Qwen3-1.7B, and 26.220M for Qwen2.5-14B.
  • 2.3 Small writer and bottleneck supervision: The writer is trained using reconstruction and forward-KL supervision, with task adaptation optionally using QA examples from another domain or the target-domain training split.The reconstruction term targets output recovery, while forward KL distills full-context next-token behavior.

3 Conversational Memory

LatentPress compresses conversational evidence into soft tokens for frozen readers and preserves LongMemEval accuracy under zero-shot transfer. Role-aware compression is stable across readers and outperforms uniform pooling, OCR compression, and text summaries.

  • Evaluation setup: LongMemEval evaluates 500 oracle-evidence questions with frozen readers, reporting accuracy and the ratio of original text tokens to compressed vectors.The writer is trained on 2,000 generic UltraChat conversations without QA labels and transfers zero-shot to held-out memory-QA conversations.
  • Accuracy and compression: 0.504 LongMemEval accuracy at 7.70× compression exceeds the 0.490 uncompressed oracle-evidence baseline.Role-aware LatentPress also reaches 0.476 and 0.478 at 4.62× and 6.27× compression, respectively.
  • Baselines: 0.426 to 0.312 DeepSeek-OCR accuracy declines across its evaluated curve, while text summarization is weakest on every reader.LatentPress stays close to or ahead of the visual baseline across the evaluated readers.
  • Compression schedule: The role-based schedule keeps user turns lossless while pooling assistant turns, preserving short answer-bearing facts during compression.Uniform pooling of every turn remains lower over the evaluated range.
  • Accuracy and compression: Role-aware LatentPress improves over uniform pooling by +0.34 to +0.45 in overall accuracy across three Qwen readers.The comparison spans Qwen2.5-7B, Qwen3-8B, and Qwen3-1.7B with reader-specific compressor heads.

4 Generalization to Long-Document QA

LongBench-QA tests LatentPress without conversational role structure under cross-domain transfer and in-domain adaptation. In-domain writers match or exceed raw-context scores at mild compression, whereas 16× compression consistently falls below raw performance.

  • Evaluation setup: LongBench-QA removes conversational role structure and evaluates six English long-document subsets with uniform compression.The experiments compare cross-domain training against target-domain task adaptation on Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B.
  • In-domain task adaptation: In-domain LatentPress exceeds uncompressed performance at milder rates but drops below it at 16× across all three readers.The pattern appears in the LongBench-QA accuracy–compression frontiers under target-domain QA supervision.
  • Cross-domain QA transfer: Cross-domain transfer roughly matches raw reading at low compression but does not consistently beat it as compression increases.Across all three readers, 4× is preferred and accuracy declines with greater compression; Qwen2.5-7B reaches 45.13 versus raw 43.80 at 4×.
  • In-domain task adaptation: 57.99 and 52.18 at 4× and 8× exceed Qwen2.5-14B’s raw 47.93, but 40.30 at 16× falls below it.Qwen2.5-7B reaches 49.06 at 4×, 43.77 at 8×, and 37.78 at 16× versus raw 43.80.
  • In-domain task adaptation: 39.62 and 36.93 at 4× and 8× exceed Qwen3-8B’s raw 30.80, while 26.12 at 16× falls below it.The most aggressive rate exposes the cost of losing verbatim detail.

5 Efficiency

LatentPress separates context efficiency into writing encoded tokens and reading them with a frozen decoder. It substantially reduces both costs relative to raw-context and OCR-based routes, while comparisons remain tied to specific evaluated settings.

  • Write cost: 43 ms per conversation is LatentPress’s write cost, versus 844–1056 ms for batched DeepSeek-OCR reconstruction.Writing uses a single forward pass rather than autoregressive generation or OCR reconstruction.
  • Scope: These speedups are specific to the evaluated models, output lengths, and batching regimes, and do not compare against all one-pass soft-token methods.The reported comparisons focus on reconstruction-based routes.
  • Read cost: 0.43–0.49 seconds per example makes LatentPress 5.0–9.2× faster than raw inference at the f8 operating point.The comparison uses warm-loaded inference-only latency on 30 LongBench-QA examples across three readers.
  • Read cost: 5.5–9.4× faster than cached DeepSeek-OCR, LatentPress reads compressed prefixes in 0.43–0.49 seconds per LongBench-QA example.DeepSeek-OCR uses a precomputed base_size=640 cache in this comparison.
  • End-to-end cost: 6.0–13.7× shorter whole-job time than cold-cache DeepSeek-OCR is achieved by in-domain LatentPress on LongBench-QA.This coarser comparison includes adapter training, prediction, and evaluation for LatentPress, versus OCR reconstruction, prediction, and evaluation for DeepSeek-OCR.

6 Limitations and Future Work

LatentPress isolates the stored-context interface rather than implementing a complete memory system. Its simple hand-specified compression policies leave adaptive allocation and broader system integration for future work.

  • Scope boundary: Oracle evidence sessions isolate compression and reading in LongMemEval, leaving retrieval, memory updates, and conflict resolution to full memory systems.The study focuses on the representation interface between stored context and a frozen reader.
  • Scope boundary: Retrieval, memory updates, and conflict resolution are not addressed by LatentPress itself.These components are left to memory systems built on top of the interface.
  • Future work: Hand-specified role-based and uniform compression rates constrain the current system, motivating learned per-segment allocation.A learned policy could preserve detail selectively under latency or memory budgets.
  • Future work: Future extensions include learned token-wise fusion and writers trained for additional readers or non-text context.The paper names tool, multimodal, and embodied traces as possible targets.

7 Related Work

Prior context compression methods use continuous vectors, token pruning, or visual reconstruction, but differ in reader training, representation interface, and pipeline structure. LatentPress keeps the reader frozen and supplies directly readable soft tokens for histories and documents.

  • Soft-token compression: Gist, AutoCompressor, and ICAE adapt an LLM-scale reader or encoder, unlike LatentPress’s frozen reader and small reader-matched adapter.xRAG also freezes the reader but compresses one retrieved passage into a single token.
  • Soft-token compression: LatentPress compresses multi-turn histories and whole documents end-to-end without a retriever.Its vectors are consumed directly by the frozen decoder rather than reconstructed as text.
  • Token pruning: Selective Context and LLMLingua drop or select prompt tokens, while TIP emphasizes that token-level training value is highly nonuniform.These methods represent pruning or selection rather than direct continuous-vector reading.
  • Visual compression: DeepSeek-OCR reconstructs text optically, whereas LatentPress writes vectors that the same frozen decoder reads directly.Reconstruction-based visual compression adds an autoregressive OCR stage before language-only answering.
  • Conversational memory: Conversational memory systems typically store and retrieve text, while LatentPress provides a complementary learned soft-token representation.The representation could sit inside retrieval, reflection, or summarization pipelines.
  • Latent reasoning: LatentPress targets input history compression, whereas Coconut and CRISP compress or shorten reasoning traces.LatentPress does not alter what the reader generates.

8 Conclusion

LatentPress demonstrates that context can be stored as continuous memory tokens and read directly by a frozen decoder rather than kept as text or reconstructed from images. Across conversational memory and long-document QA, it combines compressed-context accuracy with low writing and reading costs, while dynamic compression remains a next step.

  • Conclusion: Continuous memory tokens let a frozen decoder read compressed conversational histories and long documents directly through its embedding interface.The approach avoids text reconstruction at inference and trains only a small reader-matched adapter.
  • Conclusion: Across LongMemEval and LongBench-QA, compressed readers preserve or improve accuracy while encoded tokens are generated near real time.The decoder remains frozen and the trainable state is limited to a small adapter.
  • Conclusion: Reading compressed prefixes is much faster than reading raw context or cached OCR, satisfying the paper’s practical efficiency criteria.Dynamic compression is identified as a natural next step for pushing compression further.

A Implementation and Training Details

LatentPress trains a small reader-matched compressor while keeping the decoder frozen, using distinct supervision and compression settings for conversational and document QA. Its baselines reconstruct text or use pooled embeddings before the same frozen reader answers.

  • Writer and reader: A linear adapter maps features from deep-copied bottom transformer layers into soft tokens, while the downstream decoder and borrowed encoder remain frozen.The adapter starts from the identity and is the only trainable component in the described writer.
  • Training and evaluation: LongMemEval trains writers on UltraChat and evaluates zero-shot memory QA, while LongBench-QA uses LongMemEval-derived QA for cross-domain transfer or target-domain QA for adaptation.LongBench evaluation covers six English subsets, with five-seed mean±standard deviation reporting for in-domain runs.
  • Compression schedule: Conversation compression uses lossless user turns and assistant rates of 8, 16, or 32, whereas document experiments use uniform factors of 4, 8, or 16.Reported conversational ratios are means over the evaluation set rather than preset budgets.
  • Baselines: The text baseline supplies an abstractive summary to the frozen reader, while the visual baseline renders pages and uses DeepSeek-OCR to reconstruct text before answering.The LongBench visual pipeline follows the same reconstruction-then-text-reader protocol as the LongMemEval visual baseline.
  • Ablations: At 4.62× compression, direct embedding pooling reduces accuracy from 0.476 to 0.325, and removing assistant turns lowers it further to 0.217.The ablation uses the same frozen Qwen2.5-7B reader and evaluation protocol across variants.

C.2 Encoder-Training Ablation

Freezing the borrowed encoder layers is preferable to fine-tuning them for LatentPress: it improves accuracy across tested rates and avoids degradation at high compression. Judge-free token-F1 preserves the main ranking, while text summaries fail especially on answer-bearing categories.

  • Encoder-training ablation: Freezing the encoder beats fine-tuning it by +0.018 to +0.066 absolute accuracy at every tested assistant rate.The authors attribute fine-tuning’s weaker behavior to apparent overfitting on UltraChat and retain frozen layers thereafter.
  • Judge-free validation: Token-level F1 agrees with judge accuracy in ranking methods, with role-aware LatentPress reaching 0.251 versus DeepSeek-OCR’s 0.160 at ka=32.Absolute F1 is low because generated explanatory sentences dilute precision against short gold spans.
  • Text-summary breakdown: Text summarization preserves abstention accuracy but reduces temporal and multi-session accuracy to 0.016–0.041 on smaller readers.The breakdown attributes this collapse to abstractive summaries discarding precise user facts required by those questions.
  • Qualitative behavior: At 4–6× compression, qualitative examples retain names, durations, prices, and weekdays while correctly abstaining when queried facts were never stated.The examples use the role-aware compressor with ka=8 and a Qwen3-8B reader.

D.3 Failure Modes

LongBench-QA compression failures increase with compression and include both output pathologies and genuine information loss. The failures therefore affect formatting and decoding as well as answer semantics.

  • Failure modes: Six recurring failure modes include unanswerable collapse, format artifacts, blank output, repetition loops, reasoning-template leakage, and semantically wrong short answers.The first five are decoding or formatting pathologies; the last reflects information loss in compressed memory.
  • Compression effects: Repetition loops and the other first five pathologies become more common as compression increases, especially at 16×.The passage distinguishes these output failures from the separate category of genuine semantic information loss.
  • Evaluation integrity: The reported degradations are content and format changes rather than evaluation-harness artifacts because JSONL schema and scoring match the raw runs.The table caption and accompanying text explicitly make this comparison.

E.1 LongBench-QA Wall-Clock Comparison

End-to-end cold-cache measurements favor LatentPress over the DeepSeek-OCR pipeline on LongBench-QA. The comparison includes training or reconstruction, prediction, and official evaluation, with OCR also assessed under cache amortization.

  • Measurement scope: Cold-cache totals measure the path from raw context to scored prediction, including LatentPress adapter training or OCR reconstruction, reader prediction, and evaluation.Measurements were collected once on one NVIDIA H100 80GB GPU without variance estimates.
  • Cold-cache comparison: LatentPress is 6.0–13.7× faster than DeepSeek-OCR under cold-cache accounting.The largest stated gap is 9.9× for Qwen2.5-14B b512 versus the nearest LatentPress f8 point at 7.997×.
  • Cache amortization: OCR cache generation is reader-independent and is also amortized across four readers, yet LatentPress remains faster under that favorable accounting.Qwen3-8B OCR is reported only as a cold-cache total because its cache and reader times were not separated.
Loading 2609.01507v2…