PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC-Encoder: learning compressed text representations for large language models

Hippolyte Pilchen, Edouard Grave, Patrick Pérez

arXiv:2510.20535v2cs.CLcs.AI

TL;DR

Longer contexts increase inference costs and can exceed decoder context windows, while many soft-compression methods require adapting the target decoder. ARC-Encoder replaces token embeddings with pooled continuous representations; it reports strong results across tasks and domains, including near-full-text accuracy at a 4× pooling factor.

  • Problem

    Long contexts raise inference costs and can dilute useful information or exceed decoder context windows, while many soft-compression methods require decoder adaptation and a separate encoder per decoder.

    Source: §1 · p. 1

  • Method

    ARC-Encoder pools text into continuous representations that replace decoder token embeddings, using a shared encoder with decoder-specific projectors for multiple decoders.

    Source: §3.1 · p. 3, §3.3 · p. 4

  • Results

    ARC-Encoder reports strong results across tasks and domains, achieving nearly the same accuracy as a full-text decoder at a 4× pooling factor.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Reducing context length while leaving the target LLM unchanged leads to faster prefilling and decoding stages.

    Source: §5 · p. 12

    The method depends strongly on fine-tuning task format, limiting novel-task adaptation and long-context summarization, and struggles at higher compression ratios.

    Source: §5 · p. 12

Abstract

from arXiv · show

Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .

1 Introduction

Longer contexts raise inference costs and can dilute useful information or exceed decoder context limits. ARC-Encoder compresses context without modifying the decoder, aiming to preserve performance while supporting multiple decoders and long-context use.

1 Introduction

  • Longer contexts increase inference costs through quadratic Transformer attention and can dilute useful information or exceed an LLM’s context window.Context compression is presented as a way to reduce input length while preserving semantics needed for generation.Source: §1 · p. 1
  • ARC-Encoder produces compressed representations that replace raw input without modifying the decoder, preserving strong performance across benchmarks and in-context learning.A shared encoder can support multiple decoders, and the method also targets context extension by compressing document chunks in parallel.Source: §1 · p. 2
  • A single encoder can work with multiple decoders, requiring less than 1% new parameters per LLM and minimal adjustment for new decoders.The contribution describes decoder-specific adaptation with a shared encoder.Source: §1 · p. 2
  • ARC-Encoder can extend decoder context size by compressing large-document chunks in parallel, with competitive long-context benchmark results.Source: §1 · p. 2
  • Both pretraining and fine-tuning are key to the approach, while compressed Wikipedia representations require memory on the same order as raw text.The paper notes that this memory scale allows the representations to be precomputed.Source: §1 · p. 2

2 Related work

Prior context-compression work includes hard token pruning and soft learned representations, as well as encoder-decoder and long-context approaches. ARC-Encoder is situated among soft methods while keeping the target decoder unchanged.

2 Related work

  • Earlier encoder-decoder systems include retrieval-focused Atlas, in-context-learning-oriented RAVEN, and a smaller-encoder approach using cross-attention.The passage notes that Zhang et al. (2025) experimented with combinations of different-sized models.Source: §2 · p. 2
  • Soft compression learns continuous representations, while hard compression removes prompt tokens; prior soft methods commonly fine-tune both encoder and decoder.The cited work describes gist and memory tokens as examples of soft compression.Source: §2 · p. 2
  • Other soft-compression approaches include merged tokens, precomputed text embeddings, and decoder-based compression into a single vector.The cited passage reports xRAG pooling factors up to ×150, while noting poor performance on certain benchmarks and limited flexibility.Source: §2 · p. 3
  • Long-context methods include positional interpolation, inserted activation anchors, and lightweight encoders that pass compressed inputs to a decoder.ARC-Encoder follows the lightweight-encoder approach while producing fewer tokens and keeping the decoder untouched.Source: §2 · p. 3

3 Method

ARC-Encoder combines a trainable text encoder and projector with a frozen decoder, pooling hidden states into continuous representations that replace token embeddings. Its training includes alternating reconstruction and continuation pretraining, task-specific fine-tuning, and a shared-encoder strategy for multiple decoders.

3.1 Architecture

  • The trainable encoder and two-layer MLP projector feed a frozen decoder, where compressed continuous representations replace token embeddings.The encoder is based on an LLM transformer, and pooling reduces the sequence from n elements to n/x for pooling factor x.Source: §3.1 · p. 3

3.2 Pooling method

  • ARC-Encoder pools hidden states directly, keeping a fixed pooling factor independent of input sequence length rather than a fixed number of output representations.Source: §3.2 · p. 3
  • In the final self-attention layer, consecutive queries are averaged while keys and values remain unchanged, producing fewer queries that attend to the uncompressed keys and values.Pooling earlier in the encoder led to poorer performance.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 Training

  • Pretraining alternates reconstruction and continuation because reconstruction alone can yield representations the decoder cannot exploit for downstream tasks.Continuation replaces text subsequences with compressed representations and teacher-forces the following text; learned <Cont> and <Rec> tokens identify the task.Source: §3.3 · p. 4, Figure 2 · p. 4
  • Fine-tuning specializes the encoder for tasks while keeping the decoder unchanged and can preserve few-shot or zero-shot abilities through suitable training templates.The pooling factor may differ between pretraining and fine-tuning, with a higher pretraining factor improving models.Source: §3.3 · p. 4
  • Multi-decoder training shares an encoder but specializes each decoder’s projector and special tokens, which account for less than 1% of encoder weights.Training samples a decoder uniformly and updates its projector alongside the shared encoder.Source: §3.3 · pp. 4–5

4 Experiments

The experiments evaluate ARC-Encoder on context compression, long-context understanding, and encoder design. Results show useful compressed representations with frozen decoders, alongside trade-offs in training, pooling, efficiency, memory, and domain adaptation.

4.2 Context compression

  • ARC-Encoder outperforms the closed-book baseline and ARC4-Encoder nearly matches open-book while delivering ×1.8 gains of prefilling FLOPs, without altering the decoder.It tends to outperform baselines more on reading comprehension and summarization, but is slightly behind PISCO-like methods on FLORES, where sequences are under 30 tokens.Source: §4.2 · p. 6, Table 1 · p. 7
  • The shared encoder loses less than 1.0 point on average versus specialized counterparts; adapting it to OLMo trains only 15M parameters, though its open-book score gap remains larger.For HotpotQA, ARC4-Encoder outperforms the open-book setup when the decoder is limited by its context window.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Long Context

  • ARC8-Encoder lets Llama2 Chat process up to 8× more input than its original context window and improves long-context QA, but underperforms on GovReport and QM-Sum.The authors attribute the task differences likely to alignment between the synthetic fine-tuning data and benchmark answer-length distributions; scaling to 128k tokens is left for future work.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Compression Ablations

  • Pretraining is essential: compared with fine-tuning alone, 20k pretraining steps improve average score by approximately +16 points and 80k steps by +19 points.Omitting reconstruction or using too little continuation causes substantial drops; reconstruction first aligns representations with the decoder embedding space, while continuation encourages downstream usability.Source: §4.4 · p. 9, Table 4 · p. 9
  • Averaging contiguous tokens is the default pooling choice; memory tokens worsen as sequence length grows, while k-means is harmful especially in translation.Pooling two layers earlier gives an identical average score at ×8, moving to the encoder middle drops average performance by over 1%, and at ×4 the default beats the earlier variant by 0.5%.Source: §4.4 · pp. 9–10, Table 6 · p. 10, Table 7 · p. 10

4.5 Profiling

  • ARC4-Encoder yields a 1.8× speed-up over natural text when compressing contexts on the fly, with compression costs amortized during decoder prefilling.Profiling measures CUDA time and TFLOPs for compression, prefilling, and decoding on one H100; compressed representations can also be precomputed on another GPU.Source: §4.5 · pp. 10–11, Figure 3 · p. 11

4.6 Memory Analysis

  • Combining pooling-factor changes, a smaller MLP bottleneck, and product quantization stores English Wikipedia in 80 GiB with minimal performance impact or 20 GiB while still improving the closed-book baseline.Raw English Wikipedia requires approximately 24 GiB, making the 20-GiB representation smaller than the source text.Source: §4.6 · p. 11, Figure 4 · p. 11

4.7 In-Domain and Out-of-Domain Analysis

  • Adding HotpotQA and SQuAD training data improves performance on those domains and overall average while preserving compression efficiency and competitive accuracy elsewhere; ARC-Encoder also performs strongly across tested biomedical pooling factors.The out-of-domain tests use BioASQ and PubMedQA, whose biomedical domain is absent from the fine-tuning mixture.Source: §4.7 · p. 12, Table 8 · p. 12, Table 9 · p. 12

5 Limitations & Conclusion

ARC-Encoder compresses text into representations that replace raw inputs, reducing context length while leaving the decoder unchanged. The paper reports efficiency and cross-domain generalization, but identifies task-format dependence and other limits; compressed vectors also do not guarantee confidentiality.

5 Limitations & Conclusion

  • ARC-Encoder remains highly dependent on fine-tuning task format, limiting adaptation to novel tasks, especially long-context summarization.It also struggles at higher compression ratios and has only been used for context extension on models with relatively small native context windows.Source: §5 · p. 12
  • ARC-Encoder replaces raw LLM inputs with compressed representations, reducing context length and enabling faster prefilling and decoding without changing the target LLM.The paper reports generalization across domains and adaptation to multiple decoders, with pretraining and fine-tuning both critical to success.Source: §5 · p. 12

Statement of Broader Impact

  • Compressed representations do not guarantee confidentiality and may remain susceptible to inversion or interpretation by specialized models.The authors advise caution in privacy-sensitive settings such as healthcare or legal analysis.Source: Statement of Broader Impact · p. 13

A.1 Context Compression with benchmark train sets

The appendix reports benchmark specialization, pooling-factor transfer, and encoder–decoder pairing results. It finds that adding benchmark training data can improve specialization, 8× pretraining transfers well to 4× and 8× pooling, and a shared encoder can serve multiple decoders.

A.1 Context Compression with benchmark train sets

  • Adding HotpotQA and SQuAD training sets lets ARC-Encoders specialize on those benchmarks, outperform the open-book baseline, and retain efficiency gains without harming other-task performance.The decoder remains unaltered.Source: §A.1 · p. 18, Table 10 · p. 18

A.2 Pooling factor generalization

  • Pretraining at 8× can outperform other pretraining–fine-tuning pairs at 4× and 8× pooling, while transfer to higher pooling factors generalizes poorly.The best ARC4-Encoder and ARC8-Encoder models were trained from 8×-pretrained models; a pooling factor of 32 averaged 33.1.Source: §A.2 · p. 18, Figure 5 · p. 18

A.3 Encoder-Decoder pairs

  • A single encoder can provide the same compressed representations to Llama3.1 8B and Mistral 7B decoders, outperforming the closed-book baseline.The reported compatibility is attributed to similar architectures and training pipelines; Table 11 compares backbones paired with different decoders using average Tab. 1 scores.Source: §A.3 · pp. 18–19, Table 11 · p. 19

A.4 Further baselines evaluations

The appendix compares ARC-Encoder with ICAE-like models and describes differences in how they generate compressed representations. At matched pooling factors, ARC-Encoder outperforms the ICAE-like models while using less than twice their encoding compute.

A.4 Further baselines evaluations

  • The appendix includes further baseline evaluations in the same setting and with the same specifics as Table 1.The supplied table caption does not provide additional result values.Source: Table 12 · p. 19

A.4.1 ICAE further comparisons

  • ARC-Encoder directly pools context tokens into a variable number of representations that scales with input length, unlike ICAE-style learned memory tokens, which produce a fixed number.ARC-Encoder encoders can start from any decoder-only LLM with removed layers and bidirectional attention, whereas ICAE encoders mirror their target decoder architecture.Source: §A.4.1 · p. 19
  • At matched pooling factors, ARC-Encoder outperforms ICAE-like models while requiring less than twice the encoding compute.For some benchmarks, ARC-Encoder also outperforms ICAE-like models at higher pooling factors.Source: §A.4.1 · p. 20, Table 13 · p. 20

A.4.2 Generalization on out-of-domain professional datasets

On biomedical QA datasets absent from fine-tuning data, ARC-Encoder achieves the best reported performance, including at 8× pooling. In long-context QA and summarization, its performance remains largely consistent across context-length ranges.

A.4.2 Generalization on out-of-domain professional datasets

  • On BioASQ and PubMedQA, which are absent from fine-tuning data, ARC-Encoder achieves the best performance even at 8× pooling, close to the open-book upper bound.The compared compressor models other than LLMLingua2 were fine-tuned on the same dataset.Source: §A.4.2 · p. 20

A.5 Context length performance scaling

  • ARC-Encoder performance remains largely consistent across context-length ranges for QASPER question answering and QMSUM summarization.The authors attribute this robustness to fine-tuning on samples of varying lengths.Source: §A.5 · p. 20
  • The reported length robustness concerns long-context evaluations grouped into approximately equal-sized sample bins by context length.The bins cover QASPER and QMSUM examples.Source: §A.5 · p. 20

A.6 Needle-in-Haystack analysis

ARC-Encoder enabled precise retrieval beyond Llama2 7B Chat’s 4096-token window, though retrieved answers were not always formatted exactly. Smaller encoder variants also delivered computational benefits with Llama 3.2 3B, and the theoretical analysis estimates lower relative costs for ARC4 and ARC8.

A.6 Needle-in-Haystack analysis

  • ARC-Encoder with pooling factor 8 enabled Llama2 7B Chat to retrieve precise information beyond its 4096-token window, with retrieval more consistent across context lengths and depths.Its top score was 7 because the retrieved answer paraphrased rather than exactly matching the needle.Source: §A.6 · p. 21, Figure 7 · p. 22

A.7 On targeting a smaller LLM

  • The 1.8B- and 1.1B-parameter encoder variants paired with Llama 3.2 3B stayed close to open-book performance and delivered computational benefits.The authors say encoder size should be adapted to the target LLM to preserve the performance–efficiency trade-off.Source: Table 15 · p. 22, §A.7 · p. 22, Figure 8 · p. 23

B Theoretical generation FLOPs

  • 1.5× smaller for ARC4-Encoder and 1.9× smaller for ARC8-Encoder in the main-table setting.These are the reported relative FLOP estimates for the theoretical generation-cost analysis.Source: B Theoretical generation FLOPs · p. 24

C Training details

The training appendix specifies ARC-Encoder’s default architecture and training choices, decoder adaptation, and the synthetic and supervised data used for compression pretraining and fine-tuning.

C.1 Default setting

  • The default encoder truncates Llama 3.2 3B’s last two layers, averages queries with pooling factor 8, and uses a two-matrix MLP projector.For Llama 3.1 8B, the last eight layers are truncated; projector dimensions depend on the encoder backbone.Source: §C.1 · p. 24
  • Continuation training prefixes up to 256 text tokens to the compressed sequence, while interleaving during pretraining costs more but improves multi-decoder generalization.After fine-tuning with interleaved few-shot samples, interleaving gives no gains in the specific-decoder setting; standard continuation can substitute during pretraining.Source: • Training · p. 24, • Fine-tuning · p. 25

C.2 Hyperparameters

  • Adding OLMo 7B to the multi-decoder ARC4-Encoder requires fine-tuning only the MLP projector, for 8k steps with 100 warmup steps and a 10^-4 maximum learning rate.Figure 10 illustrates this projector-only adaptation to a new decoder.Source: Figure 10 · p. 25, §C.2 · p. 26

C.3 Context Compression fine-tuning

  • Context-compression fine-tuning mixes generated translations in 12 languages with supervised QA, summarization, and reading-comprehension data.For QA, top-five NV-Embed passages are retrieved from KILT Wikipedia sequences and may be concatenated and compressed jointly; this particularly benefits HotpotQA and CNN, with minimal impact elsewhere.Source: §C.3 · p. 26, Table 18 · p. 27

C.4.1 Pretraining

  • Instruct-decoder pretraining continues to alternate continuation and reconstruction tasks using an instruction template around the compressed representations.The Llama2 Chat template places the prefix, compressed tokens, and instruction inside the instruction prompt, followed by the continuation suffix.Source: §C.4.1 · pp. 27–28

C.4.2 Fine-tuning

  • Fine-tuning QA examples use questions generated from book or paper passages, with the full book or paper as final context and additional examples from Wikipedia chunks.The synthetic data also includes hierarchical summaries of passage groups and paraphrases; contexts longer than 500k characters are truncated and those shorter than 1k are discarded.Source: §C.4.2 · p. 28

D.1 Baselines implementation

The baseline implementations adapt xRAG, ICAE, and PISCO to the paper’s evaluation setup and describe the LLMLingua2 model used; the authors also report translation weaknesses for xRAG.

D.1 Baselines implementation

  • LLMLingua2 uses the open-source multilingual XLM-RoBERTa-large meeting-bank model following LLMLingua’s instructions.Source: §D.1 · p. 29
  • xRAG’s projector is pretrained following its original setup, then fine-tuned with interleaved compressed contexts; the authors attribute partial translations or hallucinations to compressing a sequence into one vector.The appendix illustrates translation failures with generated and ground-truth examples.Source: §D.1 · pp. 29–30
  • ICAE is re-implemented with its original design choices, pretrained on the evaluated decoders for 100k steps, and fine-tuned on the authors’ dataset after poor generalization with its original dataset.The training curve had already converged at 100k steps, half the number reported in the original paper.Source: §D.1 · p. 29
  • PISCO is re-implemented with LoRA on both encoder and decoder, the authors’ custom fine-tuning data, and 8,000 fine-tuning steps.The custom dataset excludes evaluation train sets and adds summarization and synthesized translation data; variants use fixed 128-token input chunks.Source: §D.1 · pp. 29–30

D.2 Evaluation datasets

Evaluation covers QA and summarization, translation, and zero-shot long-context understanding, with task-specific benchmarks, metrics, and input settings.

D.2 Evaluation datasets

  • QA uses HotpotQA, Natural Questions, TriviaQA, and SQuAD with Exact Match; CNN-DailyMail summarization uses Rouge-L on 1,000 development samples.When ground-truth context is absent, top-five passages are retrieved to simulate RAG; the authors note Rouge-L aligns closely with human approval in this setting.Source: §D.2 · p. 30
  • Translation uses 992 FLORES samples and averages SacreBLEU scores across English-to-Danish, French, German, and Spanish in a 5-shot setting.The reported pooling factor is per-context compression, not compression of the full prompt.Source: §D.2 · pp. 30–31
  • Long-context evaluation uses the ZeroSCROLLS validation sets for NarrativeQA, QASPER, GovReport, and QM-Sum, following Yen et al.’s task formats and instructions.The evaluation uses the full validation sets, with sample counts of 3,461, 1,726, 973, and 272, respectively.Source: §D.2 · p. 31

Related papers

Loading 2510.20535v2…