Source-linked AI summary
Read As Human: Compressing Context via Parallelizable Close Reading and Skimming
Jiwei Tang, Shilei Liu, Zhicheng Zhang, Qingsong Lv, Runsong Zhao, Tingwei Lu, Langming Liu, Haibin Chen, Yujin Yuan, Hai-Tao Zheng, Wenbo Su, Bo Zheng
TL;DR
Long-context LLM use is challenged by computational inefficiency and redundant information. RAM applies parallel query-aware hybrid compression, retaining relevant segments and vectorizing less relevant ones, and reports stronger benchmark performance with substantial speed gains.
Problem
Long-context deployment faces quadratic self-attention costs and information redundancy, creating a need for efficient compression that preserves query-relevant content.
Method
RAM partitions context into segments, encodes them with the query in parallel, retains highly relevant text, compresses less relevant segments into vectors, and trains with contrastive learning.
Results
RAM consistently outperforms strong baselines across multiple QA and summarization benchmarks and delivers approximately 12× end-to-end speedup on long inputs.
Takeaways & Limitations
RAM provides an efficient and interpretable hybrid representation that preserves verbatim tokens for high-relevance segments while using compact representations for low-relevance ones.
Takeaways & Limitations
Contrastive-learning labels for NarrativeQA are generated by Qwen3-235B-A22B-Instruct and are not guaranteed to be fully accurate.
Abstract
from arXiv · showhide
Large Language Models (LLMs) demonstrate exceptional capability across diverse tasks. However, their deployment in long-context scenarios is hindered by two challenges: computational inefficiency and redundant information. We propose RAM (Read As HuMan), a context compression framework that adopts an adaptive hybrid reading strategy, to address these challenges. Inspired by human reading behavior (i.e., close reading important content while skimming less relevant content), RAM partitions the context into segments and encodes them with the input query in parallel. High-relevance segments are fully retained (close reading), while low-relevance ones are query-guided compressed into compact summary vectors (skimming). Both explicit textual segments and implicit summary vectors are concatenated and fed into decoder to achieve both superior performance and natural language format interpretability. To refine the decision boundary between close reading and skimming, we further introduce a contrastive learning objective based on positive and negative query-segment pairs. Experiments demonstrate that RAM outperforms existing baselines on multiple question answering and summarization benchmarks across two backbones, while delivering up to a 12x end-to-end speedup on long inputs (average length 16K; maximum length 32K).
1 Introduction
Long-context LLM deployment is limited by computational inefficiency and information redundancy. RAM addresses both through parallel, query-aware hybrid reading that retains relevant segments and compresses less relevant ones.
- Transformer self-attention has quadratic time complexity with input length, while long contexts also contain redundancy that can degrade performance.
- Task-agnostic compression can lose query-relevant information because it lacks query guidance, motivating task-aware compression.
- RAM partitions context into segments, encodes them with the query in parallel, and selects close reading or skimming according to relevance.
- Highly relevant segments remain as natural-language text, whereas less relevant segments are compressed into query-guided representations.
- RAM uses contrastive learning on positive and negative query-segment pairs to improve the decision boundary between close reading and skimming.
- RAM achieves superior performance across QA and summarization benchmarks and approximately 12× end-to-end speedup on long inputs.The reported inputs average 16K tokens and reach a maximum of 32K tokens.
2 Related Work
Context compression methods are divided into task-agnostic and task-aware approaches. RAM is motivated by efficiency limitations in existing task-aware methods and combines parallel query-aware processing with adaptive retention or compression.
- Task-agnostic methods compress context without specific queries, using hard prompt compression or soft prompt compression.
- RAM instead encodes segmented context and the query in parallel, then adaptively retains relevant segments and compresses less relevant ones.
- Hard prompt compression retains explicit tokens through metric-based pruning or query-agnostic summarization, whereas soft compression maps context to non-lexical embeddings.
- Task-aware methods incorporate the query to retain task-relevant content through summaries, token merging, or low-relevance token pruning.
- Existing task-aware methods either load the full sequence at once or use iterative autoregressive compression, hindering efficiency.
3 Method
RAM compresses long contexts through query-aware parallel encoding and adaptive hybrid representations. It retains highly relevant segments as text while replacing less relevant segments with query-guided vectors, and trains relevance selection with language modeling and contrastive objectives.
- 3.1 Query-Aware Parallel Encoding: RAM encodes context segments alongside the query in parallel, avoiding full-sequence attention and iterative compression.A shared encoder produces contextualized hidden states for the segmented context and query.
- 3.1 Query-Aware Parallel Encoding: Segment representations are formed by mean-pooling last hidden states, with the query represented in the same way.For each segment, the representation averages its token-level hidden states; the query representation is constructed analogously.
- 3.2 Adaptive Compression and Training: Highly relevant segments remain verbatim, whereas less relevant segments become single query-guided vectors in the original segment order.A trainable alignment matrix bridges the explicit token and implicit vector representations, producing a mixed compressed context.
- 3.2 Adaptive Compression and Training: RAM uses cosine similarity and softmax to assign query-relevance probabilities, then retains the top-k segments while compressing the others.The retention count is determined from a sampled compression rate, and non-retained segments are sent to skimming.
- 3.2 Adaptive Compression and Training: The decoder trains on the compressed memory, query embeddings, and answer tokens with teacher-forced language modeling.The language-modeling loss is computed over answer positions.
- 3.2 Adaptive Compression and Training: A contrastive loss encourages higher query similarity for positive segments and lower similarity for negative segments, improving relevance discrimination.The total objective combines language modeling and contrastive terms; the framework targets performance, efficiency, and interpretability.
4 Experiment
RAM is evaluated across multiple QA and summarization benchmarks, backbones, compression settings, and input lengths. It consistently improves performance and efficiency while remaining robust under aggressive and varying compression.
- Main Results: RAM consistently outperforms existing baselines on all benchmarks under both 4× and 8× compression constraints.It achieves state-of-the-art EM and F1 results most of the time.
- Main Results: RAM maintains stable performance across LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct backbones and under 4× and 8× compression.This demonstrates adaptability across model architectures and compression constraints.
- Main Results: Although trained on inputs up to 20K tokens, RAM outperforms the Original Prompt and other baselines on NarrativeQA inputs up to 32K tokens.The result indicates length extrapolation beyond the training regime.
- Efficiency Analysis: RAM’s compression stage splits context into segments, encodes them with the query in parallel, and adaptively retains or compresses each segment.Highly relevant segments are retained, while less relevant segments become compact vectors before decoding.
- Efficiency Analysis: Segment-wise encoding has complexity O(Lseg·Lorg), substantially below the O(Lorg^2) cost of full-sequence encoding when Lseg ≪ Lorg.Query-guided attention adds only O(N) overhead for N segments.
- Efficiency Analysis: Under the same 32× compression constraint, RAM achieves significantly lower end-to-end latency than task-relevant baselines.Latency evaluation uses NarrativeQA inputs averaging 16K tokens and reaching 32K tokens.
- Robustness Across Compression Rates: Across compression rates from 2× to 32×, RAM maintains a clear performance advantage while Provence, LongLLMLingua, and EXIT decline as compression becomes more aggressive.All methods use Qwen3-4B as the backbone in this comparison.
- Experimental Design: The experiments include ablations of skimming, close reading, and alternative skimming, alongside segment-size and contrastive-learning analyses.The evaluation covers four QA benchmarks and MultiNews summarization, with EM and F1 used as reported metrics.
5 Conclusion
RAM is a task-aware parallel context compression framework inspired by human reading, combining close reading for relevant segments with skimming for less relevant ones. Experiments show strong effectiveness and efficiency gains.
- RAM adaptively compresses query-irrelevant segments while preserving important context through close reading.It combines explicit and implicit representations in a parallel compression framework.
Limitations
RAM’s contrastive-learning labels for NarrativeQA are generated by Qwen3-235B-A22B-Instruct, but these labels are not guaranteed to be fully accurate.
- Contrastive-learning labels for NarrativeQA are generated by Qwen3-235B-A22B-Instruct and may contain inaccuracies.The labels are described as high-precision but not fully guaranteed to be accurate.
- Despite this labeling limitation, contrastive learning based on the labels remains effective in Table 3.
A Impact of Segment Size
Under 4x compression with Qwen3-4B-Instruct, RAM remains strong and stable across three QA benchmarks as segment size varies. Low standard deviations indicate minimal performance variation across segment granularities.
- Under 4x compression, RAM achieves strong and stable results across three QA benchmarks with varying segment sizes.Qwen3-4B-Instruct is used as the backbone.
- Low standard deviations indicate minimal performance variance across different segment granularities.The passage presents this as evidence of robustness and reliability in practical deployment scenarios.
B Prompt Template of Annotation
The annotation prompt instructs Qwen3-235B-A22B-Instruct to classify NarrativeQA segments according to whether they help generate the ground-truth answer.
- Qwen3-235B-A22B-Instruct labels each NarrativeQA segment positive when it helps generate the ground-truth answer.
- Segments that do not help generate the ground-truth answer are labeled negative.
C Dataset Details
The paper evaluates RAM on question-answering and summarization benchmarks spanning multi-hop reasoning, long-document understanding, and multi-document information consolidation. These datasets provide varied settings for testing context compression.
- Question answering: NaturalQuestions pairs Google search queries with entire Wikipedia pages and requires identifying both long and short answers.The processed version uses 20 related documents per question, with one containing the correct answer.
- Question answering: HotpotQA evaluates multi-hop reasoning across Wikipedia articles and requires supporting facts to explain answers.The paper evaluates on its validation set.
- Question answering: 2WikiMQA uses Wikipedia and Wikidata to construct questions requiring synthesis across up to four documents.It covers compositional, inference, comparison, and bridge reasoning, and the paper uses its test set.
- Question answering: NarrativeQA tests understanding of complete books and movie scripts through questions requiring long-range dependencies.The collection contains 1,572 documents and 46,765 question-answer pairs.
- Summarization: MultiNews evaluates multi-document summarization using news articles paired with human-written summaries.The benchmark tests consolidation of redundant information and resolution of conflicting details across sources.
D Ethic Statement
The ethics statement says RAM uses publicly available benchmarks and open-source platforms under appropriate licenses and introduces no additional ethical concerns beyond existing context-compression approaches.
- Data and models: RAM uses data and models from publicly available benchmarks and open-source platforms under appropriate licenses.
- Risk assessment: The authors state that RAM does not introduce new ethical risks beyond existing context compression approaches.
- Risk assessment: The paper concludes that no additional ethical concerns require specific attention.
E Language Model Usage Statement
The authors used a large language model to assist with academic writing while retaining responsibility for the work’s scientific contributions.
- Language model assistance: A large language model assisted with wording, organization, readability, method descriptions, and mathematical exposition.
- Author responsibility: The authors state that the scientific contributions, experimental setup, and reported results were conceived, executed, and verified by them.
- Author responsibility: The assistance was described as editing and exposition support rather than authorship of the scientific work.