PaperlayerView exact revision on arXiv

Source-linked AI summary

BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection

Zhengpei Hu, Kai Li, Dapeng Fu, Chang Zeng, Yue Li, Yuanhao Tang, Jianqiang Huang

arXiv:2603.19635v1cs.CL

TL;DR

Long-context expansion creates inference-latency and information-utilization bottlenecks, while existing compression methods can require training or fragment semantics. BEAVER introduces training-free, structure-aware hierarchical selection and achieves competitive long-context performance with a reported 26.4× speedup at 128k context.

  • Problem

    Longer contexts create self-attention latency and diminishing information utilization, while existing compression can require training or disrupt semantic and syntactic coherence.

  • Method

    BEAVER compresses context through training-free hierarchical segment-page selection using a Segmenter, PageEncoder, and QueryPlanner.

  • Results

    Across four long-context benchmarks, BEAVER performs on par with or above SOTA methods and achieves a 26.4× speedup over LongLLMLingua at 128k context.

  • Takeaways & Limitations

    BEAVER provides a hardware-efficient, plug-and-play framework for scalable long-document understanding without parameter updates.

  • Takeaways & Limitations

    BEAVER may retain minor redundancies, struggle with deep multi-hop reasoning lacking direct query overlap, and require manual hyperparameter adjustment across different settings.

Abstract

from arXiv · show

The exponential expansion of context windows in LLMs has unlocked capabilities for long-document understanding but introduced severe bottlenecks in inference latency and information utilization. Existing compression methods often suffer from high training costs or semantic fragmentation due to aggressive token pruning. In this paper, we propose BEAVER, a novel training-free framework that shifts compression from linear token removal to structure-aware hierarchical selection. BEAVER maximizes hardware parallelism by mapping variable-length contexts into dense page-level tensors via dual-path pooling, and preserves discourse integrity through a hybrid planner combining semantic and lexical dual-branch selection with sentence smoothing. Extensive evaluations on four long-context benchmarks demonstrate that BEAVER achieves comparable performance to state-of-the-art (SOTA) methods like LongLLMLingua. Notably, on the RULER benchmark, BEAVER maintains high fidelity in multi-needle retrieval where baselines deteriorate. Regarding efficiency, BEAVER reduces latency by 26.4x on 128k contexts, offering a scalable solution for high-throughput applications. Our code is available at https://cslikai.cn/BEAVER/.

1 Introduction

Longer LLM contexts improve long-document capabilities but create inference and information-utilization bottlenecks. BEAVER addresses these challenges with training-free, structure-aware hierarchical selection and reports competitive benchmark performance with substantial latency reduction.

  • Exponential context-window growth enables codebase analysis and global understanding across multiple long documents, but introduces practical deployment bottlenecks.
  • Self-attention’s O(L2) complexity causes prefill latency to increase with context length, producing excessive time to first token and tail latency.
  • Longer contexts can yield diminishing information utilization and expose models to the “Lost in the Middle” effect.
  • Existing compression methods either incur training overhead or use unstructured token pruning that disrupts semantic and syntactic coherence.
  • BEAVER replaces linear token-wise pruning with structure-aware hierarchical selection through a Segmenter, PageEncoder, and QueryPlanner, followed by smoothing.
  • Across four long-context benchmarks, BEAVER performs on par with or above SOTA methods, while achieving a 26.4× speedup over LongLLMLingua at 128k context.

2 Related Work

Prior prompt-compression work divides into unsupervised statistical methods and supervised or specialized learning methods. BEAVER differs by using training-free, structure-aware segment-page selection to reduce deployment and coherence limitations.

  • Hard prompt compression preserves discrete tokens for interpretability and compatibility and is organized into two primary paradigms.
  • Unsupervised methods use perplexity, generation probabilities, or self-information from off-the-shelf models to identify redundant tokens or chunks.
  • Supervised and specialized methods optimize compression objectives through token-level classification, data distillation, or reinforcement learning.
  • Learning-based approaches incur high deployment costs and limited transferability, while fine-grained token pruning can disrupt syntactic coherence.
  • BEAVER shifts from token-wise pruning to training-free segment-page selection while preserving discourse integrity and addressing fragmentation.

3 Methods

BEAVER compresses long contexts through hierarchical page selection rather than token-wise pruning. Its pipeline maps segmented text into page tensors, encodes pages with complementary pooling, selects pages using semantic, lexical, and structural signals, and smooths spans to preserve sentence completeness.

  • Overall Pipeline: BEAVER comprises a Segmenter, PageEncoder, and QueryPlanner that transform long-context compression into structure-aware hierarchical selection.The selected spans are restored into coherent compressed text for downstream inference.
  • Segmenter: The Segmenter greedily packs delimiter-defined logical segments into fixed-capacity pages, minimizing padding while preserving local semantic boundaries.Long segments are split across consecutive pages, and the resulting page index tensor supports standard matrix multiplication.
  • PageEncoder: The PageEncoder combines ITF-weighted average pooling for global semantics with max pooling for salient local features, then fuses page representations.Context-adaptive weighting reduces the influence of frequent, uninformative tokens and helps retain rare keywords.
  • PageEncoder: Short queries use dense retrieval, whereas longer queries retain multiple token representations through late interaction for complex matching.This query representation feeds page-level semantic and lexical relevance scoring.
  • QueryPlanner: The QueryPlanner linearly fuses normalized semantic similarity and lexical overlap, with λ controlling their relative importance across task types.Lexical matching emphasizes exact query-token overlap, while semantic scoring supports broader relevance.
  • QueryPlanner: Structural priors preserve anchor pages, enforce query-order constraints, and guide selected spans before sentence-level smoothing extends them to sentence boundaries.Overlapping spans are merged and concatenated into the final compressed context.

4 Experiments

BEAVER is evaluated against compression baselines across four long-context benchmarks, latency settings, and ablations. It combines strong task performance with lower latency, robust retrieval, generalization, and identifiable contributions from its hierarchical design.

  • Evaluation Setup: BEAVER is compared with unsupervised, supervised, specialized, and embedding-based compression methods across LongBench, ZeroSCROLLS, RULER, and L-Eval.Evaluations use strict 2,000- and 3,000-token budgets, with gpt-3.5-turbo-instruct as the downstream model and Qwen3-8B embeddings.
  • Benchmark Performance: 40.7 on LongBench single-document QA establishes BEAVER as a new SOTA, while ZeroSCROLLS performance remains comparable to LongLLMLingua at 32.0 versus 32.7.BEAVER also records the lowest latency of 3.0s and the highest speedup of 5.2× in the reported comparison.
  • Retrieval Robustness: 83.7 average on RULER substantially exceeds LLMLingua-2’s 47.9, with 100.0% accuracy on all single-needle tasks and 99% multi-needle recall.LongLLMLingua falls below 10% on the visualized needle tasks, while LLMLingua-2 reaches 86% and 72% on single- and multi-needle tasks.
  • Generalization: 57.6 average on L-Eval surpasses LongLLMLingua at 51.5 and LLMLingua-2 at 54.6, with BEAVER ranking first on four of six datasets.The evaluation includes unseen distributions and open-weights model generalization, with gains on SFictionQA and Legal Contract QA.
  • Efficiency: 26.4× speedup at 128k context comes from 1.20s compression latency versus about 31.7s for LongLLMLingua, while BEAVER’s latency grows linearly with a flatter slope.BEAVER also achieves 5.9× and 2.3× speedups over LLMLingua2 and LLMLingua-2-small, respectively.
  • Ablation Study: Ablations show that page size, dual-path pooling, multi-vector queries, lexical and semantic matching, in-context ITF, smoothing, and hybrid selection each affect performance.Small pages reduce the average score by 1.8 points, large pages by 2.4, lexical matching removal by 6.0, and sentence smoothing adds 1.6.

5 Conclusion

BEAVER is a hardware-efficient, training-free framework for compressing long-context prompts through hierarchical segment-page selection. It reports strong question-answering and retrieval performance alongside competitive summarization fidelity and a 26.4× speedup at 128k contexts.

  • Conclusion: BEAVER transforms irregular sequence compression into optimized tensor operations through a hierarchical segment-page mechanism.Its design uses unsupervised in-context ITF weighting and a structure-aware hybrid planner to retain global semantics and lexical details.
  • Conclusion: BEAVER establishes SOTA performance in question answering and fine-grained retrieval while maintaining highly competitive fidelity in summarization tasks.The framework is presented as avoiding semantic collapse associated with probability-based pruning.
  • Conclusion: 26.4× speedup on 128k contexts demonstrates BEAVER’s reported scalability advantage over learning-based baselines.The framework is also described as a plug-and-play module requiring no parameter updates.
  • Conclusion: BEAVER provides a practical and robust foundation for efficient large-scale long-document understanding.This conclusion follows the framework’s training-free, plug-and-play positioning.

Limitations

BEAVER trades fine-grained precision for structural coherence and speed, while facing challenges in deep multi-hop retrieval and cross-domain hyperparameter transfer.

  • Page-level selection may retain minor redundancies because it is less precise than fine-grained token pruning.The design prioritizes structural integrity and speed over maximal pruning precision.
  • Semantic and lexical retrieval may struggle when deep multi-hop reasoning requires intermediate deductions rather than direct query overlap.
  • Preset weighting factors may require manual adjustment across drastically different domains, unlike end-to-end models that adapt automatically.

Ethical Considerations

BEAVER's information pruning creates risks in high-stakes use, especially when compression removes qualifiers, warnings, or material from underrepresented linguistic and cultural contexts.

  • Information pruning may remove critical qualifiers or safety constraints, including negations and warnings, from compressed contexts.The risk is particularly consequential for legal or medical analysis, where omissions could produce incorrect or unsafe downstream generations.
  • Users should verify BEAVER outputs before applying them to high-stakes decision-making tasks.Structure-aware design and sentence smoothing aim to mitigate fragmentation but do not eliminate the need for caution.
  • Embedding-model biases may propagate into page-importance scores and systematically filter text associated with minority groups.The paper identifies under-represented dialects and cultural contexts as examples requiring future fairness investigation.

B.1 Unsupervised Statistical Methods

Prompt compression methods include statistical approaches that prune content using off-the-shelf model signals and learned approaches that train specialized compression models.

  • Selective-Context and LongLLMLingua use self-information metrics to identify and prune redundant tokens or chunks.These unsupervised methods avoid specialized training but rely on heuristic information-theoretic metrics.
  • LongLLMLingua adds question-aware coarse-to-fine compression, document reordering, and post-compression subsequence recovery for long-context scenarios.
  • LLMLingua uses a budget controller and token-level iterative compression, while LLMLingua-2 frames compression as token classification with bidirectional context.
  • Sentence-BERT derives sentence embeddings through siamese and triplet structures, enabling cosine-similarity comparisons for semantic search and clustering.

C Detailed Analysis on ZeroSCROLLS

BEAVER performs robustly across ZeroSCROLLS and under aggressive RULER retrieval compression, retaining distributed information more reliably than the cited baselines.

  • ZeroSCROLLS: 32.0 and 32.4 are BEAVER's average ZeroSCROLLS scores at compression budgets of 2,000 and 3,000 tokens, respectively.It surpasses Selective-Context and is comparable to LongLLMLingua while matching or exceeding LLMLingua-2 across metrics.
  • ZeroSCROLLS: 34.0 is BEAVER's NarrativeQA score, supporting the QueryPlanner's structural priors for long-range dependency understanding.The cited task requires retaining and connecting information across long sequences.
  • Needle-in-a-Haystack Visualization: 100.0% and 99.0% are BEAVER's Single-Needle and Multi-Needle retrieval accuracies under approximately 5.3× compression.The experiment compresses 16k-token contexts to 3,000 tokens.
  • Needle-in-a-Haystack Visualization: LLMLingua-2 declines from 86.0% on Single-Needle to 72.0% on Multi-Needle retrieval.
  • Needle-in-a-Haystack Visualization: LongLLMLingua achieves only 6.0% and 9.0% accuracy on Single-Needle and Multi-Needle retrieval, respectively.The results indicate severe information loss within the 3,000-token budget.

E Additional Evaluation on L-Eval Benchmark

On out-of-domain L-Eval tasks with Qwen3-8B, BEAVER achieves the strongest average performance while preserving its advantage under a 2,000-token budget and approximately 4× compression.

  • Out-of-domain evaluation: 45.1 average score: BEAVER achieves the highest result on Qwen3-8B across the L-Eval suite.The suite covers academic QA, legal analysis, and fiction reading.
  • Out-of-domain evaluation: +1.6 points over LLMLingua-2: BEAVER outperforms the strongest baseline on average.
  • Out-of-domain evaluation: +1.9 on TPO and +0.6 on Legal Contract QA: BEAVER leads in specialized reasoning tasks.These gains are reported under the same approximate 4× compression setting.
  • Practical implications: Without external training or unstable perplexity calculations, BEAVER is presented as a lightweight, plug-and-play solution for open-source LLMs.

F Qualitative Comparison with Baseline Methods

Qualitative evaluations under severe token budgets show that BEAVER better preserves the information structures required for numerical reasoning, coherent summarization, demonstrations, and code syntax.

  • Evaluation design: 500–1,500 token budgets: the qualitative tests assess preservation of critical information under extreme compression across four representative tasks.The comparison examines semantic noise, program logic, chain-of-thought structure, and code syntax.
  • Financial QA: 500-token LongFQA: BEAVER retains scattered pricing evidence and produces the correct iPhone 15 Pro price of $1,078.80.Baselines either corrupt the 20% surcharge or lose the causal relation between base price and surcharge.
  • Government report summarization: 500-token GovReport: BEAVER preserves complete sentences for the 72-hour rule and its exceptions, yielding a coherent and accurate summary.The Segmenter respects natural language boundaries, unlike baselines that fragment syntax or lose connective logic.
  • Mathematical reasoning: 1,500-token GSM100: BEAVER treats each few-shot example as a coherent page unit and preserves chain-of-thought steps.Baselines can destroy question, reasoning, and answer boundaries, causing degradation to zero-shot guessing.
  • Code generation: 1,000-token CodeU: BEAVER prioritizes rare tokens and structural separators to protect syntactic correctness in code.Statistical pruning can remove delimiters and supervised compression can collapse complex logic into meaningless punctuation.

G.1 Analysis vs. Unsupervised Statistical Methods

Against unsupervised statistical methods, BEAVER is evaluated through Qwen3 scalability analysis and qualitative comparisons showing how probability-based pruning can fail on lightweight models and structured tasks.

  • Model scalability: 25.9 average score: LongLLMLingua performs poorly on Qwen3-0.6B, where probability distributions are noisier proxies for semantic importance.
  • Model scalability: 32.9 score: LLMLingua-2 underperforms on Qwen3-0.6B because its GPT-4-aligned compressor transfers poorly to the lightweight target model.The passage attributes the mismatch to differences between GPT-4-derived informative patterns and Qwen3-0.6B reasoning paths.
  • Qualitative failure modes: In GovReport, statistical compression corrupts syntax, producing fragments such as “Rements R” and preventing extraction of specific rules.The qualitative comparison concerns the 72-hour availability rule and waiver methods.
  • Qualitative failure modes: Retaining keywords such as “72 hours” and “waiving” is insufficient when connective logic, including “convening two legislative days,” is lost.The fragmented output preserves terms but not the relation explaining how waiver works.
Loading 2603.19635v1…