PaperlayerView exact revision on arXiv

Source-linked AI summary

QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory

Yihang Wang, Xu Huang, Bowen Tian, Yueyang Su, Lei Yu, Huaming Liao, Yixing Fan, Jiafeng Guo, Xueqi Cheng

arXiv:2408.10497v3cs.CLcs.AI

TL;DR

Long contexts increase costs and can worsen the “lost in the middle” problem, while existing compression metrics may not retain query-relevant information. QUITO-X applies the Information Bottleneck and uses cross-attention to approximate mutual information; experiments report improved performance and efficiency, with occasional gains over original-context models on long texts.

  • Problem

    Long contexts raise computational costs and inference latency, while existing self-information and perplexity metrics may not align with retaining the most important query-conditioned information.

    Source: §1 · pp. 1–2

  • Method

    QUITO-X formulates context compression through the Information Bottleneck and uses cross-attention values as a practical proxy for mutual information.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The method reduces memory usage to 50% of the most memory-efficient baseline and improves accuracy by 25% over the best-performing baseline; on long texts it occasionally surpasses models using original context.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors report that compressed long-text contexts can sometimes outperform original contexts, likely by eliminating redundancy.

    Source: §5 · p. 9

    Chunking can break global context and create semantic discontinuities between chunks; maintaining coherence across boundaries remains an open question.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Generative LLM have achieved remarkable success in various industrial applications, owing to their promising In-Context Learning capabilities. However, the issue of long context in complex tasks poses a significant barrier to their wider adoption, manifested in two main aspects: (i) The excessively long context leads to high costs and inference delays. (ii) A substantial amount of task-irrelevant information introduced by long contexts exacerbates the "lost in the middle" problem. Existing methods compress context by removing redundant tokens using metrics such as self-information or PPL, which is inconsistent with the objective of retaining the most important tokens when conditioning on a given query. In this study, we introduce information bottleneck theory (IB) to model the problem, offering a novel perspective that thoroughly addresses the essential properties required for context compression. Additionally, we propose a cross-attention-based approach to approximate mutual information in IB, which can be flexibly replaced with suitable alternatives in different scenarios. Extensive experiments on four datasets demonstrate that our method achieves a 25% increase in compression rate compared to the state-of-the-art, while maintaining question answering performance. In particular, the context compressed by our method even outperform the full context in some cases.

1 Introduction

Long contexts raise computational costs and inference latency, while irrelevant information can worsen the “lost in the middle” problem. The paper frames compression through the Information Bottleneck and evaluates a mutual-information-based objective.

1 Introduction

  • Long contexts increase computational costs and inference latency, and can exacerbate the “lost in the middle” problem.Source: §1 · p. 1
  • Existing compression metrics such as self-information and perplexity can favor nouns over relational phrases, quantifiers, or verbs, producing fragmented context and incorrect outputs.Generative compression methods also face context-window limits, hallucination, and the “lost in the middle” problem.Source: §1 · pp. 1–2
  • Information Bottleneck theory yields a mutual-information metric that the authors prove equivalent to maximizing the likelihood of the compressed generation.This aligns the proposed metric with the compressor’s optimization objective.Source: §1 · p. 2
  • The method reduces memory usage to 50% of the most memory-efficient baseline while improving accuracy by 25% over the best-performing baseline.The reported experiments concern long-context question answering.Source: §1 · p. 2

2 Related Work

Prior extractive methods use token-importance metrics that may not reflect task relevance, while the Information Bottleneck offers a framework for preserving task-relevant information during compression.

2.1 Extractive Context Compression

  • Selective Context ranks tokens by self-information, while LLMLingua methods use perplexity-based compression strategies.QUITO uses attention from a small language model to retain query-relevant context.Source: §2.1 · p. 2
  • Entropy-based metrics can favor high-entropy nouns while underweighting function words that carry relational semantics, leaving compressed context fragmented.The figure contrasts LLMLingua2’s focus on “barn” and “farmhouse” with QUITO-X retaining “in a barn near a farmhouse.”Source: §2.1 · p. 2, Figure 2 · p. 2

2.2 Information Bottleneck

  • The Information Bottleneck compresses input X into representation T while preserving task-relevant information I(T; Y) and discarding irrelevant information I(T; X).The paper derives a token-wise mutual-information metric, uses cross-attention scores as a practical proxy, and reports theoretical consistency with maximum likelihood.Source: §2.2 · pp. 2–3

3 Method

The method casts context compression as preserving query-relevant information at a fixed length, then estimates importance with cross-attention and selects high-scoring words.

3.1 Theorem

  • At a fixed compression ratio, the information-bottleneck objective maximizes information about the output given the query while penalizing information retained from the original context.Fixing the compression ratio makes the efficiency term constant, so optimization focuses on the output-relevant term.Source: §3.1 · p. 3
  • Theorem 1 states that this fixed-ratio mutual-information objective is equivalent to maximizing the expected log-likelihood of the ground-truth output.Source: §3.1 · p. 3
  • Because the ground-truth output is unavailable and sampled model probabilities do not directly yield token-level mutual information, the method uses cross-attention as a practical proxy.An encoder-decoder takes the context and query as input; attention between the output’s first token and context tokens approximates mutual information.Source: §3.1 · p. 3

3.2 Algorithm

  • Tokens are merged into words and their scores summed; Gaussian smoothing elevates neighboring words to preserve information around important units.Independent word scores can favor nouns while overlooking conjunctions, verbs, and prepositions, creating semantic ambiguity.Source: §3.1 · pp. 3–4
  • The context and query are encoded together; the decoder’s first token attends over encoded tokens, and attention scores are averaged across heads, softmax-normalized, and aggregated by word.Source: §3.2 · p. 4
  • The method retains words among the top τ proportion of smoothed scores, with τ controlling retained detail and the resulting compressed context length.The paper describes the algorithm as reducing context length while retaining essential information for downstream tasks.Source: §3.2 · p. 5

4 Experiments

Across datasets and compression settings, QUITO-X generally outperforms the tested baselines, including on long-context tasks and sentence-level compression. Ablations show gains from both the Gaussian filter and merging, especially under aggressive compression.

4.1 Datasets and Metrics

  • The evaluation spans nine datasets, covering moderate- and long-context question answering and summarization, with Exact Match for QA and ROUGE-L for Gov_Report.The long-context datasets include multi-document QA, few-shot QA, and summarization.Source: §4.1 · p. 5

4.2 Implementation Details

  • Compression uses FLAN-T5-small; QA evaluation uses LongChat-13B-16k and LLaMA3-8B-Instruct, and σ is fixed at 1 because nonzero choices did not significantly affect performance.Long inputs are divided into 512-token chunks, with experiments comparing separate chunk compression followed by merging against unified compression after merging attention scores.Source: §4.2 · p. 5

4.4 Experimental Results

  • QUITO-X consistently outperforms the listed baselines on CoQA and Quoref across compression ratios from 1.00 to 0.00, sometimes surpassing the original context.The authors attribute this to retaining critical information while removing irrelevant noise.Source: Table 2 · p. 6, §4.4 · p. 7

4.3 Baseline

  • CoQA and Quoref are compared against all listed baselines, while long-context datasets use a narrower comparison with LLMLingua2, Selective Context, and QUITO.The baselines include methods based on self-information, perplexity, distillation, and attention; QUITO-X uses an 80M-parameter encoder-decoder compressor.Source: §4.3 · pp. 5–6, Table 1 · p. 6

4.4 Experimental Results

  • Both chunking strategies outperform baselines on multi-document QA; QUITO-X also improves over baselines on TriviaQA and Gov_Report, particularly at medium and high compression.Strategy 2 is strongest on HotpotQA at ratios 0.25 and 0.50 and has a clear advantage on MuSiQue at 0.25; Gov_Report uses ROUGE-L.Source: Table 3 · p. 7, §4.4 · pp. 7–8

4.5 Ablation Study

  • At a 0.25 ratio, SQuAD information coverage rises from 71.5 to 87.8 with the Gaussian filter, while merging improves DROP information coverage by nearly 10 points.Both components improve accuracy or information coverage, with merging particularly beneficial at higher compression.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6 Comparison with Sentence-Level Compression

  • On NQ and TQA, QUITO-X outperforms sentence-level FILCO at comparable compression ratios of 25% and 50%.The comparison follows FILCO’s experimental protocol and preprocessing pipeline.Source: §4.6 · p. 8

5 Conclusion

The conclusion presents mutual information, derived from information bottleneck theory, as an optimization objective equivalent to maximizing likelihood. It reports gains in latency and performance, including occasional results above full-context models, while identifying improved long-text chunking as future work.

5 Conclusion

  • The paper derives mutual information from information bottleneck theory as its optimization objective and proves it equivalent to maximizing likelihood.Source: §5 · pp. 8–9
  • The method significantly outperforms strong baselines in inference latency and performance, and on long texts occasionally surpasses models using the original context.The authors suggest this may result from eliminating inherent context redundancy.Source: §5 · p. 9
  • More effective chunking strategies for long texts remain future work.Source: §5 · p. 9

Limitations

The limitations concern chunking long documents, the human readability of token-level outputs, and the breadth of experimental validation. The motivating observation tests whether answer-bearing substrings in DROP contexts capture mutual information.

Limitations

  • Chunking is required for long documents with smaller models, but it can break global context and create semantic discontinuities between chunks.Maintaining coherence across chunk boundaries, or quantifying fragmentation’s impact, remains open.Source: Limitations · p. 9
  • Token-level outputs may be less readable to people, although QUITO has better semantic continuity and readability than LLMLingua2 and other token-level baselines.The authors identify human interpretability of compression decisions as a potential cost.Source: Limitations · p. 9
  • Computational constraints limited experiments across diverse domains, and σ was not comprehensively tuned across datasets despite appearing stable under reasonable variations.The authors say broader validation would strengthen generalizability claims.Source: Limitations · p. 9

A.1 Motivating Observation

  • In the DROP motivating experiment, SubYi is the answer substring within Xi, assumed to capture most mutual information between Xi and Yi.The experiment uses the relative metric values on SubYi tokens to assess how well a metric approximates I(X; Y | Q).Source: §A.1 · p. 11

A.2 Experiment

The experiment compares token-scoring metrics as approximations of mutual information, using MRR to assess whether they rank answer-bearing context tokens highly. First-token cross-attention achieves significantly higher MRR than the alternatives.

A.2 Experiment

  • The experiment compares self-attention, self-information, and two cross-attention strategies: scoring from the first output token or averaging scores across generated output tokens.Flan-T5-small computes cross-attention; the two strategies differ in which output-token attention scores they use.Source: §A.2 · p. 11
  • MRR evaluates how highly each metric ranks the answer-bearing substring’s tokens, with per-example reciprocal ranks averaged across the dataset.For each example, tokens are ranked by metric value and reciprocal rank is calculated for the positions of the substring.Source: §A.2 · p. 11

A.3 Result

  • Cross-attention between the first output token and each context token yields significantly higher MRR than the other tested methods.Figure 5 reports the MRR comparison.Source: §A.3 · p. 11, Figure 5 · p. 11

B Proof of Theorem 1

The proof uses conditional mutual information between the compressed context and output, given the query, to establish the link to the likelihood objective. Terms independent of the optimization are omitted, leaving an expected log-likelihood expression.

Theorem

  • The proof begins by expanding mutual information between the compressed context and output conditioned on the query.The notation uses X for original context, Q for query, Y for output, and X̄ for the extractive compressed result.Source: B Proof of Theorem 1 · p. 11, Theorem · p. 12
  • Terms that do not affect the optimization are ignored, leaving the expected log probability of the output given compressed context and query.The resulting expression is E[log P(y | x̄, q)].Source: Theorem · p. 12
  • The proof identifies x̄, y, and q as data points sampled from X̄, Y, and Q, respectively.This completes the proof in the supplied passage.Source: Theorem · p. 12

C Information Coverage

The appendix evaluates information coverage, sensitivity to σ, computational overhead, inference time, and proxy-model size. QUITO leads on information coverage across the reported compression ratios, while σ has little effect for nonzero values.

C Information Coverage

  • Information Coverage, which measures preservation of answer-source information after compression, is highest for the reported method versus baselines on DROP and SQuAD across compression ratios from 1.0 to 0.25.At 0.25, where information loss is more severe, the method outperforms other methods by a clear margin.Source: C Information Coverage · p. 12

D Parameter Search for σ

  • Nonzero σ values have minimal impact on accuracy and Information Coverage, so the authors use σ = 1 as the default.The search tested σ values from 1 to 5 at compression ratios 0.75, 0.50, and 0.25.Source: D Parameter Search for σ · pp. 12–13

E Computational Overhead Analysis

  • Cross-attention overhead with a small proxy model is approximately one proxy-inference round; PPL-based compression has similar overhead when using the same proxy.The paper attributes both methods’ added inference time to calculations performed with the proxy model.Source: E Computational Overhead Analysis · p. 13

E.1 Inference Time per 512 Tokens

  • FLAN-T5-Small, with 80M parameters, serves as QUITO’s proxy, making added time negligible; the authors say its efficiency gains outweigh the cost.Prior works typically use larger proxy models, which makes QUITO more efficient in practice.Source: §E.1 · p. 13

F Comparison with Different FLAN-T5 Model Sizes

G Comparison with Sentence-Level Compression Methods

The comparison evaluates token-level QUITO-X against sentence-level methods on NQ and TQA using FILCO’s setup. QUITO-X outperforms FILCO at both tested compression ratios and offers precise compression control.

G Comparison with Sentence-Level Compression Methods

H Case Studies on Readability and Semantic Continuity

Case studies compare compressed outputs for readability and semantic integrity. The examples support the authors’ claim that QUITO-X produces more coherent, interpretable outputs than other token-level baselines, despite reduced syntactic completeness.

H Case Studies on Readability and Semantic Continuity

  • The case studies assess readability and semantic integrity by comparing QUITO-X with LLMLingua2, QUITO, and their variants.Source: H Case Studies on Readability and… · p. 14
  • QUITO-X produces more coherent and interpretable outputs than other token-level baselines, although token-level compression tends to reduce syntactic completeness.The authors identify transparency-sensitive applications as a setting where this difference matters.Source: H Case Studies on Readability and… · p. 14
  • In the census example, QUITO-X retains the 9.8% Italian and 9.2% English ancestry figures, while the QUITO prompt shown omits them.The original prompt contains both figures, and the case-study caption says QUITO-X retains more semantic and numeric fidelity than other token-level approaches.Source: H Case Studies on Readability and… · p. 15, Figure 9 · p. 15
  • The sports example highlights relevant touchdown-yardage details in QUITO-X’s compressed output, including the 91-yard and 15-yard plays.The figure caption characterizes these retained details as supporting accurate numerical reasoning.Source: Figure 10 · p. 16, H Case Studies on Readability and… · p. 16

Related papers

Loading 2408.10497v3…