PaperlayerView exact revision on arXiv

Source-linked AI summary

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Zifeng Cheng, Jie Zheng, Zhiwei Jiang, Shuwen Wang, Fei Shen, Shiping Ge, Qing Gu

arXiv:2609.37782v1cs.CL

TL;DR

Long-context embeddings commonly average all tokens, although documents may contain substantial redundant or uninformative content. SCSP uses chunk-level semantic compression prompts and their attention patterns to select informative tokens, and experiments report consistent improvements across baselines and fine-tuned models.

  • Problem

    Long-context embedding methods commonly use mean pooling, which weights all tokens equally despite redundancy and uninformative content in long documents.

    Source: §1 · p. 2

  • Method

    SCSP partitions documents into sentence-aware chunks, appends local semantic compression prompts, and uses prompt-isolated attention patterns to select and aggregate informative tokens.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4

  • Results

    SCSP consistently improves Vanilla Mean, Echo Mean, and HTP across contexts and backbones and enhances fine-tuned models; with LLaMA at 8,192 tokens, Echo Mean's average performance improves 60% relative to Echo Mean alone.

    Source: §4.3 · p. 6, §5 · pp. 9–10

  • Takeaways & Limitations

    Selective token aggregation is an effective alternative to conventional mean pooling in the evaluated long-context embedding settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.

1 INTRODUCTION

Existing long-context embedding methods improve information flow but commonly use uniform mean pooling, which can dilute salient semantics. SCSP uses local semantic compression prompts to select informative tokens and reports improvements across training-free and fine-tuned models.

1 INTRODUCTION

  • Prior training-free approaches address information flow under causal attention, while mean pooling still weights every token equally.Echo repeats the input; Token Prepending adds a global representation, and Hierarchical Token Prepending adds block-level representations.Source: §1 · pp. 1–2
  • Selective pooling targets the risk that redundant or uninformative tokens dilute salient semantics in long documents.The section frames identifying and selectively aggregating informative tokens without additional training as its research question.Source: §1 · p. 2
  • SCSP estimates token importance from attention elicited by local semantic compression prompts, then selects and aggregates informative token representations.Sentence-aware chunks and a prompt-isolated mask keep each prompt focused on its corresponding local context.Source: §1 · p. 2
  • SCSP consistently improves Vanilla Mean, Echo Mean, and HTP across benchmarks, and also enhances fine-tuned embedding models.The experiments report plug-and-play integration across these model types.Source: §1 · p. 2

2 RELATED WORK

Related work includes fine-tuned LLM embedding models and training-free methods that repurpose off-the-shelf LLMs. SCSP differs by using semantic-compression attention to select and aggregate informative tokens.

Finetuning LLMs for Embeddings

  • Fine-tuned LLM embedding approaches use learned semantic representations, but require substantial data and incur fine-tuning costs while specializing general-purpose models.Examples include Sentence-BERT, SimCSE, and LLM2Vec.Source: §2 · p. 2

Training-free LLMs for Embeddings

  • Training-free work repurposes off-the-shelf LLMs through prompt engineering or changes that introduce backward dependencies under causal attention.The cited prompt-engineering line includes PromptEOL, which uses a semantic compression prompt.Source: §2 · pp. 2–3
  • Echo repeats the input, Token Prepending adds a global semantic representation, and HTP adds block-level representations; SCSP instead uses prompt attention to select informative tokens.HTP also reports mean pooling as more effective than last-token pooling for long-context embeddings.Source: §2 · p. 3

3 METHOD

SCSP builds long-context embeddings by using chunk-local semantic compression prompts to identify informative tokens, then selectively pools their intermediate-layer representations. Its attention mask keeps prompts local while preserving causal information flow among document tokens.

3.1 OVERVIEW

  • SCSP uses chunk-wise semantic compression prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into a long-context embedding.The framework comprises input construction, a prompt-isolated attention mask, token selection, and selective pooling.Source: §3.1 · p. 3

3.2 INPUT CONSTRUCTION

  • The input is divided into non-overlapping chunks of complete sentences capped at X tokens, then each chunk receives an appended semantic compression prompt.The augmented sequence is fed to the LLM at once rather than processing chunks separately.Source: §3.2 · p. 4

3.3 PROMPT-ISOLATED ATTENTION MASK

  • Document tokens retain causal attention to preceding document tokens but cannot attend to prompts, while each prompt attends only to its chunk and preceding prompt tokens.A positional-encoding variant intended to keep document positions contiguous yields no performance improvement.Source: §3.3 · p. 4

3.4 SEMANTIC COMPRESSION-GUIDED TOKEN SELECTION

  • Prompt attention estimates token importance, taking the maximum across layers and heads; a token is selected when its score exceeds α times the uniform-attention baseline.The final token of each chunk’s prompt can attend to that chunk, providing the attention pattern used for token scoring.Source: §3.4 · pp. 4–5

3.5 INTERMEDIATE-LAYER SELECTIVE POOLING

  • The selected tokens’ representations are mean-pooled from an intermediate layer to produce the final long-context embedding.The paper motivates intermediate-layer representations as richer in semantic information than final-layer representations.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments evaluate SCSP on five long-context datasets using multiple embedding baselines and two LLMs. Results show consistent gains, including a substantial improvement when SCSP is combined with Echo Mean at 8,192 tokens.

4.1 DATASETS AND EXPERIMENTAL SETTINGS

  • Evaluation uses nDCG@10 on five LongBench and LongEmbed datasets, including QMSum, 2Wiki-MultiHopQA, SummScreenFD, NarrativeQA, and MultiFieldQA.Source: §4.1 · p. 6

4.2 BASELINES

  • Baselines include PromptEOL, TP variants, Vanilla Mean, Echo Mean, and HTP; all use the same output layer for representation extraction.Source: §4.2 · p. 6

4.3 RESULTS

  • 60%: with LLaMA at 8,192 tokens, SCSP with Echo Mean improves average performance over Echo Mean alone; SCSP also improves Vanilla Mean, Echo Mean, and HTP across both models.The reported cross-model comparison covers Mistral-7B-Instruct-v0.3 and LLaMA-3.1-8B-Instruct.Source: §4.3 · p. 6
  • At 8,192 tokens on Mistral-7B-Instruct-v0.3, Vanilla Mean combined with SCSP achieves the best performance among the reported methods.Source: §4.3 · p. 7
  • At 512 tokens, Echo Mean is a strong baseline, whereas at 8,192 tokens it generally adds no gains; SCSP’s gains are larger in the longer-context setting.The authors suggest that selecting salient information becomes increasingly important as context grows.Source: §4.3 · p. 7
  • SCSP adds only marginal runtime cost, unlike the substantial runtime overhead of Echo’s input repetition.Source: §4.3 · p. 7

4.4 ABLATION STUDY

The ablations examine chunk and prompt construction, prompt isolation and position encoding, and token selection and aggregation. Their results support localized semantic prompting and selective token-level aggregation.

4.4 ABLATION STUDY

  • 4.77%: removing the semantic compression prompt lowers performance; fixed-length chunking and a single document-end prompt lower it by 1.01% and 2.73%, respectively.The authors attribute the single-prompt drop possibly to long-text attention scores being influenced by relative token distances.Source: §4.4 · p. 7
  • 3.49%: retaining the original causal mask lowers performance, while making document-token positions contiguous causes a 0.35% drop.The authors suggest prompt-induced positional gaps may help distinguish and capture chunk semantics.Source: §4.4 · p. 7
  • 5.02%: aggregating each chunk’s compressed-prompt representation lowers performance, and weighting token representations by importance scores adds no improvement over SCSP.Source: §4.4 · pp. 7–8
  • Randomly selecting the same number of tokens performs close to Vanilla Mean, indicating that SCSP’s gains are not simply from pooling fewer tokens.The ablation suggests that identifying tokens capturing core document semantics matters.Source: §4.4 · p. 7

4.5 GENERALIZABILITY ACROSS DIFFERENT CONTEXT LENGTHS

Across context lengths, SCSP improves both evaluated baselines, while the gains from expanding context generally diminish as the window grows.

4.5 GENERALIZABILITY ACROSS DIFFERENT CONTEXT LENGTHS

  • SCSP consistently improves both baseline methods across multiple context lengths, and larger contexts generally improve performance with diminishing gains.Source: §4.5 · p. 8
  • At 16,384 tokens, Vanilla Mean gains further performance on SumFD and NQA, suggesting these datasets benefit from a larger context window.Source: §4.5 · p. 8

4.6 EFFECTS OF HYPERPARAMETERS

SCSP improves performance across a range of hyperparameter settings and transfers to additional LLM backbones. The preferred chunk size and selection ratio vary by baseline.

4.6 EFFECTS OF HYPERPARAMETERS

  • SCSP consistently improves Vanilla Mean and HTP across a range of chunk-size, ratio, and output-layer settings.Source: §4.6 · p. 8
  • Vanilla Mean performs best with chunk size 1,024 and ratio 1.75, whereas HTP performs best with chunk size 2,048 and ratio 2.5.Source: §4.6 · p. 8
  • At their best configurations, SCSP retains 14.03% of Vanilla Mean tokens and 8.64% of HTP tokens; the best output layer is sixth-to-last, with performance declining closer to the final layer.Source: §4.6 · p. 8

4.7 GENERALIZATION ACROSS DIFFERENT BACKBONES

  • The method is also effective on these two LLMs, demonstrating generalizability; among the four LLMs, Gemma2-9B achieves the best performance with Vanilla Mean at a context length of 8,192.Source: §4.7 · p. 9

4.8 EFFECTS OF TOKEN IMPORTANCE SCORES

Across token-importance strategies, max aggregation performs best and mean aggregation worst; selective pooling also improves both evaluated fine-tuned embedding models.

4.8 EFFECTS OF TOKEN IMPORTANCE SCORES

  • Max aggregation across layers and heads performs best among four token-importance strategies, while averaging across them performs worst.The authors suggest max aggregation better uncovers salient signals from individual layers or heads, whereas mean aggregation may smooth them out.Source: §4.8 · p. 9

4.9 RESULTS ON FINETUNED EMBEDDING MODELS

  • Selective pooling consistently improves GritLM and NV-EMBED over their mean-pooling versions.The evaluation uses final-layer representations; NV-EMBED uses latent attention, yet the authors report that semantically irrelevant tokens remain unable to capture useful information after refinement.Source: §4.9 · p. 9
  • Selective pooling improves fine-tuned models even when their token representations use latent-attention refinement.The authors interpret the NV-EMBED result as evidence that semantically irrelevant tokens may remain uninformative after that refinement.Source: §4.9 · p. 9

5 CONCLUSION

SCSP estimates token importance from semantic-compression prompts on sentence-aware chunks and uses prompt-isolated attention to selectively aggregate informative tokens. The conclusion reports improvements across context lengths and LLM backbones, as well as gains for fine-tuned embedding models.

5 CONCLUSION

  • SCSP consistently improves Vanilla Mean, Echo Mean, and HTP across different context lengths and LLM backbones.The conclusion also reports that SCSP enhances fine-tuned embedding models.Source: §5 · pp. 9–10
  • SCSP appends semantic-compression prompts to sentence-aware chunks and uses their attention patterns to estimate token importance.A prompt-isolated attention mask prevents the prompts from interfering with original document representations; selected informative tokens are then aggregated.Source: §5 · p. 9
  • The conclusion presents selective token aggregation as an effective alternative to conventional mean pooling.It describes SCSP as a training-free framework for constructing long-context embeddings.Source: §5 · pp. 9–10

A EFFECTS OF SEMANTIC COMPRESSION PROMPT

All evaluated semantic-compression prompts improve performance. Removing the opening quotation mark reduces performance by 0.77%, which the authors link to its role as a cue for semantic continuation.

A EFFECTS OF SEMANTIC COMPRESSION PROMPT

B RESULTS ON THE LOCOV1 DATASET

On LOCOV1, SCSP improves Vanilla Mean across all eight datasets, with gains spanning QA, retrieval, and long-document understanding tasks.

B RESULTS ON THE LOCOV1 DATASET

  • 55.77 to 60.77: SCSP raises Vanilla Mean’s average score across all eight LOCOV1 datasets, an absolute gain of 5.00 points.The authors report gains across QA, retrieval, and long-document understanding tasks.Source: B RESULTS ON THE LOCOV1 DATASET · p. 13
  • 12.56 points on SumFD and 9.13 points on QASPER Title: these are SCSP’s particularly pronounced gains over Vanilla Mean.The comparison is reported within the eight-dataset LOCOV1 evaluation.Source: B RESULTS ON THE LOCOV1 DATASET · p. 13
  • SCSP improves Vanilla Mean across QA, retrieval, and long-document understanding tasks.The authors state that selective aggregation can suppress less informative tokens and produce more discriminative long-context representations.Source: B RESULTS ON THE LOCOV1 DATASET · p. 13
Loading 2609.37782v1…