PaperlayerView exact revision on arXiv

Source-linked AI summary

LMK > CLS: Landmark Pooling for Dense Embeddings

Meet Doshi, Aashka Trivedi, Vishwajeet Kumar, Parul Awasthy, Yulong Li, Jaydeep Sen, Radu Florian, Sachindra Joshi

arXiv:2601.21525v1cs.CLcs.IR

TL;DR

Existing pooling methods can inadequately represent distributed or local evidence in variable-length sequences, limiting long-context embedding performance. LMK pooling inserts landmark tokens between chunks and mean-pools them, matching short-context retrieval performance while substantially improving long-context evaluations with minimal computational overhead.

  • Problem

    Existing pooling strategies must compress variable-length token representations into one vector, but [CLS] can bias toward early positions while mean pooling can dilute salient local signals.

  • Method

    LMK pooling partitions sequences into chunks, inserts landmark tokens at intervals, and mean-pools only the landmark embeddings to form the sequence representation.

  • Results

    LMK pooling maintains competitive short-context retrieval performance while substantially improving long-context evaluations across encoders, languages, and training setups.

  • Takeaways & Limitations

    LMK pooling offers a practical alternative for embeddings that balances local semantic information with global context and supports robust extrapolation beyond training lengths.

  • Takeaways & Limitations

    Adding LMK tokens increases sequence length and computation, although the overhead is modest when tokens are inserted at coarse intervals.

Abstract

from arXiv · show

Representation learning is central to many downstream tasks such as search, clustering, classification, and reranking. State-of-the-art sequence encoders typically collapse a variable-length token sequence to a single vector using a pooling operator, most commonly a special [CLS] token or mean pooling over token embeddings. In this paper, we identify systematic weaknesses of these pooling strategies: [CLS] tends to concentrate information toward the initial positions of the sequence and can under-represent distributed evidence, while mean pooling can dilute salient local signals, sometimes leading to worse short-context performance. To address these issues, we introduce Landmark (LMK) pooling, which partitions a sequence into chunks, inserts landmark tokens between chunks, and forms the final representation by mean-pooling the landmark token embeddings. This simple mechanism improves long-context extrapolation without sacrificing local salient features, at the cost of introducing a small number of special tokens. We empirically demonstrate that LMK pooling matches existing methods on short-context retrieval tasks and yields substantial improvements on long-context tasks, making it a practical and scalable alternative to existing pooling methods.

1. Introduction

The introduction frames pooling variable-length token representations into fixed-dimensional text embeddings as a central challenge, highlighting weaknesses in mean and [CLS] pooling. It presents Landmark (LMK) pooling as a simple alternative intended to improve long-context generalization while retaining local salient information.

  • Motivation: Pooling transforms variable-length token representations into a single fixed-dimensional representation for downstream retrieval, classification, and clustering tasks.The encoder output is converted through a pooling function after tokenization and insertion of special markers such as [CLS] and [SEP].
  • Motivation: Mean pooling can dilute salient or extreme features by uniformly averaging token representations.Its holistic aggregation treats all token contributions uniformly, which may weaken locally important signals.
  • Motivation: [CLS] pooling places the burden of encoding all relevant information on one position and often fails to generalize.The introduction contrasts this single-token aggregation with the distributed evidence needed for long sequences.
  • Landmark pooling: LMK pooling replaces reliance on a single initial special token with landmark tokens inserted between sequence chunks, then mean-pools their embeddings.The method adapts landmark tokens from prior work on mitigating abrupt chunking effects in retrieval-augmented generation to Dense Passage Retrieval.
  • Contributions: The paper reports substantial long-context embedding gains across domains and languages, while systematically characterizing pooling limitations and evaluating robustness across training regimes.It also positions LMK pooling as compatible with existing pretraining and retrieval systems.

2. Related Work

Dense embedding methods have advanced through stronger encoders and training objectives, but consensus remains limited on pooling token representations into document embeddings. Meanwhile, training data have expanded beyond standard benchmarks, while long-context embedding evaluation remains underexplored; this work connects pooling mechanisms with systematic long-context comparison.

  • Dense Embedders: Dense embedding approaches progressed from distance-based token transformations to stronger encoder architectures and improved training objectives.The passage attributes these developments to Gao et al. (2011), Huang et al. (2013), Devlin et al. (2019), Warner et al. (2025), and van den Oord et al. (2018).
  • Pooling Methodologies: Modern dense passage retrieval lacks consensus on aggregating token representations into a single document-level embedding.Recent models use CLS pooling, mean pooling, learned weighting, or hybrid strategies.
  • Training and Evaluation Datasets: Embedding training data have expanded from MSMarco and NQ to large, heterogeneous mixtures for modern text embedders.The passage cites Nguyen et al. (2016), Kwiatkowski et al. (2019), Chen et al. (2024), and Li et al. (2025a).
  • Training and Evaluation Datasets: Long-context embedding tasks remain underexplored, especially regarding how representation and pooling choices affect performance.The passage motivates robust evaluation of long-context embeddings and cites Li et al. (2025b), Zhu et al. (2024), and Chalkidis et al. (2021).
  • Contribution: This work links pooling mechanisms with long-context evaluation to systematically compare existing approaches against LMK pooling.The comparison is explicitly framed as an evaluation of pooling mechanisms under long-context conditions.

3. Understanding Pooling Methodologies

The section identifies complementary weaknesses in [CLS] and mean pooling: [CLS] favors early tokens through positional decay, while mean pooling can dilute concentrated salient evidence. These limitations motivate a pooling method that better preserves distributed and local information across context lengths.

  • [CLS] Pooling: [CLS] pooling inherently biases representations toward early sequence positions, and fine-tuning on shorter sequences amplifies this bias.Most embedding models fine-tune [CLS] pooling on relatively short sequences despite long-context pretraining.
  • [CLS] Pooling: RoPE attenuates attention logits with increasing relative distance, causing [CLS] to assign smaller weights to distant tokens.For [CLS] at position m = 0, the attenuation increases as token position n increases.
  • [CLS] Pooling: The [CLS] positional bias becomes more pronounced for longer sequences, limiting long-context extrapolation even when the RoPE base frequency increases.Increasing the RoPE base frequency slows positional decay and can improve inference-time extrapolation, but does not fully remove the underlying bias.
  • Mean Pooling: Mean pooling is position-independent and extrapolates well to long contexts, but equal token weighting can dilute salient evidence concentrated in a small subset.This weakness is especially relevant for short-context tasks, where mean pooling can underperform [CLS] pooling.
  • Alternative Pooling: Latent Attention selectively reweights token embeddings before mean pooling, but its resulting procedure is conceptually equivalent to mean pooling and adds learnable parameters.The additional parameters include learnable latent vectors and feed-forward layers.

4. Landmark (LMK) Pooling

LMK pooling addresses complementary weaknesses of CLS and mean pooling by inserting multiple landmark tokens throughout the sequence and mean-pooling their embeddings. Its coarse, configurable placement improves granularity robustness while keeping the added-token overhead modest.

  • LMK tokenization splits the token sequence into chunks and inserts a landmark token between chunks and after the final chunk.The construction retains an initial CLS token and produces token IDs plus an attention mask.
  • The sentence representation is computed by mean-pooling the embeddings at all valid LMK positions.The LMK index set selects positions containing the landmark token and excludes padding positions.
  • CLS pooling attenuates attention to distant positions, whereas mean pooling can dilute salient features despite each method’s representation-learning benefits.
  • LMK placement can follow sentence boundaries or fixed token intervals, with predefined intervals supporting robustness to granularity choice and task-dependent inference granularity.Sentence chunking may be unsuitable for code or languages without clear sentence delimiters, while fixed-length chunking can introduce positional bias.
  • At 32K-token evaluation length, adding one LMK token every 128 tokens adds only 256 tokens while significantly outperforming alternative pooling mechanisms.The additional computation is described as modest because LMK tokens are inserted at coarse intervals.

5. Experimental Setup

The experiments compare pooling strategies across English and multilingual retrieval settings, with separate evaluations of short- and long-context behavior. They also examine how long-context training and Landmark-token insertion granularity affect extrapolation.

  • Experimental goals: The study trains models with different pooling strategies, evaluates domain and context-length effects, and tests how long-context training influences extrapolation.Experiments are divided into English and Multilingual settings.
  • English setting: English experiments use gte-en-mlm-base and ModernBERT-base trained on MS MARCO passage and document-ranking data with hard negatives and distillation scores.Training uses 7 hard negatives for passage ranking and 1 for document ranking, for 5k steps at a learning rate of 2 × 10−5.
  • English setting: Evaluations cover MS MARCO Dev, BEIR, MTEB-v2, MIRACL Retrieval, MLDR, COIR, and LongEmbed to assess in-domain, short-context, and long-context retrieval.The out-of-domain long-context benchmarks include documents often exceeding the 8k-token pretraining context.
  • Multilingual setting: Multilingual experiments use mmBERT-base fine-tuned on multilingual data and evaluate MIRACL Hard Negatives, MLDR, LongEmbed, and Multi-EURLEX.The benchmarks span 18, 13, and 23 languages respectively, with Multi-EURLEX reported using Macro-F1 scores.
  • Pooling strategies: Pooling comparisons include LMK, CLS, Mean, 64-latent Attention, and inference-time MultiCLS, while LMK ablations vary sentence-based, fixed, and sampled granularities.Variable splitting samples granularities from {32, 64, 128, 256} during training.

6. Results & Discussion

The results show that LMK pooling is competitive on short-context retrieval and generally improves long-context and multilingual generalization, while reducing but not eliminating positional bias. LMK embeddings preserve local semantic information, though they favor earlier over later context.

  • Retrieval performance: LMK pooling performs marginally better than pooling baselines on in-domain MSMarco retrieval while remaining comparable to CLS on short-context tasks.
  • Multilingual generalization: LMK pooling maintains substantial improvements on long-context evaluations across in-domain and out-of-domain multilingual retrieval and classification tasks.
  • Long-context results: For gte-en-mlm-base, CLS is strongest on short-context tasks and Mean outperforms CLS and LMK on long-context tasks, whereas ModernBERT-base finds LMK most consistent.
  • Training-length effects: On gte-multilingual-base, CLS, Mean, and Latent Attention become competitive with LMK on MLDR but substantially underperform on out-of-domain long-context tasks.These results suggest that longer-context training reduces but does not eliminate positional bias.
  • LMK representations: Each neighboring chunk retrieves its corresponding LMK embedding with ≥58% Hit@10 accuracy, showing that contextualized LMK representations preserve substantial local semantic information.
  • LMK representations: LMK embeddings show higher affinity for left than right neighboring chunks, indicating a directional bias toward earlier context.

7. Conclusion

The paper identifies shortcomings in existing pooling mechanisms and introduces Landmark (LMK) pooling as a simple alternative. Across diverse architectures, languages, and training setups, LMK maintains competitive short-context retrieval performance.

  • Method: LMK pooling partitions sequences into chunks, inserts landmark tokens, and mean-pools their embeddings into the final representation.The method is presented as simple yet effective.
  • Motivation: Existing pooling mechanisms exhibit biases and shortcomings across various embedding tasks.
  • Empirical findings: LMK maintains competitive performance on short-context retrieval across multiple encoder architectures, languages, and training setups.

A. Additional Results

Additional experiments confirm LMK’s strong long-context extrapolation and robustness to inference-time granularity choices. Results vary by context length and pooling strategy, while multilingual evaluations are reported in full.

  • A. Additional Results: On MSMarco passage ranking with gte-en-mlm-base, LMK pooling consistently shows strong long-context extrapolation, and variable-granularity finetuning improves inference-time granularity robustness.These results are reported in Table 7 as an analogue to the ModernBERT-base experiments in Table 1.
  • A. Additional Results: On MSMarco document ranking, LMK achieves the best overall in-domain performance.Table 8 reports comparisons across in-domain and out-of-domain retrieval benchmarks.
  • A. Additional Results: ∼0.5 points separate CLS and LMK on short-context benchmarks, where CLS leads, while mean pooling degrades notably.For long-context evaluations, mean pooling performs best overall, LMK follows closely, and CLS lags substantially behind.
  • A. Additional Results: Full multilingual results are provided for MIRACL, MLDR, and Multi-EURLEX in Tables 10 to 12.These evaluations correspond to Table 2 in the main text.

B. Training and Evaluation Hyperparameters

Experiments cover English and multilingual settings under short- and long-context training and evaluation. The setups specify model families, ranking data, optimization parameters, hardware, and benchmark suites.

  • Experimental settings: Experiments span English and multilingual settings, each evaluated in short- and long-context regimes.English uses gte-en-mlm-base and ModernBERT-base; multilingual experiments use mmBERT-base and gte-multilingual-base.
  • English: English models train for 5k steps at 2 × 10−5 learning rate, with 250 warmup steps and queries capped at 128 tokens.Training uses MS MARCO passage and document ranking datasets with hard negatives and distillation scores; passage ranking uses effective query batch size 2,048.
  • Multilingual: Multilingual models train for 10k steps at 2 × 10−5 learning rate, with batch size 1,024 and maximum sequence length 512 tokens.For ablations up to 8,192 tokens, batch size is reduced to 64 due to memory constraints.
  • Training setup: Training uses BF16 precision on 8 NVIDIA H100 80GB GPUs, InfoNCE temperature τ = 0.02, shared cross-device negatives, DeepSpeed, and HuggingFace Accelerate.The pipeline is built on FlagEmbedding3.
  • Evaluation: Evaluation uses MTEB and SentenceTransformers across English and multilingual retrieval benchmarks, plus Multi-EURLEX zero-shot classification with Macro-F1.Multilingual evaluation covers 18-language MIRACL Hard Negatives, 13-language MLDR and LongEmbed, and 23-language Multi-EURLEX.

C. Dataset Information

The paper uses standard short-context retrieval datasets but notes that they provide limited exposure to true long-document structure. It therefore incorporates longer-context datasets and evaluates across in-domain, out-of-domain, monolingual, and multilingual benchmarks.

  • Short-context datasets: Standard training datasets include MSMarco Passage Ranking, TREC-DL, and Natural Questions, but primarily contain relatively short passages.These datasets have supported effective short-context retrievers while offering limited exposure to long document structures.
  • Dataset limitations: MSMarco Document Ranking provides longer documents than passage-level data but still offers limited true long-context coverage.Its document lengths remain well below those encountered in many real-world retrieval scenarios.
  • Training constraints: Training directly on long-context inputs is constrained by higher computational cost, smaller batch sizes, and longer training times.The paper presents LMK pooling as enabling strong long-context extrapolation despite predominant short-context training.
  • Evaluation datasets: Evaluation covers MSMarco Dev, BEIR-15, MTEB-v2 retrieval, and MIRACL English for English retrieval, plus MIRACL, MLDR, and Multi-EURLEX for multilingual assessment.MIRACL provides short-context evaluation across 18 languages, MLDR covers 13 languages for long-context testing, and Multi-EURLEX contains 65k legal documents across 23 languages.
  • Long-context training data: MLDR contains substantially longer documents than MSMarco Document Ranking and is used for multilingual long-context training.The MLDR mixture contains about 42k query-document pairs within roughly three million total training examples.
Loading 2601.21525v1…