LongEmbed: Extending Embedding Models for Long Context Retrieval

Benchmark

• Existing embedding models show huge room for improvement on LONGEMBED when processing long contexts.Source: §1 · p. 1

Context Extension

• Training-free strategies extend existing embedding models by several folds, regardless of whether their original context is 512 or beyond 4k tokens.Source: §1 · pp. 1–2

APE Models

• Fine-tuning APE-based models can improve extension while strictly preserving original short-context behavior, extending E5Base from 512 to 4k.Source: §1 · p. 2

RoPE Models

• NTK and SelfExtend substantially enhance RoPE-based models; NTK extends E5-Mistral to 32k with close-to-perfect passkey accuracy and state-of-the-art LONGEMBED performance.Source: §1 · p. 2

Benchmark

• LONGEMBED evaluates long-context retrieval across two synthetic and four real-world tasks with varying document lengths and dispersed target information.Source: §1 · pp. 1–2

Resources

• The paper releases the LONGEMBED benchmark and the E5Base-4k and E5-RoPEBase models to support future research.Source: §1 · p. 2

Long-Context Embeddings

• Existing long-context embedding efforts typically pre-train long-context backbones from scratch or use existing ones, then train them to produce embeddings.Source: §2 · p. 2
• This paper instead gives existing embedding models long-context capability through context-window extension.Source: §2 · p. 2

Evaluation Setting

• The paper’s benchmark setup notes that inputs exceeding each model’s supported context length are truncated.Source: §2 · p. 2

LLM Context Extension

• LLM context-extension methods include divide-and-conquer chunking, position reorganization, and position interpolation in plug-and-play settings.Source: §2 · p. 3

Efficient Extension

• Further fine-tuning on top of plug-and-play methods has been shown to improve performance, while other work simulates long inputs within short contexts.Source: §2 · p. 3

3.1 Examing Existing Retrieval Benchmarks

• LONGEMBED addresses short documents and biased target placement in existing retrieval benchmarks by requiring long candidates and broadly distributed answer-relevant information.BEIR averages fewer than 300 words per document, while LoCo has biased key-information placement.Source: §3.1 · p. 3

3.2 Synthetic Tasks in LONGEMBED

• The synthetic passkey and needle tasks insert target information at random positions, enabling controlled evaluation across context lengths up to 32k tokens.Each context length uses 50 queries with 100 candidate documents.Source: §3.2 · pp. 3–4
• LONGEMBED spans documents averaging 0.25 to 32 × 0.75k words for the synthetic tests, while limiting each task to at most 10^3 candidate documents.The candidate cap controls the quadratic computational cost of long-input evaluation.Source: Table 1 · p. 4, §3.3 · p. 5

3 The LONGEMBED benchmark

• The benchmark construction separates flexible control over length and target distribution in synthetic tasks from the realism of dispersed information in real-world datasets.Synthetic tasks provide manipulable context lengths, whereas real tasks reflect long-form QA and summarization settings.Source: §3.2 · p. 4, §3.3 · pp. 4–5

3.3 Real Tasks in LONGEMBED

• The benchmark also adapts NarrativeQA, 2WikiMultihopQA, QMSum, and SummScreenFD to test retrieval over long stories, multi-hop evidence, meetings, and screenplays.These tasks require aggregating details dispersed throughout the input rather than relying on a short span.Source: §3.3 · pp. 4–5

APE and RoPE

• RoPE encodes relative positions through rotations applied to query and key vectors, making attention depend on their relative position difference.The attention score is represented as a function of q, k, and (m − n)θ.Source: §4.1 · p. 5

Context Extension Strategies

• The paper evaluates three extension categories: divide-and-conquer, position reorganization, and position interpolation, with additional fine-tuning on selected methods.PCW splits long documents into short chunks and averages their embeddings; GP, RP, and PI modify position handling or embeddings.Source: §4.2 · p. 5

APE Extension

• For APE models, GP maps position ids to floor(pid/s), RP reuses them modulo the original context length, and PI maps them to pid/s with interpolated embeddings.These strategies extend the original position embedding matrix to the target length.Source: §4.2 · p. 6

Further Tuning

• Fine-tuning freezes original APE model parameters while updating added position embeddings, preserving behavior within the original 512-token context.The paper explores further tuning on RP and PI rather than PCW.Source: §4.2 · p. 6

RoPE Extension

• SelfExtend preserves normal relative positions within a nearest-neighbor window, while NTK interpolation rescales RoPE frequencies to extend context.SelfExtend is presented as improving on grouped relative positions, whereas NTK modifies the frequency schedule to mitigate losses from uniform frequency reduction.Source: §4.3 · p. 6

5.2 Main Results

• Existing models score as high as 64.4 points on LONGEMBED, leaving substantial room for improvement in long-context retrieval.Among 512-context models, E5Base reaches 41.0 average points; scores increase for models supporting more than 4k tokens.Source: §5.2 · p. 7
• +15.6, +20.3, and +10.9 average-score points are achieved by extended E5Base, E5-RoPEBase, and E5-Mistral models, respectively.E5Base and E5-RoPEBase are extended from 512 to 4,096 tokens, while E5-Mistral is extended from 4,096 to 32,768.Source: §5.2 · p. 7

5.3 Comparison of Extension Methods

• RoPE-specific NTK and SelfExtend methods significantly outperform PCW, PI, and GP across LONGEMBED datasets.Table 3 reports results for E5-RoPEBase and E5-Mistral across the benchmark’s datasets and extension methods.Source: §5.3 · p. 8, Table 3 · p. 8

5.2 Main Results

• Further tuning consistently improves E5Base and GTEBase across target context lengths, with GTEBase gaining approximately 5 average points.The tuning updates added position embeddings while freezing original model weights, maintaining the models’ original ability.Source: §5.3 · p. 8, Figure 5 · p. 8

5.3 Comparison of Extension Methods

• Tuning on PI consistently outperforms tuning on RP for both GTEBase and E5Base.The paper suggests fixed vectors in PI act as anchors that prevent learnable vectors from converging to suboptimal values.Source: §5.4 · p. 8, Figure 6 · p. 8
• RoPE’s advantage becomes more pronounced at longer target lengths, surpassing the fine-tuned E5Base by a large margin.E5Base and E5-RoPEBase were selected for comparison because they share training data and procedure and have comparable baseline performance.Source: §5.4 · pp. 8–9

• Training-free context window extension strategies increase existing embedding models’ input lengths by several folds on the LONGEMBED benchmark.Source: §6 · p. 9
• RoPE-based embedding models outperform APE-based models for context window extension, motivating RoPE for future embedding models.Source: §6 · p. 9

• Most explored context-extension strategies are training-free, limiting the paper’s coverage of training-based extension methods.The authors identify further fine-tuning, especially for RoPE-based models, as future work.Source: Limitations · p. 9

• The paper declares full compliance with the ACL Ethics Policy and reports no known ethical issues.Source: Ethics Statement · p. 9

Training procedure

• E5-RoPEBase follows E5’s contrastive pretraining and fine-tuning data, using CCPairs, MS-MARCO, NQ, and NLI with seven hard negatives per example.Source: A Training Details for E5-RoPEBase · p. 13

Initialization

• E5-RoPEBase replaces BERTBase’s absolute position embeddings with RoPE, then relies on subsequent pretraining to adapt the model.Source: A Training Details for E5-RoPEBase · p. 13

Pre-training length and batch size

• E5Base retains a 512-token position-embedding capability despite training at length 192, whereas E5-RoPE uses 512-token training with batch size reduced to 16k.Source: A Training Details for E5-RoPEBase · p. 13

BEIR comparison

• E5Base and E5-RoPEBase achieve comparable overall scores across 15 BEIR tasks, enabling a fair comparison of APE and RoPE for length extrapolation.Source: A Training Details for E5-RoPEBase · p. 13, Table 5 · p. 13

Plug-and-play strategies

• Table 6 summarizes hyperparameters for plug-and-play strategies including PCW, RP, GP, PI, NTK, and SE.Source: B Implementation Details for Context… · p. 13, Table 6 · p. 14

Further tuning

• Fine-tuning manipulates position ids by adding a randomly resampled skipping bias u, covering target positions from 0 through L_t − L_o.The procedure trains for 3 epochs on 2 A100 GPUs with learning rate 5e−4, batch size 512, and 100 warmup steps.Source: B Implementation Details for Context… · p. 14

Inference

• Inference uses attention scaling by default, while PI-tuned GTEBase and E5Base retain original position ids for inputs of at most 512 tokens.Source: B Implementation Details for Context… · p. 14

Dataset preparation

• LONGEMBED uses test splits for NarrativeQA and 2WikiMultihopQA, SCROLLS-processed versions for QMSum and SummScreenFD, and validation splits where SCROLLS test ground truth is unavailable.The 2WikiMultihopQA data are length-uniformly sampled to assess performance across context lengths.Source: C Further details on LONGEMBED · p. 14

• The significant gap between BM25 and E5-Mistral highlights substantial room for improvement in current long-context embedding models.Source: D BM25 Results on LONGEMBED · p. 14