PaperlayerView exact revision on arXiv

Source-linked AI summary

NoLiMa: Long-Context Evaluation Beyond Literal Matching

Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze

arXiv:2502.05167v3cs.CL

TL;DR

Long-context benchmarks can overstate retrieval ability when literal matches help models locate relevant information. NoLiMa minimizes question–needle lexical overlap and evaluates 13 long-context models, finding substantial degradation as contexts grow, including at 32K tokens.

  • Problem

    Existing long-context benchmarks often contain literal matches between questions and relevant information, leaving limited evidence about retrieval through latent associations.

  • Method

    NoLiMa embeds needles with minimal lexical overlap into irrelevant haystacks and evaluates models’ latent associative reasoning across context lengths.

  • Results

    Performance declines as context length increases; at 32K tokens, 11 of 13 models achieve only half of their short-context performance.

  • Takeaways & Limitations

    NoLiMa exposes limitations of literal matching and provides a more demanding evaluation of reasoning in long-context settings.

Abstract

from arXiv · show

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

1. Introduction

NoLiMa addresses the limitation of literal matching in long-context benchmarks by testing whether models can retrieve relevant information through latent associations. Across 13 models, performance declines as context length increases, and neither reasoning models nor CoT fully resolves the problem.

  • 1. Introduction: Literal matches can simplify retrieval and answer generation in existing long-context benchmarks.The benchmark asks whether matching cues help models locate relevant information and produce correct answers.
  • 1. Introduction: NoLiMa minimizes lexical overlap between questions and needles, requiring models to use latent associative reasoning instead of surface matching.Questions and needles are linked through real-world knowledge or commonsense associations and embedded in irrelevant haystacks.
  • 1. Introduction: At 32K tokens, 11 of 13 evaluated models achieve only half of their short-context performance.The evaluation covers state-of-the-art models claiming support for at least 128K tokens.
  • 1. Introduction: Latent-hop count and fact direction both influence task performance.NoLiMa’s analyses vary associative reasoning steps and the ordering of elements within fact statements.
  • 1. Introduction: As latent reasoning complexity grows, performance depends more on context length than needle position.The aligned-depth analysis attributes the longer-context difficulty to attention being overwhelmed without surface cues.
  • 1. Introduction: CoT prompting and reasoning-based models improve performance but do not fully mitigate failures beyond 16K tokens.The limitation persists particularly in contexts exceeding 16K tokens.

2. Related Work

Prior long-context benchmarks extend NIAH with distractors, chaining, or reasoning, but may become too difficult at short lengths or retain literal cues. These limitations motivate evaluations that separate long-context retrieval from surface matching.

  • 2. Related Work: NIAH extensions increase difficulty through additional needles, fact chaining, arithmetic, or code reasoning.Some extensions require inter-needle reasoning or other forms of in-context computation.
  • 2. Related Work: Some complex benchmarks are already difficult in short contexts, including BABILong’s 28% accuracy on counting at 0K.The cited result occurs without irrelevant background text.
  • 2. Related Work: Literal matches frequently occur between question-relevant facts and questions across synthetic retrieval and downstream QA benchmarks.Similar documents may be added as distractors, but matching cues can remain available.
  • 2. Related Work: NOLIMA’s needle-set templates use paper-specific query and needle keywords linked to a randomly selected answer character.The templates include default and inverted ordering variants.

3. NOLIMA

NOLIMA adapts the needle-in-a-haystack setup so questions retrieve needles through latent associations rather than literal overlap, while filtering haystacks for distractors and misleading answer candidates.

  • Benchmark design: The benchmark aims to make associative reasoning sufficient while removing the utility of surface-level matching, isolating latent retrieval in long contexts.Needle constraints favor simple, uniquely associated keywords and use preface phrases to separate needles from preceding context.
  • Benchmark design: NOLIMA places a single relevant needle inside an irrelevant book-snippet haystack and tests whether a model can retrieve it from a question.This preserves the basic NIAH structure while changing the relation between question and needle.
  • Benchmark design: Questions and needles minimize literal overlap, requiring models to connect question keyword Wq with needle keyword Wn through world knowledge or commonsense reasoning.For example, Dresden links to the Semper Opera House, which identifies Yuki; a two-hop Saxony relation further increases difficulty.
  • Haystack filtering pipeline: Haystack filtering removes distracting words with extreme literal or semantic similarity to question key points and information that could form plausible false answers.Contriever embeddings identify candidate distractors for manual inspection, while a semi-automatic redaction process flags unintended answers for review.
  • Haystack filtering pipeline: The filtering pipeline scans haystacks in approximately 250-token chunks using an 800-character stride and 1000-character chunk size before manual review of flagged candidates.An instruction-tuned model labels chunks as “N/A” or explains a possible candidate answer; unflagged text remains unchanged.

4. Experiments

Experiments show that NOLIMA remains easy at short contexts but exposes sharp length-generalization failures, especially when associative reasoning is required without literal cues. These failures persist across placement, reasoning-prompting, and literal-match analyses.

  • Dataset Configuration: 58 question-needle pairs use default and inverted word-order templates, with one-hop and two-hop associative examples embedded in book-based haystacks.The default places CHAR before Wn, while the inverted template places CHAR after Wn.
  • Results: 11 of 13 models perform at 32K with half or less of their base scores, despite most achieving high short-context baselines.Llama 3.1 70B falls from 94.3% base score to 42.7% at 32K, with an effective length of only 2K.
  • Latent Hops & Inversion: Two-hop examples are more difficult than one-hop examples at the same context lengths, and their performance gap widens as context length increases.GPT-4o handles both types effectively up to 4K, but the aggregate trend favors one-hop tasks as contexts grow.
  • Latent Hops & Inversion: Inverted templates are harder because the causal attention path from the question to the character must backtrace through the needle keyword.The default ordering permits a direct link from Wq to Wn, whereas inversion weakens this route in longer contexts.
  • Needle Placement Depth Analysis: In two-hop tasks, increasing context length shifts performance downward even at context edges, while aligned last-2K results remain relatively stable across needle depth.The aligned setup holds relative question-fact distance constant, implicating increased context length rather than position encoding alone.
  • CoT Prompting: CoT and reasoning-based models improve some long-context results but do not restore full-length generalization, with all models below 50% at 32K on NOLIMA-Hard.Two-hop CoT examples barely reach one-hop scores without CoT, and intermediate reasoning steps do not solve long-context association linking.
  • Ablation Study: Literal Match Effect: Literal overlap makes long-context retrieval straightforward, whereas irrelevant overlapping distractors substantially degrade length generalization.With distractors, GPT-4o has an effective length of just 1K; base scores are 93.8 for GPT-4o and 84.4 for Llama 3.3 70B.

5. Conclusion

NOLIMA shows that long-context models struggle to retrieve relevant information when literal overlap is removed, exposing attention limitations and motivating evaluations beyond surface-level retrieval.

  • NOLIMA removes literal overlaps to test whether models can infer and link relevant information within extensive irrelevant content.
  • Models often rely on surface-level cues in longer contexts, causing them to overlook correct information when misleading literal matches are present.
  • Lexical gaps between queries and relevant documents may also hinder answer extraction in search engines and retrieval-augmented generation systems.
  • NOLIMA highlights the need for benchmarks that assess deeper reasoning rather than surface-level retrieval.

A. Needle Set Design & Considerations

NOLIMA’s needle set is designed to prevent accidental contextual or linguistic cues while creating uniquely associated query–needle pairs for associative reasoning.

  • Needle templates begin with an introductory phrase or word to distinguish each needle from preceding context.A newline is appended after each needle to reduce unintended linkage with prior context.
  • The placeholders [CHAR], Wq, and Wn denote the answer character, query keyword, and needle keyword, while Def. and Inv. indicate default and inverted fact order.
  • Needle keywords are selected to be uniquely associated with query keywords rather than ambiguously linked to multiple places or concepts.The Cambridge example illustrates why common associations are insufficiently specific.
  • The design minimizes language-specific markers so performance reflects learned knowledge rather than orthographic, morphological, cultural, or geographic cues.
  • Manual curation produced 28 keyword pairs and 58 needle-question pairs after accounting for fact-statement order.

B. Models

The evaluation uses a selected set of models, with open-weight models deployed through vLLM using weights obtained from HuggingFace.

  • Open-weight models were deployed using the vLLM library, with their weights obtained from HuggingFace.

C. Task Prompt Templates & Inference Settings

The evaluation uses prompts that encourage expanded reasoning while constraining generation, alongside model-specific decoding and token limits.

  • The CoT prompt encourages models to elaborate their reasoning without using the standard “Let’s think step by step” wording.
  • Reasoning is limited to three sentences or 192 generated tokens for the extensive evaluation workload.The stated testing scope is 7,540 tests per context length.
  • Table 8 lists the selected models used in the evaluation.
  • Standard instruction-tuned models use greedy decoding, while reasoning-based and R1-based models use their specified sampling mechanisms.GPT-o1 and GPT-o3 Mini use default sampling; R1-based models use top-P sampling with p = 0.95 and temperature 0.6.
  • Reasoning-based models are capped at 1536 generated tokens, including reasoning and output tokens.

D. Distractor Design

The evaluation adds neutral distractor sentences and controls their placement to reduce interference and positional advantages. Prompt templates specify whether answers should include reasoning or only the final answer.

  • Distractor templates: Two neutral templates insert unrelated sentences about Wq appearing in or pictured in a daily newspaper.The templates are selected based on grammatical fit and remain unrelated to each needle’s question or fact.
  • Placement constraints: Distractors are placed at least 20% of the context length from the needle and restricted to the middle 20%–80% region.These constraints minimize needle interference and avoid attention advantages near context boundaries.
  • Prompt templates: Prompt templates either request a brief reasoning explanation before the answer or require only a short final answer based on explicit facts or strong inferences.The reasoning template limits explanations to three sentences.
  • Prompt templates: Table 9 documents the prompt templates used in the evaluation.The table summarizes the template configurations rather than reporting benchmark outcomes.

E. Results Beyond 32K & Recent LLMs

Additional evaluations extend NoLiMa beyond 32K tokens and include newer models, showing that performance degradation persists at long contexts despite some model-specific improvements.

  • Evaluation setup: Haystacks are randomly concatenated from filtered book snippets, enabling evaluation at 64K and 128K tokens and further extension as model limits allow.The setup is based on the dataset configuration described for the benchmark.
  • Evaluation setup: Due to cost limitations, one evaluation uses 11 needle placements per context length instead of the default 26.This adjustment reduces API usage while preserving meaningful haystack coverage.
  • Results beyond 32K: GPT-4o maintains over 50% of its base score at 128K, while Gemini 2.0 Flash drops to 16.4% at that length.The evaluation focuses on GPT-4o and Gemini 2.0 Flash at 64K and 128K contexts.
  • Recent LLMs: GPT-4.1 improves over prior models, but its effective context length remains around 16K and falls below 65% at 128K.This remains well below its claimed 1M-token context length.

F. One- & Two-hop Results

Subset analyses report strong base scores for both one-hop and two-hop NoLiMa examples, while two-hop tasks generally support shorter effective context lengths.

  • Subset comparison: Two-hop tasks generally yield shorter effective context lengths than one-hop tasks despite strong base scores for both subsets.The comparison uses separately computed base scores and thresholds for each subset.
  • Reporting: Tables 11 and 12 report one-hop and two-hop results using the same format and thresholding criteria as Table 3.Scores below 50% of each subset’s base score are shaded in red.

G. Raw Needle Placement Depth Plots

Figure 6 shows raw needle-placement depth results before smoothing, corresponding to the smoothed results presented in Figure 3.

  • Raw results: Figure 6 presents unsmoothed needle-placement depth plots using raw performance values before the moving average.The plots correspond to the smoothed results in Figure 3.
Loading 2502.05167v3…