PaperlayerView exact revision on arXiv

Source-linked AI summary

NoLiMa: Long-Context Evaluation Beyond Literal Matching

Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze

arXiv:2502.05167v3cs.CL

TL;DR

Many long-context benchmarks contain literal matches between questions and relevant information, leaving open whether models can locate facts without them. NoLiMa minimizes that overlap, and its evaluation finds that at 32K tokens, 11 of 13 models achieve only half their short-context performance.

  • Problem

    Existing long-context benchmarks often contain literal matches between queries and relevant information, leaving unclear whether models can locate facts without those matches.

    Source: §1 · pp. 1–2

  • Method

    NoLiMa minimizes lexical overlap between questions and needles and embeds needles linked to questions through associations, testing models’ latent associative reasoning.

    Source: §1 · p. 2

  • Results

    At 32K tokens, 11 of 13 models achieve only half their short-context performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NoLiMa highlights the need for benchmarks that test beyond surface-level retrieval, including cases where relevant documents have a lexical gap with the query.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

1. Introduction

NoLiMa reduces literal overlap between questions and needles to test whether models can use latent associations rather than surface matching. Across 13 long-context models, performance declines as context length grows, and the paper’s analyses show that reasoning methods do not fully overcome this challenge.

Benchmark motivation and design

  • NoLiMa minimizes literal overlap between questions and needles, requiring models to use associative links and latent reasoning instead of surface-level matching.The introduction argues that existing synthetic, NIAH-based, and document-QA benchmarks often contain literal matches that can simplify retrieval.Source: §1 · pp. 1–2

Evaluation results

  • At 32K tokens, 11 of 13 evaluated models achieve only half of their short-context performance, with noticeable declines already at 2K–8K.The decline also appears in newer models such as GPT-4.1, at context lengths relatively short compared with their claimed capacity.Source: §1 · pp. 1–2

Impact of Latent Hops and Fact Direction

  • The number of latent reasoning hops and the ordering of elements within a fact statement both influence task performance.These analyses examine how associative-step count and fact direction affect results.Source: §1 · p. 2

Context Length vs. Needle Position

  • As latent reasoning complexity grows, performance depends more on context length than needle position, and longer contexts overwhelm attention without surface cues.This conclusion comes from the paper’s aligned-depth analysis.Source: §1 · p. 2

Chain-of-Thought (CoT) Prompting and Reasoning-based Models

  • CoT prompting and reasoning-based models such as GPT-o1 (OpenAI et al., 2024) improve performance but do not fully mitigate the challenge, especially beyond 16K tokens.The prompting and models encourage step-by-step reasoning, yet long-context performance remains difficult to sustain.Source: §1 · p. 2

Ablation Tests

  • Literal matches substantially simplify answering, while using them as distractors severely impairs accuracy.The paper reports this contrast in its ablation tests.Source: §1 · p. 2

2. Related Work

Prior long-context benchmarks extend NIAH with harder retrieval and reasoning tasks, but some are difficult even at short lengths. Many benchmarks also retain literal question–context overlap that can cue retrieval; NoLiMa is designed to reduce that overlap.

Related Work

  • Some extensions become difficult even without long contexts: BABILong counting reaches 28% accuracy at 0K, while ATC tasks are overly complex below 1K (Kuratov et al., 2024; Li et al., 2024).Extensions include adding needles, fact chaining, and arithmetic or code reasoning.Source: §2 · p. 2
  • NIAH is described as the best-known and most widely used long-context benchmark, with later extensions responding to performance saturation.Those extensions add needles, chain facts, or require arithmetic and code reasoning.Source: §2 · p. 2
  • Literal matches persist from synthetic retrieval benchmarks to QA benchmarks, and similar-document distractors may not remove the cues they provide (Liu et al., 2024; Zhang et al., 2024; Bai et al., 2024; Yen et al., 2024).The passages attribute the usefulness of these cues to attention mechanisms’ ability to recall repetitive patterns (Olsson et al., 2022; Arora et al., 2024).Source: §2 · pp. 2–3
  • The paper measures question–context overlap using ROUGE-1, ROUGE-2, and ROUGE-L precision and reports that NoLiMa has much less literal overlap than other datasets.The context compared may be a needle, relevant document, or full document, depending on the benchmark setup.Source: §2 · p. 3

3. NOLIMA

NoLiMa isolates associative reasoning in long contexts by designing questions with minimal literal overlap with their relevant needles. Its haystack-filtering pipeline removes superficial distractors and plausible unintended answers.

3. NOLIMA

  • NoLiMa places a relevant needle in an irrelevant-text haystack and asks questions with minimal literal overlap, making surface-level matching unhelpful.The benchmark is designed to investigate associative reasoning without confounding from surface-level effects.Source: §3 · p. 3
  • Questions link to needle keywords through latent associations, including commonsense or world knowledge; some questions require two hops to connect them.For example, a question about Saxony requires connecting it to Dresden and then to the Semper Opera House.Source: §3 · p. 3
  • Needle sets use clear associations, randomized character names, unique keyword links, and usually preface phrases to reduce tokenization sensitivity, ethnic bias, ambiguity, and contextual spillover (Navigli et al., 2023; Jiang et al., 2024).Names already appearing in the haystacks are excluded.Source: §3 · p. 3

3.1. Haystack Filtering Pipeline

  • Haystack filtering removes words with extreme literal or high semantic similarity to question key points, and text that could support unintended answers.For distractors, Contriever embeddings and dot-product similarity identify the top 20 words per question keyword for manual review; sentences containing flagged words are removed.Source: §3.1 · p. 4
  • A semi-automatic process scans filtered haystacks in chunks, uses an instruction-tuned language model to flag possible answers, and has flagged text manually reviewed for removal.The process repeats across selected haystacks until no further removals are necessary.Source: §3.1 · p. 4

4. Experiments

Across NoLiMa experiments, performance often deteriorates sharply as context grows, and latent reasoning, position, and misleading literal overlaps help explain the challenge. Even reasoning prompts and models do not sustain strong performance at long lengths.

4.1. Dataset Configuration

  • The benchmark uses 58 question–needle pairs across five needle groups, with default and inverted word orders, and tests each needle at 26 placements in five haystacks.Haystacks are assembled from short snippets of 10 open-license books; each context-length experiment comprises 7,540 tests.Source: §4.1 · pp. 4–5

4.4. Results

  • At 32K, 11 of 13 models score at most half their base scores; Llama 3.1 70B scores 42.7% versus a 94.3% base, while GPT-4o remains above half its base at 128K.Most models with base scores above 90% have effective lengths of at most 2K; GPT-4o is an exception, though its generalization weakens beyond 8K. Model scaling generally helps, but gains diminish at larger scales.Source: Table 3 · p. 5, §4.4 · p. 6

4.2. Models

  • The evaluation covers widely used closed-source and open-weight models, with GPT-o1, GPT-o3 Mini, and DeepSeek-R1 Distill-Llama-70B added for reasoning-model analysis.Llama 3.3 70B was used for filtering, and achieved 99.8% on a control test of flagging conflicting information.Source: §4.2 · p. 5

4.3. Evaluation Setup & Metric

  • Models are tested from 250 tokens through 32K, with a base score derived from the 250-, 500-, and 1K results and effective length defined by an 85%-of-base threshold.For each question–needle example, base scoring averages over five haystacks, takes its maximum across the three shortest contexts, then averages across examples.Source: §4.3 · p. 5

4.4.1. LATENT HOPS & INVERSION

  • Two-hop questions become more difficult than one-hop questions as contexts lengthen, and inverted needle templates are harder than default templates.The paper attributes inversion difficulty to causal attention: the fact is incomplete before the character name appears, leaving weaker signals to establish the association.Source: §4.4.1 · p. 6, Figure 2 · p. 6

4.4.2. NEEDLE PLACEMENT DEPTH ANALYSIS

  • In two-hop tasks, longer contexts lower performance across the entire context window, including its edges; in last-2K tests, context length matters more than placement depth.The one-hop results show a stronger lost-in-the-middle pattern, while the two-hop results suggest that position encoding alone does not explain the decline: longer contexts make attention and retrieval harder without strong surface cues.Source: §4.4.2 · p. 7, Figure 3 · p. 7

4.4.3. COT PROMPTING

  • CoT prompting improves long-context test performance, with greater improvement in two-shot; nevertheless, two-hop CoT barely reaches one-hop without CoT and remains poor on texts of 16K tokens or longer.On NOLIMA-Hard, all models fall below 50% at 32K context length despite nearly perfect base scores.Source: §4.4.3 · p. 8

4.4.4. ABLATION STUDY: LITERAL MATCH EFFECT

  • Literal overlap makes retrieval easier, but irrelevant question-word distractors sharply hurt length generalization: with distractors, GPT-4o’s effective length is 1K.Direct questions state the needle keyword, while multiple-choice questions retain associative reasoning but offer four character-name options. Distractors lower base scores to 93.8 for GPT-4o and 84.4 for Llama 3.3 70B.Source: §4.4.4 · pp. 8–9, Figure 5 · p. 9

5. Conclusion

NoLiMa evaluates long-context reasoning without literal overlap between questions and relevant information, exposing models’ difficulty linking relevant facts amid irrelevant content. Its findings motivate evaluation beyond surface-level retrieval and approaches that handle complex long-context reasoning.

  • NoLiMa removes literal overlap between questions and relevant information, testing whether models can link relevant facts within extensive irrelevant content.The conclusion presents it as a challenging benchmark for evaluating LLM reasoning in long-context settings.Source: §5 · p. 9
  • Models struggle as context length increases, often relying on surface cues and overlooking correct facts when literal matches distract them.The conclusion identifies limits in models’ attention mechanisms as a key concern.Source: §5 · p. 9
  • In search and RAG, models may miss a correct answer in a relevant document with a lexical gap when other documents have greater query overlap.The conclusion argues that this risk underscores the need for benchmarks beyond surface-level retrieval and approaches for complex long-context reasoning.Source: §5 · p. 9

A. Needle Set Design & Considerations

NoLiMa’s needle templates are designed to reduce accidental links to preceding context and reliance on ambiguous or language-specific cues. The design uses uniquely associated keyword pairs, totaling 28 pairs and 58 needle-question pairs.

A. Needle Set Design & Considerations

  • Templates begin with an introductory phrase or word to distinguish the needle from preceding context, and a trailing newline helps mitigate accidental links when its keyword or character appears at the end.Examples include “Actually,” and “In 2013.”Source: A. Needle Set Design & Considerations · p. 13
  • Table 7 presents the needle-set templates, defining [CHAR], Wq, and Wn as the answer character, query keyword, and needle keyword, with default and inverted orders.The placeholders represent randomly selected values.Source: Table 7 · p. 13
  • Needle keywords should be uniquely associated with query keywords; “Cambridge” is ambiguous, and the design avoids language-specific markers that could reveal answers through surface cues.City names can contain orthographic, morphological, or cultural features that hint at linguistic or geographic origins.Source: A. Needle Set Design & Considerations · p. 13
  • 28 keyword pairs were manually curated across templates, yielding 58 needle-question pairs when statement order is considered.Each template contains 2–6 keyword pairs.Source: A. Needle Set Design & Considerations · p. 13

B. Models

The section identifies the evaluated models and describes how open-weight models were deployed.

B. Models

C. Task Prompt Templates & Inference Settings

The section specifies CoT prompt and scoring rules, along with decoding and output limits for standard and reasoning-based models.

C. Task Prompt Templates & Inference Settings

D. Distractor Design

Distractors are generated from two newspaper-based templates selected for each needle-question pair, then placed randomly under distance and context-position constraints.

D. Distractor Design

  • Distractors use one of two templates: an article or a photo of Wq in the daily newspaper.The templates are applied uniformly across needle-question pairs.Source: D. Distractor Design · p. 14
  • Template choice depends on whether Wq naturally includes an article, and both templates are neutral to the question’s intent and needle fact.Source: D. Distractor Design · p. 14
  • Distractors are placed randomly at least 20% of the context length from the needle and between the 20% and 80% context marks.These constraints leave a 40%-60% span of context length available for random placement.Source: D. Distractor Design · p. 14

E. Results Beyond 32K & Recent LLMs

Beyond 32K, GPT-4o retains over half its base score at 128K while Gemini 2.0 Flash falls to 16.4%; GPT-4.1 improves over prior models but remains limited at long contexts (Gemini Team et al., 2024).

E. Results Beyond 32K & Recent LLMs

  • The evaluation uses randomly concatenated book snippets to create haystacks for scalable 64K and 128K context lengths.Cost limitations reduce needle placements from 26 to 11 per context length; GPT-4o and Gemini 2.0 Flash use 127,500-token haystacks to accommodate GPT-4o’s 128,000-token input-and-output limit.Source: E. Results Beyond 32K & Recent LLMs · p. 15
  • At 128K, GPT-4o maintains over 50% of its base score, while Gemini 2.0 Flash drops to 16.4%.Table 10 reports the evaluation of both models at 64K and 128K context lengths.Source: E. Results Beyond 32K & Recent LLMs · pp. 15–16
  • GPT-4.1 improves over prior models, but its effective context length remains around 16K and its performance falls below 65% at 128K (Gemini Team et al., 2024).The evaluation extends newer models that performed more strongly at 32K to 128K using the same setup.Source: E. Results Beyond 32K & Recent LLMs · p. 16

F. One- & Two-hop Results

Across the one-hop and two-hop subsets, models score strongly at short contexts, but two-hop tasks generally have shorter effective context lengths, indicating weaker length generalization under greater reasoning demands.

F. One- & Two-hop Results

G. Raw Needle Placement Depth Plots

Figure 6 shows the needle placement depth plots corresponding to Figure 3 before the moving average used in the main figure was applied.

Related papers

Loading 2502.05167v3…