PaperlayerView exact revision on arXiv

Source-linked AI summary

RULER: What's the Real Context Size of Your Long-Context Language Models?

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg

arXiv:2404.06654v3cs.CL

TL;DR

Existing long-context evaluations rely heavily on retrieval tests that do not cover broader long-context understanding. The paper introduces RULER, a configurable synthetic benchmark with diverse tasks, and finds that nearly all evaluated models degrade on complex tasks as context length grows, while only half effectively handle 32K tokens despite claiming at least that context size.

  • Problem

    Widely used needle-in-a-haystack evaluations mainly measure retrieval and provide limited evidence about other forms of long-context understanding.

  • Method

    RULER is a configurable synthetic benchmark covering retrieval, multi-hop tracing, aggregation, and question answering across controlled sequence lengths and task complexities.

  • Results

    Nearly all models degrade on complex RULER tasks as sequence length increases, and only half effectively handle 32K sequences despite claiming context sizes of at least 32K tokens.

  • Takeaways & Limitations

    RULER provides a broader behavioral check of long-context capabilities than retrieval-only tests and is released to support further evaluation.

  • Takeaways & Limitations

    RULER’s proxy tasks have not been validated against realistic long-context tasks and should not be preferred over more realistic settings.

Abstract

from arXiv · show

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

1 Introduction

RULER addresses the limits of retrieval-focused long-context evaluation with a configurable synthetic benchmark spanning retrieval, multi-hop tracing, aggregation, and question answering. Across 17 models, performance generally deteriorates on complex tasks as context length increases, despite near-perfect vanilla NIAH results.

  • Motivation and benchmark design: RULER extends needle-in-a-haystack evaluation with diverse needles and adds multi-hop tracing, aggregation, and question answering.Its synthetic design supports controlled sequence lengths and task complexities.
  • Motivation and benchmark design: Synthetic RULER inputs reduce reliance on parametric knowledge and allow relevant or distracting information to be varied in volume and placement.These controls support customized contexts for different sequence lengths and task complexities.
  • Evaluation and findings: 17 long-context language models were evaluated with RULER across context lengths ranging from 4K to 128K.The benchmark includes Gemini-1.5, GPT-4, and 15 open-source models.
  • Evaluation and findings: Despite nearly perfect vanilla NIAH performance, almost all models show large degradation on more complex RULER tasks as sequence length increases.The result indicates that retrieval success does not guarantee robust long-context task performance.
  • Evaluation and findings: Only half of models claiming at least 32K-token contexts effectively handle 32K sequences, and almost all fall below the threshold before their claimed context lengths.Gemini-1.5 and GPT-4 consistently outperform other models under both weighted aggregation schemes.
  • Evaluation and findings: Yi-34B’s performance degrades substantially with longer inputs and greater task complexity, while larger model size correlates positively with long-context capability.At large context sizes, Yi-34B often returns incomplete answers or fails to precisely locate relevant information.

2 Related Work

Prior long-context research includes efficiency mechanisms for processing extended inputs and benchmarks using realistic, bilingual, very-long, or conversational tasks. RULER complements these efforts with synthetic, configurable evaluation.

  • Long-context language models: Flash attention, Ring attention, and sparse attention mechanisms reduce memory or computational demands for processing long contexts.Examples include shifted sparse attention, dilated attention, and attention sinks.
  • Long-context benchmarks and tasks: Existing benchmarks evaluate realistic natural-language tasks, bilingual settings, contexts exceeding 100K tokens, or long-term conversations.Examples include ZeroSCROLLS, L-Eval, LongBench, InfiniteBench, and LTM.

3 The RULER Benchmark

RULER evaluates long-context behavior across retrieval, multi-hop tracing, aggregation, and question answering with automatically generated tasks whose length and complexity can be configured. Its retrieval tasks extend NIAH with varied needles, distractors, and multi-item recall, while aggregation and QA test broader information use.

  • Benchmark design: RULER spans retrieval, multi-hop tracing, aggregation, and question answering, with task configurations controlling input length and complexity.Within its constrained domain, complexity depends on target output tokens and the context signal-to-noise ratio.
  • Retrieval: Retrieval tasks require matching end-of-sequence queries to key-value pairs embedded in long distractor contexts.The benchmark evaluates retrieval across needle and haystack types, resistance to hard distractors, and recall when multiple items must be retrieved.
  • Retrieval: Single NIAH varies keys and values across words, seven-digit numbers, and 32-digit UUIDs, with repeated noise sentences or Paul Graham essays as haystacks.This extends the vanilla single-needle setup across multiple content formats and distractor-text types.
  • Retrieval: Multi-keys NIAH inserts several needles but requires retrieving only one, making the remaining needles hard distractors.The most challenging version fills the haystack with distractor needles.
  • Retrieval: Multi-values and multi-queries NIAH require retrieving all values sharing a key or all needles with distinct keys, testing recall without missing critical information.These tasks extend associative recall from one retrieved item to multiple required outputs.
  • Aggregation: Aggregation samples synthetic vocabulary words to test extracting relevant information across long contexts, using uniform common-word sampling or frequency-controlled sampling.Common-word extraction fixes the number of common words while increasing uncommon words with sequence length; frequent-word extraction uses a rank-based frequency distribution.
  • Question Answering: Question answering inserts answer-containing paragraphs into randomly sampled distractor paragraphs, adapting short-context datasets to longer inputs.The question acts as the query, golden paragraphs as needles, and distracting paragraphs as the haystack.

4 Experiments & Results

The experiments evaluate 17 long-context language models on 13 RULER tasks across lengths from 4K to 128K, using threshold-based effective length and weighted averages for comparison. Although models perform nearly perfectly on vanilla NIAH, performance generally degrades as sequences grow, and claimed context lengths often exceed effective performance.

  • Experimental setup: 17 long-context models, including 15 open-source and two closed-source systems, are evaluated across claimed context lengths from 32K to 1M.Inference uses vLLM with BFloat16 on eight NVIDIA GPUs.
  • Experimental setup: 13 RULER tasks are tested with 500 generated examples per model, task, and length across 4K, 8K, 16K, 32K, 64K, and 128K tokens.Configurations span the four RULER categories and were selected because most models perform decently at 4K.
  • Evaluation criteria: Effective length is the maximum evaluated context length whose performance exceeds the Llama2-7B model’s 4K performance threshold.The threshold is used to compare claimed context size with the longest length a model handles satisfactorily.
  • Evaluation criteria: Weighted-average scores aggregate performance across context sizes using linearly increasing or decreasing weights to represent usage dominated by longer or shorter sequences.The two schemes are reported as wAvg. (inc) and wAvg. (dec).
  • Main results: Nearly perfect vanilla NIAH performance does not prevent large degradation on more complex RULER tasks as sequence length increases.All evaluated models fail to maintain performance above the Llama2-7B baseline at their claimed lengths, according to the main results.
  • Main results: Gemini-1.5-Pro has an effective length greater than the maximum tested length and outperforms the other evaluated models by a large margin.The reported performance at each length averages accuracy across all 13 RULER tasks.
  • Model comparisons: Large training context windows are not always necessary for strong long-context performance, since top-ranked open-source models use either 128K training or 32K inference-time extrapolation.Models trained at much larger context sizes can nevertheless be less performant.

5 Task Error Analysis

Yi-34B’s performance degrades across retrieval and non-retrieval tasks as context length, distractors, needle multiplicity, and task complexity increase. The errors include incomplete retrieval, copying, unreliable tracking, parametric answers, and hallucination.

  • At input lengths above 128K, Yi sometimes fails to return all 32 UUID digits, showing sensitivity to needle type.
  • At 256K, increasing distracting needles lowers Yi’s performance by ∼40 points in the full-haystack condition.Yi often retrieves values associated with distractor keys or nearby locations instead of precisely locating the target.
  • Increasing queries from 1 to 8 drops retrieval performance by ∼15 points, while repeated keys can produce duplicated rather than complete answers.
  • Over 80% of Yi’s CWE output at 128K is copied from the one-shot example, although copying is nonexistent for short sequences.Removing the example instead led Yi to copy the beginning of the input, likely due to attention sinks.
  • Adding chains or hops causes large variable-tracking degradation, including empty outputs and variables taken from other chains.More-hop degradation appears as context grows, while more-chain degradation is most significant beyond 128K.
  • In QA, Yi approaches its no-context baseline with distracting paragraphs, while aggregation errors involve hallucination, parametric knowledge, and inaccurate frequency discrimination.For frequent-word extraction, lowering α narrows frequency differences and makes top words harder to distinguish.

6 Model Analysis

Model analysis shows that longer training contexts generally improve long-context performance, while model size benefits performance and non-Transformer architectures lag the Transformer baseline. However, rankings can be inconsistent at long extrapolated lengths.

  • Effect of training context length: Larger training context sizes generally improve performance, but model rankings can become inconsistent on long sequences.LWM-1M is worse than LWM-512K at 256K, likely because it received insufficient training to adjust to the new RoPE base frequency.
  • Effect of model size: The 34B Yi model significantly outperforms the 6B model on RULER at 4K and in relative degradation when both use 200K training contexts.The comparison uses Yi-34B-200k, Yi-9B-200k, and Yi-6B-200k trained with the same data blend.
  • Effect of architecture: RWKV-v5 and Mamba-2.8B-slimpj significantly degrade by 8K and trail Llama2-7B by large margins through 4K.Beyond 4K, Llama2 itself shows poor length extrapolation performance.

7 Conclusion

RULER evaluates long-context capabilities beyond simple retrieval through diverse synthetic tasks and increasing context sizes. Across 17 models, performance generally deteriorates on complex tasks as input length grows, revealing persistent long-context failure modes and room for improvement.

  • RULER is a synthetic benchmark spanning retrieval, multi-hop tracing, aggregation, and question answering.
  • Across 17 long-context LMs, nearly perfect vanilla NIAH results coexist with declining performance on other RULER tasks as input length increases.
  • At large context sizes, models commonly fail to ignore distractors, use long-context information effectively, or avoid copying and parametric substitution.
  • RULER remains challenging for top-ranked open-source models as task complexity increases, while larger model sizes benefit long-context capability.

8 Limitations

RULER’s limitations concern evaluation coverage, realism, positional analysis, and prompt robustness. The authors also note that the reported model set differs between the main text and broader analyses.

  • Position control: RULER reports one score per input length and does not control the key information’s depth, limiting analysis of lost-in-the-middle effects.The authors plan to support position control in the codebase.
  • Realism: RULER’s variable tracking and frequent-word extraction tasks are proxies whose validity against realistic long-context tasks remains unverified.The authors recommend realistic settings such as NoCHA for capabilities including reasoning and instruction-following.
  • Short-context evaluation: RULER emphasizes degradation with increasing context length but does not report its more difficult short-context results.The authors caution that reasonable performance at 4K should not be interpreted as perfect capability.
  • Prompt robustness: RULER lacks a comprehensive study of prompt robustness and varies only minimally tested task hyperparameters.Examples include variable-name length and synthetic vocabulary size.
  • Evaluation scope: The main text evaluates 17 aligned models, while the broader selection contains 37 models and the analysis evaluates 11 models.The main-text set includes GPT-4, Gemini-1.5, and 15 open-source models.

B Task Configurations

RULER uses configurable synthetic tasks spanning retrieval, multi-hop tracing, aggregation, and question answering. The appendix specifies representative task variants and their configurations across these categories.

  • Task suite: RULER evaluates 13 representative task configurations across four categories, selected from a combinatorially large configuration space.The categories are retrieval, multi-hop tracing, aggregation, and question answering.
  • Retrieval: Retrieval tasks vary key-value types, haystack backgrounds, and needle quantities, including UUID retrieval and three distractor needles.The suite includes S-NIAH, MK-NIAH, line retrieval, and other existing setups.
  • Multi-hop tracing: Variable tracking uses one chain with four name-binding hops, requiring five variable names to be returned.This configuration operationalizes multi-hop tracing.
  • Aggregation: Common-word extraction requires 10 common words appearing 30 times each, while uncommon words appear three times each; frequent-word extraction uses α = 2.0.These tasks instantiate the aggregation category.
  • Question answering: The QA configurations extend SQuAD and HotpotQA to simulate long-context single-hop and multi-hop question answering.SQuAD represents single-hop QA and HotpotQA represents multi-hop QA.

C Task Correlation Analysis

RULER uses model-performance correlations across task configurations to assess whether its categories reveal distinct behaviors and to select representative tasks. The analysis compares 18 configurations before retaining 13 tasks.

  • Correlation procedure: Eight open-source models are evaluated across 18 task configurations and context sizes, representing each task by its performance vector.These vectors are used for task-correlation analysis and representative-task selection.
  • Task selection: The correlation heatmap identifies redundant tasks, which are removed so RULER preserves 13 representative tasks.The figure distinguishes words, numbers, UUIDs, and entire-haystack configurations.
  • Prompt construction: Prompt construction separates model chat templates from task templates that combine instruction, context, and query.An answer prefix is appended, and VT and CWE use one in-context demonstration.

E Passkey Retrieval and Vanilla NIAH Results

The appendix reports retrieval and category-level RULER results across context lengths from 4K to 128K. Passkey retrieval and vanilla NIAH remain nearly perfect at models’ claimed lengths, while aggregate tables cover broader task behavior.

  • Retrieval results: Almost all models achieve perfect scores at their claimed lengths in passkey retrieval and vanilla NIAH.These results are reported across lengths from 4K to 128K.
  • Aggregate results: RULER’s 13-task aggregate and its eight-task retrieval aggregate are reported across context lengths from 4K to 128K.The tables provide overall and retrieval-focused comparisons.
  • Category results: Separate tables report multi-hop tracing, aggregation, and question-answering performance across lengths from 4K to 128K.Aggregation averages two task scores, and question answering likewise averages two task scores.
Loading 2404.06654v3…