Source-linked AI summary
$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, Maosong Sun
TL;DR
Long-context LLMs are increasingly needed, but standardized evaluation beyond roughly 10K tokens remains limited while models claim support for 100K-token contexts. The paper introduces ∞BENCH, a bilingual benchmark with synthetic and realistic tasks averaging over 100K tokens, and finds substantial performance degradation in current long-context LLMs.
Problem
Existing long-context benchmarks typically use contexts around 10K tokens, limiting standardized evaluation of models intended to process more than 100K tokens.
Method
∞BENCH combines English and Chinese synthetic and realistic tasks across diverse domains, including long-dependency tasks, and evaluates state-of-the-art proprietary and open-source long-context LLMs.
Results
Current state-of-the-art long-context LLMs show severe performance degradation when context lengths scale and are not fully equipped for all ∞BENCH tasks.
Takeaways & Limitations
∞BENCH provides a broader test of long-context understanding and reasoning, while its analyses offer directions for improving long-context LLMs.
Takeaways & Limitations
The benchmark may not be sufficiently diverse or extensive, exact-match scoring depends on prompts and answer parsing, and 100K-token contexts may not cover applications involving multiple books or entire databases.
Abstract
from arXiv · showhide
Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse domains, presented in both English and Chinese. The tasks in $\infty$Bench are designed to require well understanding of long dependencies in contexts, and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. In our experiments, based on $\infty$Bench, we evaluate the state-of-the-art proprietary and open-source LLMs tailored for processing long contexts. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context. We further present three intriguing analyses regarding the behavior of LLMs processing long context.
1 Introduction
Long-context processing is increasingly important, but existing benchmarks and models remain inadequate beyond 100K tokens. ∞BENCH addresses this gap with a bilingual, multi-domain benchmark and evaluation of current long-context LLMs.
- Long-context processing is critical for document comprehension, code analysis, web navigation, and other practical LLM tasks.
- LLMs often struggle beyond 8K tokens because long-sequence training is computationally demanding and length generalization remains limited.
- Existing long-context benchmarks generally use contexts averaging around 10K tokens, while many models claim support for over 100K tokens.
- ∞BENCH is the first comprehensive benchmark with average data length surpassing 100K tokens, covering English and Chinese tasks across novels, code, math, and other domains.
- The benchmark combines human-annotated and scalable synthetic tasks, then evaluates state-of-the-art long-context LLMs to assess their effectiveness and difficulty.
- Evaluation reveals severe performance degradation as context length scales, indicating that current long-context LLMs are not fully equipped for all benchmark tasks.
2 Related Work
Prior work extends context lengths through positional encoding, inference, and infrastructure techniques, while existing benchmarks mostly remain near 10K tokens. ∞BENCH differs through substantially longer contexts and broader task coverage.
- Long-context transformers address length limits through longer-context positional encodings and post-training inference techniques.
- Inference research improves long-sequence efficiency through IO management, memory optimization, parallelized decoding, and sparse or streaming attention methods.
- L-Eval and LongBench are prominent benchmarks centered on contexts of around 10K tokens, covering tasks such as QA, summarization, retrieval, math, and code.
- The benchmark compares annotation mode, example counts, and average input/output lengths across its data statistics.
- ∞BENCH distinguishes itself from prior benchmarks through substantially longer contexts and broader task domains, including novel, code, math, dialogue, and synthetic tasks.
3 ∞BENCH
∞BENCH contains diverse realistic and synthetic tasks designed to evaluate long-context processing beyond simple retrieval. Its tasks span multiple domains and languages while testing retrieval, resolution, state preservation, sequential processing, and long-range reasoning.
- ∞BENCH encompasses 12 tasks across five domains: retrieval, code, math, novels, and dialogue.Half of the newly introduced tasks are automatically generated, while the remainder are human-annotated.
- The benchmark combines realistic contexts with synthetic contexts engineered to test specific long-context capabilities.Realistic tasks target potential practical applications, whereas synthetic tasks isolate capabilities needed for processing lengthy contexts.
- Novel-based tasks: Novel-based tasks require reasoning over entire novels through summarization, open-form QA, and multiple-choice questions.Key entity replacement creates “fake novels” to reduce the influence of memorized source material.
- Novel-based tasks: En.QA and Zh.QA require long-range dependency reasoning through aggregation or filtering rather than simple short-passage retrieval.Aggregation combines information scattered throughout a novel, while filtering identifies specific information from a larger set.
- Synthetic tasks: Retrieve.PassKey places random five-digit pass keys at 59 evenly distributed context locations, yielding 590 examples.The task embeds each key in lengthy noisy context and asks the model to retrieve it.
- Synthetic tasks: Retrieve.Number increases retrieval difficulty with 10-digit answers containing successive repetitive digits, while Retrieve.KV retrieves values from large JSON key-value objects.These tasks assess local resolution and accurate key-value identification amid indistinguishable relevant and irrelevant information.
- Synthetic tasks: Code.Run uses cascading function calls with depths from 2 to 10, restricting each function to at most one addition or subtraction call.The design tests preservation of intermediate computational state across dependent calls.
4 Experiments
The experiments evaluate proprietary and open-source long-context models on ∞BENCH using available context limits and prompt optimization. Results show strong domain variation and substantial performance degradation as input length increases.
- Baselines: The evaluation includes four baselines: GPT-4, Claude 2, Kimi-Chat, and YaRN-Mistral.The first three are proprietary models, while YaRN-Mistral is open-source.
- Experimental setup: Each model-task combination receives a prompt crafted to optimize performance on short dummy examples.Detailed prompt templates are provided separately for each model and task.
- Experimental setup: Inputs exceeding API limits are rejected, while YaRN-Mistral inputs are truncated by removing the center and joining both ends.The truncation assumes key information, including instructions and book titles, is usually near the prompt’s beginning or end.
- Main results: GPT-4 outperforms other baselines in retrieval, code, and math, with a considerably higher average score.No distinct proprietary winner emerges on novel-based tasks.
- Main results: YaRN-Mistral lags behind proprietary models in most tasks and shows almost random performance in multiple areas.The baselines generally perform better on retrieval than on other task areas.
5 Analysis
The analyses show that longer inputs generally reduce baseline performance, while answer-position effects vary across models and tasks. A context-recalling prompt substantially improves GPT-4’s Code.Debug accuracy.
- 5.1 Length Ablation: Model performance generally declines as input length increases, despite baselines being technically equipped to process extended inputs.The shortened-context comparison indicates that effectiveness diminishes significantly under longer conditions.
- 5.2 Lost in the middle: Answer-position effects are inconsistent across models and tasks rather than uniformly producing a middle-context performance drop.GPT-4 prefers early answers in Retrieval.KV but later answers in En.Dia, while Claude 2 remains relatively unaffected.
- 5.2 Lost in the middle: Kimi-Chat’s steep middle-position drop on Retrieval.KV is caused by truncation removing the answer.The figure caption attributes this specific decline to answer truncation.
- 5.2 Lost in the middle: The authors hypothesize that “Lost in the middle” appears only for specific tasks and models, leaving a broader investigation beyond the paper’s scope.They attribute differing observations partly to differences in models, context lengths, and tasks relative to prior work.
- 5.3 Context Recalling: Context recalling prompts the model to repeat relevant code before reasoning, improving GPT-4’s Code.Debug accuracy from 15.74% to 39.59%.The technique is motivated by recalling information in generation before further analysis.
6 Conclusions
The paper introduces analyses of long-context behavior and finds that current LLMs still degrade substantially on contexts exceeding 100K tokens. These results motivate improved methods for efficient long-context processing and inform future research.
- 6 Conclusions: Contexts exceeding 100K tokens expose significant performance degradation in current LLMs despite their claimed proficiency with extensive contexts.The benchmark’s empirical evidence identifies continued difficulty in processing very long contexts.
- 6 Conclusions: The analyses provide insights into LLM behavior on long-context tasks and guide future research toward improving long-context processing efficiency.The conclusion frames these directions as implications of the benchmark findings.
Limitations
The benchmark has limited coverage and uses evaluation choices that may constrain how comprehensively model capabilities are assessed. Its 100K-token scope also falls short of applications involving much larger collections.
- The benchmark may not be sufficiently diverse or extensive to comprehensively assess model capabilities.The authors describe this as a constraint common to most benchmarks.
- Exact-match scoring depends on prompt templates and answer parsing, potentially requiring redesigns for new model evaluations.
- Supporting contexts up to 100K tokens may be insufficient for analyzing multiple books or entire databases.The authors identify million-token contexts as a promising research direction.
Ethics Statement
The authors discuss both the potential safety value and risks of improving long-context processing. Sensitive content may still occur despite annotation safeguards, while detailed pre-interaction specifications could improve guardrails but increase prompt-injection risk.
- Sensitive content may persist in sourced books or code despite annotators excluding offensive language and social biases.The authors note that the benchmark evaluates long-context capabilities rather than social bias.
- Improved long-context proficiency could support more effective guardrails by incorporating detailed specifications before user interactions.
- Adding detailed pre-interaction specifications could also increase the risk of novel prompt injection attacks.
A RWKV
RWKV combines transformer and recurrent-network properties, enabling recurrent inference with O(1) complexity and context-length-independent memory usage. In this benchmark, however, RWKV-4-World-7B produced unintelligible outputs and zero performance on Retrieve.PassKey, so it was not tested on other tasks.
- A RWKV: RWKV combines transformer capacity with recurrent-network inference, enabling parallelized training and O(1) inference complexity.
- A RWKV: RWKV’s memory usage does not scale with context length, allowing it to support arbitrary-length inputs architecturally.
- A RWKV: Zero performance on Retrieve.PassKey was observed for RWKV-4-World-7B, which output unintelligible text on the benchmark.The model was not trained on inputs of this length, and the authors associate the result with train-test domain shift.
- B.1 Retrieve.PassKey: Retrieve.PassKey hides an important pass key inside a long noisy context and asks the model to find it.
- A RWKV: The authors caution that this result does not show RWKV’s architecture is incapable of handling lengthy inputs.
- B Prompt Templates: GPT-4 has a strong tendency to reject the long-arithmetic task under some templates, claiming the platform is not designed for it.
C Annotation Process
The authors perform the annotation themselves and acknowledge fatigue-related errors. They mitigate this risk through independent quality checks, while excluding key-entity replacement from some little-known or newly published novels when models could not identify the entities.
- The paper’s authors completed the annotations without payment and acknowledged the intended uses and requirements.
- Each annotation was quality-checked by at least two other annotators to mitigate fatigue-related errors.
- Some novels omit key-entity replacement because models failed to identify entities in brand-new or little-known books.
D Evaluation Process
The evaluation process used different access methods, costs, and computational resources across the evaluated models. YaRN-Mistral’s maximum output-token setting is documented separately.
- GPT-4 evaluation used the official API with default hyperparameters and cost around 5000 US dollars.
- Claude 2 evaluation involved manually entering contents on the web, requiring three authors for several weeks and about 160 US dollars in membership fees.
- Kimi-Chat was free, while YaRN-Mistral inference used one A100 80GB GPU and took roughly 10 minutes per example, requiring several days for the full benchmark.
- Table 5 reports the maximum number of output tokens used as a decoding hyperparameter for YaRN-Mistral.