Source-linked AI summary
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
TL;DR
Long-context benchmarks often do not test deep understanding and reasoning across realistic tasks. LongBench v2 addresses this gap with a challenging multitask benchmark, finding that longer inference-time reasoning improves performance beyond the human baseline.
Problem
Existing long-context benchmarks often emphasize extractive or synthetic tasks and unreliable metrics, limiting their reflection of deep understanding across diverse real-world scenarios.
Method
LongBench v2 evaluates long-context understanding and reasoning using 503 multiple-choice questions spanning six realistic task categories, with contexts from 8k to 2M words and human review.
Results
57.7% accuracy from o1-preview surpassed the 53.7% human-expert accuracy, while the best model answering directly achieved 50.1%.
Takeaways & Limitations
LongBench v2 provides a challenging evaluation standard and indicates that enhanced reasoning and increased inference-time compute are important for long-context reasoning.
Takeaways & Limitations
The benchmark is limited to English and has uneven task-length distributions, complicating fair performance comparisons across length intervals.
Abstract
from arXiv · showhide
This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding. To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds. We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint. Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%. These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2. The project is available at https://longbench2.github.io.
1 Introduction
LongBench v2 addresses whether long-context LLMs can deeply understand, learn from, and reason over long texts in realistic multitask settings. It targets limitations of existing benchmarks through challenging, broad-coverage multiple-choice evaluation and reports substantial difficulty for both humans and models.
- Motivation: Existing benchmarks often test extractive or synthetic tasks, limiting their reflection of deep understanding in diverse real-world scenarios.They also commonly use F1 and ROUGE, which the paper characterizes as unreliable for these evaluations.
- Benchmark goals: LongBench v2 targets contexts from 8k to 2M words, challenging questions, realistic scenarios, and reliable multiple-choice evaluation.The benchmark is designed so that even human experts using document search cannot answer every question correctly in a short time.
- Benchmark scope: 503 questions span six categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repositories, and structured data.The data are in English and cover 20 subtasks.
- Construction: Automated and manual reviews use 97 annotators, 24 reviewers, and three long-context LLMs to maintain data quality and difficulty.Questions judged too easy by all three automated-review models are rejected before manual review.
- Results: 57.7% accuracy from o1-preview surpassed human experts, while the best direct-answering model reached 50.1% and humans reached 53.7% within 15 minutes.The paper links the stronger o1-preview result to longer reasoning during inference.
2 Related Work
Prior long-context benchmarks include broad multitask suites and targeted task-specific evaluations, but LongBench v2 is positioned as a more challenging, reliable, and comprehensive alternative. Its design uses a uniform multiple-choice format and manual verification.
- Benchmark landscape: Existing long-context benchmarks divide into comprehensive multitask suites and targeted evaluations of tasks such as QA, summarization, retrieval, conversation, coding, and learning.The related work lists representative benchmarks and task-specific datasets across these categories.
- LongBench v2: LongBench v2 responds by combining broad task coverage with a uniform multiple-choice format and manual verification of every data point.The stated goals are greater challenge, reliability, and comprehensiveness.
3 LongBench v2: Task and Construction
LongBench v2 combines realistic long-context task coverage with a multi-stage, human-involved construction and verification process. Annotators contribute documents and questions, while automated and manual review filter and revise submissions.
- Task design: Its design principle requires contexts from 8k to 2M words, difficult non-extractive questions, and broad real-world coverage.The benchmark aims to assess holistic abilities to reason, apply, and analyze information.
- Task design: The benchmark covers six major categories divided into 20 subtasks, including single-document QA, long in-context learning, and other realistic long-context tasks.The categories are defined from testing scenarios and long-text sources.
- Human involvement: 97 university-affiliated annotators and 24 professional human experts provide and review documents, questions, answer options, and evidence.Annotators are proficient in English and reviewers are selected according to major and year of study.
- Collection pipeline: The collection pipeline has five stages: document collection, annotation, automated review, manual review, and optional data revision.Figure 2 summarizes this process, which is implemented through an online annotation platform.
- Document collection: Documents are uploaded from materials annotators have personally read or used, then converted into plain text for diverse everyday scenarios.Examples include research papers, textbooks, and novels.
- Review: Automated review uses three 128k-context LLMs, while manual reviewers assess answers, objectivity, and correctness using document search when needed.Questions exceeding the automated reviewers’ context length are truncated from the middle.
- Revision and incentives: Rejected submissions are revised according to illegal-question, insufficient-difficulty, or wrong-answer feedback until they pass review.Difficulty rewards additionally favor longer and harder submissions, including cases that models and reviewers cannot solve readily.
- Verification: Verification found 68/70 sampled items completely correct and 67/70 Google-proofed, leading to an estimated error rate of around 3%.The authors use this sample to assess correctness and whether answers can be obtained through internet search.
4 Evaluation
LongBench v2 evaluates long-context models under zero-shot and zero-shot + CoT settings, comparing model, human, and retrieval-augmented performance across six tasks. Results show that longer reasoning helps, but performance remains uneven across context lengths, tasks, and retrieval settings.
- Baselines: 57.7% accuracy is achieved by o1-preview, exceeding human experts’ 53.7% under a 15-minute limit and substantially challenging current models.The best direct-answering model reaches 50.1%, while random guessing yields 25%.
- Baselines: 3.4% average improvement comes from CoT prompting for open-source models, while longer test-time reasoning improves o1-preview over GPT-4o by 7.6%.The corresponding o1-mini versus GPT-4o-mini improvement is 8.5%.
- Baselines: 15.4% is the best-model advantage over humans on contexts shorter than 32k, but models trail human accuracy by 5.6% on 32k–128k contexts.The results identify maintaining reasoning capability at longer context lengths as an important challenge.
- Baselines: The largest model–human gap occurs on long structured data understanding, while models perform at or above human levels on single- and multi-document QA.o1-preview outperforms GPT-4o on multi-document QA, long in-context learning, and code repository understanding.
- Retrieval-Augmented Baselines: RAG evaluation retrieves the top 4–256 chunks of 512 tokens after ranking chunks by embedding similarity between the question-plus-choices query and context chunks.The resulting performance curves are compared with each model’s full 128k no-RAG score.
- Retrieval-Augmented Baselines: Qwen2.5 and GLM-4-Plus show no significant improvement beyond 32k retrieval context, whereas GPT-4o performs best with RAG at 128k but remains 0.6% below its no-RAG score.Qwen2.5 improves by 4.1% at a 32k retrieval context compared with its full 128k no-RAG setting.
5 Conclusion
LongBench v2 is presented as a challenging, expert-reviewed benchmark for long-context understanding and reasoning. Its results place human and state-of-the-art model performance in a comparable range while highlighting inference-time compute and RAG as important evaluation dimensions.
- Conclusion: LongBench v2 provides a challenging evaluation standard for long-context understanding and reasoning, with human and state-of-the-art AI performance in a comparable range.The benchmark is carefully annotated and reviewed by human experts.
- Conclusion: 57.7% accuracy is achieved by the best LLM, compared with 50.1% human performance in the conclusion’s reported comparison.The paper also presents insights into scaling inference-time compute and RAG for long-context reasoning.
6 Limitations
The benchmark has limitations in size, language coverage, and length-distribution comparability. Its 503 examples may yield less stable results, the dataset is English-only, and uneven task-length distributions complicate cross-interval comparisons.
- Benchmark size: 503 examples may produce less stable results that are more vulnerable to randomness.The authors also report that resource constraints prevented dataset expansion.
- Language: The dataset is limited to English and does not capture model performance across multiple languages.
- Length distribution inconsistencies: Uneven task-length distributions make fair comparison of a single model across length intervals difficult.The authors recommend comparing models on a per-interval basis.
B Task Descriptions
The benchmark defines diverse task descriptions spanning document understanding, in-context learning, dialogue history, code, structured data, and multiple realistic domains.
- Single-Document QA: Single-document QA covers academic, literary, legal, financial, governmental, detective, and event-ordering tasks.Questions may require reasoning, interpretation, or reconstructing events from long documents.
- Multi-Document QA: Multi-document QA requires synthesizing information from at least two relevant documents without irrelevant documents.The listed domains include academic, legal, financial, governmental, and news documents.
- Long In-context Learning: Long in-context learning includes user-guide QA, rare-language translation, and many-shot classification from anonymized examples.The many-shot tasks cover named entity recognition, relation classification, event detection, and sentiment classification.
- Long-Dialogue History Understanding: Long-dialogue understanding uses agent-game histories and multi-turn chat histories to test questions about prior interactions.The chat-history source contains over 500 sessions for each history.
- Code and Structured Data: Other tasks require careful reading of codebases or integrating information across long tables, interconnected tables, and knowledge graphs.The code task targets multiple code locations, while structured-data tasks require combining cells, tables, or entities.
VI.2. Long Structured Data Understanding (Knowledge graph reasoning)
Knowledge graph reasoning requires integrating multiple entities through complex logical queries, while the surrounding annotation workflow supports systematic question construction and verification.
- Knowledge graph reasoning: Knowledge graph questions require integrating multiple entities from a Wikidata-derived graph.Ground-truth answers are automatically derived by executing the corresponding KoPL program.
- Knowledge graph reasoning: The task uses complex logical queries from KQAPro and automatically derives answers by running programs over the graph.
- Annotation workflow: The annotation workflow asks contributors to select a task, upload documents, and propose English multiple-choice questions with supporting evidence.The guidelines emphasize challenging questions involving reasoning, summarization, integration, or complex information extraction.
- Quality control: Automated filtering removes questions answered correctly by all three review models, followed by manual review and possible revision.Reviewers can inspect documents, answers, and evidence while tracking answer time.
C.4 Data Collection Cost
Data collection cost approximately 100,000 CNY.
- Collection cost: 100,000 CNY was spent on data collection.
D More Evaluation Details
The evaluation compares LongBench v2 against a broad set of open-source and proprietary baseline models.
- The benchmark evaluates numerous open-source baselines, including GLM, Llama, Qwen, Mistral, and Cohere models.
- The benchmark also includes proprietary baselines such as GLM-4-Plus and GPT-4o-mini.
D.2 Evaluation Setting
LongBench v2 uses separate prompting configurations for direct zero-shot answering and zero-shot chain-of-thought evaluation. These settings control generation length and response formatting.
- Zero-shot setting: Zero-shot evaluation uses temperature=0.1 and max_new_tokens=128.
- Zero-shot + CoT setting: Zero-shot + CoT first generates reasoning with max_new_tokens=1024, then produces the final answer with max_new_tokens=128.
- Prompt format: The prompts present the text, question, and four answer choices before requesting the correct option.
- Prompt format: The chain-of-thought prompt asks the model to think step by step before selecting a single answer choice.
- Response format: The final response is constrained to the format “The correct answer is (insert answer here)”.
E Deferred Experimental Results
Deferred results report LongBench v2 performance with YaRN and compensated scoring for invalid outputs. YaRN improves long-context processing, particularly beyond 32k tokens, while invalid outputs are relatively uncommon.
- Qwen2.5 results using YaRN: Table 4 reports Qwen2.5 results (%) using YaRN on LongBench v2, with higher scores bolded.
- Qwen2.5 results using YaRN: YaRN significantly enhances both Qwen2.5 models’ long-context processing on LongBench v2, especially for test cases longer than 32k.
- Invalid outputs: Invalid outputs are relatively rare and do not affect the conclusions drawn from the experimental results.
- Compensated results: Compensated results assign invalid outputs a 25% accuracy rate to address refusals and answer-format parsing failures.