PaperlayerView exact revision on arXiv

Source-linked AI summary

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

arXiv:2412.15204v2cs.CLcs.AI

TL;DR

Existing long-context benchmarks often emphasize extractive or synthetic tasks rather than deep understanding across realistic scenarios. LongBench v2 tests this gap with a 503-question multitask benchmark, and o1-preview reaches 57.7% accuracy versus human experts’ 53.7% under a 15-minute time constraint.

  • Problem

    Existing long-context benchmarks often lack measures of deep understanding across diverse tasks, focusing instead on extractive questions or synthetic tasks with limited real-world applicability.

    Source: §1 · p. 1

  • Method

    LongBench v2 is a benchmark of 503 multiple-choice questions spanning six realistic task categories, with contexts ranging from 8k to 2M words.

    Source: §1 · pp. 1–2

  • Results

    57.7% accuracy: o1-preview with longer inference-time reasoning surpasses human experts’ 53.7% under a 15-minute constraint, while the best directly answering model achieves 50.1%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation suggests that scaling inference-time reasoning is relevant to addressing deep-understanding challenges in long-context scenarios.

    Source: §1 · p. 2

    The English-only dataset does not capture model performance across multiple languages.

    Source: §6 · p. 9

Abstract

from arXiv · show

This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding. To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds. We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint. Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%. These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2. The project is available at https://longbench2.github.io.

1 Introduction

LongBench v2 tests whether long-context LLMs can deeply understand, learn from, and reason over long texts, addressing limits of benchmarks centered on extractive and synthetic tasks.

1 Introduction

  • Despite context windows advancing from 8k to 128k and even 1M tokens, existing benchmarks often emphasize extractive or synthetic tasks over deep, realistic understanding (Bai et al., 2024b; Zhang et al., 2024d; Hsieh et al., 2024).The introduction also questions whether long-context models can deeply understand, learn from, and reason over the texts they process.Source: §1 · p. 1
  • Its contexts range from 8k to 2M words, with most under 128k, and the benchmark targets difficult, realistic scenarios using multiple-choice questions for reliable evaluation.Qualified questions should resist both three long-context LLMs and human reviewers searching the document for up to 3 minutes.Source: §1 · pp. 1–2
  • LongBench v2 contains 503 multiple-choice questions across 6 categories and 20 subtasks covering realistic long-context understanding scenarios.The categories are single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.Source: §1 · pp. 1–2
  • Data quality and difficulty are supported by 97 annotators, 24 reviewers, automated review by three long-context LLMs, and human checks.Questions deemed too easy when all three models answered correctly were excluded from review; reviewers assessed question appropriateness and answer correctness.Source: §1 · p. 2
  • 53.7% human-expert accuracy within 15 minutes compares with 25% random guessing; the best direct-answer model reaches 50.1%, while o1-preview reaches 57.7%.The o1-preview result surpasses human experts, and the authors identify inference-time reasoning as a promising way to tackle long-context challenges.Source: §1 · p. 2

2 Related Work

Prior long-context benchmarks include comprehensive multitask suites and targeted task-specific evaluations. The paper identifies gaps in deep reasoning and evaluation reliability, and responds with a uniform multiple-choice format and manual verification.

2 Related Work

  • Comprehensive benchmarks combine tasks such as QA, retrieval, and summarization; examples include ZeroSCROLLS (Shaham et al., 2023), L-Eval (An et al., 2024), and LongBench (Bai et al., 2024b).The paper notes that most multitask benchmarks appeared the prior year, alongside long-context models extending to 128k tokens or more.Source: §2 · p. 2
  • Existing benchmarks may reward shallow retrieval over deep reasoning, while ROUGE/F1 are unreliable and LLM-as-a-judge can be costly and biased.The authors note that many long examples have not been human-examined; their proposed response is a uniform multiple-choice format with manual verification of every data point.Source: §2 · pp. 2–3
  • Targeted long-context benchmarks evaluate specific tasks, including document QA, summarization, retrieval, conversation, coding, many-shot learning, and long-text generation.The paper distinguishes these task-specific evaluations from comprehensive multitask benchmarks.Source: §2 · p. 2

3 LongBench v2: Task and Construction

LongBench v2 is constructed to test deep reasoning over long, diverse real-world contexts through six task categories and a review-driven collection process. Verification and dataset statistics report the benchmark’s quality and scope.

3 LongBench v2: Task and Construction

  • The benchmark’s design targets contexts of 8k–2M words and questions requiring contextual reasoning rather than memory, direct extraction, or option-pattern guessing.Items are English multiple-choice questions with four choices, a ground-truth answer, and supporting evidence.Source: §3 · p. 3

3.1 Task Overview

  • LongBench v2 defines six task categories and 20 subtasks, spanning single- and multi-document QA, in-context learning, dialogue histories, code repositories, and structured data.Tasks include reasoning across documents, learning from long-context examples, and answering questions about tables and anonymized knowledge graphs.Source: §3.1 · p. 3

3.2 Data Collection

  • Annotators submitted personally read or used documents and proposed questions; 97 annotators and 24 professional experts participated in collection and review.Documents under 8,192 words or with high overlap with previous annotations were rejected; questions were designed to avoid simple retrieval and overly professional or tricky prompts.Source: §3.2 · pp. 3–4
  • Automated review flagged questions all three LLM reviewers answered correctly as too easy, followed by expert checks for qualification, correctness, and difficulty.Questions answered correctly by an expert within 3 minutes required revision; experts could stop after 15 minutes. Hard data earned an additional 50 CNY, and reviewers received 25 CNY per item.Source: §3.2 · pp. 4–6, Figure 2 · p. 5

3.3 Data Verification

  • 68/70 sampled entries were completely correct, 67/70 were Google-proof, and the authors estimate an error rate of around 3%.The sample of 70 was checked for correctness and whether answers could be found online within 15 minutes.Source: §3.3 · p. 6

3.4 Data Statistics

  • Of 503 entries, 192 were classified as easy and 311 as hard; 180 were short, 215 medium, and 108 long.Answers A–D accounted for approximately 19%, 25%, 30%, and 26%, respectively; manual review rejected 4% of submissions for illegal questions, 7% for insufficient difficulty, and 4% for wrong answers.Source: §3.4 · p. 6

4 Evaluation

The evaluation shows that LongBench v2 remains challenging for current models, compares their performance with human experts, and examines retrieval and context-free settings to probe how they handle long-context tasks.

4.1 Baselines

  • The evaluation covers 10 open-source and 7 proprietary models with 128,000-token context windows, using zero-shot and zero-shot-plus-CoT settings and middle truncation when inputs exceed a model’s window.Qwen2.5 models are evaluated without YaRN for a fair comparison.Source: §4.1 · p. 6
  • 57.7% accuracy: o1-preview exceeds human experts’ 15-minute performance by 4% on LongBench v2.Table 2 reports evaluation results alongside human expert performance; experts could answer “I don’t know” after the 15-minute limit.Source: Table 2 · p. 7, §4.1 · p. 7
  • Around 30% accuracy for smaller models contrasts with around or above 40% for larger counterparts; CoT improves open-source models by 3.4% on average.Longer reasoning also raises o1-preview over GPT-4o by 7.6% and o1-mini over GPT-4o-mini by 8.5%.Source: §4.1 · p. 7
  • 15.4% above human performance is the best model’s margin on data <32k, while it trails human accuracy by 5.6% on 32k–128k data.The models perform best relative to humans on short contexts, but the gap at longer lengths underscores the challenge of maintaining reasoning across longer contexts.Source: §4.1 · p. 7
  • The largest model–human gap is on structured-data tasks, while models match or surpass humans on single- and multi-document QA; o1-preview leads GPT-4o on multi-doc QA, long ICL, and code-repository understanding.The authors hypothesize that models’ greater exposure to document-type data during long-context training may explain their weaker structured-data understanding.Source: §4.1 · pp. 7–8, Figure 3 · p. 8

4.2 Retrieval-Augmented Baselines

  • At a 32k retrieval context, Qwen2.5 improves by 4.1% over using the full 128k context without RAG, while GPT-4o’s best RAG result at 128k is 0.6% below its no-RAG score.Qwen2.5 and GLM-4-Plus show no significant improvement beyond 32k retrieval context; the experiments also indicate that retrieval alone cannot solve the questions.Source: Figure 4 · p. 8, §4.2 · p. 8

4.3 Measuring Memorization of Context

  • 25% to 30% overall accuracy without context is comparable to random guessing; context-free performance suggests minimal memorization on tasks II, III, and VI.Without context, models perform best on tasks I and V, possibly because they encountered some documents, novels, or code repositories during training.Source: Table 3 · p. 8, §4.3 · pp. 8–9

5 Conclusion

LongBench v2 is a carefully reviewed multitask benchmark for long-context understanding and reasoning, and its evaluation reports human and best-LLM accuracy alongside insights about inference-time compute and RAG.

5 Conclusion

  • Human performance is 50.1%, while the best LLM achieves 57.7% accuracy on LongBench v2.The paper presents the benchmark as a reliable evaluation standard for developing future superhuman AI systems.Source: §5 · p. 9
  • LongBench v2 introduces a challenging multitask benchmark for long-context understanding and reasoning, carefully annotated and reviewed by human experts.Source: §5 · p. 9
  • The evaluation also provides insights into how scaling inference-time compute and RAG affect long-context reasoning.Source: §5 · p. 9

6 Limitations · Appendix · A Author Contributions

The authors note that LongBench v2’s size, English-only coverage, and uneven task-length distribution limit how its results can be interpreted. The author-contributions list assigns responsibilities across benchmark design, annotation, evaluation, writing, supervision, and fundraising.

6 Limitations

  • The 503-sample benchmark may yield unstable results, and collecting it cost 100,000 CNY and took more than two months.The authors cite resource constraints as preventing expansion at this time.Source: §6 · p. 9
  • The English-only dataset does not yet capture model performance across multiple languages.Source: §6 · p. 9
  • Uneven task-length distributions make cross-interval model comparisons difficult; the authors recommend comparing models within each length interval.They note that one model may outperform another on short tasks while losing on long tasks.Source: §6 · p. 9

A Author Contributions

B Task Descriptions … VI.1. Long Structured Data Understanding (Table QA)

The task descriptions span single- and multi-document question answering, long in-context learning, dialogue-history understanding, code-repository questions, and structured-table reasoning. They distinguish tasks by source material and the kinds of inference or integration required.

I.1–I.5. Single-Document QA (Academic, Literary, Legal, Financial, Governmental)

  • Single-document QA covers academic, literary, legal, financial, and government texts, with questions about their arguments, content, rules, policies, or analysis.Government-document questions include budget allocations, grant eligibility, and policy concerns.Source: §I.1 · p. 14, §I.2 · p. 14, §I.3 · p. 14

I.6–I.7. Single-Document QA (Detective; Event ordering)

  • Detective questions require inferring events such as a murderer or crime method after reading most of a novel, while event-ordering questions arrange four plot events into their original sequence.Source: §I.6 · p. 15, §I.7 · p. 15

II.1–II.5. Multi-Document QA

III.1–III.3. Long In-context Learning

IV.1–IV.2. Long-dialogue History Understanding

  • The long-dialogue task takes a multi-turn user–AI assistant chat history as context and asks a question requiring understanding of that history; histories are sampled from LongMemEval (Wu et al., 2024a).Source: IV.2. Long-dialogue History… · p. 16

V.1. Code Repository Understanding (Code repo QA)

  • Code-repository questions target a specific branch or commit and require careful reading across code or deep understanding of its contents.Source: §V.1 · p. 16

VI.1. Long Structured Data Understanding (Table QA)

VI.2. Long Structured Data Understanding (Knowledge graph reasoning) … C.3 Annotation Guidelines

The paper combines knowledge-graph questions with a platform-based process for collecting and reviewing challenging long-context QA. Its guidelines constrain document and question design, filter overly simple or incorrect data, and promote annotator diversity.

VI.2. Long Structured Data Understanding (Knowledge graph reasoning)

  • Knowledge-graph questions integrate multiple entities, using a Wikidata-derived graph, complex KQAPro-based queries, and KoPL programs to derive groundtruth answers.The cited works are Vrandeˇci´c and Krötzsch (2014), Cao et al. (2022), and Yao et al. (2023).Source: VI.2. Long Structured Data… · p. 16

C.1 Annotation Platform

  • The platform has main, annotation, and verification pages; annotations undergo word-count and duplicate checks, automated complexity review, and human review.Annotators submit English questions, options, answers, and evidence; reviewers cannot verify their own data.Source: §C.1 · pp. 17, 18, 21

C.2 Annotator Statistics

  • Annotators span majors and degree levels: CS 29%, Law 24%, and Economics 22%; 47% hold or pursue a Bachelor’s, 29% a Master’s, and 24% a PhD.Most are aged 20–26, and each annotator may contribute at most 20 items.Source: §C.2 · p. 21

C.3 Annotation Guidelines

  • Documents should be 8,000–2 million words, with as many as possible above 32,000; avoid overly common texts, or design niche questions for them to reduce training-set exposure.Source: §C.3 · p. 21
  • Questions and options must be in English, challenging beyond three minutes, and grounded in reasoning or information integration rather than retrieval or visual interpretation.Guidelines also discourage large-count questions, reliance on substantial outside expertise, and deliberately stilted difficulty.Source: §C.3 · pp. 21–22
  • Data is rejected if it fails requirements, all three models answer correctly, a human answers correctly within three minutes, or reviewers find the answer incorrect.Only submissions passing automated review proceed to human verification.Source: §C.3 · pp. 22–23
  • Reviewers download and open a verification file before answering under a timer, then check the annotator’s answer against the evidence; if unable after more than 15 minutes, they should not guess.Verification pays 25 CNY per piece of data verified. Annotators receive a 100 CNY basic reward per piece passing review, with conditional difficulty and document-length bonuses.Source: §C.3 · pp. 22–24

C.4 Data Collection Cost

Data collection cost approximately 100,000 CNY.

C.4 Data Collection Cost

  • Approximately 100,000 CNY was spent on data collection.Source: §C.4 · p. 24

D More Evaluation Details · D.1 Baseline Models

D.1 establishes a broad baseline suite spanning open-source and proprietary models, most with 128k-token context windows.

D.1 Baseline Models

  • Open-source baselines include GLM-4-9B-Chat (GLM et al., 2024), Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, and Llama-3.3-70B-Instruct (Dubey et al., 2024).Source: §D.1 · p. 24
  • The open-source suite also includes Llama-3.1-Nemotron-70B-Instruct (Wang et al., 2024b), Qwen2.5-7B-Instruct, and Qwen2.5-72B-Instruct (Team, 2024).Source: §D.1 · p. 24
  • Other open-source baselines are Mistral-Large-Instruct-2407 and Mistral-Large-Instruct-2411 (Jiang et al., 2023), plus c4ai-command-r-plus-08-2024 (Cohere For AI, 2024).Source: §D.1 · p. 24
  • Proprietary baselines include GLM-4-Plus (GLM et al., 2024), GPT-4o-mini-2024-07-18, and GPT-4o-2024-08-06.Source: §D.1 · p. 24
  • The proprietary suite also tests GPT-4o-2024-11-20, o1-mini-2024-09-12, and o1-preview-2024-09-12.Source: §D.1 · p. 24
  • Claude-3.5-Sonnet-20241022 is another proprietary baseline.Source: §D.1 · p. 24
  • All listed models have 128k-token context windows except Claude-3.5-Sonnet-20241022, whose window is 200k tokens.Source: §D.1 · p. 24

D.2 Evaluation Setting · D.3 Evaluation Prompts

The evaluation specifies separate decoding settings for direct zero-shot answering and two-call zero-shot chain-of-thought (CoT), with prompts that present the context and multiple-choice question and, for CoT, use generated reasoning before the final answer.

D.2 Evaluation Setting

  • Zero-shot generation uses temperature=0.1 and max_new_tokens=128.Source: §D.2 · p. 24
  • Zero-shot + CoT uses temperature=0.1 with max_new_tokens=1024 for chain-of-thought generation, then max_new_tokens=128 for the final-answer call.Source: §D.2 · p. 24

D.3 Evaluation Prompts

  • The zero-shot prompt asks the model to read the text and answer the question.Source: §D.3 · p. 24
  • The zero-shot prompt presents the question with four labeled choices, A through D.Source: §D.3 · p. 24
  • The zero-shot prompt requires the response format “The correct answer is (insert answer here)”.Source: §D.3 · p. 24
  • The first CoT prompt asks the model to read the text, answer the multiple-choice question, and “think step by step.”Source: §D.3 · pp. 24–25
  • The follow-up CoT prompt supplies the generated chain of thought and asks for the single most likely answer choice in the specified answer format.Source: §D.3 · p. 25

E Deferred Experimental Results

YaRN improves Qwen2.5’s LongBench v2 performance, particularly on longer contexts, while compensated handling of invalid outputs leaves the experimental conclusions unchanged.

Qwen2.5 Results Using YaRN

Compensated Results

  • Invalid outputs are relatively rare, and applying 25% accuracy compensation—including for human “I don’t know” responses—does not affect the experimental conclusions.The compensated results account for invalid outputs caused by refusals or answer-format errors under zero-shot prompting.Source: E Deferred Experimental Results · p. 25, Table 5 · p. 26

Related papers

Loading 2412.15204v2…