PaperlayerView exact revision on arXiv

Source-linked AI summary

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

arXiv:2308.14508v2cs.CL

TL;DR

Long-context applications expose a gap in evaluating whether language models can understand sequences beyond a few thousand tokens. LongBench addresses this with a bilingual, multitask benchmark of 21 datasets across six categories, finding that model performance declines on longer contexts and that context compression helps weaker long-context models but remains behind stronger ones.

  • Problem

    Many LLMs handle only a few thousand tokens, while applications involving books, reports, and codebases require understanding much longer sequences; comprehensive benchmarks are lacking.

  • Method

    LongBench standardizes 21 English and Chinese datasets across six long-text task categories into a unified format for automated evaluation.

  • Results

    Evaluation of eight LLMs finds performance declines as context length increases, while retrieval-based compression improves weaker long-context models but still trails stronger models.

  • Takeaways & Limitations

    LongBench and LongBench-E provide testbeds for comparing long-context modeling ability across tasks and context lengths.

  • Takeaways & Limitations

    Automatic ROUGE-L and F1 metrics may not accurately reflect response quality, particularly for models generating longer responses.

Abstract

from arXiv · show

Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability. The code and datasets are available at https://github.com/THUDM/LongBench.

1 Introduction

LongBench addresses the limited evaluation of long-context understanding with a bilingual, multitask benchmark spanning diverse long-text applications. It standardizes 21 datasets and supports automated evaluation across English and Chinese contexts.

  • Motivation: LongBench targets the gap between long-context applications and models that often handle only a few thousand tokens.The motivating applications include summarization, question answering, and repository-level code generation.
  • Benchmark: LongBench contains 21 tasks across six categories, including single-document QA, multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks.The benchmark covers both English and Chinese to evaluate bilingual long-context capabilities.
  • Benchmark: 4,750 test instances average 6,711 words for English and 13,386 characters for Chinese.These statistics quantify the benchmark’s long-context scale.
  • Evaluation: All 21 datasets are standardized into a unified format and evaluated automatically using metrics including ROUGE-L and F1.The construction combines datasets extracted from prior work, adapted for long-context evaluation, and newly created and annotated datasets.
  • Findings: Evaluation of eight models shows performance declines as context length increases, including for some models trained or fine-tuned on longer contexts.LongBench-E provides a more even length distribution to examine performance across context lengths.

2 Related Work

Prior long-context research develops efficient, recurrent, memory-based, and length-extrapolating models, while existing evaluations cover narrower task ranges. LongBench seeks broader evaluation across task types, lengths, languages, domains, and dependency patterns.

  • Long Context Modeling Techniques: Long-context modeling methods address runtime overhead and catastrophic forgetting in long sequences.Approaches include sparse or efficient computation and recurrent or memory modules.
  • Long Context Modeling Techniques: Length-extrapolation methods extend Transformer usability to longer sequences and appear in long-context models such as ChatGLM2-32k and LongChat-32k.These methods include scaled or modified positional representations and related training approaches.
  • Evaluation for Long Context Understanding: Existing long-text benchmarks such as ZeroSCROLLS and L-Eval encompass a restricted range of task types.This limits the diversity of long-text modeling patterns represented in their evaluations.
  • Evaluation for Long Context Understanding: LongBench spans six task categories with varying sequence lengths, languages, and domains, enabling more holistic evaluation of long-context modeling.Its design also covers varied length distributions and long dependency patterns.

3 LongBench: Task and Construction

LongBench formalizes long-context tasks as mapping short inputs and long contexts to answers, then constructs a bilingual benchmark from diverse datasets and task-specific procedures. Its collection includes QA, summarization, few-shot, synthetic, and code-completion settings with standardized statistics and evaluation.

  • 3.1 Problem Definition: LongBench defines each task as mapping input and context sequences (I, C) to an answer A, with C typically extending to thousands of tokens.For QA, I is the question, C is the document, and A is the answer.
  • 3.2 Dataset Construction: The benchmark organizes datasets by task-specific collection and annotation procedures, with overall statistics reported in Table 1.The construction covers both LongBench and its evenly distributed LongBench-E variant.
  • Question Answering: Single-document QA uses long NarrativeQA stories and Qasper questions over NLP papers, while multi-document QA combines supporting and distracting passages into randomly ordered contexts.Multi-document contexts are lengthened with complete Wikipedia passages or distractor documents.
  • Summarization: Summarization datasets require global understanding through government reports, meeting transcripts, multi-news articles, and long Chinese meeting segments.The benchmark represents both single- and multi-document summarization settings.
  • Few-shot Learning: Few-shot learning includes classification, summarization, and reading comprehension, with training examples concatenated into long contexts using task-specific sampling ranges.The included datasets are TREC, LSHT, SAMSum, and TriviaQA.
  • Synthetic Tasks and Code Completion: PassageCount tests full-context use by repeating and shuffling Wikipedia passages, while code completion evaluates long file-level and repository-level code contexts.Repository-level completion requires aggregating information across files.

4 Experiments

The experiments evaluate eight LLMs on LongBench and examine how context length, compression, and memorization affect long-context performance. Results show benefits from longer-context training and position scaling, but persistent degradation on longer inputs.

  • Experimental Setup: Eight LLMs are evaluated on LongBench using zero-shot prompting, with few-shot examples included in the long context and middle truncation when inputs exceed model limits.F1 and ROUGE-L are used for QA and summarization, while Edit Sim is used for code generation.
  • Results on LongBench: 62% and 19% relative improvements are obtained by ChatGLM2-6B-32k and LongChat-v1.5-7B-32k, respectively, highlighting gains from scaled positional embeddings and longer-context training.The comparison is reported across LongBench’s tasks.
  • Results on LongBench-E: GPT-3.5-Turbo-16k still drops 17% from 0-4k to 8k+ context lengths on LongBench-E despite strong overall performance.LongBench-E distributes examples more evenly across context-length ranges to assess performance changes within tasks.
  • Results on LongBench-E: Larger truncation sizes improve GPT-Turbo-3.5-16k and ChatGLM2-6B-32k scores, indicating that successful LongBench performance requires using longer context rather than truncated information alone.The experiment compares 4k and 8k truncation sizes, with maximum length defined by each model’s configuration.
  • Context Compression: Retrieval-based compression splits contexts into chunks, selects top-N chunks by embedding similarity, and feeds them with the query to the model.The study tests OpenAI Embedding, Contriever, and BM25 with chunk sizes of 200 and 500.
  • Context Compression: Summary-based compression improves performance only on VCSUM, where examples are longer than those in the other summarization datasets.The method summarizes each chunk and concatenates the summaries as compressed context.
  • Context Understanding or Memorization?: High scores without context on HotpotQA, 2WikiMultihopQA, and MusiQue indicate that memorization can affect multi-document QA evaluation.The score difference between original and no-context performance is proposed as an indicator of long-context understanding.

5 Conclusion

LongBench is introduced as a bilingual, multitask benchmark for evaluating long-context understanding across varied tasks and lengths. LongBench-E complements it with a more even length distribution, and experiments characterize current models’ long-context capabilities.

  • Conclusion: LongBench contains six task categories and 21 tasks spanning thousands to tens of thousands of tokens.LongBench-E provides a more evenly distributed set of data lengths for analyzing performance across context lengths.

6 Limitations

The benchmark has limitations in evaluation reliability and construct isolation. Automatic metrics may misrepresent response quality, while results remain coupled with instruction-following ability.

  • Evaluation Metrics: ROUGE-L and F1 may not accurately reflect response quality, particularly underestimating models that generate longer responses.Using an LLM as an evaluator may reduce this issue but introduces runtime overhead and evaluator bias.
  • Evaluation Scope: LongBench performance is coupled with instruction-following capability because its real-world-style tasks require some instruction following.This coupling complicates assessing long-context modeling independently of instruction-following ability.

A Dataset Details

The dataset details describe LongBench’s task instantiation, length distributions, document sources, and annotation process. These design choices support varied long-context evaluation across languages and domains.

  • Length Distribution: LongBench-E reports task data across 0-4k, 4k-8k, and 8k+ length ranges.Table 8 provides the number of examples in each range.
  • Task Format: LongBench represents each task as input I, context C, and answer A, with I and A generally short while C may span thousands of tokens.Table 7 lists the instantiation for each task.
  • Document Sources: MultiFieldQA documents come from academic papers, C4, WuDaoCorpora, legal judgments, Wikipedia, and government websites.The sources cover English and Chinese materials across several domains.
  • Annotation: MultiFieldQA annotation asks for clear, short-answer questions with evidence paragraphs distributed randomly throughout each document.The guidelines also require varied question types, including information extraction, summarization, and multi-hop reasoning.
  • Annotation: Each MultiFieldQA sample takes about five minutes to annotate, and cross-validation reports 100% accuracy for the annotated answers.The annotators are Ph.D. students with extensive NLP research experience.

B Evaluation Setups

LongBench uses task-specific prompts that pair an input with a long context and constrain the requested response format. The evaluation also limits output length and applies automated extraction or scoring conventions across tasks.

  • Prompt design: LongBench prompts each instance with a task-specific instruction, context, and input, then requests a constrained answer or completion.The general format treats I as the input, C as the long context, and A as the expected answer.
  • Prompt design: Question-answering prompts require concise answers, sometimes allowing “yes,” “no,” or “unanswerable” but disallowing explanations.This format appears across NarrativeQA, Qasper, and related QA tasks.
  • Prompt design: Summarization prompts request one-page or short summaries of reports, news collections, meetings, dialogues, or transcripts.The prompts specify the source material and the expected summary length or sentence format.
  • Prompt design: Few-shot and classification prompts provide examples or contextual passages before asking for a label, question type, or brief answer.These include TREC, TriviaQA, SAMSum, and LSHT task formats.
  • Prompt design: Synthetic and retrieval tasks ask models to count unique paragraphs or identify which paragraph contains an abstract.The required outputs are explicitly formatted as a number or paragraph identifier.
  • Prompt design: Code-completion prompts provide preceding code and request the next line, while evaluation caps output length to prevent non-stop generation.The code tasks are LCC and RepoBench-P.

C Radar Plot and Analysis

The radar analysis reports average scores across six major tasks in English and Chinese and cautions that task categories differ in how well their metrics distinguish model strength.

  • Radar plot: Average scores are plotted across six major tasks for English and Chinese datasets, respectively.The supplied caption identifies the language-specific averages but does not state individual values or rankings.
  • Task analysis: Summarization and code completion are less discerning because ROUGE-L and Edit Sim may not separate strong from weak models well.The authors attribute this possibility to the limited sensitivity of similarity-based metrics on these tasks.
  • Task analysis: Synthetic tasks are more discerning, with models tending toward either high scores or near-zero performance.This contrast motivates evaluating task categories separately rather than relying only on an overall average.

D Analysis on the Inter-task Correlation on LongBench

LongBench’s inter-task correlations reveal strong within-category and same-language relationships alongside several tasks with unusually low correlations.

  • Correlation patterns: Most tasks within the same category have high correlations, except PassageCount, which correlates weakly with almost all tasks.The authors associate PassageCount’s low correlation with near-random model performance on that challenging task.
  • Correlation patterns: Qasper and RepoBench-P also correlate less with other tasks, suggesting they may require different attention patterns.This is presented as a possible explanation for their distinct relationships with the remaining tasks.
  • Correlation patterns: Tasks in the same language show higher correlations, including a strong relationship among the Chinese tasks.The cited Chinese tasks are 1-4, 2-4, 3-4, 4-4, and 5-3.
  • Implication: The observed task and language relationships support integrating varied task types and languages for a more comprehensive evaluation.Figure 5 presents the inter-task correlation analysis underlying this conclusion.

E Full results on LongBench-E

The paper reports the complete LongBench-E results in Table 9, which presents performance percentages for the benchmark’s more evenly distributed context-length evaluation.

  • Full results: Table 9 reports performance percentages for all LongBench-E results.The supplied passages provide the table reference and title but no individual model scores.
Loading 2308.14508v2…