Source-linked AI summary
LooGLE: Can Long-Context Language Models Understand Long Contexts?
Jiaqi Li, Mengmeng Wang, Zilong Zheng, Muhan Zhang
TL;DR
Long-context LLM evaluation is limited by short or outdated documents and weak coverage of tasks requiring evidence integration across entire documents. The paper introduces LooGLE, a benchmark built from long, recent, diverse documents and manually designed dependency tasks, and evaluates eight LLMs. Results show strong short-dependency performance but substantial difficulty with long dependencies, motivating further progress in true long-context understanding.
Problem
Existing long-context benchmarks often use short or outdated documents and emphasize short-dependency tasks, limiting evaluation of long-context understanding.
Method
LooGLE combines extra-long, relatively new, multi-domain documents with manually designed short- and long-dependency tasks and cross-validated human question-answer pairs.
Results
Eight evaluated LLMs performed better on short-dependency tasks than long-dependency tasks, with commercial models outperforming open-source models and all models showing significant long-dependency declines.
Takeaways & Limitations
LooGLE provides an up-to-date benchmark for assessing and improving LLM comprehension and reasoning over extended dependency contexts.
Takeaways & Limitations
The benchmark includes source materials that are not license-free, requiring some sources to be excluded.
Abstract
from arXiv · showhide
Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".
1 INTRODUCTION
LooGLE is introduced to address limitations in existing long-context benchmarks, including short documents, outdated sources, and insufficient testing of long dependencies. Its evaluation finds that current LLMs still struggle substantially with long-dependency understanding.
- Motivation: Existing benchmarks often use short, outdated, domain-specific documents and mostly test retrieving information from localized passages.These choices can limit comprehensive evaluation and may introduce data-leakage concerns.
- Benchmark design: LooGLE evaluates long-context understanding with extra-long realistic documents and diverse domains.The benchmark includes 776 documents averaging 19.3k words, with many exceeding 100k words, and covers sources such as arXiv papers, Wikipedia articles, and scripts.
- Benchmark design: LooGLE combines short- and long-dependency tasks, including 1,101 manually created long-dependency question-answer instances.Long-dependency questions require integrating evidence distributed across the document.
- Contribution: LooGLE uses relatively new documents and serves as an up-to-date benchmark for assessing long-context understanding and modeling.The benchmark is intended for cutting-edge evaluation and research.
- Evaluation findings: The authors evaluate eight representative LLMs and report that larger context windows and stronger base models generally improve performance, while all models decline on long-dependency tasks.This decline indicates persistent difficulty with genuine long-context understanding.
2 RELATED WORK
Prior work develops architectures and benchmarks for long-context processing, but existing datasets commonly rely on collected data and automatic evaluation. Recent benchmarks differ in document length, data sources, and task coverage.
- Existing models: Long-context modeling research explores recurrent memory, sparse attention, external memory, and improved transformer architectures.Efficient transformers aim to reduce memory and time complexity when modeling longer texts.
- Existing datasets: ZeroSCROLLS, L-Eval, and LongBench are recent benchmarks for evaluating LLM long-context understanding.ZeroSCROLLS unifies datasets from different sources with documents averaging 10k words, but mainly reuses existing datasets and relies on automatic metrics.
- Existing datasets: Existing benchmark design varies in document collection, input formatting, task construction, and evaluation methodology.The cited discussion contrasts automatically processed existing datasets with other recent long-context evaluation efforts.
3 THE LOOGLE BENCHMARK
LooGLE is a benchmark built from recent, long documents and combines short- and long-dependency tasks to evaluate long-context understanding. Its manually constructed long-dependency questions use distributed evidence and multi-stage validation to support precise assessment.
- Dataset and task design: LooGLE combines short-dependency tasks with long-dependency summarization and question-answering across scientific papers, Wikipedia articles, and movie or TV scripts.Long-dependency QA covers multiple information retrieval, timeline reorder, computation, and comprehension and reasoning.
- Dataset and task design: The benchmark uses post-2022 documents filtered to exceed 10,000 words, including 516 research papers selected after quality filtering.The source collection also includes recent Wikipedia articles and scripts, with books excluded because most qualifying books were not license-free.
- Dataset and task design: Long-dependency QA requires aggregating evidence distributed across lengthy documents, including multiple information retrieval and numerical computation tasks.Multiple-information-retrieval questions draw on diverse evidence throughout the text, while computation questions require relating retrieved numerical data to the question.
- QA construction: Annotators generated 5 to 10 questions per document, with recommended evidence spans of at least 5,000 words and limits on repeated question types.Questions were required to be clear, precise, deterministic, and objective.
- QA construction: The three-step annotation process used independent questioner and answerer roles, answer checking, feedback-based revision, and final answer unification.Answerers read the full document, assessed question quality, and flagged questions lacking definite and unambiguous answers.
- QA construction: 1,101 high-quality long-dependency QA pairs remained after 206 of 1,137 initial pairs were identified as non-compliant, with 81.88% inter-annotator agreement.The final pairs were intended to require strong long-context understanding.
4 EVALUATION
LooGLE evaluates eight LLMs across short- and long-dependency tasks, showing strong short-task performance but substantial difficulty with long-range dependencies. Commercial models generally outperform open-source models, while retrieval helps some tasks and chain-of-thought yields mixed gains.
- Evaluation setup: Eight representative LLMs were evaluated on LooGLE using commercial and open-source baselines, including retrieval-based LlamaIndex.The evaluation also considers context truncation, automatic metrics, output formats, and chain-of-thought prompting.
- Overall results: GPT4-32k achieved the strongest overall performance, while open-source models showed a significant gap behind commercial models.GPT4-32k scored highest on every task except summarization in the overall comparison.
- Short versus long dependency: Short QA, cloze, and summarization were handled more effectively, whereas long-dependency QA remained difficult; even GPT4-32k barely exceeded 40% accuracy there.Long-dependency tasks require evidence spanning widely separated parts of the document.
- Input length: Longer context windows improved long-dependency QA, but extending input length had little effect on summarization.GPT4-8k performed much worse than GPT4-32k on long tasks, while summarization already draws heavily on information near the document’s beginning and end.
- Retrieval: Retrieval-based context compression was evaluated through LlamaIndex, with embedding choice affecting performance but not changing the study’s conclusions.The open-source all-mpnet-base-v2 embedding performed worse than text-embedding-ada-002.
- Prompting: Zero-shot chain-of-thought produced minimal gains overall, although it increased timeline-reorder accuracy by 20% and computation accuracy by 10% absolute.Few-shot chain-of-thought helped some task types but reduced performance on others, possibly because example rationales did not generalize.
- Failure analysis: Long-dependency failures reflected imprecise information extraction, factual inaccuracies, context-window constraints, and information loss from model architectures and positional encoding.Models often retrieved and combined pertinent evidence redundantly, or abstained when they recognized insufficient context.
5 CONCLUSION
LooGLE is a benchmark for assessing long-context comprehension, addressing prior dataset deficiencies with longer, newer documents and diverse dependency structures. Evaluations show that existing LLMs still struggle with intricate long-text interdependencies, including with extended context windows.
- LooGLE assesses long-context comprehension using longer passages, post-2022 documents, multi-source materials, and tasks with diverse contextual dependencies.
- Existing LLMs exhibit substantial limitations in understanding and reasoning about intricate interdependencies in lengthy texts.
- Commercial and open-source models both face challenges on long dependency tasks, despite a notable performance disparity between them.
- LooGLE provides a reference for evaluating long-context comprehension and identifying potential directions for improving LLM performance.
A MORE DETAILS OF OUR DATASET
The dataset details characterize long dependency QA through input-length and dependency-span distributions, alongside n-gram analyses of generated question-answer pairs.
- Input document lengths for long dependency QA are represented by distributions in Figure 7.
- Evidence dependency spans for long dependency QA are represented by distributions in Figure 8.
- Generated QA pairs are analyzed with an n-gram sunburst graph in Figure 9.
B TASK DEFINITION
The Cloze task is defined through a formulation process illustrated in Figure 10, which provides the task’s construction reference.
- The Cloze task formulation process is presented in Figure 10.
C TIMELINE REORDER EVALUATION METRICS
Timeline reorder outputs are evaluated by comparing numeric sequences with four deviation metrics, while associated prompts specify generation and semantic or quality judgments.
- Four metrics measure similarity between equal-length numeric sequences for timeline reorder tasks.
- LSD denotes location square deviation and compares the positions of corresponding numbers in two sequences.
- LMD denotes location mean deviation for timeline reorder evaluation.
- SD denotes swap deviation, while SDD denotes swap distance deviation.
- Swap-based metrics model conversion between sequences as a series of element swaps with weighted actions.
- Timeline reorder prompts require Roman-number answers, and evaluation separately handles semantic equivalence and summary quality.
E EXAMPLES FOR LONG CONTEXT UNDERSTANDING TASKS
The examples span short factual questions, cloze completion, long-document summarization, architectural project retrieval, timeline ordering, numerical reasoning, and questions whose answers require evidence across a document.
- Short factual questions ask for directly stated information, such as Picardo’s collaborator or the nickname “Age of Enlightenment.”
- Cloze examples require reconstructing masked entities and events from long narrative inputs, including a heist involving multiple characters.
- Long-document summarization uses extended research papers and asks models to condense their content into a summary.
- Long-dependency questions retrieve distributed evidence to identify Picardo’s architectural projects and order restoration and castle-conversion events chronologically.
- Other examples require numerical computation or broad historical reasoning, such as comparing Barcelona’s population figures or identifying a turning point in territorial expansion.
F.1 GPT4-32K
The GPT4-32K examples cover factual extraction, explanatory questions, numerical comparison, multi-part life-event reasoning, cloze completion, and technical paper summarization.
- Factual extraction examples ask for charges against individuals or reasons behind government relocation programs.
- A numerical question requires subtracting World Bank aid to Turkey from estimated damage suffered by Syria, producing 3.32 Billion.
- A multi-part question about Guy’s efforts to improve his life is answered through therapy, introspection, relationships, renovation, and personal change.
- The technical summarization example concerns Prasad’s conjecture, regular supercuspidal representations, and a local approach involving character analysis.
- The cloze example draws on a long script segment set during the 2022 Wildfire in Tulare County, involving Lu and another character.
F.2 GPT4-8K
The GPT4-8K examples include historical fact retrieval, economic explanation, timeline ordering, consistency judgment, technical summarization, and script-based cloze questions.
- Factual retrieval asks when large cattle ranches appeared, with the answer given as early as 1734.
- An economic-history question attributes Rio Grande do Sul’s late-nineteenth-century growth to trade, immigration, industry, and a wealthy bourgeois class.
- Timeline ordering requires arranging reported events involving a lost thumb, a sexual-assault report, and an internal investigation.
- A consistency question asks whether Maurice Duplessis treated the news publishing industry consistently across two terms, with the groundtruth answer being inconsistent.
- The technical example summarizes work on Prasad’s conjecture and regular supercuspidal representations, while another example uses a long Wildfire 2022 script segment for cloze completion.
F.3 GPT3.5-TURBO-16K
The GPT3.5-TURBO-16K examples test current-event fact retrieval, numerical extraction, set selection, comparative judgment, technical summarization, and script cloze completion.
- Fact-retrieval questions ask what Buckingham Palace unveiled and the percentage decline in UK house prices.
- A selection question asks which listed churches were not built during Phase 1 of construction.
- A comparative judgment asks whether Reader’s Digest and Alexander Cockburn agreed in their portrayals of the Mujahideen, with the answer being no.
- The technical summarization example covers Prasad’s conjecture, local Langlands correspondence, representation parametrization, character computations, and supporting examples.
- The cloze example uses a long Wildfire 2022 script segment involving a mute girl, a Trans Latin woman, and school experiences.
F.4 LLAMAINDEX
The LLAMAINDEX evaluation includes short factual and cloze-style questions alongside more complex script and research-paper understanding tasks. Its outputs show some correct answers but also substantial errors on multi-step or long-context questions.
- LLAMAINDEX answered factual questions about Christian rule in Tripoli and the cause of the Paris waste workers’ strike correctly.
- The model failed a script question by identifying Buck’s interaction with Tye as the turning point instead of the groundtruth answer, Sasha.
- The model undercounted Mia’s visits to the Decrepit House, answering once instead of the groundtruth three times.
- For a research-paper summarization task, the output begins with a broad review of Prasad’s conjecture and its relation to representation theory and Langlands parameters.
F.5 CHATGLM2-6B-32K
The CHATGLM2-6B-32K evaluation contains factual, comparison, cloze, multi-hop, and generation-oriented questions across historical, sports, media, mathematical, and everyday domains. Its outputs include several correct factual responses, but also reveal answer mismatches, repetition, and unsupported or incorrect interpretations.
- The model correctly described Arsenal’s improved 2022–23 Premier League position relative to 2021–22, including 84 points and a top-four finish.
- The model failed the media-frequency question, returning a list of years instead of identifying The Herald and The Auckland Star.
- Other responses include fragmented or mismatched outputs, such as a partial JavaScript answer, a masked cloze prompt, and an answer missing the requested person’s name.
- Several outputs repeat the same answer many times, as in the Rock Machine question about establishing its Australian Nomads chapter in the early 2000s.
- The model produced an incorrect option for Peter Herring’s opinion question and an overconfident answer where the text supported insufficient information.