PaperlayerView exact revision on arXiv

Source-linked AI summary

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, Xipeng Qiu

arXiv:2307.11088v3cs.CL

TL;DR

Long-context language models lack standardized, diverse benchmarks and reliable open-ended evaluation metrics. L-Eval addresses both gaps with a manually validated suite and metric study, finding weak n-gram agreement with human judgment and benefits from LIE and LLM judges.

  • Problem

    Long-context modeling lacks a high-quality multitask benchmark, and the agreement of commonly used n-gram metrics with human judgment remains uncertain.

  • Method

    L-Eval builds a 20-subtask long-context evaluation suite and studies n-gram metrics, LLM judges, and length-instruction-enhanced evaluation across 16 language models.

  • Results

    N-gram metrics often fail to correlate with human judgment, while LIE substantially improves Kendall-Tau correlation for all automatic metrics.

  • Takeaways & Limitations

    L-Eval provides a standardized testbed for comparing long-context models and developing more principled evaluation metrics.

  • Takeaways & Limitations

    Evaluating long-context datasets can be costly, with testing four datasets requiring about $100.

Abstract

from arXiv · show

Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k$\sim$200k tokens). On the other hand, we investigate the effectiveness in evalution metrics for LCLMs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment, and thus we strongly advocate for length-instruction-enhanced (LIE) evaluation and employing LLM judges. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of more principled evaluation of these models.

1 INTRODUCTION

L-Eval addresses the lack of standardized long-context benchmarks and uncertainty about evaluation metrics by combining a diverse manually validated suite with metric analysis and a 16-model study.

  • Long-context benchmarks lack the breadth of established short-prompt multitask evaluations, while n-gram metrics’ correlation with human judgment remains uncertain.The gap is especially relevant for zero-shot evaluation of long-context language models and comparisons among newly released open-source models.
  • L-Eval combines 20 sub-tasks with closed-ended and open-ended groups to evaluate long-context reasoning, understanding, and long-document information aggregation.Four tasks were annotated from scratch, four were re-annotated from public datasets, and twelve were manually cleaned from existing long-sequence datasets.
  • L-Eval prioritizes diversity and quality through varied question styles, domains, and input lengths, with manual validation after data collection.The dataset design emphasizes correctness and diversity rather than quantity.
  • Traditional lexical metrics often fail to correlate with human evaluation, whereas LLM judges perform better on open-ended tasks and LIE improves Kendall-Tau correlation for all automatic metrics.LIE incorporates the desired reference-answer length into the evaluation instruction to reduce misleading conclusions from generation-length effects.
  • A comprehensive study of 16 LLMs finds persistent commercial–open-source gaps, weak open-ended performance after context extension, and trade-offs between retrieval and reasoning from scaled positional embeddings.The study also reports that full-context models outperform traditional retrieval-based systems in experiments using GPT-3.5-Turbo.

2 RELATED WORK

Related work spans efficient long-sequence modeling and existing benchmarks, but L-Eval targets manually curated samples, standardized metric validation, and broader closed-ended coverage.

  • Long-context modeling research addresses computational bottlenecks through efficient attention mechanisms and input chunking.
  • Existing long-sequence benchmarks and language-model evaluations use classification, efficiency, accuracy, or perplexity, but perplexity may not reflect actual performance.The cited benchmark landscape includes Long Range Arena, CAB, ZeroScrolls, and LongBench.
  • L-Eval manually filters testing samples, investigates lexical and LLM metric correlations with human judgment, and includes more closed-ended tasks to reduce open-ended fairness concerns.

3 TOWARDS HIGH-QUALITY AND DIVERSE LONG CONTEXT DATASETS

L-Eval constructs a diverse long-context dataset through annotation, re-annotation, and manual correction, covering varied domains, question styles, and retrieval challenges.

  • Data construction: L-Eval’s dataset-construction pipeline combines annotation from scratch, re-annotation of public datasets, and manual filtering of existing long-sequence data.
  • Data annotation from scratch: Four from-scratch datasets draw on courses, science fiction, source code, and earning-call transcripts to test distinct long-context abilities.The resources support tasks including multiple-choice reasoning, contextual knowledge following, code understanding, and financial question answering.
  • Data annotation from scratch: Coursera uses multiple-correct-option questions from big-data and machine-learning course subtitles, making it more challenging than single-option questions.
  • Data annotation from scratch: CodeU requires inferring the output of lengthy Python programs after locating function calls and determining which functions are invoked.Function names are replaced to discourage answers based on parametric knowledge.
  • Data filtering and correction: TopicRet includes first-topic retrieval, but retrieving second and third topics is substantially harder and better distinguishes model abilities.The figure reports test accuracy for models retrieving the first topic versus the second or third topic.
  • Data filtering and correction: Existing datasets receive manual filtering and correction because prior annotation quality varies and includes unanswerable questions unrelated to context.Automatic preprocessing scripts were insufficient to correct these issues reliably.
  • Dataset statistics: L-Eval covers diverse formats including multiple choice, true or false, mathematics, code understanding, dialogue, extractive and abstractive question answering, and summarization.

4 TOWARDS STANDARDIZED LONG CONTEXT EVALUATION METRICS

L-Eval combines closed-ended exam scoring, human evaluation, LLM judging, and n-gram metrics to assess long-context models, while testing whether automated metrics align with human judgments. Its experiments show that length instructions improve correlation for some metrics and that LLM judges are generally more accurate and robust to output length than n-gram metrics.

  • Exam evaluation scores closed-ended multiple-choice tasks with exact-match accuracy, assigning each question 100 divided by the number of questions.
  • Human evaluation scores open-ended outputs from 1 to 5 on a 12-document, 85-question subset.The paper describes human evaluation as the most accurate evaluation for open-ended tasks.
  • LLM judging uses pairwise comparisons against Turbo-16k-0613, with GPT-4 and GPT-3.5 reporting win-rates in long-context settings.Because the judge cannot process the entire lengthy input, results mainly depend on the reference answer and user questions.
  • N-gram metrics such as ROUGE-L and F-1 are sensitive to ground-truth length, creating a length bias in reference-based evaluation.Claude-100k receives a 9.84 F-1 score because of undesired output length.
  • 4.1 LENGTH INSTRUCTION ENHANCED LONG CONTEXT EVALUATION: Length-instruction-enhanced evaluation exposes the reference length to the model, producing a near 50-point F-1 gap for Claude-100k when expected length is used.The instruction injects the desired word count into the original task instruction.
  • 4.1 LENGTH INSTRUCTION ENHANCED LONG CONTEXT EVALUATION: On the 85-question validation subset, length instructions raise ROUGE-L Kendall-Tau from 0.5 to 0.8, while GPT-4 evaluator correlation reaches 1.Except for GPT-4, the tested automatic metrics fail to correlate with human judgment; LLM judges are more accurate and robust to output length than n-gram metrics.

5 BENCHMARKING LLMS WITH L-EVAL

L-Eval benchmarking shows substantial differences between commercial and open-source long-context models across closed- and open-ended tasks, with performance depending on context handling and task type. Retrieval, positional scaling, and long-context tuning do not uniformly preserve instruction-following or reasoning.

  • Main results: GPT-4-32k clearly outperforms other models on L-Eval closed-ended tasks, while Claude-100k surpasses all baselines on open-ended tasks.The open-ended advantage is associated with the need to understand longer texts globally.
  • Main results: A near 20-point gap remains between the best open-source 16k models and Turbo-16k on closed-ended tasks.
  • Main results: Open-source models’ apparent proximity to GPT-Turbo under n-gram metrics is contradicted by both LLM and human evaluations.The results indicate that n-gram rankings do not accurately reflect open-ended performance gaps.
  • Main results: Retrieval-based Turbo-4k methods underperform full-context Turbo-16k because some tasks require more than simple retrieval.
  • Main results: Longer fine-tuning improves closed-ended performance but falls short on open-ended tasks, where scaled-position models can lose instructions in lengthy inputs.Llama2-16k invalid outputs increase sharply on longer samples, unlike Turbo-16k.
  • Main results: Increasing the NTK base improves topic retrieval but produces the opposite trend for lengthy-context math reasoning, which levels off for retrieval after 60,000.

6 CONCLUSION

L-Eval provides a comprehensive benchmark and evaluation framework for long-context language models, covering diverse models and long-context settings. Its experiments reveal both gains from context extension and persistent limitations relative to commercial systems.

  • Conclusion: L-Eval introduces a comprehensive suite of tasks and evaluation metrics for assessing long-context language-model capabilities.
  • Conclusion: The study tests most open-source long-context models and identifies promising gains from extending context length alongside gaps relative to commercial models.
  • Conclusion: L-Eval focuses on practical long-form documents across domains and is intended as a challenging testbed for advancing longer-context modeling.
  • Baseline models: Claude-100k targets extremely long-document understanding and has the longest context length among the evaluated models.
  • Baseline models: GPT-4-32k maintains strong reasoning over 32k-token contexts but is described as the most expensive model.
  • Baseline models: Llama2-NTK-chat extends Llama2-chat with NTK-aware positional embedding, while practical tests report random-token generation at a scale-up factor of 4 on 16k contexts.The experiments therefore use a scale-up factor of 8.
  • Baseline models: The evaluated open-source baselines include models built through positional interpolation, longer-context fine-tuning, continued pretraining, or progressive context-length increases.

A.2 HUMAN EVALUATION

The human-evaluation study compares seven models on 85 open-ended questions using ratings from three long-context researchers per sample. It finds that very long inputs increase poor open-source outputs and that lexical metrics poorly track human judgments.

  • Experimental setup: Seven commercial and open-source models are evaluated on an 85-question open-ended subset, with each sample scored by three Ph.D. annotators.
  • Evaluation scale: Human ratings use five levels ranging from totally unhelpful responses to outputs that are close to human quality or better.
  • Human evaluation results: Very long input sequences substantially increase Level-1 outputs from open-source long-context models during inference.
  • Human evaluation results: Models limited to 4k context maintain partially correct generation quality on numerous-token inputs but do not achieve high scores.
  • Human evaluation results: N-gram metrics, including F-1 and ROUGE, generally do not correlate with human evaluation results.
  • Human evaluation results: The study releases its human assessment to support development of more advanced metrics, given the cost and inefficiency of evaluating many samples with LLMs.

A.3 ANALYSIS

L-Eval analysis shows that n-gram metrics can obscure meaningful differences in long-context performance, while instruction understanding and context length strongly affect open-ended tasks. Retrieval and positional-scaling results further reveal task- and length-dependent behavior.

  • Evaluation metrics: N-gram metrics may misrepresent open-ended performance, especially when model outputs have similar scores or differing granularity.The table caption cautions that F1 results may be inaccurate in these cases, and broader comparisons show that n-gram metrics do not clearly reflect commercial–open-source gaps.
  • Open-ended tasks: Open-source LCLMs generally perform better on conventional summarization than query-based summarization and QA tasks.Query-based tasks degrade when instructions are not fully understood, particularly as input length increases.
  • Retrieval comparison: Dense retrieval outperforms BM25 by more than two points on open-ended tasks.BM25 often matches dense retrieval on closed-ended tasks, whereas semantic matching gives the dense retriever an advantage for open-ended tasks.
  • Positional scaling: Dynamic NTK scaling does not hold consistently in practical tasks, and its variants are strongly affected by the dataset’s maximum input length.NTK+bias performs best between 4k and 8k tokens, while NTK-weighted is more robust at 16k.
  • Overall results: GPT-4-32k is stronger on closed-ended tasks, whereas Claude’s 100k context helps it surpass GPT-4-32k and Turbo-16k on open-ended tasks.The comparison attributes this pattern partly to closed-ended inputs usually being shorter than 32k tokens and open-ended tasks generally containing more tokens.
  • Overall results: Topic-retrieval evaluation includes only long-context LLMs because truncating inputs for short-context models would be unfair.This design keeps the comparison aligned with the task’s full-context requirement.

B DATA COLLECTION AND ANNOTATION FOR L-EVAL

L-Eval combines heterogeneous sources and task formats to evaluate long-context reasoning, instruction following, contextual fidelity, and document-level synthesis. The collection process includes manual correction, annotation, and validation across multiple domains.

  • Data sources: L-Eval draws on sources including Coursera, corporate earnings calls, GitHub, and prior open-source datasets.These sources represent everyday and specialized domains with different challenges for long-context models.
  • Task formats: The benchmark includes multiple-choice questions, school math with lengthy chain-of-thought examples, and synthesis questions requiring global document understanding.The QuALITY-derived subset adds questions about document-wide properties such as longest sentences, dialogue speakers, names, word counts, and sentence counts.
  • QuALITY: The QuALITY subset filters 20 long stories and 202 questions, correcting or deleting items with annotation errors.It also adds manually annotated synthesis questions beyond the original extraction-oriented questions.
  • Coursera: Coursera inputs merge course subtitles into long lectures and use multiple-correct-option questions, where missing an option yields only a quarter of the total points.Questions and ground-truth answers are labeled by the authors.
  • SFiction: The SFiction task tests whether models follow contextual knowledge rather than parametric knowledge using science-fiction true-or-false questions.Many answers contradict real-world principles, so the intended answer must follow the fictional world described in the input.

B.6 CODEU (PYTHON)

CodeU evaluates long-code understanding by asking models to infer program outputs from lengthy Python codebases. Its design reduces reliance on memorized knowledge by using source code and renamed functions.

  • Task design: CodeU asks models to infer the final output of a lengthy Python program by tracing functions defined in the codebase.The prompt instructs models to identify function definitions and determine what they do before answering.
  • Data construction: The dataset uses NumPy source code and a string-processing codebase containing more than 100 functions.The string-processing functions take inputs such as strings from which email addresses can be extracted.
  • Data construction: Function names are replaced to prevent models from answering through parametric knowledge rather than code analysis.The task therefore emphasizes locating calls and determining which functions are invoked.

B.8 LONGFQA (FINANCE)

LongFQA addresses the lack of long open-ended finance QA data by annotating questions over earnings-call transcripts from six companies. Its prompts frame the model as a financial-results conference-call participant and impose answer-length requirements.

  • Dataset construction: LongFQA contains 54 questions based on six earnings-call transcripts from six corporations.The transcripts were obtained from company Investor Relations webpages and manually annotated.
  • Dataset construction: The dataset targets long open-ended question answering in finance, a domain identified as lacking such resources.Its documents are earnings-call transcripts rather than generic financial text.
  • Prompt format: Prompts ask the model to act as a member of a Financial Results Conference Call and answer questions about company actions.One example asks about Greg Dougherty’s highlighted restructuring actions at Oclaro.
  • Prompt format: The example requires an answer with a specified number of words.The exact word count is represented as a placeholder in the prompt.

B.9 CUAD (LAW)

L-Eval constructs long-context evaluation tasks from established datasets spanning legal contracts, multi-document dialogue, Wikipedia questions, novels, scientific papers, and other document types. The CUAD task uses manually filtered legal contracts with annotated question-answer pairs to assess contract-review questions.

  • Legal-domain construction: 20 CUAD documents with annotated question-answer pairs are manually filtered for the legal-domain task.CUAD is designed to support NLP research on automating legal contract review.
  • Legal-domain construction: Legal-contract prompts ask models to highlight expiration-related clauses and identify the contract’s initial-term end date.The example ground truth states that the initial term lasts five years from the Effective Date.
  • Related long-context tasks: MultiDoc2Dial supplies dialogues grounded in 2-5 relevant documents across Finance, Travel, Entertainment, and Shopping.The task includes questions requiring answers from multiple documents and domains.
  • Related long-context tasks: Additional tasks draw on Wikipedia questions, entire books and movie scripts, NLP papers, government reports, meeting transcripts, reviews, news, patents, and television transcripts.These sources support question answering, paper assistance, summarization, and review-style tasks over varied long inputs.
Loading 2307.11088v3…