PaperlayerView exact revision on arXiv

Source-linked AI summary

AcademicEval: Live Long-Context LLM Benchmark

Haozhen Zhang, Tao Feng, Pengrui Han, Jiaxuan You

arXiv:2510.17725v1cs.CLcs.AIcs.LG

TL;DR

Long-context benchmarks have rigid context lengths, labor-intensive annotation, and label-leakage concerns. AcademicEval uses arXiv papers, co-author demonstrations, and periodic updates to evaluate four hierarchical writing tasks. Results show varied model performance across automatic and judged quality assessments, with performance often declining as inputs grow.

  • Problem

    Existing long-context benchmarks are limited by rigid context lengths, labor-intensive annotation, and possible contamination from LLM training data.

    Source: §1 · p. 1

  • Method

    AcademicEval evaluates Title, Abstract, Introduction, and Related Work generation from arXiv papers, using co-author demonstrations and periodic benchmark updates.

    Source: §1 · p. 2

  • Results

    Automatic metrics often favor RALM, but judged quality finds retrieval is not always preferred for Title or Abstract and is beneficial for Related Work; performance commonly degrades with longer inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AcademicEval's evaluations expose complementary aspects of long-context modeling through overlap-oriented metrics and higher-level judged quality.

    Source: §1 · p. 2

    The causes of performance plateaus at longer contexts are not settled; the discussion attributes them partly to limited in-context use of ultra-long inputs and notes in-weights learning as another possibility.

    Source: §4.5 · p. 13

Abstract

from arXiv · show

Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval

1 Introduction

Existing long-context benchmarks face rigid context lengths, costly annotation, and possible training-data leakage. AcademicEval addresses these gaps with arXiv-based writing tasks, co-author demonstrations, and periodic updates; evaluations show varied performance across tasks and metrics.

Motivation

  • Existing long-context benchmarks have rigid context lengths, often require costly human labeling, and may be contaminated by training data.The introduction argues that holding out future data for validation and testing is an effective approach for open benchmarks.Source: §1 · pp. 1–2

Experimental findings

  • Automatic metrics often favor RALM, while judge evaluations find retrieval is not always preferred for Title or Abstract but is beneficial for Related Work.Across evaluations, performance commonly degrades with longer inputs, and co-author demonstrations yield modest gains for some model–task pairs.Source: §1 · p. 2

Contributions

  • AcademicEval evaluates Title, Abstract, Introduction, and Related Work generation at hierarchical abstraction levels, using arXiv papers as labels without manual annotation.Its co-author graph supplies expert-curated few-shot demonstrations, while periodic updates use recent arXiv papers to support live evaluation and avoid label leakage.Source: §1 · pp. 2–3

Experimental findings

  • The experiments aim to expose benchmark challenges and provide insights for improving long-context modeling.The introduction describes these as comprehensive experiments on AcademicEval.Source: §1 · p. 3

2 Related Work

Related work covers long-context modeling and benchmarks, leakage mitigation, and summarization tasks. AcademicEval is distinguished by evaluating hierarchical academic-writing tasks with real papers and a graph-based, live benchmark.

Long-context Modeling and LLM Benchmarks

  • Long-context systems include retrieval-augmented models, which retrieve short text chunks, and long-context LLMs, which expand context windows for tasks such as QA, summarization, and reasoning.RALM is commonly evaluated on retrieval benchmarks, while long-context LLMs are assessed on varied long-context tasks.Source: §2 · p. 3

Related frameworks

  • ResearchTown simulates research-community collaboration, whereas AcademicEval evaluates hierarchical writing on real academic papers using a co-author graph for demonstrations and updates.WildLong synthesizes long-context instruction data, making it complementary to AcademicEval's evaluation focus.Source: §2 · p. 3

Label Leakage in LLM Benchmarks

  • Existing leakage approaches detect contamination or use dynamic data, but some rely on labor-intensive human involvement and others on chronological splits.The related work notes little prior work directly mitigating leakage, despite methods that measure or detect it.Source: §2 · pp. 3–4

Long-context Summarization Benchmarks

  • Long-context summarization benchmarks use query-based or holistic tasks, but typically require human-assisted labels, face leakage concerns, and assess only one abstraction level.AcademicEval instead spans multiple abstraction levels in its academic-writing tasks.Source: §2 · pp. 3–4

3 AcademicEval Benchmark

AcademicEval builds a long-context academic-writing benchmark from arXiv papers and a co-author graph. It combines paper-section extraction with writing tasks spanning different abstraction levels.

3 AcademicEval Benchmark

  • AcademicEval's benchmark combines four academic-writing tasks, flexible few-shot demonstrations, and periodic data updates for live evaluation.The section introduces data collection and preprocessing before describing the tasks and update procedure.Source: §3 · p. 4

3.1 Data Curation

  • The co-author graph connects authors who have co-authored papers and stores each author's first-author papers as node features, enabling structural use of arXiv papers and later updates.The graph is constructed through breadth-first search using the arXiv API.Source: §3.1 · p. 4
  • Preprocessing removes authors without independent first-author papers, retains the maximum connected component, extracts paper sections, and adds cited-paper titles and abstracts as a Related Work corpus.Metadata and PDFs are collected through the arXiv API; PyMuPDF is used to detect section headings and extract content.Source: §3.1 · p. 4

3.2 Benchmarking LLMs over Long-context Generation Tasks with Hierarchical Abstraction

  • The benchmark uses four long-context academic-writing tasks to assess generation at different abstraction levels, with each paper sample split into sections and optionally paired with co-author demonstrations.The figure depicts a chronological split into training, validation, and test samples.Source: §3.2 · p. 4, Figure 1 · p. 5

• Title Writing.

AcademicEval varies writing-task inputs and context lengths using paper sections and few-shot demonstrations. Its live update procedure adds recent arXiv papers through co-author graph updates and pruning.

• Title Writing.

  • The four tasks predict a title, abstract, introduction, or related work from the remaining paper sections and specified inputs.Title uses the main body and abstract; Abstract excludes the conclusion; Introduction excludes the introduction; Related Work excludes that section and adds a citation corpus.Source: • Title Writing · p. 5
  • Each task's corresponding paper section serves as ground truth, and settings vary input context length through random or co-author few-shot demonstrations.The initial settings include Title-10K, Abs-9K, Intro-8K, and Related-34K; demonstrations can be increased to scale context length.Source: • Title Writing · pp. 5–6
  • A higher compression rate indicates greater text abstraction, while longer settings can make holistic information use harder and may provide more references for stronger outputs.Settings combine single-paper inputs with random-paper and co-author demonstrations; Title-30K and Title-31K-G use two demonstrations of those respective types, and Title-50K-M combines both.Source: • Title Writing · pp. 6–7

3.3 Live Evaluation with Periodic Data Updates on the Co-author Graph

  • Chronological splitting places the latest collected papers in the test set, and periodic updates refresh the benchmark with newly published arXiv papers.The live update cycle may be monthly or quarterly.Source: • Title Writing · p. 7, §3.3 · p. 7
  • The graph update adds new first-author papers to author nodes, prioritizes new co-authors by their paper activity for BFS expansion, and prunes outdated papers and inactive authors.The pruning and update process is intended to maintain graph connectivity and efficiency while collecting the latest papers.Source: §3.3 · p. 7

4 Experiments

The experiments compare standard and long-context LLMs with retrieval-augmented models across four academic writing tasks, using automatic metrics and LLM-judged preferences. Results vary by task and evaluation measure; longer inputs often reduce performance, few-shot effects are mixed, and the discussion attributes the observed plateau chiefly to imperfect long-context use.

4.1 Baselines

  • Evaluation baselines include standard LLMs with 8K context, long-context LLMs with 32K context, and sparse and dense retrievers; metrics are BERTScore and ROUGE-L, where higher scores indicate better matches.For each LLM, its tokenizer configuration is used to ensure accurate input truncation.Source: §4.1 · p. 8, §4.2 · p. 8, §4.3.1 · p. 9

4.3.2 Result Analysis

  • Performance varies across abstraction levels, often declines as context grows, and few-shot demonstrations have mixed effects, though co-author examples are generally more positive than random ones.Title tends to score higher than Abstract, which the authors suggest may reflect titles’ brevity and lower detail requirements.Source: §4.3.2 · pp. 9–11
  • RALM with LLaMA frequently leads on BERTScore and ROUGE-L, while standard models remain competitive and Hermes is the strongest long-context baseline overall.Judge preferences differ from overlap metrics: RALM is not consistently preferred across tasks, although it leads on longer Related Work settings.Source: Table 3 · p. 9, §4.3.2 · p. 10, Table 4 · p. 10

4.4.1 Evaluation Setup

  • The LLM judge reports preference win rates across novelty, feasibility, consistency, factuality, and academic style, targeting qualities beyond semantic overlap.Mixtral-8x22B-Instruct-v0.1 performs the judging.Source: §4.4.1 · p. 11

4.4.2 Result Analysis

  • Judge preferences differ partly from overlap metrics: Qwen leads Abstract and Introduction, while RALM LLaMA† leads longer Related Work settings.The judge favors Hermes on Title-10K and Mixtral on Title-31K-G; correlated contexts yield modest, model-dependent gains in some settings.Source: §4.4.2 · p. 12

4.5 Discussion

  • On Abs-9K, dense retrievers consistently outperform sparse retrievers, with Contriever achieving the best results.The discussion links this pattern to summary generation’s emphasis on semantic similarity, which dense embeddings measure well.Source: §4.5 · pp. 12–13, Figure 3 · p. 12
  • The authors attribute the longer-context performance plateau primarily to imperfect long-context utilization rather than in-weights learning.In the Title-only Abs-9K ablation, removing most context lowers LLaMA and Hermes BERTScore and ROUGE-L by 5–7 points.Source: §4.5 · p. 13, Table 6 · p. 13

5 Conclusion

AcademicEval is a live benchmark for long-context generation with hierarchical abstraction levels, and its experiments find the benchmark challenging while offering insights for future modeling and benchmark design.

5 Conclusion

  • AcademicEval uses arXiv papers for long-context generation tasks with hierarchical abstraction levels and integrates co-author-graph few-shot demonstrations for flexible context length.The benchmark requires no manual annotation and includes live evaluation designed to address label leakage.Source: §5 · p. 13
  • The experiments evaluate several LLM baselines and show that AcademicEval is a challenging long-context benchmark.The live evaluation is designed to address label leakage and support fairer evaluation.Source: §5 · pp. 13–14
  • The analysis offers insights intended to strengthen LLM long-context modeling and inform future benchmarks with more flexible and holistic evaluation.Source: §5 · p. 14

A API cost

Experiments use the Together.ai LLM API, with API costs estimated at around $300, mainly for evaluating AcademicEval’s test set.

A API cost

B Limitation and Future Improvement

AcademicEval is constrained by limited task diversity, independent section-level evaluation, and possible popularity bias in its arXiv-derived paper set. The authors propose broader data sources, cross-validation, and adjusted sampling as future improvements.

Limitations

Future Improvement

C Social Impact

The section describes anticipated benefits of automating academic writing and provides prompt designs for writing tasks and model judging.

C Social Impact

  • LLMs are proposed as aids for drafting, literature reviews, article summaries, and bibliographies, potentially freeing researchers for higher-level work.Source: C Social Impact · p. 18

D.1 LLM Prompts for Title Writing

  • Title prompts ask models to distill the target content, with variants using reference examples or multiple target contents.Retrieval queries separately request titles from an abstract.Source: §D.1 · pp. 19–20

D.2 LLM Prompts for Abstract Writing

  • Abstract prompts request roughly 200-word summaries of a target paper’s theme, findings or arguments, and conclusions.Variants provide demonstrations or combine multiple target contents.Source: §D.2 · pp. 20–21

D.3 LLM Prompts for Introduction Writing

  • Introduction prompts request 1,000–1,500 words covering background, prior research, the problem, the approach, results, and paper structure.The prompts include standard, demonstrated, and multi-target variants.Source: §D.3 · pp. 21–23

D.4 LLM Prompts for Related Work Writing

  • Related-work prompts summarize target citations or content, with variants incorporating demonstrations or multiple target papers.The prompt asks for concise summaries and permits categorizing relevant works and discussing their advantages and disadvantages.Source: §D.4 · pp. 23–25

D.5 LLM Prompts for LLM-as-a-Judge

  • The judge prompt compares two generated texts on novelty, feasibility, consistency, factuality, and academic style, then selects an overall winner.Source: §D.5 · p. 26

E Examples

This section presents generation examples for title, abstract, and introduction tasks.

  • The examples cover Title Writing, Abstract Writing, and Introduction Writing.Source: E Examples · p. 27

E.1 Title Writing

The title-writing examples show generated titles for papers on confidence alignment and multilingual information retrieval, alongside reference titles.

  • One generated CONQORD title describes confidence alignment through reinforcement learning, while another frames it as confidence calibration.Source: §E.1 · p. 27
  • For the confidence-alignment paper, generated titles range from “Aligning Confidence with Response Quality in Large Language Models using Reinforcement Learning with Dual-Component Reward Function: CONQORD Approach” to shorter CONQORD-focused formulations.The examples include zero-shot and two-shot outputs, including demonstrations selected from co-author papers or at random.Source: §E.1 · p. 27
  • For “Distillation for Multilingual Information Retrieval,” shown outputs include titles about translate-distillation and cross-language or multilingual retrieval.The examples include outputs from multiple models and both zero-shot and two-shot settings.Source: §E.1 · p. 28

E.2 Abstract Writing

The section presents two-shot generated abstracts for papers on mobile-device agents and neural ranking, alongside their ground-truth abstracts. The examples show how generated summaries describe each paper’s tasks, methods, and reported findings.

E.2 Abstract Writing

  • The neural-ranking ground truth reports positional-bias injection attacks and greater resilience in lexical models, while generated versions describe position effects and mitigation strategies.The generated abstracts mention a promotional-content classifier or a model-agnostic mitigation strategy; these are claims in the examples, not verified by the supplied ground truth.Source: §E.2 · p. 29
  • The B-MoCA ground truth describes 60 daily tasks and device randomization, while the generated abstracts emphasize evaluation across configurations and difficulty with sequential tasks.One generated version says LLM and MLLM agents struggle with complex tasks; another says they are robust across configurations but fall short on sequential decision-making.Source: §E.2 · p. 30
  • The generated B-MoCA abstracts frame the benchmark as testing mobile-device agents across randomized configurations, including layouts, languages, wallpapers, and device types.The ground truth likewise describes randomization of interface layouts and language settings to assess generalization.Source: §E.2 · p. 30

E.3 Introduction Writing

The examples present two-shot generated introductions on retrieval poisoning and multi-domain, multi-task recommendation. They describe the first as an attack on retrieval-augmented generation and the second through expert modules and fusion.

E.3 Introduction Writing

  • The generated retrieval-poisoning introduction describes invisible document injections intended to steer responses in retrieval-augmented generation applications.It presents framework analysis and attack-sequence generation as a two-step approach.Source: §E.3 · p. 31
  • The generated M3oE introduction describes shared, domain-specific, and task-specific expert modules combined through two-level fusion.It also describes AutoML as dynamically optimizing the model structure.Source: §E.3 · p. 32
  • The M3oE example frames its recommendation task as jointly addressing multiple domains and tasks while balancing user-preference representations across its expert modules.The example claims benchmark performance surpasses state-of-the-art baselines, but no supporting ground-truth passage is supplied here.Source: §E.3 · p. 32

Related papers

Loading 2510.17725v1…