Source-linked AI summary
SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
Weihang Su, Anzhe Xie, Qingyao Ai, Jianming Long, Xuanyi Chen, Jiaxin Mao, Ziyi Ye, Yiqun Liu
TL;DR
Rapid growth in academic literature makes manual survey creation increasingly impractical, while automated survey research lacks standardized evaluation. SurGE provides a benchmark and multidimensional framework for assessment; its experiments find agentic pipelines outperform standard RAG baselines but still struggle to use collected evidence.
Problem
Manual survey creation is increasingly impractical as literature grows, and survey-generation research lacks standardized evaluation protocols.
Source: §1 · p. 1Method
SurGE combines expert-written ground-truth surveys, a large academic corpus, and automated evaluation across comprehensiveness, citation accuracy, structure, and content.
Source: §1 · p. 2Results
Specialized agentic pipelines outperform standard RAG baselines but struggle to use collected evidence despite greater linguistic fluency.
Source: §1 · p. 2Takeaways & Limitations
SurGE provides a benchmark and evaluation framework for assessing survey generation, with the stated aim of supporting more trustworthy and effective systems.
Source: §8 · p. 7Ground-truth references are a pragmatic proxy for expert consensus, not a measure of absolute completeness.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
The rapid growth of academic literature makes the manual creation of scientific surveys increasingly infeasible. While large language models show promise for automating this process, progress in this area is hindered by the absence of standardized benchmarks and evaluation protocols. To bridge this critical gap, we introduce SurGE (Survey Generation Evaluation), a new benchmark for scientific survey generation in computer science. SurGE consists of (1) a collection of test instances, each including a topic description, an expert-written survey, and its full set of cited references, and (2) a large-scale academic corpus of over one million papers. In addition, we propose an automated evaluation framework that measures the quality of generated surveys across four dimensions: comprehensiveness, citation accuracy, structural organization, and content quality. Our evaluation of diverse LLM-based methods demonstrates a significant performance gap, revealing that even advanced agentic frameworks struggle with the complexities of survey generation and highlighting the need for future research in this area. We have open-sourced all the code, data, and models at: https://github.com/oneal2000/SurGE
1 Introduction
Rapid literature growth makes manual comprehensive surveying impractical, while existing automated-survey evaluations are unscalable or pipeline-dependent. SurGE pairs a method-agnostic benchmark with multidimensional quality assessment and finds that agentic systems still struggle to use gathered evidence.
1 Introduction
- Existing survey-generation evaluations are either unscalable or tied to specific pipelines, hindering fair cross-system comparison.The paper cites human evaluation as a scalability constraint and pipeline-specific frameworks as a source of self-validation bias.Source: §1 · p. 1
- SurGE formalizes survey generation as retrieving relevant papers from a corpus and synthesizing them into a structured survey.Its benchmark combines expert-validated ground-truth surveys with an academic corpus of over one million papers.Source: §1 · p. 2
- SurGE evaluates survey quality across comprehensiveness, citation accuracy, structure, and content using objective indicators and LLM-as-a-Judge metrics.The authors report high correlation between their LLM-based metrics and fine-grained expert judgments.Source: §1 · p. 2
- Specialized agentic pipelines outperform standard RAG baselines, yet struggle to use collected evidence despite greater linguistic fluency.The evaluation separates information acquisition from content synthesis to analyze performance bottlenecks.Source: §1 · p. 2
2 Task Definition
Survey generation is defined as producing a structured, comprehensive article from a topic description and an academic corpus through retrieval followed by synthesis.
2 Task Definition
- Given a topic description and academic corpus, the task is to generate a structured, comprehensive survey article.The paper denotes the topic as t, corpus as D, and generated survey as S.Source: §2 · p. 2
- Information collection retrieves a set of papers relevant to the topic from the corpus.The retrieved set is denoted R_t and is a subset of D.Source: §2 · p. 2
- Survey generation composes the article from the topic and retrieved papers, with proper citations and a reference list.Source: §2 · p. 2
3 Dataset Construction
SurGE builds its benchmark by collecting and screening reference surveys, assembling an arXiv-based retrieval corpus, and documenting dataset properties and licensing.
3 Dataset Construction
- Dataset construction proceeds from reference-survey collection and expert annotation to corpus building, statistics, and licensing considerations.Source: §3 · p. 2
3.1 Ground-Truth Survey Collection
- Ground-truth survey candidates came from 2020–2024 arXiv surveys or systematic reviews with at least 20 citations.Candidates also had to explicitly identify themselves as a survey or review.Source: §3.1 · p. 2
3.2 Expert Annotation
- The benchmark’s 205 ground-truth surveys passed two-annotator usability screening, with Cohen’s Kappa of 0.792 across 250 instances.A paper was retained only when both annotators labeled it usable.Source: §3.2 · p. 2
3.3 Academic Corpus Construction
- The retrieval corpus combines metadata for cited references, topic-related arXiv results, and random computer science papers, totaling 1,086,992 records.About 30% of cited references were unavailable, primarily because they appeared in closed-access or otherwise restricted venues.Source: §3.3 · pp. 2–3
3.4 Statistics and Analysis
- Surveys average 3.07 levels of heading depth, 42.7 heading nodes, and 65.8 cited papers.The statistics characterize both hierarchical structure and reference density; Table 1 presents benchmark statistics.Source: §3.4 · p. 3, Table 1 · p. 3
3.5 Ethical Considerations and Licensing
- The corpus contains arXiv descriptive metadata rather than PDFs or source files; the dataset and codebase use the MIT license.The authors state that metadata is designated CC0 under arXiv terms.Source: §3.5 · p. 3
4 Evaluation Framework
SurGE evaluates generated surveys against expert-written ground truth across comprehensiveness, citation accuracy, structural quality, and content quality. Its framework pairs quantitative indicators with LLM-as-a-Judge metrics and is assessed for alignment with human judgments.
4.1 Comprehensiveness
- Recall measures generated-survey reference coverage against the expert survey, treated as a proxy for expert consensus rather than absolute completeness.The ground-truth reference set is not assumed complete; highly cited, peer-reviewed surveys serve as the best available proxy for a topic’s core literature.Source: §4.1 · p. 3
4.2 Citation Accuracy
- Citation Accuracy assesses whether citations are relevant to the survey topic, appropriate to their section, and supportive of their sentence-level claims.NLI scores use entailment, neutral, and contradiction probabilities; citations missing from the corpus score zero, while ground-truth bibliography citations receive a document-level score of one.Source: §4.2 · pp. 3–4
4.3 Structural Quality
- Structure Quality Score compares generated and ground-truth hierarchical headings for structural organization, meaning, and wording using an LLM judge.The heading hierarchy is preserved in the inputs rather than flattened into lists.Source: §4.3 · p. 4
- Soft-Heading Recall measures heading alignment through semantic overlap, allowing paraphrased headings to count as matches.Its soft cardinality discounts redundant headings, and semantically similar headings reduce the union’s soft cardinality to reflect overlap.Source: §4.3 · p. 4
4.4 Content Quality
- Content Quality Score uses GPT-4o to assess five content criteria per section, averaging section scores because of context-length limits.ROUGE and BLEU are reported for reference but are not part of SurGE’s metric suite.Source: §4.4 · pp. 4–5
4.5 Meta-Evaluation: Human Alignment
- 0.707 Spearman and 0.610 Kendall correlations in w/o GT; with GT, Kendall’s τ is 0.805, showing alignment with human rankings.Doctoral researchers blind-ranked four surveys for structure and content, with and without the ground truth included.Source: §4.5 · p. 5, Table 2 · p. 5
5 Experimental Setup
The experiments use a shared retrieval-then-generation setup, with one trained Paper Retriever supplying candidate papers to several survey-generation baselines. Retriever training, baseline configurations, and implementation choices are specified for comparison.
5 Experimental Setup
- All baselines retrieve candidate papers and then organize and summarize them into a structured survey.The shared retriever supplies the top 100 candidate papers for each topic.Source: §5 · p. 5, §5.2 · p. 5
5.1 Paper Retriever
- The Paper Retriever uses a shared-parameter RoBERTa dual encoder to score topic–abstract pairs by dot product and train against relevant papers and sampled negatives.At inference, it returns the highest-scoring papers as evidence for generation.Source: §5.1 · p. 5
5.2 Baselines
- The four generation baselines are RAG, AutoSurvey, StepSurvey, and SurveyForge, each using the shared retriever and top-100 candidate pool.Their generation strategies include batch-summary fusion, outline expansion, stepwise drafting, and hierarchical memory-driven writing.Source: §5.2 · pp. 5–6
5.3 Implementation Details
- The retriever is trained for 10 epochs with AdamW at 5 × 10^-6; specialized agents share Qwen2.5-14B-Instruct, while RAG also uses GPT-4o and Gemini-2.5-Pro-Thinking.Experiments ran on a server with eight NVIDIA A100 GPUs.Source: §5.3 · p. 6
6 Experimental Results
Retrieval can recover much of the benchmark’s reference set, but end-to-end results show distinct strengths across generation systems. The findings also expose gaps between fluency, evidence use, and survey quality.
6.1 Evaluation of the Retrieval Stage
- At k = 1000, the Paper Retriever recalls 68.05% of ground-truth references; end-to-end baseline performance remains below 10%.The authors interpret this gap as evidence that information selection and synthesis, rather than corpus coverage alone, constrain generation.Source: §6.1 · p. 6
6.2 Evaluation of End-to-End Generation
- SurveyForge excels in Recall and Doc-Acc, StepSurvey achieves the highest CQS, AutoSurvey leads SQS, and specialized agents exceed RAG on SHR.Table 4 compares the systems across comprehensiveness, citation accuracy, structure, and content quality.Source: §6.2 · p. 6, Table 4 · p. 7
- RAG-Qwen scores competitively on ROUGE-L and BLEU but has the lowest Citation Accuracy and Recall, showing that fluent output can remain poorly grounded.The authors recommend fact-centric metrics such as Doc-Acc and CQS over n-gram metrics alone.Source: §6.2 · p. 6
- All specialized agents achieve higher SHR than standard RAG baselines, while AutoSurvey records the best overall SQS.The authors associate these structural results with explicit planning and structured drafting.Source: §6.2 · p. 6
- SurveyForge with Qwen-14B surpasses RAG-Gemini in Citation Accuracy, while RAG-Gemini significantly outperforms RAG-Qwen.The authors suggest structured agentic substeps can compensate for limited model capacity.Source: §6.2 · p. 6
7 Related Work
Survey generation developed from multi-document summarization toward full-length manuscript production using LLM-based agentic pipelines. It remains distinct from standard RAG and long-context evaluation, motivating task-specific assessment.
7.1 Survey Generation
- Before LLMs, survey generation was framed as multi-document summarization, but it also requires large-scale literature organization, cross-paper synthesis, and hierarchical discourse construction.Earlier models had difficulty with long-context modeling and global planning.Source: §7.1 · p. 7
- LLM-based pipelines decompose survey writing into stages such as collection, outlining, drafting, and refinement, but prior work focused more on pipeline construction than standardized evaluation.AutoSurvey [50] and StepSurvey [18] use hierarchical strategies; SurveyForge [52] adds a Scholar Navigation Agent and memory-driven writing.Source: §7.1 · p. 7
7.2 Retrieval-Augmented Generation
- RAG typically retrieves relevant documents and conditions generation on that evidence, with extensions including dynamic, graph, parametric, and agentic RAG.The passage situates survey generation alongside, but distinct from, this retrieve-and-generate paradigm.Source: §7.2 · p. 7
- Unlike standard RAG or long-input benchmarks, survey generation requires hierarchical planning, multi-paper synthesis, citation coverage, and claim-to-citation faithfulness.The passage identifies the lack of standardized evaluation and describes SurGE as a method-agnostic benchmark.Source: §7.2 · p. 7
8 Conclusion
SurGE combines a large academic corpus, expert-written ground-truth surveys, and automated evaluation across four dimensions. Experiments identify incomplete topic coverage and reference hallucination as challenges for current LLM-based systems.
8 Conclusion
- SurGE provides a large academic corpus, expert-written ground-truth surveys, and automated assessment of comprehensiveness, citation accuracy, structure, and content.The authors report incomplete topic coverage and reference hallucination among current LLM-based survey-generation systems.Source: §8 · p. 7