PaperlayerView exact revision on arXiv

Source-linked AI summary

SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs?

Zhaojun Sun, Xuzhou Zhu, Xuanhe Zhou, Xin Tong, Shuo Wang, Jie Fu, Guoliang Li, Zhiyuan Liu, Fan Wu

arXiv:2510.03120v2cs.CL

TL;DR

Automated academic surveys can be fluent and well structured, but a rigorous reader-aligned benchmark is needed to assess their depth and usefulness. SurveyBench pairs curated topics with multidimensional content evaluation and quiz-based answerability tests. Its evaluations find that generated surveys still fall short of human-authored surveys in technical detail, reasoning, and core-idea abstraction.

  • Problem

    Existing survey-generation methods lack a rigorous, reader-aligned benchmark that evaluates survey quality beyond surface fluency and tests whether surveys answer readers’ informational needs.

    Source: §1 · pp. 1–2

  • Method

    SurveyBench combines curated topics and human-authored references with hierarchical content metrics and content- and quiz-based evaluation.

    Source: §1 · p. 2, §3.2.1 · p. 5, §3.3.2 · p. 6

  • Results

    LLM-generated surveys show structural coherence but fall short of human-written surveys in technical detail, reasoning, and core-idea abstraction.

    Source: §1 · p. 2, §4.1 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    SurveyBench supports evaluation beyond surface fluency by assessing survey structure, content depth, richness, and quiz answerability.

    Source: §1 · p. 2, §3.2.1 · p. 5, §5 · p. 10

Abstract

from arXiv · show

Academic survey writing, which distills vast literature into a coherent and insightful narrative, remains a labor-intensive and intellectually demanding task. While recent approaches, such as general DeepResearch agents and survey-specialized methods, can generate surveys automatically (a.k.a. LLM4Survey), their outputs often fall short of human standards and there lacks a rigorous, reader-aligned benchmark for thoroughly revealing their deficiencies. To fill the gap, we propose a fine-grained, quiz-driven evaluation framework SurveyBench, featuring (1) typical survey topics source from recent 11,343 arXiv papers and corresponding 4,947 high-quality surveys; (2) a multifaceted metric hierarchy that assesses the outline quality (e.g., coverage breadth, logical coherence), content quality (e.g., synthesis granularity, clarity of insights), and non-textual richness; and (3) a dual-mode evaluation protocol that includes content-based and quiz-based answerability tests, explicitly aligned with readers' informational needs. Results show SurveyBench effectively challenges existing LLM4Survey approaches (e.g., on average 21% lower than human in content-based evaluation).

1 INTRODUCTION

SurveyBench evaluates whether automated surveys meet reader needs for clear answers and useful insights, addressing the gap between fluent generation and human-level survey quality. It combines curated topics, multidimensional scoring, and quiz-based evaluation, and finds generated surveys still fall short of human-authored ones.

1 INTRODUCTION

  • High-quality surveys must cover extensive relevant literature, organize chapters logically, categorize methods precisely, articulate insights, and offer reasoned future directions.Human survey writing can take months or even a year, making it costly and vulnerable to becoming outdated.Source: §1 · p. 1
  • Existing general research agents and survey-specific methods automate literature retrieval and synthesis, but outputs can have weak references, incomplete coverage, shallow insights, and few actionable comparisons.General metrics such as ROUGE, BERTScore, and citation density do not by themselves capture these shortcomings against high-quality human surveys.Source: §1 · p. 2
  • A reader-aligned benchmark must assess survey structure, content, and multimodal richness while testing whether surveys inform readers and offer forward-looking insights.Representative high-quality surveys are scarce and domain-specific, and LLM-as-judge evaluations can miss reader perspectives.Source: §1 · p. 2
  • LLM-generated surveys remain markedly behind human-written surveys in content richness and quiz-based assessments, especially topic-specific quizzes.They can nevertheless be fluent, well structured, and provide basic instructional value.Source: §1 · p. 2
  • SurveyBench combines curated topics, hierarchical quality metrics, and content- and quiz-based evaluation to assess surveys against readers’ informational needs.Its benchmark covers topics drawn from 11,343 recent arXiv papers and 4,947 high-quality surveys.Source: §1 · p. 2

2 AUTOMATIC SURVEY WRITING PIPELINES

Existing LLM4Survey methods generally follow a human-like workflow, retrieving and preparing literature before generating and refining an outline and survey text.

2 AUTOMATIC SURVEY WRITING PIPELINES

  • Existing LLM4Survey methods generally mimic the workflow of human authors.Their pipelines organize work around literature retrieval, outline generation, and content generation.Source: §2 · pp. 2–3
  • Methods retrieve candidate papers from databases or open-access sources, then filter and preprocess references to improve relevance, coverage, and downstream structure.SURVEYX uses semantic clustering and embedding–LLM filtering; SURVEYFORGE also considers publication date and citation impact.Source: §2 · pp. 2–3
  • Outline pipelines draft structures from retrieved references and refine them through merging, deduplication, reorganization, or best-of-N self-refinement.LLM×MAPREDUCE-V2 batches literature into partial outlines and uses entropy-driven convolution; SURVEYX uses attribute trees to guide headings.Source: §2 · p. 3
  • Content pipelines generate sections in parallel, sequentially, or through a tree process, then refine text for clarity, consistency, and citation quality.AUTOSURVEY and SURVEYFORGE use neighboring context for refinement, while SURVEYX retrieves from an attribute forest to filter and rewrite paragraphs.Source: §2 · p. 3

3 SURVEYBENCH

SurveyBench addresses the challenges of selecting diagnostic, representative survey topics and evaluating survey quality beyond surface fluency. It constructs a curated benchmark and evaluates structure, content, richness, and answerability using human-aligned prompts and quizzes.

3 SURVEYBENCH

  • SurveyBench presents a framework spanning benchmark construction, dataset statistics, and evaluation procedures.The framework is introduced as an overview of the benchmark and its evaluation process.Source: §3 · p. 3

3.1 BENCHMARK CONSTRUCTION

  • Benchmark construction must select topics that are typical, diverse, and diagnostic, while applying fine-grained evaluation criteria beyond surface fluency.The topic-selection challenge stems from the rapid growth of papers across arXiv and Google Scholar.Source: §3.1 · p. 3

3.1.1 SURVEY TOPIC PREPARATION

  • Topic preparation narrows 127 computer-science candidates to 20 vetted topics using recent-paper clusters and filters for volume, diversity, influence, overlap, survey impact, and coverage depth.The authors sample 4,947 survey papers and use embedding-based clustering and t-SNE to inspect recent arXiv literature.Source: §3.1.1 · pp. 3–4

3.1.2 EVALUATION PROMPT PREPARATION

  • Evaluation prompts align content-based judgments with human-authored standards and constrain quiz answers to retrieved evidence, with scoring for accuracy, completeness, relevance, and support.Content prompts compare generated surveys with a high-quality human survey; quiz prompts require supporting passage identifiers and permit an explicit no-content response.Source: §3.1.2 · p. 4

3.2.1 EVALUATION METRICS

  • Metrics assess outline coverage, relevance, and structure; content coverage, depth, focus, coherence, and fluency; and the proportion of non-text elements relative to chapter length.Quiz-based evaluation also uses win-rate.Source: §3.2.1 · p. 5

3.2.2 EVALUATION QUIZ SET

  • General quizzes progress from basic coverage and taxonomy to technical depth and higher-order insight, while topic-specific quizzes are generated from verified survey passages and checked for grounded, substantive answers.General quizzes test concepts, taxonomies, history, algorithmic principles, practical guidance, performance analysis, and insights; topic-specific quiz-answer pairs undergo length, completeness, and source-alignment checks.Source: §3.2.2 · pp. 5–6

3.3 SURVEY EVALUATION

SurveyBench evaluates survey writing with fairness controls, LLM-judged content assessments, and quiz-based tests of answerability. Its protocol compares generated surveys with human references and tests whether surveys support grounded answers to quizzes.

3.3.1 FAIRNESS-GUARANTEED SURVEY WRITING

  • Survey generation avoids human-survey leakage by barring OPENAI-DEEPRESEARCH from consulting existing surveys and limiting other methods to retrieved papers’ titles and abstracts.The restriction is intended to avoid bias from referencing human-written surveys.Source: §3.3.1 · p. 6

3.3.2 CONTENT-BASED SURVEY EVALUATION

  • Content evaluation uses LLM judges to assess surveys both without human references and, in a separate full-document setting, against high-quality human-written counterparts.Without references, surveys receive document-level or chapter-level scores; chapter scores are averaged across sections and topics.Source: §3.3.2 · p. 6

3.3.3 QUIZ-BASED SURVEY EVALUATION

  • Quiz evaluation retrieves outline-relevant sections and quiz-similar paragraphs, then prompts an LLM to answer from that context with supporting evidence.Answers are checked for correctness and grounding; topic-specific answers score zero when evidence is insufficient, regardless of plausibility.Source: §3.3.3 · p. 6
  • The evaluation reports content results in Table 3 and quiz-based results in Table 4, while Figure 5 depicts error distributions and token utilization.Table 3's caption specifies human-reference settings, including scores of 5 for outline and content quality and three richness metrics.Source: Table 3 · p. 7, Table 4 · p. 7, Figure 5 · p. 7

4 EXPERIMENTS

Experiments benchmark four survey-writing methods using content, richness, quiz, and fine-grained analyses. Generated surveys can approach human performance on some content measures, but the reported findings identify gaps in richness, detail, reasoning, and coverage.

4.1 OVERALL PERFORMANCE RESULTS

  • OPENAI-DR scores only 4% below humans in content focus, while LLM×MR-V2 lags by just 9% in outline relevance against human references.LLM×MR-V2 has the highest outline quality, whereas OPENAI-DEEPRESEARCH has the highest content quality but the lowest outline quality.Source: §4.1 · p. 7
  • Human-written surveys score about 5.56 times higher than OPENAI-DEEPRESEARCH on richness, while LLM×MR-V2 averages 10.95 tables versus 5.45 for humans.LLM×MR-V2's overall richness score remains moderate at 5.04, with the passage attributing this in part to its considerably longest outputs.Source: §4.1 · pp. 7–8
  • All four methods receive relatively low quiz scores; generated surveys often give surface-level explanations and perform poorly on content-specific quizzes.OPENAI-DEEPRESEARCH's high-level overview lacks the fine-grained discussion needed to answer a quiz about differences between multilingual instruction-adaptation strategies.Source: §4.1 · p. 8
  • LLM-generated surveys struggle to connect concepts through analogy, including the relationship between rendering tiles and pixels and CUDA blocks and threads.The reported finding links these omissions to a lack of deep cross-domain coverage and reasoning ability.Source: §4.1 · p. 8
  • Generated surveys seldom synthesize overarching ideas, and most quizzes probing main aspects, methods, or dimensions remain unanswered.The reported explanation is that summaries are often rewritten from sources without clear self-assessment of importance, relevance, or ordered discussion.Source: §4.1 · p. 8
  • Most generated surveys include forward-looking sections, while human authors may omit them according to individual emphasis.The methods follow a standardized structural template and almost never leave out such discussions.Source: §4.1 · p. 8

4.2 FINE-GRAINED ANALYSIS

  • OPENAI-DEEPRESEARCH shows strength in algorithmic principles and structural classification but weakness in comparative performance analysis; AUTOSURVEY has the most technology-related errors.LLM×MAPREDUCE-V2 and SURVEYFORGE have nearly identical error distributions, with shallow understanding of deeper algorithmic mechanisms.Source: §4.2 · p. 8
  • OPENAI-DEEPRESEARCH uses the fewest tokens, while LLM×MR-V2 consumes over 33.7% more; all methods score higher on old than new topics.LLM×MAPREDUCE-V2 scores 4.39 in average outline quality and 4.29 in average content quality on old topics; the case study also finds the human RL survey outperforms AUTOSURVEY in organization, depth, and coverage.Source: §4.2 · p. 9, §4.3 · p. 9, Figure 6 · p. 9

5 CONCLUSION

SurveyBench combines curated topics, human-aligned scoring, and content- and quiz-based assessment to evaluate automatic academic survey writing beyond surface fluency.

5 CONCLUSION

  • SurveyBench assesses automatic survey writing beyond surface fluency, with results showing structural coherence but shortfalls in technical detail, reasoning, and core idea abstraction.Source: §5 · p. 10

A DETAILS OF TOPICS AND HUMAN SURVEYS

Table 6 is a survey table with release-time information.

A DETAILS OF TOPICS AND HUMAN SURVEYS

  • Table 6 presents survey entries alongside their release times.Source: Table 6 · p. 11

B EXPERIMENTAL RESULTS OF “WITHOUT HUMAN AS REFERENCE”

Tables 7 and 8 report evaluation results without using a human-written survey as reference, at document and chapter levels, respectively.

B EXPERIMENTAL RESULTS OF “WITHOUT HUMAN AS REFERENCE”

  • Table 7 reports outline and document-level content quality results without a human-written survey as reference.The caption does not provide the table's values.Source: Table 7 · p. 11
  • Table 8 reports chapter-level content quality results without a human-written survey as reference.The caption does not provide the table's values.Source: Table 8 · p. 11

C OPENAI-DEEPRESEARCH PROMPTS

The appendix includes an initial requirement prompt and a further requirement prompt for OpenAI-DeepResearch.

C OPENAI-DEEPRESEARCH PROMPTS

  • Figure 7 is the initial requirement prompt.Its contents are not stated in the supplied passage.Source: Figure 7 · p. 11
  • Figure 8 is a further requirement prompt.Its contents are not stated in the supplied passage.Source: Figure 8 · p. 12

D WORKFLOW OF TOPIC-SPECIFIC QUIZ GENERATION

The workflow generates topic-specific Q–A pairs toward a target count, using segment sampling and progressively relaxed quality requirements when needed.

D WORKFLOW OF TOPIC-SPECIFIC QUIZ GENERATION

  • The algorithm takes a target count N, questions per segment k, and maximum attempts m, and returns a list of N Q–A pairs.It initializes an empty result list and sets the quality threshold to 0.7.Source: D WORKFLOW OF TOPIC-SPECIFIC QUIZ… · p. 12
  • Each attempt samples up to min(⌈q/k⌉, 10) segments, where q is the remaining pair count, and generates up to min(k, q) pairs per unprocessed segment.The algorithm lowers its quality threshold by 0.1 after 0.3m attempts or when no segments are found, with a floor of 0.3.Source: D WORKFLOW OF TOPIC-SPECIFIC QUIZ… · p. 12
  • If sampling stalls after more than 80% of segments were processed, the workflow resets the processed-segment list to allow reuse.It then falls back to additional pairs from successful segments with relaxed validation, and if fewer than 0.8N pairs exist, generates from any available segments with minimal requirements.Source: D WORKFLOW OF TOPIC-SPECIFIC QUIZ… · p. 12

E TOPICS COLLECTED FROM CS CONFERENCES

The appendix lists topics collected from calls for papers at representative computer science conferences, organized by category and topic.

E TOPICS COLLECTED FROM CS CONFERENCES

  • The catalog includes reinforcement learning and robotics, data governance, machine-learning evaluation, data-management systems, and human-centric data management topics.Examples include data quality, provenance, benchmarking, query processing, and user interfaces.Source: Table 9 · p. 13, E TOPICS COLLECTED FROM CS CONFERENCES · p. 14
  • Vision topics span recognition, scene understanding, segmentation, motion and embodied vision, and vision theory, explainability, ethics, and applications.Listed applications include autonomous driving, computer vision for robotics, medical and biological vision, and vision-language reasoning.Source: Table 9 · p. 13, E TOPICS COLLECTED FROM CS CONFERENCES · p. 14
  • The listed categories also cover big-data and machine-learning systems, ML infrastructure, data-driven applications, and optimization.Optimization topics include robust, stochastic, convex, and non-convex optimization.Source: E TOPICS COLLECTED FROM CS CONFERENCES · pp. 14–15
  • Additional areas include machine learning for sciences, representation learning and AI methods, operating systems, and applied machine-learning and vision systems.The catalog names applications such as creative AI and language, speech, and audio applications.Source: E TOPICS COLLECTED FROM CS CONFERENCES · p. 15
Loading 2510.03120v2…