SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models

• Existing RAG approaches rely on textual relevance without assessing literature quality, risking the inclusion of low-impact or marginal studies.Source: §1 · pp. 1–2
• Large-scale benchmarks remain lacking for rigorous comparison with human-written surveys, and citation quality, structural consistency, and domain variation are underexplored.Source: §1 · p. 2
• QUAL-SG expands candidate references through citation graphs and reranks articles using quality indicators, while the study benchmarks state-of-the-art LLMs under different levels of human involvement.Source: §1 · p. 2
• SurveyGen contains over 4,200 human-written surveys with section-level structures, cited references, and metadata on citation performance, author influence, and venue reputation.Source: §1 · p. 2
• QUAL-SG significantly improves citation reliability, overall content quality, and structure consistency in generated surveys.Source: §1 · p. 2

2.1 Task Design

• The evaluation distinguishes fully LLM-based generation, standard RAG, and human-guided writing with human-selected references and a predefined outline.The three settings vary how much external literature and human input the LLM receives.Source: §2.1 · pp. 2–3

2.2 SurveyGen: Dataset Construction

• SurveyGen contains 4,205 surveys, 115,376 sections, 242,143 directly cited references, and 5,062,596 references cited by those papers.Surveys with fewer than 30 references or fewer than three top-level sections were removed.Source: §2.2 · p. 3

2.3 Quality-Related Indicators Supplement

• SurveyGen supplements papers with citation-performance, author-influence, and venue-reputation indicators from OpenAlex.The indicators include citation counts, author h-index and publication totals, and venue h-index, i10-index, and CORE status.Source: §2.3 · p. 4

2.4 QUAL-SG: QUALity-aware Literature Retrieval for Survey Generation

• QUAL-SG expands semantically retrieved candidates by adding papers cited by at least two candidates, helping include influential works with weaker topic similarity.The initial candidate set is retrieved by abstract-embedding similarity, with its size exceeding the human survey’s reference count.Source: §2.4 · pp. 4–5
• QUAL-SG re-ranks candidates using topical relevance, academic impact, and content diversity, then selects the top K papers for generation.Academic impact combines citation performance, author influence, and venue reputation with adjustable weights; K matches the reference count of the corresponding human survey.Source: §2.4 · p. 5

3.1 Baselines

• Experiments use 120 highly cited SurveyGen surveys, with 30 each from Biology, Medicine, Psychology, and Computer Science, and compare Fully-LLMGen, Naive-RAG, and human-written surveys.Six LLMs serve as generation agents across the experiments.Source: §3.1 · p. 5

3.2 Evaluation Metrics

• Automatic evaluation measures citation overlap and accuracy, semantic similarity, ROUGE, key-point recall, and structural consistency against human-written surveys.Structural consistency uses section overlap and an LLM judge’s structural-alignment rating.Source: §3.2 · pp. 5–6
• Human evaluation rates topic relevance, information coverage, critical analysis, and overall quality against human-written surveys.Human-written surveys are treated as ground truth for both automatic and human evaluations.Source: §3.2 · pp. 5–6

4.1 Main Results

• 82.43% similarity and 15.36% ROUGE-L make LLaMA-3.1-70B the Task 1 content leader, while Claude 3.7-Sonnet leads overall across citation quality, KPR, and structural consistency.Closed-source models surpass open-source models in citation quality and structural consistency, while open-source models deliver comparable content generation.Source: §4.1 · p. 6
• 35.84% best citation accuracy contrasts with 46.59% key point coverage and 14.89% structural overlap against human-written surveys.The authors report that these gaps indicate relying solely on LLMs is insufficient for reliable reference generation.Source: §4.1 · p. 6
• 16.73% citation F1: QUAL-SG exceeds Naive-RAG by 10.80% and Fully-LLMGen by 8.97%, while also improving content and structural metrics over both baselines.Content gains are Similarity +0.73%, ROUGE-L +1.40%, and KPR +3.66%; structural gains are LLM evaluation +0.16 on a 5-point scale and semantic overlap +12.54%.Source: §4.1 · pp. 6–7, Table 3 · p. 7
• 84.39% content similarity and 17.16% ROUGE-L go to LLaMA-3.1-70B in Task 3, while Claude-3.7-Sonnet leads KPR at 54.67%.Task 3 provides references and outlines extracted from human-written surveys; its content quality improves over Task 2, which has no human intervention.Source: Table 4 · p. 7, §4.1 · p. 7

4.2 Further Analysis of Reference Selection

• QUAL-SG aligns most closely with human-written surveys in citation-count and temporal reference distributions; Fully-LLMGen favors less-cited papers and often misses recent studies.Human-written surveys tend to favor papers from the preceding decade; the analysis attributes Fully-LLMGen’s omission of recent studies to outdated training data.Source: §4.2 · pp. 7–8, Figure 3 · p. 7

4.3 Cross-Disciplinary Comparison of LLMs

• Across disciplines, content similarity shows no significant differences, but ROUGE-L and structural consistency do; KPR differences are not significant for individual models.ROUGE-L and KPR are generally higher in Computer Science and Psychology than in Medicine and Biology; LLaMA-3.1-70B leads ROUGE-L and Claude-3.7-Sonnet leads KPR.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 Comparison with Other Ranking Models

• QUAL-SG outperforms UPR in retrieval and uses weighted aggregation to handle multiple ranking criteria more robustly than RankGPT.The comparison reports retrieval-stage performance because generation depends mainly on the selected references.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Human Evaluation Results

• Human evaluators generally find Task 3 more acceptable, yet generated surveys still lack sufficient information coverage and critical analysis despite comparable topic relevance.The authors identify high-quality reference selection and effective outline construction as important preprocessing steps for guiding more reliable survey generation.Source: §4.5 · p. 8

5 Discussion and Future Directions

• Fully LLM-based and RAG-based approaches have not reached human-level performance; generated surveys remain limited in information coverage and in-depth analysis despite strong topical relevance.The discussion also identifies hallucinated references and factual inaccuracies as critical challenges; RAG reduces hallucinations, but retrieved papers can remain misaligned with human preferences.Source: §5 · p. 9
• Future work could use citation networks, human citation behavior, and human-annotated data for reference selection, while adding full texts, human-in-the-loop control, verification, and long-document modeling.The authors state that using abstracts alone limits information coverage because they do not capture broader paper details.Source: §5 · p. 9

Real-World Applicability and Deployment

• The modular retrieval, co-citation, and re-ranking components can be extended; users can replace databases and customize ranking strategies for their needs.The authors also note that open-source models such as LLaMA-3.1-70B can offer flexibility based on budget, deployment needs, and infrastructure.Source: Real-World Applicability and Deployment · p. 9

6 Conclusion

• SurveyGen supports scientific survey generation, and QUAL-SG improves on semantic-similarity RAG in citation quality, content quality, and structural consistency.The paper also reports that more accurate references and a well-structured outline produce surveys more aligned with human-written ones, though citation and content quality still need improvement to meet human expectations.Source: §6 · p. 9

Input Limitation

• The framework uses only abstracts and bibliographic metadata, which may limit generated surveys’ depth and completeness relative to surveys based on full papers.The restriction is attributed to copyright reasons.Source: Limitations · p. 10

Post-generation Refinement

• The authors omit post-generation polishing, citation formatting, and structural adjustments to reduce API costs, and do not generate figures, tables, or diagrams.They note these post-processing steps could improve personalization and overall quality; Claude-3-Haiku may perform better on longer surveys because of extended context handling.Source: Limitations · p. 10

Data Contamination

• Some open-access surveys or references may have appeared in LLM training data, and the evaluation does not explicitly control for this possible contamination.The authors say it could slightly alter performance estimates, but shared exposure across mainstream-LLM baselines is unlikely to affect relative comparisons.Source: Limitations · p. 10

Evaluation Sample Scope

• Empirical evaluation covers 120 relatively short surveys across multiple disciplines, selected to balance cost and feasibility.The authors expect similar trends across the full dataset but encourage broader benchmarking.Source: Limitations · p. 10

Data Collection, Ethics, and Licensing

• SurveyGen uses S2ORC, released under CC BY-NC 4.0, and its metadata excludes sensitive personal data; the dataset is intended for non-commercial academic research, not ranking individuals or venues.The authors state that metadata collection complies with source terms and that the dataset will be released under the same non-commercial license.Source: Ethics Statement · p. 10

Caution about Use of LLMs

• Because generated surveys may contain factual inconsistencies, the authors advise critical review and say they are for reference, not substitutes for peer-reviewed articles or expert judgment.Source: Ethics Statement · p. 10

A Related Work

• SurveyGen adds extensive referenced-paper, author, and venue metadata and supports multidiscipline evaluation, unlike SciReviewGen’s computer-science focus, while related datasets address single- or multi-document summarization.Prior LLM survey systems use RAG and structured generation approaches, including staged drafting, AttributeTree preprocessing, and quality control.Source: A Related Work · p. 14

B Input and Output Settings

• Task 1 generates a survey from its topic alone, Task 2 adds retrieved literature and quality metadata, and Task 3 supplies human-selected references and a human-written outline.Task 3 removes retrieval and represents a human-guided writing setting.Source: B Input and Output Settings · p. 14

C Implementation Details

• QUAL-SG retrieves 300 topic-relevant papers, adds 50 frequently co-cited candidates, then ranks papers using citation performance, author influence, and venue reputation.The weights are 0.5, 0.2, and 0.3 respectively; the final subset matches the corresponding human-written survey’s reference count, and weights can be changed.Source: C Implementation Details · p. 15

D.1 Metric Formulations

• Citation evaluation compares generated or retrieved references with human-selected references using precision and recall, counts matches above 0.95 textual similarity, and separately checks generated-reference title matches in S2ORC.Source: §D.1 · p. 15
• Key Point Recall checks whether points extracted from a human-written survey appear in the generated survey, while structural evaluation uses a 0.8 semantic-similarity threshold for section matches.Higher KPR means more human-survey key points are covered.Source: §D.1 · pp. 15–16
• The human-evaluation table reports results across tasks for five computer-science surveys per task, all generated using Claude-3.7-Sonnet.Task 2 surveys use the QUAL-SG pipeline.Source: Table 6 · p. 16

D.2 Comparison with Existing Metrics

• Citation precision complements recall because recovering many human-selected references can still overstate quality when many irrelevant references are added.The authors illustrate this with 80% recall alongside over 20 additional irrelevant references.Source: §D.2 · p. 16
• Structural evaluation compares generated outlines with human-written survey structures, enabling identification of section types that are covered, missing, or over-generated.The authors use human-written, carefully designed and peer-reviewed outlines as the structural reference.Source: §D.2 · p. 16

E Human Evaluation Protocol

• Fifteen computer-science surveys were evaluated by three second-year PhD annotators across topic relevance, information coverage, critical analysis, and overall rating.Five surveys were randomly selected for each task; each annotator had published at least one peer-reviewed paper.Source: E Human Evaluation Protocol · p. 16
• Each generated survey was paired with its human-written counterpart, and identifying information was removed so annotators did not know which was machine-generated.For each pair, annotators judged whether the generated survey was better, comparable, or worse.Source: E Human Evaluation Protocol · p. 16
• Table 7 presents the QUAL-SG literature-retrieval ablation study, with best results bolded and second-best results underlined.Source: Table 7 · p. 17
• Table 8 presents a case study of two SurveyGen surveys; an asterisk marks a highly co-cited reference added without semantic-similarity retrieval.The S2ORC ID identifies the article in the S2ORC corpus.Source: Table 8 · p. 17
• The corresponding human-evaluation results are summarized in Table 6 of Section 4.5.Source: E Human Evaluation Protocol · p. 17

F.1 Ablation Study

• Removing academic-impact ranking (-7.72%) and co-cited-paper expansion (-5.98%) caused the largest declines, while relevance-based ranking and content diversity also contributed positively.QUAL-SG performance declined in every ablation setting.Source: §F.1 · p. 17

F.2 Case Analysis

• In two Computer Science and Biology cases, QUAL-SG recovered all key human-selected papers, whereas Fully-LLMGen and Naive-RAG missed crucial references.Naive-RAG found only one valid reference out of seven; co-citation expansion added semantically distant but influential papers, and quality ranking selected impactful studies.Source: §F.2 · p. 17

G Topic Examples for Survey Generation

• The topic examples list survey titles, identifiers, and citation counts across Biology, Medicine, Psychology, and Computer Science.Examples include “Microbial Enzymes in Pollutant Bioremediation” in Biology and “Multimodal Machine Learning Taxonomy” in Computer Science.Source: G Topic Examples for Survey Generation · p. 18

H SurveyGen Data Format

• The SurveyGen data format represents corpus papers with metadata and sections, and records references with matched paper IDs and metadata.Paper metadata can include titles, abstracts, authors, and journal information; sections contain paragraphs and citations.Source: H SurveyGen Data Format · p. 19

Prompt for Survey-type Article Classification

• The survey-type classification prompt asks whether an abstract declares survey intent, reviews existing work, and synthesizes trends, gaps, and future directions.The requested output is a single TRUE or FALSE judgment.Source: Prompt for Survey-type Article… · p. 20

Prompt for Topic Relevance Evaluation

• The topic-relevance prompt uses a five-point relevance scale and gives outline and subsection-writing tasks that rely on supplied papers and supported citations.The instructions call for 3–7 thematic sections and subsection content of at least 300 words, with claims supported by academic literature.Source: Prompt for Topic Relevance Evaluation · pp. 20, 24

Prompt for Structural Consistency Evaluation

• The evaluator compares the LLM-generated and human-written outlines, scores structural consistency from 1 to 5 using alignment and coverage criteria, and returns only the score.The scale ranges from very poor alignment, with major omissions or irrelevant sections, to excellent alignment in structure and content coverage.Source: Prompt for Structural Consistency… · p. 25