SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing

1 Introduction

• Rapidly growing research literature makes it difficult to gain in-depth knowledge of a field, while human survey writing is time-consuming and hard to keep current.Survey papers are described as integrating studies and summarizing developments and trends.Source: §1 · p. 1
• Automated surveys can have poorly organized outlines, miss influential literature, and lack fine-grained, objective evaluation of outline and reference quality.The introduction also notes structural imbalance, irrelevant citations, and overlooked contributions.Source: §1 · p. 2
• SURVEYFORGE combines heuristic outline generation using topic-relevant literature and human-survey structures with memory-driven retrieval and content refinement.Its introduction describes a two-stage framework and a temporal-aware reranking engine for subsection literature.Source: §1 · p. 2
• SURVEYFORGE demonstrates strengths in outline structure, reference relevance, and coherent, comprehensive content generation.The introduction reports these as strengths across multiple dimensions.Source: §1 · p. 2
• SurveyBench is introduced as a multidimensional benchmark for assessing outline quality, reference quality, and content quality.The contribution statement describes its purpose as supporting objective evaluation.Source: §1 · p. 2

Autonomous Scientific Discovery

• Scientific-discovery systems range from idea-generation methods to pipelines covering experimental design and manuscript writing.Examples include a hybrid literature-and-brainstorming approach and a closed-loop LLM-driven framework.Source: §2 · p. 2

Automated Survey Generation

• Earlier automated survey methods selected and organized paper sentences or recursively generated content from template trees, with the latter remaining inflexible.The passage describes these earlier approaches as often lacking coherence and readability.Source: §2 · p. 3
• LLM-based survey approaches include hierarchical catalogue generation with fixed reference papers, AutoSurvey, and HiReview’s hierarchical exploration of paper relationships.The passage says catalogue generation is limited to outline generation, while HiReview relies on two-hop citation networks from existing surveys rather than commonly cited papers.Source: §2 · p. 3
• The related-work discussion says AutoSurvey does not focus on human academic writing styles and key references, both considered important for high-quality surveys.Source: §2 · p. 3

3 Method

• SurveyForge has two stages—outline generation and content writing—and introduces SurveyBench to evaluate automated survey generation.The outline stage uses research papers and existing survey structures; the content stage uses a memory-driven scholar navigation agent.Source: §3 · p. 3

3.1 Heuristic Outline Generation

• A top-down heuristic approach uses human-written survey outlines and topic-relevant literature to build comprehensive, academically organized outlines.It retrieves papers and outlines to identify thematic areas and relationships, then recursively develops second-level headings and queries.Source: §3.1 · pp. 3–4

3.2 Memory-Driven Content Generation

• Content generation proceeds through literature retrieval followed by parallel content creation using SANA and an LLM.The two steps are performed sequentially by the Scholar Navigation Agent and the LLM.Source: §3.2 · p. 4

3.2.1 SANA: Scholar Navigation Agent

• SANA uses outline-level literature as memory for query decomposition and retrieval, then reranks papers by relevance, citation impact, and publication recency.Its reranking groups papers into two-year periods and retains highly cited papers; a hyperparameter sets the number used for each subsection.Source: §3.2.1 · pp. 4–5

3.2.2 Parallel Generation and Refinement

• Parallel section generation reduces generation time and supports survey length, while a refinement stage addresses possible repetition across independently generated sections.The refinement stage operates on the draft formed by concatenating the section contents.Source: §3.2.2 · p. 5

3.3 Multi-dimensional Evaluation Benchmark

• SurveyBench contains approximately 100 human-written surveys across 10 topics, and SAM evaluates reference, outline, and content quality.SAM-R measures reference overlap, SAM-O scores outline quality from 0 to 100, and SAM-C evaluates content quality.Source: §3.3 · pp. 5–6

4.1 Experimental Settings

• The evaluation selects approximately 100 human-written surveys across 10 topics, uses AutoSurvey as the baseline, and aligns the retrieval timeline with its settings.Source: §4.1 · p. 6

4.2 Main Results

• Input Coverage rises from 0.12 to 0.22 (Claude-3-Haiku) and 0.07 to 0.20 (GPT-4o mini); Reference Coverage rises from 0.23 to 0.40 and 0.20 to 0.42, respectively.High-quality human-written surveys score 0.63 on Reference Coverage.Source: §4.2 · pp. 6–7
• Across SurveyBench’s reference, outline, and content metrics, SURVEYFORGE substantially improves over AutoSurvey, though it remains below expert-crafted surveys.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Comparison with Human Evaluation

• Automated outline Score Win Rate is 73.00% and Comparative Win Rate 75.00%, near human evaluation’s 74.00%; content Score Win Rate is 69.00% versus experts’ 70.00%.Source: Table 2 · p. 7, §4.3 · p. 8

4.2 Main Results

• Outline quality rises from 82.25 to 86.58 with Claude-3-Haiku and 83.10 to 86.62 with GPT-4o mini; content quality rises from 73.87 to 76.34 and 75.05 to 77.06, respectively.Source: §4.2 · p. 7

4.4 Ablation Study

• Outline quality scores rise from 81.78 without structural guidance to 84.58 with random survey outlines and 86.67 with domain-specific outlines.Source: Table 3 · p. 7, §4.4 · p. 8
• Content-generation ablations associate memory in retrieval and sub-query processing with better literature quality, while temporal-aware reranking supports a more comprehensive, balanced reference collection.Source: Table 4 · p. 8, §4.4 · p. 8

• SURVEYFORGE combines heuristic outline generation with memory-driven content generation to produce high-quality surveys.Source: §5 · p. 8
• The paper introduces a multidimensional benchmark to comprehensively assess survey quality.Source: §5 · p. 8
• The authors aim to reduce the learning curve for researchers entering unfamiliar fields and promote cross-disciplinary knowledge integration.Source: §5 · p. 8

• LLMs can struggle to analyze and synthesize relationships across sources, lacking some critical thinking and originality of human-authored work.Source: Limitations · p. 9
• These limits constrain the generated content’s ability to reflect research trends or offer forward-looking insights.Source: Limitations · p. 9
• LLM hallucinations also affect the accuracy of generated content and citations.Source: Limitations · p. 9
• Future work could better capture interconnections among references to improve generated content’s coherence, depth, and scholarly value.Source: Limitations · p. 9

• The framework relies on publicly available datasets and research papers to comply with copyright and intellectual property laws.Source: Ethics Statement · p. 9
• The framework is designed to augment human expertise in summarizing existing literature.Source: Ethics Statement · p. 9
• Users are encouraged to critically evaluate outputs for ethical research alignment and possible biases or incomplete summaries.Source: Ethics Statement · p. 9

A Appendix

• Due to page limits, supplementary material provides further details and visualizations, including database construction and SurveyBench.Source: A Appendix · p. 12
• The appendix adds discussion of generated versus human-written surveys and details on human evaluation and inter-rater agreement.Source: A Appendix · p. 12
• Other supplementary sections cover open-source-model experiments, time and economic costs, qualitative results, generated-survey examples, and prompts.Source: A Appendix · p. 12

A.1 Database Construction

• The Research Paper Database contains about 600,000 papers, while the Survey Outline Database contains 20,000 review articles for content generation and structural guidance.The paper database stores titles and abstracts; the outline database stores titles, abstracts, and extracted outlines.Source: §A.1 · p. 12
• MinerU extracts hierarchical section and subsection headings from survey articles, and Claude-3.5-sonnet-20241022 refines and standardizes the extracted outlines.Rule-based extraction can introduce noise because papers vary in formatting and structure.Source: §A.1 · p. 12
• The databases are encoded with gte-large-en-v1.5 to capture semantic relationships and enable similarity-based retrieval.The combination of structured human-written examples and semantic encoding supports outline generation and content retrieval.Source: §A.1 · p. 12

A.2 Details of SurveyBench

• SurveyBench covers ten trending computer-science topics, with high-quality human-written surveys curated by 20 doctoral researchers with expertise in those areas.The surveys are predominantly from the last two years; selected surveys serve as content-evaluation references.Source: §A.2 · p. 12
• Assessment criteria adapt peer-review guidelines into measurable components, including topic uniqueness and structural balance for outline evaluation.Researchers with at least two years of top-CS-venue reviewing experience contributed to developing the criteria.Source: §A.2 · p. 12
• Each topic has 330–994 collected references, and the selected high-quality evaluation surveys have 128–1,690 citations.Table 5 summarizes each topic’s unique reference count and the citation count of its selected evaluation survey.Source: §A.2 · p. 13, Table 5 · p. 13

A.3 Discussion about Generated Surveys and Human-written Surveys

• The system struggles to connect publications deeply, often listing references mechanically instead of comparing temporal innovations or methodological evolution.The authors attribute this limitation mainly to identifying core literature and constructing deep logical relationships across long-form knowledge.Source: §A.3 · p. 13
• Generated surveys occasionally contain inaccurate citations or academic claims, which may affect their reliability.The authors identify content and citation accuracy as an area needing further improvement.Source: §A.3 · p. 13
• The authors propose comprehensive knowledge-association networks, using core-entity extraction and citation graphs, as a possible way to improve identification of deep inter-publication connections.The proposal is framed as future work.Source: §A.3 · p. 13

A.4 Details of Human Evaluation and Inter-rater Agreement

• Twenty active computer-science PhD researchers with topic-specific publication experience evaluated the surveys using shared guidelines, training, and a double-blind process.Experts were paid $50 per hour, and each survey took approximately 1–3 hours to evaluate.Source: §A.4 · pp. 13–14
• For 100 surveys across 10 topics, LLM–human Cohen’s kappa was 0.7177 for outline and 0.6462 for content, while human–human kappa was 0.7921 and 0.7098.The authors report good consistency for evaluating long-form content as well as outline quality.Source: §A.4 · p. 14

A.6 Details of Time and Economic Cost

• At $0.43 per survey with GPT-4-mini-2024-07-18, SurveyForge generates approximately 64k-token surveys in about 10 minutes.Generation averages 2.37M input tokens and 0.13M output tokens; actual duration may vary with API rate limits.Source: §A.6 · p. 14

A.7 Qualitative Results

• Baseline outlines often had weak hierarchy and coherence, fragmented paper-by-paper subsections, and redundancy; SurveyForge produced more coherent, hierarchically organized outlines.The comparison is presented in Figure 4.Source: §A.7 · pp. 14–15, Figure 4 · p. 16

A.5 Additional Experiments with Open-Source Models

• 87.42 outline quality for DeepSeek-v3 (Liu et al., 2024) approached the human-survey score of 87.62, with 0.2554 Input Coverage and 0.4553 Reference Coverage.DeepSeek-v3 scored 79.20, 80.17, and 81.07 on content structure, relevance, and coverage, respectively, for a mean of 80.15. Its cost was $0.37 per survey versus $0.43 for GPT-4o-mini.Source: §A.5 · p. 14, Table 7 · p. 15

A.8 Example of Generated Survey

• SurveyForge-generated surveys can synthesize field knowledge and suggest research directions, as illustrated by a survey of multimodal large language models.That example discusses scalability, cross-modal integration, efficient training and inference, ethical and societal concerns, and applications including healthcare and education, alongside computational and privacy challenges.Source: §A.8 · p. 15

A.9 Prompt Used

• Generation prompts use relevant literature and human-survey structures to guide original, logically connected outlines, while content prompts require literature-supported writing and length constraints.Evaluation prompts score outline uniqueness, balance, hierarchy, and organization; content assessment first reviews human-written surveys on the same topic for context.Source: §A.9 · pp. 15, 18
• Evaluators assess only relevance or structural quality, compare the generated review with a human-written review, and treat the latter as a reference rather than an absolute standard.Each prompt requests a single numerical score from 0 to 100 with no additional commentary.Source: §A.9 · pp. 21–22
• Relevance is scored out of 100 across topic alignment (35), content appropriateness (35), and information focus (30).The rubric defines relevance through topic fit, appropriate information, and focus on key aspects of the subject.Source: §A.9 · p. 21
• Structural quality is scored out of 100 across logical flow and organization (35), hierarchical structure (35), and format and presentation (30).The prompt defines structure in terms of organization, logical flow, connected development of ideas, and presentation.Source: §A.9 · p. 22