PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSurvey: Large Language Models Can Automatically Write Surveys

Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, Yue Zhang

arXiv:2406.10252v2cs.IRcs.AIcs.CL

TL;DR

Rapidly growing research output makes comprehensive literature surveys difficult, while LLM-based survey writing faces context, knowledge, and evaluation challenges. AutoSurvey retrieves papers, drafts and refines survey sections, and evaluates candidates; the authors report that it outperforms naive RAG and matches human performance in content and citation quality while being efficient.

  • Problem

    Rapid literature growth makes comprehensive survey synthesis difficult, while LLMs face context-window limits, incomplete or hallucinated references, and a lack of scalable evaluation benchmarks.

    Source: §1 · pp. 1–2

  • Method

    AutoSurvey retrieves current papers, generates an outline and parallel subsection drafts, refines the merged survey, and evaluates candidates with multiple LLMs.

    Source: §1 · pp. 2–3, §2 · p. 4

  • Results

    AutoSurvey outperforms naive RAG-based LLM generation and matches human performance in content and citation quality, while also being highly efficient.

    Source: §6 · p. 9

  • Takeaways & Limitations

    AutoSurvey offers a scalable approach to synthesizing research literature for rapidly evolving fields such as artificial intelligence.

    Source: §6 · p. 9

    Manual analysis found misinterpretation accounted for 10% of unsupported-claim errors, including claims that misrepresented a cited paper’s focus.

    Source: §5 · p. 9

Abstract

from arXiv · show

This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys in rapidly evolving fields like artificial intelligence. Traditional survey paper creation faces challenges due to the vast volume and complexity of information, prompting the need for efficient survey methods. While large language models (LLMs) offer promise in automating this process, challenges such as context window limitations, parametric knowledge constraints, and the lack of evaluation benchmarks remain. AutoSurvey addresses these challenges through a systematic approach that involves initial retrieval and outline generation, subsection drafting by specialized LLMs, integration and refinement, and rigorous evaluation and iteration. Our contributions include a comprehensive solution to the survey problem, a reliable evaluation method, and experimental validation demonstrating AutoSurvey's effectiveness.We open our resources at \url{https://github.com/AutoSurveys/AutoSurvey}.

1 Introduction

Rapidly expanding AI literature makes comprehensive survey writing difficult, while LLM-based automation faces context, knowledge, and evaluation constraints. AutoSurvey combines retrieval, parallel drafting, refinement, and evaluation; experiments report strong citation and content quality across survey lengths.

1 Introduction

  • Over 4,000 papers mentioning “Large Language Model” in titles or abstracts appeared on arXiv in the first four months of 2024, outpacing comprehensive scholarly synthesis.The paper links the volume and complexity of research to difficulty producing traditional human-authored surveys.Source: §1 · p. 1
  • Despite rising survey numbers, research-topic clusters still reveal areas lacking comprehensive surveys.Figure 1’s 2024 data extends only through April, with a bar forecasting the full year.Source: §1 · p. 1, Figure 1 · p. 2
  • LLM survey generation is constrained by output windows, incomplete or hallucinated parametric knowledge, and a lack of scalable quality benchmarks.The paper notes that surveys may require tens of thousands of tokens, and that training data can exclude recent studies.Source: §1 · p. 2
  • AutoSurvey combines parallel outline and subsection generation with systematic revision, while retrieval brings recent papers into outlines and subsection drafts.The system also uses multiple LLMs and expert-refined metrics to assess citation recall and precision, plus content coverage, structure, and relevance.Source: §1 · pp. 2–3
  • At 64k tokens, citation recall and precision are 82.25% and 77.41% for AutoSurvey, versus 68.79% and 61.97% for naive RAG-based LLMs, approaching human scores of 86.33% and 77.78%.The cited figures are recall followed by precision for each system.Source: §1 · p. 3
  • At 64k tokens, AutoSurvey scores 4.73 for coverage, 4.33 for structure, and 4.86 for relevance, versus human scores of 5.00, 4.66, and 5.00.The paper also reports that the mixture of models has the highest Spearman correlation with human rankings, at 0.5429.Source: §1 · p. 3
  • The authors describe AutoSurvey as the first system to explore large-model agents for extensive academic surveys and propose survey evaluation criteria aligned with human preferences.They present those criteria as a reference for future related research.Source: §1 · p. 3

2 Methodology

AutoSurvey creates surveys through four phases: retrieving papers and building an outline, drafting subsections, integrating and refining them, and evaluating candidate documents. Parallel work supports drafting, while contextual refinement and iterative selection produce the final survey.

2 Methodology

  • The algorithm’s four phases run from initial retrieval and outline generation through parallel subsection drafting and refinement to evaluation of candidate surveys.Its inputs are a survey topic and publications database; its output is the selected refined survey.Source: §2 · p. 4

Initial Retrieval and Outline Generation

  • The process begins by retrieving topic-relevant publications and generating a structured outline with subsection titles and descriptions.When the retrieved pool exceeds an LLM’s context window, papers are split into batches, outlines are generated, and then consolidated.Source: §2 · p. 4

Subsection Drafting

  • Specialized LLMs draft outline subsections in parallel, retrieving references relevant to each subsection and citing them in the generated content.Generated references are extracted and mapped to corresponding arXiv papers.Source: §2 · p. 4

Integration and Refinement

  • Each draft is refined using its preceding and following subsection context, then merged to improve readability, reduce redundancy, and maintain coherence.Refinement also checks and corrects citation errors.Source: §2 · p. 4

Rigorous Evaluation and Iteration

  • A Multi-LLM-as-Judge evaluation guides further refinement, and the best survey is selected from N candidates.The algorithm repeats generation trials before evaluation selects the final document.Source: §2 · pp. 4–5

3 Experiments

Experiments compare AutoSurvey with human-authored surveys and naive RAG-based LLM generation, assessing speed, citation quality, and content quality. They also test evaluation alignment, component ablations, writer choice, iteration, and whether surveys help answer literature questions.

Main Results

  • AutoSurvey balances near-human coverage, relevance, and citation quality with much lower time cost, though human writing leads in structure and overall quality.Source: Table 2 · p. 6, §3 · pp. 6–7
  • At 64k tokens, AutoSurvey produces 73.59 surveys per hour, versus 0.07 for human writing and 12.56 for naive RAG-based LLM generation.Source: Table 2 · p. 6, §3 · p. 6
  • For 8k-token surveys, citation recall and precision are 82.48 and 77.42 for AutoSurvey, versus 78.14 recall and 71.92 precision for naive RAG-based generation.Source: Table 2 · p. 6, §3 · p. 7
  • At 16k tokens, AutoSurvey averages 4.60 in content quality, with 4.66 coverage, 4.33 structure, and 4.86 relevance, matching human writing and surpassing naive RAG generation.Source: Table 2 · p. 6, §3 · p. 7

Meta Evaluation

  • The mixture of evaluation models has the highest LLM–human ranking correlation, with Spearman’s rho of 0.5429, indicating moderate positive correlation.Source: Figure 3 · p. 7, §3 · p. 7

Ablation Study

  • Removing retrieval significantly degrades citation quality, while removing reflection slightly affects content quality, especially structure and coherence.Source: Table 3 · p. 7, §3 · p. 7

Base Writer Ablation

  • GPT-4 slightly outperforms Claude-haiku and Gemini-1.5-Pro in overall content quality, while Claude-haiku remains competitive with human standards.Source: §3 · pp. 7–8, Table 4 · p. 8

Iteration Ablation

  • Increasing iterations from 1 to 5 slightly improves overall content quality, with diminishing returns after the second iteration.Source: Figure 4 · p. 8, §3 · p. 8

4 Related Work

Prior work tackles long-form generation through context extension, memory, and hierarchy, while automated writing systems have largely focused on shorter texts. AutoSurvey applies hierarchical outlining and refinement to academic reviews as long as 64k tokens.

Long-form Text Generation

  • Earlier long-text methods extend context or add memory, but can require costly retraining or incur long generation times.Hierarchical methods can be more efficient but may ignore long-range dependencies and reduce content quality.Source: §4 · p. 8
  • AutoSurvey uses hierarchical generation, a structured outline, and content refinement to address drawbacks of prior long-form approaches [38].Source: §4 · p. 9

Automatic Writing

  • Earlier automated-writing methods address faithfulness, structure, and style, but the cited systems focus on texts shorter than 4k; AutoSurvey targets academic reviews up to 64k [39] [40] [41] [42] [43] [44] [27] [45] [46] [47] [20].Examples include RAG, IRP, PaperRobot, and STORM.Source: §4 · p. 9

5 Limitation

Manual analysis of 100 unsupported claims identifies misalignment, misinterpretation, and overgeneralization as citation-error types. Overgeneralization is the largest category, at 51%.

  • 51% of the 100 examined unsupported claims were overgeneralizations, the largest error category, indicating reliance on parametric knowledge [34].One example extends a paper's discussion of multi-task few-shot learning to an influence on in-context learning that it does not explicitly address.Source: §5 · p. 9
  • 39% were misalignments, such as citing a paper that mentions GDPR as if it proposed GDPR [32].Source: §5 · p. 9
  • 10% were misinterpretations, where a related source was incorrectly represented, such as attributing a claim about in-context learning to a paper focused on meta-out-of-context learning [33].Source: §5 · p. 9

6 Conclusion

AutoSurvey combines retrieval, outline generation, parallel subsection drafting, integration, and evaluation to automate comprehensive literature surveys. Experiments report better performance than naive RAG-based generation and content and citation quality matching human performance, alongside high efficiency.

  • AutoSurvey is presented as a systematic approach to comprehensive survey writing, and experiments report that it outperforms naive RAG-based generation while matching human content and citation quality.The approach includes initial retrieval, outline generation, parallel subsection drafting, integration, and rigorous evaluation.Source: §6 · p. 9

A Detail of Topics and Human-writing Surveys

The authors selected 20 LLM-topic surveys, prioritizing topic breadth and citation count. Table 6 lists their basic information.

B Details of Implementations

AutoSurvey indexes paper titles and abstracts for retrieval, then maps generated citation titles back to database entries.

B Details of Implementations

  • Paper title-and-abstract embeddings are stored without chunking because the 8k context window exceeds each abstract’s length.The embedding model is nomic-embed-text-v1.5.Source: B Details of Implementations · p. 13
  • Related papers are retrieved by abstract and ranked by similarity to the query.Source: B Details of Implementations · p. 13
  • Generated citation titles are embedded as queries and mapped to the closest database paper title, and survey lengths are grouped into token-size categories.Lengths from 8k to 16k are classified as 8k, 16k to 32k as 16k, and so on.Source: B Details of Implementations · p. 13

C Details of Evaluation

The evaluation defines citation-bearing sentences as claims and uses three experienced PhD students to score generated surveys with shared criteria.

C Details of Evaluation

D Cost Analysis

Table 7 reports the average tokens used to generate a 32k-token survey and the costs of using different LLMs.

D Cost Analysis

  • The cost analysis reports average generation tokens for a 32k-token survey and costs across different LLMs.Source: D Cost Analysis · p. 13
  • Table 7 presents the cost of AutoSurvey.Source: Table 7 · p. 14

E Societal Impact and Ethical Considerations

AutoSurvey may generate surveys across fields using specialized databases, but its reliance on large-model performance means citation errors remain and the output is for reference only. The appendix prompts specify how outlines, subsections, citations, and coherence are handled.

E Societal Impact and Ethical Considerations

  • AutoSurvey can draw on specialized databases to generate surveys across fields, potentially filling gaps in existing reviews, but large-model reliance means citation errors remain and outputs are intended for reference only.Evaluation participants volunteered and received ample compensation; experimental data comes from arXiv and is permitted for non-commercial use.Source: E Societal Impact and Ethical… · p. 14

F Prompt used in AutoSurvey

Loading 2406.10252v2…