SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentation

1. Introduction

• Survey writing is labor-intensive and depends on authors’ domain knowledge and judgment, while rapid publication growth makes it harder for researchers to follow developments.Surveys synthesize research, highlight limitations and challenges, and offer insights for future directions.Source: §1 · p. 1
• Existing systems often use limited sources and text-based ranking, while leaving planning and revision largely model-internal rather than using human-created guidance.The paper identifies published-survey structures and peer-review feedback as time-consuming human knowledge not systematically incorporated into prior frameworks.Source: §1 · p. 2
• The framework retrieves literature from multiple sources, clusters topics into an initial outline, refines it with human-written survey outlines, and uses peer-review comments to guide draft revision.LLM-powered agents handle literature retrieval, outline generation, survey writing, and refinement.Source: §1 · p. 2
• The framework combines multi-source retrieval with citation-network analysis and re-ranking to select representative papers for grounded survey drafting.It also uses hierarchical outlines from published surveys and real review feedback to support structure and multi-round revision.Source: §1 · p. 2
• Across two domains, SurveyAgent-HKA outperforms mainstream baselines in citation quality, structural consistency, and content quality, with each module contributing to overall performance.A user study further validates the practical value of the generated surveys.Source: §1 · p. 2

2.1. Automated scientific survey generation

• Earlier automated survey methods aggregated representative sentences, while LLM-based systems later used retrieval, outline generation, and survey drafting [14].Other work evaluates reference accuracy, hallucination, coverage, and human preference [29] [31] [32].Source: §2.1 · p. 3
• Existing survey-generation studies often rely on fragmented sources and text-based ranking, lack human-written feedback, and provide limited cross-domain evaluation and comparison with human-written ground truth.The paper notes that most prior studies focus on computer science and use arXiv as a primary source.Source: §2.1 · p. 3
• SurveyAgent-HKA differentiates itself through multi-source retrieval, full-text use, human-derived outline and revision guidance, and cross-domain evaluation against human-written standards.Table 1 summarizes these framework capabilities relative to mainstream baselines.Source: §2.1 · p. 3, Table 1 · p. 4

2.2. Multi-agent systems for text generation

• Multi-agent systems have supported long-form writing and related tasks, but their application to scientific writing remains underexplored.The paper frames survey writing as a multi-stage workflow suited to task decomposition and role-based agent collaboration.Source: §2.2 · p. 4

3.1. Problem formalization

• The task is to retrieve relevant papers from a large academic corpus and generate a structured, long-form survey that incorporates claims from those papers.The formalization separates paper retrieval from survey generation.Source: §3.1 · p. 4

3.2. SurveyAgent-HKA: The overall architecture

• Specialized retrieval and writing agents handle literature search, ranking, extraction, outlining, drafting, review, and refinement, while a central router delegates tasks and shared memory stores intermediate outputs and references.The memory module supports information sharing, reduces duplicate or conflicting content, and tracks references for final validation.Source: §3.2 · pp. 5–6

3.3.1. Literature retrieval and citation-based expansion

• The system expands topic-based retrieval with papers co-cited by at least two retrieved articles and queries multiple academic databases, deduplicating results by DOI.The co-citation step is intended to recover potentially relevant foundational work that semantic similarity may miss.Source: §3.3.1 · p. 7

3.3.2. Dynamic survey outline generation and enhancement

• Clustered paper abstracts are summarized to build an initial hierarchical outline, which is iteratively refined using similar human-written survey outlines as few-shot guidance.The outline guides later survey generation and is intended to align its structure and language with human writing standards.Source: §3.3.2 · pp. 8–9

3.3.3. Topic-focused literature retrieval and re-ranking

• Section-title queries retrieve more targeted papers, which are re-ranked by topical relevance, academic impact, and recent popularity to select representative works for each section.Academic impact combines citation performance, author influence, and venue reputation; recent popularity uses citations from the past three years and total citations.Source: §3.3.3 · pp. 9–10

3.3.4. Information extraction from candidate papers

• Candidate-paper knowledge cards use extracted full-text PDFs when available, but fall back to abstracts when PDFs cannot be downloaded.The extraction template organizes background, methodology, datasets, and results; figures and tables are removed during text cleaning.Source: §3.3.4 · p. 10

3.3.5. Reference-grounded survey drafting and peer review-guided refinement

• Peer-review comments from topic-similar published surveys inform a checklist that guides targeted revisions, with additional retrieval or content supplementation available before final polishing.Subsections are drafted in parallel and grounded in section-relevant documents before assembly.Source: §3.3.5 · pp. 10–11

4.1. Baselines

• Six baselines span direct RAG, staged drafting and refinement, human-survey references, and deep-research approaches.The comparison includes Naive-RAG, AutoSurvey, SurveyX, SurveyGen, SurveyForge, and DeepResearch.Source: §4.1 · p. 11

4.2.1. Dataset for survey generation

• Evaluation uses 20 computer-science survey targets and 10 medicine targets drawn from a 4.2K-survey dataset, with human-written surveys as gold standards.The computer-science targets are shared with prior comparisons; the medicine targets are selected from SurveyGen.Source: §4.2.1 · p. 11

4.2.2. Peer-review comments collection for survey papers

• Review feedback comes from OpenReview and F1000 Research in computer science and DeepReview and F1000 Research in medicine, totaling 480 published survey papers.Survey-type papers are identified through title keywords and a secondary LLM verification process.Source: §4.2.2 · pp. 11–12

4.3.1. Automatic evaluation

• Automatic metrics assess citation selection, semantic heading matches above 0.8, semantic and ROUGE-L content overlap, and key-point recall against human-written surveys.Heading alignment uses first- and second-level headings; key-point recall checks coverage of points extracted from the reference surveys.Source: §4.3 · p. 12, §4.3.1 · pp. 12–13

4.3.2. Human evaluation

• Human evaluation compares coverage, structure, and relevance for 10 computer-science surveys; 30 students rate 10 generated surveys on usefulness across three aspects.Pairwise comparisons include initial versus final drafts and generated versus human-written surveys.Source: §4.3.2 · p. 14

4.4. Implementation details

• Retrieval combines Semantic Scholar with arXiv for computer science or PubMed for medicine, then reranks section-related papers using citation, author-impact, and venue-reputation signals.The first retrieval selects 200 papers per topic: 150 from Semantic Scholar and 50 from the supplementary database; the final section reference set contains 50 papers.Source: §4.4 · pp. 14–15
• The agent workflow clusters papers into an outline, refines it with human-survey outlines, drafts with domain-specific templates, and revises using checklists from 20 related surveys; optimization stops after three iterations.The sequence can be re-invoked for additional retrieval, ranking, and writing when refinement needs more references or content.Source: §4.4 · pp. 15–16

5.1. Overall comparison with baselines

• SurveyAgent-HKA achieves the best citation quality in both domains, the highest structural match, and strong key-point recall, while ranking second in content overlap behind DeepResearch.Citation F1 is 16.94% in computer science and 19.39% in medicine; structural relevance scores are 3.5/5 and 3.9/5, and key-point recall is 53.85% and 57.07%, respectively.Source: §5.1 · pp. 16–17

5.2.1. Citation performance with different retrieval strategies

• Combining citation-based signals with recent relevance outperforms similarity-only retrieval across K, while multiple data sources improve search accuracy and recall over a single database.Citation-based gains slow beyond K=50; Semantic Scholar outperforms arXiv and PubMed in citation quality, though its advantage over PubMed is smaller in medicine.Source: §5.2.1 · p. 17, Figure 3 · p. 17, §5.2.2 · p. 18

5.2.3. Distribution analysis of the retrieved references

• Retrieved references align more closely with human-selected distributions across four dimensions, but substantial differences remain because human selection can include older or inaccessible works.The analysis compares citation count, author and journal H-index, and publication time gap; similarity-based systems retrieve more recent, lower-citation papers, while human selection remains difficult to replicate.Source: §5.2.3 · pp. 18–19, Figure 5 · p. 19

5.3.1. Statistical analysis of the generated survey outline

• The model achieves the strongest outline F1 matches, while second-level headings are harder and computer-science outlines are more challenging than medical ones.F1 is 38.53% (H1) and 33.42% (H2) in computer science, and 45.38% (H1) and 33.85% (H2) in medicine. Matched H1 headings cluster at survey beginnings and endings, while H2 matches concentrate in the middle; few-shot performance peaks at 10 examples before declining with excess examples.Source: §5.3.1 · pp. 19–20, Table 8 · p. 20, §5.3.2 · pp. 20–21

5.4.1. Analysis of peer review comments on survey revision

• Peer-review-guided revisions improve citation, structure, and content, with three revision rounds balancing quality gains against added time and token costs.Across computer science and medicine, citation F1 rises by 3.86% and 2.62%; H1/H2 F1 by 9.31%/6.19% and 4.92%/6.71%; and ROUGE/KPR by 2.81%/10.25% and 4.22%/8.48%. In computer science, V3 achieves the highest overall performance, after which gains become marginal.Source: §5.4.1 · p. 21, §5.4.2 · p. 22

5.6.1. Comparison across LLM-generated and Human-written surveys

• Human evaluation finds the final version improves over the initial draft, but human-written surveys retain an advantage; users find generated surveys useful, though research insights are weaker.The final version’s structure and coverage improve by more than 40%; 83 user-study responses rate surveys mostly between 3 and 4. LLM evaluations show reasonable agreement with human judgments, including Cohen’s κ of 0.82 for annotator content judgments.Source: §5.6.1 · p. 23, Figure 9 · p. 23, Figure 10 · p. 23

5.7. Cost and efficiency analysis

• The workflow takes about 12 minutes for survey writing, while a case study shows coherent topic coverage but less depth in challenges and future directions.Writing costs range from $0.50 with GPT-4o-mini to $0.89 with Claude-3.5-Haiku; the case study’s application section overlaps with human-selected sections in three of five subchapters, while two of 11 introduction references match human choices.Source: §5.7 · p. 25, §5.9 · pp. 26–27

5.8. Ablation experiments

• Ablations show that citation expansion, re-ranking, topic clustering, few-shot learning, knowledge extraction, and peer-review feedback each contribute to survey quality.Removing re-ranking reduces citation F1 by 4.83% in computer science and 4.56% in medicine; removing few-shot learning lowers structural F1 by 7.02% and 4.36%. Removing peer-review feedback reduces KPR to 43.60% and 48.59%, respectively.Source: §5.8.1 · p. 25, Table 14 · p. 25, §5.8.2 · p. 26

5.10. Error analysis

• The most frequent errors concern research insights and cross-paper comparisons, indicating stronger individual-study summaries than deeper synthesis.The error taxonomy was derived from recurring human-evaluation explanations and user-study limitations, then refined through pilot inspection.Source: §5.10 · p. 27
• Missed or poorly ranked studies and copyright-limited full texts can restrict evidence to titles and abstracts, weakening detailed comparisons and critical discussion.The framework can make fuller use of open-access papers.Source: §5.10 · p. 28
• Human input is needed earlier for literature selection, evidence verification, and outline refinement because errors at those stages are difficult to correct during writing.Source: §5.10 · p. 28

6.1. Implications

• The framework integrates literature retrieval, paper ranking, knowledge extraction, survey generation, and evaluation to improve the efficiency of producing long-form documents.The discussion frames automatic surveys as a way to reduce information overload, particularly for early-stage researchers.Source: §6.1 · p. 28
• Network-analysis and NLP methods can provide landscape views, while RAG and modular-agent systems generate review drafts; SurveyAgent-HKA is positioned as complementary to these approaches.The discussion describes possible integration of topic structures, citation networks, and impact signals into retrieval, ranking, and outline construction.Source: §6.1 · p. 29
• The implementation uses publicly accessible databases and tools, and the authors report that API-based writing incurs almost no annotation or training costs.The discussion also says the dataset and evaluation protocols establish a basis for systematic comparisons across models and domains.Source: §6.1 · p. 29

6.2. Limitations

• The study focuses on textual surveys, uses a relatively small dataset, and leaves applicability across other disciplines, venues, lengths, and emerging areas for further examination.The authors also state that rigorous length- or venue-aware evaluation would require additional matched human surveys and comparable annotations.Source: §6.2 · p. 29
• Some evaluation surveys may have appeared in backbone pretraining, potentially affecting absolute scores, although the authors consider this less likely to fully explain relative differences.The study excluded target surveys and alternative versions from retrieval and outline enhancement; compared methods used the same survey set and backbone models.Source: §6.2 · p. 29
• Generated surveys still lag expert-written surveys in key references, balanced and coherent outlines, cross-paper comparisons, and research insights, so human verification and refinement remain necessary.The authors frame generated surveys as support for literature exploration rather than substitutes for expert-written surveys.Source: §6.2 · p. 29

Conclusion

• Across computer science and biomedical domains, SurveyAgent-HKA outperforms mainstream baselines in citation quality, structural consistency, and content quality.The conclusion also reports that ablation studies confirm contributions from the framework's modules.Source: §7 · p. 30
• The framework retrieves literature across sources, refines topic-based outlines with human-crafted outlines, and re-ranks subtopic papers using citation analysis.It extracts structured knowledge for drafting and uses recurring issues from peer reviews of published surveys to guide iterative revision.Source: §7 · p. 30

Future work

• Future work includes incorporating images and tables, testing larger datasets and additional disciplines, and developing a web-based system for customized surveys.Source: §7 · p. 30

A.1. Survey examples

• The comparison set contains 30 surveys: computer-science examples from AutoSurvey and medical examples from SurveyGen.Source: §A.1 · p. 31

A.2. Peer review examples

• Figure 12 presents two peer-review examples from the dataset.Source: §A.2 · p. 31

• The human evaluation criteria are provided in Table 19.Source: §A.3 · p. 31
• Table 20 presents knowledge extraction trees for papers in Computer Science and Medicine.Source: Table 20 · p. 32

• Knowledge extraction trees capture paper details such as datasets and results that abstracts usually omit.They condense full text into a structured, concise format intended to retain relevant content while reducing token cost for survey generation.Source: §A.4 · p. 32

• SurveyAgent-HKA significantly improves all four key metrics over Naive-RAG, AutoSurvey, and SurveyForge in Computer Science.Against SurveyX, gains are significant for Citation F1 and KPR, while H1(F1) and H2(F1) improvements are positive but not significant.Source: §A.5 · p. 32
• In Medicine, SurveyAgent-HKA significantly outperforms Naive-RAG across all metrics and SurveyGen in structural consistency and KPR.Source: §A.5 · p. 32
• The authors test topic-level paired differences with the Wilcoxon signed-rank test; some stronger-baseline comparisons remain non-significant at the limited evaluation scale.Source: §A.5 · p. 32

A.6. Resource-controlled comparison

• Single-source retrieval and abstract-only evidence lower performance versus the full system, but the controlled version still beats baselines on most metrics.Source: §A.6 · p. 32, Table 22 · p. 33
• In Computer Science's controlled setting, citation F1 is 13.46% for SurveyAgent-HKA versus 13.97% for SurveyForge, while structural consistency and content quality are better.The controlled setting uses single-source retrieval and abstracts only; Table 22 compares controlled and full settings.Source: §A.6 · pp. 32–33, Table 22 · p. 33

A.7. Prompts used in this study

• Prompts combine task instructions, context, input data, and output format, with human-informed observations optionally guiding task instructions.Outline generation requires JSON output and brief section-setting explanations, unlike survey generation.Source: §A.7 · p. 33
• The authors acknowledge prompt variation but place its effects outside the study's scope; the prompts are available in supplementary files rather than the main paper.Source: §A.7 · pp. 33–34