Source-linked AI summary
Instruct Large Language Models to Generate Scientific Literature Survey Step by Step
Yuxuan Lai, Yupeng Wu, Yidan Wang, Wenpeng Hu, Chen Zheng
TL;DR
Scientific literature surveys are time-consuming to produce and difficult to generate coherently within LLM output and API-cost constraints. The paper uses step-by-step prompting, and its Qwen-long system placed third in the NLPCC 2024 evaluation, 0.03% behind the second-place team.
Problem
Scientific surveys take increasing researcher time as publications grow, and their length, coherence demands, and information volume challenge LLM generation.
Source: §1 · pp. 1–2Method
A sequence of prompts generates the survey title, abstract, hierarchical headings, and content, with content conditioned on the generated structure.
Source: §1 · p. 2, §3.2 · p. 4Results
Third place in the NLPCC 2024 task came with an overall score 0.03% below the second-place team.
Source: §5 · p. 11Takeaways & Limitations
The method’s reported generation expense is 0.1 RMB per survey, which the authors say enhances its practical value.
Source: §1 · p. 3Without reference-paper content, LLM hallucinations prevent the method from ensuring the accuracy and reliability of generated citations and analysis.
Source: §4.4 · p. 11
Abstract
from arXiv · showhide
Abstract. Automatically generating scientific literature surveys is a valuable task that can significantly enhance research efficiency. However, the diverse and complex nature of information within a literature survey poses substantial challenges for generative models. In this paper, we design a series of prompts to systematically leverage large language models (LLMs), enabling the creation of comprehensive literature surveys through a step-by-step approach. Specifically, we design prompts to guide LLMs to sequentially generate the title, abstract, hierarchical headings, and the main content of the literature survey. We argue that this design enables the generation of the headings from a high-level perspective. During the content generation process, this design effectively harnesses relevant information while minimizing costs by restricting the length of both input and output content in LLM queries. Our implementation with Qwen-long achieved third place in the NLPCC 2024 Scientific Literature Survey Generation evaluation task, with an overall score only 0.03% lower than the second-place team. Additionally, our soft heading recall is 95.84%, the second best among the submissions. Thanks to the efficient prompt design and the low cost of the Qwen-long API, our method reduces the expense for generating each literature survey to 0.1 RMB, enhancing the practical value of our method.
1 Introduction
The system uses staged prompting to generate survey structure and content, addressing long-output, coherence, and API-cost challenges. With Qwen-long, it placed third in the NLPCC 2024 task and generated surveys for 0.1 RMB each.
1 Introduction
- Long surveys exceed single-pass output limits, while sequential generation from all prior content risks forgetting earlier information or losing structural consistency.Transmitting and generating the large amount of survey information can also make LLM API usage expensive.Source: §1 · p. 2
- The method prompts an LLM to generate the title, abstract, hierarchical headings, and main content sequentially, then conditions content on the generated headings.The authors say this lets headings retain a high-level perspective and reduces API usage compared with conditioning on all previous content.Source: §1 · p. 2
- The system placed third in NLPCC 2024 Scientific Literature Survey Generation, scoring 61.11 overall, just 0.03% behind the second-place team; its soft heading recall was 95.84%, the second highest among all submissions.Source: §1 · p. 2
- 0.1 RMB per survey is the reported generation expense, which the authors attribute to efficient prompts and the low-cost Qwen-long API.The paper presents this low cost as enhancing the method’s practical value.Source: §1 · p. 3
2 Related Works
Prior work on automatic literature surveys was limited, while related approaches explored retrieval, long-context modeling, and outlines for long-form generation.
2 Related Works
- AutoSurvey [7] equips LLMs with initial retrieval and outline generation for automatic literature surveys, against a background of few systematic methodologies for the task.The paper also notes surveys claimed to be written with LLM assistance [8].Source: §2 · p. 3
- The paper addresses the NLPCC 2024 survey-generation task; related work explored long-context attention [11], while studies [13] [14] report that such models may not fully use extensive context.Source: §2 · p. 3
- Outline-based approaches used detailed outlines for long-story generation [15] and a retrieval-augmented pipeline for long blogs [16]; this paper likewise uses outlines for survey generation.Source: §2 · p. 3
3 Methodology
The method takes reference papers and key subjects as inputs and produces a structured survey. It plans the survey in stages, then generates section content using selected references and the resulting outline.
3.1 Task Formulation
- Given reference papers and key subjects, the task is to produce a survey with a title, abstract, and main body organized into heading–content pairs.Source: §3.1 · p. 4
3.2 Overview
- The six-step pipeline has two phases: outline generation produces title, section headings, and abstract in sequence; subsection and content generation then uses the outline and selected references.The first phase conditions each step on prior outputs; the second keeps each subsection aware of the overall structure while reducing input length and API cost.Source: §3.2 · pp. 4–5
3.3 Prompt Design
- Title generation uses the subject and references; section headings follow as a 6–10-heading outline, and the abstract is generated from that title and outline.If the initial heading count falls outside 3–25, the system regenerates the headings; it does not require strict adherence to the suggested 6–10 range.Source: §3.3 · pp. 5–6
- Before drafting, the system selects a subset of references for each section to reduce API costs and avoid using the same reference group across sections.The selected references are used to generate subsection headings and content for the relevant section.Source: §3.3 · p. 6
- For each section, subsection headings use its title, outline, and selected references; content prompts provide the full outline and relevant references, and repeat the subsection title to maintain focus.The authors note that lengthy prompts can distract the LLM from generating the requested subsection content.Source: §3.3 · pp. 6–7
4 Experiments
Experiments evaluate the step-by-step survey-generation system on the NLPCC 2024 task, reporting its benchmark placement, evaluation measures, and case-study performance. They also document practical constraints, including citation reliability and outline length.
4.1 Dataset
- The dataset contains 500 training and 200 test arXiv surveys; no separate validation set was used, and training data informed prompt design only.The authors did not use training data for parameter tuning or in-context learning.Source: §4.1 · pp. 7–8
4.2 Implement Detail
- Qwen-long generation cost about 0.10 RMB per survey; long reference lists were truncated to 80–100 references, with odd/even selection when content triggered refusal.Generating all 200 test surveys cost 20.86 RMB.Source: §4.2 · p. 8
4.4 Evaluation Result
- The system placed third, with an overall score 0.03% below the second-place team; its 95.84% soft heading recall ranked second, trailing the leading team by 1.17%.Source: §4.4 · p. 9
4.3 Evaluation Metric
- The overall score averages ROUGE1, ROUGE2, and ROUGEL, Soft Heading Recall, and human scores for fluency, structure, citations, relevance, and analytical scope.Soft Heading Recall evaluates survey structure using generated and reference chapter titles; human evaluation considers five listed aspects.Source: §4.3 · pp. 8–9
4.4 Evaluation Result
- The XAI comparison shows a broader generated title and outline, abstract coverage of GANs, RNNs, and healthcare, and expanded ethical-accountability discussion [57].The generated content discusses privacy, bias, and algorithmic accountability in healthcare.Source: §4.4 · p. 10
- Without reference-paper content, hallucinations weaken human evaluation, and the method cannot ensure citation and analysis accuracy; incorporating that content is proposed as a remedy.The authors identify factual accuracy and reliability as targets for future improvement.Source: §4.4 · p. 11
4.5 Case Study
- 97.78% soft heading recall in the example accompanies an outline judged too lengthy and detailed, prompting a proposal for future concision instructions.The example outline captures explainable AI, saliency maps, black-box models, ethics, and future directions.Source: §4.5 · p. 11
- The example captures common abstract and content points with the reference, but some citations remain hallucinated and may be mitigated with reference-paper content.The authors describe the generated abstract and citations as appropriate while noting this remaining issue.Source: §4.5 · p. 11
5 Conclusion
The paper presents step-by-step prompting for structured, relevant scientific literature surveys; its Qwen-long system placed third in the NLPCC 2024 evaluation, while citation and content reliability remain challenges.
5 Conclusion
- The Qwen-long system placed third in the NLPCC 2024 evaluation, with an overall score 0.03% below the second-place team.Source: §5 · p. 11
- The approach uses a series of instructions to generate well-structured and relevant survey content step by step.Source: §5 · p. 11
- Citation and content accuracy and reliability remain challenges because of LLM hallucinations and the absence of reference content.Source: §5 · p. 11