Source-linked AI summary
Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models
Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, Monica S. Lam
TL;DR
Grounded, organized long-form article generation remains difficult because research and outlining must precede writing. STORM automates this pre-writing process through perspective-guided, source-grounded conversations and outline curation, and its articles outperform an outline-driven retrieval baseline in organization and coverage. Expert evaluation also exposes source-bias, verifiability, and unrelated-fact association challenges.
Problem
It remains unclear how LLMs can produce grounded, organized, full-length Wikipedia-like articles, whose pre-writing stage requires research and planning.
Method
STORM discovers diverse perspectives, conducts multi-turn conversations grounded in trusted Internet sources, and uses the collected information to curate an article outline.
Results
STORM improves outline and article quality, with more articles judged organized and broad in coverage than those from an outline-driven retrieval-augmented baseline.
Takeaways & Limitations
Experienced Wikipedia editors unanimously found STORM helpful for their pre-writing stage, while expert feedback surfaced new challenges for grounded long-form writing.
Takeaways & Limitations
Machine-written articles still lag behind well-revised human-authored articles in neutrality and verifiability, and collected information may inherit dominant-source bias or promotional content.
Abstract
from arXiv · showhide
We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplored problem poses new challenges at the pre-writing stage, including how to research the topic and prepare an outline prior to writing. We propose STORM, a writing system for the Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. For evaluation, we curate FreshWiki, a dataset of recent high-quality Wikipedia articles, and formulate outline assessments to evaluate the pre-writing stage. We further gather feedback from experienced Wikipedia editors. Compared to articles generated by an outline-driven retrieval-augmented baseline, more of STORM's articles are deemed to be organized (by a 25% absolute increase) and broad in coverage (by 10%). The expert feedback also helps identify new challenges for generating grounded long articles, such as source bias transfer and over-association of unrelated facts.
1 Introduction
Although LLMs write fluently, it remains unclear how to produce grounded, organized long-form articles such as full Wikipedia pages. Such expository writing requires thorough research and planning before drafting.
- LLMs have demonstrated impressive writing capabilities, but their use for grounded, full-length Wikipedia-like articles remains unclear.
- Expository writing requires thorough research and planning during the pre-writing stage.
Topic
Generating Wikipedia-like articles from scratch requires automating research and outlining, tasks that prior systems often bypass or simplify. STORM addresses this with multi-perspective question asking, trusted-source retrieval, and evaluations using FreshWiki and expert feedback.
- Topic: Prior Wikipedia-generation work often assumes reference documents or an outline, bypassing the difficult pre-writing work of collecting and organizing sources.
- Topic: Direct generation and simple topic retrieval can lack detail, hallucinate, and miss information needed for long-tail topics.
- Topic: STORM discovers diverse perspectives, asks iterative questions, and grounds answers in trusted Internet sources to research a topic before outlining.
- Topic: FreshWiki and outline-quality metrics support evaluation of pre-writing, alongside feedback from experienced Wikipedia editors.
- Topic: Experienced editors found STORM superior to an outline-driven RAG baseline in breadth and organization, while identifying Internet bias and fabricated connections as challenges.
2 FreshWiki
FreshWiki is designed to evaluate long-form grounded writing without relying on training-leaked Wikipedia content. The setup focuses on researching references, constructing multi-level outlines, and measuring their coverage against human-written articles.
- The task asks systems to find references and generate full-length articles whose sentences cite those references.
- FreshWiki selects recent, high-quality Wikipedia articles to reduce data leakage from language-model pretraining.
- The dataset retains plain text while excluding structured and multimodal information to simplify construction.
- Generation is decomposed into pre-writing an outline and writing the article from the topic, references, and outline.
- Heading soft recall and heading entity recall measure how well generated multi-level headings cover those in human-written articles.
3 Method
STORM automates pre-writing by discovering perspectives, conducting grounded multi-turn research conversations, and refining an outline before article generation. The resulting outline and references then guide section-by-section writing.
- 3 Method: STORM automates pre-writing through effective question asking, outline creation, and subsequent expansion into a grounded full-length article.
- Perspective Discovery: STORM discovers perspectives by surveying related Wikipedia articles and uses them to guide parallel question asking.
- Simulating Conversations: In each simulated conversation, a perspective-specific writer asks questions using the topic and conversation history, enabling follow-up questions.
- Simulating Conversations: STORM decomposes complex questions into searches, filters results using Wikipedia guidelines, and synthesizes trustworthy sources into answers and references.
- Creating the Article Outline: The system first generates a topic-only draft outline, then refines it using the draft and simulated conversations before article writing.
- Article Generation: During writing, section and subsection headings retrieve relevant references so the language model can generate the article section by section.
4 Experiments
The experiments evaluate outline and article quality using FreshWiki samples, recall-based outline metrics, article-quality metrics, and LLM-based baselines. STORM is implemented with zero-shot prompting, specified language models, and search grounding.
- Experimental Setup: 100 FreshWiki samples with human-written articles no longer than 3000 words are randomly selected for evaluation.
- Evaluation: Outline quality is assessed with heading soft recall and heading entity recall, where higher recall indicates greater coverage relative to the human-written article.
- Evaluation: Article quality is evaluated using ROUGE, entity recall, five rubric-based aspects, and verifiability.
- Baselines: The baselines are Direct Gen, RAG, and outline-driven RAG, differing in direct generation, topic-level retrieval, and section-level retrieval.
- Implementation: STORM uses zero-shot prompting with DSPy, gpt-3.5-turbo for question asking, gpt-3.5-turbo-instruct for other components, and You.com search grounding.
- Implementation: Final article results use gpt-4 because gpt-3.5 was not faithful to sources when generating cited text.
5 Results and Analysis
STORM produces stronger outlines and full-length articles than the comparison systems, while ablations show that conversation and outlining contribute to performance. Citation support remains imperfect, and the authors identify source bias and over-association as challenges.
- Outline Results: STORM creates more topic-specific, higher-recall outlines than Direct Gen, RAG, and RAG-expand.RAG-expand improves RAG through additional search and refinement, but STORM still performs better.
- Article Results: STORM outperforms oRAG on entity recall and evaluator scores for Interest Level, Relevance and Focus, and Coverage.The authors note that the evaluator LLM may overrate machine-generated text.
- Citation Quality: 84.83% of generated sentences are judged supported by their citations.The remaining unsupported cases primarily involve improper inferences and inaccurate paraphrasing rather than nonexistent content.
- Ablation Studies: Removing perspectives or conversations weakens outline creation, and the full STORM pipeline produces the highest recall among the ablation variants.The no-conversation variant performs much worse, while the full pipeline also discovers more unique sources.
- Ablation Studies: Removing the outline stage significantly deteriorates performance across all metrics.
6 Human Evaluation
Human evaluation by experienced Wikipedia editors found that STORM-generated articles were broader, deeper, and better organized than oRAG outputs, while also revealing verifiability challenges.
- STORM articles exhibited greater breadth and depth than outputs from the oRAG baseline.
- 25% more STORM articles met the Organization rating threshold of 4 or higher than oRAG articles.Editors used a 1–7 rating scale, with values ≥4 indicating good quality.
- 10% more STORM articles met the Coverage rating threshold of 4 or higher than oRAG articles.The comparison involved 20 article pairs, each evaluated by two Wikipedia editors.
- Editors identified low Verifiability in over half of the examined cases where they preferred oRAG, often due to red-herring connections or overspeculation.These issues involve unverifiable links between retrieved facts or between information and the topic, rather than simple factual hallucinations.
- STORM was unanimously judged helpful for editors’ pre-writing stage, while 80% thought it could help edit a Wikipedia article on a new topic.For usefulness to the Wikipedia community overall, 70% agreed and 10% disagreed.
7 Related Works
Prior work has widely used retrieval to ground language-model outputs, but retrieval-augmented generation remains less studied for long-form article writing and question-driven research.
- Retrieval-augmented generation supplies language models with external knowledge at inference time and is used for knowledge-intensive tasks, citation generation, and attributed question answering.
- Despite extensive work on retrieval for question answering, using RAG for long-form article generation remains less investigated.
- RAG systems can vary in both retrieval source and timing, ranging from domain databases or documentation to the whole Internet and from one-time retrieval to repeated retrieval during generation.
- Question-asking research addresses clarification and decomposition, while human learning research frames questions as tools for acquiring new knowledge.
8 Conclusion
The paper introduces STORM to automate pre-writing for Wikipedia-like articles and evaluates it through the FreshWiki dataset and outline-quality criteria. Human editors found the system helpful for pre-writing.
- STORM is an LLM-based system that automates the pre-writing stage for creating Wikipedia-like articles from scratch.
- The authors curate FreshWiki and establish criteria for evaluating both outline and final article quality.
- Automatic and human evaluations show that STORM’s question-asking mechanism improves outline and article quality.
- Experienced Wikipedia editors unanimously agreed that STORM was helpful for their pre-writing stage.
Limitations
The paper’s generated articles still lag behind well-revised human articles in neutrality and verifiability, while Internet-source bias and fabricated connections remain unresolved challenges.
- Machine-written articles lag behind well-revised human-authored articles, particularly in neutrality and verifiability.
- Information collected during research may remain biased toward dominant Internet sources and may include promotional content.
- The identified verifiability problems extend beyond factual hallucination and include fabricated connections between unrelated facts.
- The task setup considers only free-form text, whereas high-quality Wikipedia articles also contain structured data and multimodal information.
Ethics Statement
The work addresses ethical risks in grounded article generation by restricting experiments to offline evaluation and identifying bias in Internet sources as a key concern. It also notes that the study is limited to English Wikipedia and reports safeguards against misinformation and disruption to live Wikipedia pages.
- Ethical safeguards: The studies avoid publishing generated content online, apply strict accuracy checks, and do not interact with live Wikipedia pages.These measures are intended to reduce misinformation dissemination and disruption to Wikipedia communities.
- Source bias: The primary risk is that Internet sources containing biased or discriminatory content can transfer those biases into generated Wikipedia articles.The system relies on search retrieval without a post-processing module for filtering source content.
- Source bias: Improving viewpoint coverage in retrieval and adding content sifting are identified as critical next steps for better neutrality and balance.
- Scope: The study considers only English Wikipedia articles, leaving multilingual extension as a direction for future work.The authors note that more topics lack Wikipedia pages in non-English languages.
- System design: STORM automates pre-writing by discovering perspectives, simulating information-seeking conversations, and creating comprehensive outlines.The framework is implemented as a general grounded long-form writing system rather than a domain-specific prompt.
E Error Analysis
The error analysis finds that STORM articles are preferred by editors but still contain recurring problems, especially emotional language, source-driven lack of neutrality, and unsupported connections between unrelated facts.
- Overall findings: Editors found STORM articles preferable, while still identifying multiple problems in the machine-generated articles.The free-form comments were analyzed to summarize the major issues in Table 11.
- Source bias: The primary issue is emotional language and lack of neutrality caused primarily by non-neutral Internet sources.The authors identify retrieval-source bias as a direction for future research.
- Inferential errors: Another major issue is the red herring fallacy, in which models over-associate unrelated facts.Addressing this challenge requires high-level sensemaking rather than only fact-level verification.