Source-linked AI summary
LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation
Gregory Hok Tjoan Go, Khang Ly, Anders Søgaard, Amin Tabatabaei, Maarten de Rijke, Xinyi Chen
TL;DR
The growth of scientific publications makes timely, comprehensive literature reviews difficult, while automated writing remains limited in readability and factual accuracy. LiRA addresses this gap with a multi-agent workflow that plans, writes, edits, reviews, and grounds citations. Across two datasets, it outperforms tested baselines on most evaluation metrics, with especially strong citation-quality results, while remaining applicable to real-world retrieved references.
Problem
Publication growth makes systematic literature reviews difficult to produce timely, while prior automation focuses mainly on retrieval and screening rather than readable, factual writing.
Source: §1 · p. 1Method
LiRA decomposes literature-review writing into specialized LLM agents for outlining, subsection writing, editing, reviewing, and citation grounding.
Source: §3 · p. 2, §3.5 · p. 3Results
Across both datasets, LiRA outperforms baseline systems on most evaluation metrics, including Citation Quality F1 scores of 0.76 on SciReviewGen and 0.73 on ScienceDirect.
Source: §4.4 · pp. 4–5Takeaways & Limitations
LiRA demonstrates the potential of an out-of-the-box agentic workflow to improve the reliability, writing quality, and practical usability of automated literature reviews.
Source: §1 · p. 1, §7 · p. 6Results may be irreproducible because all experiments use gpt-4o-mini, and the lack of open-source datasets limits generalizability across scientific fields.
Source: §7 · p. 6
Abstract
from arXiv · showhide
The rapid growth of scientific publications has made it increasingly difficult to keep literature reviews comprehensive and up-to-date. Though prior work has focused on automating retrieval and screening, the writing phase of systematic reviews remains largely under-explored, especially with regard to readability and factual accuracy. To address this, we present LiRA (Literature Review Agents), a multi-agent collaborative workflow which emulates the human literature review process. LiRA utilizes specialized agents for content outlining, subsection writing, editing, and reviewing, producing cohesive and comprehensive review articles. Evaluated on SciReviewGen and a proprietary ScienceDirect dataset, LiRA outperforms current baselines such as AutoSurvey and MASS-Survey in writing and citation quality, while maintaining competitive similarity to human-written reviews. We further evaluate LiRA in real-world scenarios using document retrieval and assess its robustness to reviewer model variation. Our findings highlight the potential of agentic LLM workflows, even without domain-specific tuning, to improve the reliability and usability of automated scientific writing.
1 Introduction
Literature reviews, especially systematic reviews, are increasingly difficult to produce as publication volume grows, while automation has focused mainly on retrieval and screening rather than readable, factual writing. LiRA addresses this gap with an out-of-the-box agentic workflow for accurate, readable review generation.
1 Introduction
- Growing publication volume and expert-labor requirements make timely systematic literature reviews increasingly difficult to produce.Existing automation research largely targets retrieval and screening, leaving the task of compiling comprehensive reviews comparatively under-explored.Source: §1 · p. 1
- Only limited prior work automates literature-review writing, particularly work addressing readability and hallucination mitigation.Source: §1 · p. 1
- LiRA proposes an agentic LLM workflow that emulates human writing, incorporates literature-review guidelines, and requires no task-specific pre-training or fine-tuning.The workflow also targets reduced hallucination through citation-grounded generation.Source: §1 · pp. 1–2
- The study evaluates similarity to human-written reviews, writing quality, citation use, and real-world performance with retrieved references.Source: §1 · pp. 1–2
2 Related Work
Prior work establishes the importance and cost of systematic literature reviews but offers limited automation for writing, readability, factuality, and real-world usability. LiRA responds with an agentic workflow focused on these underdeveloped aspects.
Agentic workflows
- Agentic workflows decompose complex tasks into smaller steps and have shown promise in fields including medicine and law.Related work compares this decomposition with aspects of human cognition.Source: §2 · p. 2
- Automated writing research remains sparse, and existing open-source literature-writing systems do not adequately address output length, readability, or factuality.Source: §2 · p. 2
Literature review
- Systematic literature reviews reduce personal bias through systematic collation and synthesis but remain time-consuming and costly, motivating automated alternatives.Source: §2 · p. 2
- LiRA addresses these gaps by using LLM agents to generate literature reviews automatically while evaluating readability and factuality.Source: §2 · p. 2
3 The LiRA Framework
LiRA decomposes literature-review writing into specialized agents for planning, subsection generation, editing, review, and citation grounding. The resulting pipeline combines modular generation with iterative quality control and structured inter-agent communication.
3 The LiRA Framework
- LiRA emulates the human literature-review process through specialized agents for structural planning, writing, refinement, and factual verification.This decomposition produces a modular and scalable pipeline.Source: §3 · p. 2
3.1 Outline Drafter Agent
- The outline drafter builds unified section and subsection structures from reference abstracts or full texts, using suggested supporting papers to guide later writing.It considers up to 50 references and can adapt an approximately eight-section, four-subsection heuristic.Source: §3.1 · p. 2
3.2 Subsection Writer Agent
- The subsection writer generates sections in parallel from descriptions and retrieved references, producing modular long-form synthesis rather than a single-pass review.References are selected with FAISS, and the article title, abstract, and conclusion are written after the body.Source: §3.2 · p. 3
3.3 Editor Agent
- The editor improves transitions, vocabulary, fluency, and cohesion across the assembled draft while preserving its factual content.If the output reaches the context limit, continuation is prompted using the original input and edited content.Source: §3.3 · p. 3
3.4 Reviewer Agent
- The reviewer evaluates intermediate outputs against completeness, transparency, clarity, and contribution criteria, triggering regeneration when quality thresholds are not met.The review loop runs for up to three rounds before fallback progression.Source: §3.4 · p. 3
3.5 Citation Behavior
- Citation grounding uses full article titles as semantic anchors during generation, then converts them into numbered references and redacts hallucinated titles during evaluation.This design is intended to improve factual consistency and support fair comparison.Source: §3.5 · p. 3
3.6 Additional Implementation Details
- LiRA implements agent memory, parallel processing, structured documents, shared message pools, context filtering, and selective Zero-Shot Chain-of-Thought prompting.These mechanisms support feedback use, communication efficiency, and processing within model context limits.Source: §3.6 · p. 3
4 Experiments
LiRA is evaluated against direct prompting, MASS-Survey, and AutoSurvey across similarity, writing quality, citation reliability, and datasets spanning computer science and multiple other fields. Across both datasets, LiRA generally produces concise, coherent reviews with stronger citation reliability and competitive coverage.
4.1 Baselines
- The experiments evaluate direct prompting, MASS-Survey, and AutoSurvey as baselines under a shared gpt-4o-mini backbone.Direct prompting tests single-pass generation; MASS uses sequential clustering and drafting; AutoSurvey retrieves references, constructs outlines, and refines subsections.Source: §4.1 · pp. 3–4
4.2 Metrics / 4.3 Datasets
- Evaluation covers content similarity to human-written reviews, writing quality through automatic and human assessments of coverage, structure, and relevance, and citation reliability using CQF1, which balances precision and recall.SciReviewGen contains 10,000 computer-science reviews, with a randomly sampled 125-review evaluation set; human assessments use 30 sample snippets for preferences and 15 article snippets for 1-to-5 scoring against human-written baselines.Source: §4.2 · p. 4, §4.3 · p. 4
4.4 Results
- LiRA consistently outperforms baseline systems on most evaluation metrics across SciReviewGen and ScienceDirect.The comparison covers alignment with human-written reviews, writing quality, and citation reliability.Source: §4.4 · p. 4
- LiRA achieves the highest ROUGE scores, while AutoSurvey’s slightly higher heading/entity recall coincides with outputs averaging 50,000 tokens versus LiRA’s 22,000.Because recall metrics do not normalize for length, the results support LiRA’s concise, information-dense output rather than a verbosity advantage.Source: §4.4 · pp. 4–5
- LiRA achieves the best overall writing quality and structural coherence, while AutoSurvey is marginally stronger in coverage but weaker in organization and readability.Expert evaluations also favored LiRA for structure on ScienceDirect and, on SciReviewGen, over human-written reviews given the outlines.Source: §4.4 · pp. 4–5
- LiRA records the highest Citation Quality F1 scores on both datasets: 0.76 on SciReviewGen and 0.73 on ScienceDirect, versus AutoSurvey at ≤0.63.The metric balances citation precision and recall, capturing irrelevant or hallucinated citations as well as missing necessary citations.Source: §4.4 · p. 5
5 Potential Modifications on LiRA
LiRA is tested with an alternative reviewer model and with retrieved rather than gold references to assess robustness and real-world applicability.
5 Potential Modifications on LiRA
- The modifications section evaluates alternative reviewer models and document retrieval as robustness checks for LiRA deployment.These tests address potential reviewer self-bias and operation without gold references.Source: §5 · p. 5
5.1 Using a different reviewer model
- Using gemma3:4b as a different reviewer model had little overall impact across metrics, suggesting the pipeline may work with varied LLM configurations.The study notes that bias mitigation itself cannot be properly measured with the current metrics.Source: §5.1 · p. 6
5.2 Retrieval Usage
- Retrieved references produced results similar to the baseline researcher setting, with only two statistically significant differences.The retrieval experiment used embedding-similarity search to obtain as many references as the human-written review listed.Source: §5.2 · p. 6
6 Deployment
LiRA’s planned deployment uses LangGraph with gpt-4o-mini and adds embedding-based retrieval so users can generate reviews from topic queries.
6 Deployment
- Deployment will use Python LangGraph and gpt-4o-mini from AzureOpenAI, while allowing other supported model endpoints.The paper specifically mentions ollama as another possible endpoint.Source: §6 · p. 6
- A user’s review topic will be enriched by an LLM and used for embedding-based retrieval from a replaceable scientific-article collection.The retrieval system is intended for novel reviews that are not based on existing reviews.Source: §6 · p. 6
7 Conclusion, Limitations, and Future Work
LiRA integrates research-before-writing and refinement, performs well across the evaluated automated-review task, and shows improved citation behavior in real-world use.
7 Conclusion, Limitations, and Future Work
- LiRA outperformed all tested open-source methods across varying output lengths and reduced hallucination through improved citation behavior.The conclusion characterizes these findings as positive across the proposed research questions.Source: §7 · p. 6
- Using gpt-4o-mini throughout the experiments makes the results potentially irreproducible, motivating evaluation with seedable models.The paper identifies this as a main area for improvement.Source: §7 · p. 6
- LiRA does not yet cover primary-study handling or risk-of-bias assessment in randomized trials, limiting the completeness of end-to-end review pipelines.The paper suggests integrating screening and search-criteria definition in future work.Source: §7 · p. 6
A Metrics Measuring the Similarity with Human-Written Articles
The paper uses textual similarity and entity-recall metrics to compare generated reviews with human-written articles, covering both structure and scientific content.
A Metrics Measuring the Similarity with Human-Written Articles
- ROUGE-L measures recalled units from a human-written reference, while heading soft recall measures similarity between predicted and gold headings.Heading soft recall uses embedding-based similarity rather than exact heading matches, with higher scores indicating more comprehensive outlines.Source: A Metrics Measuring the Similarity… · pp. 9–10
- Heading and article entity recall measure the proportion of named entities from the gold review reproduced in generated headings or full text.The implementation uses SciSpacy’s en_core_sci_lg model to handle scientific terminology.Source: A Metrics Measuring the Similarity… · p. 10
B LLM Evaluation
LLM evaluation scores generated reviews for coverage, structure, and relevance, while a separate SME procedure compares human- and LiRA-written samples using article snippets and outlines.
B LLM Evaluation
- Prometheus-7b-v2.0 evaluates generated reviews three times, averaging scores to account for evaluator randomness and using fixed generation settings.Coverage and relevance are evaluated section by section, with further splitting when paired inputs exceed the context budget.Source: B LLM Evaluation · p. 10
- Structure is judged from article outlines, while coverage, structure, and relevance receive rubric-based scores from 1 to 5.Reported results are means of averaged scores and averaged standard deviations across the repeated runs.Source: B LLM Evaluation · p. 10
C Sample SME Evaluation
- SME evaluation presents titles, article snippets, and outlines so experts can choose between human-written and LiRA-written articles.The SciReviewGen evaluation used three SMEs to assess preferences for 30 sample snippets.Source: C Sample SME Evaluation · p. 10
D Citation Quality
Citation quality is measured by LLM-based entailment between article claims and cited reference abstracts, producing recall, precision, and a length-adjusted F1 score.
D Citation Quality
- For each claim, an NLI model checks whether the listed references support it, then aggregates these judgments into citation recall and precision.The metric also evaluates whether individual references are related to particular claims.Source: D Citation Quality · p. 10
- The citation metric applies a scaling factor to recall because longer articles contain more claims and require more citation relationships to be tracked.The authors describe the effort as roughly exponential in the number of claims, penalizing articles that are too short.Source: D Citation Quality · p. 10
- The scaling factor uses k = 0.01 and the average number of claims per article, with the resulting adjusted recall contributing to the final F1 score.The scaling term asymptotically approaches 1 as the claim count becomes large.Source: D Citation Quality · pp. 10–11
- Only reference abstracts are evaluated, and overly long claim-reference pairs are split so that support can be detected in any segment.This setup is intended to resemble how subject matter experts perform the task, but assumes the remaining reference content is also supported when one split is.Source: D Citation Quality · p. 11
E Ablation and Modification
The paper tests design modifications and retrieval-setting robustness, finding modestly negative effects from added agents while documenting nonparametric significance testing for retriever comparisons.
E Ablation and Modification
- Figures 4 and 5 report the editor ablation and researcher-agent modification using textual similarity, citation quality, and Prometheus evaluation results.The editor ablation was motivated by the expectation that editing is less impactful than most other system agents.Source: E Ablation and Modification · p. 11, Figure 4 · p. 11, Figure 5 · p. 12
- Adding a researcher agent that extracts key points and limitations from reference abstracts tends to reduce overall performance by marginal amounts.The paper suggests the abstracts may already contain sufficient information, limiting the benefit of additional processing.Source: E Ablation and Modification · p. 11
- Removing the editor generally improves final results slightly, although retaining it modestly improves article coverage and relevance.The authors characterize this as a trade-off between article quality and similarity.Source: E Ablation and Modification · p. 11
F Retrieval Setting Statistical Tests
- Retriever-setting comparisons use two-tailed Mann–Whitney U-tests for ROUGE and recall metrics because the results were generally non-normal.Prometheus results use the same test after averaging runs by sample; normality checks use Shapiro–Wilk and α = 0.05.Source: F Retrieval Setting Statistical Tests · pp. 11–12, Table 6 · p. 12
- Table 6 presents statistical significance tests for the retriever experiment, while Table 7 provides the rubric criteria and descriptions used for evaluation.The rubric is adapted from Wang et al. (2024).Source: Table 6 · p. 12, Table 7 · p. 13