PaperlayerView exact revision on arXiv

Source-linked AI summary

Wyvern: An Agentic Framework for Generating Grounded Multimodal Reports

Beatrice Alessandra Motetti, Emilien Guandalino, Daniele Jahier Pagliari, Alessio Burrello, Lorenz K. Müller, Konstantin Berestizshevsky, Lukas Cavigelli

arXiv:2608.14446v1cs.AI

TL;DR

Generative tools can help synthesize rapidly growing information, but often lack grounding. Wyvern uses coordinated agents to generate grounded multimodal technical reports, whose usefulness exceeded alternatives in 63% to 100% of instances and whose citation recall improved up to 2.3×.

  • Problem

    Rapidly growing heterogeneous information makes it difficult for researchers to stay informed, while generative synthesis tools often lack information grounding.

  • Method

    Wyvern coordinates specialized LLM-based agents to generate grounded multimodal technical reports with figures and claims auto-revision against collected evidence.

  • Results

    Wyvern reports were perceived as more useful than alternatives in 63% to 100% of instances, with citation recall improving up to 2.3× over considered methods.

  • Takeaways & Limitations

    Within the evaluated comparisons, Wyvern combined higher perceived report usefulness with improved citation recall.

  • Takeaways & Limitations

    Wyvern depends on search APIs whose results may not comprehensively cover available information and may vary with query time, location, and indexing policies.

Abstract

from arXiv · show

In the current artificial intelligence-driven innovation era, the pace of knowledge growth is accelerating, and is hard to keep up with. While generative models are increasingly used to synthesize content, they often lack in information grounding. To address these peculiarities of our time, we propose Wyvern, a multi-agent framework for the automated generation of grounded, multimodal technical reports. Wyvern allows for the generation of multimodal outputs, integrating images, tables, and text with supporting references in a unified report. Additionally, a particular focus is placed on the grounding of the content, with the implementation of a claims auto-revision stage. We conduct a human evaluation study to assess the quality of our proposed framework. The results show that the figures' informativeness is perceived as superior to that of a recent baseline in 87% of cases. Furthermore, Wyvern's reports are rated as more useful than those produced by three alternative methods in 63% to 100% of instances. We also carry out automatic evaluations showing that Wyvern gains up to 2.3$\times$ in citation recall and 1.6$\times$ in citation precision with respect to the baselines.

1 Introduction

Wyvern is a multi-agent framework for generating grounded, multimodal technical reports from web-retrieved information. Its coordinated specialized-agent workflow includes informative-image selection and claims auto-revision to improve content reliability and verify textual grounding, and it is evaluated through human and automatic comparisons.

  • Framework and motivation: Its specialized LLM-based agents each perform narrow tasks, coordinating retrieval, processing, report production, and evidence verification.This structured workflow is intended to enhance reliability, modularity, and overall content quality.
  • Evaluation: Human evaluation compares Wyvern with STORM, WebThinker, and WikiAutoGen, while automatic evaluation examines the predictive power of evaluation methods relative to human judgment.The reported human assessment covers figures and factuality in generated reports.
  • Framework and motivation: Wyvern generates grounded, multimodal technical reports on user-selected topics by retrieving relevant information from the web.The framework is designed as a multi-agent system for automatic report generation.
  • Key contributions: Wyvern includes a module for retrieving, selecting, and positioning informative images within the textual report.The image component supports the framework’s multimodal output and aims to increase report informativeness.
  • Key contributions: A claims auto-revision stage verifies whether textual statements are grounded in the collected evidence.The verification process checks statements against previously retrieved information.

2 Related Work

Related work combines retrieval and agentic planning for long-form writing, while emerging systems add multimodal content and claim-level factuality verification. Wyvern builds on these directions by unifying multimodality with explicit claims verification and revision, and by evaluating citation quality over atomic claims.

  • Long-form generation: Retrieval-Augmented Generation and web search address LLMs’ limited context and static parametric knowledge in long-form text generation.These strategies support more up-to-date and comprehensive information retrieval.
  • Long-form generation: Long-form expository systems use either top-down outline expansion or dynamic planning based on recursive task decomposition.STORM and Co-STORM use multi-perspective agents for retrieval and outline planning, while AutoSurvey targets literature surveys.
  • Multimodal reports: Multimodal report generators incorporate retrieved images or structured chart representations into textual reports.WikiAutoGen retrieves web images at agent-proposed positions, whereas Multimodal DeepResearcher generates visualizations from LLM-produced JavaScript code.
  • Factuality evaluation: Factuality research verifies generated text through source entailment or external ground-truth answers, using sentence-level or atomic-fact claim units.FActSCORE, Self-Checker, and Chain-of-Verification represent different approaches to decomposing and verifying claims.
  • Factuality evaluation: Wyvern defines citation quality over atomic claims derived from full sentences, avoiding cases of partial support.This differs from sentence-level citation recall and precision used by Gao et al. (2023).

3 Proposed framework

Wyvern is a sequential, multi-agent framework that searches and filters web sources, generates report text and visual elements, and performs a final groundedness review. Its grounding stage decomposes the report into atomic claims and revises or removes statements unsupported by cited references.

  • Framework overview: Wyvern organizes report generation into sequential modules for search, report generation, and grounding, with each phase handled by ensembles of agents.The report-generation module integrates text with informative figures and an overview table, while the grounding module verifies claims against cited references.
  • Search module: The search phase builds a knowledge database of references and summaries from user-specified topics, generated queries, retrieved webpages, and quality filters.Documents are parsed into Markdown, filtered by token length and completeness, deduplicated, summarized, and checked for semantic outliers.
  • Report generation module: The text-generation pipeline creates multiple draft outlines, merges them into a final outline, and expands sections simultaneously using reference summaries and supporting citations.Section agents select salient content, write subsections, and concatenate the expanded sections into the report draft.
  • Report generation module: Wyvern selects and places figures by generating textual descriptions, ranking candidate images against the report outline, and filtering them again using the relevant section text.The figure pipeline can operate hierarchically when descriptions exceed the LLM context length.
  • Report generation module: The framework enriches reports with a comparative overview table that summarizes shared and informative aspects across the reference set and assigns it to an appropriate outline section.Reasoning agents extract differentiating concepts, construct the table with an explanation, and determine its placement.
  • Grounding module: The final groundedness enhancement decomposes report text into atomic claims and independently reviews each claim, revising or removing statements unsupported by cited references.This stage is performed by an ensemble of agents after text and visual content have been incorporated.

4 Experiments

Wyvern is evaluated against STORM, WebThinker, and WikiAutoGen through human and automated assessments of report quality and citation verifiability. The human study uses randomized pairwise comparisons, while automated evaluation examines established report-quality rubrics and citation recall and precision.

  • Baselines: Wyvern is compared with STORM, WebThinker, and WikiAutoGen using the Serper retriever and reproducibility-oriented model choices.STORM’s proprietary models are replaced with DeepSeek-V3, while WebThinker uses WebThinker-QwQ-32B and Qwen2.5-32B-Instruct.
  • Human evaluation: The human evaluation involves 27 computing-systems researchers, engineers, and Ph.D. students who compare Wyvern reports with baseline reports on selected topics.Each reviewer receives one Wyvern report and one baseline report, yielding nine distinct comparisons with each baseline.
  • Human evaluation: Pairwise human assessments cover structure, relevance, coverage, presentation, figure and table informativeness, engagement, and usefulness.The questionnaire also includes absolute grading, alongside relative assessments designed to compare Wyvern directly with baselines.
  • Automated evaluation: Citation recall and citation precision are used to measure the verifiability and relevance of report citations.Citation recall counts supported claims, whereas citation precision evaluates whether cited references are relevant to the associated claims.
  • Automated evaluation: Automated evaluation uses pairwise comparisons under rubrics covering interest, coherence and organization, relevance and focus, and broad coverage.Both report-order permutations are tested to assess their effect on evaluator-model outcomes.

5 Results

Wyvern outperforms baselines in human-assessed usefulness and achieves strong factuality and citation-grounding scores, with claims revision improving citation recall. Automatic evaluation correlates imperfectly with human judgments and remains vulnerable to ordering bias.

  • Human evaluation: Wyvern reports were judged more useful than STORM’s, WebThinker’s, and WikiAutoGen’s in 100.00%, 62.50%, and 87.50% of cases, respectively.Wyvern was qualitatively better on all other considered aspects.
  • Human evaluation: 4.25 out of 5 was Wyvern’s highest absolute score, for the overview table, followed by 4.22 for factuality.Engagement received the lowest overall score, at 3.65.
  • Citation grounding: 73.79% citation recall exceeded STORM and WikiAutoGen by 21.50 and 42.34 percentage points, respectively.Citation recall reached 94.64% when paragraphs without citations were excluded.
  • Citation grounding: 60.92% to 73.79% was the increase in citation recall after applying the claims revision stage.Both citation recall and precision improved significantly after claims revision.
  • Automatic evaluation: Automatic evaluation achieved good correlation with human scores for STORM and WikiAutoGen but failed to capture some human-assessment trends.The evaluation used DeepSeek-V3 and Qwen3-32B across four qualitative rubrics.
  • Automatic evaluation: 36.11% and 30.56% of comparisons showed ordering bias for DeepSeek-V3 and Qwen3-32B, respectively.This indicates substantial room for improving robustness in automated report evaluation.

6 Conclusions

Wyvern is a multi-agent framework for generating grounded, multimodal technical reports on given topics. Its modular agent ensembles and final evidence-verified atomic-claim decomposition improve citation recall by up to 2.3× over considered methods.

  • Contributions: Wyvern enables grounded, multimodal technical report generation on given input topics.It is presented as a multi-agent framework.
  • Contributions: Up to 2.3× citation recall improvement comes from decomposing text into atomic claims verified against collected evidence.The final verification stage is integrated into the report-generation pipeline and compared with considered methods, including STOR.

Limitations

Wyvern’s reports depend on web search APIs and retrieved webpage quality, limiting information coverage and reproducibility. The framework was evaluated only in English, while multilingual generalization remains for future study.

  • Web retrieval dependence: Wyvern depends on auxiliary search APIs and the quality of webpages they retrieve to generate grounded multimodal reports.The framework uses source-filtering and selection pipelines, but ultimately relies on search-engine results and generates multiple distinct queries for broad topic coverage.
  • Web retrieval dependence: Search queries and API reliance do not always ensure comprehensive, optimal information coverage.Using multiple distinct queries seeks broad coverage, but does not guarantee that available information is comprehensively or optimally retrieved.
  • Reproducibility: Search-engine results are not fully reproducible because they can vary with inquiry time, location, and evolving indexing policies.This limitation follows from the inherent characteristics of search APIs.
  • Multilingual generalization: Wyvern was evaluated exclusively on English-language topics, and English retrieval queries can penalize, though not necessarily exclude, resources in other languages.A translator module could translate non-English topics into English, but multilingual generalization remains to be analyzed in future work.

Ethical considerations · A Human evaluation · A.1 Relative assessment

Wyvern’s evaluation combines ethical caveats about source and model bias with a relative human assessment against baseline reports. The study used randomized pairwise comparisons, and Wyvern was systematically preferred over STORM while showing mixed results against WebThinker.

  • Ethical considerations: Wyvern may inherit systematic biases and scientific inaccuracies from web sources and ethical misalignment from open-source LLMs.The framework currently lacks an explicit mechanism to filter or correct biased or erroneous information.
  • Ethical considerations: Wyvern does not amplify biases or inaccuracies beyond their presence in the original sources.
  • A Human evaluation: The human-evaluation appendix reports the complete unaggregated questions and results for both relative assessment and absolute grading.
  • A Human evaluation: All evaluators consented to publication of aggregated anonymous response statistics and portions of the generated reports.
  • A.1 Relative assessment: The relative assessment gave each participant one Wyvern report and one baseline report without revealing their generation methods.The 27 participants were divided into three groups of 9, with report order and naming randomized to avoid positional bias.
  • A.1 Relative assessment: Wyvern was systematically preferred over STORM on all considered rubrics.
  • A.1 Relative assessment: 37.50% was Wyvern’s favor rate for engagement against WebThinker, where WebThinker otherwise obtained a higher preference rate only for structure and engagement.The reported structure concerns included redundancy across sections and excessive overall length.

A.2 Absolute grading … B.2 Absolute grading

The supplementary evaluations detail Wyvern’s human absolute-grading results and automatic assessments, including strong report-layout and neutrality scores. They also reveal substantial evaluator-ordering bias and model-dependent rankings, complicating robust quantitative interpretation.

  • A.2 Absolute grading: Absolute grading provides a detailed rubric-level assessment of Wyvern’s report quality using the same rubrics as the relative assessment.Participants evaluated reports without knowing their generation methodology, receiving only a generic report index.
  • A.2 Absolute grading: 4.52 and 4.50 are the highest average scores, covering figure positioning and overview-table positioning, respectively.Rubric (7), concerning tone neutrality, follows with an average of 4.48.
  • A.2 Absolute grading: Table 4 aggregates scores from 23 completed questionnaires on a [1,5]-range, with selected rubric endpoints labeled by qualitative descriptors.Rubrics (6), (7), and (14) used adjective pairs spanning clarity, neutrality, and satisfaction.
  • A.2 Absolute grading: 3.65 is the lowest average score, measuring the technical report’s engagement and thought-provoking level on rubric (15).The report’s tone neutrality receives an average score of 4.48, described as satisfactorily neutral.
  • B Automatic evaluation: Automatic evaluation uses four criteria: interest level, coherence and organization, relevance and focus, and broad coverage, under relative and absolute setups.The relative-assessment results are reported using Prometheus 2-7B as evaluator model.
  • B.1 Relative assessment: 50.00% is Prometheus 2-7B’s preference rate for every method and criterion because it consistently selects the first report as best.This behavior persists when both report orderings are included in the evaluation.
  • B.2 Absolute grading: Automatic absolute grading gives one report to an evaluator, which assigns [1,5] scores for four criteria using four evaluator models.The models are Prometheus-13B, Prometheus 2-7B, Qwen3-32B, and DeepSeek-V3.

B.3 Claims evaluation

The study assessed whether LLM-based automatic evaluation can reliably judge claim grounding by comparing LLM judgments with human annotations. On 60 randomly sampled claims, the two agreed at a high rate.

  • Claims evaluation: 93.33% agreement was observed between LLM judgments and human annotations across 60 randomly sampled claims.The comparison evaluated whether automatic means could effectively assess grounding.

C Cost analysis

Wyvern’s cost analysis breaks down token usage across its modules and agents, while noting that resource parsing can substantially increase end-to-end latency for large, figure-rich document collections.

  • Token statistics were computed on 6 reports using DeepSeek-V3 as the base model and DeepSeek-R1-0528 as the reasoning model.
  • Agents 2–7 in the search module consume approximately equal input-token counts because each analyzes the collected documents individually.
  • Agents 11 and 18 use high input and completion token counts because they are responsible for the report’s tex…
  • Resource parsing can significantly increase end-to-end latency when processing many long documents containing numerous figures.Parsing falls outside the token-usage analysis.

D Failure cases

Despite positive overall evaluation, evaluators identified weaknesses involving report structure, multimodal figure handling, and claim grounding. Some grounding errors originated from inaccuracies in cited sources rather than Wyvern itself.

  • Overall assessment: The human evaluation study identified the main weaknesses and directions for improvement despite the framework’s overall positive evaluation.The study served both as an evaluation of the framework and as a means of identifying failure cases.
  • Report structure: Evaluators noted excessive length, redundancy, fragmented structure, and an uneven breadth–depth balance that overemphasized peripheral topics.These issues were identified by a small number of evaluators.
  • Figure handling: Insufficient textual descriptions in input sources sometimes produced overly general figure descriptions, imprecise placement, and vague in-text explanations.The system’s parsing tool also occasionally cropped images imprecisely, causing minor inconsistencies.
  • Claims grounding: Reported grounding problems included partially correct explanations, confusing terminology, and one clearly false statement.Manual inspection showed that the false claim faithfully reflected a factual inaccuracy in its cited source.
Loading 2608.14446v1…