PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepWeaver: Bridging the Evidence Synthesis Gap in Open-Ended Question Answering

Xujia Wang, Yizhe Zhang, Bin Xu, Lei Hou, Juanzi Li

arXiv:2608.18988v1cs.CLcs.AI

TL;DR

Open-ended QA still lacks an effective way to organize noisy, fragmented retrieved evidence into comprehensive, well-cited answers. DeepWeaver structures and revises evidence-grounded claims, improving evidence synthesis and citation quality across LoQA and DeepResearch Bench.

  • Problem

    Organizing extensive, noisy, fragmented retrieved evidence into comprehensive answers remains a key challenge beyond retrieval.

  • Method

    DeepWeaver organizes evidence into Thought Block Chains and uses subordinate chains to inspect overlooked evidence, discover claims, and revise the main structure.

  • Results

    DeepWeaver improves answer comprehensiveness and citation quality on LoQA and DeepResearch Bench, also achieving deeper insights on the latter.

  • Takeaways & Limitations

    The results support explicit evidence weaving as a mechanism for bridging retrieval and final answer generation in open-ended QA.

  • Takeaways & Limitations

    LoQA focuses on Chinese water-environment questions, limiting coverage of other languages, domains, and document types.

Abstract

from arXiv · show

Retrieve-then-generate pipelines are commonly used to produce deep-research answers for open-ended questions, but retrieval alone is insufficient: LLMs must organize noisy and fragmented evidence into comprehensive, well-cited answers. We refer to this process as evidence synthesis. However, direct generation often underuses evidence, misaligns citations, and collapses diverse information into shallow summaries, exposing an evidence synthesis gap between retrieval and generation. Thus, we propose DeepWeaver, a novel framework that weaves noisy retrieved evidence into comprehensive answers by maintaining Thought Block Chains (TBCs), a structured representation that groups claims, salient information, keywords, and supporting evidence. DeepWeaver uses subordinate TBCs to inspect residual evidence, commit TBC revisions, and discover new claims before final generation. We evaluate DeepWeaver on open-ended QA over both knowledge bases and the web, and introduce LoQA, a high-density benchmark for evidence synthesis. Across multiple LLMs, DeepWeaver improves content sufficiency, citation grounding, and detail preservation on LoQA, while achieving deeper insights and higher citation quality on DeepResearch Bench. These results show that evidence weaving is an effective mechanism for bridging retrieval and generation in open-ended QA. Our code is available at https://github.com/KlozeWang/DeepWeaver.

1 Introduction

Open-ended QA faces an evidence synthesis gap: retrieval provides noisy, fragmented evidence, but LLMs often fail to organize it into comprehensive, detailed, well-cited answers. DeepWeaver addresses this gap by weaving evidence into Thought Block Chains and is evaluated on LoQA and DeepResearch Bench.

  • Introduction: Retrieval alone is insufficient because LLMs often underuse noisy, fragmented, knowledge-dense evidence, obscuring fine-grained details and producing underdeveloped claims.Prior approaches mainly improve evidence presentation, compression, filtering, restructuring, decomposition, or outline-guided writing rather than deeply synthesizing evidence.
  • Introduction: DeepWeaver structures noisy retrieved evidence into Thought Block Chains, whose thought blocks store claims, keywords, salient information, and supporting evidence fragments.It identifies overlooked evidence, generates subordinate TBCs to inspect residual evidence, and weaves revised TBCs into answers.
  • Introduction: LoQA contains 100 water-environment research questions paired with evidence from 500 Chinese knowledge-base books and over 100M characters.The benchmark tests comprehensive evidence use, preservation of domain-specific details, and well-cited answer generation.
  • Introduction: DeepWeaver improves content sufficiency, citation grounding, and detail preservation on LoQA, while achieving deeper insights and higher citation quality on DeepResearch Bench.DeepResearch Bench contains PhD-level research tasks across diverse fields.

2 Task Formulation

The task asks QA systems to synthesize scattered evidence fragments into comprehensive, grounded answers to open-ended research questions. LoQA operationalizes this challenge with dense noisy contexts and evaluates content sufficiency, citation grounding, and detail preservation.

  • 2 Task Formulation: QA systems must synthesize large, scattered evidence sets into comprehensive answers for realistic open-ended information-seeking questions.The evidence may span multiple claims and sources, and retrieval alone does not establish whether systems can fully exploit it.
  • 2.1 Definition: Given a question q and retrieved evidence pool E, the system generates answer y that thoroughly resolves q while remaining grounded in the evidence.
  • 2.1 Definition: Evidence synthesis quality is assessed by content sufficiency, detail preservation, and citation grounding.These dimensions respectively concern covering major claims, retaining fine-grained information, and grounding claims in relevant evidence.
  • 2.2 The LoQA Benchmark: LoQA uses a Chinese water-environment knowledge base containing 500 books and over 100M characters to represent dense, cross-subtopic technical questions.The domain includes pollution control, water quality assessment, and watershed management, with complex causal relations and practical open-ended questions.
  • 2.2 The LoQA Benchmark: 100 open-ended research-style questions are selected from 3,000 LLM-generated candidates using retrieval comprehensiveness, concreteness, domain depth, and expression completeness.Final questions have the highest average similarity to retrieved fragments, ensuring rich evidence.
  • 2.2 The LoQA Benchmark: Each LoQA question contains about 100 retrieved 1,024-token evidence fragments plus 100 random fragments, totaling about 200 × 1024 = 205K tokens of evidence.The added random fragments simulate a noisier context and approach the context window of many LLMs.
  • 2.2 The LoQA Benchmark: Citation grounding is measured with Citation Count (CC), Relevant Count (RC), and Relevant Ratio (RR) over cited relevant and irrelevant evidence fragments.DeepSeek-V3.2 judges fragment relevance, partitioning E into relevant ER and irrelevant EI before computing CC, RC, and RR.
  • 2.2 The LoQA Benchmark: LoQA evaluates content sufficiency through key-term and argument coverage and detail preservation through recovery of technical terms, concepts, and short phrases in cloze blanks.Detail Preservation (DP) is the proportion of cloze blanks that can be correctly recovered from the generated answer.

3 Method

DeepWeaver bridges retrieved evidence and answer generation with a Thought Block Chain (TBC) that organizes claims alongside their keywords, salient information, and supporting evidence. It refines this structure through Draft, Subordinate, and Commit stages to recover overlooked evidence and produce focused, woven claims.

  • Thought Block Chain: The TBC decomposes an answer into thought blocks, each linking a candidate claim to keywords, salient information, and supporting evidence fragments.For block b_i, E_i ⊆ E denotes the evidence subset linked to claim c_i.
  • Thought Block Chain: Unlike a plain outline, the TBC organizes fine-grained woven claims, explicitly maps claims to evidence, and reduces generation’s context burden.This design targets question-resolving answers rather than survey-like summaries.
  • Weaving Stages: DeepWeaver maintains a main TBC through three ordered stages: Draft, Subordinate, and Commit.Draft uses direct generation over question q and evidence pool E, then extracts an initial main TBC from the draft.
  • Weaving Stages: The Subordinate stage inspects residual evidence to find neglected, weakly covered, or insufficiently reflected fragments and discover missing aspects, details, and alternative support.It generates a subordinate TBC from residual evidence as a local inspector of the main TBC.
  • Weaving Stages: Commit merges overlapping claims and their associated information and evidence, then discards irrelevant, redundant, or weakly supported claims.Merging integrates multi-dimensional explanations, while discarding prevents the TBC from over-expanding.

q, MERGE(T 0

DeepWeaver iteratively refines woven claims by revisiting subordinate and commit stages while sampling evidence across rounds to manage context length. It then generates answer sections from claim-linked evidence subsets and composes them into the final answer.

  • Evidence refinement: DeepWeaver repeats subordinate and commit stages for n rounds to refine woven claims and incorporate additional supporting evidence.Iterative refinement enables the framework to revise claims while adding evidence.
  • Evidence refinement: Draft and subordinate stages randomly sample r evidence fragments from E and residual evidence Rt, with r < |E|, across multiple revision rounds.This exposes the model to the full evidence pool without processing an excessively long context in one pass.
  • Final generation: After weaving, DeepWeaver generates each answer section Si from block metadata and linked evidence subset Ei as focused local context.The generation function is Si = GENERATE(q, ci, si, Ei).
  • Final generation: The final answer is composed from all generated sections, decomposing context pressure into claim-level generation and improving citation grounding through smaller, relevant evidence subsets.The cited strategy links the generated sections Si while grounding each section in its corresponding evidence.

4 Experiments

DeepWeaver is evaluated on LoQA, a high-density knowledge-base benchmark, and DeepResearch Bench, a web-based deep-research benchmark. It improves evidence synthesis quality across content sufficiency, citation grounding, detail preservation, and citation quality while exposing limitations of direct generation under noisy evidence.

  • Benchmarks: LoQA contains 100 practical water-environment questions paired with long-context evidence from a 500-book knowledge base.The benchmark tests whether systems can synthesize fragmented retrieved evidence into comprehensive answers.
  • Benchmarks: DeepResearch Bench contains 100 PhD-level tasks across 22 fields and evaluates answer quality, citation accuracy, and effective citations.RACE measures comprehensiveness, insight, instruction following, and readability, while FACT measures citation accuracy and effective-citation count.
  • Main results: DeepWeaver consistently improves evidence synthesis quality across instruct and thinking backbones and substantially outperforms direct generation and evidence-based QA baselines.Compared baselines include RAG, Skeleton-of-Thought, Plan-and-Solve, Chain-of-Agents, and LongRefiner.
  • Main results: With Qwen3-30B-A3B-Instruct-2507, DeepWeaver surpasses E-RAG by 15.5% in Argument Sufficiency, +14.7 Relevant Citations, 14.5% in Relevant Citation Ratio, and 16.6% in Detail Preservation.The reported gains span content sufficiency, citation grounding, and detail preservation.
  • Evidence synthesis gap: E-RAG does not outperform E120-RAG despite receiving more evidence, while oracle ER-RAG still lags behind DeepWeaver, showing that more evidence or noise removal alone is insufficient.The results suggest that additional retrieved chunks increase context burden and can cause useful information to be overlooked.

M into the final answer. In w/o commit, the system

DeepWeaver benefits from subordinate-and-commit refinement, generalizes across LLM backbones, and improves evidence synthesis for web-based deep research. Two refinement rounds offer the best cost–performance trade-off, while evidence weaving improves claim-level organization and citation quality beyond direct generation or coarse webpage-level summarization.

  • Ablation study: Removing subordination degrades all three evidence-synthesis dimensions, while removing commitment preserves AS and DP but produces more overlapping and redundant blocks.The results indicate that initial TBCs are often incomplete and require refinement, whereas commitment helps maintain concision and coherent claim weaving.
  • Ablation study: n = 2 improves content sufficiency, citation grounding, and detail preservation over one round, whereas a third round adds no substantial improvement.The second subordinate-and-commit step enriches claims and the final TBC; n = 2 provides the best overall performance and cost–performance trade-off.
  • Backbone generalizability: On DeepSeek-V4-Flash, DeepWeaver raises Recall from 24.2 to 31.6, AS from 75.4% to 86.5%, RC from 31.0 to 44.6, and DP from 60.0% to 67.4%.The reported gains support consistent improvements across backbone models rather than dependence on a specific LLM.
  • Backbone generalizability: DeepWeaver with Qwen3-30B-A3B-Instruct-2507 outperforms direct generation with the stronger long-context model DeepSeek-V3.2.This result indicates that explicitly weaving evidence is more effective than relying solely on stronger long-context capabilities.
  • Web-based deep research: On DeepResearch Bench, DeepWeaver achieves the best overall RACE score and substantially improves effective citation and citation accuracy over WebWeaver on FACT.DeepWeaver uses fine-grained evidence chunks and claim-level structures, unlike WebWeaver’s coarse webpage-level outline summaries.
  • Web-based deep research: Overall performance gains show that evidence weaving benefits web-based deep research as a downstream module beyond knowledge-base QA despite its trade-off.DeepWeaver reuses WebWeaver’s retrieved pages and section titles, treating pages as evidence fragments for section-level answer generation rather than changing retrieval.

5 Related Work

Related work spans retrieval-augmented and attributed generation, long-context evidence integration, and long-form grounded writing for research-oriented answers. These approaches retrieve or organize evidence and improve attribution, but the cited work also identifies limitations in noisy long contexts and open-ended question answering.

  • Retrieval-Augmented and Attributed Generation: Retrieval-augmented generation incorporates external knowledge into question answering through sparse term-frequency or keyword retrieval and dense vector retrieval.The passage situates both sparse and dense retrieval as established approaches for incorporating external knowledge.
  • Long-Context Evidence Integration: Larger context windows expose models to more retrieved evidence, yet models may overlook relevant information in long or noisy inputs.Context-refinement methods such as RECOMP, Chain-of-Note, and LongRefiner improve upstream evidence presentation by compressing, filtering, or restructuring passages.
  • Long-Form Grounded Writing and DeepResearch Agents: Long-form grounded writing uses outline-guided and reflection-driven generation mainly for survey generation, while citation-oriented work improves attribution and verifiability.The passage distinguishes these research directions from methods primarily designed to answer open-ended questions.

6 Conclusion

DeepWeaver bridges the evidence synthesis gap in open-ended question answering by organizing noisy retrieved evidence into Thought Block Chains and iteratively weaving it into grounded claims. The framework introduces LoQA and improves answer quality, detail preservation, and citation grounding across knowledge-base and web-based deep-research settings.

  • Framework: DeepWeaver organizes noisy evidence into Thought Block Chains that connect fine-grained evidence fragments with evidence-grounded woven claims.This replaces treating retrieved evidence as a flat long-context input.
  • Evidence weaving: Iterative evidence weaving identifies overlooked evidence, refines incomplete claims, and improves detail preservation and citation grounding.
  • Benchmark: LoQA is introduced as a high-density evidence benchmark for evaluating evidence-grounded answer generation.
  • Evaluation: Experiments on knowledge-base QA and web-based deep research show consistent answer-quality improvements across multiple LLM backbones.
  • Implication: The findings indicate that open-ended QA requires explicit evidence synthesis mechanisms between retrieval and generation, especially with noisy and fragmented evidence.

7 Limitations · A Appendix · A.1 The LoQA Benchmark

DeepWeaver advances evidence synthesis for retrieval-augmented open-ended question answering, but the LoQA evaluation is limited to Chinese water-environment questions. DeepResearch Bench provides a complementary evaluation setting, while broader coverage remains future work.

  • 7 Limitations: LoQA focuses on Chinese water-environment questions, limiting coverage across languages, domains, and document types.The domain offers dense technical evidence and complex practical problems but does not represent all evaluation settings.
  • 7 Limitations: DeepResearch Bench is used as a complementary setting to evaluate DeepWeaver beyond LoQA.This evaluation supplements the benchmark’s narrow domain focus.
  • 7 Limitations: Broader evaluation across languages, domains, and document types remains future work.The stated limitation follows from LoQA’s focus on Chinese water-environment questions.

A.1.1 Data Statistics · A.1.2 Construction Details

LoQA is a high-density benchmark built from extensive noisy water-environment evidence and designed to evaluate evidence synthesis through localized, fine-grained targets. Its construction combines expert-corpus filtering, professional question generation, noisy evidence retrieval, multiple reference answers, and evaluations of citation grounding, content sufficiency, and detail preservation.

  • A.1.1 Data Statistics: LoQA contains 100 questions paired with an average of 206K evidence tokens and 91.79 relevant chunks per question.The benchmark includes 2,045 atomic scoring points, 4,779 cloze blanks, and reference answers averaging 9,858.64 tokens per question.
  • A.1.2 Construction Details: LoQA is built from a water-environment expert corpus filtered from approximately 30,000 hydraulic-engineering PDF books to 800 relevant books.The filtering uses tags and LLM analysis of titles, keywords, and descriptions.
  • A.1.2 Construction Details: Extract-Kit processing produces approximately 360 MB of clean text for the benchmark corpus.
  • A.1.2 Construction Details: Three LLMs generate 2,700 candidate questions from sampled seed-document fragments under requirements for decontextualization, concision, answerability, and multisource knowledge dependence.The models are GLM-4-Plus, DeepSeek-V3.2, and Qwen3-235B-A22B-Thinking-2507, each producing three questions per seed fragment.
  • A.1.2 Construction Details: The evidence pipeline retrieves approximately 90–100 highest-cosine-similarity fragments from a 500-book knowledge base after segmenting it into 1,024-token fragments and encoding them with GLM-Embedding-3.
  • A.1.2 Construction Details: Adding about 100 random fragments yields 200 evidence fragments and an evidence pool of approximately 200K tokens per question.The random sampling simulates realistic noisy retrieval.
  • A.1.2 Construction Details: LoQA creates five reference answers by generating answers from four disjoint relevant-evidence partitions and the full relevant-evidence set.Partition-based answers support local coverage under lower context burden, while the full-evidence answer supports global synthesis.
  • A.1.2 Construction Details: Atomic arguments, cloze blanks, and evidence-relevance annotations evaluate citation grounding, content sufficiency, and detail preservation through constrained extraction and verification tasks.The targets include evidence-supported answer aspects and domain-specific terms, concepts, and short phrases; evaluation avoids holistic subjective scoring of long-form answers.

A.1.3 Comparison with Prior Benchmarks · A.2 Details of Extending DeepWeaver to DeepResearch Bench

LoQA isolates evidence synthesis in open-ended QA by pairing questions with fixed, extensive, noisy evidence pools and evaluating citation grounding, content sufficiency, and detail preservation. DeepWeaver is extended to DeepResearch Bench in a controlled setting that fixes the search layer while testing evidence-to-answer generation.

  • A.1.3 Comparison with Prior Benchmarks: LoQA complements prior benchmarks by isolating evidence synthesis quality rather than explanatory QA, short-form verification, long-context understanding, or end-to-end web research.Each question is paired with a fixed, extensive, and noisy evidence pool.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: DeepResearch Bench evaluates end-to-end web research, so DeepWeaver is tested with the same search system and comparable inference cost to isolate evidence-to-answer generation.DeepWeaver does not include a search module.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: WebWeaver supplies search: its planner iteratively searches web pages, refines a hierarchical outline, assigns pages to outline nodes, and extracts evidence into a memory bank.The writer then fills the outline from collected evidence.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: DeepWeaver uses only first-level outline titles O1 and their corresponding raw web pages W, excluding WebWeaver’s detailed outline and extracted evidence memory.This keeps the search layer fixed while isolating evidence weaving during long-form generation.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: For each first-level outline node, the system rewrites it into a section task and routes critical or evidence-heavy sections to DeepWeaver, using direct generation otherwise.The rewriting and routing step consumes only a few hundred tokens and has negligible cost.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: 4.41 first-level outline nodes occur per question on average, with 2.35 routed to DeepWeaver, representing 59.39% of items.Routed items average 98 chunks and 92.75K raw tokens; LoQA averages about 200K tokens per question.
  • A.2 Details of Extending DeepWeaver to DeepResearch Bench: After section generation, lightweight polishing improves fluency and readability, while added introductions and conclusions satisfy DeepResearch Bench’s complete-answer format.The core comparison concerns synthesizing web evidence into comprehensive, insightful, and well-cited answer content.

A.3 Cost Analysis … B Ethics Statement

The supplementary sections quantify DeepWeaver’s lightweight cost, specify its staged prompting and TBC operations, illustrate evidence weaving in two case studies, and state safeguards for copyrighted source materials and data use.

  • A.3 Cost Analysis: $0.313 vs. $0.303: DeepWeaver’s monetary cost is comparable to WebWeaver when using the same searcher and excluding web-search and page-access costs.Only LLM-side computation is measured, with local inference on Qwen3-30B-A3B-Instruct-2507 using 2×A100 GPUs.
  • A.3 Cost Analysis: 35K tokens versus 24K tokens: DeepWeaver passes more original webpage content to the writer while reducing cost per observed token from 1.26×10−5 to 8.94×10−6.The comparison is made under comparable cost.
  • A.3 Cost Analysis: DeepWeaver’s main additional cost is the Subordinate stage, whose output-token usage is only slightly higher than Draft and Summary while remaining lightweight overall.The Subordinate stage inspects residual evidence and enriches the Thought Block Chain.
  • A.4 Prompt Templates: The prompt templates represent each Thought Block Chain with an argument, keywords, key information, reference chunks, and covered-evidence retrieval chunks.The example includes claims, salient information, and evidence-fragment indices as structured fields.
  • A.4 Prompt Templates: The staged templates generate drafts, extract logic blocks, inspect uncovered evidence, screen cross-group merge candidates, commit merges, discard unsuitable blocks, and expand retained arguments.Discard criteria include off-topic, redundant, incorrect, and low-impact information.
  • A.5 Case Study: In LoQA, DeepWeaver links separate water-treatment claims with subordinate claims about circular resource use and multidimensional evaluation, reframing treatment as resource recovery.The initial claims cover water reuse, energy recovery, material recovery, ecological evaluation, and economic feasibility.
  • A.5 Case Study: In DeepResearch Bench, DeepWeaver connects sports-tutoring functions—including posture correction, performance evaluation, personalized training, injury prediction, and tactical analysis—with three levels of multimodal fusion.The fusion levels are signal-level, feature-level, and decision-level fusion.
  • B Ethics Statement: The authors use licensed books without claiming ownership, release only limited excerpts with annotations and metadata, and state that the material cannot substitute for or reconstruct the originals.Each book contributes approximately 1,000 OCR-derived tokens for non-commercial academic research, reproducibility, and evaluation.

C Dataset License and Terms of Use

DeepWeaver releases its annotations, metadata, and evaluation code under Apache-2.0, while data use CC BY-NC 4.0 and copyrighted book excerpts remain with their rights holders.

  • C Dataset License and Terms of Use: Annotations, metadata, and evaluation code are released under Apache-2.0, while data are released under CC BY-NC 4.0.Copyrighted book excerpts remain the property of their respective rights holders.

Case Study on LoQA · Case Study on DeepResearch Bench · M , RETRIEVECOVERED(E, H, T (n)

The case studies illustrate DeepWeaver’s evidence synthesis across a complex LoQA question and a DeepResearch Bench sports-intelligence question. The synthesized answers preserve detailed mechanisms, evaluation dimensions, and application-specific insights from fragmented evidence.

  • Case Study on LoQA: DeepWeaver reframes industrial water treatment from compliance-oriented discharge toward high-quality reclaimed water, using separation, graded reuse, cascading use, and double-membrane treatment.The target applications include industrial cooling, boiler makeup, municipal uses, and agricultural irrigation.
  • Case Study on LoQA: The LoQA synthesis expands treatment goals to coordinated recovery of water, energy, nutrients, metals, and valuable by-products while minimizing pollution and supporting low-carbon operation.Examples include anaerobic biogas recovery, nitrogen and phosphorus recovery, and recovery of metals, paper-making alkali, or microbial protein.
  • Case Study on LoQA: The LoQA answer further proposes integrated resource-recovery process packages with source separation and distributed treatment, while identifying technology cost, standards, incentives, finance, and market models as implementation conditions.The passages specifically mention DESAR-inspired classification, high-cost advanced oxidation and membrane separation, third-party governance, subsidies, and green finance.
  • Case Study on LoQA: Its proposed evaluation framework replaces single treatment-efficiency measures with multidimensional indicators covering resource recovery, ecological impact, economic feasibility, process performance, and system robustness.Examples include water recovery rate, energy net output, carbon footprint, lifecycle cost, investment return, automation, and shock-load resistance.
  • Case Study on DeepResearch Bench: The sports case study shows SITLS delivering personalized technical feedback and objective team-dynamics analysis through edge AI, multimodal sensing, and spatiotemporal mapping.Applications include correcting golf and gymnastics technique, quantifying coordination and synchronization, identifying formation weaknesses, and supporting individualized training or teaching recommendations.
  • Case Study on DeepResearch Bench: SITLS supports proactive injury prevention by combining biomechanical stress and physiological fatigue signals to identify asymmetrical loading, declining HRV, and other early warning patterns.The system compares data against an athlete’s baseline and enables intervention before clinical injury occurs.
  • Case Study on DeepResearch Bench: For DeepResearch Bench, SITLS performs real-time motion analysis by fusing raw IMU and synchronized video signals into a dynamic 3D movement model for deviation detection and immediate feedback.Signal-level fusion uses accelerometer, gyroscope, and magnetometer streams with video, improving noise reduction and motion-artifact suppression.
Loading 2608.18988v1…