Source-linked AI summary
Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation
Chenghao Zhang, Guanting Dong, Yufan Liu, Tong Zhao, Xiaoxi Li, Zhicheng Dou
TL;DR
Verifiable multimodal deep research must synthesize open-ended evidence while interleaving text with visually grounded content. Ptah addresses this with a staged multi-agent harness and verifier checks, and reports credible, professionally interleaved reports with improved multimodal quality and readability. The authors note that modular stages improve controllability but introduce manually defined boundaries.
Problem
Open-ended deep research lacks deterministic ground truth and requires reliable interleaving of textual arguments with visual evidence.
Method
Ptah coordinates specialized agents through Planning, Research, and Writing, maintaining visual-aware plans, claim-grounded evidence, source-aligned images, and verifier checks.
Results
PTAH consistently produces credible and professionally interleaved reports, while PTAHEval measures image content and presentation quality alongside textual reliability.
Takeaways & Limitations
The work advances multimodal deep research toward evidence-grounded, visually informative, and human-centric report generation.
Takeaways & Limitations
The modular three-stage design introduces manually defined boundaries because existing open-source models have constrained reasoning capabilities.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have advanced autonomous agents from deep search, which retrieves concise factual answers, to deep research, which synthesizes scattered evidence into long-form reports. However, verifiable multimodal deep research remains challenging due to open-ended synthesis without deterministic ground truth and the need to interleave textual arguments with visual evidence. We propose Ptah, a multi-agent harness for interleaved report generation. Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages, where specialized agents construct visual-aware plans, collect claim-grounded evidence, maintain source-aligned images in a Visual Working Memory, and compose reports through declarative multimodal tool use. A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow. We further introduce PtahEval, an evaluation protocol that augments existing benchmarks with image-level and presentation-level assessments. Experiments on deep research benchmarks show that Ptah produces more reliable, visually informative, and usable human-facing multimodal reports than strong baselines. Our code is released at https://github.com/SnowNation101/Ptah
1 Introduction
PTAH addresses the verification and multimodal interleaving challenges of open-ended deep research by coordinating specialized agents, visual-aware intermediate states, and stage-wise checks. PTAHEval evaluates image content and multimodal presentation, while experiments report credible, high-quality, readable reports.
- Open-ended deep research lacks deterministic ground truth and must interleave textual arguments with visual evidence.
- Existing systems lack stage-wise verification and often treat images as post-hoc decoration, weakening factual reliability and visual alignment.
- PTAH coordinates specialized agents, external tools, research states, and verification signals across Planning, Research, and Writing.
- PTAHEval assesses Image Content Quality and Multimodal Presentation Quality, with experiments reporting high-quality, credible, professionally interleaved reports and strong textual reliability.
- The visual-aware workflow maintains explicit textual coverage, claim-grounded evidence, citations, numerical data, and source-aligned visual candidates as intermediate artifacts.
- Verifier hooks check protocol compliance, factual grounding, citation fidelity, visual relevance, and cross-modal consistency before workflow advancement.
2 Related Work
Related work progresses from tool-augmented deep search for deterministic answers to open-ended report generation and interleaved multimodal output. Existing methods still generally lack explicit verification and cross-modal consistency checks.
- Deep-search systems extend RAG with iterative retrieval, evidence verification, and richer agent action spaces, but primarily target closed-end questions with deterministic answers.
- Recent systems apply deep search to open-ended, long-form reports, yet single-agent architectures struggle to combine multi-hop reasoning with broad information coverage.
- Recent multimodal language models mainly understand interleaved inputs and generally cannot generate interleaved outputs.
- Interleaved generation follows unified multimodal-model and tool-augmented-agent paradigms, supported by dedicated benchmarks.
- Existing interleaved-generation methods generally lack explicit verification and cross-modal consistency checks.
3 Task Formulation
The task formulates multimodal deep research as producing a rendered web report from a plain-text query. Reports are ordered sequences of textual and visual blocks generated from a maintained research state and interaction trajectory.
- Given a plain-text query q, the system produces a multimodal research report r and rendered web page h.
- The report is represented as an ordered sequence of content blocks, each either a textual segment or a visual element.
- The harness maintains research state comprising the query, structured working state, and prior interaction history.
- The structured working state includes intermediate plans, evidence, citations, numerical data, and visual candidates that tools can update during reasoning.
- After research-state construction, the report is sampled and rendered by serializing interleaved blocks into HTML and displaying a webpage.
4 PTAH: Verifiable Multi-Agent Harness
PTAH orchestrates multimodal research through planning, evidence collection, visual-memory construction, declarative composition, and verifier-guided refinement. Its agents preserve source alignment and check research artifacts before progression.
- PTAH runs Planning, Research, and Writing, assigning visual-aware planning, claim-grounded evidence collection, source-aligned images, and report composition to specialized agents.
- 4.1 Planning: Visual-Aware Research State Initialization: The Planner Agent specifies section goals, expected evidence, visual placement, communicative roles, and suitable visual forms.
- 4.1 Planning: Visual-Aware Research State Initialization: Verifier checks plans using rule-based protocol and format validation plus LLM-based assessments of query coverage, coherence, and visual–argument relevance.
- 4.2 Research: Researcher Agents independently produce packages containing findings, claim-grounded evidence, numerical data, tables, references, and writing instructions.
- 4.2 Research: Visual Working Memory stores filtered, source-aligned images selected according to planning requirements using rule-based filtering and VLM evaluation.
- 4.2 Research: Research packages are verified for citation support, goal coverage, numerical and reference consistency, and visual relevance, with failed packages revised.
- 4.3 Writing: The Writer Agent jointly generates text and image directives, while image reference, search, or generation operations realize required visuals.
- 4.3 Writing: Verifier-guided test-time scaling refines sections, images, and overall organization before returning the multimodal report.
5 PTAHEval Evaluation Protocol
PTAHEval extends deep-research evaluation beyond text by assessing both image content and multimodal presentation in rendered reports. It preserves existing benchmark questions and text metrics while adding image- and layout-level judgments.
- PTAHEval addresses the insufficiency of text-focused evaluation for reports combining textual arguments, visual evidence, and rendered layouts.
- The protocol preserves original benchmark questions and text-oriented metrics while adding evaluation over the generated multimodal report artifact.
- Image Content Quality measures whether individual images are clear, relevant, informative, and aligned with surrounding text.
- Multimodal Presentation Quality evaluates the rendered report under realistic reading conditions, including layout, spacing, visual placement, and image–text organization.
- Both ICQ and MPQ dimensions use five-point Likert scores alongside original benchmark metrics to assess textual reliability, image quality, and presentation.
6 Experiments
Experiments compare PTAH with direct-generation, text-only, and multimodal baselines on content, visual quality, and credibility. PTAH consistently leads across these dimensions, while ablations show benefits from verification and test-time scaling.
- Experimental Setup: PTAH is evaluated on DeepResearch Bench and DeepConsult against direct-generation, text-only search, and multimodal baselines using shared retrieval settings.
- Main Results: PTAH consistently surpasses prior approaches in textual content quality, visual quality, and factual credibility.
- Main Results: 45.16 overall score on DeepResearch Bench makes PTAH the best system, with leading Insight/Depth and Readability.
- Main Results: 16.18 average on DeepConsult exceeds twice the best baseline, with PTAH leading across all reported dimensions.
- Visual Quality: PTAH achieves the highest scores across all Image Content Quality and Multimodal Presentation Quality dimensions on DeepResearch Bench.
- Credibility: 87.53 Citation Accuracy with 9.64 effective citations per task substantially outperforms baselines on DeepResearch Bench.
- Human Evaluation: Human annotators favor PTAH over LLM-I for Image Content Quality and over LLM-I and WebThinker for Multimodal Presentation Quality.
- Ablation Studies: Removing the Verifier leaves only 68 of 100 tasks producing final reports and substantially reduces citation validity and factual correctness.
7 Conclusion
PTAH is a verifiable multimodal deep-research harness that combines staged orchestration, verification, and multimodal evaluation. Experiments, human studies, and ablations show credible, professionally interleaved reports with evidence-grounded visual content.
- PTAH decomposes research from query to rendered report into Planning, Research, and Writing stages.
- A Verifier Agent applies rule-based and LLM-based checks for factual grounding, citation fidelity, and cross-modal consistency across stages.
- PTAHEval augments existing benchmarks with image-content and presentation metrics for multimodal reports.
- Experiments, human studies, and ablations show that PTAH consistently produces credible and professionally interleaved reports.
Limitations
The system remains constrained by the reasoning capabilities of existing open-source models, so PTAH uses manually defined stage boundaries for controllability and validation.
- Existing open-source models limit stable autonomous workflows for long-horizon multimodal search and generation.
- PTAH’s sequential modular design introduces manually defined boundaries, but enables granular monitoring, intermediate validation, and independent module optimization.
A More Implementation Details
PTAH integrates specialized retrieval, generation, editing, visualization, and code-execution tools across its multi-agent workflow. External search and model APIs are replaceable interfaces rather than core design commitments.
- Implementation interfaces: Serper, Jina Reader, and hosted model APIs function as replaceable interfaces that can be substituted with alternative tools or local deployments.Hosted APIs primarily reduce the local GPU cost of large-scale experiments.
- Tool allocation: PTAH assigns text retrieval to the PLANNER and RESEARCHER, while the WRITER uses visual retrieval, synthesis, code execution, and image editing.These tools support information gathering, draft composition, precise visualization, and subsequent visual refinement.
- Retrieval tools: Text Search retrieves relevant webpages, parses them into Markdown, and summarizes them with Qwen3-32B for key information extraction.The pipeline uses Google Search for URLs and Jina Reader for webpage parsing.
- Visual tools: Image Search maps text queries to top-K matching images to retrieve factual visual evidence for entities and other real-world subjects.It uses Google Image Search through Serper.
- Visual tools: Image Generation synthesizes thematic or abstract illustrations from textual descriptions, while Image Editing modifies retrieved or generated images using textual instructions.The implementations use Qwen-Image and Qwen-Image-Edit-2509 through SiliconFlow.
- Visualization tools: Code Execution runs generated Python scripts in an isolated sandbox to render precise charts and plots from structured data.This tool supports rigorous data visualization during report composition.
C Dataset Details
The evaluation uses DeepResearch Bench and DeepConsult to assess long-form research capabilities, while PTAH applies stage-wise verification and curated visual evidence construction. Research outputs are checked for structure, citations, content consistency, and image relevance.
- Datasets: DeepResearch Bench contains 100 PhD-level tasks across 22 domains, split evenly between English and Chinese.Tasks require multi-step web exploration, information integration, analytical reasoning, and comprehensive reports.
- Datasets: DeepConsult contains 102 business and consulting queries spanning market analysis, investment assessment, industry evaluation, financial modeling, technology trends, and strategic planning.Its queries target structured, insightful, and actionable reports for realistic consulting scenarios.
- Verification: The Verifier operates across Planning, Research, and Writing using rule-based and LLM-based checks for correctness, consistency, and quality.Verification is integrated throughout the workflow rather than restricted to a final output check.
- Planning verification: Planning verification checks output structure, tool-call schemas, search strategy rationality, and outline completeness.Rule-based checks validate formatting and schemas, while rubric-based checks assess the Planner’s reasoning trajectory.
- Research verification: Research verification enforces exact URL consistency and evaluates exploration depth, completeness, and alignment between synthesized findings and retrieved content.These checks target citation fidelity and research reliability.
- Writing verification: Writing verification checks image-tool syntax and consistency with research findings while iteratively improving report clarity and fidelity.The Verifier provides feedback that guides Writer refinement.
- Visual evidence curation: PTAH downloads webpage images, filters low-quality or irrelevant visuals, applies VLM-based retention decisions, and builds a section-level Visual Working Memory.Filtering removes low-resolution, tiny, extreme-aspect-ratio, and SVG images before semantic selection.
F Efficiency Analysis
PTAH’s multimodal workflow adds substantial computation, concentrated in research and refinement, but parallel research reduces latency. Human evaluation and ablation results indicate that the generated reports improve practical reading experience and multimodal presentation while preserving textual quality.
- Pipeline efficiency: 1015 seconds is PTAH’s average full-pipeline latency on DeepResearch Bench, with Research taking 459 seconds and Test-Time Scaling taking 243 seconds.The main costs arise from evidence collection, image-pool construction, verifier-guided refinement, and multimodal report refinement.
- Parallel research: 65.4% is the wall-clock reduction from parallel section-level research, lowering Research latency from 1328 seconds to 459 seconds.Sequential execution is 2.89× slower, showing that multi-agent parallelism offsets some added computation.
- Verifier efficiency: A stronger DeepSeek-R1 Verifier raises Planning latency from 192 to 853 seconds and Research latency from 459 to 1408 seconds.The increase reflects longer reasoning and additional revision rounds, creating a quality–efficiency trade-off.
- Human evaluation: PTAH is compared with WebThinker on 20 sampled DeepResearch Bench reports using readability, usability, information acquisition efficiency, and overall preference.Four evaluators include two AI PhD students and two AI undergraduate students.
- Human evaluation: 88.75% is PTAH’s win-or-tie rate for readability and usability, while information acquisition efficiency reaches 96.25% and overall preference reaches 95.00%.The evaluation reports these rates against WebThinker across the four user-centric dimensions.
- Visual ablation: Removing images changes DRB overall only from 45.16 to 45.10, while preserving the same Planning–Research–Writing pipeline and stage-wise verification.The text-oriented benchmark does not directly reward relevant or helpful visual evidence.
- Visual ablation: 3.71 to 3.29 is the drop in MPQ average after images are removed from PTAH reports.The ablation indicates that visuals materially contribute to multimodal presentation quality rather than merely changing surface appearance.