PaperlayerView exact revision on arXiv

Source-linked AI summary

TVIR: Building Deep Research Agents Towards Text-Visual Interleaved Report Generation

Xinkai Ma, Zhiqi Bai, Dingling Zhang, Pei Liu, Yishuo Yuan, He Zhu, Jiakai Wang, Qianqian Xie, Yifan Zhao, Xinlong Yang, Hao Cong, Zhiheng Yao, Fengxia Xie, Zihao Xu, Haoran Xu, Zhaohui Wang, Minghao Liu, Shirong Lin, Yingshui Tan, Yuchi Xu, Wenbo Su, Zhaoxiang Zhang, Bo Zheng, Jiaheng Liu

arXiv:2606.02320v2cs.CL

TL;DR

Existing deep research systems and benchmarks remain largely text-centric, providing limited evaluation of whether visual evidence is reliable and aligned with analysis. TVIR introduces a benchmark, hierarchical agent framework, and dual-path evaluation, with TVIR-Agent variants achieving the strongest aggregate performance across evaluated systems.

  • Problem

    Existing deep research benchmarks and systems are predominantly text-centric, with limited evaluation of visual evidence fidelity, provenance, and alignment with analysis.

  • Method

    TVIR combines a 100-task multimodal benchmark, a hierarchical multi-agent framework that integrates traceable visual evidence, and joint textual-visual assessment.

  • Results

    TVIR-Agent variants achieve the strongest aggregate performance among evaluated systems, with Claude-4.5-Sonnet attaining the best Overall score and Visual Assessment score.

  • Takeaways & Limitations

    TVIR supports evidence-driven multimodal deep research and highlights the importance of explicitly integrating visual assets throughout report generation and evaluation.

Abstract

from arXiv · show

Deep Research Agents have shown strong capability in multi-step information retrieval, reasoning, and long-form report generation, but existing benchmarks and systems remain predominantly text-centric, with limited evaluation of whether visual elements are factually reliable and well aligned with the surrounding analysis. To address this gap, we introduce TVIR (Text-Visual Interleaved Report Generation), which includes TVIR-Bench, a benchmark of 100 expert-curated multimodal deep research tasks that require visual elements to serve specific analytical sub-goals, and TVIR-Agent, a hierarchical multi-agent framework that serves as a strong baseline for constructing outlines, retrieving images, generating charts with traceable sources, and composing reports through context-aware sequential writing. We further develop a dual-path evaluation framework that combines Textual Assessment and Visual Assessment. Experiments across nine deep research systems show that TVIR-Agent achieves strong overall performance, underscoring the importance of explicit multimodal design and evaluation for evidence-driven report generation.

1 Introduction

Existing deep research systems remain predominantly text-centric, overlooking whether visual evidence is semantically grounded and aligned with analysis. TVIR addresses this gap with a 100-task benchmark, a hierarchical multimodal agent, and dual-path textual and visual evaluation.

  • Motivation: Existing benchmarks and agent frameworks emphasize textual coherence, depth, and citation support while overlooking integrated visual evidence in professional reports.High-quality reports commonly interleave narrative analysis with charts, diagrams, and images.
  • TVIR-BENCH: TVIR-BENCH comprises 100 expert-curated multimodal tasks requiring retrieved images and code-generated charts to support specific analytical sub-goals.Its design prevents visuals from being appended post hoc and spans diverse domains and complexity levels.
  • TVIR-AGENT: TVIR-AGENT is a hierarchical multi-agent framework that plans multimodal constraints, instantiates traceable images and charts, and writes context-aware interleaved reports.Its components include a Planner, Visual Asset Instantiation module, Writer, and Polisher.
  • Evaluation: The evaluation suite combines Textual Assessment of grounding, consistency, and depth with Visual Assessment of figure quality, chart fidelity, and cross-modal alignment.This dual-path framework jointly audits textual and visual report quality.
  • Evaluation: Experiments with nine representative deep research systems show that current LLMs often favor decorative visuals over evidential ones despite strong textual fluency.The finding identifies a gap in evidence-based multimodal reasoning.

2 Related Work

Existing deep research agents and benchmarks remain predominantly text-centric, while multimodal efforts often treat visuals as auxiliary charts or evaluate them coarsely. TVIR addresses this gap through end-to-end text-visual interleaved report generation with fine-grained textual and visual assessment.

  • Deep Research Agent: Existing DRAs, including WebThinker and WebWeaver, primarily support long-horizon retrieval, reasoning, and report generation in text-centric settings.Multimodal systems such as Multimodal DeepResearcher and FinSight typically treat visuals as auxiliary outputs and mainly rely on charts.
  • Deep Research Benchmark: Existing deep research benchmarks mainly evaluate text-only report generation, including DeepResearch Bench and DeepResearch Bench II.MultimodalReportBench considers reports with code-generated charts, but its evaluation remains coarse-grained.
  • Deep Research Benchmark: MMDeepResearch-Bench studies image–text inputs with report visuals grounded in task-provided images, whereas TVIR-BENCH evaluates end-to-end interleaved generation with fine-grained textual and visual assessment.This contrast distinguishes TVIR-BENCH from benchmarks focused on code-generated charts or task-provided images.

3 TVIR-BENCH

TVIR-BENCH is an expert-driven benchmark of 100 balanced multimodal deep-research tasks spanning diverse domains, languages, complexity levels, and functional question types. Its reports are audited through complementary textual and visual assessments using structured evidence extraction and fine-grained metrics.

  • Task Design: Tasks follow five principles: role-driven, demand-oriented, deep research, frontier-focused, and multimodal integration.These principles target realistic user needs, substantive analytical synthesis, and explicit multimodal elements rather than simple retrieval.
  • Construction Pipeline: Dataset construction uses expert topic proposals, LLM-based drafting, review and revision by three domain experts, and checklist compilation.The review addresses design compliance, factual accuracy, and logical coherence before accepted tasks receive actionable evaluation checkpoints.
  • Benchmark Composition: TVIR-BENCH contains 100 multimodal deep-research tasks, evenly split between Chinese and English, spanning 10 domains and three balanced complexity levels.Its sub-questions cover eight high-level functional types, including trend prediction, mechanism explanation, and comparative analysis.
  • Evaluation Framework: Evaluation combines Textual Assessment and Visual Assessment, whose fine-grained metric scores are averaged and generally normalized to 0–100 using an LLM-as-a-Judge.Textual assessment covers citation support, instruction alignment, writing quality, analytical depth and breadth, and factual and logical consistency.
  • Evaluation Framework: Visual assessment evaluates multimodal composition, figure quality, captions, figure–context integration, and chart–source consistency.Reports are preprocessed to extract references, fact–citation pairs, and figure elements with captions, visual content, context, and citation indices.

4 TVIR-AGENT

TVIR-Agent is a four-stage framework for text–visual interleaved report generation. It plans evidence-grounded outlines, instantiates verified visual assets, writes sections with contextual evidence, and polishes report-level references and figures.

  • Overview: TVIR-Agent produces a report through four stages built around planning, visual instantiation, sequential writing, and final report-level polishing.The framework is developed based on MiroThinker for text–visual interleaved report generation.
  • Planning: The Planner parses tasks, retrieves information, and synthesizes outline units containing section plans, visual requirements, and research notes with citations and source URLs.The notes record key findings and provide factual grounding, credibility, and traceability for later stages.
  • Visual asset instantiation: The Image Searcher retrieves and verifies visual candidates, while the Chart Generator validates data across sources and executes plotting code in a sandbox.Image verification uses heuristic filtering and visual question answering; charts preserve their original data-source URLs.
  • Report writing: The Writer generates sections sequentially using each outline unit, prior-section global context, research notes, and additional searches when evidence is insufficient.It inserts visual assets according to their descriptions and composes Markdown with charts and retrieved images interleaved with text.
  • Report polishing: The Polisher removes uncited references, deduplicates and renumbers citations globally, then sequentially renumbers figures and updates their in-text references.Reference deduplication uses URL and normalized content.

5 Experiments

Across nine evaluated deep research systems, TVIR-Agent variants achieve the strongest aggregate performance and show particular advantages in cross-modal alignment, evidence grounding, and chart fulfillment. Results also reveal meaningful variation by task complexity, language, domain, tool-allocation strategy, and evaluation judge.

  • Overall performance: TVIR-Agent variants achieve the strongest aggregate performance, with the Claude-4.5-Sonnet variant ranking first overall and Manus-1.6 strongest among commercial systems.TVIR-Agent (GLM-4.7) achieves the best TA score, indicating that system strengths differ across aggregate and textual dimensions.
  • Fine-Grained Strengths and Limitations: 68.64 on CS gives TVIR-Agent (GLM-4.7) a 21.11-point lead over Claude-4.5-Sonnet w/Search, while 74.49 on FCQ gives TVIR-Agent (Claude-4.5-Sonnet) an 8.35-point lead over Manus-1.6.These results indicate stronger evidence grounding and more reliable cross-modal alignment in TVIR-Agent variants.
  • Performance across Task Complexity Levels: As task complexity increases, IA generally declines while ADB tends to improve, reflecting greater demands on multimodal coordination, instruction tracking, and long-horizon reasoning.More complex tasks are harder for systems to satisfy fully but appear to encourage additional visual-detail behavior.
  • Performance across Languages: Chinese tasks yield a mild overall advantage, especially in TA, although several commercial systems achieve comparable or stronger results on English and rankings remain broadly stable.The language subsets are parallel benchmark slices rather than direct translations, incorporating language-specific cultural and real-world context.
  • Performance across Domains: TVIR-Agent variants remain strong across domains, while History & Society and Education & Culture are easier and Technology & Intelligence, Finance & Business, and Law & Policy are more challenging.The harder domains involve denser terminology, faster-changing factual content, and stricter citation-grounding requirements.
  • Tool Usage Analysis within TVIR-AGENT: TVIR-Agent (GLM-4.7) reaches an AEC of 102.41 through heavier search and scraping, whereas TVIR-Agent (Claude-4.5-Sonnet) reaches an AEC of 86.14 and the highest chart fulfillment rate of 94.61%.The findings suggest that variant strengths depend on both backbone capability and how tools are allocated between retrieval and chart generation.

6 Conclusion

The paper introduces TVIR, a unified benchmark and agentic framework for text–visual interleaved report generation, and reports strong overall performance with improved evidence grounding and cross-modal alignment.

  • TVIR unifies a benchmark and agentic framework for text–visual interleaved report generation.
  • TVIR-BENCH is expert-curated and uses dual-path evaluation covering textual and visual assessment.
  • TVIR-AGENT is a hierarchical multi-agent framework that explicitly models visual evidence throughout planning and writing.
  • Extensive experiments show strong overall performance and improved evidence grounding and cross-modal alignment.

A Data Design Details · A.1 Definitions of Task Design Principles

The task-design principles define realistic users and coherent research demands, while requiring deep evidence synthesis, frontier relevance, and meaningful multimodal integration. Together, these principles specify tasks that reflect practical research needs and evaluate more than surface-level aggregation.

  • A.1 Definitions of Task Design Principles: Tasks should center on a concrete professional user whose research need plausibly arises in practice.Examples include policymakers, clinicians, investment analysts, researchers, and engineers.
  • A.1 Definitions of Task Design Principles: Each task should describe a specific user context and objective rather than addressing a vague or generic audience.
  • A.1 Definitions of Task Design Principles: Tasks should express a clear, structured research demand within one coherent domain or topic.Multiple sub-questions are allowed when they remain logically connected and address defined information goals.
  • A.1 Definitions of Task Design Principles: Deep-research tasks should require evidence synthesis, causal or explanatory reasoning, claim comparison, and conclusions or recommendations.Relevant evidence may include academic findings, statistical data, case materials, or expert viewpoints.
  • A.1 Definitions of Task Design Principles: Tasks should target recent developments, emerging challenges, and open questions rather than relying mainly on static background information.Expected topics include new technologies, policy changes, market shifts, and scientific breakthroughs salient within the past two to three years.
  • A.1 Definitions of Task Design Principles: Tasks should explicitly integrate multimodal information so visual elements meaningfully support the research process instead of serving as decoration.Examples include web-retrieved images and code-generated charts based on real, publicly accessible data.

A.2 Definitions of Task Complexity Levels · A.3 Definitions of High-Level Functional Types

TVIR-BENCH defines task complexity through increasing length, multimodal requirements, structural demands, and coordination burden. It also assigns sub-questions to eight predefined high-level functional types that specify their intended analytical roles.

  • A.2 Definitions of Task Complexity Levels: Low-complexity tasks contain about 130 English words and one to three multimodal requirements, assessing focused deep research with limited structural burden.They may use a compact paragraph or brief outline without explicit bullet-point decomposition.
  • A.2 Definitions of Task Complexity Levels: The complexity levels scale from concise, lightly structured requests to explicitly organized tasks spanning multiple analytical dimensions.The progression is reflected in increasing task length, multimodal requirements, sub-question structure, and coordination demands.
  • A.2 Definitions of Task Complexity Levels: Medium-complexity tasks contain about 260 English words and two to four multimodal requirements, demanding clearer structure and coordinated text-visual evidence use.They usually organize three to four major points, each with one to three sub-questions, increasing instruction-following and cross-source synthesis demands.
  • A.2 Definitions of Task Complexity Levels: High-complexity tasks contain about 390 English words and three to five multimodal requirements, imposing substantial long-horizon planning and multimodal integration demands.They require explicit structure, typically four to five major points with two to four sub-questions each.
  • A.3 Definitions of High-Level Functional Types: TVIR-BENCH predefines eight high-level functional types to specify the intended analytical role of each sub-question.Each sub-question is designed to primarily instantiate one functional type.
  • A.3 Definitions of High-Level Functional Types: Table 9 provides the definitions of the eight high-level functional types used in TVIR-BENCH.The table serves as the reference for these predefined analytical functions.

B Fine-Grained Evaluation Metrics Computation Details

Fine-grained metrics are computed on a normalized [0, 1] scale and then linearly rescaled to [0, 100] for presentation.

  • Fine-grained metrics are normalized to [0, 1] before linear rescaling to [0, 100] for presentation.

B.1 Textual Assessment Metrics

Textual assessment evaluates reports through citation support, instruction alignment, writing quality, analytical depth and breadth, and factual and logical consistency. These metrics combine pair- and checklist-level judgments with report-level rubrics and contradiction-based scoring.

  • Citation Support: Citation Support evaluates extracted fact–citation pairs using supported, partially supported, and unsupported judgments.If no fact–citation pair is available, the score is set to 0.
  • Instruction Alignment: Instruction Alignment evaluates checklist items using fully satisfied, partially satisfied, and not satisfied judgments.The metric is computed from the resulting checklist-item assessments.
  • Writing Quality: Writing Quality is assessed with a report-level rubric containing DWQ = 4 dimensions.The dimension scores are aggregated into a normalized report-level score.
  • Analytical Depth & Breadth: Analytical Depth & Breadth is assessed with a report-level rubric containing DADB = 5 dimensions.The dimension scores are aggregated into a normalized report-level score.
  • Factual & Logical Consistency: Factual & Logical Consistency maps the number of distinct internal contradiction issues to a discrete 10-point score and then normalizes it.The mapping is defined by a contradiction-count-to-score table.

B.2 Visual Assessment Metrics

Visual Assessment Metrics evaluate multimodal composition and figure quality alongside caption quality, figure–context integration, and chart–source consistency. Scores are normalized report-level or per-element assessments, with absent or unusable visual components receiving zero where specified.

  • Multimodal Composition: Multimodal Composition uses a report-level rubric with DMC = 2 dimensions.If no figure element is present, the score is set to 0.
  • Figure Quality: Figure Quality combines Image Quality and Chart Quality.Image Quality evaluates resolution, aspect ratio, sharpness, and contrast, while Chart Quality uses a binary checklist with DCQ = 10 items.
  • Figure Quality: Image Quality weights are calibrated by regression on expert ratings, and near-duplicate images receive a perceptual-hash penalty.If no image is present, the score is set to 0.
  • Caption and Context Metrics: Figure Caption Quality and Figure–Context Integration each use DFCQ = 3 and DFCI = 3 judged dimensions, respectively.Missing captions or missing figure context are assigned scores of 0 in the implementation.
  • Chart–Source Consistency: Chart–Source Consistency maps each chart’s contradiction-issue count to a discrete 10-point score and assigns 0 to charts without usable cited references.Charts with no usable cited references therefore receive zero for this metric.

C Evaluation Implementation Details

The evaluation handles multimodal report-level metrics through textual report representations because current LLMs may hallucinate when processing long reports and many figures jointly. It also filters unusable visuals and revises unreliable captions before text-based judgment.

  • Metric Evaluation: Instruction Alignment and Multimodal Composition are evaluated from textual report representations, with judges relying mainly on figure captions and insertion markers.This design addresses the unreliability of single-pass evaluation for lengthy text and numerous figures considered jointly.
  • Visual Validation: Unusable visual elements—including missing figures, broken links, corrupted files, empty paths, and pseudo-visual substitutes—are replaced with empty values and treated as invalid.Examples of pseudo-visual content include ASCII diagrams and Mermaid-style text graphics.
  • Caption Revision: Captions with any Figure Caption Quality dimension at most 6 are treated as unreliable and replaced with judge-generated revisions based on direct figure access.Figures without captions also undergo generated revision.

D Experimental Implementation Details … D.4 Agreement Metrics

The experimental implementation standardizes report collection and evaluation across systems, while assessing run stability, evidence-backed information quantity, and evaluator agreement through complementary metrics. Agreement is measured both by shared pairwise preferences and by correlation of system-level scores and rankings.

  • D.1 Report Collection: Six commercial closed-source systems are evaluated through official web interfaces using a unified prompt for text–visual interleaved report generation.The requested output is adapted to each platform’s native support as Markdown or HTML.
  • D.1 Report Collection: HTML reports from Claude-4.5-Sonnet w/Search and Genspark Deep Research are converted to Markdown before preprocessing and scoring.The conversion verifies preservation of textual content and accurate citation, attribution, and provenance alignment.
  • D.2 Variance Across Independent Runs: Three independent evaluations under the same setup produce consistently small metric standard deviations, indicating strong run-to-run stability.Aggregate TA, VA, and Overall metrics show particularly limited variation.
  • D.3 Average Effective Citations: Average Effective Citations (AEC) measures the average quantity of verifiably supported information provided per task through cited references.Unlike normalized Citation Support, AEC rewards sustained evidence grounding at larger informational scale.
  • D.4 Agreement Metrics: Pairwise Agreement Rate (PAR) measures how often two evaluators produce the same pairwise preference over report pairs for the same task.Preferences compare evaluator scores for two reports, with equality treated as a tie.
  • D.4 Agreement Metrics: Overall Pearson Correlation (OPC) compares evaluators’ system-level average score patterns, whereas Overall Spearman Correlation (OSC) compares their system rankings.OPC uses Pearson correlation, while OSC uses rank-based Spearman correlation.

E Prompt Templates … Definition Of An Issue

This section defines prompts for multimodal deep-research task generation, report extraction, evaluation, and report production. The prompts jointly assess textual quality, evidence provenance, analytical depth, logical consistency, chart quality, and multimodal composition.

  • USER PROMPT: The benchmark-generation prompt asks for domain-specific Deep Research tasks centered on a topic, using extensive web search and role-, demand-, frontier-, and multimodal-oriented design principles.Tasks are generated in a specified language and complexity level.
  • Prompt for Reference Entries Extraction: Reference extraction returns citation numbers and URLs in order, while fact–citation extraction separately identifies factual claims from the report body.Reference extraction excludes citation facts from the body and outputs a JSON list.
  • Prompt for Figure Elements Extraction: Figure extraction records each figure’s caption, file path or URL, contextual text, and citation numbers appearing in its caption or legend.Explicit figure references are captured with their explanatory text; otherwise, nearby context and a <figure> placeholder are inserted.
  • Prompt for Citation Support: Evaluation prompts assess citation support, instruction alignment, writing quality, analytical depth and breadth, and factual and logical consistency.Instruction alignment uses Pass, Partial, and Fail scores, while consistency evaluation counts distinct contradictions independently of citation or reference issues.
  • Prompt for Factual & Logical Consistency: The consistency rubric defines an issue as unresolved statements that cannot all be true simultaneously and counts repeated manifestations of one contradiction as a single issue.Examples include conflicting quantities, scopes, assumptions, judgments, conclusions, or claims of generalizability.
  • Prompt for Chart Quality: Chart-quality evaluation uses binary checklist scores to judge layout, readability, contrast, alignment, visibility, decoration, information load, and redundant labeling.A visible defect on a criterion produces a score of 0, whereas a flawless chart receives 1.
  • 2. Multimodal Variety & Richness: Multimodal-composition evaluation examines document-level layout, quantity, variety, and richness, using topic-dependent subtype counts and hard score boundaries based on valid embedded figures or concrete paths and URLs.It distinguishes data-heavy, technical-architectural, and balanced topics and excludes placeholders, text-only diagrams, and references without actual figures or locators.
  • E.4 Prompts for Report Generation: Report-generation prompts require richly illustrated, well-structured reports that place visuals at key argumentation points, number and interpret them, cite searched content and chart data sources, and provide complete accessible references.The prompts specify external images and tool-generated charts, including bar charts, line charts, heatmaps, radar charts, flowcharts, and architecture diagrams.
Loading 2606.02320v2…