TVIR: Building Deep Research Agents Towards Text-Visual Interleaved Report Generation

Motivation

• Existing deep research systems prioritize textual coherence, depth, and citation support while often treating visuals as decorative, leaving fidelity, provenance, and text–visual alignment insufficiently evaluated.This mismatch undermines reliability for high-stakes analytical work and encourages superficial visual inclusion over evidential rigor.Source: §1 · p. 1, Figure 1 · p. 2

TVIR-BENCH

• TVIR-BENCH comprises 100 expert-curated tasks requiring retrieved images and code-generated charts to serve specific analytical sub-goals rather than being appended post hoc.The benchmark targets end-to-end generation of long-context multimodal research reports across diverse domains and complexity levels.Source: §1 · pp. 1–2, Figure 1 · p. 2

TVIR-AGENT

• TVIR-AGENT uses a hierarchical pipeline with planning, visual asset instantiation, context-aware writing, and polishing to integrate traceable images and charts into report reasoning.Its modules explicitly model multimodal constraints and treat visuals as integral reasoning components rather than optional embellishments.Source: §1 · p. 2

Evaluation

• The evaluation suite combines Textual Assessment of grounding, consistency, and depth with Visual Assessment of figure quality, chart fidelity, and cross-modal alignment.This dual-path design jointly audits textual and visual report quality.Source: §1 · p. 2

Empirical insight

• Experiments across nine representative deep research systems show that current LLMs often produce fluent text while prioritizing decorative rather than evidential visuals.The finding highlights a significant gap in existing paradigms for evidence-based multimodal reasoning.Source: §1 · p. 2

Deep Research Agent

• Existing deep research agents are mostly text-centric, while multimodal systems typically treat visuals as auxiliary outputs centered on charts; TVIR-AGENT integrates retrieved images and code-generated charts throughout the workflow.Examples include WebThinker (Li et al., 2025), WebWeaver (Li et al., 2026a), Multimodal DeepResearcher (Yang et al., 2026), and FinSight (Jin et al., 2025).Source: §2 · p. 2

Deep Research Benchmark

• Existing deep research benchmarks mainly evaluate text-only report generation, while MultimodalReportBench evaluates code-generated charts with coarse-grained assessment.The text-only benchmarks include DeepResearch Bench (Du et al., 2026) and DeepResearch Bench II.Source: §2 · pp. 2–3
• MMDeepResearch-Bench studies image–text inputs with visuals grounded in task-provided images, whereas TVIR-BENCH evaluates end-to-end text-visual interleaved generation using fine-grained textual and visual assessment.This distinction positions TVIR-BENCH as evaluating both report generation and visual quality at a finer granularity.Source: §2 · p. 3

3.1 Data Design

• Tasks are designed around realistic, role-driven, demand-oriented, frontier-focused, and multimodal research needs, with low, medium, and high integration requirements.The design targets substantive analytical synthesis rather than simple information retrieval and varies instruction-following demands across three complexity levels.Source: §3.1 · p. 3

3.2 Data Construction

• Construction combines expert topic proposals, LLM drafting, review by three domain experts, revision, and checklist compilation for verifiable task-specific evaluation.Review covers design compliance, factual accuracy, logical coherence, and multimodal validity, including accessible images and code-generated charts based on real data.Source: §3.2 · p. 3

3.3 Dataset Statistics

• TVIR-BENCH contains 100 tasks—50 Chinese and 50 English—covering 10 domains, three balanced complexity levels, and eight roughly balanced functional types.The functional types include trend prediction, mechanism explanation, and comparative analysis.Source: §3.3 · p. 4

3.4 Evaluation Framework

• The evaluation framework has complementary Textual Assessment and Visual Assessment components, each averaging fine-grained metrics scored on a normalized 0–100 scale.Unless otherwise specified, metrics are evaluated using an LLM-as-a-Judge.Source: §3.4 · p. 4

3.4.1 Report Preprocessing

• Report preprocessing extracts references and URLs, fact–citation pairs, and figure elements with captions, visual content, context, and citation indices for downstream assessment.Extracted URLs are used to retrieve corresponding webpage text via the Serper API.Source: §3.4.1 · p. 4

3.4.2 Textual Assessment (TA)

• Textual Assessment measures citation support, instruction alignment, writing quality, analytical depth and breadth, and factual and logical consistency.Citation Support and Instruction Alignment use 1, 0.5, or 0 judgments, while report-level metrics assess organization, reasoning, coverage, and contradictions.Source: §3.4.2 · pp. 4–5

3.4.3 Visual Assessment (VA)

• Visual Assessment evaluates multimodal composition, figure quality, caption quality, figure–context integration, and chart–source consistency.It covers figure organization, intrinsic image and chart quality, caption accuracy, narrative integration, and consistency between charts and cited sources.Source: §3.4.3 · p. 5

4.1 Research-Grounded Planning

• The Planner retrieves information and synthesizes a structured outline with section summaries, visual requirements, and citation-backed research notes for factual grounding and traceability.Each research note records a citation, source URL, and key findings.Source: §4.1 · p. 5

4.2 Visual Asset Instantiation

• The Image Searcher retrieves and verifies visual candidates, while the Chart Generator validates data across sources and executes plotting code in a sandbox to produce charts.The resulting augmented outline preserves captions, descriptions, and provenance, including original data-source URLs for charts and source webpage URLs for images.Source: §4.2 · pp. 5–6

4.3 Context-Aware Sequential Writing

• The Writer composes sections sequentially using each outline unit, prior-section global context, research notes, and additional web research when evidence is insufficient.It places visuals according to their descriptions and interleaves Markdown text with charts from local paths and retrieved images from online URLs.Source: §4.3 · p. 6

4.4 Global Index Polishing

• The Polisher removes uncited references, globally deduplicates and renumbers citations, and sequentially renumbers figures while updating their in-text references.Reference deduplication uses URL and normalized content.Source: §4.4 · p. 6

5.2 Main Results

• TVIR-Agent variants achieve the strongest aggregate performance on TVIR-BENCH, with the Claude-4.5-Sonnet variant best overall, GLM-4.7 best on TA, and Claude-4.5-Sonnet best on VA.Manus-1.6 is the strongest commercial system, while text-only Gemini-3-Pro Deep Research cannot be evaluated on VA or Overall.Source: §5.2 · pp. 6–7, Table 1 · p. 7

5.3 Further Analysis

• TVIR-Agent variants show stronger evidence grounding and cross-modal alignment, with GLM-4.7 scoring 68.64 on CS versus 47.53 for Claude-4.5-Sonnet w/Search and Claude-4.5-Sonnet scoring 74.49 on FCQ versus Manus-1.6’s 66.14.FLC remains relatively weaker for several strong systems, making long-form factual and logical consistency a shared challenge.Source: §5.3 · p. 7
• Across analyses, increasing task complexity generally lowers IA but raises ADB, while TVIR-AGENT variants consistently rank among the top systems across both language subsets.Source: §5.3 · pp. 7–8

5.4 Validation of the Evaluation Framework

• Evaluation validation finds near-perfect extraction reliability, 96.53% webpage-text retrieval success, strong human alignment, and robust agreement between GPT-5.2 and Gemini-2.5-Pro judges.The human alignment study covers 8 systems and 100 tasks, with an Overall inter-annotator PAR of 74.20.Source: §5.4 · pp. 9–10, Table 5 · p. 9, Table 6 · p. 10

5.5 Ablation Studies

• Removing any TVIR-Agent component lowers overall performance, while removing the Chart Generator causes the largest drop, reducing VA from 78.62 to 60.91 and Overall from 73.92 to 63.84.The Image Searcher improves all metrics, whereas removing research notes has a relatively small effect, supporting complementary research-planning and visual capabilities.Source: §5.5 · p. 10, Table 8 · p. 10

6 Conclusion

• TVIR unifies TVIR-BENCH, dual-path textual and visual assessment, and TVIR-AGENT, which models visual evidence throughout planning and writing.The benchmark is expert-curated, and the agent is hierarchical and multi-agent.Source: §6 · p. 10
• TVIR-AGENT achieves strong overall performance and improves evidence grounding and cross-modal alignment over existing systems.The conclusion reports these gains from extensive experiments but provides no numerical values.Source: §6 · p. 10
• Current deep research systems remain much stronger at textual synthesis than at integrating visual assets, highlighting a key limitation for trustworthy multimodal research.TVIR is presented as a foundation for future work on trustworthy multimodal deep research agents.Source: §6 · p. 10

A.1 Definitions of Task Design Principles

• Tasks should center on a concrete professional user, objective, constraint, or decision scenario to anchor research needs in authentic use cases.Examples include policymakers, clinicians, investment analysts, researchers, engineers, or a biopharmaceutical R&D leader assessing a therapeutic direction.Source: §A.1 · p. 23
• Tasks should express a coherent, bounded research demand with logically connected sub-questions and explicit information goals that support organization and evaluation.Instructions should avoid ambiguity, redundancy, contradiction, and unconstrained opinion prompts while remaining operational even when complex.Source: §A.1 · p. 23
• Deep research tasks should require multi-source evidence synthesis, causal or explanatory reasoning, comparison of competing claims, and conclusions or actionable recommendations.They should also address source conflicts, data limitations, interpretive uncertainty, or weaknesses in methods and assumptions, forming a loop from evidence gathering through critique to conclusions.Source: §A.1 · p. 23
• Tasks should focus on recent developments, emerging challenges, and open questions, preferably using authoritative sources from 2024 onward and the latest available data.For unsettled frontier topics, tasks may require comparing institutional or scholarly viewpoints and analyzing the basis of disagreement.Source: §A.1 · p. 23
• Tasks should make multimodal information integral to the research process by requiring visuals that improve interpretation, explanation, or decision support rather than decoration.Requests may naturally ask models to visualize, plot, show, illustrate, or provide an explained figure, with the intended visual type implied by the analytical need.Source: §A.1 · p. 23
• Required retrieved images must be publicly available online, while generated charts must use real, publicly accessible data and serve a specified analytical objective.Examples include heatmaps for regional policy differences, architecture diagrams for model innovations, and time-series plots for market growth.Source: §A.1 · p. 23

A.2 Definitions of Task Complexity Levels

• Low-complexity tasks are about 200 Chinese characters or 130 English words, include one to three multimodal requirements, and assess focused research with limited coordination.They may use a compact paragraph or brief outline without explicit bullet decomposition.Source: §A.2 · p. 24
• Medium-complexity tasks are about 400 Chinese characters or 260 English words, contain two to four multimodal requirements, and typically organize three to four major points.Each major point usually includes one to three sub-questions, increasing demands on instruction following, cross-source synthesis, and coordinated text-visual evidence.Source: §A.2 · p. 24
• High-complexity tasks are about 600 Chinese characters or 390 English words, contain three to five multimodal requirements, and use four to five major points.Each major point contains two to four sub-questions, requiring long-horizon planning, detailed instruction tracking, and multimodal integration across analytical dimensions.Source: §A.2 · p. 24
• Across levels, increasing task complexity raises the expected length, number of multimodal requirements, structural explicitness, and coordination demands.The progression moves from focused requests with limited structure to explicitly organized tasks spanning multiple analytical dimensions.Source: §A.2 · p. 24

A.3 Definitions of High-Level Functional Types

• TVIR-BENCH predefines eight high-level functional types to specify the intended analytical role of sub-questions.Each sub-question is designed to primarily instantiate one functional type.Source: §A.3 · p. 24
• Table 9 provides the definitions of the eight high-level functional types used in TVIR-BENCH.The supplied passage identifies the table’s scope but does not provide the individual type definitions.Source: Table 9 · p. 24, §A.3 · p. 24

• Fine-grained metrics are first computed in [0, 1] and then linearly rescaled to [0, 100] for presentation.Source: B Fine-Grained Evaluation Metrics… · p. 24

Citation Support

• Citation Support scores extracted fact–citation pairs by whether each fact is supported, partially supported, or unsupported.When no fact–citation pair is available, the score is set to 0.Source: §B.1 · pp. 24–25

Instruction Alignment

• Instruction Alignment scores checklist items according to whether they are fully satisfied, partially satisfied, or not satisfied.Source: §B.1 · p. 25

Writing Quality

• Writing Quality is assessed with a report-level rubric containing four dimensions and a normalized aggregate of judge scores.Source: §B.1 · p. 25

Analytical Depth & Breadth

• Analytical Depth & Breadth is assessed with a report-level rubric containing five dimensions and a normalized aggregate of judge scores.Source: §B.1 · p. 25

Factual & Logical Consistency

• Factual & Logical Consistency converts the judge-identified number of distinct internal contradiction issues into a discrete 10-point score and then normalizes it.Source: §B.1 · p. 25, Table 10 · p. 25

Multimodal Composition

• Multimodal Composition uses a report-level rubric with two dimensions, while reports without figure elements receive a score of 0.Source: §B.2 · p. 26

Figure Quality

• Figure Quality combines image and chart quality, using calibrated image sub-scores, duplicate penalties, and checklist-based chart evaluation.Image quality considers resolution, aspect ratio, sharpness, and contrast; chart quality uses a 10-item binary checklist. Missing images, charts, or figure elements are assigned zero in the specified component or aggregate scores.Source: §B.2 · p. 26

Figure Caption Quality

• Figure Caption Quality scores each figure element on three dimensions and assigns zero when figure elements or captions are missing.Source: §B.2 · p. 26

Figure–Context Integration

• Figure–Context Integration scores each figure element on three dimensions, assigning zero when figure elements or their contextual information are missing.Source: §B.2 · p. 27

Chart–Source Consistency

• Chart–Source Consistency maps each chart’s contradiction count to a discrete 10-point score, with zero assigned to absent charts or charts lacking usable cited references.The contradiction-count mapping is provided in Table 11.Source: §B.2 · p. 27, Table 11 · p. 27

Evaluation Representation

• Instruction Alignment and Multimodal Composition are judged from textual reports, with captions and insertion markers indicating multimodal satisfaction and organization.This avoids requiring current LLMs to process lengthy text and many figures jointly in one pass.Source: C Evaluation Implementation Details · p. 27

Visual Validation

• Unusable visual elements—including missing figures, broken links, corrupted files, empty paths, and pseudo-visual text—are replaced with empty values and treated as invalid.The preprocessing prevents invalid references from misleading text-based multimodal evaluation.Source: C Evaluation Implementation Details · p. 27

Caption Correction

• Captions scoring at most 6 on any Figure Caption Quality dimension are replaced with judge-generated revisions, while missing captions are generated when possible.The corrected report is used for subsequent evaluation to better reflect the figure’s actual multimodal content.Source: C Evaluation Implementation Details · p. 27

D.1 Report Collection

• All commercial systems receive a unified multimodal prompt, with output-format and chart-generation instructions adapted to platform support; Gemini-3-Pro Deep Research remains text-only in this setup.Reports are collected through official web interfaces rather than APIs, and the same prompt is provided for fairness and consistency.Source: §D.1 · p. 28
• HTML reports from Claude-4.5-Sonnet w/Search and Genspark Deep Research are converted to Markdown while preserving text, provenance, citations, charts, images, and their positions.All reports, including TVIR-AGENT variants, are normalized into the same storage format before evaluation.Source: §D.1 · p. 28

D.2 Variance Across Independent Runs

• Standard deviations are consistently small across systems, including TA, VA, and Overall, indicating stable evaluations and robust reported performance differences.All results are independently evaluated three times under the same setup, with standard deviations reported in Table 12.Source: §D.2 · p. 28

D.3 Average Effective Citations

• Average Effective Citations measures the average quantity of verifiably supported information delivered per task, complementing normalized citation-support quality.AEC rewards systems that sustain strong evidence grounding at larger informational scale, whereas normalized support can favor fewer well-supported statements.Source: §D.3 · pp. 28–29
• AEC computes each task’s effective citation count by summing support scores over extracted fact–citation pairs and averaging across the full benchmark.The support scores distinguish supported, partially supported, and unsupported pairs.Source: §D.3 · pp. 28–29

D.4 Agreement Metrics

• Pairwise Agreement Rate measures how often two evaluators make the same pairwise preference over report pairs for the same task, treating equal scores as ties.Preferences are determined by comparing the evaluators’ scores for each report pair.Source: §D.4 · p. 29
• Overall Pearson Correlation measures agreement in absolute system-level score patterns, whereas Overall Spearman Correlation measures agreement in system rankings.OPC uses Pearson correlation, while OSC uses Spearman rank correlation over the evaluators’ system-level average-score vectors.Source: §D.4 · p. 30

SYSTEM PROMPT Project Background

• Multimodal Deep Research benchmarks must assess textual generation, external-image appropriateness, chart correctness, and semantic and logical coordination between text and visuals.The motivation is that existing text-centric benchmarks overlook whether visuals provide evidence, support conclusions, and remain aligned with the report narrative.Source: SYSTEM PROMPT Project Background · p. 31

USER PROMPT

• The task-generation prompt requests one domain-specific Deep Research task centered on a topic, following role-driven, demand-oriented, frontier-focused, and multimodal design principles.It requires extensive web search and supports configurable language and complexity settings.Source: USER PROMPT · p. 35

Prompt for Reference Entries Extraction

• Reference and citation prompts separately extract ordered reference entries and body-level fact–citation pairs, preserving citation indices and available reference URLs.Reference extraction excludes citation facts from the report body and returns structured JSON records.Source: Prompt for Reference Entries Extraction · p. 36, Prompt for Fact–Citation Pairs… · p. 36

Prompt for Figure Elements Extraction

• Figure extraction records each image, chart, graph, or diagram with its exact caption, contents, contextual text, and citation indices from captions or legends.If no explicit figure reference exists, the surrounding text is captured with a figure placeholder at the figure’s position.Source: Prompt for Figure Elements Extraction · p. 37

Prompt for Citation Support

• Evaluation prompts assess citation support, instruction alignment, writing quality, analytical depth and breadth, and factual–logical consistency as distinct report qualities.Instruction alignment uses Pass, Partial, and Fail scores, while consistency evaluation counts internal contradictions rather than external factual errors or citation issues.Source: Prompt for Citation Support · p. 38, Prompt for Instruction Alignment · p. 38, Prompt for Writing Quality · p. 40

Prompt for Factual & Logical Consistency

• The consistency evaluator defines one issue as unresolved factual or logical statements that cannot all be true simultaneously, counting repeated manifestations of the same contradiction once.Examples include conflicting quantities, scopes, trends, premises, judgments, conclusions, and claims of generalizability.Source: Prompt for Factual & Logical Consistency · pp. 46–47

Prompt for Chart Quality

• Chart-quality auditing uses binary pass/fail checks for overlap, alignment, visibility, composition, readability, rendering, decoration, information load, and redundant labels.Any visible defect fails the relevant criterion, covering layout, typography, distinguishability, encoding, and conciseness.Source: Prompt for Chart Quality · p. 48, A. Layout & Structure · p. 48

Prompt for Multimodal Composition

• The multimodal composition prompt evaluates document-level organization, quantity, variety, and richness rather than the local relevance or factual correctness of individual figures.It instructs evaluators to judge how effectively multimodal elements are incorporated into the overall document design.Source: Prompt for Multimodal Composition · p. 49

1. Multimodal Layout & Quantity Appropriateness

• The layout rubric assesses whether figures are appropriately placed, distributed, proportionate, and supportive of complex content across the report.It penalizes visual gaps, clustering, detachment from discussion, under-illustration, and overload, using defined score bands from 1–2 through 9–10.Source: §1 · p. 49

2. Multimodal Variety & Richness

• The variety rubric distinguishes retrieved images and code-generated charts, counts appropriate subtypes, and requires topic-aligned balance between visual source types.Data-heavy topics emphasize varied charts, technical or architectural topics emphasize varied retrieved images, and balanced topics use both types.Source: §2 · p. 50
• Variety scores are determined exclusively by topic category, source-type presence, and subtype counts under hard score-range boundaries.The rubric requires intermediate variables and forbids subjective quality judgments, suggestions for additional content, or personal preferences from changing the score range.Source: §2 · pp. 50–51
• Only actual embedded figures or concrete non-empty locators count as valid multimodal elements; placeholders, unsupported references, and ASCII diagrams do not.Reports without valid multimodal content should receive correspondingly weak layout, quantity, and variety scores.Source: §2 · p. 51

1. Visual Accuracy

• The figure-caption rubric scores visual accuracy by comparing the caption’s identified subjects, variables, and relationships against visible figure evidence.Scores range from 9–10 for a complete match to 1–2 for contradiction, with intermediate bands covering omissions, incompleteness, and materially incorrect descriptions.Source: Prompt for Figure Caption Quality · p. 52, §1 · p. 52

2. Minimum Necessary Information

• Captions must provide the minimum information needed for independent figure interpretation while remaining clear, accessible, and free of unsupported claims.The rubric evaluates sufficiency and clarity separately, and requires a revised caption when any dimension scores 6 or lower.Source: §2 · pp. 52–53

Prompt for Figure–Context Integration

• The evaluation compares captions against visual evidence, assesses caption quality across three dimensions, requires concrete figure-based justifications, assigns each dimension a 1–10 score, and evaluates figure integration with surrounding context.Source: §2 · p. 53, Prompt for Figure–Context Integration · p. 54
• Figure–context integration is judged by how well the visual relates to and supports the surrounding report context.The evaluation examines the figure, its surrounding paragraphs, and the placeholder marking where the figure appears.Source: Prompt for Figure–Context Integration · p. 54

1. Contextual Relevance

• Contextual relevance measures semantic alignment between the figure and the specific topics, variables, phenomena, or cases discussed nearby.Scores range from 9–10 for direct depiction of the local subject matter to 1–2 for no meaningful relationship.Source: §1 · p. 54

2. Narrative Coherence

• Narrative coherence measures whether the text clearly references, transitions to, and interprets the figure within the report flow.A seamless 9–10 requires explicit reference and meaningful discussion, whereas a disruptive 1–2 reflects random placement or contradiction of the narrative sequence.Source: §2 · pp. 54–55

3. Visual Information Value

• Visual information value measures whether the figure conveys information that text alone would communicate inefficiently or inadequately.The rubric ranges from 9–10 for visually essential information to 1–2 when the visual adds no meaningful necessity.Source: §3 · p. 55
• The figure-integration procedure requires evidence-based justification and a 1–10 score for each dimension, with improvement suggestions for any score of 6 or lower.When all dimensions score at least 7, the suggestion field must be empty.Source: §3 · p. 55

Prompt for Chart–Source Consistency

• Chart–source consistency is assessed from the chart, caption, and cited reference snippets, treating those snippets as the only permissible evidence.The prompt excludes aesthetic concerns unless they prevent verification and does not judge the real-world credibility of sources or penalize missing evidence by itself.Source: Prompt for Chart–Source Consistency · p. 56

Definition Of An Issue

• A chart issue exists when a chart claim is contradicted by the references under the same or materially equivalent claim context.Distinct underlying problems count separately, while multiple symptoms from one root cause count as one issue; unrelated contradictions remain separate.Source: Definition Of An Issue · p. 56
• Chart auditing extracts claims from the chart and caption, verifies them against the supplied references, reports each distinct issue with quoted evidence and ref_idx, and counts the total.The required output is a JSON object containing the total issue count and structured issue records.Source: Definition Of An Issue · pp. 56–57

E.4 Prompts for Report Generation

• Report-generation prompts require rich visuals at key argumentative points, uniformly numbered and titled figures explicitly interpreted in text, cited online evidence, and complete reference entries with URLs.The requirements cover charts and external images, underlying chart data sources, standardized in-text citations, and unified reference formatting across multiple Deep Research systems.Source: §E.4 · p. 57, Prompt for Grok-4.1-Thinking DeepSearch · p. 58