PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Jiacheng Cai, Zijin Hong, Zheng Yuan, Huachi Zhou, Qinggang Zhang, Xiao Huang

arXiv:2609.05518v1cs.CVcs.AI

TL;DR

Existing multimodal benchmarks provide limited coverage of open-domain multi-hop evidence discovery and cross-modal reasoning paths. CrossModalQA builds a graph-guided benchmark across five paths, and experiments find that incomplete retrieval and multi-image reasoning remain major challenges.

  • Problem

    Existing benchmarks often use short evidence chains or small provided context sets and cover few modality transitions, limiting evaluation of open-domain discovery across cross-modal paths.

    Source: §1 · p. 2

  • Method

    CrossModalQA uses multimodal knowledge-graph subgraphs to construct and verify evidence-linked questions across five reasoning paths in an open-domain text-and-image corpus.

    Source: §1 · p. 3

  • Results

    Existing multimodal RAG systems struggle to retrieve complete evidence chains, and incomplete retrieval can underperform a closed-book MLLM; multi-image retrieval and reasoning remain bottlenecks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    CrossModalQA provides a testbed for diagnosing multimodal retrieval and reasoning bottlenecks across reasoning paths and depth.

    Source: §5 · p. 10

Abstract

from arXiv · show

Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.

1 INTRODUCTION

Existing multimodal benchmarks leave open-domain multi-hop evidence discovery and several cross-modal reasoning paths underexplored. CrossModalQA evaluates these challenges with evidence-linked questions, and experiments expose retrieval completeness and multi-image reasoning as persistent bottlenecks.

1 INTRODUCTION

  • Multimodal RAG must retrieve heterogeneous textual and visual evidence, ground entities across modalities, and compose complementary information into a complete reasoning chain.This extends text-based RAG, which grounds generation in external evidence because model knowledge is incomplete and difficult to update.Source: §1 · pp. 1–2
  • Existing benchmarks often cover only one or two modality transitions and use short chains or small provided context sets, leaving open-domain evidence discovery and complex paths underexplored.Without explicit supporting evidence, retrieval failures are harder to distinguish from reasoning and generation errors.Source: §1 · pp. 2–3
  • CrossModalQA contains 1,863 question–answer pairs over 4,987 Wikipedia articles and 4,431 Wikimedia Commons images, spanning five cross-modal reasoning paths.Every question requires retrieving and composing evidence across multiple multimodal-knowledge-graph nodes; V and T denote modalities traversed during reasoning, not the input modality.Source: §1 · p. 3
  • CrossModalQA uses subgraph-conditioned question generation, rule-based consistency checking, and LLM verification to create questions averaging 3.50 reasoning hops, with more than 40% requiring at least four hops.Each question retains an explicit multimodal supporting subgraph, enabling fine-grained evaluation of retrieval, cross-modal evidence linking, reasoning, and answer generation.Source: §1 · p. 3
  • Incomplete evidence retrieval can make multimodal RAG systems underperform a closed-book MLLM, while complete textual and visual evidence substantially improves answer accuracy.The analysis also finds that retrieval contributes more than simply scaling the answer generator, and that image-set questions remain particularly challenging.Source: §1 · p. 3

2 RELATED WORK

Prior work extends retrieval-augmented generation to text-and-image corpora, while benchmark coverage still lacks systematic control of modality paths and reasoning depth. CrossModalQA addresses this gap with five reasoning categories and multi-hop questions.

2.1 MULTIMODAL RETRIEVAL-AUGMENTED GENERATION

  • Multimodal RAG retrieves candidate evidence from modality-specific or shared representations, may rerank and filter it, and uses an MLLM to answer from the query and selected context.Recent systems also use hierarchical search, cross-modal reranking, evidence filtering, and agentic retrieval strategies.Source: §2.1 · pp. 3–4

2.2 MULTIMODAL RAG BENCHMARKS

  • Existing benchmarks often lack clear modality-specific reasoning paths and systematic joint control of path diversity and hop complexity; CrossModalQA covers five categories with multi-hop evidence composition.Its questions average 3.50 hops, and multi-image intersection and image-set questions are more concentrated at two or three hops.Source: §2.2 · p. 4, Figure 2 · p. 5
  • The field has shifted from small candidate sets toward larger heterogeneous corpora, raising evaluation from local selection to retrieving, filtering, and integrating evidence at scale.Benchmarks assess retrieval of visual or textual evidence and its integration with external knowledge.Source: §2.2 · p. 4

3 CROSSMODALQA

CrossModalQA is a multi-hop multimodal benchmark organized around five reasoning paths over visual and textual evidence. Its retained knowledge graphs support question generation, evidence tracing, and evaluation of retrieval and answer quality.

3.1 DATASET OVERVIEW

  • CrossModalQA contains 1,863 question-answer pairs from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images, covering all five cross-modality reasoning-path categories.The paper describes it as the first multimodal RAG benchmark to cover all five categories.Source: §3.1 · p. 5

3.2 REASONING-PATH TAXONOMY

  • Its five modes are vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning.The modes span visual entity identification followed by textual reasoning, image-based attribute answering, linked visual groundings, shared answers across branches, and visual comparison over candidate sets.Source: §3.2 · pp. 5–6

3.3 MULTI-HOP MULTIMODAL REASONING

  • Questions compose evidence across image nodes, entity nodes, and textual relations in retained multimodal subgraphs; image, entity, and relation transitions each count as one hop.Hop depth is the largest shortest-path distance from any retained evidence node to the answer node.Source: §3.3 · p. 6
  • 3.50 hops on average, with over 40% of questions requiring at least four hops; multi-image and image-set questions concentrate more at two or three hops.Those two modes target retrieval of multiple relevant images, with less emphasis on long reasoning chains.Source: §3.3 · p. 6

3.4 MULTIMODAL KG-GUIDED QA GENERATION

  • The construction process selects visually grounded article domains, extracts subject-relation-object triples, links entities to verified images, and builds a multimodal knowledge graph.A vision-language model checks entity-image consistency before valid pairs are linked to graph nodes.Source: §3.4 · pp. 6–7
  • For each question, a connected subgraph grounds LLM-generated QA pairs, which pass rule-based consistency checks and LLM verification for mode conformity and multimodal dependence.Source: §3.4 · p. 7

3.5.1 RETRIEVAL PERFORMANCE

  • Retrieval is evaluated with Recall@k, Precision@k, and Hit@k at k values 1, 5, and 10, macro-averaged across questions.The evidence items may be images, text articles, or their union; the measures capture gold-evidence recovery, retrieved-item relevance, and whether any gold item appears in the top k.Source: §3.5.1 · p. 7

3.5.2 GENERATION PERFORMANCE

  • Generation quality is measured over atomic answer facts with atom-level F1 and exact match, which requires no unsupported facts and no omitted reference facts.Reference answers and model responses are represented as sets of atomic facts.Source: §3.5.2 · p. 7

4 EXPERIMENTS

Experiments show that retrieval errors can make multimodal RAG baselines underperform closed-book GPT-4o, while complete evidence retrieval matters more than generator scaling. Results also vary with reasoning path, and image-set questions remain especially challenging.

4.1 EXPERIMENTAL SETUP

  • The evaluation compares M2RAG and mKG-RAG retrieval variants, uses GPT-4o as the generator for RAG methods, and measures retrieval and answer quality.Retrieval metrics are Recall@5, Recall@10, Precision@5, and Hit@5; answer metrics are exact match and micro-F1.Source: §4.1 · pp. 7–8

4.2 ABLATION STUDY

  • Joint gold text-and-image evidence substantially outperforms no retrieval or either single-modality setting in overall answer accuracy.The comparison supports the benchmark’s dependence on complementary evidence from both modalities.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 RESULT ANALYSIS

  • Every evaluated multimodal RAG baseline underperforms closed-book GPT-4o when retrieval omits required evidence and adds distracting context.Table 3 reports exact-match and F1 scores; the analysis attributes the gap to retrieval errors that impair answer accuracy.Source: Table 3 · p. 8, §4.3 · pp. 8–9

4.2 ABLATION STUDY

  • Gold images do not consistently lose to joint gold evidence because supplying a target image can reveal its entity early and shorten the evidence path.The authors interpret this as an oracle shortcut for T-to-V and V-to-T-to-V questions.Source: §4.2 · p. 8

4.3 RESULT ANALYSIS

  • The effective retrieval modality aligns with each task’s reasoning direction: text retrieval matters for V-to-T and multi-image intersection, while visual retrieval matters for paths ending in target images.The analysis draws this correspondence from oracle and baseline results.Source: Table 2 · p. 8, Table 3 · p. 8, §4.3 · p. 9
  • Image-set reasoning is the most challenging mode, with incomplete image retrieval and limited multi-image reasoning as compounding bottlenecks.Table 4 reports retrieval performance, while Table 2 reports oracle-evidence answer performance by mode.Source: Table 2 · p. 8, §4.3 · p. 9, Table 4 · p. 9
  • Gold evidence yields substantially larger accuracy gains than replacing the answer generator with a stronger model.The authors identify accurate retrieval as the dominant factor in end-to-end performance.Source: §4.3 · p. 9

5 CONCLUSION

CrossModalQA evaluates open-domain multimodal RAG across diverse cross-modal reasoning structures. Its experiments expose persistent challenges in complete evidence retrieval and multi-image reasoning.

5 CONCLUSION

  • CrossModalQA poses multi-hop questions over Wikipedia articles and Wikimedia Commons images across five reasoning-path categories.Each instance requires composing distributed textual and visual evidence.Source: §5 · p. 10
  • Multimodal knowledge graph-guided construction retains supporting subgraphs, enabling evaluation by reasoning path and depth.Source: §5 · p. 10
  • Experiments with representative multimodal RAG systems reveal substantial limitations in complete evidence retrieval and multi-image reasoning.The benchmark is presented as a testbed for diagnosing these bottlenecks.Source: §5 · p. 10

A.1 DOMAIN-TARGETED ARTICLE SELECTION

The construction pipeline targets visually rich article domains, then grounds extracted entities in Wikimedia images and filters mismatched entity-image pairs.

A.1 DOMAIN-TARGETED ARTICLE SELECTION

  • Candidate articles come from the English FineWiki subset, and GPT-5.4-nano assigns them to eight categories, with quotas for visually rich categories A–G and none for Else.The selected article pool totals 10,000; articles unused by retained questions are later discarded.Source: §A.1 · p. 13, Table 5 · p. 13
  • The pipeline retains 4,987 articles after discarding those not referenced by any retained question.Source: §A.1 · p. 13

A.2 IMAGES COLLECTION

  • Extracted entity mentions are linked to Wikidata, candidate images are collected from prioritized image-related properties, and visual verification filters mismatched pairs.The verification step is intended to improve final-dataset visual grounding quality.Source: §A.2 · p. 13

A.3 DATASET CONSTRUCTION DETAILS

Dataset construction samples multimodal knowledge-graph subgraphs under mode-specific constraints and generates questions from those subgraphs. The supplied figure captions describe hop and answer-quality views of the benchmark.

A.3 DATASET CONSTRUCTION DETAILS

  • The multimodal graph represents factual triples and image-attachment edges, while sampling configurations set edge, image-entity, expansion-depth, and hop-count constraints.The sampler draws at most αT_m candidates for a target count T_m and attempt multiplier α.Source: §A.3 · p. 13
  • Figure 3 presents average reasoning hops by mode, while Figure 4 describes EM by reasoning hop and QA mode under GPT-4o with gold texts and images.Figure 4 excludes one- and seven-hop cases because of insufficient sample sizes for reliable evaluation.Source: Figure 3 · p. 14, Figure 4 · p. 14
  • GPT-5.4 generates question-answer pairs according to the sampled subgraphs.Source: §A.3 · p. 14

A.4 ADDITIONAL EXPERIMENTAL SETTINGS

The settings hold GPT-4o fixed for answer generation and adapt mKG-RAG to text-only questions. Additional results describe hop- and mode-level scoring and GPT-4o mini performance under oracle-evidence settings.

A.4 ADDITIONAL EXPERIMENTAL SETTINGS

  • Figure 5 plots overall Micro-F1 by reasoning hop and per-mode Micro-F1, excluding 1-hop and 7-hop cases for insufficient sample sizes.Scores use a 0–100 scale rounded to the nearest integer; parentheses give QA-instance counts.Source: Figure 5 · p. 15
  • Table 6 reports GPT-4o mini Exact Match and micro-F1 by question mode and overall average under different oracle-evidence settings.Gold texts and gold images denote ground-truth evidence from the corresponding modality.Source: Table 6 · p. 15
  • Using GPT-4o as the answer-generation model in a fair, controlled comparison minimizes variation from underlying language models, so results more directly reflect methods’ retrieval, context construction, and multimodal evidence integration.Source: §A.4 · pp. 14–15
  • mKG-RAG is evaluated with text-to-text retrieval for textual evidence and text-to-image retrieval for image-side evidence, since questions contain no query image.The framework uses two-stage retrieval: candidate-document recall followed by query-relevant graph-evidence retrieval.Source: §A.4 · p. 15
  • Mode-specific subgraphs are sampled by traversing factual edges from graph seeds, retaining image-bearing entities and associated image edges, then filtering by size and duplicate signatures.The algorithm also applies hop and edge-budget constraints, though the supplied algorithm text is incomplete.Source: §A.4 · p. 16

B EVALUATION METRICS

Retrieval metrics score ranked evidence against mode-appropriate gold evidence, with macro-averaging so each question contributes equally. Generation metrics evaluate atomic answer facts using per-question exact match and dataset-level micro-F1.

B.1 RETRIEVAL PERFORMANCE.

  • Retrieval is evaluated in the evidence universe relevant to each setting, using Recall@k, Precision@k, and Hit@k for k ∈ {5, 10}.Recall measures gold-evidence coverage, precision measures relevance within the top-k list, and Hit checks whether any gold item is retrieved.Source: §B.1 · pp. 16–17
  • Macro-averaged retrieval scores give every question equal weight, regardless of its number of gold evidence items.This avoids disproportionate weight for questions with larger evidence sets.Source: §B.1 · p. 17

B.2 GENERATION PERFORMANCE.

  • Generation evaluation atomizes reference answers into gold facts and model responses into candidate facts, then judges atom-level true positives, false positives, and false negatives.Exact match requires no unsupported candidate atoms and no omitted gold atoms; factual correctness is reported as micro-averaged atom-level F1.Source: §B.2 · pp. 17–18

C ADDITIONAL RESULTS

CrossModalQA averages 3.50 reasoning hops, and performance varies with both hop count and question mode. Additional analyses describe text-only querying and knowledge-graph-guided question generation as design choices for evaluating and constructing multimodal evidence chains.

C ADDITIONAL RESULTS

  • 3.50 is the benchmark-wide average number of reasoning hops required across CrossModalQA questions.Figure 3 reports average hops by question mode.Source: C ADDITIONAL RESULTS · p. 18
  • Three-hop questions perform worse than both two-hop and four-hop questions, likely because the most difficult image-set questions require three hops.The authors conclude that question modality also matters for difficulty; hop count alone does not fully account for performance.Source: C ADDITIONAL RESULTS · p. 18
  • GPT-4o mini oracle-evidence results show the same modality sensitivities as GPT-4o: vision-to-text and multi-image intersection are more sensitive to text retrieval, while text-to-vision and vision-to-text-to-vision are more sensitive to image retrieval.These findings are reported for the additional oracle-evidence settings.Source: C ADDITIONAL RESULTS · p. 18

D.1 WHY TEXT-ONLY QUESTIONS

  • Text-only questions test whether systems can turn textual descriptions of visual attributes into cross-modal retrieval without using a provided image as a retrieval shortcut.The paper frames this as a more stringent and practically relevant assessment of multimodal retrieval and reasoning.Source: §D.1 · p. 18

D.2 WHY USE A MULTIMODAL KG FOR QA GENERATION

  • The multimodal KG enables controlled sampling of connected text-and-image evidence structures and links generated questions back to supporting edges and images.The authors say this provides control over reasoning complexity and stronger evidence-provenance guarantees than direct generation from raw documents or images.Source: §D.2 · p. 18
Loading 2609.05518v1…