PaperlayerView exact revision on arXiv

Source-linked AI summary

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

arXiv:2504.20734v5cs.CLcs.AIcs.CVcs.IRcs.LG

TL;DR

Existing RAG systems often rely on text-only or single-modality corpora, limiting their ability to address heterogeneous information needs. UniversalRAG routes each query to modality- and granularity-specific corpora, and outperforms modality-specific and unified baselines across 10 benchmarks.

  • Problem

    Existing RAG approaches mainly use text-only or single-modality corpora, limiting support for queries requiring heterogeneous knowledge sources.

  • Method

    UniversalRAG predicts relevant modality–granularity pairs, retrieves from corresponding independent corpora, and jointly uses the results to ground generation.

  • Results

    UniversalRAG outperforms modality-specific and unified baselines across 10 benchmarks spanning diverse modalities and granularities, with robust out-of-distribution performance.

  • Takeaways & Limitations

    The results support adaptive retrieval across modalities and granularities as a practical approach for grounding responses with heterogeneous external knowledge.

  • Takeaways & Limitations

    Routing accuracy may depend on noisy automatically annotated training data because existing benchmarks lack ground-truth modality and granularity labels.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, an any-to-any RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis. Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines.

1. Introduction

UniversalRAG addresses the limits of single-modality RAG by routing queries across modality- and granularity-specific corpora, enabling grounded responses to diverse information needs. It outperforms all baselines across 10 datasets while improving efficiency and maintaining out-of-distribution robustness.

  • Motivation: Existing RAG methods typically target a single corpus and modality, limiting their ability to answer queries requiring text, images, videos, or combined evidence.RAG grounds responses in query-relevant external knowledge to improve factual accuracy (Lewis et al., 2020; Gao et al., 2023; Chen et al., 2024a).
  • Method: UniversalRAG introduces modality-aware routing that predicts required modalities and retrieves from corresponding modality-specific corpora, including multiple corpora for cross-modal queries.The retrieved knowledge is jointly used to ground the generated response, avoiding the need to force all modalities into one embedding space.
  • Method: The framework organizes each modality into multiple granularity levels because overly fine entries can dilute context, whereas overly coarse entries can combine unrelated information.Its corpora include paragraphs, documents, tables, images, clips, and videos; examples include paragraph-level document segmentation and short video clips.
  • Results: Across 10 datasets spanning diverse modalities and granularities, UniversalRAG outperforms all baselines by large average margins, improves efficiency, and remains robust out of distribution.The reported gains are attributed to modality-aware retrieval and appropriate granularity selection.

2. Method

UniversalRAG routes each query to the most appropriate modality–granularity pair before targeted retrieval, avoiding modality gaps caused by unified multimodal corpora. It supports both trained and training-free routers and retrieves at resolutions matched to query complexity.

  • Modality-Aware Retrieval: Unified-space retrieval can favor text items over the required modality, whereas modality-aware routing improves the probability of retrieving content from that required modality under sufficient modality bias.Proposition 2.1 establishes this result when modality bias exceeds the semantic-relevance variance.
  • Modality-Aware Retrieval: UniversalRAG first identifies relevant modalities and then performs targeted retrieval within preserved modality-specific corpora.This two-stage design addresses the modality gap arising when heterogeneous corpora are aggregated into one shared embedding space.
  • Granularity-Aware Retrieval: UniversalRAG extends routing across multiple granularity levels, enabling retrieval of fine-grained details or broader context according to query complexity and information scope.The router predicts modality–granularity pairs, after which modality-specialized retrievers obtain content for LVLM generation.
  • Router: A central router determines the optimal modality and granularity of knowledge for each query.This routing decision is the key component enabling UniversalRAG’s universal, one-for-all retrieval framework.
  • Router: UniversalRAG supports training-based routing from benchmark-derived targets and training-free routing prompted through frontier models such as Gemini (Gemini Team, 2023).The training-based approach addresses missing ground-truth routing labels by exploiting inductive biases in benchmark tasks.

3. Experiment

UniversalRAG is evaluated across diverse modalities and granularities, where it consistently achieves the best average performance. Results support modality-aware, cross-modal, and granularity-aware routing while also indicating efficiency and generalization benefits.

  • Datasets and Methods: UniversalRAG is evaluated on a benchmark spanning seven modalities and granularities, including no-retrieval, text, table, image, clip, video, and cross-modal RAG scenarios.The experiments compare against 12 baselines organized into naïve, unimodal, unified-embedding multimodal, and UniversalRAG-related categories.
  • Overall Results: UniversalRAG consistently achieves the best average performance across diverse RAG scenarios and LVLMs, outperforming unimodal and unified-embedding baselines.Performance is highest when retrieved modality and granularity align with query information needs; mismatches cause significant degradation.
  • Effectiveness of Cross-Modal Retrieval: Cross-modal retrieval improves tasks requiring complementary evidence, including table-text reasoning in HybridQA and text-image reasoning in WebQA.These gains arise because some queries cannot be adequately addressed by routing to only one modality.
  • Effectiveness of Modality Routing: Modality routing mitigates unified-embedding modality bias: VLM2Vec-V2 and GME retrieve almost exclusively or predominantly text regardless of the required modality.The analysis uses 200 sampled queries per benchmark and compares VLM2Vec-V2, GME, and UniversalRAG with Qwen3-VL-2B.
  • Effectiveness of Multigranularity: Granularity-aware corpus selection consistently improves performance by avoiding insufficient context for multi-hop reasoning and excessive context such as irrelevant full videos.Adding granularity levels can produce further gains, but improvements are not strictly monotonic because context sufficiency trades off against noise.
  • Efficiency of Modality-Specific Retrieval: Modality- and granularity-aware routing improves efficiency by restricting retrieval to relevant sources instead of searching a unified mega-corpus, yielding sub-linear latency growth as corpus size increases.Routing overhead is small relative to the search-space reduction at scale.

4. Related Work

Prior work has extended language models and retrieval-augmented generation from text toward visual and multimodal domains, but typically assumes fixed modality or retrieval granularity. UniversalRAG instead performs query-level routing across modalities while addressing varying information specificity.

  • Large Vision Language Models: Large vision-language models extend language models to visual inputs and video using image or video encoders aligned with language representations.The progression includes CLIP-based image encoding followed by models with diverse encoders and video extensions.
  • Retrieval-Augmented Generation: Conventional RAG retrieves from textual corpora, while recent methods incorporate multimodal sources such as images and videos but assume fixed single-modality retrieval.This fixed-modality assumption limits adaptability to real-world queries requiring different knowledge sources.
  • Retrieval Granularity: Most RAG systems use fixed retrieval granularity, although query needs vary in specificity and affect performance and efficiency across text and video retrieval.UniversalRAG addresses this setting through query-level routing across modalities.

5. Conclusion

UniversalRAG retrieves knowledge from corpora with diverse modalities and granularities through adaptive routing. Its theoretical and empirical results support heterogeneous knowledge grounding across 10 benchmarks.

  • Conclusion: UniversalRAG retrieves from corpora spanning diverse modalities and granularities, addressing the limitations of modality gaps and fixed-granularity retrieval.Its modality- and granularity-aware routing dynamically selects the most suitable knowledge sources for each query.
  • Conclusion: The framework’s routing mechanism is further justified through theoretical results.
  • Conclusion: Empirical evaluations across 10 benchmarks demonstrate UniversalRAG’s potential as an adaptive solution for grounding LVLMs with heterogeneous external knowledge.The authors position it as a step toward one-for-all RAG that unifies fragmented corpus-specific RAG systems.

Limitations

UniversalRAG’s routing mechanism is central but may require high-quality training samples because existing datasets and benchmarks lack ground-truth labels for ideal modality or granularity per query.

  • The central routing mechanism may require high-quality training samples, while existing datasets and benchmarks lack labels identifying each query’s ideal modality or granularity.

Ethical Considerations · A. Additional Details on Dataset

UniversalRAG is designed to integrate with LVLMs and compatible retrieval corpora while reducing hallucinations through corpus-specific routing, but retrieved or generated content may pose privacy, harm, and bias risks. The paper also documents the experimental datasets, their target modalities and corpus/query sizes, and a 3:7 training-testing split.

  • Ethical Considerations: UniversalRAG can be integrated with any LVLM and compatible retrieval corpus through corpus-specific routing.This integration is described as reducing hallucination.
  • Ethical Considerations: Corpus-specific routing is presented as a mechanism for reducing hallucination when UniversalRAG is integrated with LVLMs and retrieval corpora.
  • Ethical Considerations: Retrieved outputs may contain private, harmful, or biased content depending on the underlying corpora.
  • Ethical Considerations: Generated outputs may likewise contain private, harmful, or biased content depending on the underlying corpora or LVLM internalized knowledge.
  • Ethical Considerations: The paper recommends safeguard mechanisms and filtering techniques during both retrieval and generation to mitigate these risks.
  • A. Additional Details on Dataset: Table 7 summarizes each experimental dataset and its corresponding knowledge corpus, including target modality type and query and corpus sizes.Each dataset is divided into a 3:7 training-to-testing ratio.

A.1. In-Domain Dataset · A.2. Out-of-Domain Dataset · A.3. Evaluation Metrics

The evaluation spans in-domain and out-of-domain benchmarks across text, tables, images, and videos, with corpus construction adapted to each modality and reasoning requirement. Performance is reported using task-appropriate accuracy, overlap, semantic-similarity, and custom metrics.

  • A.1. In-Domain Dataset: In-domain evaluation covers diverse retrieval settings, from retrieval-free MMLU (Hendrycks et al., 2021) to text, multimodal, and video benchmarks.The suite includes MMLU, Natural Questions, HotpotQA, HybridQA, MRAG-Bench, WebQA, InfoSeek, LVBench, and VideoRAG benchmarks.
  • A.1. In-Domain Dataset: Text-corpus construction matches query complexity: NQ uses paragraphs of at most 100 words, while HotpotQA groups related documents that can exceed 4K tokens for multi-hop reasoning.NQ samples 2,000 development QA pairs; HotpotQA samples 2,000 test QA pairs.
  • A.1. In-Domain Dataset: HybridQA separates tables and textual evidence to test modality-specific routing, while MRAG-Bench evaluates multimodal image queries against a corpus containing all collected images.HybridQA samples 2,000 development QA pairs, whereas MRAG-Bench evaluates all 1,353 questions.
  • A.1. In-Domain Dataset: Video evaluation includes LVBench and VideoRAG benchmarks, with queries reformulated or timestamped to support retrieval over long videos and shorter relevant clips.LVBench focuses mainly on segments shorter than five minutes and uses available videos; VideoRAG-Wiki and VideoRAG-Synth lack timestamp annotations and use GPT-4o to identify video evidence.
  • A.2. Out-of-Domain Dataset: Out-of-domain preprocessing adapts to retrieval modality and reasoning scope, using paragraph or document corpora for text, sampled image pools for Visual-RAG, and reformulated video queries for CinePile.2WikiMultiHopQA aggregates annotated candidate contexts, Visual-RAG samples five images per category, and CinePile samples 10 questions for each of 144 available videos.
  • A.3. Evaluation Metrics: Metrics are selected by answer format: Top-1 Accuracy for multiple choice, Exact Match and F1 for short answers, custom InfoSeek accuracy, and ROUGE-L or BERTScore for longer answers.Top-1 Accuracy measures correctness; Exact Match and F1 measure exact agreement and word-level overlap, while ROUGE-L and BERTScore capture sequence overlap and semantic similarity.

B. Additional Implementation Details · C. Theoretical Analyses of UniversalRAG

UniversalRAG combines visual and textual retrieval signals, uses scene detection and configurable training-based routing for implementation, and formally analyzes routing effectiveness, multigranularity, and efficiency.

  • B. Additional Implementation Details: A retrieval ensemble weights visual similarity at 0.8 and combines it with textual similarity for visual element retrieval.Textual inputs are image captions for images and scripts for videos.
  • B. Additional Implementation Details: Textual similarity uses image captions for images and scripts for videos.
  • B. Additional Implementation Details: PySceneDetect detects scene boundaries from content changes to support processing long videos.
  • B. Additional Implementation Details: Training-based routers use a lightweight classifier head with multi-label logits, multi-hot targets, and binary cross-entropy loss.
  • B. Additional Implementation Details: Routers train for 5 epochs at learning rate 2e-5 with LoRA rank r=32, then select modalities and granularities above inference threshold 0.8.
  • C. Theoretical Analyses of UniversalRAG: The theoretical analysis covers modality-routing effectiveness, multigranularity, and the efficiency of modality-aware routing.

C.1. Effectiveness of Modality Routing … D.1. Additional Results using Different LVLMs

UniversalRAG’s modality-aware routing is theoretically justified to avoid modality bias, while multigranularity improves query-specific response quality and modality-specific retrieval reduces asymptotic latency. Across 10 benchmarks with InternVL3.5-8B and Molmo2-4B, UniversalRAG outperforms all baselines and approaches Oracle performance.

  • C.1. Effectiveness of Modality Routing: When modality bias α is sufficiently large relative to semantic-relevance variance, unified retrieval has lower required-modality retrieval probability than modality-aware routing.The analysis models unified similarity as a modality-bias term plus semantic relevance and noise.
  • C.1. Effectiveness of Modality Routing: For sufficiently large α/σ, unified embedding retrieval is strictly worse than modality-specific retrieval after routing, whose top-1 required-modality probability equals routing accuracy r.The unified-retrieval bound decays to zero as α/σ increases, whereas routed retrieval retains probability r.
  • C.1. Effectiveness of Modality Routing: With |R| = |S| = 10^12, p = 0.8, and σ = 0.01, routing accuracy p ≃ 0.17 suffices for routed retrieval to outperform unified retrieval.The result underscores modality-aware routing because multimodal encoders exhibit inherent modality biases.
  • C.2. Effectiveness of Multigranularity: If different queries favor different granularities, routing each query to its preferred granularity achieves strictly higher expected response quality than any fixed-granularity policy.This theoretical result supports empirical improvements reported in Tables 4 and 9.
  • C.3. Efficiency of Modality-Specific Retrieval: For modality-granularity corpora of size N, UniversalRAG has latency C + T(N) versus T(kN) for unified retrieval and therefore achieves strictly lower asymptotic latency.The advantage yields linear-in-k speedup for exact retrieval and a constant-factor asymptotic speedup when T(m) = Θ(log m).
  • D.1. Additional Results using Different LVLMs: UniversalRAG outperforms all baselines across 10 benchmarks with InternVL3.5-8B and Molmo2-4B, achieving average scores comparable to Oracle.These results indicate robustness and generalizability across different LVLM generators.

D.2. Additional Results on Multigranularity · D.3. Detailed Results on Out-of-Domain Dataset · E. Modality Gap in Unified Embedding Space

Additional results show that multigranularity improves router performance, while UniversalRAG generalizes strongly to out-of-domain datasets. Unified multimodal embedding spaces exhibit modality-specific clustering, motivating modality alignment.

  • D.2. Additional Results on Multigranularity: Training-based routers consistently perform better when granularity is incorporated, according to results across three router models.The analysis complements results for two training-free models and examines granularity in settings without labeled data.
  • D.2. Additional Results on Multigranularity: For two training-free models, Table 4 examines how the number of granularity levels correlates with end-to-end performance.This analysis leverages UniversalRAG’s flexibility in scenarios without labeled data.
  • D.3. Detailed Results on Out-of-Domain Dataset: UniversalRAG consistently outperforms all baselines on average across out-of-domain datasets, with training-free routers showing strong generalization to unseen queries.Trained routers perform relatively worse than on in-domain datasets but remain robust.
  • D.3. Detailed Results on Out-of-Domain Dataset: Out-of-domain generation results are reported separately for each dataset, enabling detailed comparison of UniversalRAG variants and baseline methods.The passage reports average superiority overall but does not provide individual dataset values here.
  • E. Modality Gap in Unified Embedding Space: Figure 7 visualizes the unified embedding spaces of six multimodal encoders and reveals clustering by modality.Text embeddings, shown in green, are farther from embeddings of other modalities.
  • E. Modality Gap in Unified Embedding Space: Recent methods including E5-V, GME, and Qwen3-VL-Embedding aim to align modalities more closely and narrow the embedding-space gap.The passage frames these methods as responses to the modality separation observed in the PCA visualization.

F. Qualitative Results

Qualitative case studies show that UniversalRAG improves retrieval by routing queries to appropriate modalities and granularity levels, including combinations of sources when needed. The cases also reveal failures when modality requirements are ambiguous or span multiple modalities.

  • Granularity selection: Clip-level retrieval focuses on smaller, relevant video segments, avoiding the irrelevant content of full-video retrieval and missed evidence from uniformly sampled frames.This targeted retrieval produces a more accurate answer for queries requiring only a specific video segment.
  • Failure cases: Routing fails in some ambiguous or cross-modal cases, including examples where GPT-5’s prior knowledge leads its training-free prediction away from the inductive ground-truth label.These failures indicate that modality selection is not universally reliable when the information requirement is unclear or distributed across modalities.
  • Modality routing: UniversalRAG routes the WebQA query to the image modality and correctly identifies the balloons as red, white, and blue, unlike TextRAG and VideoRAG.TextRAG lacks relevant visual details, while VideoRAG is better suited to temporal reasoning tasks.
  • Granularity selection: Document-level retrieval supplies richer context for multi-entity reasoning, whereas paragraph-level retrieval can center on only one entity and produce an incorrect answer.The HotpotQA case demonstrates that appropriate text granularity is necessary when reasoning requires combining information across sources.
  • Cross-modal retrieval: UniversalRAG can select multiple modality-granularity combinations, combining paragraphs for factual information with tables for structured knowledge such as numerical values.This cross-modal capability provides more comprehensive evidence when a query primarily relies on one modality but benefits from another.
Loading 2504.20734v5…