PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation

Chenghao Zhang, Guanting Dong, Xinyu Yang, Zhicheng Dou

arXiv:2510.17354v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Existing retrieval approaches do not provide a unified framework for mixed-modal queries and corpora, despite their prevalence in real-world settings. The paper introduces Nyx, a retriever trained with NyxQA and VLM feedback; experiments report improved retrieval and reasoning across mixed-modal scenarios and modality combinations.

  • Problem

    Existing retrieval approaches have difficulty with complex interwoven text-image structures, and a unified framework for mixed-modal queries and corpora remains undeveloped.

    Source: §1 · p. 1

  • Method

    Nyx combines a four-stage web-based NyxQA construction pipeline with contrastive pretraining on synthetic and public data and feedback-driven fine-tuning to align retrieval with downstream VLM preferences.

    Source: §1 · p. 2

  • Results

    Nyx consistently enhances retrieval accuracy and downstream reasoning in mixed-modal scenarios, with substantial improvements over unimodal and multimodal baselines across modality combinations.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The experiments underscore the promise of unified mixed-modal retrieval for next-generation URAG systems.

    Source: §6 · p. 9

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus. However, existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images). In this paper, we address the challenge of Universal Retrieval-Augmented Generation (URAG), which involves retrieving and reasoning over mixed-modal information to improve vision-language generation. To this end, we propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for URAG scenarios. To mitigate the scarcity of realistic mixed-modal data, we introduce a four-stage automated pipeline for generation and filtering, leveraging web documents to construct NyxQA, a dataset comprising diverse mixed-modal question-answer pairs that better reflect real-world information needs. Building on this high-quality dataset, we adopt a two-stage training framework for Nyx: we first perform pre-training on NyxQA along with a variety of open-source retrieval datasets, followed by supervised fine-tuning using feedback from downstream vision-language models (VLMs) to align retrieval outputs with generative preferences. Experimental results demonstrate that Nyx not only performs competitively on standard text-only RAG benchmarks, but also excels in the more general and realistic URAG setting, significantly improving generation quality in vision-language tasks.

1 Introduction

Real-world documents interleave text and images, while existing retrieval approaches and unified embedding models have difficulty handling these structures and their relationships. Nyx addresses this URAG gap with NyxQA and a two-stage retriever, and improves retrieval accuracy and downstream reasoning in mixed-modal scenarios.

1 Introduction

  • Current MRAG approaches often overlook spatial and logical relationships between images and text, limiting their ability to capture fine-grained interactions for downstream reasoning.The described approaches include divide-and-conquer and cross-modal retrieval.Source: §1 · p. 1
  • Web documents range from text and images to paired or arbitrarily interleaved content, while prior unified embedding models mainly target pure or neatly aligned inputs.The paper says MRAG applications of those embedding approaches remain largely unexplored, and URAG requires handling mixed-modal queries and corpora.Source: §1 · pp. 1–2
  • NyxQA addresses realistic URAG data scarcity with a four-stage web pipeline that builds a corpus, generates and filters QA pairs, then mines hard negatives for contrastive pretraining.The dataset includes mixed-modal QA, a corresponding document corpus, and contrastive pretraining data.Source: §1 · p. 2
  • Nyx uses contrastive pretraining with Matryoshka Representation Learning, then feedback-driven fine-tuning to align retrieval with downstream VLM preferences.The first stage uses NyxQA and public datasets; the second produces the final retriever, Nyx.Source: §1 · p. 2
  • Nyx consistently enhances retrieval accuracy and downstream reasoning performance in challenging mixed-modal scenarios.The authors describe it as suitable for URAG tasks.Source: §1 · p. 2

2 Related Work

Prior multimodal retrieval work spans dual-path, cross-modal, and embedding-based approaches, alongside VLM encoders and synthetic-data methods. The paper identifies a remaining gap: a unified retrieval framework for mixed-modal queries and corpora in URAG is undeveloped.

2 Related Work

  • MRAG strategies include dual-path retrieval [13] [40], cross-modal retrieval [6] [50], and treating multimodal documents as images [42].These strategies retrieve text, images, or image-text pairs from external corpora to support VLM generation.Source: §2 · p. 2
  • A unified URAG framework remains undeveloped despite real-world queries, corpora, and intermediate deep-search queries containing mixed-modal inputs.The cited deep-search paradigms use iterative retrieval.Source: §2 · p. 2
  • Embedding retrievers encode queries and documents in a shared space; CLIP [22] [27] [39] [51] and BLIP-2 [17] [32] support cross-modal retrieval tasks.The passage lists image-to-text, text-to-image, and image-to-image retrieval.Source: §2 · p. 2
  • VLM encoders [23] [24] [46] [55] and synthetic-data or contrastive-learning methods [2] [54] [56] [29] [43] still leave text-to-text and general interleaved URAG retrieval unsupported.MME [53] uses synthetic data for interleaved text-image retrieval on the wikiHow task; most retrievers are trained independently of downstream VLMs.Source: §2 · p. 2

3 Methodology

The methodology defines URAG over ordered text-and-image content, constructs NyxQA from web documents, and trains Nyx through mixed-modal pretraining followed by VLM-guided fine-tuning.

3.1 Problem Formulation: URAG

  • URAG retrieves relevant documents from a mixed-modal corpus for a mixed-modal query, then uses the retrieved content to guide a textual response.Mixed-modal content is an ordered sequence of textual segments and images.Source: §3.1 · pp. 2–3

3.2 NyxQA: A Dataset for URAG

  • NyxQA combines mixed-modal questions, a corresponding document corpus, and contrastive pretraining examples; its four-stage construction samples web documents, generates QA, filters and formats samples, and mines hard negatives.The pipeline uses a VLM to generate question-answer pairs from document segments and an existing retriever to mine training triplets.Source: §3.2 · p. 3
  • For sampled text-only or image-containing chunks, a VLM generates up to five context-independent QA pairs, including questions about visual content, yielding samples from pure text to multi-image questions.Image tags identify images in the chunk.Source: §3.2 · p. 3
  • Post-processing filters contextual references and image mismatches, refines QA for clarity and completeness, and formats each sample with three distractors and its gold document.The correct answer is shuffled among four options.Source: §3.2 · p. 4
  • Hard-negative mining uses each QA question and its gold document as a query-positive pair, selecting five differing documents from the top ten returned by mmE5.These examples form NyxQA's contrastive pretraining set.Source: §3.2 · p. 4

3.3 Nyx: Training Paradigm

  • Nyx is first pretrained as a general-purpose retriever on public and synthetic mixed-modal data, including NyxQA and text-only retrieval datasets.The pretraining mix includes MMEB, synthetic mmE5 triplets, HotpotQA, 2WikiMultiHopQA, and MuSiQue.Source: §3.3 · pp. 4–5
  • Nyx-pretrained is fine-tuned with VLM feedback: candidate retrieval contexts are tested by a VLM, and documents from the first context meeting a quality threshold form preference examples for contrastive training.Instances with no qualifying context are discarded.Source: §3.3 · p. 5

4 Main Experiments

The experiments evaluate Nyx in text-only and multimodal RAG settings and compare its retrieval representations on MMEB. Results show gains from VLM-feedback fine-tuning across generation and embedding evaluations.

4.1 Experimental Setup

  • Evaluation spans text-only and multimodal RAG datasets, using accuracy for multiple-choice tasks and EM/F1 for open-ended QA.For the main comparison, each retriever’s top document is paired with the question and passed to Qwen2.5-VL-7B; SciQA uses one lecture and two example retrievals.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Results on Generation Performance

  • Nyx-pretrained gains 9% on HotpotQA and 6% on Bamboogle over 11B mmE5, while Nyx raises MMQA F1 from 35.97% to 44.50% and NyxQA accuracy from 74.83% to 81.83%.Nyx achieves the best performance across the multimodal benchmarks after VLM-feedback fine-tuning.Source: §4.2 · p. 6, Table 1 · p. 6
  • On NyxQA, McNemar tests find significant retrieval-performance differences between mmE5 and Nyx-pretrained (p=0.0000) and between Nyx-pretrained and Nyx (p=0.0001).The reported test statistics are 19.0631 and 15.7538, respectively.Source: §4.2 · p. 6

4.3 Embedding Capability Analysis

  • On MMEB, adding OOD text and NyxQA data is associated with a 1.5% overall drop, while VLM-feedback fine-tuning on OOD tasks yields a 2.1% overall improvement over mmE5-Qwen2.5-3B.The improvement holds across all evaluated capabilities.Source: §4.3 · p. 7

5 Quantitative Analysis

The analysis examines training scale, retrieved-document count, generator transfer, embedding dimensions, and how retrieved evidence relates to answer correctness. Nyx shows gains across these settings, with some improvements depending on generator size and evidence quality.

5.1 Impact of Data Scale on URAG Performance

  • NyxQA accuracy improves steadily with training-data scale along a trend described as logarithmic-linear.The authors interpret this trend as evidence that increasing data scale can predictably improve URAG generalization.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Effect of Retrieved Document Count

  • Adding retrieved documents improves all three retrievers, with diminishing gains as document count rises; Nyx performs best, including with fewer documents.The experiment varies the number of documents supplied to Qwen2.5-VL-7B from 0 to 16.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Generalization Across Generators

  • Nyx outperforms direct answering across InternVL3 sizes, with gains above 0.2 points for InternVL3-2B and InternVL3-14B; performance is not monotonic with model size.The comparison appears in Figure 5(b), and the results suggest generator size alone does not reliably predict RAG performance.Source: §5.3 · pp. 7–8, Figure 5 · p. 7

5.4 Effect of MRL

  • The 1024-dimensional Nyx variant achieves accuracy comparable to the 2048-dimensional version while halving storage, and 512- and 256-dimensional versions retain strong performance.Table 3 reports NyxQA results across output dimensions.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 Impact of Retrieved Docs on Generation

  • In the MMQA case study, Nyx retrieves Felicia Day with both the correct title and visual evidence, unlike mmE5 and Nyx-pretrained.mmE5 retrieves a face-focused image-text pair; Nyx-pretrained identifies the entity but retrieves text that does not support the answer.Source: §5.5 · p. 8, Figure 7 · p. 8
  • On NyxQA, higher proportions of golden documents correspond to higher answer accuracy, while nearly half of answers remain correct with non-golden documents.The retrieval-versus-answer correctness distributions are shown as Sankey diagrams; the authors note that VLM preferences for non-golden evidence can diverge from human intuition.Source: §5.5 · p. 8, Figure 6 · p. 8

6 Conclusion

NyxQA is presented as a large-scale dataset reflecting real-world URAG, with text, images, and interleaved combinations. Nyx combines contrastive pretraining with Matryoshka Representation Learning and VLM-feedback fine-tuning, and improves over unimodal and multimodal baselines across modality combinations.

6 Conclusion

  • NyxQA is described as a large-scale, comprehensive dataset reflecting real-world URAG with text, images, and interleaved combinations.The dataset supports retrieval over arbitrarily mixed-modal questions and corpora.Source: §6 · p. 8
  • Nyx improves over unimodal and multimodal baselines across all modality combinations.The authors describe the approach as promising for next-generation URAG systems.Source: §6 · p. 9
  • Nyx is pretrained with contrastive learning and Matryoshka Representation Learning on public and synthetic data, then fine-tuned with feedback from a downstream vision-language generator.The feedback fine-tuning aligns retrieval relevance with generation utility.Source: §6 · p. 9

A Training Details

Nyx is trained from Qwen2.5-VL-3B on a single eight-GPU node, using LoRA and memory-efficiency measures. The setup specifies batch, resolution, optimizer, and distributed-training settings.

A Training Details

  • Nyx training uses Qwen2.5-VL-3B on one node with 8 NVIDIA A800-SXM4-80GB GPUs and LoRA rank 8.Each GPU processes 20 samples with gradients accumulated over 4 steps, for an effective batch size of 640; visual inputs are capped at 400×28×28 pixels.Source: A Training Details · p. 12
  • Training uses AdamW with a 1e-5 learning rate, 0.05 warmup ratio, maximum gradient norm 5.0, and contrastive temperature 0.02.The negative sampling ratio is 1; training uses bf16, gradient checkpointing, and a linear learning-rate scheduler.Source: A Training Details · p. 12
  • DeepSpeed ZeRO stage 2 partitions model states across devices to reduce memory use and support communication optimizations.The setup combines ZeRO stage 2 with gradient checkpointing and mixed precision for memory and training efficiency.Source: A Training Details · p. 12

B Baseline Retriever Models

The experiments compare Nyx with text and multimodal retrievers spanning contrastive text embeddings, joint image-text representations, and VLM-based retrieval. The baselines are presented as recent models for assessing the proposed method.

B Baseline Retriever Models

  • E5 is a text-embedding series trained contrastively with weak supervision from a curated large-scale text-pair dataset.The passage reports strong performance in zero-shot settings and after fine-tuning.Source: B Baseline Retriever Models · p. 12
  • CLIP jointly learns visual and textual representations in a shared embedding space using contrastive training on image-text pairs.The passage notes zero-shot image classification, image search, and text-to-image retrieval among its uses.Source: B Baseline Retriever Models · p. 12
  • VLM2Vec converts VLMs into a unified embedding space and uses task-specific instructions to integrate text and images into fixed-dimensional vectors.Its MMEB training spans four meta-tasks across 36 datasets.Source: B Baseline Retriever Models · p. 12
  • mmE5 is a multimodal multilingual embedding model that leverages high-quality synthetic datasets.The synthetic data covers varied tasks, modality combinations, and languages, with VLM self-evaluation and refinement.Source: B Baseline Retriever Models · p. 12
  • VisRAG-Ret embeds document images with a VLM and maps query and document images into a shared space using a bi-encoder.Directly processing document images preserves visual layout and content rather than relying on parsed text.Source: B Baseline Retriever Models · p. 12

C Dataset Details

The evaluation covers text-only retrieval and reasoning datasets as well as multimodal question-answering benchmarks. For text-only tasks, separate Wikipedia-derived training and evaluation corpora are constructed; multimodal datasets use task-specific corpora or mixed-modal collections.

C Dataset Details

  • The evaluation datasets cover text-only linguistic retrieval and reasoning alongside multimodal cross-modal understanding and MRAG.The authors describe the combined set as an evaluation framework for effectiveness and generalization.Source: C Dataset Details · p. 12
  • HotPotQA tests multi-hop questions requiring synthesis across Wikipedia articles; evaluation uses 250 held-out validation examples.The dataset contains 90,447 training samples.Source: C Dataset Details · p. 12
  • 2WikiMultihopQA combines Wikidata knowledge with Wikipedia passages and includes annotated reasoning chains; its test evaluation uses 250 examples.The dataset has 15,000 training samples.Source: C Dataset Details · p. 12
  • Bamboogle tests compositional reasoning with manually curated multi-hop questions, some requiring up to four inference steps; its test set has 125 examples..Source: C Dataset Details · p. 12
  • MuSiQue evaluates sequential multi-hop inference, where each step depends on the preceding one, using 250 held-out development examples.The dataset comprises 19,938 training examples.Source: C Dataset Details · p. 12
  • The four text-only benchmarks use separate Wikipedia-derived training and evaluation corpora built by retrieving and deduplicating top-20 passages per question with E5.The training corpus supports feedback collection; the evaluation corpus is used for final benchmarking.Source: C Dataset Details · pp. 12–13
  • MultimodalQA combines text, tables, and images into a 285,370-instance mixed-modal corpus for a task with 23,817 training and 2,411 testing examples..Source: C Dataset Details · p. 13
  • ScienceQA evaluation retrieves one lecture and two example questions as external support, using lecture and training-QA corpora.The dataset has 12,726 training examples and 4,241 testing examples.Source: C Dataset Details · p. 13

D Details for Raw QA Generation

For raw QA generation, the authors sampled 10,000 documents from Cmix and used InternVL3-78B to generate up to five context-independent question–answer pairs per document, with prompts tailored to whether documents contained images.

D Details for Raw QA Generation

  • 10,000 Cmix document instances were sampled, and InternVL3-78B was prompted to generate up to five context-independent question–answer pairs for each.The process used different generation strategies depending on whether a document contained images.Source: D Details for Raw QA Generation · p. 13
  • The raw QA process uses a text-only prompt when the input document contains only textual content, and a multimodal QA prompt when it contains both text and images.Source: D Details for Raw QA Generation · p. 13
  • Both prompt types require questions to be answerable from the given content and concise, with the text-only prompt asking for no more than five pairs.The text-only prompt also requires questions to be independent of unstated background information.Source: D Details for Raw QA Generation · p. 13
  • For documents containing text and images, questions may address images but must identify the relevant image or images explicitly.The examples distinguish image-specific questions from questions about text, and require the target text to be specified when relevant.Source: D Details for Raw QA Generation · p. 13
Loading 2510.17354v1…