PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment

Zhan-Lun Chang, Dong-Jun Han, Seyyedali Hosseinalipour, Mung Chiang, Christopher G. Brinton

arXiv:2609.08188v1cs.AIcs.IR

TL;DR

The paper addresses the gap between semantic retrieval and evidence that actually helps VLMs answer correctly, without human document-level relevance labels. It uses a two-stage generator-in-the-loop framework for text retrieval and reranking, and reports consistent gains over rank-order, random, and REPLUG-style baselines across two VQA datasets and multiple objectives.

  • Problem

    Standard retrievers and rerankers optimize semantic similarity even when retrieved documents fail to support correct generator answers, creating a preference gap.

  • Method

    A frozen VLM generates a hypothetical text rationale for dense retrieval and labels candidate documents by answer correctness to train a LoRA-adapted reranker with preference pairs.

  • Results

    Generator-guided preferences consistently outperform rank-order, random, and REPLUG-style likelihood baselines across VQA-X and A-OKVQA under multiple alignment losses.

  • Takeaways & Limitations

    Answer-level feedback from the deployed generator is a useful supervision signal for aligning reranking with downstream answer utility.

  • Takeaways & Limitations

    Pairwise training excludes queries whose candidate pools contain only helpful or only unhelpful documents, using only part of the available utility information.

Abstract

from arXiv · show

Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap: documents that appear relevant may not help the generator produce a correct answer. Motivated by this, we propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations. Our framework consists of two stages: in Stage 1, a VLM generates a hypothetical text passage from the image-query pair, which is used as the retrieval query for dense text search, bridging the image-to-text modality gap. In Stage 2, a cross-encoder reranker adapted with low-rank adaptation (LoRA) is fine-tuned using answer-supervised preference pairs mined from the frozen VLM: given the dataset answer label, a candidate document is labeled positive if the VLM produces the correct answer when given that document as context, and negative otherwise. This generator-guided signal is compatible with multiple alignment loss functions, including contrastive (triplet) loss, pairwise direct preference optimization (DPO), and supervised fine-tuning (SFT), and supports periodic re-mining to refresh preference pairs as the reranker improves. Experiments on VQA-X and A-OKVQA with Qwen3.5-2B and Qwen3-VL-4B-Instruct show that our proposed framework consistently outperforms rank-order, random, and REPLUG-style likelihood baselines under various alignment losses and pool size settings, suggesting that answer-level generator feedback is an effective supervision signal for preference alignment.

I. INTRODUCTION

RAG supplies external evidence for hallucination-prone VLMs, but semantic retrieval can select documents that fail to support correct answers. The paper proposes generator-guided preference alignment to train rerankers from answer labels rather than costly document-level annotations.

  • VLMs remain vulnerable to hallucinations, static knowledge cutoffs, and limited reasoning transparency despite their broad capabilities.
  • Semantic similarity does not guarantee answer utility: highly relevant documents may fail to provide evidence needed for correct generation.
  • Manual document-level relevance annotation is costly because each candidate must be judged by whether it improves generator answer correctness.
  • The framework separates multimodal query translation from generator-aligned reranking, using HyDE to create text retrieval queries and answer-supervised preference mining for reranker training.
  • Generator-guided preference mining consistently outperforms rank-order, random, and REPLUG-style likelihood mining across the considered alignment objectives.

B. Multimodal Retrieval and the Semantic Gap

MM-RAG must bridge multimodal queries to text evidence while addressing the gap between semantic relevance and generator usefulness. This paper decouples those problems by using a frozen VLM for modality translation and answer-level utility supervision.

  • MM-RAG retrieval must align images and questions with external evidence, but coarse image-text alignment can miss fine-grained objects, relationships, and structural details.
  • The framework translates each image-question pair into a hypothetical textual rationale, enabling text-only dense retrieval without training a dedicated multimodal retriever.
  • A separately trained cross-encoder reranker prioritizes documents by answer utility rather than semantic similarity after modality bridging.
  • Likelihood-based generator feedback provides an indirect proxy, whereas the proposed signal directly tests whether a candidate document helps the generator produce the ground-truth answer.
  • The supervision is generator-specific and document-level, differing from general-purpose LLM judgments or synthetic instruction data.

A. Problem Formulation

The paper formulates reranking as answer-utility alignment over text candidates retrieved from multimodal queries. Its staged architecture uses a frozen VLM for both modality conversion and utility supervision, while training only a text-based reranker.

  • A. Problem Formulation: Each query contains an image v and question q, while corpus documents provide textual evidence for the frozen generator.
  • A. Problem Formulation: The retriever forms qtext by concatenating q with a generated rationale, then retrieves a K-document candidate pool Cq.
  • A. Problem Formulation: Answer utility u(d) equals 1 exactly when the frozen generator produces the correct answer using document d as context.
  • A. Problem Formulation: The preference gap arises because semantic relevance scores can favor documents that do not support correct reasoning.
  • A. Problem Formulation: The learning objective is to train reranker parameters without human-annotated document relevance labels, using utility computed from generator outputs and dataset answers.
  • B. Architecture Overview: Stage 1 is optimized for broad corpus recall, whereas Stage 2 learns which recalled evidence improves the frozen generator’s answer behavior.
  • B. Architecture Overview: The frozen generator alone processes images, translating image-question pairs and evaluating candidate usefulness; the cross-encoder operates exclusively on text.

C. Datasets and Document Corpora

The datasets unify each multimodal query as an image-question pair and use dataset-specific textual explanations as retrieval documents. Stage 1 converts the multimodal query into text, retrieves candidates, and passes them to a generator-aligned reranking stage.

  • A-OKVQA uses image-question queries and rationale documents containing background information needed for commonsense or world-knowledge answers.
  • VQA-X uses image-question queries and textual justifications as retrievable evidence for answer explanations.
  • HyDE generates an image-grounded rationale that is concatenated with the question to form a text-only retrieval query.
  • The text query is encoded for dense retrieval, with corpus documents indexed in their entirety and candidate sets cached for Stage 2.
  • Stage 2 replaces indirect semantic or likelihood supervision with feedback on whether each candidate enables the frozen generator to produce the ground-truth answer.
  • Preference mining evaluates only the top-K′ candidate prefix and labels documents according to generator correctness, while rank-order and random strategies provide controlled alternatives.

IV. RERANKER ALIGNMENT AND PREFERENCE MINING

The alignment procedure iteratively mines generator-informed preference pairs, fine-tunes the reranker with LoRA, and refreshes the pairs as the reranker changes. Generator evaluation uses answer correctness and early exit to limit repeated VLM computation.

  • The procedure combines generator-guided preference mining, cost-reduction mechanisms, and periodic preference re-mining.
  • Iterative alignment: The iterative loop mines a preference dataset, fine-tunes the reranker for consecutive epochs, and then re-mines pairs using the updated reranker.
  • Preference mining: For each query, the frozen VLM evaluates candidates in reranker-score order and selects the first document producing a correct answer as d+ and the first producing an incorrect answer as d−.
  • Training and evaluation: The alignment procedure includes three loss functions and four preference-mining strategies.

A. Iterative Reranker Alignment

Reranker alignment initializes preference data with the base cross-encoder, periodically re-mines pairs under the current model, and updates LoRA parameters using alternative alignment losses. The objectives differ in whether they impose absolute score targets, relative margins, or reference-relative preferences.

  • Initialization: Initialization mines preference data with the unmodified base cross-encoder before LoRA adapters influence candidate ordering.
  • Iterative training: Periodic re-mining replaces stale pairs with pairs reflecting the current reranker ranking, although additional refreshes may yield diminishing returns after saturation.
  • Optimization: Each training sample supplies positive and negative logits, which are passed to the selected loss and used to update only the LoRA parameters with AdamW.
  • Alignment losses: SFT assigns target scores of 1 and 0 to positive and negative documents, respectively, imposing absolute utility targets.
  • Alignment losses: Triplet loss enforces s+ to exceed s− by a margin, with m = 0.2, and produces no loss once that margin is satisfied.
  • Alignment losses: Pairwise DPO encourages the adapted reranker to prefer d+ more strongly than the base reference reranker while anchoring updates to the initial ranking behavior.

C. Preference-Mining Strategies

Preference pairs are mined from the Stage 1 candidate pool by evaluating reranker-ranked documents with the frozen VLM and labeling them by answer correctness. The procedure caches evaluations, stops once both outcomes are found, and skips queries lacking either outcome.

  • Preference-pair construction: All mining strategies use the same pairwise format, but differ in how they assign positive and negative labels.The distinction concerns supervision strategy rather than the downstream SFT, triplet, or DPO objective.
  • Generator-guided mining: The proposed strategy evaluates the reranker’s top-K′ candidates with the frozen VLM and labels the first correct prediction positive and first incorrect prediction negative.Candidates are processed in descending reranker-score order.
  • Generator-guided mining: The algorithm terminates early after identifying both d+ and d−, reducing further VLM evaluations.A preference pair is added only when both documents are found.
  • Generator-guided mining: Candidate evaluations are cached using document-and-position keys before the VLM is queried.This persistent cache avoids repeating evaluations for the same retrieved candidate position.
  • Generator-guided mining: Queries are silently skipped when their retained candidate pool contains no correct or no incorrect VLM prediction.Such queries cannot form a complete positive-negative pair.

2) Controlled Baselines:

The controlled baselines isolate the effect of answer-utility supervision by holding the retrieval pools, reranker, generator, training procedure, and losses fixed while changing pair construction. They compare random labels, existing reranker order, and ground-truth-token likelihood against constrained answer generation.

  • Controlled comparison: All baselines share the Stage 1 pools, reranker architecture, LoRA procedure, and alignment losses, differing only in how d+ and d− are defined.This design tests whether answer-utility supervision is the source of observed gains.
  • Random baseline: The random baseline samples positive and negative documents uniformly without replacement and uses neither reranker scores nor VLM feedback.It provides a noise-sensitivity reference for the alignment losses.
  • Rank-order baseline: The rank-order baseline selects the highest-scored candidate as positive and the lowest-ranked candidate as negative without consulting the VLM.It tests whether reinforcing semantic ranking alone improves performance.
  • REPLUG-style likelihood baseline: REPLUG-L ranks candidates by average token-level log-likelihood of the ground-truth answer conditioned on the image, question, document, and preceding answer tokens.The highest-scoring document is positive and the lowest-scoring document is negative.
  • REPLUG-style likelihood baseline: Unlike generator-guided mining, REPLUG-L evaluates all K′ candidates before selection and can favor documents whose constrained answer generation would still be incorrect.Its score conditions on supplied ground-truth answer tokens rather than requiring the VLM to produce the answer.

V. EXPERIMENTS

Experiments use a shared default configuration across VQA-X and A-OKVQA, combining dense retrieval, LoRA reranking, frozen VLM generators, and controlled preference-mining comparisons. The setup reports results across multiple training-signal tables and task-specific evaluation procedures.

  • Default configuration: Stage 1 retrieves 100 candidates per query with Qwen3-Embedding-8B and FAISS, while Stage 2 uses the gte-reranker-modernbert-base cross-encoder.Preference mining defaults to the top 10 retrieved candidates, with re-mining disabled unless varied explicitly.
  • Generators and comparisons: The study evaluates frozen Qwen3-VL-4B-Instruct and Qwen3.5-2B generators under training-signal comparisons summarized in Tables III and IV.Both tables use a top-10 candidate pool and two training epochs.
  • Default configuration: The reranker is fine-tuned with LoRA using rank r = 16, scaling factor α = 32, dropout 0.1, AdamW, learning rate 5 × 10^-5, batch size 16, and 2 epochs.Pairwise DPO uses β = 0.1.
  • Experimental variations: The main comparison disables periodic re-mining, while re-mining frequency is studied separately in Table VII.The REPLUG-style baseline scores all 10 candidates using ground-truth-token log-likelihood.
  • Evaluation: A-OKVQA is evaluated as multiple choice, while decoding is deterministic greedy generation and accuracy is exact match against the correct choice index.The output is parsed first as a leading valid answer letter, with a fallback standalone-letter regex.

B. Results and Analysis

The experiments assess generator-guided supervision against alternative mining strategies, candidate-pool sizes, and periodic re-mining. Comparisons hold major system components fixed and evaluate consistency across datasets, generators, losses, and random seeds.

  • Evaluation dimensions: The evaluation covers generator-guided training signals versus alternative mining strategies, robustness to Stage 1 pool size, and periodic preference re-mining.These dimensions correspond to Tables III–VII.
  • Controlled evaluation: Direct comparison with prior retriever-generator alignment systems is difficult because they often use different tasks, corpora, generators, or trainable components.The study therefore uses controlled comparisons with fixed system components.
  • Statistical reporting: Trained results are reported as mean ± standard deviation over three random seeds that vary LoRA initialization, data order, and applicable pair sampling.Frozen generators and deterministic decoding remain unchanged across seeds.
  • Statistical reporting: Strategy quality is assessed by whether gains remain consistent across datasets, generators, alignment losses, and random seeds.The analysis considers consistency in addition to mean accuracy.

1) Generator-Guided Mining Outperforms All Baselines:

Generator-guided preference mining achieves the strongest evaluated accuracy across generators, datasets, and alignment losses. It remains effective across candidate-pool sizes, while answer-level supervision outperforms semantic ordering, random pairing, and likelihood-based alternatives.

  • The generator strategy achieves the highest accuracy in evaluated cells across both generators, datasets, and alignment losses.With Qwen3-VL-4B-Instruct, it reaches 96.27% / 87.00% under Contrastive (Triplet) and 96.11% / 87.16% under Pairwise DPO; with Qwen3.5-2B, Contrastive reaches 94.82% / 84.67%.
  • Rank-order selftraining often fails to improve over the untrained base, while random pair construction can degrade reranker accuracy.Under Contrastive training, rank-order achieves 94.72% / 85.07% versus the base 94.75% / 85.04%, while Random reaches 92.96% on VQA-X for Qwen3-VL-4B-Instruct and 89.13% for Qwen3.5-2B.
  • REPLUG-style likelihood supervision improves VQA-X but reduces A-OKVQA accuracy relative to the base cross-encoder.For Qwen3-VL-4B-Instruct, REPLUG-L reaches 95.85%, 95.79%, and 95.69% on VQA-X but only 82.03%–82.65% on A-OKVQA versus the base 85.04%.
  • Answer-utility preference mining, rather than a universally superior alignment loss, is the decisive factor; Contrastive (Triplet) is a strong default and Pairwise DPO can be comparable.Contrastive is strongest with Qwen3.5-2B, while Pairwise DPO reaches comparable accuracy with Qwen3-VL-4B-Instruct under the same generator-guided signal.
  • The generator strategy is stable across pool sizes, with larger gains for Qwen3.5-2B and strong performance even using Top 2 candidates.For Qwen3-VL-4B-Instruct, accuracy changes by 0.39 pp on VQA-X and 0.78 pp on A-OKVQA from Top 2 to Top 10; for Qwen3.5-2B, the changes are 2.06 pp and 3.22 pp, respectively.
  • Random selection has the highest pool-size variance, whereas the proposed strategy with Top 2 outperforms evaluated baselines at Top 10.For Qwen3.5-2B, Random varies from 87.73%–89.13% on VQA-X; at Top 10, the proposed strategy leads REPLUG-L by 0.42 pp / 4.97 pp for Qwen3-VL-4B-Instruct and 2.38 pp / 7.69 pp for Qwen3.5-2B on VQA-X / A-OKVQA.

3) Impact of Periodic Re-mining:

Periodic re-mining refreshes generator-derived preference pairs as the reranker evolves, generally improving accuracy over fixed preferences, though gains depend on model and dataset and can saturate. The framework also retains practical limits involving answer-label dependence, computational overhead, generator specificity, and pairwise pool filtering.

  • Impact of Periodic Re-mining: Periodic re-mining generally improves performance over a fixed preference dataset, but the benefit of higher frequency depends on the model and dataset.The study compares no re-mining, two re-mines, and four re-mines during four training epochs.
  • Impact of Periodic Re-mining: 97.45±0.09% versus 97.44±0.09% on VQA-X shows that two and four re-mines are effectively tied for Qwen3-VL-4B-Instruct.The reported seed variation indicates saturation rather than a meaningful gain from additional refreshes.
  • Impact of Periodic Re-mining: 89.58% versus 87.00% on A-OKVQA gives four re-mines the best Qwen3-VL-4B-Instruct result, a +2.58 pp improvement over no re-mining.For Qwen3.5-2B, four re-mines also produce the best results on both datasets: 95.16% on VQA-X and 84.67% on A-OKVQA.
  • Limitations: The framework still depends on answer-labeled training examples to identify documents that help the frozen generator answer correctly.Future alternatives include self-supervised consistency signals, confidence-based pseudo-labels, and other weak supervision.
  • Limitations: Preference mining adds computational cost because the VLM evaluates multiple candidates, and periodic re-mining repeats those evaluations as the reranker evolves.The paper suggests selective candidate evaluation, adaptive schedules, and lightweight surrogate models as possible reductions.
  • Limitations: The reranker is generator-specific and pairwise training excludes queries whose candidate pools lack both a helpful and an unhelpful document.These constraints limit universal transfer and discard some available utility information.
Loading 2609.08188v1…