PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment

Zhan-Lun Chang, Dong-Jun Han, Seyyedali Hosseinalipour, Mung Chiang, Christopher G. Brinton

arXiv:2609.08188v1cs.AIcs.IR

TL;DR

The paper addresses the gap between semantic retrieval and evidence that actually helps VLMs answer correctly, without human document-level relevance labels. It uses a two-stage generator-in-the-loop framework for text retrieval and reranking, and reports consistent gains over rank-order, random, and REPLUG-style baselines across two VQA datasets and multiple objectives.

Abstract

from arXiv · show

Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap: documents that appear relevant may not help the generator produce a correct answer. Motivated by this, we propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations. Our framework consists of two stages: in Stage 1, a VLM generates a hypothetical text passage from the image-query pair, which is used as the retrieval query for dense text search, bridging the image-to-text modality gap. In Stage 2, a cross-encoder reranker adapted with low-rank adaptation (LoRA) is fine-tuned using answer-supervised preference pairs mined from the frozen VLM: given the dataset answer label, a candidate document is labeled positive if the VLM produces the correct answer when given that document as context, and negative otherwise. This generator-guided signal is compatible with multiple alignment loss functions, including contrastive (triplet) loss, pairwise direct preference optimization (DPO), and supervised fine-tuning (SFT), and supports periodic re-mining to refresh preference pairs as the reranker improves. Experiments on VQA-X and A-OKVQA with Qwen3.5-2B and Qwen3-VL-4B-Instruct show that our proposed framework consistently outperforms rank-order, random, and REPLUG-style likelihood baselines under various alignment losses and pool size settings, suggesting that answer-level generator feedback is an effective supervision signal for preference alignment.

I. INTRODUCTION

RAG supplies external evidence for hallucination-prone VLMs, but semantic retrieval can select documents that fail to support correct answers. The paper proposes generator-guided preference alignment to train rerankers from answer labels rather than costly document-level annotations.

  • VLMs remain vulnerable to hallucinations, static knowledge cutoffs, and limited reasoning transparency despite their broad capabilities.Source: §I · p. 1
  • Semantic similarity does not guarantee answer utility: highly relevant documents may fail to provide evidence needed for correct generation.Source: §I · p. 1
  • Manual document-level relevance annotation is costly because each candidate must be judged by whether it improves generator answer correctness.Source: §I · p. 2
  • The framework separates multimodal query translation from generator-aligned reranking, using HyDE to create text retrieval queries and answer-supervised preference mining for reranker training.Source: §I · p. 2
  • Generator-guided preference mining consistently outperforms rank-order, random, and REPLUG-style likelihood mining across the considered alignment objectives.Source: §I · p. 2

B. Multimodal Retrieval and the Semantic Gap

MM-RAG must bridge multimodal queries to text evidence while addressing the gap between semantic relevance and generator usefulness. This paper decouples those problems by using a frozen VLM for modality translation and answer-level utility supervision.

A. Problem Formulation

The paper formulates reranking as answer-utility alignment over text candidates retrieved from multimodal queries. Its staged architecture uses a frozen VLM for both modality conversion and utility supervision, while training only a text-based reranker.

A. Problem Formulation

  • Each query contains an image v and question q, while corpus documents provide textual evidence for the frozen generator.Source: A. Problem Formulation · p. 4
  • The retriever forms qtext by concatenating q with a generated rationale, then retrieves a K-document candidate pool Cq.Source: A. Problem Formulation · p. 4
  • Answer utility u(d) equals 1 exactly when the frozen generator produces the correct answer using document d as context.Source: A. Problem Formulation · p. 4
  • The preference gap arises because semantic relevance scores can favor documents that do not support correct reasoning.Source: A. Problem Formulation · p. 4
  • The learning objective is to train reranker parameters without human-annotated document relevance labels, using utility computed from generator outputs and dataset answers.Source: A. Problem Formulation · p. 4

B. Architecture Overview

  • Stage 1 is optimized for broad corpus recall, whereas Stage 2 learns which recalled evidence improves the frozen generator’s answer behavior.Source: B. Architecture Overview · p. 5
  • The frozen generator alone processes images, translating image-question pairs and evaluating candidate usefulness; the cross-encoder operates exclusively on text.Source: B. Architecture Overview · p. 5

C. Datasets and Document Corpora

The datasets unify each multimodal query as an image-question pair and use dataset-specific textual explanations as retrieval documents. Stage 1 converts the multimodal query into text, retrieves candidates, and passes them to a generator-aligned reranking stage.

IV. RERANKER ALIGNMENT AND PREFERENCE MINING

The alignment procedure iteratively mines generator-informed preference pairs, fine-tunes the reranker with LoRA, and refreshes the pairs as the reranker changes. Generator evaluation uses answer correctness and early exit to limit repeated VLM computation.

  • The procedure combines generator-guided preference mining, cost-reduction mechanisms, and periodic preference re-mining.Source: §IV · pp. 7–8

Iterative alignment

  • The iterative loop mines a preference dataset, fine-tunes the reranker for consecutive epochs, and then re-mines pairs using the updated reranker.Source: §IV · p. 8

Preference mining

  • For each query, the frozen VLM evaluates candidates in reranker-score order and selects the first document producing a correct answer as d+ and the first producing an incorrect answer as d−.Source: §IV · p. 8

Training and evaluation

  • The alignment procedure includes three loss functions and four preference-mining strategies.Source: §IV · pp. 7–8

A. Iterative Reranker Alignment

Reranker alignment initializes preference data with the base cross-encoder, periodically re-mines pairs under the current model, and updates LoRA parameters using alternative alignment losses. The objectives differ in whether they impose absolute score targets, relative margins, or reference-relative preferences.

Initialization

Iterative training

  • Periodic re-mining replaces stale pairs with pairs reflecting the current reranker ranking, although additional refreshes may yield diminishing returns after saturation.Source: A. Iterative Reranker Alignment · p. 8

Optimization

  • Each training sample supplies positive and negative logits, which are passed to the selected loss and used to update only the LoRA parameters with AdamW.Source: A. Iterative Reranker Alignment · p. 8

Alignment losses

C. Preference-Mining Strategies

Preference pairs are mined from the Stage 1 candidate pool by evaluating reranker-ranked documents with the frozen VLM and labeling them by answer correctness. The procedure caches evaluations, stops once both outcomes are found, and skips queries lacking either outcome.

Preference-pair construction

  • All mining strategies use the same pairwise format, but differ in how they assign positive and negative labels.The distinction concerns supervision strategy rather than the downstream SFT, triplet, or DPO objective.Source: C. Preference-Mining Strategies · p. 9

Generator-guided mining

2) Controlled Baselines:

The controlled baselines isolate the effect of answer-utility supervision by holding the retrieval pools, reranker, generator, training procedure, and losses fixed while changing pair construction. They compare random labels, existing reranker order, and ground-truth-token likelihood against constrained answer generation.

Controlled comparison

  • All baselines share the Stage 1 pools, reranker architecture, LoRA procedure, and alignment losses, differing only in how d+ and d− are defined.This design tests whether answer-utility supervision is the source of observed gains.Source: 2) Controlled Baselines · p. 9

Random baseline

  • The random baseline samples positive and negative documents uniformly without replacement and uses neither reranker scores nor VLM feedback.It provides a noise-sensitivity reference for the alignment losses.Source: 2) Controlled Baselines · p. 9

Rank-order baseline

  • The rank-order baseline selects the highest-scored candidate as positive and the lowest-ranked candidate as negative without consulting the VLM.It tests whether reinforcing semantic ranking alone improves performance.Source: 2) Controlled Baselines · pp. 9–10

REPLUG-style likelihood baseline

  • REPLUG-L ranks candidates by average token-level log-likelihood of the ground-truth answer conditioned on the image, question, document, and preceding answer tokens.The highest-scoring document is positive and the lowest-scoring document is negative.Source: 2) Controlled Baselines · p. 10
  • Unlike generator-guided mining, REPLUG-L evaluates all K′ candidates before selection and can favor documents whose constrained answer generation would still be incorrect.Its score conditions on supplied ground-truth answer tokens rather than requiring the VLM to produce the answer.Source: 2) Controlled Baselines · p. 10

V. EXPERIMENTS

Experiments use a shared default configuration across VQA-X and A-OKVQA, combining dense retrieval, LoRA reranking, frozen VLM generators, and controlled preference-mining comparisons. The setup reports results across multiple training-signal tables and task-specific evaluation procedures.

Default configuration

  • Stage 1 retrieves 100 candidates per query with Qwen3-Embedding-8B and FAISS, while Stage 2 uses the gte-reranker-modernbert-base cross-encoder.Preference mining defaults to the top 10 retrieved candidates, with re-mining disabled unless varied explicitly.Source: A. Implementation Details · p. 10

Generators and comparisons

Default configuration

  • The reranker is fine-tuned with LoRA using rank r = 16, scaling factor α = 32, dropout 0.1, AdamW, learning rate 5 × 10^-5, batch size 16, and 2 epochs.Pairwise DPO uses β = 0.1.Source: A. Implementation Details · p. 10

Experimental variations

  • The main comparison disables periodic re-mining, while re-mining frequency is studied separately in Table VII.The REPLUG-style baseline scores all 10 candidates using ground-truth-token log-likelihood.Source: A. Implementation Details · p. 10

Evaluation

  • A-OKVQA is evaluated as multiple choice, while decoding is deterministic greedy generation and accuracy is exact match against the correct choice index.The output is parsed first as a leading valid answer letter, with a fallback standalone-letter regex.Source: A. Implementation Details · pp. 10–11

B. Results and Analysis

The experiments assess generator-guided supervision against alternative mining strategies, candidate-pool sizes, and periodic re-mining. Comparisons hold major system components fixed and evaluate consistency across datasets, generators, losses, and random seeds.

Evaluation dimensions

  • The evaluation covers generator-guided training signals versus alternative mining strategies, robustness to Stage 1 pool size, and periodic preference re-mining.These dimensions correspond to Tables III–VII.Source: B. Results and Analysis · p. 11

Controlled evaluation

  • Direct comparison with prior retriever-generator alignment systems is difficult because they often use different tasks, corpora, generators, or trainable components.The study therefore uses controlled comparisons with fixed system components.Source: B. Results and Analysis · p. 11

Statistical reporting

  • Trained results are reported as mean ± standard deviation over three random seeds that vary LoRA initialization, data order, and applicable pair sampling.Frozen generators and deterministic decoding remain unchanged across seeds.Source: B. Results and Analysis · p. 11
  • Strategy quality is assessed by whether gains remain consistent across datasets, generators, alignment losses, and random seeds.The analysis considers consistency in addition to mean accuracy.Source: B. Results and Analysis · p. 11

1) Generator-Guided Mining Outperforms All Baselines:

Generator-guided preference mining achieves the strongest evaluated accuracy across generators, datasets, and alignment losses. It remains effective across candidate-pool sizes, while answer-level supervision outperforms semantic ordering, random pairing, and likelihood-based alternatives.

  • The generator strategy achieves the highest accuracy in evaluated cells across both generators, datasets, and alignment losses.With Qwen3-VL-4B-Instruct, it reaches 96.27% / 87.00% under Contrastive (Triplet) and 96.11% / 87.16% under Pairwise DPO; with Qwen3.5-2B, Contrastive reaches 94.82% / 84.67%.Source: 1) Generator-Guided Mining Outperforms… · p. 11
  • Rank-order selftraining often fails to improve over the untrained base, while random pair construction can degrade reranker accuracy.Under Contrastive training, rank-order achieves 94.72% / 85.07% versus the base 94.75% / 85.04%, while Random reaches 92.96% on VQA-X for Qwen3-VL-4B-Instruct and 89.13% for Qwen3.5-2B.Source: 1) Generator-Guided Mining Outperforms… · pp. 11–12
  • REPLUG-style likelihood supervision improves VQA-X but reduces A-OKVQA accuracy relative to the base cross-encoder.For Qwen3-VL-4B-Instruct, REPLUG-L reaches 95.85%, 95.79%, and 95.69% on VQA-X but only 82.03%–82.65% on A-OKVQA versus the base 85.04%.Source: 1) Generator-Guided Mining Outperforms… · pp. 11–12
  • Answer-utility preference mining, rather than a universally superior alignment loss, is the decisive factor; Contrastive (Triplet) is a strong default and Pairwise DPO can be comparable.Contrastive is strongest with Qwen3.5-2B, while Pairwise DPO reaches comparable accuracy with Qwen3-VL-4B-Instruct under the same generator-guided signal.Source: 1) Generator-Guided Mining Outperforms… · pp. 11–12
  • The generator strategy is stable across pool sizes, with larger gains for Qwen3.5-2B and strong performance even using Top 2 candidates.For Qwen3-VL-4B-Instruct, accuracy changes by 0.39 pp on VQA-X and 0.78 pp on A-OKVQA from Top 2 to Top 10; for Qwen3.5-2B, the changes are 2.06 pp and 3.22 pp, respectively.Source: 2) Robustness to Stage 1 Candidate… · p. 12
  • Random selection has the highest pool-size variance, whereas the proposed strategy with Top 2 outperforms evaluated baselines at Top 10.For Qwen3.5-2B, Random varies from 87.73%–89.13% on VQA-X; at Top 10, the proposed strategy leads REPLUG-L by 0.42 pp / 4.97 pp for Qwen3-VL-4B-Instruct and 2.38 pp / 7.69 pp for Qwen3.5-2B on VQA-X / A-OKVQA.Source: 2) Robustness to Stage 1 Candidate… · p. 12

3) Impact of Periodic Re-mining:

Periodic re-mining refreshes generator-derived preference pairs as the reranker evolves, generally improving accuracy over fixed preferences, though gains depend on model and dataset and can saturate. The framework also retains practical limits involving answer-label dependence, computational overhead, generator specificity, and pairwise pool filtering.

Impact of Periodic Re-mining

  • Periodic re-mining generally improves performance over a fixed preference dataset, but the benefit of higher frequency depends on the model and dataset.The study compares no re-mining, two re-mines, and four re-mines during four training epochs.Source: 3) Impact of Periodic Re-mining · p. 12
  • 97.45±0.09% versus 97.44±0.09% on VQA-X shows that two and four re-mines are effectively tied for Qwen3-VL-4B-Instruct.The reported seed variation indicates saturation rather than a meaningful gain from additional refreshes.Source: 3) Impact of Periodic Re-mining · p. 12
  • 89.58% versus 87.00% on A-OKVQA gives four re-mines the best Qwen3-VL-4B-Instruct result, a +2.58 pp improvement over no re-mining.For Qwen3.5-2B, four re-mines also produce the best results on both datasets: 95.16% on VQA-X and 84.67% on A-OKVQA.Source: 3) Impact of Periodic Re-mining · p. 12

Limitations

  • The framework still depends on answer-labeled training examples to identify documents that help the frozen generator answer correctly.Future alternatives include self-supervised consistency signals, confidence-based pseudo-labels, and other weak supervision.Source: §VI · p. 13
  • Preference mining adds computational cost because the VLM evaluates multiple candidates, and periodic re-mining repeats those evaluations as the reranker evolves.The paper suggests selective candidate evaluation, adaptive schedules, and lightweight surrogate models as possible reductions.Source: §VI · p. 13
  • The reranker is generator-specific and pairwise training excludes queries whose candidate pools lack both a helpful and an unhelpful document.These constraints limit universal transfer and discard some available utility information.Source: §VI · p. 13

Related papers

Loading 2609.08188v1…