RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness

1. Introduction

• Existing approaches rely on costly proprietary labelers, while the community lacks knowledge about high-quality open-source feedback and inference-time feedback guidance.The introduction also notes that human annotation is labor-intensive and difficult to scale across misalignment.Source: §1 · p. 1
• RLAIF-V generates responses under matched sampling conditions and evaluates atomic claims separately, targeting data efficiency and preference accuracy with open-source labelers.Its self-feedback method uses reward scores from DPO-aligned models, with length normalization to suppress bias toward shorter responses.Source: §1 · pp. 1–2
• RLAIF-V 7B reduces Object HalBench object hallucination by 80.7%, while self-labeled OmniLMM 12B reduces it by 76.8% and MHumanEval overall hallucination by 32.4%, surpassing GPT-4V.The 7B model uses feedback from LLaVA-NeXT 34B; OmniLMM 12B serves as its own labeler.Source: §1 · p. 2
• The authors report state-of-the-art trustworthiness among open-source and proprietary MLLMs and plan to release the framework’s code, data, and model weights.The introduction describes the framework as open-source feedback alignment for MLLMs.Source: §1 · p. 2

2.1. Response Generation

• Candidates generated under the same input and decoding settings share similar styles, helping preference learning focus on trustworthiness differences.The strategy samples with different random seeds while keeping the input and decoding parameters invariant.Source: §2.1 · p. 3

2.2. Feedback Annotation

• Atomic claims become yes/no questions scored by an open-source MLLM; response score is the negative count of claims with pno > pyes, and higher-scored responses form preference pairs.The process filters pairs to limit training cost and removes pairs where the preferred response is too short to constrain average length difference.Source: §2.2 · pp. 3–4

2.3. Iterative Feedback Learning

• At each iteration, the latest instruction model generates candidates, the labeler scores them, and DPO training produces the next instruction model.The updated model supplies candidates for the next round, updating the feedback distribution.Source: §2.3 · p. 4

2.4. Self-Feedback for Inference-time Scaling

• The DPO-aligned model supplies its own reward as feedback; averaging token-level scores addresses shorter-response bias before best-of-N selection.The highest-scoring response among sampled candidates is selected as the prediction.Source: §2.4 · p. 4

3.1. Experimental Setup

• Evaluation spans five trustworthiness benchmarks and MMStar helpfulness; RefoMB adds free-format tasks, while comparisons include general, feedback-trained, hallucination-focused, and proprietary baselines.RefoMB contains 120 images and 360 instructions across eight tasks; its reported dev split has 99 instructions.Source: §3.1 · pp. 4–6

3.2. Main Results

• RLAIF-V reaches state-of-the-art open-source trustworthiness and surpasses GPT-4V; Object HalBench hallucination falls 80.7% for LLaVA 1.5 and 76.8% for OmniLMM, while MHumanEval overall hallucination is 35.6%.Table 1 reports the main experimental results, including response- and mention-level hallucination rates and trustworthiness and overall win rates.Source: Table 1 · p. 5, §3.2 · p. 6

3.3. Ablation Study

• Deconfounded candidate responses yield the best performance on both tasks, while replacing them with human annotations causes significant performance loss.The authors attribute the diminished effectiveness of RLHF-V data to its limited model selection and limited hallucination-distribution similarity with LLaVA 1.5 7B.Source: Table 2 · p. 5, §3.3 · pp. 6–7
• Divide-and-conquer feedback improves human agreement and performance over direct overall response scoring, and outperforms VL-Feedback when trained on the same amount of data.The comparison includes different labeler models and measures pair agreement and Object HalBench performance.Source: Table 3 · p. 6, §3.3 · p. 7

3.4. Analysis

• Combining RLAIF-V feedback with heuristic HA-DPO or human-annotated RLHF-V feedback further improves trustworthiness on Object HalBench.Figure 3 reports response-level no-hallucination rate.Source: §3.4 · p. 7, Figure 3 · p. 7
• First-iteration feedback from OmniLMM with RLAIF-V reduces hallucinations in other MLLMs, with improvements sometimes exceeding those of the model that generated the feedback.The analysis trains LLaVA 1.5 7B and 13B, MiniCPM-V, and MiniCPM-V 2; Figure 4 reports Object HalBench and AMBER reductions.Source: Figure 4 · p. 7, §3.4 · p. 8
• At inference time, RLAIF-V reward improves Object HalBench trustworthiness for LLaVA 1.5 7B and Qwen-VL-Chat more than either model's PPL baseline.For LLaVA 1.5 7B best-of-64, length normalization shifts the average selected-response length difference from -7.7 to +3.9 words.Source: Figure 5 · p. 7, §3.4 · p. 8

Learning from Feedback.

• DPO is widely adopted for preference alignment, but multimodal methods that omit its continuous-reward perspective may obtain suboptimal outcomes.The related-work discussion also notes that DPO uses static pairwise data, motivating RLAIF-V's iterative acquisition of fresh feedback from the current model's output distribution.Source: §4 · p. 8

Feedback Collection for MLLMs.

• Existing MLLM feedback methods rely on costly human annotation or proprietary labelers, and prior AI-feedback approaches focus on image-captioning hallucinations involving three object-related categories.RLAIF-V instead uses open-source feedback across visual question answering, scene-text understanding, and image captioning.Source: §4 · p. 8

Hallucination Reduction without Feedback.

• Hallucination-reduction methods without feedback include caption rewriting, image-informed decoding, confident-object selection, distribution contrast, and logical-loop correction.The cited approaches use distinct decoding or correction strategies rather than preference feedback.Source: §4 · p. 8

• RLAIF-V enhances MLLM trustworthiness through open-source AI feedback, with experiments reporting state-of-the-art performance in both generative and discriminative trustworthiness.The paper says aligning with high-quality feedback can substantially improve trustworthiness without sacrificing performance on other tasks.Source: §5 · pp. 8–9
• RLAIF-V introduces deconfounded candidate sampling and divide-and-conquer evaluation to improve feedback efficiency and quality.It also applies length normalization to self-feedback guidance and reports that its feedback generalizes to different MLLMs.Source: §5 · pp. 8–9
• The conclusion reports that trustworthiness improves without sacrificing performance on other tasks.Source: §5 · p. 9
• Future work will explore collecting more complex model feedback for logical reasoning and complex task-solving capabilities.Source: §5 · p. 9

A. Extended Related Work

• RLAIF-V differs from related feedback methods by using iterative open-source feedback across diverse tasks, while prior methods often rely on proprietary models or heuristic preference pairs.Other hallucination-reduction work includes decoding, image-use, and response-correction approaches.Source: A. Extended Related Work · p. 13

B. RefoMB

• RefoMB contains 120 images, each paired with three instructions, and assesses eight capabilities spanning perception and reasoning.Its evaluation addresses reliability concerns in judging open-ended responses by supplying comprehensive image descriptions averaging 706 words.Source: B. RefoMB · p. 13, §B.1 · p. 13

B.1. GPT-4 as Evaluator

• GPT-4 evaluates response trustworthiness and helpfulness by comparing model and competitor responses against a comprehensive image description and instruction.GPT-4V is selected as the competitor; trustworthiness is gauged by hallucinations and helpfulness by assistance toward the instruction's goal.Source: §B.1 · p. 14

B.2.1. Image and Description Collection

• The image-description process combines prompts focused on different image aspects, followed by human correction and verification to improve coverage and accuracy.Source: §B.2.1 · p. 14

B.2. Benchmark Construction

• RefoMB images were selected from multiple benchmarks, while descriptions were drafted from GPT-4V outputs and then expanded, corrected, and verified by human annotators.Each description received verification from at least two graduate students and at least three rounds of additions and modifications.Source: §B.2.1 · p. 14

B.2.2. Instruction Design

• RefoMB instructions target eight capabilities, including fine-grained and coarse perception, OCR, creative generation, and attribute, relation, time-series, and logical reasoning.The material also reports dev/test splits of 99 and 261 images, respectively, and says the test split would be released six months after the dev split.Source: §B.2.2 · pp. 14–15

B.3. Analytical Results

• The analytical section examines RefoMB reliability and compares GPT-4 with GPT-4V as evaluators.Figure 6 presents the distribution of RefoMB instructions.Source: Figure 6 · p. 14, §B.3 · p. 15

B.3.1. Reliability Analysis of RefoMB

• RefoMB exhibits higher human agreement and more evenly distributed win and loss counts than MMHal-Bench.The comparison sampled 100 response pairs from each benchmark and asked human annotators to mark evaluation decisions as agree or disagree.Source: §B.3.1 · p. 15, Table 5 · p. 15

B.3.2. GPT-4 or GPT-4V as Evaluator

• GPT-4V's image-perception hallucinations, including errors about object existence and counts, motivate using annotated image descriptions with text-only GPT-4 evaluation.Source: §B.3.2 · p. 15

B.4. Example of Evaluation Results

• Figure 11 provides an example of a RefoMB evaluation result.Source: §B.4 · p. 15

B.5. RefoMB Dev Split Evaluation Results

• Table 6 reports dev-split trustworthiness and overall win rates for each category.Source: §B.5 · p. 15

B.6. RefoMB Test Split Evaluation Results

• Table 7 reports test-split trustworthiness and overall win rates for each category.Source: §B.6 · p. 15

C. Implementation Details

• The section introduces implementation details and experimental results for the RLAIF-V framework.Source: C. Implementation Details · p. 15

C.1. Different Combine Strategies

• REJ-C obtains better pairwise accuracy and promising hallucination reduction on Object HalBench and MHumanEval.Table 9 reports performance of different combine strategies, with human agreement and MHumanEval included.Source: §C.1 · p. 15

C.2. Response split and question generation

• The response split and question conversion models were trained on 2k examples each collected from Llama 3 70B.Data collection and fine-tuning took 1.2h and 0.5h, respectively, on an 8xA100 80G machine.Source: §C.2 · p. 16

C.3. No divide-and-conquer Feedback Collection

• For self-rewarding feedback, the open-source MLLM is asked to score a response’s holistic helpfulness and trustworthiness.The prompt is listed in Table 13.Source: §C.3 · p. 16

D.1. Response Generation Model

• RLAIF-V uses open-source model feedback without human labor, while RLHF-V relies on human annotators to identify and correct hallucinations.The authors note RLHF-V’s high cost and report that its dataset contains no responses from LLaVA 1.5 7B.Source: §D.1 · p. 16

Prompts for Descriptions Collection

• A separate prompt asks an expert to describe image components thoroughly enough for someone who has not seen the image to reconstruct its contents.Source: Prompts for Descriptions Collection · p. 17
• The image-description prompt requests comprehensive, accurate descriptions that address each question’s sub-questions and permits justified inferences.The prompt also specifies at least 700 words and gives additional instructions for mathematical problems, code, and subject-matter content.Source: Prompts for Descriptions Collection · p. 17

D.2. Hallucination Distribution

• The RLHF-V dataset’s concentration on two models limits shared hallucination distributions, while correlations between some model pairs are negative.The cited pairs are Mufin [65] and LLaVA 1.5 7B, and LLaVA 1.0 and LLaVA 1.5 7B.Source: §D.2 · p. 16, Prompts for Descriptions Collection · p. 17

• The paper provides additional qualitative results in this section, shown in Figures 12 and 13, to better reflect the effectiveness of its method.Source: E. Qualitative Results · p. 17
• Table 9 presents performance comparisons among different combine strategies, including human agreement and MHumanEval.Source: Table 9 · p. 18
• Table 10 reports the proportions of responses generated by different MLLMs in the RLHF-V dataset.Source: Table 10 · p. 18

F. Potential Impact and Limitations

• RLAIF-V targets trustworthy visual-language conversation by enabling open-source MLLMs to learn from peer feedback rather than relying on proprietary MLLMs or human feedback.The authors hope the framework will help community teams make MLLMs more trustworthy.Source: F. Potential Impact and Limitations · p. 18
• Training MLLMs incurs costs, and RLAIF-V models still hallucinate despite marked improvement.The authors identify further improvement in trustworthiness as an area for exploration.Source: F. Potential Impact and Limitations · p. 18
• The authors state that broader MLLM use could have either positive or negative impacts as AI tools.Source: F. Potential Impact and Limitations · p. 18

Prompts for Self-Rewarding Feedback Collection

• Table 13 contains prompts for collecting self-rewarding feedback.Source: Table 13 · p. 27