Source-linked AI summary
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, Huaxiu Yao
TL;DR
VLLMs can hallucinate image-ungrounded answers because their separately trained vision and language representations are not perfectly aligned. POVID uses ground-truth answers with AI-generated dispreferences in DPO, reducing hallucinations and improving performance across broad benchmarks.
Problem
VLLMs can generate content that is not grounded in the image because image-text representations remain insufficiently aligned.
Method
POVID treats ground-truth responses as preferred and generates dispreferred responses by injecting GPT-4V hallucinations or distorting images, then trains with DPO.
Results
POVID reduces hallucinations and improves performance across broad hallucination, comprehensive, and open-sourced VLLM benchmarks.
Takeaways & Limitations
AI-generated dispreferences provide a scalable alternative to human feedback for aligning image and text modalities in VLLMs.
Abstract
from arXiv · showhide
Instruction-following Vision Large Language Models (VLLMs) have achieved significant progress recently on a variety of tasks. These approaches merge strong pre-trained vision models and large language models (LLMs). Since these components are trained separately, the learned representations need to be aligned with joint training on additional image-language pairs. This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations. In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. First, we prompt GPT-4V to inject plausible hallucinations into the correct answer. Second, we distort the image to trigger the inherent hallucination behavior of the VLLM. This is an automated approach, which does not rely on human data generation or require a perfect expert, which makes it easily scalable. Finally, both of these generation strategies are integrated into an RLHF pipeline via Direct Preference Optimization. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches. Our data and code are available at https://github.com/YiyangZhou/POVID.
1. Introduction
VLLMs can hallucinate image-ungrounded content despite capable vision and language components, motivating modality alignment through preference tuning. POVID uses AI-generated dispreferences and reports reduced hallucination alongside improved VLLM performance.
- VLLMs may generate imagined objects, scenes, incorrect spatial relationships, or categories despite high-quality visual features and factual language models.
- The paper attributes VLLM hallucinations to insufficient alignment between image and text modalities and reduced attention to image information.
- Prior preference methods may compare two incorrect responses, while corrective feedback approaches can require costly human data gathering.
- POVID uses ground-truth multimodal instructions as preferred answers and generates dispreferred responses by injecting plausible hallucinations with GPT-4V or adding image noise.
- 12.4% average improvement is reported over other VLLM preference-tuning methods, alongside reduced hallucinations and better modality alignment.
2. Preliminaries
The preliminaries define VLLMs as autoregressive multimodal models and describe preference-based optimization. DPO learns from preferred and dispreferred responses using a fixed preference dataset, while POVID frames preference generation around hallucination mitigation.
- Vision Large Language Models: VLLMs autoregressively predict text responses from prompts containing both images and text, supporting tasks such as image captioning and VQA.
- Direct Preference Optimization: RL fine-tuning maximizes expected output rewards while a KL-divergence term limits deviation from a reference policy.
- Direct Preference Optimization: The objective uses a prompt dataset D, with β controlling the balance between its terms.
- Direct Preference Optimization: DPO simplifies preference optimization by learning from preferred and dispreferred responses in a fixed dataset.
- POVID framework: POVID's preference-generation framework separates hallucinating textual responses from triggering inherent hallucination patterns during training.
3. Constructing Preferences to Aligning Modalities in VLLMs
POVID constructs preference pairs for VLLMs by retaining ground-truth answers as preferred responses and generating dispreferred responses through hallucinated text or distorted images. These pairs are integrated into DPO fine-tuning to improve alignment between image and text modalities.
- Constructing Preferences: POVID uses original ground-truth answers as preferred responses and generates dispreferred responses with two complementary strategies.The strategies hallucinate textual answers and trigger inherent hallucination patterns through image distortion.
- Hallucinating Textual Responses: GPT-4V transforms correct answers into dispreferred responses by injecting plausible hallucinations tailored to captioning, VQA, and reasoning tasks.Captioning perturbations include object co-occurrence, logical relationships, and attributes; reasoning perturbations alter logical relationships or entity information.
- Mitigating Inherent Hallucination Patterns: Image distortion introduces noise that disrupts visual understanding and encourages responses based more on textual context or learned object associations.The noise level should remain within a reasonable range so the image remains interpretable while inherent hallucination patterns are activated.
- Mitigating Inherent Hallucination Patterns: Under noisy conditions, the model may assign higher likelihood to “fork” than the ground-truth “plate” after the prompt mentions a knife.With increased noise, “pixel” can become predominant because of visible noise patterns in the image.
- Preference Training: POVID combines textual-hallucination and distorted-image dispreferred responses in a reformulated DPO loss.The loss uses coefficients α, β1, and β2 to balance preferred and dispreferred terms, including responses generated from noisy images.
4. Experiment
POVID is evaluated against preference-tuning methods and open-source VLLMs across hallucination, comprehensive, ablation, fine-grained, and modality-alignment analyses. It reduces hallucinations, improves broad VLLM performance, and shifts attention toward image information.
- Comparison with Different Preferences in VLLMs: POVID improves hallucination benchmarks by 31.78% across the evaluated measures, reducing hallucinations in generated responses.The comparison evaluates POVID against other preference construction approaches on hallucination and comprehensive benchmarks.
- Comparison with Open-Sourced VLLMs Models: POVID outperforms other open-source VLLMs on five of eight benchmarks, while Qwen-VL-Chat leads on two.The comparison includes models using different image and text encoders.
- Ablation Studies: Image distortion improves performance across all ablation benchmarks, while hallucinated responses from ground truth also significantly enhance performance.Combining both strategies yields the best performance.
- Fine-grained Performance Analysis: POVID excels at image captioning and detailed descriptions on LLaVA-Bench, while consistently outperforming alternative preference strategies across other categories.The fine-grained analysis covers conversation, detail description, and complex reasoning.
- Modality Alignment Analysis: Attention-map analysis shows that POVID prioritizes image information more than LLaVA-1.5, whose text-context emphasis can produce hallucinations.The comparison focuses on image captioning and VQA tasks and tracks attention assigned to generated textual tokens over image-text inputs.
5. Related Work
Related work covers methods for developing VLLMs, improving image-text alignment, and applying preference alignment to reduce hallucinations.
- VLLMs and VLLM Hallucination: VLLMs combine large-scale pretrained vision models with LLMs and require alignment between image and text modalities.Prior work explores instruction tuning, larger training datasets, and local feature enhancement for better alignment.
- Preference Alignment: Preference alignment includes feedback-based approaches using human or AI-generated feedback and approaches based on prompt guidance.
- Preference Alignment: LLaVA-RLHF uses human-annotated preferences to reduce hallucinations, while preference distillation targets improved VLLM capabilities.
6. Conclusion
POVID addresses modality alignment by generating dispreferred responses with GPT-4V and distorted images, then integrating them through Direct Preference Optimization. Across multiple benchmarks, it mitigates hallucinations while improving overall model performance.
- Conclusion: POVID generates dispreferred responses by injecting plausible hallucinations with GPT-4V and using distorted images to trigger inherent VLLM hallucinations.Both response-generation strategies are integrated into an RLHF framework through Direct Preference Optimization.
- Conclusion: POVID mitigates hallucinations and boosts overall model performance across multiple benchmarks.
- Conclusion: The approach addresses challenges in modality alignment for large vision-language models.
A.1. Training Setup
The training setup uses separate training phases and specifies a diffusion-noise adjustment formula for the second phase.
- Training phases: The first training phase runs for 3 epochs, while the second phase runs for 1 epoch.
- Compute: Training takes 20 hours on one A100 80G GPU.
- Noise adjustment: The second phase adjusts diffusion noise ξ using a sigmoid-based formula with bounds 10^-5 and 0.5 × 10^-2.The formula is ξ = Sigmoid(l_t) × (0.5 × 10^-2 − 10^-5) + 10^-5.
A.2. Construction of the Dispreference Dataset
The dispreference dataset construction uses prompts for image captioning and reasoning tasks, with training hyperparameters documented separately in Table 5.
- Prompt design: Prompts are designed to elicit dispreference patterns in image captioning and reasoning tasks.
- Task categories: Image-captioning prompts provoke imaginative descriptions, while reasoning prompts stimulate inferential thought processes.
- Training documentation: Table 5 presents the training hyperparameters.
B. Details about baselines
The paper evaluates multitasking with several benchmarks covering perception, reasoning, recognition, OCR, spatial awareness, language generation, math, and domain adaptability. It also documents two GPT-4V prompt types whose outputs are structured as image, preferred data, and dispreferred data.
- MME measures perception and cognition across 14 subtasks.
- MMBench assesses perception and reasoning through top-level ability dimensions and their specific constituent abilities.
- MM-Vet evaluates recognition, knowledge, OCR, spatial awareness, language generation, and math.
- LLaVA-Bench tests complex tasks and domain adaptability using 24 diverse images and 60 questions paired with manual descriptions.
- Table 6 presents two GPT-4V prompt types producing records formatted as image, preferred data, and dispreferred data.