PaperlayerView exact revision on arXiv

Source-linked AI summary

RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, Tat-Seng Chua

arXiv:2312.00849v2cs.CLcs.CV

TL;DR

MLLMs often generate image-ungrounded hallucinations, undermining trustworthiness in real-world applications. RLHF-V uses segment-level correctional feedback with dense direct preference optimization, reducing hallucinations and achieving strong trustworthiness with efficient data use.

  • Problem

    MLLMs frequently produce responses that are not factually grounded in associated images, making them untrustworthy for real-world applications.

  • Method

    RLHF-V collects direct segment-level corrections of hallucinations and applies dense direct preference optimization to those fine-grained preferences.

  • Results

    RLHF-V substantially improves trustworthiness across five benchmarks; using 1.4k preference data, it reduces object hallucination rate by 34.8% and surpasses LLaVA-RLHF trained on 10k data.

  • Takeaways & Limitations

    RLHF-V achieves state-of-the-art trustworthiness among open-source MLLMs and better robustness than GPT-4V against over-generalization hallucinations.

  • Takeaways & Limitations

    Traditional overall-ranking feedback remains ambiguous for complex responses, and coarse rankings make precise credit assignment difficult.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects human preference in the form of segment-level corrections on hallucinations, and performs dense direct preference optimization over the human feedback. Comprehensive experiments on five benchmarks in both automatic and human evaluation show that, RLHF-V can enable substantially more trustworthy MLLM behaviors with promising data and computation efficiency. Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the hallucination rate of the base MLLM by 34.8%, outperforming the concurrent LLaVA-RLHF trained on 10k annotated data. The final model achieves state-of-the-art performance in trustworthiness among open-source MLLMs, and shows better robustness than GPT-4V in preventing hallucinations aroused from over-generalization. We open-source our code, model, and data at https://github.com/RLHF-V/RLHF-V.

1. Introduction

RLHF-V addresses hallucinations in MLLMs by collecting fine-grained correctional human feedback and applying dense direct preference optimization. Experiments show substantially improved trustworthiness with promising data and computation efficiency.

  • Motivation: MLLM hallucinations produce responses that are not factually grounded in associated images, limiting practical use in high-stakes applications.Even GPT-4V contains obvious hallucinations in 45.9% of responses.
  • Motivation: Traditional RLHF faces annotation ambiguity because complex responses may combine advantages and flaws, while the optimal response remains unknown.Coarse-grained rankings also make it difficult to allocate credit precisely and can contribute to reward hacking and behavior degeneration.
  • RLHF-V: RLHF-V collects human feedback as direct segment-level corrections to hallucinated content, yielding clear, dense preferences and optimal responses.The procedure is designed to reduce linguistic variance and non-robust bias in feedback allocation.
  • RLHF-V: RLHF-V introduces dense direct preference optimization, which directly optimizes the policy against fine-grained preferences and gives hallucinated segments stronger feedback.DDPO is presented as an efficient supervised alternative addressing the traditional RLHF objective.
  • Results: 34.8% reduction in object hallucination rate is achieved using 1.4k preference data, surpassing LLaVA-RLHF trained on 10k preference data.Comprehensive experiments on five benchmarks report substantially enhanced MLLM trustworthiness with promising data and computation efficiency.
  • Results: RLHF-V achieves state-of-the-art trustworthiness among open-source MLLMs and better robustness than GPT-4V against hallucinations from over-generalization.The work releases its code, data, and model weights.

2. Human Preference Collection

RLHF-V collects human preference data through fine-grained corrections that distinguish preferred behavior from bias and linguistic variation. Annotators correct hallucinated segments, producing factually optimal responses and incremental preference pairs.

  • Underlying Factors: Human preference differences comprise preferred behavior Yp, shallow non-robust bias Ys, and linguistic variance Yn.Yp captures trustworthy and helpful behavior; Ys can cause reward hacking, while Yn makes preference learning more difficult.
  • Challenges: Overall ranking labels create annotation ambiguity because complex responses may contain both advantages and flaws, leaving optimal responses unknown.They also provide sparse, coarse-grained credit that can be misallocated through linguistic variance or toward non-robust bias.
  • Fine-grained Correction: Segment-level corrections directly replace hallucinated portions of flawed outputs, yielding factually optimal preferred outputs and incremental preference pairs.The procedure is intended to make annotation clearer and allocate feedback more precisely to desirable behavior.
  • Data Statistics: 1.4k prompts were annotated, producing responses averaging 64.4 words and 2.65 corrected segments.Correction types included objects (41.2%), positions (20.3%), numbers (16.5%), attributes (10.0%), actions (5.3%), and miscellaneous types (6.8%).

3. Method

RLHF-V aligns MLLM behavior using dense direct preference optimization over fine-grained human corrections, while also addressing vision-language mismatch as a source of hallucination. Its DDPO variant gives greater weight to corrected segments, and the training recipe uses higher-quality visual question-answering data to counter noisy supervision.

  • Dense Direct Preference Optimization: RLHF-V uses dense direct preference optimization to learn from fine-grained correctional human feedback.Unlike traditional RLHF, the approach directly optimizes the policy against dense preferences rather than fitting a reward model and training multiple models.
  • Dense Direct Preference Optimization: DDPO scores responses as weighted aggregations of unchanged and corrected segments, assigning stronger preference feedback to corrected content.The weighting hyperparameter γ > 1 increases the contribution of corrected segments, while N = |yu| + γ|yc| normalizes response length.
  • Mitigating Hallucination from VL Mismatch: The method targets hallucinations arising from vision-language mismatch caused by noisy text supervision and semantics-changing image augmentation.The paper identifies low-quality web or instruction-tuning text and careless image augmentation as mismatch sources.
  • Mitigating Hallucination from VL Mismatch: Posttraining on high-quality human-labeled VQAv2 data can counter noisy supervision and significantly reduce hallucination rates.The paper reports that simply fine-tuning on VQAv2 also enhances instruction-following capabilities.

4. Experiments

The experiments evaluate RLHF-V’s trustworthiness, helpfulness, data efficiency, scalability, generalization, robustness, and component contributions across multiple benchmarks. RLHF-V substantially reduces hallucinations while preserving helpfulness and generalizes across MLLMs.

  • Main Results: RLHF-V achieves state-of-the-art trustworthiness among open-source models across the reported hallucination evaluations.It outperforms strong general and hallucination-specialized models.
  • Main Results: 75.8% relative points lower hallucination on Object HalBench and 34.8% lower hallucination on MHumanEval are reported for the base model Muffin.Improvements span response-level and mention-level rates, as well as object, position, and number hallucinations.
  • Main Results: RLHF-V maintains strong helpfulness on MMHalBench, LLaVA Bench, and VQAv2, with results strong and comparable to the base model.The reported results indicate improved trustworthiness without sacrificing helpfulness.
  • Scaling Feedback Data: Hallucination rate and hallucinated-segment count decrease significantly and rapidly as RLHF-V receives more fine-grained correctional feedback.The authors interpret this trend as evidence of effective and efficient learning signals for behavior alignment.
  • Data Efficiency: Using only 200 preference data, RLHF-V reaches a hallucination rate comparable to a model trained with an order of magnitude more ranking-based preference data.The comparison uses RLHF-V’s fine-grained corrections versus 2.2k LLaVA-RLHF human preference data.
  • Generalization: RLHF-V reduces LLaVA’s hallucination count by 13.8 relative points and hallucination rate by 5.9 relative points, while OmniLMM-12B reaches 4.5% mention-level hallucination on Object HalBench.The framework is reported to improve trustworthiness across different MLLM base models.
  • Robustness: RLHF-V shows the smallest hallucination-rate change under over-generalization, outperforming GPT-4V in robustness.The authors attribute this robustness to feedback that clarifies boundaries between reasonable and excessive generalizations.
  • Ablation Study: Vanilla DPO degrades performance relative to DDPO, while instruction tuning on VQAv2 reduces hallucination and untrustworthy augmentation harms results.The ablation isolates contributions from the preference-learning algorithm, traditional annotated data, and image augmentation.

5. Related Work

Related work covers MLLM construction, behavior alignment, and hallucination reduction. RLHF-V distinguishes itself by using fine-grained correctional feedback to address limitations of ranking-based alignment and existing hallucination-mitigation methods.

  • Multimodal Large Language Models: MLLMs commonly connect visual encoders with LLMs and train through pretraining followed by instruction tuning.Pretraining learns visual-text semantic mappings, while instruction tuning enables instruction following.
  • Multimodal Large Language Models: Current MLLMs continue to suffer serious hallucinations despite extensive multimodal training efforts.Hallucinated content is not factually grounded in associated images.
  • Behavior Alignment for LLMs: Instruction tuning and RLHF are major approaches for aligning language-agent behavior with human preferences.RLHF is widely used because likelihood maximization does not fully match human preference.
  • Reducing Hallucination for MLLMs: Existing hallucination-reduction methods include negative-response instruction data, response-length limiting, iterative refinement, and traditional RLHF.These approaches face limitations including reduced helpfulness, unknown positive behaviors, label ambiguity, learning inefficiency, or complex training.
  • Reducing Hallucination for MLLMs: RLHF-V presents the first fine-grained correctional human-feedback framework for MLLM behavior alignment.It systematically addresses different hallucination sources during MLLM training.

6. Conclusion

The paper presents RLHF-V as a framework for improving MLLM trustworthiness through fine-grained correctional human feedback. Experiments report state-of-the-art trustworthiness, especially for challenging long-form responses, while maintaining helpfulness.

  • Conclusion: RLHF-V targets the practical application barrier posed by hallucinations in MLLMs through behavior alignment from fine-grained correctional feedback.The framework collects correctional feedback from human annotators.
  • Conclusion: RLHF-V achieves state-of-the-art trustworthiness, particularly on challenging long-form responses, while maintaining strong helpfulness.The authors identify future directions including MLLM-generated preferences and applying the framework to LLM hallucinations.

Contributions

The work contributes a correctional-feedback framework, human preference data, and experiments demonstrating trustworthy MLLM behavior. It also releases the code, data, and model weights.

  • Contributions: RLHF-V aligns MLLM behavior through fine-grained correctional human feedback.The framework is presented as a novel contribution.
  • Contributions: The authors collect high-quality human preference data to provide human-aligned learning signals for MLLMs.
  • Contributions: Comprehensive experiments demonstrate state-of-the-art trustworthiness among open-source MLLMs.The project releases code, data, and model weights.
  • Contributions: The study includes a zoom-in comparison of RLHF-V and GPT-4V to analyze their behaviors.

A.1. Hallucination Patterns

RLHF-V and GPT-4V differ in how hallucinations are distributed across responses: GPT-4V produces more detailed descriptions, but its hallucinations are concentrated in fewer responses.

  • GPT-4V generates 2.1 times more object mentions per response than RLHF-V.The authors attribute this difference mainly to GPT-4V’s 7.6-times-larger resolution and more powerful LLM backbone.
  • 17.3% lower hallucination rate occurs for GPT-4V than RLHF-V on the comprehensive ALL metric.Their Object and Position hallucination rates are comparable in HumanEval.
  • The top 45.6% hallucinated responses of GPT-4V contribute 75% of hallucination counts, compared with 64.6% of RLHF-V responses.This indicates that GPT-4V’s hallucinations are concentrated in a smaller subset of responses.

A.2. Distillation against GPT-4V

Distilling GPT-4V increases the model’s descriptive detail, but the paper reports that this exacerbates hallucinations when the model’s foundational capabilities cannot support the added demand.

  • 1.8 times more object mentions are produced after fine-tuning on 1.2k GPT-4V visual instruction samples.The increase is measured relative to the origin model.
  • GPT-4V distillation remarkably exacerbates hallucinations when the model’s fundamental capabilities are insufficient for the demanded detail.The authors connect this result to supervising models with instruction data beyond their foundational capabilities.
  • Visual instruction data should match the model’s foundational capability rather than simply become stronger.This conclusion is stated as a scope condition for choosing distillation targets.

B. Qualitative Results

Qualitative comparisons show that RLHF-V generally hallucinates less than open-source baselines, while GPT-4V describes more image details and is more vulnerable to over-generalization.

  • RLHF-V typically exhibits less hallucination than LLaVA-RLHF and InstructBLIP in short-form and long-form question answering.The comparison is reported across Figures 5, 6, 7, and 8.
  • GPT-4V is more descriptive about image details than RLHF-V.For example, GPT-4V mentions black dots across each tile that RLHF-V does not describe.
  • RLHF-V is more resistant to over-generalization than GPT-4V.GPT-4V falsely mentions scene-related but unseen objects such as exhaust, hood, and bottle.

C. Implementation Details

The implementation uses full-parameter training with a longer warm-up, reports a short multi-GPU training time, and evaluates against several multimodal baselines and benchmarks.

  • Less than 1 hour on 8 A100 GPUs is required for the total training process.All model parameters are trainable during training.
  • A 10% training-step warm-up is applied because it empirically makes training more stable.The same warm-up setting is used for all experiments.
  • RLHF-V is compared with LLaVA, Muffin, and LLaVA 1.5 as state-of-the-art baseline models.The cited implementation section introduces these baseline systems and their training scales.

D.2. Benchmarks

The evaluation uses Object HalBench for object hallucination, additional multimodal capability benchmarks, and qualitative comparisons that highlight different hallucination types and reasoning errors.

  • Object HalBench: Object HalBench evaluates object hallucination in detailed image descriptions using response-level and mention-level hallucination rates.The evaluation samples 300 COCO validation images, uses eight description prompts, and extracts mentioned objects with ChatGPT before calculating scores.
  • Evaluation materials: Object HalBench evaluation instructions are listed separately from the training description instructions and the object-mention extraction prompt.These materials are provided in Tables 5–7, with Table 6 listing Object HalBench instructions.
  • Multimodal capabilities: LLaVA Bench assesses multimodal conversation, detailed description, and complex reasoning using 30 image-question pairs for each capability.The benchmark passage identifies these three evaluation capabilities and their image-question-pair structure.
  • Qualitative analysis: Qualitative figures compare models by highlighting correct answers, hallucinations, scene-related hallucinations, and unreasonable reasoning or extensions.Figures 5–10 use colored highlights to distinguish these behaviors across qualitative results.
Loading 2312.00849v2…