PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

arXiv:2309.14525v1cs.CVcs.CL

TL;DR

LMMs can generate hallucinated text when multimodal information is misaligned, motivating improved vision-language alignment. The paper augments instruction data and adapts RLHF with factual information, reporting improved performance and reduced hallucination while noting unresolved capability trade-offs.

  • Problem

    Multimodal misalignment can produce text inconsistent with associated images, creating a need for improved vision-language alignment.

  • Method

    The paper enriches vision instruction-tuning data with human-authored image-text pairs, adapts RLHF using human preferences about hallucinations, and augments the reward model with factual information.

  • Results

    LLaVA-RLHF improves human alignment, reduces hallucination, encourages truthfulness and calibration, and improves 13B performance over LLaVA-SFT+ by 3% on MMBench.

  • Takeaways & Limitations

    The approach demonstrates a data-efficient adaptation of RLHF for multimodal alignment and uses factual augmentation to counter reward hacking.

  • Takeaways & Limitations

    Applying RLHF can dampen the performance of small-sized LMMs, while optimal data mixtures, larger-model scaling, and misalignment beyond fine-tuning remain unresolved.

Abstract

from arXiv · show

Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the task of vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-language model is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 94% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We opensource our code, model, data at https://llava-rlhf.github.io.

1 INTRODUCTION

LLaVA-RLHF adapts RLHF to improve multimodal alignment and reduce hallucinations in LMMs, while Fact-RLHF augments reward signals with factual information to address reward hacking. The work also strengthens supervised training with human-annotated multimodal data and introduces MMHAL-BENCH for hallucination-focused evaluation.

  • RLHF for multimodal alignment: LLaVA-RLHF adapts RLHF to vision-language alignment by collecting human preferences that emphasize multimodal consistency and hallucination reduction.Annotators compare responses, prioritizing better alignment and fewer hallucinations; 10K preferences cost approximately $3000.
  • Factually Augmented RLHF: Fact-RLHF augments reward signals with image captions and ground-truth multiple-choice options to improve reward-model use of factual information and prevent reward hacking.The method targets cases where high reward-model scores do not improve human judgments.
  • Instruction-tuning data: High-quality human-annotated multimodal data are added to GPT-4-generated instruction data through converted VQA-v2, A-OKVQA, and Flickr30k tasks.The resulting mixture supports supervised fine-tuning of LLaVA-SFT+ models.
  • Evaluation: MMHAL-BENCH evaluates real-world multimodal alignment with varied questions spanning 12 COCO object categories and 8 task types, emphasizing hallucination penalties.The benchmark is reported to align well with human evaluations when scores are adjusted for anti-hallucinations.

2 METHOD

The method adapts multimodal RLHF to align vision-language responses, combining preference-based reinforcement learning with factual augmentation and higher-quality instruction data. It also introduces design choices to reduce hallucinations and reward hacking during training and evaluation.

  • Multimodal RLHF: Multimodal RLHF trains an SFT-initialized policy to maximize reward-model scores for image-and-prompt queries, optionally using a per-token KL penalty.The pipeline comprises supervised fine-tuning, preference modeling, and reinforcement learning.
  • Augmenting LLaVA with High-Quality Instruction-Tuning: Instruction tuning augments synthetic LLaVA data with human-annotated VQA-v2, A-OKVQA, and Flickr30k examples converted into conversational tasks.The mixture includes 83k VQA-v2 yes/no queries, 16k A-OKVQA multiple-choice questions, and 23k Flickr30k grounded captions.
  • Multimodal Preference Modeling: Human annotators compare response pairs by prioritizing multimodal alignment and minimizing hallucinations, while the reward model learns to score the preferred response higher.Preference data contains an image, prompt, two responses, and the preferred-response index.
  • Factually Augmented RLHF: The method addresses reward hacking because a weak reward model may fail to detect hallucinations, while iterative collection of fresh human feedback is costly and underuses existing annotations.Fact-RLHF instead makes existing human-annotated data and external factual information available to reward modeling.
  • Factually Augmented RLHF: Fact-RLHF augments reward-model judgment with factual information such as image captions, annotated rationales, and ground-truth answer options.The augmented inputs provide five COCO captions for general COCO questions and annotated rationales for A-OKVQA questions.
  • Factually Augmented RLHF: The training design adds symbolic correctness and response-length penalties because ground-truth options provide factual signals and verbose outputs tend to introduce more hallucinations.Correctness penalties target answers diverging from VQA-v2 or A-OKVQA ground-truth options.

3 EXPERIMENTS

The experiments evaluate LLaVA-RLHF across capability, human-alignment, and hallucination-focused benchmarks, including the newly introduced MMHAL-BENCH. Results show that high-quality supervised data improves capabilities, while RLHF primarily improves human alignment, with Fact-RLHF addressing reward hacking across alignment benchmarks.

  • MMHAL-BENCH: MMHAL-BENCH evaluates hallucinations using 96 image-question pairs spanning eight question categories and 12 object topics.Its questions are open-ended and designed around common hallucination types, including attributes, adversarial objects, comparison, counting, and spatial relations.
  • MMHAL-BENCH: MMHAL-BENCH uses adversarially designed questions selected to induce hallucinations in the original LLaVA13BX336 model and evaluates responses with GPT-4 aided by category names and human answers.The images come from OpenImages validation and test sets, while the questions are newly designed across 12 COCO object meta-categories.
  • Capability benchmarks: 52.1% on MMBench and 82.7% F1 on POPE are achieved by LLaVA-SFT+7B, improving over original LLaVA by 13.4% and 6.7%, respectively.The gains are attributed to adding high-quality human-annotated multimodal data during supervised fine-tuning.
  • Capability benchmarks: 57.5% on MMBench and 82.9% on POPE are achieved by LLaVA-SFT+13BX336, while RLHF effects on capability benchmarks are mixed across model scales.LLaVA-RLHF shows subtle degradations at 7B but improves over LLaVA-SFT+ by 3% on MMBench at 13B.
  • Human-alignment benchmarks: 2.05 at 7B and 2.53 at 13B are obtained on MMHAL-BENCH, while LLaVA-RLHF improves LLaVA-SFT+ by over 10% on LLaVA-Bench.The reported results indicate stronger human alignment than high-quality instruction data alone, which improves capability benchmarks more than human-alignment benchmarks.
  • Fact-RLHF ablation: Fact-RLHF improves both LLaVA-Bench and MMHAL-BENCH, whereas standard RLHF improves LLaVA-Bench but underperforms on MMHAL-BENCH.The paper attributes standard RLHF’s MMHAL-BENCH degradation to longer responses that exploit a naive reward model instead of reducing hallucinations.

4 RELATED WORK

Recent LMMs build on advances in LLMs by integrating visual features and fine-tuning language models, but GPT-4-generated instruction data can create image-text misalignment. The paper addresses this misalignment through RLHF.

  • Flamingo, mPLUG-Owl, QWen-VL, and LLaVA represent prominent approaches to building and scaling LMMs.
  • LLaVA pioneered GPT-4-generated vision-language tuning data, but syntactic generation can produce prevalent image-text misalignments.
  • The paper presents RLHF as the first approach addressing this multimodal misalignment.

5 DISCUSSIONS & LIMITATIONS

The discussion attributes LMM hallucinations to low-quality instruction data and behavior cloning, while identifying important boundaries for RLHF and MMHAL-BENCH. These include reduced performance in small models, unresolved scaling and modality questions, and possible benchmark incentives for evasiveness.

  • Low-quality instruction-tuning data, often synthesized by GPT-4, is identified as a salient contributor to hallucinations in LMMs.
  • Behavior cloning can condition LMMs to speculate on uncertain visual content because labelers lack insight into the model’s visual perception.
  • Applying RLHF can inadvertently dampen the performance of small-sized LMMs despite improving alignment, hallucination reduction, truthfulness, and calibration.
  • Optimal data mixtures, scaling to larger models, and misalignment during pre-training or in other modalities remain unresolved.
  • Short or evasive responses can receive high MMHAL-BENCH scores, creating a trade-off between honesty and helpfulness.

6 CONCLUSION

The paper combines human-authored multimodal data, RLHF, factual reward augmentation, and hallucination-focused evaluation to address vision-language misalignment. LLaVA-RLHF is reported to improve performance across benchmarks and support more reliable, human-aligned LMMs.

  • The authors enrich GPT-4-generated instruction data with human-authored image-text pairs to address multimodal misalignment.
  • They adapt RLHF by having human evaluators identify the more hallucinated response and training the VLM against simulated preferences.
  • Factually Augmented RLHF adds image captions and other factual information to the reward model to counter reward hacking and improve performance.
  • MMHAL-BENCH evaluates real-world LMM alignment with an emphasis on penalizing hallucinations.
  • LLaVA-RLHF, the first VLM trained with RLHF, shows a notable performance surge across benchmarks.

A SOURCE OF MULTIMODAL HALLUCINATION

Figure 3 identifies two SFT-related sources of hallucination: hallucinated GPT-4 synthesis and labelers’ uncertainty about what LMMs know or perceive. Both can encourage unsupported speculation.

  • GPT-4-synthesized instruction data can itself contain hallucinations.
  • Instruction-data labelers lack insight into what LMMs know or see, teaching them to speculate on uncertain content.

B DETAILED EVALUATION RESULTS ON MMHAL-BENCH

The appendix provides detailed evaluation results for different large multimodal models on MMHAL-BENCH in Table 6.

  • The full MMHAL-BENCH evaluation results are included in Table 6.
  • The section directs readers to Table 6 for the complete benchmark results.
  • Table 6 reports the detailed MMHAL-BENCH evaluation results for different LMMs.

C DETAILED EVALUATION RESULTS ON POPE

The appendix presents full evaluation results on POPE, with Table 7 defining the benchmark’s reported metrics and notation.

  • Table 7 contains the full evaluation results on POPE.

D AMAZON MECHANICAL TURK DESIGN FOR HUMAN FEEDBACK DATA COLLECTION

The data-collection appendix documents the crowdworker instruction template and provides few-shot examples, with examples included in Tables 8–10.

  • The crowdworker instruction template is shown in Table 2.
  • The appendix demonstrates few-shot examples provided to crowdworkers.
  • POPE accuracy denotes prediction accuracy, while “Yes” represents the probability of a positive model answer.
  • Tables 8–10 contain examples provided to the crowdworkers.

E EXAMPLE QUESTIONS OF MMHAL-BENCH

MMHAL-BENCH covers 12 common object categories and eight question types where LMMs commonly hallucinate, illustrated through example tables.

  • MMHAL-BENCH covers 12 common object categories and 8 hallucination-prone question types.
  • Question types: Object-attribute questions test incorrect descriptions of individual-object visual attributes such as color and shape.
  • Question types: Adversarial-object questions test whether models identify referred objects that do not exist in the image.
  • Question types: Comparison questions test incorrect comparisons of attributes across multiple objects.
  • Question types: Counting questions test models’ ability to count the named objects correctly.
  • Question types: Additional examples address spatial relations, image environments, holistic descriptions, and text in images.

F DETAILS ON IMPLEMENTATIONS AND HYPERPARAMETERS

The implementation details specify PPO-based RLHF settings and a GPT-4 evaluation procedure for judging multimodal responses, with hallucination defined by factual grounding in image context.

  • RLHF implementation: RLHF fine-tuning uses LoRA with rank r = 64 in both attention and feed-forward network modules.
  • PPO hyperparameters: PPO training uses batch size 512, two epochs of gradient steps, cosine-decayed learning rate 3 × 10−5, gradient clipping at 1, and four rounds of held-out RL data.The training comprises around 500 PPO steps.
  • Reward settings: Symbolic rewards penalize responses that diverge from predetermined binary or multiple-choice ground-truth options, while length penalties depend on response tokens and question type.
  • MMHAL-BENCH evaluation: MMHAL-BENCH evaluation supplies GPT-4 with image contents, the question, a human-generated answer, and the LMM response for hallucination analysis and rating.
  • Hallucination criterion: Hallucination is judged as information generated by an LMM that is not present or implied in the image or previous conversation.False claims about objects, actions, emotions, or other image details count as hallucinations.
  • Evaluation criteria: The evaluation does not require a response to answer directly, include every object, or match the human answer’s detail, provided it makes no false image-grounded claims.Detailed analysis or reasoning is encouraged when it contains no false claims.
Loading 2309.14525v1…