PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models

Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, Feng Zheng

arXiv:2407.11422v1cs.CV

TL;DR

LVLM instruction tuning often lacks fine-grained reasoning supervision, leaving models to learn from responses without their rationales. The paper adds positive and negative rationale learning through reflective instruction tuning and the REVERIE dataset. Experiments report gains across multiple benchmarks, including a 12.7-point improvement on POPE over a specified baseline.

  • Problem

    Many LVLM instruction-tuning datasets provide direct responses without intermediate rationales, leaving fine-grained reasoning supervision underexplored.

    Source: §1 · pp. 2–3

  • Method

    Reflective instruction tuning trains LVLMs on rationales that justify correct responses and explain errors in incorrect ones, using the REVERIE dataset.

    Source: §1 · p. 3

  • Results

    83.8 on POPE, a 12.7-point improvement over baseline LLaVA-1.0-7b-lora; reflective tuning with REVERIE also yields gains across multiple LVLM benchmarks.

    Source: §1 · p. 4, §5.1 · p. 9

  • Takeaways & Limitations

    The experiments support incorporating fine-grained reflective rationales into visual instruction tuning as an approach to mitigating hallucinations.

    Source: §6 · p. 14

Abstract

from arXiv · show

Large vision-language models (LVLMs) have shown promising performance on a variety of vision-language tasks. However, they remain susceptible to hallucinations, generating outputs misaligned with visual content or instructions. While various mitigation strategies have been proposed, they often neglect a key contributor to hallucinations: lack of fine-grained reasoning supervision during training. Without intermediate reasoning steps, models may establish superficial shortcuts between instructions and responses, failing to internalize the inherent reasoning logic. To address this challenge, we propose reflective instruction tuning, which integrates rationale learning into visual instruction tuning. Unlike previous methods that learning from responses only, our approach entails the model predicting rationales justifying why responses are correct or incorrect. This fosters a deeper engagement with the fine-grained reasoning underlying each response, thus enhancing the model's reasoning proficiency. To facilitate this approach, we propose REVERIE, the first large-scale instruction-tuning dataset with ReflEctiVE RatIonalE annotations. REVERIE comprises 115k machine-generated reasoning instructions, each meticulously annotated with a corresponding pair of correct and confusing responses, alongside comprehensive rationales elucidating the justification behind the correctness or erroneousness of each response. Experimental results on multiple LVLM benchmarks reveal that reflective instruction tuning with the REVERIE dataset yields noticeable performance gain over the baseline model, demonstrating the effectiveness of reflecting from the rationales. Project page is at https://zjr2000.github.io/projects/reverie.

1 Introduction

LVLM instruction tuning often supervises responses without their intermediate reasoning, a gap that may encourage shortcuts and hallucinations. Reflective instruction tuning adds positive and negative rationales, and REVERIE supplies large-scale annotated data for this approach.

1 Introduction

  • The introduction identifies missing fine-grained reasoning supervision as a gap in response-only instruction tuning that risks superficial shortcuts and hallucinatory outputs.Without rationales, models may not internalize the reasoning logic connecting instructions to responses.Source: §1 · pp. 2–3
  • Reflective instruction tuning trains models to generate rationales that justify correct responses and explain errors in incorrect ones.Positive rationales provide visual details and step-by-step reasoning; negative rationales highlight errors to aid discrimination.Source: §1 · p. 3
  • REVERIE provides 115k instructions with positive responses, an average of 1.2 negative responses, and rationales for both response types.The dataset contains 254k instruction-response-rationale training instances.Source: §1 · p. 3
  • The rationale data is placed after response prediction in a conversation format, decoupling rationale generation from response prediction.This design is intended to support reflective tuning across diverse tasks.Source: §1 · pp. 3–4
  • Experiments on two LVLMs and multiple benchmarks show performance gains over baselines, with negative rationale use adding improvement beyond positive rationale alone.The authors report this as evidence for incorporating reflective rationales into visual instruction tuning.Source: §1 · p. 4

2 Related Work

Related work covers LVLM architectures and instruction-tuning datasets, learning from reasoning steps, and hallucination mitigation. The paper positions rationale supervision as an underexplored direction for visual instruction tuning.

Large vision-language models

  • LVLMs commonly combine a visual encoder, language model, and connector, and are trained with vision-language pretraining and visual instruction tuning.Prior work also improves models through stronger perception or language models, connectors, mixture-of-experts designs, and task-specific datasets.Source: §2 · p. 4

Learning from reasoning steps

  • Reasoning-step learning has been studied in language and vision-language tasks, but rationale learning in visual instruction tuning remains underexplored because large annotated datasets are scarce.LLaVA-Instruct-158k includes some step-by-step rationales, but these are positive-only and limited to open-ended reasoning tasks.Source: §2 · p. 5

Hallucination in LVLMs

  • Hallucination research includes evaluation benchmarks and mitigation methods such as balanced instruction data, contrastive learning, and decoding strategies.The paper also focuses on dataset-based mitigation, emphasizing fine-grained reasoning supervision.Source: §2 · p. 5

3 The REVERIE Dataset

REVERIE addresses the scarcity of visual instruction data with positive and negative reflective rationales, including hard negative responses. Its construction combines multimodal annotation with consistency filtering, and the resulting dataset spans multiple image sources and task formats.

3 The REVERIE Dataset

  • REVERIE is introduced to provide large-scale visual instruction data with detailed rationales for positive and negative responses, including hard negatives.The paper says existing datasets often contain few positive rationales and omit hard-negative rationales.Source: §3 · p. 5
  • The collection pipeline uses Gemini-Vision-Pro to annotate instructions, responses, and rationales, then filters samples with inconsistent positive and negative rationales.The consistency check is performed with Gemini-Pro.Source: Figure 2 · p. 6, §3 · p. 6
  • Visual Genome images are annotated with multi-step instructions and confusing negative responses, while A-OK-VQA and ScienceQA data broaden task diversity.For multiple-choice questions, incorrect choices are included and receive negative rationales.Source: §3 · pp. 6–7
  • After filtering, REVERIE contains 71,558 images and 115,280 instructions, with 138,897 negative responses and 254,177 training instances across four task formats.The image sources include Visual Genome, COCO, and ScienceQA.Source: §3 · p. 7
  • Over 50% of REVERIE rationales exceed 25 words and contain more than 8 nouns, indicating substantial information and knowledge in the rationales.The distributions of task formats and rationale statistics are shown in Fig. 3.Source: §3 · p. 8, Figure 3 · p. 8
  • Rationale learning follows response generation in a separate conversation turn, so response prediction relies on the image and instruction rather than prior rationale context.The authors use this format to reduce interference with response prediction and improve adaptability across tasks.Source: §4 · pp. 8–9

5 Experiments

The experiments compare reflective tuning with baselines, test rationale and training-format choices, and inspect example rationales. Results favor REVERIE across several benchmarks and show that the effects vary by rationale type and context.

5.1 Main Results

  • 83.8 on POPE is 12.7 points above LLaVA-1.0-7b-lora; REVERIE-1.0-7b-lora also improves by 321.2 on MME and 27.2 on MMBench.The comparison covers six benchmarks.Source: §5.1 · p. 9, Table 2 · p. 10
  • REVERIE-1.5-7b-lora gains on ScienceQA, MME, and GQA over its baseline and remains competitive on other benchmarks.Adding ScienceQA response data alone lowered MME and GQA performance for the baseline.Source: §5.1 · p. 10
  • Using both positive and negative rationales yields the best performance on most metrics, while positive-only and negative-only training favor different MMBench scores.Positive-only training scores higher on CP and FP-C; negative-only training scores higher on AR and RR.Source: Table 3 · p. 10, §5.1 · p. 11, Table 4 · p. 11

5.2 Ablation Studies

  • Explicit guidance alone gives the best overall prompt performance across four benchmarks; excessive constraints and unfiltered inconsistent rationale data degrade performance on most benchmarks.The prompt comparison and filtering experiments are reported in Tables 6 and 7.Source: §5.2 · p. 12, Table 6 · p. 12, Table 7 · p. 12
  • Separate contexts for positive and negative rationales outperform both joint-context ordering variants across four benchmarks.The authors suggest that exposing a rationale to another rationale may introduce information leakage.Source: Table 8 · p. 12, §5.2 · p. 13

5.3 Qualitative Analysis

  • In case analyses, REVERIE produces more necessary rationale details and identifies image information that competing models miss, while baseline rationales include errors or inconsistencies.The examples include incorrect or hallucinated baseline rationales and misinterpretation of two sub-images.Source: §5.3 · pp. 13–14, Figure 4 · p. 13, Figure 5 · p. 14

6 Conclusion

The conclusion identifies missing fine-grained reasoning supervision as a contributor to LVLM hallucinations. It presents reflective instruction tuning and REVERIE as a rationale-based approach, with experiments showing improved performance and the importance of such supervision for mitigating hallucinations.

6 Conclusion

  • The paper identifies lack of fine-grained reasoning supervision during visual instruction tuning as a contributor to LVLM hallucinations.The conclusion frames this as a key contributor.Source: §6 · p. 14
  • Reflective instruction tuning trains models on rationales for both positive and negative answers, and REVERIE supplies annotations for both.REVERIE is described as the first large-scale visual instruction-tuning dataset with comprehensive positive and negative rationale annotations.Source: §6 · p. 14
  • Experiments show that reflective instruction tuning with REVERIE enhances LVLM performance, supporting the importance of fine-grained reasoning supervision in mitigating hallucinations.Source: §6 · p. 14

A Prompts for Dataset Annotation

The annotation prompts guide generation of multi-step reasoning instructions, paired positive and negative responses, and rationales that explain answer prediction using image details and relevant outside knowledge. A consistency check is used to filter rationale data.

A Prompts for Dataset Annotation

B Implementation Details

The implementation follows the LLaVA training recipes for two model variants and evaluates them on six benchmarks. Supplementary details also describe rationale-prompt variants and length-specific prompts for two source datasets.

B Implementation Details

  • Both REVERIE variants train for one epoch with global batch size 128, LoRA attention dimension 128, alpha 256, and 3% linear warmup followed by cosine decay.REVERIE-1.0 uses learning rate 2e-5; REVERIE-1.5 uses 2e-5 for its MLP connector and 2e-4 for other parameters.Source: B Implementation Details · p. 21
  • Evaluation uses POPE, GQA, ScienceQA, MMBench, MME, and MM-Vet.Source: B Implementation Details · p. 21
  • The prompt ablation compares explicit guidance, separate positive and negative prompts, and length-specific prompts; the latter accommodate shorter rationales from ScienceQA and A-OK-VQA.A-OK-VQA rationales are described as 1–2 sentences, while ScienceQA rationales often omit image-detail descriptions.Source: B Implementation Details · pp. 21–22, Table I · p. 22

C Additional Results

Reflective tuning consistently improves performance on two additional LVLMs and outperforms baselines on MMHal-Bench for both hallucination rate and average score.

Results on additional LVLMs

  • Reflective tuning consistently improves performance on MoE-LLaVA and LLaVA-Phi3, two additional models.MoE-LLaVA uses a mixture-of-experts design, while LLaVA-Phi3 incorporates a stronger LLM.Source: C Additional Results · p. 22

Results on MMHal-Bench

  • Table III omits the term “lora” to save space.The table’s results use gpt-4-turbo-2024-04-09 as evaluator.Source: Table III · p. 23
  • On MMHal-Bench, reflective tuning consistently outperforms baselines on hallucination rate and average score.Table III uses gpt-4-turbo-2024-04-09 as evaluator.Source: C Additional Results · p. 23, Table III · p. 23

D Visualizations

The visualizations show REVERIE examples and a prompt for consistency-based data filtering; the dataset rationales combine visual information, outside knowledge, and underlying logic.

Fig. A5 and Fig. A6

Fig. A4

  • Figure A4 presents a prompt for consistency-based data filtering.Source: Figure A4 · p. 24

Related papers

Loading 2407.11422v1…