Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

1 INTRODUCTION

• LVLM hallucinations—responses inconsistent with visual input—raise misinformation concerns and restrict reliability and broader real-world deployment.Source: §1 · p. 1
• Hallucinations can stem from visual-recognition deficiencies as well as language priors; complex scenes can lead to miscounts and other incorrect visual interpretations.The introduction argues that this diversity may limit how well existing contrastive decoding methods generalize.Source: §1 · pp. 1–2
• DeGF generates an image from the initial response and uses it as an auxiliary visual reference to verify or revise the response through complementary or contrastive decoding.The text-to-image model is motivated as reversing image-conditioned response generation; discrepancies between the original and generated images provide self-feedback at response and token levels.Source: §1 · p. 2
• DeGF consistently outperforms state-of-the-art approaches in mitigating LVLM hallucinations across six benchmarks.The evaluation includes LLaVA-1.5, InstructBLIP, and Qwen-VL; qualitative studies and GPT-4V-aided evaluation also report improved accuracy and detailedness on LLaVA-Bench.Source: §1 · pp. 2–3

Hallucination in LVLMs

• LVLMs extend language models to visual inputs and perform multimodal tasks, but can generate text that contradicts visual content.Source: §2 · p. 3
• Training-based mitigation includes RLHF, auxiliary supervision, negative or noisy data, and post-hoc revision, but often requires additional data and costly training.Source: §2 · p. 3
• Training-free alternatives include contrastive and auxiliary-information-guided decoding; DeGF adds recursive text-to-image feedback during decoding.Source: §2 · p. 3

Text-to-Image Synthesis

• Diffusion models produce high-quality, detailed images and show strong vision-language alignment, supporting downstream uses such as classification and semantic segmentation.Source: §2 · p. 3
• Prior work used Stable Diffusion XL-generated image pairs as fine-tuning data, whereas DeGF uses a pretrained diffusion model for feedback without modifying the LVLM.The cited prior work reports improved performance on several VQA tasks after fine-tuning with the additional data.Source: §2 · p. 3

3 METHOD

• DeGF is a training-free algorithm that recursively improves LVLM responses using text-to-image generative feedback.It generates an image from the initial response to serve as an auxiliary visual reference.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PRELIMINARY: DECODING OF LVLMS

• LVLM decoding samples each response token from a distribution conditioned on the image, query, and previously generated tokens.The process continues autoregressively until the response sequence is complete.Source: §3.1 · p. 4

3.2 VISUAL REFERENCE GENERATION

• A pre-trained diffusion model generates an auxiliary image from the initial LVLM response, visualizing its descriptions for feedback.The generated image serves as a reference for assessing the initial response.Source: §3.2 · p. 4
• ρ = −0.63 links CLIP similarity to CHAIRI on CHAIR, with lower similarity accompanying higher response-level hallucination rates.Figure 3 plots average CHAIRI by CLIP-similarity bin.Source: Figure 3 · p. 5, §3.2 · p. 5
• Hallucinatory POPE tokens show longer-tailed and higher JS-divergence density than non-hallucinatory tokens.The comparison uses the first response token for POPE’s yes-or-no questions.Source: Figure 3 · p. 5, §3.2 · p. 5, §3.3 · p. 5

3.3 SELF-CORRECTING DECODING WITH GENERATIVE FEEDBACK

• At each token, JS divergence between original- and generated-image distributions selects complementary enhancement below γ or contrastive revision at or above γ.Aligned predictions are combined; predictions with significant discrepancy use the generated image as a contrasting reference.Source: §3.3 · pp. 5–6
• The per-token distance is a symmetric measure in [0, 1] of how closely the two next-token distributions align.It is based on JS divergence between distributions conditioned on the original and generated images.Source: §3.3 · p. 5

4 EXPERIMENTS

• The experiments use LLaVA-1.5, InstructBLIP, and Qwen-VL across six benchmarks, comparing regular decoding and state-of-the-art decoding and correction baselines.The benchmarks include object-hallucination, attribute-hallucination, multimodal-understanding, fine-grained-recognition, and challenging-question evaluations.Source: §4 · p. 6

4.1 RESULTS AND DISCUSSIONS

• On POPE, DeGF achieves state-of-the-art accuracy across three LVLMs and all negative-sampling settings, with gains up to 5.24% accuracy, 6.33% precision, and 2.79% F1 over the second-best approach.The method also has a lower performance drop in popular and adversarial settings than other baselines.Source: §4.1 · pp. 7–8, Table 1 · p. 7
• On CHAIR, DeGF beats the second-best approach by 3.0% and 2.6% on CHAIRS across two LVLMs, while increasing recall and response length versus regular decoding.The maximum new-token limit is 64.Source: §4.1 · p. 8, Table 2 · p. 8
• DeGF significantly outperforms compared methods on MME-Hallucination, with total-score gains of +18.19 on LLaVA-1.5 and +21.11 on InstructBLIP, and leads in 6 of 8 settings.It also improves the challenging color subset and achieves superior MMBench performance.Source: §4.1 · p. 8, Table 3 · p. 8
• On MMVP, DeGF raises LLaVA-1.5 performance from 22.67% to 27.33%, improving fine-grained visual recognition over regular decoding.The comparison with other hallucination-mitigation methods is shown in Figure 4.Source: §4.1 · p. 9, Figure 4 · p. 9
• On LLaVA-Bench, DeGF responses are more accurate and detailed than regular decoding in the case study, and GPT-4V-aided evaluation outperforms VCD and M3ID.Figure 5 contrasts the responses; Table 4 reports GPT-4V-aided evaluation on LLaVA-1.5.Source: §4.1 · p. 9, Figure 5 · p. 9, Table 4 · p. 9

4.2 ABLATION STUDIES

• Setting γ to 0 or 1 lowers POPE accuracy by 0.6% or 1.1%, respectively, while the default γ = 0.1 is optimal in 3 of 4 evaluated metrics.The endpoints correspond to using contrastive or complementary decoding for all tokens.Source: §4.2 · pp. 9–10, Table 5 · p. 10
• Across Stable Diffusion variants, DeGF performance remains largely unaffected and improves over regular decoding; SD-v1.5 is the default for faster generation than SD-XL-v1.0 (3.8 s/image vs. 11.3 s/image).On CHAIR with LLaVA-1.5 and maximum token length 128, DeGF has 4.04× the latency and 1.21× the GPU memory use of regular decoding, but is more efficient than OPERA and HALC.Source: §4.2 · p. 10, Table 6 · p. 10, §4.3 · p. 10

5 CONCLUSION

• The efficiency comparison reports average inference latency per instance and peak GPU memory for each method on one RTX A6000 Ada GPU.Source: Table 7 · p. 10
• Across six benchmarks, DeGF consistently outperforms state-of-the-art methods in mitigating LVLM hallucinations.The method uses generated-image feedback to inform self-correcting decoding.Source: §5 · p. 10

ETHICS STATEMENT

• The work focuses on improving the reliability of current LVLMs and reports no human subjects, sensitive data, or privacy or security concerns.Broader ethical and societal implications are discussed in Appendix A.Source: ETHICS STATEMENT · p. 11

APPENDIX

• The appendix directs readers to sections on limitations and broader impacts, implementation and licensing details, additional results, case studies, and future work.Source: APPENDIX · p. 16

Limitations

• Reliance on pretrained text-to-image models adds computational complexity and generation time, potentially slowing responses and limiting real-time suitability.The authors say DeGF is better suited to settings prioritizing accuracy and reliability over speed.Source: A LIMITATIONS AND BROADER IMPACTS · p. 16

Broader Impacts

• Using generative feedback is intended to improve LVLM accuracy and reliability and reduce the spread of misinformation in multimodal systems.Source: A LIMITATIONS AND BROADER IMPACTS · p. 16

B.1 BENCHMARKS AND METRICS

• POPE evaluates object hallucinations with yes-or-no object-presence questions, while CHAIR evaluates open-ended captioning using hallucinated-object metrics and response recall and length.POPE uses random, popular, and adversarial subsets; the listed benchmarks also include MME-Hallucination, MMBench, MMVP, and LLaVA-Bench.Source: §B.1 · pp. 16–17

B.2 MORE IMPLEMENTATION DETAILS

• Implementation uses default prompts and decoding parameters, modifies prompts to elicit visual details for binary-question benchmarks, and retains CHAIR's detailed-caption prompt.For the second query, the original prompt is used with both original and generated images; β is set to 0.1 for CHAIR and 0.25 for other tasks.Source: §B.2 · pp. 17–18

B.3 DETAILS OF OTHER BASELINES

• The compared baselines are VCD, which contrasts original and distorted-image predictions; M3ID, which contrasts visual and text-only predictions; and RITUAL, which uses original and transformed images.The appendix gives implementation settings for these baselines, including VCD's 500 noise steps and RITUAL's default κ = 3.Source: §B.3 · p. 18

B.4 DATASET AND CODE LICENSING

• The appendix lists licenses for POPE, MMVP, CHAIR, LLaVA-Bench, MME-Hallucination, and the LLaVA, VCD, InstructBLIP, and RITUAL codebases.Source: §B.4 · p. 18

C MORE EXPERIMENTAL RESULTS AND ANALYSIS

• Supplementary results report CHAIR benchmark performance with maximum new-token limits of 128 and 256, and MME-Hallucination average scores across four subsets plus total scores, averaged over three random seeds.Source: Table C1 · p. 19, Table C2 · p. 19, Table C3 · p. 19

C.1 ADDITIONAL RESULTS ON CHAIR

• DeGF achieves competitive CHAIR performance across two LVLMs in long-sequence generation with maximum token limits of 128 and 256.Source: §C.1 · p. 19

C.2 FULL RESULTS ON MME-HALLUCINATION

• The MME-Hallucination results cover three LVLMs.Source: §C.2 · p. 20

C.3 FULL RESULTS ON MMBENCH

• MMBench results cover overall performance and six Level-2 abilities on the MMBench-dev set.The abilities are Logical Reasoning, Attribute Reasoning, Relation Reasoning, Fine-grained Perception (Single Instance and Cross Instance), and Coarse Perception.Source: §C.3 · p. 20, Table C4 · p. 20

• DeGF consistently outperforms other methods on MM-Vet with random sampling and greedy decoding, notably in OCR, spatial awareness, and math.Source: §C.4 · p. 20

C.5 RESULTS ON POPE USING GREEDY DECODING

• α1 = 3 and α2 = 1 consistently perform best across two benchmarks; setting them to 0 reduces the decoding approaches to standard decoding.The hyperparameters control the relative influence of the original and generated images in next-token prediction.Source: §C.5 · pp. 20–21, Table C8 · p. 22, Table C9 · p. 22
• β = 0 is suboptimal on both benchmarks; β = 0.25 works best for POPE yes-or-no questions, while β = 0.1 works best for open-ended CHAIR.Source: §C.5 · p. 21, Table C10 · p. 23

C.8 SCALING UP THE LVLMS

• With 13B LLaVA-1.5, DeGF improves performance over other approaches across all POPE subsets, while scaling from 7B to 13B does not alleviate hallucinations.Source: §C.8 · p. 21, Table C11 · p. 23

C.9 SPEEDING UP OUR APPROACH

• 10.21 seconds per instance with 10 diffusion steps and a 96-token initial-response cap maintains robust performance.Separately, reducing diffusion steps from 50 to 10 lowers latency by 2.85 seconds per instance, and reducing the initial-response cap from 128 to 96 lowers it by 0.72 seconds; further token reduction degrades performance.Source: §C.9 · p. 22, Table C12 · p. 24, Table C13 · p. 24

C.10 QUANTITATIVE ASSESSMENT OF GENERATED IMAGE QUALITY

• The SD-v1.5-generated images achieve an average CLIPScore above 30 across all four MME subsets.DALL-E 3 scores 32.0 for comparison.Source: §C.10 · pp. 22–23, Table C14 · p. 24

D.1 DETAILS ABOUT GPT-4V-AIDED EVALUATION

• GPT-4V rates open-ended LVLM responses from 1 to 10 for accuracy and detailedness, judging image consistency and response depth or specificity.Source: §D.1 · pp. 23–24

D.2 MORE QUALITATIVE RESULTS

• LLaVA-Bench case studies compare regular decoding with DeGF using LLaVA-1.5, with GPT-4V evaluations and color-coded hallucinated and accurate content.Source: §D.2 · p. 24, Figure D1 · p. 26, Figure D2 · p. 27
• Across all four MME subsets, generated images include success and failure cases, yet are described as consistently high-quality and realistic visualizations of the initial responses.Source: §D.2 · p. 24, Figure D3 · p. 28, Figure D4 · p. 29

E FUTURE WORK

• Future work proposes evaluating more LVLMs and benchmarks, including relation and multiple-object hallucination, and integrating generative feedback into instruction tuning to potentially avoid inference-time overhead.Source: E FUTURE WORK · p. 24