PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment

Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arık, Tomas Pfister

arXiv:2405.18654v3cs.CV

TL;DR

MLLMs can generate descriptions of objects or attributes absent from, or unverifiable in, an image. DPA fine-tunes them using phrase-altered correct and hallucinated response pairs. Evaluations report reduced object hallucination while retaining or improving general vision-language performance.

  • Problem

    MLLMs can generate objects or attributes absent from, or unverifiable in, the input image, challenging reliable use.

    Source: §1 · p. 1

  • Method

    DPA fine-tunes MLLMs on correct and hallucinated response pairs made by selectively altering ground-truth phrases, reducing hallucinated-token likelihood relative to correct ones.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Evaluations show DPA mitigates object hallucinations while retaining or improving performance on general vision-language tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DPA mitigates object hallucinations across existence and attribute errors, visual illusions, and complex charts, and across diverse vision-language tasks.

    Source: §6 · p. 10

    For image-description hallucination evaluation, the authors also use AMBER because of limitations in descriptions.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Despite their significant advancements, Multimodal Large Language Models (MLLMs) often generate factually inaccurate information, referred to as hallucination. In this work, we address object hallucinations in MLLMs, where information is generated about an object not present in the input image. We introduce Data-augmented Phrase-level Alignment (DPA), a novel loss which can be applied to instruction-tuned off-the-shelf MLLMs to mitigate hallucinations, while preserving their general vision-language capabilities. To fine-tune MLLMs with DPA, we first generate a set of `hallucinated' and `correct' response pairs through generative data augmentation by selectively altering the ground-truth information of the correct responses at a phrase level. The DPA loss is then used to train MLLMs to reduce the likelihood of hallucinated phrases compared to the correct ones. Our thorough evaluation on various benchmarks confirms the effectiveness of DPA in mitigating hallucination while retaining the out-of-the-box performance of the MLLMs on general tasks. For instance, MLLMs finetuned with DPA, which we refer to as Hallucination Attenuated Language and Vision Assistant (HALVA), improve F1 by up to 13.4% on hallucination visual question-answering and reduce the hallucination rate by up to 4.2% on image description tasks.

1 INTRODUCTION

Object hallucination—generated descriptions of objects or attributes absent from or unverifiable in an image—challenges reliable use of MLLMs. The introduction presents DPA as phrase-level fine-tuning to reduce hallucinations while retaining general vision-language performance.

1 INTRODUCTION

  • Object hallucination includes generated objects or attributes absent from, or unverifiable in, the input image.Examples include a model describing a toothpick among utensils and affirming a tie in a wedding-cake image.Source: §1 · pp. 1–2
  • Earlier approaches address hallucination during inference, pretraining, or fine-tuning, with inference methods potentially increasing serving costs and pretraining methods requiring massive datasets.The introduction also notes that pretraining methods cannot be applied to off-the-shelf MLLMs.Source: §1 · p. 2
  • Sequence-level alignment overlooks that hallucinations are local, producing a coarse, noisy signal that can degrade general vision-language capabilities.Figure 2 contrasts HALVA with existing fine-tuning approaches, which diverge from their LLaVA-v1.57B base model.Source: §1 · p. 2, Figure 2 · p. 2
  • DPA uses selectively altered ground-truth phrases to create correct and hallucinated response pairs for fine-tuning MLLMs.The objective targets hallucinated phrases rather than penalizing every token in a response.Source: §1 · pp. 2–3
  • DPA mitigates object hallucinations while retaining or improving performance on general vision-language tasks.The paper names models fine-tuned with DPA HALVA.Source: §1 · p. 3

2 METHOD: DATA-AUGMENTED PHRASE-LEVEL ALIGNMENT (DPA)

DPA targets object hallucinations that can arise from spurious concept co-occurrences by pairing correct responses with versions containing altered ground-truth phrases. Its phrase-level alignment penalizes hallucinated tokens, while a token-wise KL regularizer constrains divergence from the model’s initial abilities.

Generative data augmentation

  • DPA augments correct responses by replacing true concepts with hallucinated ones absent from the image, while keeping the response structure intact.The altered concepts can include objects, attributes, actions, and locations.Source: §2 · p. 3, Figure 3 · p. 3
  • Hallucinated concepts come from closed-set co-occurrences in an object-centric dataset or open-set concepts prompted from an LLM; yes-or-no answers are also inverted.The resulting correct and hallucinated response pairs are used for DPA training.Source: §2 · p. 3

Proposed phrase-level loss

  • The alignment loss compares correct and hallucinated phrase likelihoods, penalizing hallucinated tokens rather than every token in the response.This localized feedback distinguishes DPA from sequence-level preference optimization methods.Source: §2 · p. 4
  • A token-wise KL regularizer using a frozen reference model limits divergence from the initial model and is combined with the alignment loss in the DPA objective.The reference and trainable models start from the same checkpoint; α controls divergence during training.Source: §2 · pp. 4–5

3 EXPERIMENT SETUP

The experiments use Visual Genome-based instruction data and fine-tune LLaVA-v1.5 and VILA-v1.5 variants as HALVA. Evaluation spans object-hallucination, other hallucination, and general vision-language benchmarks.

Training data

  • The training set contains 21.5K vision-language instructions with correct and hallucinated responses, built from Visual Genome images and annotations.Correct responses use descriptive and non-descriptive instructions and are generated with Gemini Vision Pro; hallucinated responses are created by augmentation.Source: §3 · p. 5

Implementation details

  • LLaVA-v1.5 and VILA-v1.5 are the base models; training freezes vision encoders and projection layers and updates the LLM with LoRA for one epoch.The resulting checkpoints are called HALVA; training takes 1.5–3 hours for 7B and 13B variants.Source: §3 · p. 5

Evaluation setup

  • Evaluation covers four object-hallucination benchmarks, HallusionBench, and five general vision-language benchmarks, with all evaluations run thrice and average scores reported.GPT-4-based evaluation also reports standard deviations because performance varies slightly with randomness.Source: §3 · p. 5

4 RESULTS

HALVA reduces hallucination across object-focused and broader vision-language evaluations while maintaining or improving general-task performance. The results also examine how the KL weight affects model divergence and show qualitative examples of more accurate responses.

4 RESULTS

  • On image description and visual QA, HALVA outperforms HA-DPO and EOS on hallucination; EOS also reduces description detail, and both methods underperform on general tasks.Figure 2 reports that HA-DPO and EOS substantially diverge from the base model, unlike HALVA.Source: §4 · pp. 5–6

4.1 EVALUATION ON OBJECT HALLUCINATION

  • 675.0: HALVA13B gains 31.7 points on MME-Hall; HALVA7B scores 2.25 on MMHal-Bench, exceeding 7B RLHF, DPO, and SFT variants.On MMHal-Bench, those variants score 2.05, 1.97, and 1.76, respectively; HALVA13B also lowers hallucination rate to 0.45 versus 0.57 for LLaVA-RLHF.Source: Table 2 · p. 6, §4.1 · pp. 7–8, Table 4 · p. 8
  • 50.0→41.4: HALVA7B lowers sentence-level CHAIR (Cs) versus LLaVA-v1.5 7B while maintaining similar generation length.HALVA7B generates 98 tokens versus 100.9 for its base model.Source: Table 1 · p. 6, §4.1 · p. 6
  • 7.8→6.6 hallucination and 51%→53% ground-truth object coverage: HALVA7B improves both AMBER image-description measures over its base model.HALVA also improves discriminative-task F1 scores, including attribute F1 from 64.6% to 80.0% for HALVA7B.Source: Table 3 · p. 7, §4.1 · p. 7

4.2 EVALUATION ON HALLUCINATION BENCHMARKS BEYOND OBJECT HALLUCINATION

  • HALVA improves HallusionBench overall accuracy by 1.86% for HALVA7B, 2.16% for HALVA13B, and 1.21% for HALVA13B/384 versus their base models.HALVA13B/384 also reduces Yes/No bias from 0.19 to 0.02.Source: Table 5 · p. 8, §4.2 · p. 8

4.3 EVALUATION ON NON-HALLUCINATION BENCHMARKS

  • HALVA maintains or improves performance over its base models on general vision-language benchmarks, while HA-DPO7B and EOS7B show statistically significant performance drops.HALVA7B improves on MME by 16.3, MM-Vet by 1%, and LLaVA-BW by 1.8%, while retaining VQA-v2 performance.Source: §4.3 · p. 8, Table 6 · p. 9

4.4 ABLATION STUDY

  • α=0.4 works optimally for HALVA7B; at α=0.01 the model diverges substantially, while increasing α keeps it closer to the base model.Figure 5 plots model divergence against changes in relative log-probability of hallucinated tokens.Source: §4.4 · p. 9, Figure 5 · p. 9

4.5 QUALITATIVE ANALYSIS

  • In qualitative examples, HALVA avoids LLaVA-v1.5’s unsupported objects and Yes-biased answers, and shows some ability to mitigate visual-illusion hallucinations.The examples include a parked airplane, cloud-color questions, and counting people.Source: §4.5 · p. 9, Figure 6 · p. 9

5 RELATED WORK

Related work covers multimodal models, alignment methods, and hallucination mitigation across training and inference stages. DPA is positioned as a directly applicable fine-grained finetuning objective that aims to retain general vision-language abilities.

Background

  • MLLMs combine LLMs with vision encoders to extend visual perception, understanding, and reasoning; publicly available LLaVA and VILA models provide off-the-shelf bases for DPA.VLMs commonly align image and text features through contrastive learning on large image-text datasets.Source: §5 · p. 10

Alignment

  • Unlike sequence-level RLHF and DPO, DPA uses phrase-level alignment and a token-wise forward KL regularizer to constrain divergence.The paper notes that reverse-KL-based RL fine-tuning may encourage mode-seeking behavior and lower generation diversity.Source: §5 · p. 10

Hallucination mitigation

  • Hallucination mitigation spans pretraining, inference, and finetuning; DPA is a fine-grained objective for direct finetuning that targets hallucinations while retaining general abilities.The related-work discussion contrasts DPA with methods that train reward models to provide subsequence-level feedback.Source: §5 · p. 10

6 CONCLUDING REMARKS

DPA pairs selectively altered hallucinated responses with correct responses, then uses a phrase-level loss to reduce hallucinated-token likelihood. The appendix outlines the remaining material and provides a minimal implementation sketch.

6 CONCLUDING REMARKS

  • DPA creates hallucinated and correct response pairs by selectively changing ground-truth phrases, then uses its loss to reduce hallucinated tokens’ relative likelihood.The conclusion says the method mitigates object hallucinations, including existence and attribute errors, visual illusions, and complex-chart hallucinations, while maintaining or improving general vision-language performance.Source: §6 · p. 10

Appendix

  • The appendix lists pseudocode, DPA-versus-DPO distinctions, additional experiments, implementation details, qualitative results, and limitations.The supplied implementation is described as minimal, with some intermediary steps omitted for brevity.Source: Appendix · p. 17, A DPA PSEUDO CODE · p. 17

A DPA PSEUDO CODE

  • The pseudocode takes an image-language input, correct and augmented hallucinated responses, and a reference pair, then computes phrase-level alignment and token-wise KL divergence.It compares accumulated log probabilities for correct and hallucinated phrases and adds a token-wise KL term.Source: A DPA PSEUDO CODE · p. 17

B DISTINCTION BETWEEN OURS DPA AND DPO-BASED HALLUCINATION

DPA constructs negatives by changing specific ground-truth phrases and applies alignment loss to hallucinated phrases, unlike DPO’s sequence-level objective. The authors distinguish the methods in both their training pairs and their loss granularity.

B DISTINCTION BETWEEN OURS DPA AND DPO-BASED HALLUCINATION

  • DPA’s objective combines phrase-level alignment loss with token-wise KL divergence, and its stated setup reuses the correct input-response pair as the reference sample.The text defines the trained and frozen reference models, and identifies correct and hallucinated responses as the paired samples.Source: B DISTINCTION BETWEEN OURS DPA AND… · p. 18
  • DPA applies phrase-level alignment to hallucinated-phrase tokens, whereas DPO maximizes a response-level reward margin and penalizes all tokens in a hallucinated response.DPA also applies token-level KL regularization to retain the base model’s vision-language capabilities.Source: B DISTINCTION BETWEEN OURS DPA AND… · p. 18
  • DPA negatives selectively alter ground-truth words or phrases while preserving the rest of each correct response; HA-DPO changes responses at the sequence level.Figure S1 contrasts the phrase-level changes used for DPA with HA-DPO’s sequence-level changes.Source: B DISTINCTION BETWEEN OURS DPA AND… · p. 18, Figure S1 · p. 19

C.1 ABLATION ON LOSS

The ablation varies the KL coefficient α, which controls how closely the trained model stays aligned with its reference. The preferred range differs between the 7B and 13B variants, while very low α can harm language generation.

C.1 ABLATION ON LOSS

  • α between 0.3 and 0.4 performs better for HALVA7B, while α between 0.4 and 0.6 gives HALVA13B its highest performance.The authors use α = 0.4 for HALVA7B and α = 0.5 for HALVA13B by default; above 0.4, HALVA7B behaves similarly to the base model.Source: §C.1 · p. 20, Figure S2 · p. 20, Figure S3 · p. 20
  • A very low α of 0.01 may harm language generation, producing degeneration or descriptions that omit key details.The examples use the 7B variant.Source: §C.1 · p. 20, Figure S4 · p. 21

C.2 ABLATION ON GENERATIVE DATA AUGMENTATION

The appendix reports augmentation and divergence ablations, detailed benchmark results, and a comparison testing whether DPA’s gains arise from its data, objective, or both. Across these analyses, reported outcomes include task-specific benefits and broad hallucination mitigation.

C.2 ABLATION ON GENERATIVE DATA AUGMENTATION

  • Open-set hallucinated descriptions benefit generative-task CHAIR performance, while adding Yes-or-No responses boosts AMBER F1; combining splits gives broader improvements or competitive results.The study uses 21.5K samples and 28K unique correct–hallucinated phrase pairs based on 5K hallucinated objects; training curves remain stable across data splits.Source: §C.2 · p. 22, Table S1 · p. 22, Table S2 · p. 22

C.3 ABLATION ON DIVERGENCE MEASURE

  • Using seen reference samples gives better divergence estimates and better overall benchmark performance; initializing reference and trained models from the same checkpoint achieves optimal results.The ablations use HALVA7B; Table S3 reports the reference-data and reference-model comparisons.Source: §C.3 · p. 23, Table S3 · p. 23

C.4 DETAILED RESULTS OF MME-HALL

  • DPA mitigates or retains base-model performance across MME-Hall’s existence, count, position, and color categories, unlike methods with reported category tradeoffs or overall degradation.The comparison includes HA-DPO, EOS, VCD, and Woodpecker.Source: §C.4 · p. 23, Table S4 · p. 23

C.5 DETAILED RESULTS OF AMBER

  • HALVA reduces image-description hallucination from 7.8 to 6.6 and raises object coverage from 51% to 53%; discriminative-task F1 improves by up to 13.4%.The authors report outperforming HA-DPO, EOS, and Woodpecker on these AMBER results.Source: §C.5 · p. 24, Table S5 · p. 24

C.6 DETAILED RESULTS OF MMHAL-BENCH

  • DPA mitigates adversarial, comparison, relation, and holistic hallucinations on both HALVA7B and HALVA13B, and improves 6 of 8 subcategories for both 13B variants.The authors report that HA-DPO and EOS can fail across these broad categories and may worsen baseline performance.Source: §C.6 · p. 24, Table S6 · p. 24

C.7 DETAILED RESULTS OF HALLUSIONBENCH

  • HALVA7B and HALVA13B improve HallusionBench Hard Set performance by 4.34%-6.90%; Yes/No bias falls from 0.31 to 0.17 and 0.38 to 0.20, respectively.For HALVA13B/384, Yes/No bias falls from 0.19 to 0.02; zero is ideal.Source: §C.7 · p. 24, Table S7 · p. 25

C.8 A CRITICAL ANALYSIS OF OUR PROPOSED DPA

  • Applying DPA’s augmentation to HA-DPO does not produce HALVA’s performance boost, supporting the authors’ conclusion that gains combine the KL-regularized phrase-level objective and augmentation.DPA cannot be applied directly to HA-DPO’s descriptive-response data because it requires aligned correct and hallucinated phrases.Source: §C.8 · pp. 24–25, Table S8 · p. 25

C.9 RESULTS ON POPE

The supplementary analyses describe POPE’s coverage limits, compare response linguistic quality, and assess statistical significance across general and hallucination tasks. Training hyperparameters and asset licenses are also documented.

C.9 RESULTS ON POPE

  • POPE tests object presence with yes-or-no questions but omits attributes, relations, descriptive tasks, and broad visual-concept coverage.Its questions use 500 images and 79 unique objects, motivating the use of AMBER and MME as primary benchmarks.Source: §C.9 · p. 25

C.10 RESULTS ON LINGUISTIC QUALITY

  • HALVA7B receives the same linguistic-quality ratings as LLaVA 1.5 7B across grammatical correctness, fluency, detailedness, and choice of words.Ratings use GPT-4o-mini on 100 detailed descriptions, scored from 0 to 10.Source: §C.10 · pp. 25–26, Table S10 · p. 26

C.11 STATISTICAL ANALYSIS

  • DPA avoids statistically significant drops on general tasks and improves hallucination-task performance, while HA-DPO and EOS show general-task drops and deterioration on two of six hallucination tasks.HALVA7B improves significantly on CHAIR, AMBER generative, and AMBER discriminative tasks; both 13B variants improve across all setups, with significant gains on AMBER discriminative tasks.Source: §C.11 · p. 27, Table S11 · p. 27, Table S12 · p. 28

D.1 TRAINING HYPERPARAMETERS

D.2 LICENSES OF EXISTING ASSETS USED

  • The supplementary asset notes identify Visual Genome as publicly available under CC BY 4.0 and list Apache 2.0 licenses for LLaVA-v1.5 and VILA-v1.5.The weights used in the work are also referenced.Source: §D.2 · p. 31

D.3 GENERATIVE DATA AUGMENTATION SETUP

The generative augmentation setup uses templates and examples to prepare correct and hallucinated descriptions for DPA training. It includes one-sentence captions, short and detailed descriptions, and yes-or-no questions.

D.3 GENERATIVE DATA AUGMENTATION SETUP

  • Templates and complete examples show the generation of correct, closed-set hallucinated, and open-set hallucinated descriptions, with Gemini Vision Pro used to prepare responses.Instructions cover multiple description types and training examples also include yes-or-no questions.Source: §D.3 · p. 31
  • One training example changes a correct description of a cat beside a bird into a hallucinated description of a dog beside a leaf.The prompt asks for a brief image description.Source: §D.3 · p. 40
  • A second example changes a correct description of a baseball game into a hallucinated description of a hockey game.The prompt requests a terse but informative picture summary.Source: §D.3 · p. 40

E.1 QUALITATIVE COMPARISON BETWEEN HALVA AND LLAVA-V1.5

Qualitative examples contrast HALVA with LLaVA-v1.5 and VILA-v1.5 on image descriptions and questions. The examples highlight fewer hallucinated details and more consistent responses from HALVA.

E.1 QUALITATIVE COMPARISON BETWEEN HALVA AND LLAVA-V1.5

  • HALVA correctly describes image details and spatial relations in examples where LLaVA-v1.5 hallucinates objects or misstates positions.The examples include an unsupported carrot, people, pedestrian, and handbag, as well as incorrect positions for cars and traffic lights.Source: §E.1 · pp. 44–45
  • HALVA mitigates a co-occurrence-related hallucination in a baseball scene and describes the image with necessary details.The passage contrasts LLaVA-v1.5’s hallucinated pitcher-associated objects with HALVA’s description.Source: §E.1 · p. 46
  • HALVA’s responses are described as more consistent than LLaVA-v1.5’s, and it supports answers with accurate image details.One example distinguishes a dog standing on a skateboard from touching the ground.Source: §E.1 · pp. 47–48

E.2 QUALITATIVE COMPARISON BETWEEN HALVA AND VILA-V1.5

  • HALVA is described as more consistent than VILA-v1.5 in a horse-presence example and provides accurate supporting image details.The cited example says HALVA confirms two horses on both occasions.Source: §E.2 · p. 49

F LIMITATIONS

The paper focuses on object hallucinations in MLLMs. It identifies other hallucination forms as directions for future work and notes possible extensions to bias and harmful language generation.

F LIMITATIONS

  • The paper’s scope is object hallucinations; other forms, including modality misalignment and over-reliance on language, are not rigorously explored.The authors leave rigorous exploration of these directions for future work.Source: F LIMITATIONS · p. 49
  • Generalization to other hallucination forms is described as promising, but the paper leaves further exploration for future research.The cited passage does not claim a rigorous evaluation of those forms.Source: F LIMITATIONS · p. 49
  • The authors suggest DPA might be adapted to bias and harmful language generation, but leave those applications unexplored.Source: F LIMITATIONS · p. 49

Related papers

Loading 2405.18654v3…