Multi-Modal Hallucination Control by Visual Information Grounding

• The visual prompt dependency measure (PDM) compares output likelihood with and without image information to assess grounding, is annotation-free, and low dependency alone does not establish hallucination.Source: §1 · pp. 1–2
• PDM decreases as more tokens are generated, a conditioning dilution or fading memory effect that may lead to hallucinations.Source: §1 · p. 2
• M3ID is a training-free, low-overhead inference-time intervention that amplifies visual prompt influence over the language prior to reduce hallucinations while preserving fluency.Source: §1 · p. 2
• For users with model weights, DPO preference optimization favors continuations with higher visual prompt dependency to further reduce conditioning dilution and hallucinations.Source: §1 · p. 2
• M3ID and DPO reduced hallucinated objects in captioning by 25% and 28%, respectively, and improved POPE accuracy by 21% and 24% over the base model.Source: §1 · p. 2

Hallucinations in VLMs

• Studies report that VLMs often describe objects absent from images, with hallucinations especially pronounced for common or frequently co-occurring objects [1] [6] [9] [27].Other work addresses instruction-following with absent objects [11] and proposes post-hoc hallucination identification and correction [31].Source: §2 · p. 2

Context-dependent decodings

• Search decoding can be accurate but repetitive, whereas sampling is more diverse but prone to topic drift [5, 8, 15].Context-aware decoding [21] and Pointwise Mutual Information decoding [16] [25] aim to align generations with context or factual data.Source: §2 · p. 2
• Mutual-information decoding has steered language-model outputs toward input faithfulness [25] and dialogue diversity or relevance [7], but this work applies it to multimodal grounding.It also varies penalization of marginally likely outputs over time to counter progressive forgetting of the visual prompt.Source: §2 · p. 3

• PDM compares token distributions with and without image conditioning; high values indicate prompt-specific tokens, while low values indicate prompt-neutral tokens.The analysis mainly uses PDM-H, based on Hellinger distance.Source: §3 · p. 3
• As token position increases, PDM-H falls while hallucinated-object counts rise in LLaVA captions on MS COCO, linking weaker visual conditioning with more hallucinations.Conditioning dilution is not necessarily harmful if the relevant image information has already been extracted, but reliance on language priors can facilitate hallucinations when important details are missing.Source: Figure 3 · p. 3, §3 · p. 4
• Similar conditional and unconditioned likelihoods for fluent or predictable tokens can reflect contextual pressure, not necessarily increased hallucination risk.This can occur for prepositions and conjunctions or fine-grained objects such as “Peanut butter.”Source: §3 · pp. 3–4

4.1. M3ID: Improving grounding at inference time

• M3ID models conditioned log probabilities as an interpolation between a non-forgetting distribution and an unconditioned model, with the image’s influence decaying as γ_t = exp(−λt).A smaller γ_t means the conditioned distribution relies less on the image and more on the unconditioned model.Source: §4.1 · p. 4
• M3ID counteracts fading memory by amplifying conditioned tokens that differ from the unconditioned model, while suppressing correction when the conditioned model is highly confident.Late in generation, the difference corresponds to favoring tokens with higher mutual information with the visual input; the confidence threshold avoids penalizing obvious tokens that need not depend on the image.Source: §4.1 · pp. 4–5

4.2. M3ID+DPO to learn more grounded policies

• M3ID+DPO [19] trains the VLM to prefer continuations sampled from the M3ID-improved conditioned distribution over negatives sampled from the unconditioned distribution.To make negatives more informative, the method appends the first sentence generated by the conditioned VLM to the prompt before sampling unconditioned continuations.Source: §4.2 · p. 5

5. Experiments

• Experiments evaluate captioning and VQA on MS COCO, measuring ungrounded objects with CHAIR and POPE object perception with binary Yes/No questions.CHAIRi measures the fraction of hallucinated objects, CHAIRs the fraction of captions with at least one hallucination, and Cover the fraction of annotated objects mentioned.Source: §5 · pp. 5–6
• Baselines include multinomial sampling, PMI [25], Contrastive Decoding, and annotation-based training methods; M3ID+DPO requires no annotations.The experiments mainly use LLaVA, with evaluations also extended to InstructBLIP.Source: §5 · pp. 5–6

5.1. VLM grounding on captioning

• 27%/21% relative improvements over LLaVA 7B and 26%/29% over LLaVA 13B on CHAIRi/CHAIRs show M3ID reduces caption hallucinations across model sizes.Cover improves on 7B and decreases by less than 2.2% on 13B; longer captions also retain improved grounding. M3ID+DPO further reduces hallucinated objects and improves Cover versus M3ID and LLaVA+LURE.Source: Table 1 · p. 6, §5.1 · pp. 6–7

5.2. VLM grounding on VQA

• 15% and 24% accuracy improvements over LLaVA 7B and 13B, respectively, show M3ID+DPO further improves POPE performance over inference-time M3ID.M3ID alone improves accuracy over standard LLaVA decoding by 8% and 21% for 7B and 13B, respectively; M3ID+DPO is close to labeled training baselines without annotations.Source: Table 2 · p. 7, §5.2 · p. 7
• On POPE, prompt-template tokens between image and output create dilution despite Yes/No answers, so M3ID uses offset t=t0, the intervening-token count.The base LLaVA models answer “Yes” 84%/83.7% of the time; M3ID reduces this ratio to 72.9%/61.8% for 7B/13B.Source: §5.2 · p. 7

5.3. Ablations

• High λ or α strengthens M3ID’s intervention, but excessive correction can raise hallucinations and harm Cover; too-high α can also disrupt fluency.Adding contextual-pressure compensation does not significantly change POPE results, which the paper frames as binary classification rather than open-ended generation.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

6. Conclusions

• M3ID maximizes mutual information between generated text and visual context and integrates at inference time with any pretrained autoregressive VLM.The authors present it as a flexible, cost-effective approach to vision-language grounding.Source: §6 · p. 8
• Object hallucinations stem from excessive reliance on the language prior, and inference-time M3ID can significantly reduce them without training.The authors distinguish this cause from poor understanding of the visual modality.Source: §6 · p. 8
• M3ID requires two inference passes; batching conditioned and visually masked queries may avoid added inference time but uses more memory.The method can also suppress objects likely under the unprompted language prior; the authors say proper hyperparameter selection can mitigate this.Source: §6 · p. 8
• The authors propose future work on structured captions that become progressively more detailed while remaining image-grounded.They suggest encoding this preference in model weights to address the suppression of obvious objects.Source: §6 · p. 8
• Human- or AI-annotated preference pairs assessed for grounding are a promising direction for future investigation.M3ID paired with DPO is also presented as an option when training is feasible, with reduced hallucinatory behavior.Source: §6 · p. 8

A. Experimental details

• CHAIR evaluation uses 5,000 MS COCO 2014 validation images and detects hallucinated objects by comparing caption mentions with annotated objects and synonyms [10] [20].Source: A. Experimental details · p. 11
• POPE evaluation uses 3,000 questions each for random, popular, and adversarial object conditions on MS COCO 2014 validation images [9] [10].Accuracy is measured with the template “Is a ⟨object⟩present in the image?”.Source: A. Experimental details · p. 11
• Decoding hyperparameters are tuned on 500 non-overlapping validation images, and DPO uses the SFT fine-tuning configuration with β = 0.1 [19].The DPO setup uses AdamW, batch size 128, learning rate 2 × 10−5, cosine annealing, warm-up ratio 0.03, ZeRO-2, and LoRA rank 64, scaling 16, and dropout 0.05.Source: A. Experimental details · p. 11

B. Context-dependent decodings

• PMI Decoding [25] boosts source-related tokens under output uncertainty, but its penalty weight for marginally likely continuations is constant over time.For multimodal use, the input image replaces the source text.Source: B. Context-dependent decodings · p. 11
• Contrastive Decoding [7] amplifies expert-versus-amateur predictions after a plausibility constraint; its multimodal penalty is also time-independent.Source: B. Context-dependent decodings · pp. 11–12
• M3ID penalizes image-unconditioned language-prior probabilities when contextual pressure is low, with the penalty increasing over generation through a time-dependent parameter.The decay rate is controlled by λ; the initial strength t0 is 0 for captioning and 10 for POPE.Source: B. Context-dependent decodings · p. 12

C. Multi-modal Direct Preference Optimization

• Under a Bradley–Terry preference model, DPO [19] maximizes expected reward while regularizing to the initialization policy, increasing preferred and reducing poorly grounded completion likelihood.Source: C. Multi-modal Direct Preference… · p. 12
• Positive captions are generated with M3ID, while negative continuations come from a visually masked VLM completing the positive captions’ first sentences; DPO trains on these pairs.Source: C. Multi-modal Direct Preference… · pp. 12–13

• The section presents further results complementing the experiments in the main text.Source: D. Further results · p. 13

D.1. Prompt dependency measures

• PDM-H considers the full vocabulary distribution, whereas PDM-R focuses on the most likely token; PDM-R = 1 means both conditioned and unconditioned models rank the same token highest.Thus, greedy generation without the conditioning image would produce the same continuation when PDM-R = 1.Source: §D.1 · pp. 13–14
• After about 30 generated tokens, standard decoding’s PDM-R approaches 1, while M3ID maximizes both PDMs and counters language-prior effects in longer captions.PMI and Contrastive Decoding raise PDM distances early, but their effect becomes negligible near the middle and end of generation.Source: §D.1 · p. 14
• Estimating the conditioning-dilution decay rate from Fig. 3 with log-coordinate linear regression gives λ = 0.016, close to the cross-validated optimum of 0.02.This offers a way to reduce the computational complexity of M3ID hyperparameter search.Source: §D.1 · p. 14
• PDM-H and PDM-R rise on tokens requiring visual information, such as objects and attributes, and fall on articles or when caption context already makes a token likely.For example, “kitchen” is already likely from text alone, and articles such as “the” and “a” can have PDM-R = 1.Source: Figure 6 · p. 13, §D.1 · p. 14

Detailed captioning and stochasticity due to sampling

• With detailed captions, cover rises from approximately 55% to about 70%, and M3ID and M3ID+DPO achieve the best results in minimizing object hallucinations despite higher CHAIR values.The table also reports standard deviations from stochastic sampling.Source: §D.2 · p. 14

Introduced errors

• For 13B (7B), M3ID corrects hallucinations in 18% (16.2%) of instances while introducing new hallucinations in only 2.4% (3.0%) of cases.Source: Table 6 · p. 15, §D.2 · p. 15

Overcompensation

• Excessive language-prior penalties can make M3ID omit context-predictable objects, such as the man or dominant van, despite their presence in the image.Source: Table 7 · p. 15, §D.2 · pp. 15–16

Preserving language structure

• At α = 1, stronger correction activates more often, first harms syntax and then produces apparently random, sometimes Cyrillic, tokens, disrupting language fluency and structure.Source: Table 8 · p. 16, §D.2 · p. 16

D.3. InstructBLIP

• M3ID applies to VLMs with fine-tuned or untuned base LLMs, provided visual conditioning can be dropped to compute the language prior.InstructBLIP [4] is evaluated as an example of this broader applicability.Source: §D.3 · p. 16
• For InstructBLIP [4], which connects the vision encoder and LLM through a Q-Former, unconditioned predictions are computed by masking image tokens in the Q-Former.This differs from the procedure used for LLaVA.Source: §D.3 · p. 16
• On CHAIR, M3ID significantly reduces hallucinations with InstructBLIP compared with standard generation.The evaluation uses image captioning on MS COCO.Source: §D.3 · p. 16, Table 9 · p. 17

D.4. Conditioning dilution in VQA

• Longer POPE prompts result in higher errors, while M3ID with an offset significantly improves performance as the output token moves farther from image content.The offset t0 accounts for the number of tokens between the output token and the image tokens.Source: Table 10 · p. 17, §D.4 · p. 17
• In binary VQA, prompt and question length affect visual prompt dependency by changing the distance between the yes/no response and image tokens.The tested format is [Img][Model Prompt][Question], with prompts ranging from empty to instructions that include neutral sentences.Source: §D.4 · p. 17
• For POPE, M3ID uses γt = e−λ(t−t0), with t0 set to the number of tokens between the output token and image tokens.This offset accounts for the response’s distance from the image content.Source: §D.4 · p. 17