Source-linked AI summary
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, Nenghai Yu
TL;DR
MLLM hallucinations threaten reliable image-grounded assistance, and existing remedies can require additional data, models, or knowledge. OPERA uses over-trust-penalized beam decoding with retrospection-allocation to address summary-token aggregation, achieving generalized hallucination reduction across MLLMs and metrics. Its gains are limited when strong model bias or weak visual perception remains, and for short answers under 10 tokens.
Problem
MLLMs can generate statements that conflict with user-provided images, while many existing mitigation methods require extra data, external models, or knowledge.
Method
OPERA combines an over-trust penalty with retrospection-allocation during inference to counter self-attention patterns that over-rely on summary tokens.
Results
OPERA demonstrates generalized hallucination reduction across various MLLM models, benchmarks, and hallucination metrics; on Shikra, it achieves approximately 35% improvement on DoLa.
Takeaways & Limitations
OPERA provides an inference-time hallucination-mitigation method without additional data, knowledge, or training costs.
Takeaways & Limitations
OPERA faces challenges when model bias is too strong, visual perception is weak, or answers are short sequences under 10 tokens.
Abstract
from arXiv · showhide
Hallucination, posed as a pervasive challenge of multi-modal large language models (MLLMs), has significantly impeded their real-world usage that demands precise judgment. Existing methods mitigate this issue with either training with specific designed data or inferencing with external knowledge from other sources, incurring inevitable additional costs. In this paper, we present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy, serving as a nearly free lunch to alleviate the hallucination issue without additional data, knowledge, or training. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns manifested in the self-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on a few summary tokens, but not all the previous tokens. Such partial over-trust inclination results in the neglecting of image tokens and describes the image content with hallucination. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary. With extensive experiments, OPERA shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality. Our code is available at: https://github.com/shikiw/OPERA.
1. Introduction
MLLM hallucinations can undermine trustworthy image-based assistance, while existing mitigation methods often require extra data, models, or knowledge. OPERA links hallucinations to partial over-trust in summary-token aggregation and addresses it through penalty-based decoding with retrospection-reallocation.
- Motivation: MLLM hallucinations include inaccurate objects, attributes, quantities, and locations, creating risks for practical applications such as autonomous driving.These errors can produce wrong judgments from road-scene images and potentially cause serious traffic accidents.
- Motivation: Existing hallucination-reduction methods can incur substantial costs through extra instruction-data annotation or external knowledge and models.
- Observation: Many hallucinations begin after columnar attention patterns on low-information tokens, which can function as summary tokens guiding subsequent generation.The observed patterns often occur on tokens such as full stops or quotation marks, while subsequent content contains reasoning or hallucinations.
- Observation: As generated text grows, MLLMs may over-trust nearby summary tokens, attenuate image information, and hallucinate content from learned associations such as “road” to “cars.”The paper relates this aggregation behavior to anchor-token observations in language models.
- Observation: CHAIR scores increase when more anchor tokens appear in context, showing a positive relation between summary-token accumulation and hallucination occurrence.
- Approach: OPERA applies an over-trust penalty during beam-search candidate selection and uses retrospection-allocation to reselect tokens when prior summary-token patterns are detected.The penalty derives a column-wise metric from a local self-attention window.
- Approach: OPERA mitigates hallucinations during inference without additional data, external knowledge, or training, and is evaluated across MLLMs and hallucination assessments.The authors describe the method as a nearly free-lunch approach.
2. Related Work
Related work covers multimodal model development, hallucination definitions and mitigation, and decoding strategies. Existing approaches include training and external-model interventions, while decoding methods trade off efficiency, diversity, coherence, or hallucination mitigation.
- Multimodal Large Language Models: MLLMs combine modalities such as text, images, and audio, using feature alignment and instruction fine-tuning to support image-text interaction.Shikra additionally incorporates grounding data for understanding image-grounding knowledge.
- Hallucination: In MLLMs, hallucination primarily concerns faithfulness to user-provided images, while existing countermeasures often require extra data or more powerful external models or knowledge.
- Decoding Strategies: Greedy decoding is efficient but can produce repetitive text, whereas beam search tracks multiple hypotheses to seek a more optimal sequence.
- Decoding Strategies: Top-k and nucleus sampling introduce randomness to increase diversity, with nucleus sampling selecting a dynamic set whose cumulative probability reaches p.
- Decoding Strategies: DoLa mitigates MLLM hallucinations by contrasting mature- and premature-layer logits and rescaling their increments during generation.
3. Method
OPERA decodes MLLM outputs by detecting knowledge aggregation patterns in local self-attention and penalizing over-trusted candidates during beam search. Its Retrospection-Allocation strategy detects persistent patterns and rolls back decoding to reselect a subsequent token.
- Over-Trust Logit Penalty: OPERA builds on beam search and augments candidate selection with an accumulative over-trust penalty in the beam score.Candidates with larger penalties receive lower selection priority.
- Over-Trust Logit Penalty: The method analyzes a local self-attention window over generated tokens, excluding image and prompt tokens, and uses maximum multi-head attention weights after renormalization.The window begins after the image and prompt tokens; k denotes its size.
- Over-Trust Logit Penalty: Column-wise products of scaled lower-triangular attention values produce scores whose maximum represents the strength of a knowledge aggregation pattern.The upper triangle is filled with zeros before score computation.
- Over-Trust Logit Penalty: The penalty is applied to a candidate set formed from the top-Ncan logits of each beam, limiting prediction while incorporating the detected pattern metric.The candidate set has size Ncan × Nbeam.
- Retrospection-Allocation Strategy: The penalty can detect aggregation patterns only after subsequent tokens are generated, and some cases remain where all candidates are penalized after hallucination has already occurred.This limitation motivates the more aggressive rollback and reselection strategy.
- Retrospection-Allocation Strategy: Retrospection-Allocation responds to delayed pattern detection by measuring repeated maximum-score locations across recent tokens and rolling back when overlap reaches threshold r.The method reselects the next token from candidates not previously selected and constrains rollback locations to be non-decreasing.
4. Experiment
Experiments evaluate OPERA across four MLLMs, decoding baselines, hallucination metrics, GPT-assisted assessments, and ablations. OPERA consistently reduces hallucination while preserving answer detail and general text quality, with robustness across settings.
- 4.1. Setup: Evaluation covers InstructBLIP, MiniGPT-4, LLaVA-1.5, and Shikra against greedy, nucleus, beam-search, and DoLa decoding.Beam search and OPERA use Nbeam = 5, while nucleus sampling uses p = 0.9.
- 4.2. Quantitative Results: OPERA surpasses all baseline decoding methods on CHAIR sentence-level and instance-level hallucination metrics for both long and short descriptions.The evaluation uses MSCOCO images and restricts maximum generated lengths to 512 and 64 tokens.
- 4.2. Quantitative Results: 30.4% improvement over greedy decoding on HSR and 15.4% improvement over DoLa on HWR are reported on VG-100K.Figure 7 evaluates SPI, WPI, HSPI, HWPI, HSR, and HWR; larger SPI and WPI, but smaller hallucination measures, indicate better performance.
- 4.2. Quantitative Results: 27.5% improvement over Beam search is achieved in GPT-4V evaluation while maintaining answer detailedness.GPT-4V assessment is used to evaluate hallucination beyond object existence, including attributes, locations, and relations.
- 4.2. Quantitative Results: OPERA attains the highest POPE average F1 scores among decoding strategies, although gains are marginal for brief Yes-or-No answers.The method is reported to be especially effective for hallucinations in lengthy sequences.
- 4.2. Quantitative Results: OPERA generally preserves generated-text quality and maintains or improves performance on the MME and MMBench benchmarks.The method also handles repetition by retrospectively reallocating token selection toward alternatives such as the eos token.
- 4.3. Ablation Study on Hyper-parameters: Removing both components reduces OPERA to standard Beam search, while either the over-trust penalty or retrospection-reallocation strategy reduces hallucinations.The over-trust penalty contributes relatively more to final performance; default settings use Ncan = 5, σ = 50, α = 1, and r = 15.
5. Limitation & Social Impact
OPERA has two main limitations: it cannot address every hallucination, especially those driven by strong model biases or weak visual perception, and it gains little on short answers. The paper reports no potential social harm and suggests OPERA may inspire further cost-free mitigation approaches.
- Limitations: OPERA cannot address all kinds of MLLM hallucinations.Failure cases include hallucinations caused by strong learned biases and insufficiently robust visual perception.
- Limitations: Strong associations in the model’s learned content can make suitable candidates difficult to find during retrospection-reallocation.For example, the model may hallucinate cars after road is mentioned because of a strong road–cars bias.
- Limitations: OPERA faces challenges when MLLMs are misled by similar shapes, colors, or low-resolution images.These cases are constrained by the model’s underlying visual capabilities.
- Limitations: OPERA demonstrates marginal gains for hallucinations in short answers under 10 tokens.The paper attributes this boundary primarily to the hysteresis of knowledge aggregation patterns and reports stronger performance on long sequences.
- Social impacts: The paper identifies no potential social harm from OPERA and presents it as an inspiration for cost-free hallucination mitigation research.It suggests such approaches may generalize across different MLLMs.
6. Conclusion
The paper introduces OPERA, an inference-time decoding method that mitigates MLLM hallucinations without additional data, knowledge, or training costs. Its approach targets self-attention aggregation patterns, and experiments report superior hallucination reduction across various MLLMs and metrics.
- 6. Conclusion: OPERA mitigates MLLM hallucinations during decoding without requiring additional data, knowledge, or training costs.It combines an Over-trust Penalty with a Retrospection-Allocation strategy.
- 6. Conclusion: OPERA links hallucinations to self-attention knowledge aggregation patterns that focus on summary tokens while neglecting image tokens.The proposed decoding method uses this observation to address content hallucination.
- 6. Conclusion: Experiments show OPERA’s superiority in reducing hallucination across various MLLMs and metrics.The conclusion states this result at the paper level without restricting it to a single benchmark or model.