PaperlayerView exact revision on arXiv

Source-linked AI summary

Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective

Zihao Yue, Liang Zhang, Qin Jin

arXiv:2402.14545v2cs.CLcs.CV

TL;DR

Multimodal hallucinations can arise when overly detailed training data encourages models to continue generating beyond their visual perception limits. The paper studies EOS decisions and develops selective EOS supervision and EOS-based data filtering. Both approaches significantly reduce hallucinations, while the study is limited to generative tasks and multimodal models.

  • Problem

    Overly detailed training data can hinder timely termination and cause multimodal models to generate content beyond their visual perception limits.

  • Method

    The paper analyzes EOS decisions and uses selective EOS supervision plus data filtering to preserve or improve appropriate termination behavior.

  • Results

    Selective EOS Supervision reduces LLaVA-1.5 hallucinations by 26% at the sentence level and 27% at the instance level, while filtering a small portion of training data also significantly reduces hallucinations.

  • Takeaways & Limitations

    Models retain an inherent ability to align generation length with visual perception, which can be exploited using regular instruction data and data filtering.

  • Takeaways & Limitations

    The study covers only generative tasks and multimodal models, and addresses only one source of hallucinations among several causes.

Abstract

from arXiv · show

Large Multimodal Models (LMMs) often suffer from multimodal hallucinations, wherein they may create content that is not present in the visual inputs. In this paper, we explore a new angle of this issue: overly detailed training data hinders the model's ability to timely terminate generation, leading to continued outputs beyond visual perception limits. By investigating how the model decides to terminate generation with EOS, the special end-of-sentence token, we find that the model assesses the completeness of the entire sequence by comparing the generated text with the image. This observation suggests that the model possesses an inherent potential of making proper EOS decisions based on its visual perception to avoid overly lengthy outputs. To take advantage of such potential, we explore two methods to mitigate multimodal hallucinations: a training objective that enables the model to reduce hallucinations by learning from regular instruction data, and a data filtering strategy to prevent harmful training data from exacerbating model hallucinations. Both methods significantly improve the hallucination performance of LMMs, without requiring any additional data or knowledge.

1 Introduction

The paper identifies overly detailed training data as an overlooked source of multimodal hallucinations because it can push LVLMs to generate beyond their visual perception limits. It analyzes EOS decisions and proposes selective supervision plus data filtering to improve timely termination and reduce hallucinations.

  • Motivation: Overly detailed captions can exceed LVLMs’ visual perception, encouraging hallucinated details as models fit the training data’s length and detail distribution.This is especially problematic for subtle, small, or easily confusable image features.
  • Motivation: Models should ideally terminate generation when their visual perception limits are reached, but explicit supervision for this boundary is difficult to construct.The paper therefore examines the model’s existing EOS decision behavior as an alternative source of supervision.
  • EOS analysis: EOS prediction integrates the generated sequence with the visual input, suggesting that models can assess whether the text is complete relative to what they perceive.Manipulating visual and textual context changes EOS tendency in ways consistent with semantic completeness.
  • Approach: Selective EOS Supervision modifies maximum-likelihood training so regular-token positions do not suppress the model’s inherent EOS tendency, while EOS-labeled positions retain standard learning.This allows regular instruction data to train better termination decisions.
  • Approach: The paper also filters harmful training data using metrics that rank examples by their positive or negative effects on EOS tendency.Removing a small portion of data significantly reduces hallucinations without additional data or knowledge.

2 EOS Decision

The EOS analysis shows that termination decisions use the entire generated sequence and compare textual completeness with the image. EOS probability rises as descriptions become more complete relative to the visual input, revealing an intrinsic mechanism that can support hallucination mitigation.

  • 2.1 Information Basis of EOS Decision: When predicting EOS, the model relies more on preceding sentences than on the current sentence, unlike regular-token prediction where the current sentence remains highly important.This indicates that EOS decisions integrate information from the entire generated context.
  • 2.1 Information Basis of EOS Decision: EOS prediction considers the current state of the entire sequence, whereas non-EOS prediction emphasizes information nearer the current target position.The comparison uses saliency-based information-flow analysis across image tokens, preceding sentences, and the current sentence.
  • 2.2 Semantic Comparison for EOS Decision: Reducing image information increases EOS tendency, while adding image information or hiding text decreases it, supporting comparison between generated text and visual input.These interventions preserve sequence length while changing multimodal contextual semantics.
  • 2.2 Semantic Comparison for EOS Decision: As the sequence lengthens, EOS tendency increases, and the model is more likely to stop when the image has been described more completely.The result links termination to relative textual completeness rather than length alone.
  • 2.3 Discussion: These findings suggest that models retain the potential to adjust generation length to visual perception even after fitting overly detailed training distributions.Timely termination may indicate that the generated text adequately captures the visual information the model can perceive.

3 Mitigating Multimodal Hallucinations

The paper mitigates multimodal hallucinations by preserving models’ inherent ability to terminate generation according to visual perception, using Selective EOS Supervision and EOS-based data filtering.

  • 3.1 Selective EOS Supervision for Training: Selective EOS Supervision excludes EOS from non-EOS probability distributions while retaining vanilla MLE when EOS is the label.This preserves the model’s tendency to stop when generated text adequately represents perceived visual information, without requiring additional data or expert models.
  • 3.1.2 Results: 26.4% and 26.6% decreases in CHAIRS and CHAIRI for LLaVA-1.5 (7b) follow full-data fine-tuning with the proposed learning objective.The method also outperforms truncation, length-penalty decoding, VCD, and OPERA, although recall decreases by 1.8% for LLaVA-1.5 (7b).
  • 3.2 Data Filtering: Removing 20% of the highest-scoring harmful data reduces sentence-level and instance-level hallucinations by 23.7% and 23.2%, respectively.Reversed filtering exacerbates hallucinations, while random removal does not improve sentence-level performance.
  • 3.2 Data Filtering: The filtering strategy changes model generation length substantially without greatly changing training-data length, indicating that it acts through EOS supervision rather than length distributions.Reversed filtering nearly doubles average model-generation length while leaving average training-data length almost unchanged.
  • 3.3 Discussion: The two approaches retain complementary practical advantages: the training objective is broadly applicable and easy to deploy, whereas filtered data is compatible with various training methods.The objective is especially suited to further training existing models, while filtering can be paired with different training procedures.

4 Related Works

Prior work addresses multimodal hallucinations through training, inference, and post-hoc correction strategies, while this paper introduces complementary training- and data-focused approaches.

  • Hallucination-mitigation research targets training, inference, and post-hoc correction.Training methods use hallucinatory data as negative examples through preference optimization or contrastive learning; inference methods suppress parametric biases or penalize inferior attention; post-hoc methods revise outputs or prompt self-correction.
  • Training-based approaches include preference optimization and contrastive learning to improve resistance to hallucinations.
  • This paper proposes a new learning objective and a data filtering strategy within the training and data perspectives.

5 Conclusion

The paper identifies overly detailed training data as a source of multimodal hallucinations and studies EOS decisions as a way to align generation length with visual perception. It develops Selective EOS Supervision and Scoring EOS Supervision to mitigate hallucinations.

  • Overly detailed training data can prevent timely termination, causing outputs to exceed the model’s visual perception limits and hallucinate.
  • EOS analysis shows that models assess generated-sequence completeness relative to the image and retain potential to terminate according to visual perception.
  • The paper proposes Selective EOS Supervision and Scoring EOS Supervision to improve EOS decisions and remove training data that hinders timely termination.

Limitations

The paper’s evaluation is limited to generative detailed image-description tasks and multimodal models, while its solutions address only one source of hallucination.

  • The evaluation covers generative tasks, specifically detailed image description, rather than broader tasks such as classification-oriented VQA.
  • The solutions are examined only on multimodal models, although the authors note they could technically apply to unimodal large language models.
  • The methods target timely sequence termination, addressing only the simplest source among the various causes of hallucinations.The authors state that fully solving hallucination remains a substantial challenge.

Ethics Statement

The authors report no significant ethical concerns and note that the work uses public, commonly used data and models. They caution that inherited biases and limitations may remain.

  • The authors anticipate no significant ethical concerns for this work.
  • The study uses publicly available and commonly used data and models, whose biases and limitations may be inherited by the findings.

A.1 Experiment in Figure 1

The training analysis focuses on the latter half of fine-tuning, where performance stabilizes, and uses Gaussian noise to reduce image semantics over T=500 steps. Training costs are also reported under an 8-GPU setup.

  • A.1 Experiment in Figure 1: The analysis focuses on the latter half of three-epoch fine-tuning, where performance begins to stabilize after early fluctuations.The model is fine-tuned on LLaVA-Instruction-150K and evaluated on the same data used in Section 2.
  • A.1 Experiment in Figure 1: Gaussian noise is gradually added to images over T=500 steps to reduce their semantic information, following the forward diffusion process.The implementation follows Leng et al. (2023).
  • A.1 Experiment in Figure 1: The proposed training objective and data filtering strategy do not noticeably increase training costs on a setup with 8 NVIDIA RTX A6000 GPUs.All experimental results use single runs with greedy decoding.

B Additional Results

Additional analyses show that information is aggregated through sentence boundaries before prediction, while EOS decisions depend on the relative semantic completeness of text with respect to the image.

  • B Additional Results: Information from regular content tokens converges at sentence-ending periods before being aggregated to the next-token prediction position.This pattern appears in both EOS and non-EOS predictions and is illustrated through normalized information-flow comparisons.
  • B Additional Results: The model’s EOS probability varies with target position, providing a direct view of how its termination tendency changes across a sequence.Figure 8 reports predictive EOS probabilities at different positions.
  • B Additional Results: Replacing or augmenting the image produces similar EOS effects, indicating that the model evaluates text’s relative semantic completeness against perceived visual information rather than absolute richness alone.The comparison supports assessing whether the generated text encompasses the image’s perceived information.

B.3 Selective EOS Supervision

Selective EOS Supervision increases and stabilizes EOS prediction, shortening captions while reducing hallucinations and largely preserving correct content; score balancing makes the filtering strategy more effective.

  • B.3 Selective EOS Supervision: Nearly 3/4 of objects omitted by the further-trained models are hallucinations, indicating that shorter outputs remove predominantly incorrect content.The omitted objects are those mentioned by the original model but absent from the further-trained models.
  • B.3 Selective EOS Supervision: Selective EOS Supervision raises and stabilizes EOS prediction tendency, whereas MLE produces no change, suggesting improved timely sequence termination.The EOS trend is shown during further training on Detail23K.
  • B.3 Selective EOS Supervision: The method reduces hallucinated objects while largely preserving correct objects in generated captions.Average correct and hallucinated object counts are reported in Table 6 for LLaVA-1.5 (7b).
  • B.3 Selective EOS Supervision: Spos and Sneg have comparable magnitudes, while Sfinal is approximately normally distributed with zero mean, addressing concerns that non-EOS positions dominate the combined score.The distributions are examined on LLaVA-Instruction-150K.
  • B.3 Selective EOS Supervision: Balancing the positive- and negative-position scores through Sfinal is more effective than the alternative and avoids the greater recall loss associated with that alternative.The supplied results identify balanced scoring as the most desirable outcome.

B.5 MME and POPE Evaluation

The proposed techniques target hallucinations in generative tasks by adjusting termination behavior, but they do not improve performance on the Yes-or-No MME and POPE benchmarks.

  • B.5 MME and POPE Evaluation: The methods do not yield performance gains on MME or POPE, which evaluate perception, cognition, and object hallucination through Yes-or-No questions.Their effectiveness is therefore demonstrated for generative tasks rather than these broader VQA evaluations.
  • B.5 MME and POPE Evaluation: The techniques address generative hallucinations by adjusting the model’s propensity to conclude outputs appropriately.This scope differs from the perception and object-hallucination challenges represented by MME and POPE.
  • B.5 MME and POPE Evaluation: Timely stopping is characterized as an important hallucination source, but addressing it alone does not solve multimodal hallucinations because their origins are multifaceted.The broader problem remains open for further investigation.

B.6 Qualitative Results

Qualitative examples show that both EOS-based methods reduce late-stage hallucinations by helping models stop after covering what they can visually perceive.

  • Both methods better preserve the model’s ability to stop generation after covering visually perceivable content, preventing overly lengthy and inaccurate outputs.The qualitative results correspond to Selective EOS Supervision and Scoring EOS Supervision in Figures 11 and 12.
  • Baseline models often hallucinate toward the ends of their outputs while attempting to include excessive image details beyond their visual perception limits.
  • The examples support why simply truncating sequences can reduce hallucinations, while the proposed methods retain higher information density by stopping more appropriately.
Loading 2402.14545v2…