PaperlayerView exact revision on arXiv

Source-linked AI summary

EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

Andrés Villa, Juan León Alcázar, Motasem Alfarra, Vladimir Araujo, Alvaro Soto, Bernard Ghanem

arXiv:2501.02699v1cs.CVcs.AI

TL;DR

Instruction-tuned vision-language models can generate responses that depart from image ground truth, while existing mitigation often targets language or fusion components. EAGLE improves visual grounding by tuning the visual encoder independently, then replacing the default encoder without additional alignment or instructional training. It reduces hallucination metrics across benchmarks and architectures, with reported relative improvements of 11.2% on MMVP and 6.3% on MERLIM.

  • Problem

    Instruction-tuned vision-language models can produce open-set responses unrelated to image ground truth, and mitigation approaches have primarily focused on the language model or adapter.

    Source: §1 · p. 2

  • Method

    EAGLE tunes the visual encoder to improve fine-grained grounding, then plugs it into a trained IT-VLM without further modification or alignment training.

    Source: §1 · p. 2

  • Results

    EAGLE improves hallucination metrics across three standard benchmarks and six diverse architectures, with 11.2% relative improvement on MMVP and 6.3% relative improvement on MERLIM.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The tuned visual encoder can be integrated into IT-VLMs without additional alignment or instructional training.

    Source: §1 · p. 2, §5 · p. 8

    EAGLE uses object segmentations for fine-grained representation, but the relative scarcity of instance-segmented image data constrains that supervision source.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models and vision transformers have demonstrated impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE, is fully agnostic to the LLM or fusion module and works as a post-pretraining approach that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without additional instructional training. As a result, EAGLE achieves a significant reduction in hallucinations across multiple challenging benchmarks and tasks.

1. Introduction

Instruction-tuned vision-language models combine an instruction-fine-tuned vision transformer with a language model, but can produce outputs unrelated to image ground truth. EAGLE addresses this by improving the visual encoder, with reported hallucination reductions across benchmarks and models without additional alignment or tuning.

1. Introduction

  • IT-VLMs combine an LLM with an instruction-fine-tuned ViT through a small adapter or connector, supporting zero-shot multimodal tasks such as VQA, image captioning, and classification.The fusion module is much smaller than the other components.Source: §1 · p. 2
  • Hallucinations are open-set language responses unrelated to image ground truth, and can reflect decoder language bias overriding visual semantics.The passage attributes this account to prior work.Source: §1 · p. 2
  • Earlier hallucination-mitigation methods primarily target the LLM or adapter, whereas this work focuses exclusively on the visual component.The cited approaches include changes to adapter training or architecture, LLM pre-training, and instructional data.Source: §1 · p. 2
  • Prior visual-encoder approaches add encoders and require further IT-VLM fine-tuning, while EAGLE directly enforces grounding for common object classes and preserves the global feature descriptor.The refined ViT is described as encoding fine-grained object-class distributions and integrating without additional adaptation.Source: §1 · p. 2
  • EAGLE optimizes the ViT in isolation and replaces the default visual encoder at inference without further modification to the trained IT-VLM.The approach is agnostic to the LLM and fusion module.Source: §1 · p. 2
  • EAGLE reduces hallucinations across three benchmarks and six IT-VLMs; its visual modules can be integrated without additional alignment or tuning.The reported relative improvements are 11.2% on MMVP and 6.3% on MERLIM.Source: §1 · p. 2

2. Related Work

Related work addresses hallucinations through instructional data, adapter design, and visual grounding. EAGLE instead enhances fine-grained representation in CLIP-based encoders while retaining zero-shot capabilities and replacing default encoders without architectural changes or alignment training.

2. Related Work

  • Hallucinations include ungrounded multimodal outputs, which challenge system reliability and trustworthiness; proposed contributing factors include language bias and limited fine-grained visual encoding.The cited work discusses decoder biases [41], linguistic cues [15] [21], and visual-encoder limitations [34].Source: §2 · p. 2
  • Instructional-data approaches use greater data granularity; InstructBLIP combines 13 multimodal datasets, while LLaVA-v1.5 selects four academic datasets and converts them into conversational formats using GPT.The passage cites [1] for data granularity, for InstructBLIP, and [24] for LLaVA-v1.5.Source: §2 · p. 3
  • Adapter designs align the modality encoders and prompt the language decoder from visual embeddings; InstructBLIP uses question-conditioned Q-Former selection of visual features [20].The adapter is identified as critical to alignment [20].Source: §2 · p. 3
  • Prior visual-encoder ensembles [19] [34] can enrich representations but increase computational demand; EAGLE enhances fine-grained CLIP-based representations without sacrificing zero-shot capabilities.EAGLE requires no extra encoders, architectural modifications, or alignment training, and is reported to reduce hallucinations across six IT-VLMs on three benchmarks.Source: §2 · p. 3

3. Methodology

EAGLE adapts visual-language alignment to the feature sequence used by IT-VLMs, whose tokens are less language-aligned than the CLS representation. It trains fine-grained object representations while aiming to preserve the global feature descriptor and transfer capabilities.

3. Methodology

  • IT-VLMs use a feature sequence without the CLS token, and Table 1 indicates a substantial alignment gap between that sequence and the CLS representation.For two representative VLMs, the CLS token is more aligned with language embeddings and more effective in zero-shot recognition.Source: §3 · p. 3, Table 1 · p. 3
  • EAGLE aligns fine-grained visual tokens with object-language representations while preserving the global feature structure for integration into pretrained IT-VLMs.The method uses instance segmentation masks to pool object-associated features and aligns them with language embeddings.Source: §3 · p. 3, Figure 2 · p. 4, §3.1 · p. 4

3.1. Language Feature Alignment

  • Masked average pooling zeros feature tokens outside an instance mask before averaging the remaining tokens into an object representation.For images with multiple segmented instances, EAGLE randomly samples one mask.Source: §3.1 · p. 4
  • EAGLE combines instance-level contrastive alignment with a cross-entropy loss that permits multiple masks to correspond to one semantic class.The final loss integrates both terms without extra hyperparameters.Source: §3.1 · p. 4
  • EAGLE uses OpenImages V7 instance-segmentation labels across 350 object classes, without instructional or VQA training data.The segmented dataset contains 944,037 images and is much smaller than VLM pretraining datasets.Source: §3.1 · pp. 4–5
  • GaLore addresses feature-distribution drift from fine-tuning through low-rank gradient approximations, with updates focused on MLPs and attention linear layers.The reported drift otherwise causes substantial drops in zero-shot accuracy.Source: §3.1 · p. 5

4. Experimental Evaluation

The evaluation tests EAGLE-tuned visual encoders first in VLMs and then across instructional models and hallucination benchmarks. It also examines how visual-token performance and training choices affect the results.

4. Experimental Evaluation

  • EAGLE ViTs reduce VLM false-positive predictions by up to 19.53%, and improve MMVP results across most scenarios, including a 1.49% average boost for EVA-01-CLIP-g-14’s unchanged CLS token.OpenAI CLIP-L-14-336’s feature sequence also improves, surpassing its CLS token’s average performance.Source: §4 · p. 5, §4.1 · p. 6

4.2. Hallucinations in Instructional Models

  • Across POPE, MMVP, and MERLIM, EAGLE improves all evaluated IT-VLMs without alignment or tuning; MMVP gains 2.3% absolute and 11.2% relative, while MERLIM gains 2.73% absolute and 6.3% relative.The benchmark results are summarized in Tables 4 and 5.Source: Table 4 · p. 6, §4.2 · p. 7, Table 5 · p. 7
  • EAGLE does not remove MERLIM prompt bias: it raises precision for both semantically equivalent prompts while preserving their original relative ranking.The evaluated MERLIM subset contains edited images where an entire object category was removed.Source: §4.2 · pp. 6–8
  • LLaVA-v1.5* gains 5.34% on MMVP and 2.93% on MERLIM after instructional tuning with EAGLE’s visual encoder, while directly plugged-in EAGLE models retain 98.5% of original POPE performance.The retrained LLaVA-v1.5* also surpasses the reported 3.3% MMVP improvement in [34].Source: §4.2 · pp. 7–8

4.3. Training Strategy Analysis

  • GaLore is identified as the most important choice for preserving ImageNet-1K zero-shot accuracy while improving sequence embeddings; training the CLS token further lowers false positives but compromises zero-shot performance and generalization.The strategy analysis also reports reduced MS-COCO false positives for sequence embeddings.Source: §4.3 · p. 8, Table 6 · p. 8

5. Conclusions and Limitations

EAGLE directly enhances fine-grained visual grounding in the visual encoder and improves hallucination metrics across benchmarks and architectures. Its use of instance segmentations is constrained by scarce segmentation data.

5. Conclusions and Limitations

  • EAGLE improves hallucination metrics across three standard benchmarks and six diverse architectures, with consistent improvements on challenging MMVP-VLM scenarios.The visual encoder integrates into IT-VLMs without additional alignment or instructional training.Source: §5 · p. 8
  • EAGLE enhances fine-grained visual grounding directly within the visual encoder, rather than relying on additional instructional data, LLM fine-tuning, adapter changes, or multiple visual encoders.The approach is described as straightforward and scalable.Source: §5 · p. 8
  • Instance-segmentation data is relatively scarce, so EAGLE uses parameter-efficient fine-tuning such as GaLore to help prevent overfitting and preserve generalization.The authors encourage exploring surrogate sources of fine-grained supervision.Source: §5 · p. 8

Multimodal Models

Figure 3 compares original and EAGLE-tuned visual encoders in three illustrative IT-VLM scenarios. The figure marks correct and incorrect predicted nouns in green and red, respectively.

Multimodal Models

  • Figure 3 compares responses from IT-VLMs using original visual encoders with responses using corresponding EAGLE-tuned encoders.Original and EAGLE-tuned outputs appear in pink and orange boxes, respectively.Source: Figure 3 · p. 11
  • The three scenarios include LLaVA-1.5, its EAGLE-tuned version, and BLIP-2 with FlanT5xl.The figure labels these models LLA, LLA*, and BT5.Source: Figure 3 · p. 11
  • Green and red markings distinguish correct from incorrect predicted nouns in the examples.Source: Figure 3 · p. 11

6. Linear Probing of EAGLES-Tuned Models

EAGLE-tuned models retain linear-probing performance comparable to the original models, including on the untrained CLS token. Their sequence features improve zero-shot accuracy, with minor degradation in CLS-token zero-shot accuracy.

6. Linear Probing of EAGLES-Tuned Models

  • EAGLE significantly improves zero-shot accuracy of the feature sequence, with only minor degradation in zero-shot accuracy of the CLS token.Source: §6 · p. 11
  • For sequence-feature linear probing, logits are computed for each embedding in the sequence and then averaged.The analysis also evaluates CLS-token embeddings and uses the official implementation provided by.Source: §6 · p. 11
  • EAGLE-tuned models show linear-probing performance comparable to original models for both CLS-token and sequence features.Table 7 summarizes this comparison, including the CLS token, which remains untrained in the proposed pipeline.Source: Table 7 · p. 11, §6 · p. 11

7. Qualitative Examples

Qualitative examples show EAGLE-tuned encoders capturing small or subtle visual details in IT-VLM responses and sequence-feature image-text alignments. The examples include additional objects, partially occluded objects, and image pairs differing in specific details.

7. Qualitative Examples

  • Figure 4 presents two visual examples per model to compare original and EAGLE-tuned encoders’ fine-grained sequence-feature alignment.Source: Figure 4 · p. 12
  • In the Figure 3 examples, EAGLE-tuned encoders add recognition of a car and object arrangement, a motorcycle and bus, and a small partially occluded clock.The examples use LLaVA-1.5, its EAGLE-tuned version, and BLIP-2.Source: §7 · pp. 11–12
  • On MMVP-VLM examples, EAGLE-tuned sequence features correctly align image-text pairs even when images differ in subtle, specific details.Figure 4 compares EVA-01-CLIP-g-14 and OpenAI CLIP-L-14-336 with their EAGLE-tuned versions; green and red arrows mark correct and incorrect alignments.Source: §7 · p. 12, Figure 4 · p. 12

8. Training Hyperparameters

EAGLE training adds hyperparameters to the original EVA01-CLIP-g-14 settings, applying the same additions to both visual encoders and using two A100 GPUs.

8. Training Hyperparameters

  • Two A100 GPUs (80GB) are used to tune EVA01-CLIP-g-14 and OpenAI CLIP-L-14-336.The same added hyperparameters apply to both models.Source: §8 · p. 12
  • EAGLE training adds hyperparameters to the original EVA01-CLIP-g-14 settings provided by.Table 8 summarizes the added hyperparameters, but their values are not included in the supplied passage.Source: §8 · p. 12, Table 8 · p. 12
  • The added hyperparameters are shared between EVA01-CLIP-g-14 and OpenAI CLIP-L-14-336.Source: §8 · p. 12

Related papers

Loading 2501.02699v1…