Source-linked AI summary
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, Jitao Sang
TL;DR
MLLM hallucinations can misrepresent images and matter for deployment and reliability, while existing evaluations are costly and incomplete. AMBER provides an LLM-free benchmark and pipeline covering generative and discriminative tasks and three hallucination types, then evaluates mainstream MLLMs and analyzes mitigation strategies. The results show persistent hallucinations across evaluated models, with discriminative tasks generally producing more hallucinations than generative tasks and attribute hallucination posing a substantial challenge.
Problem
MLLM hallucinations can produce image-unfaithful content with harmful consequences, while existing evaluation methods are costly and insufficiently broad.
Method
AMBER is an LLM-free benchmark and evaluation pipeline covering generative and discriminative tasks plus existence, attribute, and relation hallucinations.
Results
Evaluations of mainstream MLLMs found persistent hallucinations; discriminative tasks generally produced more hallucinations than generative tasks, and attribute hallucination remained challenging.
Takeaways & Limitations
AMBER supports low-cost, multi-dimensional analysis of MLLM hallucinations and informs suggested mitigation strategies.
Takeaways & Limitations
Attribute and relation evaluations remain limited to discriminative tasks, and toolkit-based object extraction can make mistakes.
Abstract
from arXiv · showhide
Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences. Therefore, evaluating MLLMs' hallucinations is becoming increasingly important in model improvement and practical application deployment. Previous works are limited in high evaluation costs (e.g., relying on humans or advanced LLMs) and insufficient evaluation dimensions (e.g., types of tasks and hallucinations). In this paper, we propose an LLM-free multi-dimensional benchmark AMBER, which can be used to evaluate both generative task and discriminative task including existence, attribute and relation hallucination. Based on AMBER, we design a low-cost and efficient evaluation pipeline. Additionally, we conduct a comprehensive evaluation and detailed analysis of mainstream MLLMs including GPT-4V(ision), and also give guideline suggestions for mitigating hallucinations. The data and code of AMBER are available at https://github.com/junyangwang0410/AMBER.
1 Introduction
MLLM hallucinations can be harmful and are important to evaluate, but existing methods are costly and do not cover all task and hallucination types. AMBER addresses these gaps with an LLM-free benchmark and evaluation pipeline spanning generative and discriminative tasks and existence, attribute, and relation hallucinations.
- MLLMs may generate plausible content that is unfaithful to images, creating harmful consequences when users over-rely on them.
- Existing evaluation methods often incur high costs through human or additional-LLM evaluation and omit some task or hallucination dimensions.
- AMBER is an LLM-free benchmark covering both generative and discriminative tasks and existence, attribute, and relation hallucinations.
- The proposed pipeline supports low-cost evaluation, while AMBER is used to analyze nine mainstream MLLMs, including GPT-4V, and suggest mitigation strategies.
2 Related Work
MLLMs offer strong multimodal capabilities but can produce deceptive, image-unfaithful responses. Existing hallucination evaluation is costly or incomplete, motivating AMBER as an LLM-free multi-dimensional benchmark.
- MLLMs integrate visual modules with LLMs to support multimodal multitask learning, but their responses can remain unfaithful to the input images.
- Hallucinations are associated with insufficiently diverse or erroneous training data, lost image attention during generation, and information loss in visual encoders.
- Human- or GPT-4-based evaluation is relatively reliable but expensive, while detection-model approaches require collecting descriptions and training a detector.
- AMBER addresses high cost, LLM dependence, and incomplete task and hallucination coverage with an LLM-free multi-dimensional benchmark.
3 The AMBER Benchmark
AMBER constructs an LLM-free benchmark and evaluation pipeline for generative and discriminative MLLM hallucinations across existence, attribute, and relation dimensions. It evaluates responses using object annotations, task-specific prompts, and complementary metrics.
- Dataset construction: The dataset contains 1,004 manually selected images with clear content, well-defined objects, and annotations validated for accuracy.Images are sourced from open-source test sets and copyright-free repositories, with filtering intended to avoid training-data overlap and incomplete annotations.
- Dataset construction: Annotations represent visible objects, object attributes, direct-contact relations, and counterfactual target objects likely to be imagined by MLLMs.Attributes cover state, number, and human or animal action, while incorrect annotations support counterfactual prompts.
- Prompting MLLMs: AMBER uses image descriptions for generative evaluation and type-specific yes/no prompts for discriminative evaluation.Discriminative prompts test state, number, action, relation, and counterfactual object existence.
- Evaluation pipeline: For generative responses, AMBER extracts nouns and filters them against annotated objects before computing hallucination and coverage measures.The pipeline obtains responses from MLLMs using image-and-instruction inputs, then processes nouns against the annotated object list.
- Metrics: AMBER evaluates generative hallucinations with CHAIR, Cover, Hal, and Cog, while discriminative evaluation uses accuracy, precision, recall, and F1.The AMBER Score integrates generative CHAIR with discriminative F1; precision and recall are calculated on hallucinatory questions, while accuracy covers all questions.
4 Results
AMBER reveals substantial and varied hallucinations across MLLMs, with GPT-4V strongest overall but still imperfect. Hallucinations depend on task type, response length, resolution, model scale, and training data.
- Overall Results: GPT-4V ranks highest overall, with the fewest hallucinations, strongest generative-task object coverage, and best performance across all three discriminative hallucination types.Qwen-VL ranks second and excels in three of four metrics.
- Generative Task: Generative hallucinations remain substantial: most MLLMs exceed 30% Hal, while even advanced models produce about 5% hallucinatory objects on open-ended instructions.CogVLM and Qwen-VL show advantages on CHAIR, Hal, and Cog, whereas MiniGPT-4 has high Cover but more hallucinations.
- Discriminative Task: Discriminative performance shows a strong affirmative-response bias, with recall substantially below precision across MLLMs.The pattern suggests models are easily misled by hallucinatory question content.
- Multi-dimensional Results: Relation hallucination is hardest, with most models near 0.7 accuracy; attribute hallucination is also challenging, with the best F1 around 0.8.Existence hallucination is less severe, although precision is near 1 while recall lags.
- Hallucination Position: Hallucinatory objects cluster in the middle and latter response portions, possibly reflecting snowballing or error accumulation.The analysis examines four relatively low-hallucination models in the generative task.
- Response Length and Tendency: Longer responses increase both object coverage and hallucinations, exposing a trade-off between conservative generation and response quality.Negative responses in training are suggested to counter affirmative bias and reduce erroneous content.
- Visual Resolution: Increasing image resolution generally decreases hallucinations, while GPT-4V is minimally affected and therefore robust to resolution changes.AMBER Score variation with resolution is reported in Table 5.
- LLM Scale: LLM scale has inconsistent effects across MLLMs, suggesting hallucination formation may depend more on vision or connectivity modules.MiniGPT-4 and LLaVA-1.5 exhibit opposing trends across generative and discriminative tasks.
5 Conclusion
The paper introduces AMBER as an LLM-free, multi-dimensional benchmark for evaluating MLLM hallucinations across tasks and hallucination types. It uses AMBER to analyze mainstream models and identify factors and mitigation suggestions relevant to improving reliability.
- Conclusion: AMBER evaluates hallucinations in both generative and discriminative MLLM tasks across existence, attribute, and relation types.The benchmark is designed to address high evaluation costs and insufficient dimensional coverage.
- Conclusion: The study analyzes mainstream MLLMs with AMBER, explores factors related to hallucinations, and provides suggestions for mitigation and future reliability improvements.Future work will investigate hallucination mitigation more deeply.
6 Limitations
AMBER improves evaluation efficiency, cost, and quality but has limitations in task coverage, extraction accuracy, and annotated object diversity.
- Attribute and relation hallucination evaluations are limited to discriminative tasks because effective generative descriptions cannot be guaranteed.
- Language-toolkit extraction can misclassify adjectives or compound-noun components as hallucinated objects.Examples include identifying “orange” as a noun and “water” within “water bottle” as a separate object.
- AMBER’s annotated hallucinatory target objects remain limited, so less common image objects may be overlooked.The paper recommends expanding target objects as MLLM recognition capabilities improve.
A.1.1 Data Source
AMBER draws images from the MS-COCO 2014 test set and Unsplash, using sources with no training annotations or permissive licensing.
- AMBER sources images from the MS-COCO 2014 test set and Unsplash.
- The MS-COCO 2014 test set has no annotations and is therefore not used as training data.
- Unsplash images use the Creative Commons Zero license, allowing free use without attribution.
A.1.2 Data Dtatistics
AMBER’s annotations cover 337 objects, exceeding existing benchmarks by more than four times.
- 337 objects are covered by AMBER’s resultant annotations.The dataset statistics table reports the number of categories and objects represented in the data.
- AMBER covers more than four times as many objects as existing benchmarks, including 80 specific objects in COCO.
A.1.3 Object Distribution
AMBER evaluates hallucinations across broad object coverage, multiple hallucination dimensions, task types, and model behaviors. Analyses examine object preferences, attribute weaknesses, scale effects, and GPT-4V’s residual errors.
- Object Distribution: AMBER covers 14 relatively balanced object categories without a significant long-tail phenomenon.It additionally covers Nature, Architecture, and Street View, while extending categories such as Fruit.
- Attribute Hallucination: Attribute hallucination analysis separates state, number, and action into refined evaluations.
- Attribute Hallucination: MLLMs generally perform poorly on state and number attributes, motivating greater coverage of these object details in training data.
- Attribute Hallucination: MLLMs generally perform well on action attributes, with available open-source models outperforming GPT-4V in this perspective.
- Hallucinatory Object Preferences: Hallucinatory-object preferences vary by model: open-source MLLMs tend toward people, whereas GPT-4V tends toward background objects such as sky, sun, and clouds.When people are present, boat, backpack, and bench are frequent hallucinated objects.
- Ablation Results: Increasing LLM size reduced generative hallucinations and improved entity coverage for LLaVA-1.5, while MiniGPT-4 improved generative quality without significantly increasing hallucinations.For LLaVA-1.5, the 13B model performed poorly on the discriminative task.
- GPT-4V Examples: Even GPT-4V produces absurd hallucinations, while AMBER accurately detects them in evaluation examples.