PaperlayerView exact revision on arXiv

Source-linked AI summary

ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

Yahan Tu, Rui Hu, Jitao Sang

arXiv:2409.09318v4cs.CLcs.CV

TL;DR

MLLMs can produce plausible descriptions that do not match image content, and static hallucination benchmarks may be exposed to data contamination. ODE dynamically builds graph-based concept combinations and samples for evaluating object hallucinations. Its evaluations find more pronounced hallucination rates on ODE-generated samples than on static benchmarks, and the samples also support analysis and fine-tuning.

  • Problem

    MLLM outputs can appear plausible while lacking fidelity to image content, making hallucination a persistent challenge.

    Source: §1 · p. 1

  • Method

    ODE uses a graph of object concepts, attributes, and associations to generate open-set samples for existence- and attribute-level hallucination evaluation.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    ODE evaluations find more pronounced hallucination rates than static benchmarks, with patterns differing across distributions, tasks, and models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ODE-generated samples support hallucination-pattern analysis and fine-tuning on identified errors or generated data to reduce hallucinations.

    Source: §1 · p. 2

Abstract

from arXiv · show

Hallucination poses a persistent challenge for multimodal large language models (MLLMs). However, existing benchmarks for evaluating hallucinations are generally static, which may overlook the potential risk of data contamination. To address this issue, we propose ODE, an open-set, dynamic protocol designed to evaluate object hallucinations in MLLMs at both the existence and attribute levels. ODE employs a graph-based structure to represent real-world object concepts, their attributes, and the distributional associations between them. This structure facilitates the extraction of concept combinations based on diverse distributional criteria, generating varied samples for structured queries that evaluate hallucinations in both generative and discriminative tasks. Through the generation of new samples, dynamic concept combinations, and varied distribution frequencies, ODE mitigates the risk of data contamination and broadens the scope of evaluation. This protocol is applicable to both general and specialized scenarios, including those with limited data. Experimental results demonstrate the effectiveness of our protocol, revealing that MLLMs exhibit higher hallucination rates when evaluated with ODE-generated samples, which indicates potential data contamination. Furthermore, these generated samples aid in analyzing hallucination patterns and fine-tuning models, offering an effective approach to mitigating hallucinations in MLLMs.

1. Introduction

Static hallucination benchmarks can be vulnerable to data contamination, while plausible MLLM outputs may lack fidelity to image content. ODE dynamically creates open-set evaluations and experiments find more pronounced hallucinations on its samples than on static benchmarks.

1. Introduction

  • Static benchmarks use fixed test data with limited distributions, risking training-data overlap that can inflate performance metrics.The paper reports better performance on COCO2014 than on newer Internet images under the same semantic distribution.Source: §1 · pp. 1–2
  • ODE is designed for open-set evaluation, with novel data at both sample and distribution levels and dynamic generation across modalities and levels of granularity.The authors also describe applying ODE to general and specialized scenarios, including settings with limited data.Source: §1 · p. 2
  • ODE models object concepts, attributes, and their associations as a graph, then selects combinations using Standard, Long-tail, Random, and Fictional criteria.It generates semantic scenes and queries for existence- and attribute-level object hallucination evaluation.Source: §1 · p. 2
  • ODE evaluations find more pronounced hallucination rates than static benchmarks, with differences across distributions, tasks, and models.The paper also reports that ODE-identified errors or ODE-generated data can be used for fine-tuning that reduces hallucinations.Source: §1 · p. 2
  • The authors report that ODE-generated samples support hallucination-pattern analysis and fine-tuning to reduce hallucinations.They state that targeted refinement on ODE-identified errors or fine-tuning on ODE-generated data improves model performance.Source: §1 · p. 2

2. Related Work

Prior work spans object-hallucination benchmarks, contamination detection, and dynamic evaluation methods. ODE is positioned against fixed evaluation sets by dynamically combining concepts to create open-set samples and distributions.

2.1. Hallucination Evaluation in MLLMs

  • Existing hallucination benchmarks cover object existence, attributes, relations, visual reasoning, events, and other phenomena, but closed-set testing remains an unresolved issue.The passage names CHAIR [22], POPE [14], AMBER [26], HallusionBench [6], Hal-Eval [10], CorrelationQA, VHTest [9], and HaloQuest.Source: §2.1 · p. 2

2.2. Data Contamination

  • Prior work discusses LLM benchmark contamination and detection strategies, while ODE's dynamically distributed, continuously updated data is presented as avoiding contamination risks in static hallucination datasets.The cited discussion includes Zhou et al. [35] and Ni et al. [21] on contamination, and dynamic evaluation strategies by Fan [5], Lei, and Zhu [37].Source: §2.2 · p. 3

2.3. Dynamic Evaluation

  • Dynamic evaluation ranges from costly crowd-based challenges to automated sample generation, rewriting, expansion, and mining.ODE instead dynamically combines meta-concepts and attributes to adapt visual content and textual prompts.Source: §2.3 · p. 3

3. Methodology

ODE constructs graph-based object scenes, samples concept pairs and attributes across distribution criteria, and generates images and inquiries for hallucination testing. Its pipeline supports both general and specialized domains.

3. Methodology

  • ODE's four-stage pipeline models scenarios with a graph, designs semantic scenes, generates and filters images, and creates inquiry templates.The stages are graph modeling, scene design, image generation and filtering, and inquiry-template design.Source: §3 · p. 3

3.1. Graph-Based Conceptual Modeling

  • A weighted graph represents object concepts, their attributes, and inter-concept associations based on real-world co-occurrence.Its nodes include entity and environment concepts; attributes include state and action, while quantity is considered during testing. Edge weights encode co-occurrence strength.Source: §3.1 · pp. 3–4

3.2. Construction of Semantic Scenes

  • ODE selects two concept nodes to form each test-image scene, using entity-entity or entity-environment combinations.The graph-based approach is described as applicable to general and specific domains, including customized detection for specific fields.Source: §3.2 · p. 4, §3.1 · p. 4

3.2.1. Selection Criteria

  • Four criteria sample concept combinations: Standard uses highest co-occurrence, Long-tail uses moderate co-occurrence, Random ignores co-occurrence, and Fictional uses pairs without a co-occurrence record.Standard and Long-tail use commonly observed attributes; Random and Fictional pairs use randomly chosen attributes.Source: §3.2.1 · p. 4

3.2.2. Scene Content

  • Scenes pair two entity categories or an entity with an environmental category, with attributes included or omitted in some samples.The examples include a dog with a frisbee and a deer with a forest, supporting analysis of concept-specific hallucination tendencies.Source: §3.2.2 · p. 5

3.3. Generation and Filtering of Images

  • Text-to-image models generate test images from prompts, and ODE filters out images that lack expected entities at the required confidence threshold.Multiple random-seed variations are produced per test case; the example filtering threshold is below 0.65 confidence.Source: §3.3 · p. 5

3.4. Structuring of Inquiries

  • Inquiry templates evaluate object existence and state, action, and quantity attributes through image-description prompts and yes-or-no questions.Discriminative inquiries also ask about nonexistent objects or attributes to examine hallucinations.Source: §3.4 · p. 5

4. Experiments

ODE evaluates MLLM hallucinations with varied distributions and generated images, and its results show performance differences across distributions, tasks, and models. Fine-grained analysis and dynamic updates extend evaluation beyond fixed samples.

4.1. Setup

  • ODE’s image generation and evaluation setup uses 808 test images from 40 concept combinations and 8,786 factual and hallucination question pairs.The concepts were drawn from AMBER; eight MLLMs were evaluated using official hyperparameters, with generative hallucination metrics and discriminative classification metrics.Source: §4.1 · p. 5

4.2. Effectiveness of Synthetic Images

  • Under the same semantic distribution, COCO2014 images yield higher test performance than Internet and ODE-generated images, while the latter two differ minimally.Synthetic and natural-image features are highly similar in a two-dimensional CLIP feature space, supporting synthetic images as an alternative for open-set evaluation.Source: Figure 4 · p. 5, §4.2 · p. 6

4.3. Main Results

  • MiniGPT-4 and LLaVA, despite static-benchmark F1-scores around 80–90, score lower on ODE standard concept pairs.The authors interpret this as possible reliance on memorized correlations or training-data noise, and argue that static benchmarks can overestimate capabilities.Source: §4.3 · p. 6
  • Performance varies across distributions: models do well on high-frequency concepts, while random and fictional samples raise hallucinations, especially in attribute recognition.Long-tail concepts pose a different challenge: models perform adequately in generative tasks but struggle with object discrimination; the reported results also vary by task and model.Source: §4.3 · pp. 6–7, Figure 5 · p. 6

4.4. Dynamic Benchmarking Updates

  • ODE’s proposed benchmark updates add target-concept pairs, transform attribute combinations, and expand the knowledge base to cover more distributional scenarios.The updates are iterative; a second Stable-Diffusion 1.5 evaluation reports long-tail performance differences across models.Source: §4.4 · p. 7, Table 2 · p. 7
  • A second evaluation using Stable-Diffusion 1.5-generated data reports model performance variations under a long-tail distribution; significant discrepancies indicate differing levels of model comprehension across concepts.Table 2 covers generative and discriminative tasks, including object existence hallucination, across two datasets and reports the difference between datasets.Source: §4.4 · p. 7, Table 2 · p. 7

5. Methodological Applications

ODE-generated samples support analysis of concept-linked hallucination patterns, domain-specific evaluation, and fine-tuning. Experiments report improved performance after fine-tuning, while cluster analysis links higher hallucination rates to contextual similarity or visual ambiguity.

5.1. Hallucination Tendencies in Evaluation Results

  • Clusters combining contextually similar or visually ambiguous concepts, such as indoor and outdoor “car” and “chair,” have higher hallucination rates.The analysis uses a LLaVA fact-hallucination concept-pair frequency matrix and identifies four clusters, including indoor and traffic-scene concepts.Source: §5.1 · p. 7, Table 3 · p. 7

5.2. ODE in Domain-Specific Scenarios

  • ODE’s flexible concept selection and customizable image generation support rare concept combinations for data-scarce domains such as transportation.The authors describe these samples as supplementing real-world scene data and report that fine-tuning with distributionally diverse ODE images improves reliability in specialized domains.Source: §5.2 · pp. 7–8, Figure 6 · p. 8

5.3. Fine-Tuning Enhancements

  • Fine-tuning LLaVA on ODE-identified error samples improves performance on AMBER across generative and discriminative tasks.The fine-tuning setup also uses 800 sample pairs per distribution type for the object-existence hallucination task; Tables 4 and 5 report the task comparisons.Source: §5.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

6. Conclusions

The paper addresses data contamination in MLLM hallucination evaluation with a dynamic open-set protocol. Its initial application evaluates object hallucinations at existence and attribute levels in visual question answering, and its results are described as more reliable than static benchmarks.

6. Conclusions

  • The paper targets data contamination in the evaluation of multimodal large language model hallucinations.Source: §6 · p. 8
  • ODE is a dynamic open-set protocol initially applied to object hallucinations at existence and attribute levels in visual question answering.Source: §6 · p. 8
  • The paper reports that ODE’s experimental results are more reliable than those of static benchmarks.Source: §6 · p. 8

A. Contamination and Synthetic Validity

ODE-generated images perform slightly below Internet data on discriminative tasks, while their feature space closely resembles uncontaminated real images. The comparison also cautions that score differences can reflect semantic differences between Internet and synthetic images.

A. Contamination and Synthetic Validity

B. Evaluation Results

ODE results vary by attribute, distribution, task type, and model. Random, Fictional, and Long-tail conditions expose weaknesses that are less apparent under Standard distributions.

B. Evaluation Results

  • Models perform well on state attributes but show weaknesses on action and number attributes, especially in rare or unseen scenarios.The authors associate state-attribute performance with strong semantic associations in training data.Source: B. Evaluation Results · p. 11
  • Fictional and Random distributions cause sharp performance declines, while Long-tail results highlight difficulty generalizing to sparse data.Standard distributions support high performance, which the authors attribute to frequent training exposure.Source: B. Evaluation Results · p. 11
  • LLaVA-1.5 has the best overall performance with balanced precision and recall, while InstructBLIP has high precision but poor recall.MiniGPT-4 and mPLUG-Owl struggle with generalization, particularly in Random and Fictional contexts.Source: B. Evaluation Results · p. 11
  • Generative tasks emphasize semantic fluency, whereas discriminative tasks require detailed attribute recognition, producing different performance gaps.Source: B. Evaluation Results · p. 11
  • The authors identify training-data diversity and balance, attribute-specific fine-tuning, and robust data augmentation as priorities for mitigating relational hallucination.Source: B. Evaluation Results · p. 11

Related papers

Loading 2409.09318v4…