PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Open-Vocabulary Caption Hallucinations

Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, Hadar Averbuch-Elor

arXiv:2312.03631v4cs.CVcs.AI

TL;DR

Closed-vocabulary object lists limit how hallucinations in image captions are evaluated and mitigated. The paper introduces OpenCHAIR for open-vocabulary evaluation and MOCHa for mitigation, reporting improved performance across diverse hallucination types and captioning models.

  • Problem

    Existing hallucination methods largely rely on closed-vocabulary object lists, which do not capture the diversity of open-vocabulary hallucinations in modern image captioning.

    Source: §1 · pp. 1–2

  • Method

    OpenCHAIR evaluates open-vocabulary object hallucinations, while MOCHa uses multi-objective reinforcement learning to optimize caption fidelity and adequacy without a closed object list.

    Source: §1 · pp. 1–2

  • Results

    MOCHa improves performance across diverse hallucination types and captioning architectures, as shown by OpenCHAIR and other metrics.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    OpenCHAIR and MOCHa provide a flexible framework for measuring and reducing object hallucinations while preserving caption quality across model sizes and architectures.

    Source: §6 · pp. 8–9

    OpenCHAIR measures object hallucinations, not nonobject hallucinations such as hallucinated attributes or relations between entities.

    Source: §7 · p. 9

Abstract

from arXiv · show

While recent years have seen rapid progress in image-conditioned text generation, image captioning still suffers from the fundamental issue of hallucinations, namely, the generation of spurious details that cannot be inferred from the given image. Existing methods largely use closed-vocabulary object lists to mitigate or evaluate hallucinations in image captioning, ignoring the long-tailed nature of hallucinations that occur in practice. To this end, we propose a framework for addressing hallucinations in image captioning in the open-vocabulary setting. Our framework includes a new benchmark, OpenCHAIR, that leverages generative foundation models to evaluate open-vocabulary object hallucinations for image captioning, surpassing the popular and similarly-sized CHAIR benchmark in both diversity and accuracy. Furthermore, to mitigate open-vocabulary hallucinations without using a closed object list, we propose MOCHa, an approach harnessing advancements in reinforcement learning. Our multi-objective reward function explicitly targets the trade-off between fidelity and adequacy in generations without requiring any strong supervision. MOCHa improves a large variety of image captioning models, as captured by our OpenCHAIR benchmark and other existing metrics. Code and models can be found at: https://github.com/assafbk/mocha_code

1 Introduction

Image captioning models can produce spurious details that undermine confidence, while closed-vocabulary methods miss the diversity of real-world hallucinations. The paper proposes open-vocabulary evaluation with OpenCHAIR and mitigation with MOCHa.

1 Introduction

  • Hallucinated details can damage user confidence or lead to uncritical acceptance of fallacious, potentially dangerous content.Source: §1 · p. 1
  • Closed-vocabulary object lists simplify hallucination evaluation and mitigation but fail to capture the diversity of hallucinations in modern captioning models.Source: §1 · pp. 1–2
  • OpenCHAIR addresses the lack of an open-vocabulary benchmark by using generative models to create data and perform evaluation.Source: §1 · pp. 1–2
  • OpenCHAIR captures diverse open-vocabulary hallucinations more effectively than CHAIR, complementing CHAIR’s evaluation of 80 common object types on natural images.Source: §1 · pp. 1–2
  • MOCHa jointly optimizes sequence-level fidelity and semantic adequacy, avoiding the limits of optimizing either goal alone.Source: §1 · p. 2
  • MOCHa improves performance across diverse hallucination types, metrics, captioning architectures, and model sizes without requiring direct supervision.Its optimization uses learned text-based metrics and a multi-objective reinforcement-learning reward.Source: §1 · p. 2

2 The OpenCHAIR Benchmark

OpenCHAIR extends object-hallucination evaluation beyond a fixed vocabulary by pairing synthetic images with diverse object captions and evaluating free-text predictions. Its construction and LLM-based evaluation provide a complementary alternative to CHAIR’s fixed-vocabulary approach.

2 The OpenCHAIR Benchmark

  • OpenCHAIR targets arbitrary objects and free-text predictions, complementing CHAIR’s closed set of 80 common objects on natural images.Its approximately 5K images illustrate diverse object types in context.Source: §2 · pp. 2–3
  • The benchmark uses Llama-2 to generate diverse captions, selects and balances concrete nouns, then uses Stable Diffusion XL to create paired images.Manual filtering removed 3% of the 5K generated image-caption pairs to assure alignment and general quality.Source: §2 · p. 3
  • Example images are paired with ground-truth captions to illustrate the benchmark’s diverse coverage of object types.Source: Figure 3 · p. 3
  • Unlike CHAIR, OpenCHAIR counts “guitar” and “man” as hallucinations in the illustrated example, without relying on a fixed vocabulary or synonym list.Source: Figure 4 · p. 3, §2 · p. 3
  • OpenCHAIR parses predicted captions for objects and prompts an LLM to check whether each object appears in the paired ground-truth caption.The hallucination rate is the number of hallucinated objects divided by the total objects considered; LLM “no” answers count as hallucinations.Source: §2 · p. 3

3 The MOCHa Framework

MOCHa uses reinforcement learning to mitigate open-vocabulary caption hallucinations by jointly optimizing fidelity and semantic adequacy. Its reward combines learned quality signals, and PPO applies iterative updates to the captioning model.

3 The MOCHa Framework

  • MOCHa iteratively samples caption batches from a model and updates it to favor captions with higher multi-objective rewards.The process repeats until convergence.Source: §3 · pp. 3–4, Figure 5 · p. 4

3.1 Reward Function

  • The reward balances fidelity, measured through NLI-based contradiction with references, against adequacy, measured with BERTScore, while KL regularization constrains drift from the initial policy.The fidelity–adequacy weighting is controlled by α, and the KL penalty helps preserve output diversity and limit reward over-optimization.Source: §3.1 · p. 4

3.2 Learning Procedure

  • PPO optimizes the expected reward with a clipped surrogate objective intended to avoid instability.MOCHa repeats the RL optimization step after collecting a full batch of rewards.Source: §3.2 · pp. 4–5, §3 · p. 4

4 Experiments and Results

The experiments evaluate OpenCHAIR’s coverage and human-judged accuracy, then test MOCHa across captioning models and compare it with prior methods. The reported results show broader object coverage, higher OpenCHAIR accuracy than CHAIR, and improvements from MOCHa across most measures and model architectures and sizes.

OpenCHAIR Analysis

  • 2.4K unique object types in OpenCHAIR represent an approximately 30-fold increase over CHAIR’s 80, with 22% of OpenCHAIR types appearing only once.Both benchmarks contain approximately 5K images; 53% of OpenCHAIR object types appear at most three times.Source: §4 · p. 5
  • OpenCHAIR outperforms CHAIR in per-object predictive accuracy despite its broader scope, with the reported difference attributed to finer-grained matching rather than coarse synonym lists.The authors note a trade-off between true and false positives because some objects present in images may be absent from OpenCHAIR’s generated ground-truth descriptions.Source: §4 · p. 5, Table 1 · p. 5, Figure 6 · p. 5

MOCHa Results

  • MOCHa-optimized captions in the qualitative examples remain image-consistent while preserving a satisfying level of detail.The examples use BLIP-Large on COCO with beam search decoding using five beams.Source: Figure 8 · p. 6, §4 · p. 7

Experimental Setup

  • The evaluation uses MS-COCO’s Karpathy split, reporting captioning metrics alongside CHAIR, OpenCHAIR, NLI, and BERTScore.The tested models include BLIP, BLIP-2, and GIT; the reported training uses language-model fine-tuning followed by PPO with α = 0.5.Source: §4 · p. 6

MOCHa Results

  • MOCHa improves most measures across model architectures and sizes, including general caption quality and both closed- and open-vocabulary hallucination metrics.Against TLC-A, it improves open-vocabulary hallucination rate without the caption-quality degradation described for that method; LURE degrades performance across metrics in this captioning setting.Source: §4 · p. 7, Table 2 · p. 7

Ablations

  • MOCHa’s joint reward improves the fidelity–adequacy trade-off over the base model, while fidelity-only optimization harms general quality and adequacy-only optimization fails to improve hallucinations.Varying α traces a Pareto frontier; removing the fidelity reward increases hallucinations, while removing adequacy reduces descriptive detail.Source: Figure 9 · p. 6, §4 · p. 7

5 Related Work

Prior work measures hallucinations through similarity metrics or object-level evaluation and mitigates them through methods often restricted to closed vocabularies. The paper positions its benchmark and mitigation approach within this landscape, distinguishing caption evaluation from related open-vocabulary VQA assessment and supervised RLHF approaches.

Measuring VLM Hallucinations

  • Similarity-based metrics estimate text fidelity but do not provide a discrete hallucination count, while POPE assesses VQA model answers rather than predicted captions.Figure 10 organizes metrics and mitigation algorithms and marks the automatic hallucination-rate metric used to evaluate each algorithm.Source: §5 · pp. 7–8, Figure 10 · p. 8

Reducing VLM Hallucinations

  • Many caption-mitigation methods target object hallucinations within closed vocabularies, whereas MOCHa addresses the open-vocabulary setting.LURE also proposes an open-setting method, but it was not automatically evaluated in that setting because no benchmark existed.Source: §5 · p. 8

Deep RL for VLM Text Generation

  • Related RL-based VQA methods require laborious human annotation for supervised reward models, while this approach requires no explicit supervision.RLHF for language and multimodal models commonly relies on manual human preference annotations to train reward models.Source: §5 · p. 8

6 Conclusion

The conclusion presents open-vocabulary evaluation and mitigation as a framework for reducing caption hallucinations while preserving caption quality across models.

6 Conclusion

  • OpenCHAIR measures open-vocabulary hallucinations, while MOCHa optimizes captioning models to reduce them while preserving caption quality.The authors report this reduction on OpenCHAIR and other existing metrics, and state that the methods apply across model sizes and architectures.Source: §6 · pp. 8–9

7 Limitations

OpenCHAIR measures object hallucinations but not hallucinated attributes or relations, while captioners may behave differently on its synthetic images than on natural images. The work is limited to image captioning and does not fully solve hallucinations.

7 Limitations

  • OpenCHAIR does not directly measure hallucinated attributes or relations, although MOCHa also targets these non-object hallucinations.The authors identify extending the benchmark to attribute-, relation-, and predicate-level hallucinations as future work.Source: §7 · p. 9
  • Captioning models may perform differently on OpenCHAIR’s synthetic images than on natural images, though the authors report empirical correlation with other metrics and human intuition.The authors note that OpenCHAIR uses synthetic images.Source: §7 · p. 9
  • The work focuses on image captioning and does not solve the hallucination problem completely.The authors point to VQA and visual instruction-following as other VLM tasks where hallucinations remain a challenge.Source: §7 · p. 9

8 Ethics Statement

The authors expect this work on measuring and mitigating VLM hallucinations to support reliability and performance measurement in real-world systems, while cautioning that foundation models used in the work could propagate biases. An interactive visualization presents BLIP-Large captions with and without MOCHa for sampled test images.

8 Ethics Statement

  • Foundation models used to construct and evaluate OpenCHAIR and calculate MOCHa rewards could propagate biases, so further research is anticipated before relying on the work beyond research settings.The authors expect the work to increase VLM reliability and measurement ability for real-world systems.Source: §8 · p. 9

A Interactive Visualization

  • The interactive visualization compares BLIP-Large with and without MOCHa on 350 randomly selected MS-COCO and Flickr30K test images using NLI contradiction-probability differences.Samples are ordered by n-gram similarity between predictions, placing the most different first.Source: A Interactive Visualization · p. 12

B.1 MOCHa Implementation Details

The appendix details MOCHa training, OpenCHAIR construction and evaluation, and comparisons with LURE and TLC-A. These details describe the reward adjustment and implementation settings, the benchmark pipeline, and the comparison procedures.

B.1 MOCHa Implementation Details

  • MOCHa shifts rewards to zero mean by subtracting the minibatch sample mean before adding the KL penalty.This adjustment is described as reducing variance in gradient estimates.Source: §B.1 · p. 12

B.2 OpenCHAIR Implementation details

  • OpenCHAIR captions are generated with LLMs from COCO captions, expanded toward rare objects, and rendered as synthetic images with Stable Diffusion XL.The pipeline filters captions for rare objects, defined as appearing in the dataset’s lowest 10th percentile, before using them as few-shot examples for further LLM generation.Source: §B.2 · pp. 12–13

Evaluation on the OpenCHAIR Benchmark

  • OpenCHAIR extracts caption objects and uses an LLM to judge whether each appears in the ground-truth caption; objects judged absent count as hallucinations.Objects receiving answers other than yes or no are ignored; the reported OpenCHAIR rate is hallucinated objects divided by hallucinated plus existing objects.Source: Evaluation on the OpenCHAIR Benchmark · p. 13

B.3 LURE Comparison

  • The LURE comparison applies its pretrained revisor to BLIP-Large and BLIP-2 beam-search captions, supplying token probabilities for the top-scoring beam before evaluating revised captions.The setup uses five beams and evaluates the revised captions with various metrics.Source: §B.3 · p. 13

B.4 TLC-A Comparison

  • TLC-A’s recommended 99% precision threshold was unattainable for the considered models without sacrificing most recall, so the authors selected a threshold by validation-set CHAIR performance.The selected threshold was 0.33, achieving 98.3% precision and 84% recall on the validation set; the precision-recall curve is shown in Figure 11.Source: Figure 11 · p. 13, §B.4 · p. 14

C.1 Full Quantitative Results

MOCHa improves hallucination scores on difficult images, while OpenCHAIR covers far more object types than CHAIR. OpenCHAIR evaluation with Llama-3-8B-Instruct shows similar evaluation and improvement trends to Llama-2-70B-chat-hf.

C.1 Full Quantitative Results

  • 7 points (20%) improvement in OCH score for BLIP-Large on images prone to object hallucination; the reported hard-question gains are 10 points for BLIP-Base, 4 for BLIP-2, and 13 for GIT-Base.Such images occur on approximately half of the images; easy objects detected both before and after tuning reduce the overall OCH improvement.Source: §C.1 · p. 14

C.3 OpenCHAIR Efficiency

  • The Instruct model shows evaluation and overall improvement trends similar to Llama-2-70Bchat-hf (4-bit quant.); Table 9 shows OCH scores improve after MOCHa-tuning.Source: §C.3 · p. 15

C.4 Additional Ablations

MOCHa’s ablations show that its reward components and optimization choices address distinct failure modes, while comparisons test its performance across models, datasets, and evaluation settings.

C.4 Additional Ablations

  • Using both fidelity and adequacy rewards reduces hallucinations while preserving or improving caption quality; removing either degrades hallucination or text-quality performance.Figure 14 illustrates hallucinations without fidelity and generic captions without adequacy.Source: §C.4 · p. 15, Table 10 · p. 17, Figure 14 · p. 18
  • OpenCHAIR covers significantly more unique object types than CHAIR, according to the benchmark coverage comparison.OpenCHAIR objects are identified using the parsing method described in Section B.2.Source: Figure 13 · p. 15
  • Without the KL penalty, optimization improves the NLI-based reward but degrades other text-quality measures, while the penalty mitigates over-optimization.The no-KL captions are generic; the full-reward model produces descriptive captions consistent with the image.Source: §C.4 · p. 15, Table 7 · p. 17
  • PPO outperforms SCST across metrics; the authors relate this to prior findings that PPO avoids optimization instabilities.SCST is described as the common name for REINFORCE in image captioning.Source: §C.4 · pp. 15–16, Table 7 · p. 17

C.5 Additional Comparisons

  • Older dedicated reduced-hallucination methods with weaker backbones are outperformed by modern VLMs on all reported metrics, including CHAIR.The comparison uses the MS-COCO Karpathy test split.Source: §C.5 · p. 16, Table 8 · p. 17
  • MOCHa outperforms LLaVa-RLHF on open-vocabulary hallucinations over 100 OpenCHAIR samples, using greedy decoding for both models in the fair comparison.LLaVa-RLHF was also evaluated with stochastic sampling; its generations were terminated after one sentence for comparison.Source: §C.5 · p. 16, Table 11 · p. 18
  • Zero-shot MOCHa tuning on Flickr30K improves optimized metrics while preserving similar values on other text-quality metrics; on CC3M, NLI P(Contradict) improves 12.9% and BERTScore 1.44%.The model was tuned on COCO, not either evaluation dataset.Source: §C.5 · p. 17, Table 12 · p. 18
  • OpenCHAIR evaluations by small and large LLMs correlate well and capture the same trends in MOCHa model improvement.The table describes relative improvement in OpenCHAIR scores for MOCHa-tuned models.Source: Table 9 · p. 17

D.1 Similarity Based Metrics

The discussion contrasts similarity-based hallucination metrics with explicit benchmark evaluation and describes several closed-vocabulary mitigation methods alongside MOCHa’s open-vocabulary approach.

D.1 Similarity Based Metrics

  • CLIPScore and related embedding metrics assess text-image similarity, but CLIP can score contradictory hard negatives highly; OpenCHAIR instead makes predicted captions directly examinable.Other discussed methods use object-annotation or detection similarity, or learned fidelity metrics trained on generated scene graphs.Source: §D.1 · p. 18

D.2 Closed Vocabulary Algorithms

  • MOCHa’s improvements on OpenCHAIR are similar before and after manual filtering, consistent with the small proportion of generated data removed.The table compares MOCHa improvements across various models.Source: Table 13 · p. 19
  • Tuning α allows MOCHa-trained captioners to produce different output lengths.The table reports results for BLIP-Large over MS-COCO.Source: Table 14 · p. 19
  • Closed-vocabulary methods include debiased captions, detector or decoder changes, object-token confidence filtering, masking object lists, and LLM-guided decoding; MOCHa does not rely on a closed object list.The passage identifies UD-L [Biten et al., 2021], CIIC, TLC [Petryk et al., 2023], ObjMLM [Dai et al., 2023], and Woodpecker [Yin et al., 2023].Source: §D.2 · pp. 18–19

Related papers

Loading 2312.03631v4…