PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

Spencer Whitehead, Jacob Phillips, Sean Hendryx

arXiv:2409.00238v1cs.CLcs.CV

TL;DR

Multimodal hallucination detection is important but existing approaches generally do not localize erroneous spans, while fine-grained annotations are expensive. The paper formulates detection as sequence labeling and pre-trains detectors on automatically corrupted grounding data, improving fine-tuning sample efficiency and showing that grounding annotations matter.

  • Problem

    Existing multimodal hallucination detectors generally frame detection as classification without localizing hallucinated spans, while fine-grained annotations are costly to collect and scale.

  • Method

    The paper predicts hallucinated response-token spans and pre-trains detectors on data created by replacing grounded phrases with plausible, visually incorrect text-only-LM proposals.

  • Results

    Pre-training on corrupted grounding data improves sample efficiency when fine-tuning across model and data scales, with grounding annotations important for these improvements.

  • Takeaways & Limitations

    Corrupted grounding data provides a scalable pre-training resource for hallucination detectors, and its grounding signal is valuable for learning to detect hallucinations.

  • Takeaways & Limitations

    The pre-training grounded-conversation data and image-description fine-tuning data have a distribution shift that could cause results to vary.

Abstract

from arXiv · show

Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a classification task. In this work, we first pose multimodal hallucination detection as a sequence labeling task where models must localize hallucinated text spans and present a strong baseline model. Given the high cost of human annotations for this task, we propose an approach to improve the sample efficiency of these models by creating corrupted grounding data, which we use for pre-training. Leveraging phrase grounding data, we generate hallucinations to replace grounded spans and create hallucinated text. Experiments show that pre-training on this data improves sample efficiency when fine-tuning, and that the learning signal from the grounding data plays an important role in these improvements.

1 Introduction

The paper frames multimodal hallucination detection as end-to-end sequence labeling and proposes corrupted-grounding pre-training to improve sample efficiency under costly annotation requirements.

  • Multimodal hallucinations can undermine reliability, especially when users rely on model outputs for decisions or assistance.
  • The task predicts which response-token spans are hallucinated from an image, prompt, and response, without requiring predefined spans.
  • Fine-grained hallucination annotations are costly to collect and scale, creating a need for more sample-efficient detectors.
  • The method automatically creates corrupted grounding data by replacing grounded spans with plausible but visually incorrect phrases from a text-only LM.
  • Pre-training on corrupted grounding data improves fine-tuning sample efficiency, reaching up to +7 F1 with 500 fine-tuning samples.
  • Experiments show improvements across model and data scales, while grounding annotations provide an important learning signal.

2 Related Work

Prior work largely detects multimodal hallucinations through classification or tool-based systems without localizing hallucinated spans. This paper instead studies end-to-end span localization and sample-efficient detector training.

  • Chen et al. (2024) develop a tool-based system for hallucination detection across multiple multimodal tasks.
  • Gunjal et al. (2024) formulate hallucination detection as classification without localization, while Wang et al. (2023a) train an evaluator on synthetic data without localization.
  • This work explores end-to-end detection without predefined spans and improves detector sample efficiency using corrupted grounding data.

3 Hallucination Detection

The paper defines multimodal hallucination detection as token-level sequence labeling: given an image and prompt-response pair, the model identifies hallucinated response segments.

  • The input consists of an image and an associated prompt-response pair, and the output marks hallucinated and non-hallucinated response spans.
  • Unlike prior classification setups, models predict a binary label for each response token without receiving predefined spans.
  • Evaluation uses span F1 at a specified intersection-over-union threshold, requiring models to identify spans accurately.

4 Corrupted Grounding Data

The method generates scalable pre-training examples by corrupting grounded spans with text-only language-model proposals. In-filled spans become hallucinations, while retained phrases preserve grounded content.

  • Figure 2 compares fine-tuning-only models with models using pre-training followed by fine-tuning at 500, 1k, and 10k samples.
  • The approach targets annotation-efficient learning because human fine-grained labels are expensive, whereas grounding data can be created automatically at scale despite noise.
  • The approach uses multimodal data with grounding annotations to create corrupted text for detector pre-training.
  • Grounded spans are masked, then a text-only LM proposes contextually plausible phrases that are likely incorrect for the visual context.
  • The procedure restricts original-phrase generation and samples diverse proposals to increase the likelihood of hallucinated replacements.
  • A random subset of masked spans receives proposed phrases and is labeled hallucinated, while remaining spans retain their original phrases.

5 Experiments

The experiments evaluate detector sample efficiency across model sizes and fine-tuning scales, comparing direct fine-tuning with corrupted-grounding pre-training followed by fine-tuning. Pre-training improves low-data performance, while grounded spans and plausible LM-generated hallucinations provide important learning signals.

  • 5.1 Benchmarking Detector Sample Efficiency: 25.30% F1 versus 17.98% at 500 samples for LLaVA-1.6 13B shows that pre-training improves low-data detection performance.For this model, the 500-to-10k performance gap decreases from 13.54% to 6.22% with pre-training.
  • 5.1 Benchmarking Detector Sample Efficiency: At 10k samples, LLaVA-1.6 13B provides the strongest fine-tuned baseline with 31.52% F1.The experiments use 500, 1k, and 10k training subsets and report test F1 at an IoU threshold of 0.5.
  • 5.1 Benchmarking Detector Sample Efficiency: Pre-training yields consistent sample-efficiency improvements across the evaluated LLaVA-1.5 and LLaVA-1.6 models and 7B and 13B sizes.The gains are most pronounced at 500 and 1k fine-tuning samples, and larger models tend to benefit more at lower data scales.
  • 5.1 Benchmarking Detector Sample Efficiency: 81.63% F1 on pre-defined-span classification shows that the fine-tuned detection models can also perform the related classification task.Classification predictions are obtained by majority vote over token labels within each given span.
  • 5.2 Ablations: Random-span masking lowers performance across data scales, and can even hurt 10k-sample fine-tuning, indicating that grounded spans matter.Random in-filling also harms lower-data performance, but less than using random, ungrounded spans.
  • 5.2 Ablations: Pre-training outperforms augmenting the fine-tuning data with corrupted-grounding examples, possibly because of distribution or noise differences.The corrupted-grounding data uses 121k samples from Grounded Visual Chat and T5 to propose hallucinated phrases.

6 Conclusions

The paper frames hallucination detection as token-level localization and uses corrupted grounding data for pre-training to improve detector sample efficiency. Its experiments show improvements across model and data scales, with grounded spans important to those gains.

  • 6 Conclusions: The paper combines sequence-labeling detection with corrupted-grounding pre-training to improve sample efficiency across model and data scales.Grounded spans provide an important learning signal for the pre-training improvements.

7 Limitations

The approach is constrained by annotation noise, nontrivial resources for scaling grounding data, distribution shift, and errors in automatically corrupted examples. Despite these issues, the data remains suitable for pre-training, and cleaner data may improve results.

  • Task noise: Span annotations contain localization noise, including inconsistent punctuation boundaries, which affects both fine-tuning predictions and evaluation.The authors use IoU thresholds instead of exact matching to account for this noise.
  • Costs of scaling grounding data: Creating scalable grounding data still requires nontrivial resources, including multimodal models and systems for matching text spans to bounding boxes.This process is less expensive and more scalable than human annotation but is not resource-free.
  • Distribution shift between pre-training and fine-tuning: The grounded-conversation pre-training data differs from M-HalDetect’s primarily image-description data, so distribution shift could cause results to vary.The authors identify controlling for this shift and measuring generalization as a future direction.
  • Errors in corrupted grounding data: Some automatically proposed hallucinated phrases remain valid but less specific, indicating errors in corrupted grounding data.The authors report performance improvements despite these cases, while noting that removing such noise may yield further gains.

8 Ethical Considerations

The method is intended to improve multimodal reliability by detecting hallucinated spans, but the authors note that it could also be repurposed to encourage hallucinations. Such misuse could increase misinformation risks for users.

  • 8 Ethical Considerations: Although the method targets hallucination detection and reliability, it could potentially be repurposed to encourage hallucinations during multimodal-model alignment.The authors warn that this misuse could negatively affect users and pose misinformation risks.

A Further Ablations

The ablations compare pre-training with alternative training choices and examine whether model weights should remain frozen. They also frame the comparison of pre-training and augmentation across M-HalDetect data scales.

  • Augmentation: Pre-training on the generated data benefits the model more than augmenting fine-tuning with that data across M-HalDetect data scales.The reported sample efficiency of augmentation is noticeably worse, motivating pre-training as the initialization strategy.
  • Ablation design: The ablations test augmentation versus pre-training and assess whether preserving base-model features improves the detector.These comparisons target the training design decisions underlying the proposed approach.
  • Freezing weights: Fully tuning the model is consistently more effective than freezing the base model during pre-training.This suggests that further adapting the model’s learned features is useful.

B Classification Results

The detection setup is adapted to sentence- and span-level classification, while pre-training on corrupted grounding data improves classification sample efficiency. Proprietary GPT models perform strongly on sentence classification but do not localize hallucinations reliably.

  • Classification Results: Adapted detection models trained on the 10k split demonstrate that detection models can also be evaluated for classification, but their results are not directly comparable with Gunjal et al. (2024) because the base models differ.The comparison supports generality of the detection setup rather than a direct model ranking.
  • Classification Results: Pre-training on corrupted grounding data improves classification sample efficiency, although the gain is smaller than for the more challenging detection task.Figure 4 compares fine-tuning-only models with models pre-trained on corrupted grounding data before fine-tuning at 500, 1k, and 10k samples.
  • Proprietary LM Results: Reliable token-level predictions were difficult to obtain from GPT-4 Turbo and GPT-4o for the detection task, so the proprietary-model evaluation uses the simpler sentence-classification setting.The authors identify reliable prompting for token-level detection as a direction for future work.
  • Proprietary LM Results: GPT-4 Turbo and GPT-4o have strong sentence-classification performance, with GPT models slightly outperforming the specifically fine-tuned model from Gunjal et al. (2024).The adapted detector exceeds GPT-4 Turbo but remains below GPT-4o; unlike these alternatives, it localizes hallucinations.
  • Corrupted Grounding Data: The approach uses grounded image-prompt-response triples and is compatible with phrase-grounding datasets, with experiments using 121,907 samples from GVC.GVC contains 449,144 grounded spans across 121,909 samples, averaging 3.684 grounded spans per sample.
  • Corrupted Grounding Data: Corrupted grounding data is generated by masking grounded spans, using T5 to propose replacements that cannot repeat the original phrase, and replacing a randomly sampled 75–100% of proposals.Samples are corrupted with probability 0.95, while the remaining masked spans retain their original phrases.
  • Task Setup: Detection is framed as sequence labeling that predicts boundaries and hallucination labels for contiguous spans, whereas classification uses pre-defined spans and majority voting over their token predictions.The adapted detectors are evaluated with span-level weighted F1 (wF1), matching Gunjal et al. (2024).

I.1 Corrupted Grounding Data

The corrupted grounding data replaces selected grounded spans with hallucinated phrases and assigns hallucination labels for detector pre-training. Qualitative examples show improved span coverage after pre-training, alongside failure cases and noise in the generated data.

  • Data construction: Selected grounded spans are masked and replaced with hallucinated phrases, while unaffected grounded spans remain unchanged.The procedure randomly selects a subset of grounded spans, so some grounded content is not corrupted.
  • Data quality: Analysis of 50 samples found that 66% of proposed phrases were actual hallucinations, while generic phrases accounted for 18% and other spurious errors 6%.The remaining analyzed categories include semantic matches, which are described separately in the supplied passage.
  • Data quality: Because the generated data contains noise, it is better suited for pre-training, yet significant sample-efficiency improvements remain.The authors suggest filtering, candidate re-ranking, or stronger masked language models as possible ways to reduce noise.
  • Qualitative results: Pre-training examples show more correct, contiguous hallucination spans than fine-tuning alone, though one case favors the fine-tuned model.For LLaVA-1.6 13B fine-tuned on 500 samples, PT+FT improves two examples but detects only part of a hallucinated span in another.
Loading 2409.00238v1…