PaperlayerView exact revision on arXiv

Source-linked AI summary

PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training

Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, Chunhua Shen

arXiv:2503.06486v1cs.CVcs.AI

TL;DR

MLLM hallucinations are a persistent problem, especially for dense image captioning, where detailed descriptions must remain faithful to visual content and existing metrics lack fine-grained completeness assessment. The paper introduces graph-based HalFscore and trains PerturboLLaVA with adversarially perturbed text to reduce reliance on language priors. It reports hallucination suppression without added training or inference costs and improved performance across multimodal benchmarks.

  • Problem

    Dense image captioning requires detailed, visually faithful descriptions, but existing hallucination metrics lack fine-grained analysis and do not assess caption completeness.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    HalFscore compares language graphs for generated captions and ground truth, while PerturboLLaVA uses adversarially perturbed text during training to reduce reliance on language priors.

    Source: §1 · pp. 1–2, A Fine-Grained & Comprehensive… · p. 4

  • Results

    PerturboLLaVA suppresses hallucinations without additional training or inference costs and boosts performance across multimodal benchmarks; versus LLaVA1.5, HalFscore precision increases by +6.2.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The paper presents perturbative visual training as a scalable, efficient approach that integrates into existing training pipelines with minimal additional cost.

    Source: §1 · p. 2

Abstract

from arXiv · show

This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs) particularly for dense image captioning tasks. To tackle the challenge, we identify the current lack of a metric that finely measures the caption quality in concept level. We hereby introduce HalFscore, a novel metric built upon the language graph and is designed to evaluate both the accuracy and completeness of dense captions at a granular level. Additionally, we identify the root cause of hallucination as the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded descriptions without incurring additional computational overhead. PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved performance across general multimodal benchmarks.

1 INTRODUCTION

MLLMs continue to hallucinate visual content, and dense image captioning makes reliable, detailed descriptions especially important. The paper introduces HalFscore to assess caption accuracy and completeness, then proposes perturbative visual training to reduce reliance on language priors; it reports fewer hallucinations without added training or inference costs.

1 INTRODUCTION

  • MLLMs can produce descriptions that do not reflect the image, and scaling parameters or training data has not proven effective at mitigating this problem.Dense captioning requires comprehensive detail while maintaining fidelity to visual content.Source: §1 · p. 1
  • HalFscore evaluates dense-caption accuracy and completeness at a fine-grained level by comparing language graphs for generated captions and ground truth.The metric identifies incorrect and missing concepts, aggregating precision and recall.Source: §1 · p. 1
  • The paper attributes hallucinations to models’ over-reliance on pretrained linguistic knowledge, which can draw generation away from visual information.As text accumulates, models may favor preceding language patterns; a green banana may therefore be described as yellow.Source: §1 · pp. 1–2
  • Perturbative training inserts adversarial text that aligns with general knowledge but conflicts with the image, encouraging the model to scrutinize visual content when predicting tokens.The method is described as shifting the conditional distribution toward the image and away from perturbation text.Source: §1 · p. 2
  • PerturboLLaVA suppresses hallucinations without additional training or inference costs and improves performance across multimodal benchmarks.The paper says this makes the method more suitable for real-world applications.Source: §1 · p. 2

2 RELATED WORK

The related work describes MLLM architectures and persistent hallucinations, surveys evaluation gaps in open-ended tasks, and contrasts training-based and decoding-based mitigation approaches.

2.1 MULTIMODAL FOUNDATIONAL MODEL

  • MLLMs combine a vision encoder, language-model backbone, and projector, typically trained through modality-alignment pretraining and instruction fine-tuning; hallucinations remain persistent.Figure 2 compares methods, highlighting hallucinations in red and detailed descriptions in blue, and reports richer details from PerturboLLaVA.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 EVALUATION OF HALLUCINATION

  • Open-ended hallucination metrics lack fine-grained analysis: CHAIR focuses on objects, while MMHalBench gives a holistic score without detailed analysis or completeness assessment.Open-ended tasks require detailed image understanding and dense outputs.Source: §2.2 · p. 3

2.3 MITIGATION OF HALLUCINATION

  • Training-based mitigation can require reward models or extra training phases, while decoding methods are training-free but do not address hallucinations’ training-time roots.The paper proposes a training strategy intended to avoid the additional overhead of prior training-based approaches.Source: §2.3 · pp. 3–4

3 HUALLUCINATION F-SCORE

The paper argues that existing hallucination metrics miss fine-grained caption errors and completeness, and introduces HalFscore to compare generated and ground-truth concept graphs.

3 HUALLUCINATION F-SCORE

  • Existing metrics are limited: CHAIR matches objects but not attributes or relations, while MMHalBench’s holistic score lacks detail and prior metrics omit caption completeness.This can favor short, confident captions despite missing image details.Source: §3 · p. 4

A Fine-Grained & Comprehensive Hallucination-Metric

  • HalFscore compares graphs built from model captions and dense ground-truth annotations to identify hallucinated and omitted concepts.The paper uses dense annotations for 1,000 images from the DCI dataset as ground truth.Source: A Fine-Grained & Comprehensive… · p. 4
  • Captions are parsed into triplets that represent entities and their relations or attributes, then organized into graphs with entity nodes and relation edges.Matching the graphs identifies hallucinated and omitted concepts using GPT-4o.Source: A Fine-Grained & Comprehensive… · p. 5

4 MITIGATION OF HALLUCINATION VIA PERTURBATIVE VISUAL TRAINING

PerturburboLLaVA trains multimodal models with contextually adaptive, misleading text so they rely more on visual input. Its mathematical explanation frames this training as steering predictions away from language-prior influence, while the method adds no inference overhead.

4.1 PERTURBATIVE VISUAL TRAINING

  • Perturbative visual training inserts contextually adaptive, misleading text alongside image-question tokens to encourage responses grounded in image content.The perturbation is part of the input, with no direct loss computed on it.Source: §4.1 · p. 5
  • Perturbations are designed to be plausible, align with pretrained knowledge, and vary semantically; GPT-4o generates them using the image, question, and answer without disclosing the answer.The design aims to make perturbations realistic but misleading.Source: §4.1 · p. 5
  • PerturboLLaVA avoids preference-data collection and reward-model training, adds minimal training cost, and introduces no inference overhead.Its effectiveness scales with the quality and diversity of perturbation texts.Source: §4.1 · p. 6

4.2 MATHEMATICAL EXPLANATION

  • Because the desired image-based prediction without language-prior interference is difficult to model directly, the method adds perturbed text before the instruction.The account also states that p(xk) can be ignored in a sufficiently uniform dataset.Source: §4.2 · p. 7
  • The mathematical account treats the prediction tendency induced by perturbed text as an unoptimizable language-prior term and describes training as steering toward predictions without that influence.The passage characterizes the resulting model as relying on image information to answer questions.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Experiments report stronger HalFscore and general multimodal results for PerturboLLaVA, while analyses show trade-offs as perturbation strength increases. The evaluation also covers benchmark comparisons, human alignment, and compatibility with decoding strategies.

5.1 EXPERIMENTS SETTINGS

  • Experiments replicate LLaVA1.5 training on its dataset and apply GPT-4o-generated perturbations to 160k VQA examples, comparing with RLAIF-V, OPERA, and VCD.The RLAIF-V setup uses a LLaVA-Next 34B reward model, which may transfer capabilities and make comparisons less fair; the metric is also evaluated on six multimodal models.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • On HalFscore, PerturboLLaVA improves over LLaVA1.5 by +6.2 in precision, +0.7 in recall, and +2.4 in Fscore, leading comparators in precision and Fscore.The metric also separates object, attribute, and relation hallucinations; relation hallucinations remain an area for improvement.Source: Table 2 · p. 8, §5.2 · p. 8
  • PerturboLLaVA avoids the general-performance degradation seen with VCD and RLAIF-V, gaining +1.6 in MMB, +0.3 in SEEDImage, and +1.2 in CCBench.The authors attribute these gains to perturbation training that encourages more effective use of image information.Source: Table 3 · p. 8, §5.2 · p. 9
  • On CHAIR, PerturboLLaVA beats LLaVA1.5, OPERA, and VCD but trails RLAIF-V; HallusionBench improves +0.6, while OPERA gains +0.2 and VCD maintains baseline.The paper describes HallusionBench as covering multiple-choice and vision-reasoning tasks in addition to dense-caption evaluation.Source: Table 3 · p. 8, §5.2 · pp. 8–9

5.3 ANALYSES

  • Adding OPERA at inference to PerturboLLaVA further improves HalFscore by 0.6, CHAIRs by 3 points, HallusionBench by +0.1, and CCBench by 0.4.The paper presents the training strategy as complementary to post-SFT optimization methods.Source: Table 3 · p. 8, §5.3 · p. 10
  • Increasing perturbation strength reduces hallucinations on HalFscore and CHAIR but shortens captions, lowers recall, and consistently reduces general-performance metrics.The analysis compares three insertion approaches, from warning the model about perturbations to providing no guidance.Source: Table 5 · p. 9, §5.3 · p. 9, Table 4 · p. 9
  • HalFscore precision and recall align with human evaluations, and its precision correlation is higher than that of MMhalbench.The user study compares four methods, with each method paired against others 12 times.Source: Table 6 · p. 10, §5.3 · p. 10
  • Random text perturbations still improve HalFscore and HallusionBench over LLaVA1.5, but targeted perturbations are more effective.The authors suggest relevant perturbations exert a stronger disruptive effect that prompts greater focus on the image.Source: §5.3 · pp. 9–10

6 CONCLUSION

The conclusion presents HalFscore as a concept-level measure of dense-caption quality and PerturboLLaVA as a training strategy for reducing reliance on linguistic priors. It reports improved performance over leading approaches without additional training or inference overhead.

6 CONCLUSION

  • HalFscore enables fine-grained analysis of hallucinations involving objects, attributes, and relations, while also reflecting overall captioning capability.The conclusion presents the metric as a way to evaluate multimodal models' dense captions at the concept level.Source: §6 · p. 10
  • PerturboLLaVA guides multimodal models to reduce reliance on linguistic priors and prioritize image information during training.Source: §6 · p. 10
  • The method outperforms leading state-of-the-art approaches without additional training or inference overhead.Source: §6 · p. 10

A.1.1 STATISTIC OF IMAGE-CAPTION PAIR

The evaluation image data comprises 1,000 manually selected DCI images, with indoor and outdoor scenes represented in different proportions. The selection was intended to support evaluation across varied scenarios.

A.1.1 STATISTIC OF IMAGE-CAPTION PAIR

  • 1,000 images were manually selected from DCI to provide high-quality, diverse image data.Source: §A.1.1 · p. 14
  • 69.3% of the selected images are outdoor scenes and 30.7% are indoor scenes.Source: §A.1.1 · p. 14, Table 8 · p. 14
  • Indoor and outdoor scenes were further divided into categories to support evaluation of hallucinations across varied scenarios.Source: §A.1.1 · p. 14

A.1.2 CONSTRUCTION OF CONCEPT GRAPH

The concept-graph construction pipeline parses captions into subject–object–relation triplets, then organizes repeated entities as nodes and relations as directed edges. The resulting graph supports hallucination and omission analysis.

A.1.2 CONSTRUCTION OF CONCEPT GRAPH

  • Caption parsing extracts triplets representing subjects, objects, and their relations, including attributes, states, and actions.Examples include <Boy, behind, Cabin>, <jacket, is, blue>, and <Girl, Wearing, Pants>.Source: §A.1.2 · p. 14
  • Identical subjects or objects become graph nodes, while triplet relations become directed edges.The resulting concept graph contains entity concepts as nodes and relation concepts as edges.Source: Figure 6 · p. 14, §A.1.2 · p. 15
  • The graph organizes concrete and abstract caption concepts to enable subsequent hallucination and omission analysis.Source: §A.1.2 · p. 15

A.1.3 ANALYSIS OF HALLUCINATION

The supplied passages identify Figure 9 as a VLM graph and describe an example analysis of hallucination and omission, but do not report the analysis details.

A.1.3 ANALYSIS OF HALLUCINATION

A.2 IMPLEMENTATION DETAILS OF PERTURBOLLAVA

Perturbation text is generated in two Q&A rounds, then inserted before the first-round question in each training conversation. Rotating hint prompts tell the model how to treat the text while reducing reliance on a fixed prompt pattern.

A.2 IMPLEMENTATION DETAILS OF PERTURBOLLAVA

  • Perturbation text is generated through two Q&A rounds: the first drafts and refines misleading text, and the second validates and supplements it.The first round analyzes the image, question, and correct answer; it also checks that the text is sufficiently detailed and complex.Source: §A.2 · p. 16
  • The generated perturbation is inserted before the first-round question in each multi-turn SFT conversation, using the format <image><hint prompt1><perturbation text><hint prompt2>.The training text is based on image elements and world knowledge but contradicts the actual image.Source: §A.2 · pp. 16–17
  • The first hint identifies a hallucination perturbation, the second directs the model to disregard it and answer from the image, and both hints vary across instances.Multiple hint variations are randomly selected to avoid overfitting to a fixed prompt pattern.Source: §A.2 · p. 16

A.3 QUANTIFYING THE ADDITIONAL TRAINING OVERHEAD OF PERTURBOLLAVA

This section quantifies PerturboLLaVA’s training overhead and documents related experimental materials, including protocols, a proposed mechanism, user-study scoring, caption-graph prompts, and prompts for identifying omissions and generating misleading perturbation text.

A.3 QUANTIFYING THE ADDITIONAL TRAINING OVERHEAD OF PERTURBOLLAVA

  • PerturboLLaVA increases memory consumption by 2.6% and training time by 6.5% relative to the original SFT stage.The comparison is summarized in Table 9.Source: Table 9 · p. 17, §A.3 · pp. 17–18

A.5 EXPERIMENTS DETAILS

  • Object HalBench uses 1,000 randomly selected MSCOCO2014 validation images, while Object HalBench and HalFscore evaluations use the same prompt and beam search with Nbeams set to 5.The LLaVA1.5 reproduction uses official open-source data and the XTuner framework; Table 10 lists training settings.Source: Table 10 · p. 18, §A.5 · pp. 19–20

A.6 UNDERSTANDING THE MECHANISM OF PERTURBOLLAVA FROM ANOTHER PERSPECTIVE

  • The mechanism analysis treats µ(x) as the multimodal model’s language prior and proposes that perturbed text x+ increases the training gradient; optimal x+ maximizes the gap of µ(xt|x<t,x+).The analysis defines the multimodal model as initialized from the pure language model.Source: §A.6 · p. 22

A.7 DETAILS OF USER STUDY

  • The user study compares LLaVA 1.5, MiniCPM-2.6, Internvl2, and PerturboLLaVA 12 times per pair, then uses Bradley–Terry maximum likelihood estimates to score ability.The model assigns the i-th method probability θi/(θi+θj) of outperforming method j.Source: §A.7 · pp. 22–23

A.8 THE PROMPT FED TO GPT-4O

  • The prompt asks the model to identify objects and attributes, then clearly related object pairs, assigning each relationship an integer strength from 1 to 10 and returning results in English as a numbered list.It specifies object and relationship record formats and uses a record delimiter as the list delimiter.Source: §A.8 · p. 24

Prompt Used to Construct Graph

  • The graph-construction prompt directs the model to identify object nodes with attributes and relationship edges with descriptions and strengths scored from 1 to 10.It specifies node identifiers and a graph-structured output format.Source: Prompt Used to Construct Graph · p. 27

Prompt used to Analyze Halluciation

  • The hallucination-analysis prompt compares VLM entries against ground-truth objects, attributes, and relationships, treating unmatched or significantly different entries as hallucinations.Repeated incorrect objects count once, and relationships involving an already-counted incorrect object do not count separately.Source: Prompt used to Analyze Halluciation · p. 28

Prompt used to Analyze Omission

  • The omission-analysis prompt compares ground-truth entries against VLM output, counting absent objects, attributes, or relationships as missing unless they involve an already-counted missing object.Semantic equivalents are treated as matches, and repeated missing objects count once.Source: Prompt used to Analyze Omission · p. 31
  • The omission-analysis prompt compares ground-truth and VLM lists and excludes relationships separately when their linked object is already missing.It asks for the missing serial numbers under these counting rules.Source: Prompt used to Analyze Omission · p. 33

Prompt used to Generate Perturbation Text(Round 1)

  • Round 1 prompts generate lengthy, confident, plausible perturbations that subtly mislead through language-based reasoning rather than direct correction or visual validation.The prompt illustrates this with examples that reinterpret an image toward an incorrect conclusion.Source: Prompt used to Generate Perturbation… · pp. 34–35

Prompt used to Generate Perturbation Text(Round 2)

  • Round 2 prompts review perturbations for subtle contradiction, no disclosure of the correct answer, grounding in observable image content, and layered misleading reasoning.The perturbation should remain plausible while leading toward a wrong conclusion.Source: Prompt used to Generate Perturbation… · p. 36
Loading 2503.06486v1…