Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment

1. Introduction

• Ungrounded MLLM descriptions pose substantial risks in applications requiring high robustness and faithfulness, including healthcare and autonomous driving.MLLMs can generate descriptions not grounded in visual input [5] [31] [37].Source: §1 · p. 1
• Static-image hallucination methods cannot address object and temporal action hallucinations that commonly occur in videos.Some prior approaches also add computational overhead that slows inference.Source: §1 · pp. 1–2
• Frame-by-frame video processing makes it difficult for MLLMs to capture accurate object appearance and temporal action dynamics.Vista-LLaMA equalizes visual and textual token influence but processes video frame by frame.Source: §1 · p. 2
• SANTA combines hallucinative self-augmentation with tracklet-phrase contrastive alignment to jointly target object and action hallucinations in video captions.The augmentation identifies phrase-level spurious correlations and turns original captions into contrasted negatives; alignment matches objects and relation-aware actions with visual and temporal phrases.Source: §1 · p. 2

2.1. Mitigating Object Hallucination in MLLMs

• Image-focused decoding and alignment methods address object hallucinations, but their computational costs or static-image scope limit their reach to video action hallucinations.VCD and DeCo add decoding-time burden; HACL and HALVA align image and text representations.Source: §2.1 · p. 2

2.2. Mitigating Action Hallucination in MLLMs

• Action hallucination requires tracking evolving object states and cross-frame interactions, unlike static image relationship evaluation.Tri-HE and R-Bench assess static image relationships, not the multi-frame reasoning needed to distinguish motions such as going up versus down stairs.Source: §2.2 · p. 2
• Frame-by-frame processing cannot exploit the temporal dynamics needed for action faithfulness, so SANTA jointly aligns object and action tracklets with textual tokens.Vista-LLaMA applies visual-token influence across generated language tokens; Vriptor [37] incorporates voice-over and audio transcriptions.Source: §2.2 · pp. 2–3

3.1. Problem Formulation

• The framework defines object hallucination as an object noun absent from the video and action hallucination as an action verb absent from it.Source: §3.1 · p. 3

3.2. Mitigating Video-Level Hallucination

• Hallucinative self-augmentation generates captions by selecting high-probability tokens outside ground-truth object and action tokens, including their synonyms and hypernyms.These captions are used as contrasted negatives to reflect the MLLM’s potential hallucinative correlations.Source: §3.2 · pp. 3–4
• Video-caption contrastive learning attracts each input video’s visual features to its paired caption tokens and pushes away hallucinative and randomly sampled video-caption negatives.Source: §3.2 · p. 4

3.3. Mitigating Object and Action-Level Hallucinations with Hallucinative Negatives

• Tracklet-phrase contrastive learning aligns spatial-temporal visual features with language phrases to support object- and action-faithful captions.The framework targets regional objects and relation-guided actions in dynamic video.Source: §3.3 · pp. 4–5
• Regional object alignment associates video object-tracklet features with object phrases, using hallucinated object phrases as hard negatives.The tracklets are derived with visual grounding methods, while object phrases are parsed from the caption.Source: §3.3 · p. 4
• A perceiver-based action squeezer uses learnable queries and related object tracklets to derive action features for relation-guided action alignment.Cross-attention lets the queries capture action information in the involved object tracklets.Source: §3.3 · p. 4

3.4. Summary of SANTA

• SANTA jointly optimizes object-action tracklet-phrase alignment, video-caption alignment, and next-token prediction, with α and β weighting the alignment groups.The next-token cross-entropy loss is included to retain fluent generation.Source: §3.4 · p. 5

4.1. Experimental Setup

• Setup uses 42,715 MiraData training videos and evaluates MiraData-9k and FactVC captioning, VidHal QA, and Dream1k descriptions; weighted-HalFscore adds synonym-aware similarity and tf-idf weighting.HalFscore uses exact string matching and can miss synonyms; FactVCF1 cannot be applied directly to MiraData-9k because CLIP is limited to 78 tokens.Source: §4.1 · p. 5

4.2. Quantitative Evaluation

• On MiraData-9k, SANTA leads F1obj/F1act by +4.02%/+5.54% over the second-best method with Falscore and +3.77%/+7.7% with weighted-Falscore; it also leads FactVC precision, recall, and F1.The comparisons cover three captioning subtasks on MiraData-9k and video captioning on FactVC.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6
• SANTA achieves the highest accuracy on both object- and action-level hallucination QA tasks, surpassing all baselines.This extends the reported caption-hallucination results to hallucination-focused video question answering.Source: Table 3 · p. 6, §4.2 · p. 6

4.3. tSNE Visualization

• Across 250 MiraData-9k pairs, SANTA narrows the vision-language gap and makes hallucinated language features more distinguishable from ground-truth features.The visualization covers video-caption, object tracklet-phrase, and action tracklet-phrase features.Source: §4.3 · pp. 6–7, Figure 3 · p. 7

4.4. Qualitative Evaluation

• In examples from MiraData-9k and FactVC, SANTA matches the ground truth more faithfully while HACL hallucinates an action, and in one case an object and action.The examples contrast “occasionally glancing around” with the video and “broom”/“continues to sweep” with the depicted mopping.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Analysis

• SANTA boosts object- and action-level scores on LLaVA-Video across all three captioning subtasks and yields further gains on Qwen2.5-VL; Dream1k overall F1 rises from 32.5 to 32.7.The backbone results support gains across different MLLM architectures.Source: §4.5 · pp. 7–8, Table 4 · p. 8
• Region-level object alignment improves F1obj/F1act over Lg + Lvideo, action alignment adds further gains, and the complete SANTA configuration achieves the best results.The reported HalFscore gains are +5.5%/+13.3% for region-level object alignment, +0.8%/+3.5% for added action alignment, and +2.4%/+0.3% for complete SANTA; weighted-HalFscore gains are +4.3%/+6.1%, +1.1%/+3.9%, and +1.7%/+1.2%, respectively.Source: §4.5 · pp. 7–8, Table 5 · p. 8
• Self-generated hallucinative negatives outperform both SFT and the GPT4o-generated hard-negative variant across the reported MiraData-9k captioning evaluations.The ablation evaluates F1obj and F1act for overall, main-object, and background captioning.Source: §4.5 · p. 8, Figure 5 · p. 8

• SANTA combines hallucinative self-augmentation with tracklet-phrase contrastive alignment to encourage visual grounding and improve object and action faithfulness in captions.The alignment links regional objects and relation-guided actions to their corresponding visual and temporal phrases.Source: §5 · p. 8

7.1. Training Data

• MiraData captions provide three types of video descriptions; GPT-4o parses ground-truth captions into action-and-object relation triples, then WordNet adds synonyms and hypernyms to the token sets.Each triple contains an action verb and two related object instances; the enriched tokens are used to represent visual facts in the video.Source: §7.1 · p. 11
• Grounded-SAM2 generates object-instance segmentation masks from ground-truth caption information; the resulting crops identify object-specific visual tokens for object and relation-guided action alignment, conducted offline.Source: §7.1 · p. 11

7.2. Additional Evaluation Setups

• MiraData-9k evaluation parses ground-truth captions into object and action token sets, then applies HalScore and weighted-HalScore metrics to assess hallucination mitigation methods.Source: §7.2 · p. 11

7.3. Comparisions

• Comparisons include SFT, DeCo, HALVA, and HACL, using public code and aligned training data and training/inference hyperparameters matched to the base models.Source: §7.3 · p. 11

More Detailed Results on Video Description

• SANTA achieves the highest overall F1-score on Dream1k, outperforming HALVA and HACL and surpassing the original LLaVA-Video baseline.The benchmark includes videos from five diverse sources.Source: §8 · pp. 11–12

Results on General Video Question Answering

• +2.7 with subtitles and +0.9 without subtitles are SANTA’s VideoMME accuracy improvements over HACL on the same LLaVA-Video backbone.Both methods remain slightly below the original backbone; the authors hypothesize caption-only post-training contributes and suggest incorporating QA data.Source: Table 7 · p. 12, §8 · p. 12

Analysis of Robustness with the Impact of Object Tracklet Quality

• +2.1 Avg. F1Obj and +0.6 Avg. F1Act under HalFscore, and +2.9 Avg. F1Obj and +1.1 Avg. F1Act under weighted-HalFscore, are SANTA’s gains over HACL at t=0.15.The lower detection-confidence threshold induces false positives, testing robustness to noisy object tracklets.Source: Table 8 · p. 12, §8 · p. 12

Results on Hallucinative Negative and Object/Action Tracklets

• In the surfing example, sea waves make “heap of sand...” a hallucinative negative, while action tracklets link objects such as the surfer and pink surfboard.GPT-4o determines the object relationships from the ground-truth caption; Figure 8 shows its prompting template.Source: §9 · pp. 12–13, Figure 8 · p. 15

Additional Results on MiraData-9k

• SANTA’s MiraData-9k captions exhibit object and action faithfulness in the qualitative comparison with HACL.Figure 7 marks action faithfulness in green, action hallucination in red, object faithfulness in blue, and object hallucination in orange.Source: §9 · p. 13, Figure 7 · p. 14