MiD-VLM: Detecting and Grounding Global Semantic Misinformation in Image-Caption Pairs

1. Introduction

• Because an image may remain unaltered while its caption changes the narrative, pixel-artifact methods may miss semantic inconsistencies.Detecting these cases requires reasoning across image and caption meaning.Source: §1 · p. 1
• DGM4+ extends DGM4 with 5,000 pairs targeting global scene and semantic mismatches, including text swaps and attribute changes.The task predicts pair authenticity, manipulation type, and misleading caption tokens.Source: §1 · p. 1
• MiD-VLM combines SigLIP image-text representations and DINOv2 scene features through a projection MLP, alongside LoRA-adapted Qwen2-VL-7B.Separate heads predict authenticity, manipulation type, and misleading caption-token grounding.Source: §1 · pp. 1–2
• 0.9679 is MiD-VLM’s binary authenticity-classification F1 score on the DGM4+ validation split.The introduction also identifies token-level grounding as a continuing challenge.Source: §1 · p. 2

2.1. Multimodal Misinformation Detection

• Multimodal misinformation detection considers whether image and text convey mutually consistent information, beyond the authenticity of each modality alone.NewsCLIPpings studies authentic but out-of-context image-caption pairs.Source: §2.1 · p. 2, §2 · p. 2
• DGM4 and HAMMER address joint detection and grounding, while DGM4+ shifts focus to global scene and semantic inconsistencies [8] [10].The related-work discussion reports HAMMER accuracy below 20% on the extended dataset.Source: §2.1 · p. 2

2.2. Vision-Language Models

• Vision-language models provide pretrained backbones for cross-modal reasoning, but full fine-tuning can be computationally expensive for relatively small datasets.Qwen2-VL is cited as a foundation for adapting pretrained models to specialized cross-modal tasks.Source: §2.2 · p. 2

2.3. Parameter Efficient Fine Tuning

• LoRA adapts selected attention projections of Qwen2-VL through low-rank updates while keeping most backbone weights frozen.This parameter-efficient strategy avoids updating the full set of pretrained parameters.Source: §2.3 · p. 2

3.1. Problem Formulation and Dataset

• The model outputs pair authenticity, manipulation type, and indices of misleading caption tokens.The type labels are origin, text swap, and text attribute.Source: §3.1 · p. 3, §3 · p. 3
• DGM4+ contains approximately 5K pairs split into 3,500 training, 750 validation, and 750 test examples, with manipulation-class and affected-token annotations.The validation set contains 254 authentic and 446 manipulated examples; test labels are withheld.Source: §3.1 · p. 3

3.2. Architecture Overview

• The three-stage architecture extracts features with frozen encoders, fuses and projects them, then injects the representation as a learned prefix into LoRA-adapted Qwen2-VL-7B.Three shallow heads produce binary, type, and grounding outputs.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. Dual Encoder Feature Extraction

• The dual encoder supplies image and text embeddings, which are concatenated with DINOv2’s CLS token and projected by a two-layer MLP.The MLP uses GELU activation and layer normalization.Source: §3.3 · p. 3

3.4. Multi-Task Prediction Heads

• Three task-specific heads predict authenticity, manipulation type, and token grounding from a shared representation.The grounding head scores 64 token positions, and the three losses are weighted 0.5 for binary classification, 0.3 for grounding, and 0.2 for manipulation type.Source: §3.4 · p. 3

4. Experiments and Results

• Training froze SigLIP and DINOv2 while applying LoRA to Qwen2-VL-7B; early stopping halted training after five epochs.Training used four NVIDIA A100 GPUs.Source: §4 · p. 4

4.1. Evaluation Metrics

• The PP-MisDet Challenge metric combines binary detection, textual grounding, and manipulation-type classification, weighted 0.5, 0.3, and 0.2, respectively.Source: §4.1 · p. 4

4.2. Validation Performance

• At epoch 4, validation reached 96.13% accuracy, an F1 score of 0.9679, and loss of 0.0818; performance declined slightly at epoch 5.The epoch 4 checkpoint was selected.Source: Table 2 · p. 4, §4.2 · p. 4

4.3. Encoder Ablation

• Adding DINOv2 raised textual grounding from 0.8050 to 0.8375 and produced the highest overall score, 0.9282, while SigLIP-only had the highest binary F1.Source: Table 3 · p. 4, §4.3 · p. 4

4.4. Text Grounding Analysis

• At τ = 0.3, the dual-encoder produced no grounded tokens for 207 of 446 manipulated validation samples, despite achieving the strongest grounding F1.The authors suggest that global fused representations may be insufficient for finer-grained token localization.Source: §4.4 · p. 4

• MiD-VLM combines frozen SigLIP and DINOv2 representations with a LoRA-adapted Qwen2-VL backbone for joint detection, manipulation-type classification, and textual grounding.Source: §5 · p. 4
• The model achieved a best validation F1 of 0.9679 while training only a small fraction of the backbone parameters.Encoder ablations showed that DINOv2 complements SigLIP by improving textual grounding and the overall challenge score.Source: §5 · p. 4
• Token-level grounding difficulties indicate that strong global detection does not necessarily translate to precise localization, motivating finer-grained cross-modal representations.Source: §5 · p. 4