PaperlayerView exact revision on arXiv

Source-linked AI summary

SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection

Peng Qi, Zehong Yan, Wynne Hsu, Mong Li Lee

arXiv:2403.03170v1cs.MMcs.AIcs.CLcs.CVcs.CY

TL;DR

Out-of-context misinformation reuses authentic images in misleading contexts, while existing methods lack convincing explanations. SNIFFER adapts InstructBLIP through two-stage tuning and adds tools and retrieval for internal and external checking. It exceeds the original MLLM by over 40%, outperforms state-of-the-art methods, and provides persuasive explanations.

  • Problem

    OOC misinformation uses authentic images in false contexts, and existing methods assess image-text consistency but lack convincing explanations for their judgments.

    Source: §1 · p. 2

  • Method

    SNIFFER applies two-stage instruction tuning to InstructBLIP and augments internal image-text checking with external verification using tools and retrieved evidence.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    SNIFFER exceeds the original MLLM by over 40%, surpasses state-of-the-art detection methods, and provides precise explanations validated quantitatively and by humans.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Quantitative and human assessments support the precision and persuasiveness of SNIFFER’s explanations.

    Source: §1 · p. 2

    During annotation, users tended to label news items as fake more often than they usually would when reading news on social media.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Misinformation is a prevalent societal issue due to its potential high risks. Out-of-context (OOC) misinformation, where authentic images are repurposed with false text, is one of the easiest and most effective ways to mislead audiences. Current methods focus on assessing image-text consistency but lack convincing explanations for their judgments, which is essential for debunking misinformation. While Multimodal Large Language Models (MLLMs) have rich knowledge and innate capability for visual reasoning and explanation generation, they still lack sophistication in understanding and discovering the subtle crossmodal differences. In this paper, we introduce SNIFFER, a novel multimodal large language model specifically engineered for OOC misinformation detection and explanation. SNIFFER employs two-stage instruction tuning on InstructBLIP. The first stage refines the model's concept alignment of generic objects with news-domain entities and the second stage leverages language-only GPT-4 generated OOC-specific instruction data to fine-tune the model's discriminatory powers. Enhanced by external tools and retrieval, SNIFFER not only detects inconsistencies between text and image but also utilizes external knowledge for contextual verification. Our experiments show that SNIFFER surpasses the original MLLM by over 40% and outperforms state-of-the-art methods in detection accuracy. SNIFFER also provides accurate and persuasive explanations as validated by quantitative and human evaluations.

1. Introduction

Out-of-context misinformation reuses authentic images in misleading contexts, while existing detection methods lack convincing explanations. SNIFFER adapts an MLLM for detection and explanation, combining internal image-text checks with retrieval-based external verification.

1. Introduction

  • OOC misinformation repurposes authentic images with misleading text, so the deception lies in the context rather than image alteration.The passage describes this as a prevalent way to mislead audiences.Source: §1 · p. 2
  • Existing approaches assess image-text consistency or compare external references, but convincing explanations remain underexplored and are needed for trust and debunking.Attention visualizations may highlight regions without explaining why they indicate misinformation.Source: §1 · p. 2
  • General-purpose MLLMs can reason visually and generate explanations, but may misunderstand instructions, hallucinate, and lack access to current information or external tools.OOC detection can require distinguishing news events that are not identifiable from image content alone.Source: §1 · p. 2
  • SNIFFER uses two-stage instruction tuning on InstructBLIP, first aligning generic objects with news entities and then tuning for OOC discrimination using GPT-4-generated instructions.The instruction data includes judgments and explanations.Source: §1 · p. 2
  • SNIFFER exceeds the original MLLM by over 40%, surpasses state-of-the-art detection methods, achieves comparable results with 10% of the training data, and provides explanations assessed quantitatively and by humans.The paper reports precise and persuasive explanations.Source: §1 · p. 2

2. Related Work

Prior misinformation detectors use varied signals, while OOC methods check image-text consistency or compare claims with external references. Explainability and instruction-data construction remain central concerns for this work.

2. Related Work

  • Misinformation systems combine cues such as linguistic patterns, image tampering, multimodal inconsistency, user response, and propagation structure, [38] [42], [39] [54],,.OOC reuses authentic images in similar but incorrect contexts, limiting traditional detection methods.Source: §2 · pp. 2–3
  • OOC detectors use pretrained multimodal models for image-text consistency or compare claims with retrieved references and web evidence [29] [36], [16] [17] [41],,.Another approach reframes detection around semantically different captions describing the same image.Source: §2 · p. 3
  • Existing methods often do not explain their judgments, and an interpretable MLLM-based detector remains constrained by general-domain MLLM capability.SNIFFER instead uses two-stage instruction tuning to generate judgments and explanations together.Source: §2 · p. 3
  • MLLM instruction tuning uses instruction-formatted data to improve generalization, with self-instruction among approaches for constructing training examples [9], [18] [27], [48].This work extends general-domain MLLMs to OOC misinformation detection through instruction tuning.Source: §2 · p. 3

3. Method

SNIFFER adapts InstructBLIP to news-domain concepts and OOC detection, then combines internal image-text checks with retrieval-based verification. An LLM integrates both checks into a final judgment and explanation.

3. Method

  • SNIFFER feeds an image-text pair to internal cross-modal checking and compares the caption with image-retrieved textual evidence for external claim-evidence checking; an LLM combines both outcomes into a judgment and explanation.The final LLM reasoning draws on both checks and the initial caption.Source: §3 · p. 3

3.1. Base MLLM

  • InstructBLIP uses a Q-Former to produce instruction-aware visual features from a frozen image encoder and pass them as soft prompts to a frozen LLM.The Q-Former interacts with instruction text and image embeddings through self-attention and cross-attention, respectively.Source: §3.1 · p. 3

3.2. Instruction Tuning

  • SNIFFER is trained sequentially, first adapting general-domain InstructBLIP to news and then to OOC detection.This is the two-stage procedure described for adapting the model to the domain and task.Source: §3.2 · p. 3, Figure 3 · p. 4
  • Stage 1 addresses coarse object labels and subtle entity swaps by aligning news-domain image features with fine-grained entity names using NewsCLIPpings image-caption pairs.ChatGPT-4 generates 11 brief-description questions; one is sampled per image-caption pair, and training updates only the Q-Former.Source: §3.2 · pp. 3–4
  • Because supervised OOC data lacks paired judgments and explanations, GPT-4 generates instructions from captions and an image description, identifying one likely inconsistency.The process yields 35,536 OOC instructions and an equal number of real-sample instructions.Source: §3.2 · pp. 4–5

3.3. Reasoning Process

  • For internal checking, SNIFFER uses visual entities recognized by Google Entity Detection API as supplementary information, then checks image-text inconsistency.The paper notes that the model's training corpus limits access to current information.Source: §3.3 · p. 5
  • For external checking, the LLM assesses whether a caption is supported by text retrieved through image search, then integrates both checks into an interpretable final decision.The final reasoning also uses the initial caption and explains the decision.Source: §3.3 · p. 5

4. Performance Study

The performance study evaluates SNIFFER on NewsCLIPpings and examines detection performance, component contributions, explanation quality, data efficiency, cross-dataset generalization, and comparison with GPT-4V.

4.1. Experimental Setup

  • Experiments report results on the Merged/Balance subset, balanced across retrieval strategies and positive/negative samples, with 71,072 training, 7,024 validation, and 7,264 test samples; accuracy is measured over all samples (All).Source: §4.1 · p. 5

4.2. Performance Comparison (Q1)

  • SNIFFER outperforms all baselines and exceeds CCN by over 3.7%, despite using only part of the textual evidence.Baselines trained from scratch perform worse than pretrained multimodal baselines.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Ablation Studies (Q2)

  • OOC tuning improves InstructBLIP by over 35 percentage points, while visual entities add 5 points in accuracy.Retrieved evidence can yield high fake-sample recall, but noise lowers real-sample recall; fewer than 60% of samples have associated evidence.Source: Table 2 · p. 6, §4.3 · pp. 6–7

4.4. Explainability Analysis (Q3)

  • After OOC tuning, response ratios for entity outputs fall while accuracy across all three explanation points rises.Visual entities slightly raise response ratios; explanation accuracy improves across components, with ent t accuracy exceeding ent v by 17%.Source: Figure 5 · p. 7, §4.4 · p. 7, Figure 6 · p. 7
  • After reading SNIFFER’s explanations, users changed 87% of initially real judgments to fake and gained confidence on 42% of initially fake judgments.In data annotation, users tended to label news fake more frequently than in ordinary social-media reading.Source: §4.4 · pp. 7–8, Figure 7 · p. 8

4.5. Practical Setting

  • With 10% of training data, SNIFFER- matches baselines without external evidence; with 25%, it surpasses them.SNIFFER- uses only OOC tuning on InstructBLIP.Source: Figure 8 · p. 8, §4.5 · p. 8
  • Trained on NewsCLIPpings, SNIFFER outperforms CMCS by a large margin on News400 and TamperedNews across difficulty levels.Smaller k values make the image replacements more similar and detection more challenging.Source: Figure 9 · p. 8, §4.5 · p. 8

4.6. Comparison with GPT-4V (Q6)

  • On 400 randomly sampled test examples, SNIFFER exceeds GPT-4V by 11% in classification accuracy.The sample comprised 200 fake and 200 real examples.Source: Table 3 · p. 8, §4.6 · p. 8

5. Conclusion

SNIFFER is an InstructBLIP-based model for OOC misinformation that provides judgments and explanations. The paper reports state-of-the-art detection and precise, persuasive explanations.

5. Conclusion

  • SNIFFER achieves state-of-the-art detection performance and generates precise, persuasive explanations.The conclusion describes the model as providing both judgment and explanation.Source: §5 · p. 8
  • The authors construct GPT-4-assisted multi-perspective instruction data and tune InstructBLIP for news-domain alignment and OOC detection.They also integrate external tools and retrieval.Source: §5 · p. 8
  • SNIFFER is designed to output both a detection judgment and an explanation for out-of-context misinformation.Source: §5 · p. 8

A. Behaviors of open-sourced general-purpose MLLMs

Open-source general-purpose MLLMs show instruction-following and hallucination problems on OOC misinformation detection. InstructBLIP also tends to describe people with broad nouns rather than specific names.

A. Behaviors of open-sourced general-purpose MLLMs

  • InstructBLIP often paraphrases the caption instead of giving a clear judgment, indicating misunderstanding of the requested task.Source: A. Behaviors of open-sourced… · p. 9
  • LLaVA-Adapter-V2 and Woodpecker hallucinate, respectively misattributing caption content to the image and assigning the image an incorrect context.The examples include LLaVA-Adapter-V2 assigning an erroneous Apollo 11 meaning and Woodpecker describing a rocket launch in the 1980s.Source: A. Behaviors of open-sourced… · pp. 9–10
  • Only 27% of InstructBLIP responses containing person nouns use fine-grained proper names; the other 73% use common nouns.Source: A. Behaviors of open-sourced… · p. 10

B. Instruction Data Construction

The instruction-data construction uses generated questions for news-image description and GPT-4 prompts to formulate OOC-specific instructions from caption inconsistencies.

B. Instruction Data Construction

C. Case Studies

Three representative cases compare SNIFFER and GPT-4V. Their contrasting judgments illustrate SNIFFER's use of image-text and claim-evidence clues, while also showing cases where GPT-4V succeeds or errs.

C. Case Studies

  • In the first case, SNIFFER identifies a fake claim using image-text inconsistency and claim-evidence irrelevance, while GPT-4V misses the inconsistencies.The case is presented among three representative comparisons.Source: C. Case Studies · pp. 10–11, Table 6 · p. 11
  • In the second case, SNIFFER incorrectly labels the claim real when no relevant webpages are retrieved, but GPT-4V correctly identifies it as fake.GPT-4V reasons from the Pope's attire and event security conditions.Source: C. Case Studies · p. 12
  • In the third case, SNIFFER correctly classifies a real story about stranded passengers, whereas GPT-4V wrongly calls it fake based on the orderly crowd and absent signage.The passage notes that more than half of social-media news stories do not have their content words represented in images.Source: C. Case Studies · p. 12
  • Across the cases, SNIFFER integrates image-text inconsistency and claim-evidence relevance, while GPT-4V's world knowledge can help detect anomalies but also accompany false-news judgments on real stories.Source: C. Case Studies · p. 12
Loading 2403.03170v1…