PaperlayerView exact revision on arXiv

Source-linked AI summary

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

Chenhao Dang, Dantong Zhu, Jun Yang, Conghui He, Weijia Li

arXiv:2606.24112v2cs.AI

TL;DR

Existing multimodal misinformation benchmarks simplify real-world verification, despite posts combining multilingual narratives, multiple images, graded veracity, and cross-modal distortions. ReMMD addresses this gap with a realistic benchmark and persistent-memory verifier, achieving the best benchmark result while reducing verification cost.

  • Problem

    Existing evaluations simplify multimodal verification to isolated claims, single images, coarse verdicts, or one manipulation source instead of realistic multilingual, multi-image posts.

  • Method

    ReMMD combines an agent-oriented benchmark with a verifier that decomposes posts, retrieves and reuses evidence, and predicts veracity, distortion labels, and rationales.

  • Results

    ReMMD-Agent with GPT-5.2 sets the best ReMMDBench result while reducing cost by 17.5% versus MMD-Agent and 79.9% versus T2-Agent.

  • Takeaways & Limitations

    Realistic multimodal misinformation detection is best framed as evidence selection, grounding, and explanation rather than visual authenticity alone.

  • Takeaways & Limitations

    Future systems still need source-aware memory, temporal retrieval, multilingual entity linking, and metrics separating visual edits, verdicts, and misleading mechanisms.

Abstract

from arXiv · show

Multimodal misinformation detection is increasingly important because viral posts now combine long multilingual narratives, several images, mixed provenance, and subtle cross-modal framing errors. Existing benchmarks and methods remain poorly matched to this setting: they usually isolate short captions, single images, binary labels, or one manipulation source, while agentic verification remains costly under realistic evidence search. We present ReMMD, a realistic multilingual multi-image agentic verification framework for multimodal misinformation detection. ReMMD includes ReMMDBench, a real-world multimodal misinformation detection benchmark with 500 samples, 2,756 images, five monolingual evaluations, two cross-lingual settings, three text-length tiers, multi-image posts, five-way veracity labels, eight distortion labels, evidence provenance, and rationales. It also includes ReMMD-Agent, a persistent-memory verifier that decomposes posts into atomic points, builds a reusable evidence set, and predicts structured veracity verdicts, fine-grained distortion diagnoses, and explanatory rationales. Across proprietary systems, open LVLMs, MMD-Agent, and T$^2$-Agent, ReMMD-Agent obtains the best five-way veracity performance, with 41.80% accuracy and 39.12% macro-F1 using GPT-5.2, while reducing cost by 17.5% relative to MMD-Agent and 79.9% relative to T$^2$-Agent. The project is available at https://dang-ai.github.io/ReMMD.

Introduction

ReMMD addresses the gap between simplified multimodal misinformation benchmarks and operational verification by combining ReMMDBench, a realistic multilingual multi-image benchmark, with ReMMD-Agent, a persistent-memory verifier. The framework supports structured veracity and distortion judgments with rationales while improving verification cost relative to prior agents.

  • ReMMDBench: ReMMDBench evaluates multimodal misinformation under realistic conditions using multilingual, multi-image, varied-length samples and agent-oriented verification settings.It includes single-text, multi-image samples, three length tiers, five monolingual evaluations, and two cross-lingual transfer settings.
  • ReMMDBench: ReMMDBench provides five-way veracity labels, eight distortion labels, evidence provenance, and rationales instead of reducing fact-checking to binary truth judgments.The benchmark is designed to capture graded verdicts and fine-grained attribution across textual, visual, and cross-modal distortions.
  • ReMMD-Agent: ReMMD-Agent decomposes posts into atomic claims and image bindings, retrieves multimodal evidence, and incrementally maintains a reusable persistent memory bank before judgment.A structured judge predicts L1 veracity, L2 distortion labels, and an L3 rationale from the accumulated evidence state.
  • ReMMD: ReMMD combines ReMMDBench and ReMMD-Agent into a realistic multilingual multi-image agentic verification framework for multimodal misinformation detection.The evaluation spans closed-source and open-source agents across three backbone families and five total model sizes.
  • Evaluation: 17.5% lower GPT-5.2 cost than MMD-Agent and 79.9% lower than T2-Agent accompany ReMMD-Agent’s best ReMMDBench result.The framework further analyzes performance by language, text length, distortion, transfer, and cost.

Related Work

Prior multimodal misinformation benchmarks evaluate visual manipulation and cross-modal discrepancies, while recent work identifies vulnerabilities in large vision-language model perception. Agentic verification addresses these weaknesses by decomposing claims, asking targeted questions, and using retrieval or visual tools.

  • Datasets and benchmarks: Multimodal misinformation benchmarks evaluate visual manipulations and cross-modal discrepancies, including image repurposing, unimodal bias, spatial localization, and semantic attribution.
  • Datasets and benchmarks: ReMMDBench constructs controlled multilingual multi-image samples by planning language, text length, visual provenance, and label conditions before evidence-based validation.Validation checks evidence, distortion annotations, text–image consistency, and image provenance before inclusion.
  • LVLMs and agentic verification: Large vision-language models remain vulnerable to grounding errors, stale or adversarial evidence, and temporal contamination despite their suitability as multimodal verifiers.
  • LVLMs and agentic verification: Agentic verification improves robustness by decomposing claims, asking targeted questions, and invoking retrieval or visual tools.

ReMMDBench

ReMMDBench uses an automated multi-agent pipeline to transform authentic multilingual posts into controlled, image-dense misinformation samples with engineered structural misalignments. Its hierarchical annotations and validator-based quality control support rigorous five-way veracity, distortion, rationale, and reliability evaluation.

  • Data construction: ReMMDBench transforms authentic posts from diverse trending topics and debunking sources into controlled samples with prescribed veracity, distortion, language, and visual configurations.A Planner coordinates specialized Text, Search, and Image Workers that retrieve evidence and manipulate visual assets.
  • Benchmark characteristics: Image counts average 2.35 in short tiers and 10.05 in long tiers, while two cross-lingual settings migrate source material and visuals into different target languages.The pipeline deliberately engineers semantic misalignments and contextual displacement during multilingual migration.
  • Annotation scheme: Each sample has hierarchical L1 veracity, L2 distortion, and L3 rationale annotations, distinguishing five ordered veracity conditions and textual, visual, and cross-modal errors.The taxonomy separates fabrication, factual-basis distortion, misleading context, synthetic or edited visuals, and semantic, contextual, or pragmatic inconsistency.
  • Quality control: Cohen’s κ reached 0.91 against two independent professional annotators across all 500 benchmark samples, validating the automated quality-control pipeline.A Validator agent checks whether L1 verdicts follow the central claim and whether L2 tags are physically grounded.

ReMMD-Agent

ReMMD-Agent decomposes multimodal posts into atomic claims and observations, retrieves reusable evidence through persistent memory, and produces structured veracity, distortion, and rationale outputs. Its cross-lingual and multimodal retrieval procedures target regional evidence while auditing translation, visual, provenance, and cross-modal inconsistencies.

  • Agent overview: ReMMD-Agent outputs a five-way veracity label, an eight-dimensional distortion vector, and a concise rationale after reasoning over explicit evidence states.The system judges decomposed atomic units rather than the multimodal bundle directly.
  • Parsing: The parser converts text and images into atomic points containing minimal claims or visual observations, retrieval queries, visual cues, and point types.Predefined fields cover entities, temporal anchors, locations, events, and core claims.
  • Evidence retrieval: The retrieval stage maintains a sample-level memory bank asynchronously enriched by scheduled sweeps across news aggregates and fact-checking domains.During inference, targeted web and social calls retrieve evidence using parsed queries and visual cues.
  • Evidence retrieval: TopK is set to 20, enabling an auditable persistent evidence state that supports cross-claim evidence reuse.Evidence relevance is evaluated with cosine similarity over Qwen3-Embedding-8B representations.
  • Structured judgment: The structured judge assigns each atomic point a supported, contradicted, or unverified state before inferring the overall veracity from central claims and cross-modal bindings.Cross-lingual verification expands retrieval toward regional evidence, adds English or Chinese bridge queries, and audits translation-induced semantic discrepancies.

Experiments

Experiments show that ReMMD-Agent improves structured five-way, multi-image verification through claim-centered evidence organization, atomic parsing, memory reuse, and visual auxiliary analysis. These gains remain stable across challenging text and language conditions while reducing verification cost relative to prior agents.

  • Evaluation setup: Evaluation covers the full 500-sample ReMMDBench split across Manus, ChatGPT, MMD-Agent, T2-Agent, and ReMMD-Agent.Model-backed agents use shared Serper retrieval and backbones including GPT-5.2, Gemma4-31B, Qwen3.6-27B, Qwen3.5-9B, and Qwen3.5-4B.
  • Main results: ReMMDBench remains difficult for all systems, while ReMMD-Agent improves calibration of graded veracity across backbone families.General-purpose assistants are competitive on some L2 metrics but weaker on L1, and GPT-5.2 gives the best L1 performance.
  • Main results: Additional search alone is insufficient: MMD-Agent struggles with partial-truth labels, while T2-Agent’s expanded reasoning does not consistently improve veracity.MMD-Agent remains useful for distortion-oriented comparison, but evidence must be organized around the right claims.
  • Ablation: Atomic parsing and memory reuse reduce information noise, support provenance aggregation and cross-image evidence reuse, and improve both L1 and L2.Visual auxiliary analysis is especially important for L2 because visual edits and cross-modal mismatches can diagnose distortion before final veracity judgment.
  • Efficiency: 17.5% lower per-sample cost than MMD-Agent and 79.9% lower than T2-Agent demonstrate architectural efficiency rather than increased computational expenditure.Evidence reuse across atomic points avoids repeated tool-augmented reasoning-path expansion, supporting dynamic benchmarks and high-concurrency deployments.
  • Robustness: ReMMD-Agent is more stable than baselines across text-length tiers and language slices, although low-level forensics and discourse-level support remain complementary challenges.Atomic parsing handles noisy long-context entities and references, while memory reuse reduces superficially related but temporally or geographically mismatched retrieval.

Discussion

The discussion frames realistic multimodal misinformation detection as evidence selection: systems must connect sources to claims or images and diagnose how evidence, visuals, and conclusions interact. It also highlights that evidence-supported reasoning, multilingual grounding, and mechanism-specific evaluation are key directions for future systems.

  • Evidence selection: Realistic MMD is an evidence-selection problem requiring fine-grained labels because real evidence can support wrong conclusions.Retrieval helps only when each source is tied to the claim or image it verifies.
  • Evidence selection: Visual authenticity alone is insufficient because real images can be misused, while synthetic images do not automatically falsify accompanying text.Verification must assess the relationship among claims, evidence, and images rather than image authenticity in isolation.
  • Model scale and evidence: Qwen3.5-9B outperforms Qwen3.6-27B on several metrics under the same ReMMD-Agent pipeline.The discussion attributes this pattern to evidence-provided schema following, partial-evidence calibration, and uncertainty handling rather than a general reversal of model scale.
  • Future directions: Future systems should improve source-aware memory, temporal retrieval, multilingual entity linking, and metrics that separately evaluate visual edits, verdicts, and misleading mechanisms.Rationales should identify the claim, evidence, and image-text relation, while multilingual cases require local entity and source grounding beyond translation.

Conclusion … E Agent Adaptation Details

ReMMD introduces a realistic benchmark and evidence-management agent for multilingual, multi-image misinformation verification, with structured veracity and distortion outputs. The benchmark supports reproducible annotation, operational label boundaries, and adapted multi-image evaluation of existing agents.

  • Conclusion: ReMMDBench combines multilingual posts, multiple images, graded veracity, distortion labels, rationales, and evidence-centered analysis for realistic verification.ReMMD-Agent decomposes posts into checkable units, reuses retrieved evidence through memory, and judges veracity and distortion from explicit evidence.
  • A Benchmark Examples: Figure 5 presents non-sensitive benchmark samples featuring multilingual text, multiple images, hierarchical labels, rationales, and evidence-centered analysis.
  • B Additional Benchmark Statistics: The benchmark retained 500 samples after screening and controlled rewriting from an initial collection of 54,468 posts gathered across six listed sources.The collection period ran from March 1, 2026, through May 10, 2026.
  • B Additional Benchmark Statistics: The five L1 classes are close to balanced, while average L2-label counts rise monotonically from True to False, indicating that severe misinformation often accumulates distortions.
  • C Label Boundary Notes: L1 assigns exactly one veracity verdict, whereas L2 records every applicable distortion type.T2 captures altered claim scope or meaning, T3 captures misleading context, and V1–V2 may co-occur for edited real images containing generated content.
  • D Human–AI Annotation Agreement: Human annotators achieved 0.96 exact agreement and 0.97 linear-weighted κ on L1 across the complete benchmark.The agreement analysis uses exact agreement for identical ordered verdicts and weighted κ for chance-corrected severity disagreements.
  • D Human–AI Annotation Agreement: The Annotator Agent achieved 0.90 exact agreement and 0.91 weighted κ against human consensus, while Agent–consensus L2 exact set match was 0.77.Human–human L2 exact set match was 0.83, reflecting the stricter multi-label evaluation.
  • E Agent Adaptation Details: MMD-Agent and T2-Agent received the full post text unchanged without ReMMDBench gold labels or rationales during inference.MMD-Agent deduplicates and caps images at 10, while T2-Agent retains document-order images but uses the first image as its visual anchor for original MCTS and tool semantics.

F Reproducibility Materials and Repeated Runs

The project repository provides the materials and configurations needed to reproduce the experiments, while all reported results are averaged over three runs.

  • The repository includes complete code and exact prompt templates for reproducing the experiments.
  • Decoding configurations, failure-handling rules, and retrieval budgets are provided.
  • Three runs are used to compute every experimental result reported in the paper.

G MMFakeBench Transfer Setting

In the MMFakeBench transfer setting, ReMMD-Agent substantially outperforms MMD-Agent and T2-Agent on binary fake detection despite using the same Qwen3.5-9B backbone and retrieval backend. Its advantage persists beyond ReMMDBench-specific five-way veracity and eight-label distortion definitions.

  • Transfer setup: The transfer experiment evaluates the official 10,000-instance MMFakeBench test split, distributed as 70% fake and 30% true.All compared agents use Qwen3.5-9B and Google Search by Serper API.
  • Transfer results: 0.824 accuracy and 0.871 Macro-F1: ReMMD-Agent outperforms MMD-Agent at 0.592 accuracy and 0.673 Macro-F1 and T2-Agent at 0.639 accuracy and 0.715 Macro-F1.The transfer task uses binary fake detection rather than five-way veracity and eight distortion labels.
  • Analysis: ReMMD-Agent’s advantage suggests its benefit is not limited to ReMMDBench-specific label definitions.T2-Agent performs more search, but its additional reasoning loop does not compensate for weaker evidence routing with a smaller open-source backbone.

H Additional Fine-Grained Results

Fine-grained analyses across text lengths, languages, and distortion labels show that ReMMD-Agent generally leads when verification requires organizing and aligning evidence. Performance advantages vary by setting: GPT-5.2 is strongest for verdict assignment in shorter and medium tiers, Qwen3.5-9B excels on several macro-F1 slices, and MMD-Agent remains best for synthetic-visual cues.

  • Short-text analysis: ReMMD-Agent leads short-text L1 results, especially with GPT-5.2, while local visual cues keep assistant baselines competitive on L2.Exact match remains low across systems.
  • Medium-text analysis: ReMMD-Agent/GPT-5.2 gives the strongest medium-tier verdict performance, whereas ReMMD-Agent/Qwen3.5-9B achieves the best L2 macro-F1.Baseline L1 macro-F1 remains unstable because partial evidence must be assigned to the correct severity class.
  • Long-text analysis: ReMMD-Agent/Qwen3.5-9B achieves the highest long-tier L1 and L2 macro-F1, while ReMMD-Agent/GPT-5.2 records the highest accuracy.The long tier averages about ten images per sample and rewards binding claims, images, and retrieved sources.
  • Language-slice analysis: ReMMD-Agent improves verdict and distortion performance in every language, with the largest L1 gains in Japanese and French where MMD-Agent is weakest.T2Agent occasionally improves French L1 over MMD-Agent but drops sharply on L2.
  • Distortion-label analysis: ReMMD-Agent is strongest on evidence-alignment labels T2 Distortion, V2 Visual Editing, C1 Semantic Inconsistency, and C2 Contextual Inconsistency, but MMD-Agent leads V1 Synthetic Visual Content.The V1 exception indicates that low-level generation artifacts and forensic cues remain complementary to evidence retrieval.

I Additional Benchmark Distributions and Confusion Matrices

ReMMDBench’s distributions deliberately reflect multilingual, multi-image, mixed-provenance misinformation, while confusion matrices show that systems struggle most with calibrated judgments among neighboring middle veracity labels. ReMMD-Agent mitigates conservative prediction drift by separating supported, contradicted, and unverified atomic points.

  • Benchmark distributions: ReMMDBench includes cross-lingual cases, frequent textual, visual, and cross-modal distortions, mixed image provenance, and multiple text-length tiers.These design choices prevent systems from optimizing for a single language, manipulation family, provenance type, or post length.
  • Image-count distribution: The image-count distribution has an intentional long tail toward ten and eleven images, reflecting carousel-style posts with central, decorative, repeated, or weakly related images.Agents must identify which images support the claim rather than treating the entire image set as undifferentiated visual evidence.
  • Confusion matrices: Direct prompting and T2-Agent avoid confident True predictions and concentrate probability mass around middle labels, indicating conservative bias under misinformation uncertainty.ReMMD-Agent reduces this drift by forcing judgments over supported, contradicted, and unverified atomic points.
  • Confusion matrices: Errors concentrate among neighboring middle labels, where systems must assess the centrality of contradicted evidence rather than merely detect suspicious cues.The heatmaps show substantial diagonal mass, but partial-truth cases remain intrinsically ambiguous.
Loading 2606.24112v2…