Source-linked AI summary
AIRD: Adversarial Learning Framework for Image Repurposing Detection
Ayush Jaiswal, Yue Wu, Wael AbdAlmageed, Iacopo Masi, Premkumar Natarajan
TL;DR
Image repurposing spreads misinformation by pairing untampered images with falsified metadata, and detection research faces a lack of training and evaluation data. AIRD adversarially trains a counterfeiter and detector using reference knowledge, achieving state-of-the-art performance across three verification tasks.
Problem
Image repurposing spreads misinformation using untampered images and falsified metadata, while detection methods lack training and evaluation data.
Source: §1 · p. 2Method
Results
AIRD achieves state-of-the-art performance on Google Landmarks location-identity, IJBC-IRD subject-identity, and Painter by Numbers painting-artist verification.
Source: §1 · p. 2Takeaways & Limitations
IJBC-IRD adds a face-image dataset with subject-identity metadata for future image-repurposing research.
Source: §1 · p. 2IJBC-IRD uses still images only, excluding motion frames because video frames may have motion blur and lack discriminative facial features.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
Image repurposing is a commonly used method for spreading misinformation on social media and online forums, which involves publishing untampered images with modified metadata to create rumors and further propaganda. While manual verification is possible, given vast amounts of verified knowledge available on the internet, the increasing prevalence and ease of this form of semantic manipulation call for the development of robust automatic ways of assessing the semantic integrity of multimedia data. In this paper, we present a novel method for image repurposing detection that is based on the real-world adversarial interplay between a bad actor who repurposes images with counterfeit metadata and a watchdog who verifies the semantic consistency between images and their accompanying metadata, where both players have access to a reference dataset of verified content, which they can use to achieve their goals. The proposed method exhibits state-of-the-art performance on location-identity, subject-identity and painting-artist verification, showing its efficacy across a diverse set of scenarios.
1. Introduction
Image repurposing uses untampered images with falsified metadata to spread misinformation, creating a need for automated semantic-integrity assessment. AIRD models detection as adversarial training between a counterfeiter and a watchdog detector.
1. Introduction
- Image repurposing reuses unmanipulated images with falsified metadata to mislead about a similar but different entity or event.The examples span lookalikes, incorrect locations, and wrongly claimed artists.Source: Figure 1 · p. 1, §1 · p. 2
- AIRD adversarially trains a counterfeiter and detector and can be trained without manipulated-metadata training data.The framework models the real-world interplay between an actor fabricating metadata and a watchdog verifying image-metadata consistency.Source: §1 · p. 2
- The framework uses a reference dataset of unmanipulated packages as world knowledge, but evaluates structured metadata because natural-language generation remains an open problem.The authors note that AIRD could also address caption entity manipulations after named-entity extraction.Source: §1 · p. 2
- AIRD achieves state-of-the-art performance on Google Landmarks location-identity, IJBC-IRD subject-identity, and Painter by Numbers painting-artist verification.The authors describe the results across these datasets as illustrating the model’s generalization capability.Source: §1 · p. 2
- The paper introduces IJBC-IRD, a face-image dataset with subject-identity metadata, to support research on face-image repurposing detection.It is a variant of IJB-C.Source: §1 · p. 2
2. Related Work
Prior work addresses textual rumors, pixel-level image manipulation, and semantic inconsistency in image packages. AIRD applies adversarial learning to model image repurposing as a contest between malicious actors and watchdogs.
2. Related Work
- Rumor-detection methods analyze text, post propagation, or audience response, while pixel-level manipulation methods inspect watermarks or image artifacts.These approaches are not targeted at image-based information manipulation with image analysis of semantic claims.Source: §2 · p. 2
- Prior semantic-integrity work addressed copied captions and manipulated entities or locations, while AIRD proposes a generalized image-repurposing framework.The cited prior work includes image-caption packages and packages with GPS information.Source: §2 · p. 2
- AIRD uses adversarial learning to model the real-life interplay between malicious actors and watchdogs, with a counterfeiter and detector using reference data.The framework’s Metadata Generator fabricates metadata, while its Consistency Verifier assesses the query’s claimed metadata using retrieved evidence.Source: §2 · p. 3, Figure 2 · p. 3
3. Adversarial Image Repurposing Detection
AIRD models image repurposing as an informed contest: a counterfeiter and detector exploit a shared reference set for deception and verification. Its design combines retrieval, candidate generation, evidence aggregation, and adversarial training.
3. Adversarial Image Repurposing Detection
- AIRD pairs a counterfeiter that fabricates metadata with a detector that verifies query packages, with both models using the reference set.Their parameters are learned through adversarial training.Source: §3 · p. 3
3.1.EncodingandRetrieval fromReferenceDataset
- For scalable retrieval, AIRD uses modality-specific encodings and approximate search rather than brute-force nearest-neighbor search over large reference sets.Its cascaded index uses IVFPQ+R followed by exact-cosine-similarity reranking.Source: 3.1.EncodingandRetrieval… · pp. 3–4
3.2. Counterfeiter Model C
- The counterfeiter retrieves similar images with dissimilar metadata, scores the candidates against the original package, then uses a sharp choice distribution to fabricate metadata.The weighted candidate metadata are summed to produce the fabricated metadata.Source: §3.2 · p. 4
3.3. Detector Model D
- The detector retrieves evidence using the query image and metadata separately, then combines query and retrieved information within and across modalities to judge integrity.The consistency verifier performs the combinations and produces the integrity judgment.Source: §3.3 · pp. 4–5
3.4. Training AIRD
- Training adds randomly sampled fake metadata and metadata from the most similar image-based retrieval as easy and hard negatives for the verifier.The structured-metadata encoder receives no direct gradients from the verifier, a design intended to stabilize training and encourage robustness.Source: §3.4 · p. 5
4. Experimental Evaluation
The evaluation tests AIRD on three identity-verification datasets, examines retrievals and counterfeiter-generated candidates, and compares AIRD with baseline and prior models. AIRD outperforms the alternatives across the reported metrics, while retrieval performance varies with dataset difficulty.
4.1. Benchmark Datasets
- The benchmarks cover landmark location identity, face subject identity, and painting artist identity; IJBC-IRD is a still-image subset of IJB-C selected for realistic face-repurposing scenarios.IJBC-IRD focuses on subjects likely to be used for identity repurposing and includes subjects with at least five media; the evaluation uses 16,377 images across 1,649 subjects.Source: §4.1 · pp. 5–6
- Each dataset is split into stratified 80% training and 20% testing sets, with training data also serving as the reference dataset.Retrievals for counterfeiter fake candidates and detector evidence were precomputed to speed training.Source: §4.1 · p. 6
4.2. Indexing and Retrieval Performance
- Retrievals are usually visually similar to the query and share its metadata, but similar images with different metadata can cause errors.The paper attributes these errors to cascading semantic-encoding and approximate nearest-neighbor search errors.Source: §4.2 · p. 6
- At K=3, retrieval performance is best on IJBC-IRD, followed by Google Landmarks and then Painter by Numbers.MAP@K and Precision@K measure accurate metadata matching; the paper describes artist identity as subtler than subject or location identity.Source: §4.2 · p. 7, Table 1 · p. 7
4.3. Fake Candidates for Counterfeiter
- The fake-candidate examples show that the datasets contain visually similar images with dissimilar metadata, while audiences in real cases see a fake candidate without the query image.The paper says this presentation makes it easy to fool people.Source: §4.3 · p. 7
4.4. Baseline and State-of-the-Art Models
- The comparison includes non-learning baselines based on similarities between query and retrieved packages, plus a metadata predictor that checks predicted against claimed metadata.The non-adversarial detector is an ablation trained with real pairs and easy and hard negatives.Source: §4.4 · p. 7
- NAD provides the non-adversarial AIRD ablation, trained on real image-metadata pairs and easy and hard negatives.Its comparison with AIRD is used to assess the additional contribution associated with adversarial training.Source: §4.4 · p. 8, §4.5 · p. 8
4.5. Results
- AIRD outperforms all other models on every reported metric across the three datasets; NAD beats other baselines but trails AIRD, and AIRD exceeds DMM by a large margin.The paper attributes AIRD’s additional boost over NAD to adversarial training and says DMM is not suitable for image repurposing with structured identity metadata.Source: Table 2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
5. Conclusion
The conclusion presents AIRD as an adversarial framework in which a counterfeiter and detector use a reference dataset. It reports that AIRD outperforms baseline and prior state-of-the-art models across three identity-verification datasets.
5. Conclusion
- AIRD pairs a counterfeiter that fabricates metadata for untampered images with a detector that uses reference-dataset evidence to verify package integrity.Both models use retrieved information from the reference dataset to their advantage.Source: §5 · p. 8
- The evaluation spans location identity on Google Landmarks, subject identity on IJBC-IRD, and painting artist identity on Painter by Numbers.Source: §5 · p. 8
- AIRD outperforms all baseline models and prior state of the art on all metrics across the evaluated datasets.Source: §5 · p. 8