Source-linked AI summary
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
Zhenglin Huang, Jinwei Hu, Xiangtai Li, Yiwei He, Xingyu Zhao, Bei Peng, Baoyuan Wu, Xiaowei Huang, Guangliang Cheng
TL;DR
Realistic social-media deepfakes expose gaps in dataset diversity and task coverage. The paper introduces SID-Set and the VLM-based SIDA framework for detection, localization, and explanation, reporting superior or equivalent performance across SID-Set and other benchmarks. The authors note dataset scale and generator diversity as remaining limitations.
Problem
Existing deepfake datasets lack sufficient social-media diversity and comprehensiveness, often covering limited imagery, manipulation types, or forensic tasks.
Method
The paper constructs SID-Set with real, fully synthetic, and tampered images and proposes SIDA, a VLM-based framework for detection, localization, and explanation.
Results
SIDA demonstrates superior or equivalent performance on SID-Set and other benchmarks while detecting fake types, localizing tampered regions, and explaining its decisions.
Takeaways & Limitations
SID-Set and SIDA provide a unified benchmark and framework for social-media image deepfake detection, localization, and explanation.
Takeaways & Limitations
SID-Set may require more images, and relying exclusively on FLUX may create data skew that affects performance on diverse datasets.
Abstract
from arXiv · showhide
The rapid advancement of generative models in creating highly realistic images poses substantial risks for misinformation dissemination. For instance, a synthetic image, when shared on social media, can mislead extensive audiences and erode trust in digital content, resulting in severe repercussions. Despite some progress, academia has not yet created a large and diversified deepfake detection dataset for social media, nor has it devised an effective solution to address this issue. In this paper, we introduce the Social media Image Detection dataSet (SID-Set), which offers three key advantages: (1) extensive volume, featuring 300K AI-generated/tampered and authentic images with comprehensive annotations, (2) broad diversity, encompassing fully synthetic and tampered images across various classes, and (3) elevated realism, with images that are predominantly indistinguishable from genuine ones through mere visual inspection. Furthermore, leveraging the exceptional capabilities of large multimodal models, we propose a new image deepfake detection, localization, and explanation framework, named SIDA (Social media Image Detection, localization, and explanation Assistant). SIDA not only discerns the authenticity of images, but also delineates tampered regions through mask prediction and provides textual explanations of the model's judgment criteria. Compared with state-of-the-art deepfake detection models on SID-Set and other benchmarks, extensive experiments demonstrate that SIDA achieves superior performance among diversified settings. The code, model, and dataset will be released.
1. Introduction
Existing deepfake resources are limited in diversity and comprehensiveness for realistic social-media imagery. This paper introduces SID-Set and SIDA to support detection, localization, and explanation.
- Proposed framework: SIDA combines deepfake detection, localization, and textual explanation, whereas existing methods address only some of these tasks.The framework is presented as a more comprehensive solution for social-media image forensics.
- Dataset gaps: Existing datasets focus mainly on facial imagery and often use simple scenarios or outdated generation techniques.These limitations make them less representative of realistic non-facial social-media forgeries.
- Dataset gaps: Existing resources commonly separate deepfake detection from tampered-region localization and cover limited manipulation types.A comprehensive benchmark should include fully generated and edited images.
- Proposed resource: 300K images comprise SID-Set: 100K real, 100K synthetic, and 100K tampered images with extensive annotations.The dataset targets high-quality, diverse social-media images, including subtle alterations and complex scenes.
- Realism: SID-Set images are designed to be difficult to distinguish from genuine images through visual inspection.The reported challenges include alterations affecting only dozens of pixels, natural-looking local manipulations, and complex scenes.
2. Related Work
Prior work spans facial and non-facial deepfake datasets, conventional detection models, and multimodal systems for detection or localization. SIDA extends this direction with a social-media-focused dataset and a unified detection, localization, and explanation framework.
- Image deepfake datasets: Deepfake datasets historically emphasized facial imagery, while newer resources include non-facial data, diverse generation methods, and richer annotations.SID-Set is positioned as specifically tailored to social-media images and recent generation models.
- Deepfake detection: Traditional deepfake detection commonly treats the task as classification using CNNs or Transformers to identify visual artifacts.Prior work also explores augmentation, adversarial training, and reconstruction to improve precision or generality.
- Large multimodal models: Vision-language models integrate visual and textual information, while LISA-style systems apply language-model reasoning to image segmentation.Grounding multimodal models can localize image contents based on linguistic information.
- Large multimodal models: Recent multimodal methods formulate detection as visual question answering or combine forensic knowledge with mask-aware extraction for localization.Examples include AntifakePrompt, ForgeryGPT, and FakeShield, which also pursue interpretable outputs.
- SIDA: SIDA differs by pairing a large social-media deepfake dataset with VLM-based detection, forgery localization, and explanations.The paper presents this combination as its distinction from existing approaches.
3. Benchmark
SID-Set is a social-media benchmark designed to cover real, fully synthetic, and tampered images, including realistic non-facial manipulations and localization targets. Its construction combines diverse image sources with object and attribute editing pipelines and textual annotations.
- Motivation: The benchmark addresses datasets that mainly emphasize facial content or only one deepfake type, despite non-facial and mixed cases in social media.
- Tampered-image generation: The tampering pipeline extracts caption objects with GPT-4o, obtains masks with Language-SAM, builds replacement dictionaries, and regenerates images with Latent Diffusion.
- Benchmark scope: SID-Set evaluates whether models distinguish real, synthetic, and tampered images and localize altered regions.
- Dataset composition: SID-Set contains 100K real images from OpenImages, 100K synthetic images generated with FLUX, and 100K tampered images.
- Tampered-image generation: Tampering includes object replacement and attribute modification, such as changing a cat to a dog or altering an object’s characteristics.
- Annotations: The benchmark includes textual judgment-basis descriptions for 3,000 images and quality control by five annotation experts.
4. Method
SIDA extends a vision-language model with detection and segmentation tokens to classify image authenticity and localize tampered regions. It combines detection, mask, and text-generation objectives, using image features and annotated descriptions for training.
- Architecture: SIDA adds <DET> and <SEG> tokens to a vision-language model for detection, segmentation, and explanatory text generation.
- Detection: The <DET> representation passes through a detection head to classify images as real, fully synthetic, or tampered.
- Localization: For tampered images, SIDA extracts segmentation features, combines them with detection features through attention and a residual connection, and predicts masks from image features.
- Training: Training uses detection cross-entropy and a weighted BCE-DICE mask loss, followed by text-generation fine-tuning on descriptions from 3,000 images.
- Training: SIDA is trained on SID-Set together with filtered MagicBrush data, with descriptions generated for 3,000 randomly selected images.
5. Experiments
Experiments evaluate SIDA for detection, localization, robustness, generalization, ablation, and qualitative explanation on SID-Set and related benchmarks. SIDA achieves the best reported localization performance, remains resilient to common perturbations, and demonstrates accurate outputs alongside failure cases.
- Detection Evaluation: SIDA achieves better or comparable detection results than evaluated methods on SID-Set, although retrained LGrad attains the highest tampered-image accuracy and F1.The comparison uses open-source models directly and after fine-tuning on SID-Set.
- Localization Results: SIDA achieves the best forgery-localization performance among the compared IFDL methods and fine-tuned LISA.The comparison includes PSCC-Net, MVSS-Net, HIFI-Net, and LISA-7B-v1.
- Robustness Evaluation: SIDA remains resilient to JPEG compression, resizing, and Gaussian noise despite not being explicitly trained on degraded data.The evaluation covers six degradation scenarios, including two JPEG qualities, two resizing factors, and two noise variances.
- Generalization Evaluation: SIDA achieves superior performance on the DMimage dataset, demonstrating adaptability beyond SID-Set.The comparison uses the other methods’ original hyperparameters and pretrained weights.
- Ablation Study: Removing or replacing the attention module with fully connected layers significantly reduces detection and localization performance.The ablation identifies attention as important for feature interaction and task accuracy.
- Qualitative Results: Qualitative examples show accurate tampered-region detection and textual explanations, while failure cases reveal unresolved localization errors.The paper presents these examples as areas for future improvement.
6. Conclusion and Discussions
The paper presents SID-Set and SIDA as resources and a framework for social-image deepfake detection, localization, and explanation. It also identifies dataset scale, generator diversity, and localization reliability as areas requiring further work.
- Conclusion: SID-Set contains 100k real, 100k fully synthetic, and 100k tampered images for social-image deepfake detection, localization, and explanation.The conclusion describes the dataset as supporting all three tasks.
- Conclusion: SIDA uses vision-language models to detect fake types, localize tampered regions, and explain its decisions.The paper presents this integration as a direction for future research.
- Limitations: The authors identify the need for more images because the complexity of real social-media environments exceeds the current dataset scale.They specifically note that SID-Set includes 100k fully synthetic and 100k tampered images.
- Limitations: Using only FLUX may skew the data domain, motivating future inclusion of additional generation methods.The authors state that this issue was not significantly evident in their experiments but could affect performance on diverse datasets.
- Limitations: Some tampered regions are not reliably detected, leaving room for improvement in localization.This limitation is reported despite relatively strong performance on the authors’ dataset.
A. Experiment Settings
The experiments compare detection methods using established implementations, controlled preprocessing and training settings, and reported related-work baselines. Methods unable to localize are evaluated only with classification labels.
- Detection Methods: CnnSpot, FreDect, Fusing, Gram-Net, UnivFD, LGrad, and LNP are tested and retrained using AIGCDetecBenchmark2, while AntifakePrompt follows its official repository settings.The comparison evaluates methods using classification labels because they cannot handle localization tasks.
- Training Configuration: Testing disables noise preprocessing, while retraining uses specified blur, JPEG, batch-size, crop-size, optimizer, and epoch settings.Most listed methods are trained for 10 epochs with Adam and a batch size of 64; method-specific settings vary.
- Related-Work Comparison: Related-work comparisons include methods that created their own datasets, as indicated by an asterisk in Table 8.The table caption defines the asterisk convention but does not report additional performance details here.
B. Detailed Comparison
SIDA is compared with detection and localization methods across broader manipulation scenarios, while also providing fine-grained explanations. Visual comparisons show clearer tampered-region boundaries, alongside documented failure cases.
- SIDA handles fully synthetic, tampered, and real images, extending beyond methods focused on tampered-versus-real detection.
- SIDA combines segmentation masks with detailed textual explanations, adding an interpretive layer to localization results.
- Compared with FFAA, FakeShield, and ForgeryGPT, the approach uses a larger or more realistic dataset and produces fine-grained outputs across multiple tasks.
- SIDA detects tampered-area borders more precisely and clearly than PSCC-Net and LISA after both comparison models were retrained on SID-Set for 10 epochs.
- Figure 11 includes failures where SIDA misses tampered areas, produces no mask, or cannot generate fine-grained masks.
- The authors attribute these shortcomings to insufficient tampered-image training data and limited precision on subtle or intricate manipulations.
D. Detailed DataSet Creation Process
The dataset creation process includes generated descriptions and tampered-image construction organized around object replacement and attribute modification. The paper illustrates these procedures with prompts and examples.
- GPT-4o prompts were designed separately for real, fully synthetic, and tampered images to generate corresponding descriptions.
- Figures 12–14 show the prompts, while Figures 15–17 provide generated-description examples for the three image types.
- Tampered-image generation uses separate directories for replacing entire objects and modifying object attributes.
E. Experts and Human Evaluation
Human evaluation supports model and data-quality decisions through expert assessment of generated images and textual descriptions. The evaluation also includes prompts for classification, masking, and authenticity analysis.
- Five experts undertook three evaluation tasks for the dataset and generated outputs.
- Experts examined approximately 1,000 images and selected FLUX and latent-diffusion as default generative models based on output quality and consistency.
- Images received realism scores from 0 to 5, and those scoring below 3 were secondarily reviewed before flawed images were excluded.
- Experts reviewed 3,000 GPT-4 descriptions for accuracy, clarity, and consistency with their images and prompts.
- The evaluation prompts ask whether an image is real, fully synthetic, or tampered and request a mask for tampered content.
- A fully synthetic-image prompt requests technical and human-centered explanations of how artificiality can be identified.
1. Technical Analysis
The technical-analysis prompts direct SIDA to inspect visual inconsistencies associated with image manipulation. The dataset materials also distinguish fully synthetic and tampered-image examples and organize manipulation directories by operation type.
- The technical-analysis prompt asks the model to inspect lighting, color, shading, and reflections for unnatural or inconsistent patterns.
- Prompts and outputs are provided for fully synthetic and tampered images, including example outputs for each category.
- The classification prompt requires identifying an image as real, fully synthetic, or tampered and masking the tampered object or part.
- SID-Set directories organize tampering examples by object replacement and attribute modification.