PaperlayerView exact revision on arXiv

Source-linked AI summary

ViGText: Deepfake Image Detection with Vision-Language Model Explanations and Graph Neural Networks

Ahmad ALBarqawi, Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, NhatHai Phan

arXiv:2507.18031v2cs.CVcs.AIcs.LG

TL;DR

Deepfake detectors can struggle to generalize to customized generators, and captions may lack the specificity needed to reveal subtle inconsistencies. ViGText combines image patches and VLLM-generated explanations in a dual-graph framework. The paper reports strong generalization and adversarial robustness, while its evaluation remains focused on fully synthetic images and visual inputs.

  • Problem

    Deepfake detectors can degrade on user-customized generators, and captions may not capture the detail needed to reveal subtle visual inconsistencies.

    Source: §I · p. 2

  • Method

    ViGText links image-patch and VLLM-explanation graphs and analyzes their spatial, frequency, and textual information with a GNN.

    Source: §I · p. 2

  • Results

    ViGText demonstrates generalization to fine-tuned and user-customized deepfakes, resistance to adversarial manipulations, and state-of-the-art detection performance.

    Source: §VIII · p. 13

  • Takeaways & Limitations

    The framework combines visual and textual evidence to detect subtle artifacts and inconsistencies in synthetic images.

    Source: §VIII · p. 13

    The evaluation focuses on fully synthetic images, not real-world partial manipulations such as face swaps, reenactments, or spliced content.

    Source: §VII · p. 13

Abstract

from arXiv · show

The rapid rise of deepfake technology, which produces realistic but fraudulent digital content, threatens the authenticity of media. Traditional deepfake detection approaches often struggle with sophisticated, customized deepfakes, especially in terms of generalization and robustness against malicious attacks. This paper introduces ViGText, a novel approach that integrates images with Vision Large Language Model (VLLM) Text explanations within a Graph-based framework to improve deepfake detection. The novelty of ViGText lies in its integration of detailed explanations with visual data, as it provides a more context-aware analysis than captions, which often lack specificity and fail to reveal subtle inconsistencies. ViGText systematically divides images into patches, constructs image and text graphs, and integrates them for analysis using Graph Neural Networks (GNNs) to identify deepfakes. Through the use of multi-level feature extraction across spatial and frequency domains, ViGText captures details that enhance its robustness and accuracy to detect sophisticated deepfakes. Extensive experiments demonstrate that ViGText significantly enhances generalization and achieves a notable performance boost when it detects user-customized deepfakes. Specifically, average F1 scores rise from 72.45% to 98.32% under generalization evaluation, and reflects the model's superior ability to generalize to unseen, fine-tuned variations of stable diffusion models. As for robustness, ViGText achieves an increase of 11.1% in recall compared to other deepfake detection approaches. When facing targeted attacks that exploit its graph-based architecture, ViGText limits classification performance degradation to less than 4%. ViGText uses detailed visual and textual analysis to set a new standard for detecting deepfakes, helping ensure media authenticity and information integrity.

I. INTRODUCTION

ViGText addresses the difficulty of generalizing deepfake detection to customized generators and the challenge of combining detailed textual and visual evidence. It uses VLLM explanations and image patches in a dual-graph framework, with background on deepfakes, GNNs, and vision-language models.

II. BACKGROUND

  • Deepfakes are synthetic media made by altering or replacing a person’s appearance or voice; GNNs process graph data by exchanging messages across edges and aggregating information at nodes.The background also describes VLLMs as combining visual and textual data for tasks including captioning, question answering, classification, and segmentation.Source: §I · p. 1, §II · p. 3

I. INTRODUCTION

  • Existing detectors can degrade on user-customized or fine-tuned generators, while image captions may lack the detail needed to identify subtle inconsistencies.The paper also identifies meaningful integration of text and visual data as a challenge.Source: §I · p. 2
  • ViGText links image-patch and explanation graphs, using spatial and DCT-derived frequency features before a GNN classifies images as real or fake.Each patch is a node, adjacent patches are connected, and each explanation is linked to the patches it describes.Source: §I · p. 2
  • ViGText’s contributions include generalization to user-customized fine-tuned models, resilience to adversarial attacks, and eight new test sets derived from fine-tuned Stable Diffusion 3.5 variants.The new sets are intended to broaden evaluation across generative models.Source: §I · pp. 2–3

III. THREAT MODEL AND ASSUMPTIONS

The threat model considers adversaries who seek to evade detection with fine-tuned generators and subtle semantic edits, and defenders aiming to generalize detection across variants. ViGText integrates image patches and VLLM explanations in graphs, using their visual-textual relationships to detect inconsistencies that broad captions may miss.

III. THREAT MODEL AND ASSUMPTIONS

  • The adversary seeks evasion and may use fine-tuned generators or localized StyleCLIP edits, while the defender aims to generalize detection across variants and withstand adversarial attacks.The paper treats these StyleCLIP edits as fully synthetic manipulations, not as real-world partial edits such as face swaps or reenactments.Source: §III · pp. 3–4

IV. THE PROPOSED VIGTEXT

  • ViGText constructs image-patch and explanation graphs, integrates them, and applies a GNN to detect deepfakes.The method uses spatial and frequency information in patch embeddings and links explanations to the patches they describe.Source: §IV · p. 4

A. From Captions to Explanations

  • Generic captions can miss visual manipulation cues, whereas VLLM explanations can describe specific details such as reflections, shadows, and textures.ViGText compares the explanations with image content to identify discrepancies, rather than relying on VLLMs alone to classify images.Source: A. From Captions to Explanations · pp. 4–5

B. Problem Formulation

ViGText formulates deepfake detection as binary classification from images and generated explanations, while emphasizing meaningful visual–textual integration rather than simple concatenation.

B. Problem Formulation

  • The classifier maps an image and its explanation to a real-or-fake label and is optimized for accuracy on labeled image–explanation examples.The formulation also specifies robustness to allowable perturbations and generalization to unseen data through thresholds.Source: B. Problem Formulation · p. 5
  • Visual and textual information must be integrated carefully to use the complementary information in images and explanations.The formulation identifies integration as a central consideration for understanding the benefit of textual information.Source: B. Problem Formulation · p. 5

C. Visual and Textual Integration

  • DE-FAKE concatenates caption and image embeddings, an approach described as failing to capture detailed interdependencies.ViGText instead uses a graph-based model to relate explanations and visual data.Source: C. Visual and Textual Integration · p. 5
  • The integration experiment compares ViGText with DE-FAKE [21] using the same explanations as textual input.Table I is identified as the performance comparison for this controlled input setting.Source: C. Visual and Textual Integration · p. 5, Table I · p. 5
  • 90% accuracy is reported for DE-FAKE [21] when richer explanations are concatenated with image features.The result indicates that replacing captions with explanations alone did not yield accurate detection in this comparison.Source: C. Visual and Textual Integration · p. 5
  • ViGText’s graph structure captures interdependencies between textual explanations and visual features, improving detection performance in the comparison.The passage contrasts this integration with arbitrary embedding concatenation.Source: C. Visual and Textual Integration · p. 5

D. Explanation-Patch Integrated Graph Construction

ViGText builds patch-specific explanations and combines word and image-patch graphs, linking textual descriptions to corresponding visual regions for GNN analysis.

Explanation Generation

  • A grid overlay prompts the VLLM to produce explanations tied to localized image regions, rather than one unlocalized full-image description.Grid-based associations support precise cross-modal edges; patch size trades fine detail against global context.Source: D. Explanation-Patch Integrated Graph… · pp. 5–6

Graph Integration

  • ViGText treats VLLM explanations as evidence to cross-check against visual patch features, not as a standalone trust anchor.The passage says the GNN learns from matching and deliberately mismatched image–explanation pairs to detect cross-modal inconsistencies.Source: D. Explanation-Patch Integrated Graph… · p. 6

Text Graph Construction

  • Explanation graphs represent words as nodes, grammatical relationships as edges, and word embeddings as node features.The word nodes are connected to the image-graph node for the patch associated with that explanation.Source: D. Explanation-Patch Integrated Graph… · pp. 6–7

Image Graph Construction

  • Each image patch receives averaged spatial and DCT-derived frequency features, which form the feature representation of an image-graph node.Adjacent patch nodes are connected, and the same image feature extractor processes spatial and DCT-transformed patches.Source: D. Explanation-Patch Integrated Graph… · p. 6

Graph Integration

  • The integrated graph combines image patches and explanation words, with cross-modal links associating each word node with its corresponding patch.The figure’s example illustrates only two links; the caption specifies that the implementation connects all explanation-graph nodes to corresponding patch nodes.Source: D. Explanation-Patch Integrated Graph… · p. 7, Figure 7 · p. 7

V. EXPERIMENTS

Experiments assess ViGText’s detection, generalization, and robustness on curated diffusion-image and adversarial face-image datasets, including attacks designed with knowledge of its graph pipeline.

V. EXPERIMENTS

  • ViGText is reported to achieve state-of-the-art detection across multiple classification metrics, with generalization, adversarial robustness, design flexibility, and computational cost also assessed.The paper characterizes computational cost as comparable to, or tolerable relative to, existing state-of-the-art approaches.Source: §V · p. 7

A. The Setup, Dataset, and Baselines

  • The Stable Diffusion dataset uses balanced real and fake data and tests generalization on 16 fine-tuned SD 1.5 variants plus eight SD 3.5 LoRA variants.The SD 3.5 extension uses LoRA variants because full fine-tuning an 8-billion-parameter model would require impractical resources for most use cases.Source: A. The Setup, Dataset, and Baselines · pp. 7–8
  • The balanced StyleCLIP face dataset evaluates robustness to semantic image manipulations crafted with EfficientNet, ViT, and CLIP-ResNet surrogates.The described manipulations include changing expressions, adding accessories, or modifying facial attributes without perceptible noise.Source: A. The Setup, Dataset, and Baselines · p. 8
  • A targeted attack assumes access to ViGText’s training data and graph pipeline, then uses a high-performing surrogate to train StyleGAN2-generated evasive images.The surrogate uses two graph convolutional layers and Dinov2 and Jina embeddings; its accuracy, recall, precision, and F1 each exceed 95%.Source: A. The Setup, Dataset, and Baselines · p. 8
  • The evaluation does not cover coordinated attacks that target both images and VLLM-generated explanations.The paper states that such attacks would require white-box access to both components and are left for future research.Source: A. The Setup, Dataset, and Baselines · p. 8
  • Experiments report accuracy, precision, recall, and F1 and compare ViGText with DCT [20], DE-FAKE, and UnivCLIP.The ViGText experiments use a consistent three-layer GAT architecture and Qwen2-VL-7B-Instruct to generate explanations.Source: A. The Setup, Dataset, and Baselines · pp. 8–9

B. Performance Analysis

ViGText outperforms the compared methods on the reported detection metrics and generalizes across fine-tuned Stable Diffusion variants, including SD 3.5 LoRA models.

B. Performance Analysis

  • ViGText consistently outperforms the compared state-of-the-art techniques across the reported quality metrics on both datasets.Table III presents performance on the respective dataset testing sets; the supplied passage does not state individual table values.Source: B. Performance Analysis · p. 9, Table III · p. 9

C. Generalization

  • Across 24 fine-tuned Stable Diffusion testing sets, ViGText outperforms baselines on all metrics, including SD 1.5 FM and SD 1.5 and 3.5 LoRA variants.The results are summarized across eight FM variants and 16 LoRA variants.Source: C. Generalization · p. 9, Figure 8 · p. 9
  • The paper attributes generalization to graph structural-topology learning and content-agnostic frequency features that reduce dependence on image content and training-distribution characteristics.The passage describes these features as capturing subtle features invariant to image content.Source: C. Generalization · pp. 9–10
  • The robustness figure compares ViGText and baselines on adversarially manipulated StyleCLIP test sets crafted with EfficientNet, ViT, and CLIP-ResNet surrogates.The supplied passages describe the figure’s evaluation setting but do not report its specific performance values.Source: Figure 9 · p. 10

D. Robustness

ViGText is evaluated against foundation-model attacks, an informed surrogate attack, resolution changes, geometric and appearance-based warps, and FGSM/PGD attacks. Across these settings, the reported results show resilience, including smaller degradation under an informed attack than under the original metrics.

D. Robustness

  • Across foundation-model attacks, resolution changes, geometric and appearance-based warps, and FGSM/PGD attacks, ViGText consistently outperforms other methods or achieves the highest tested accuracy.The foundation-model tests use surrogate models EfficientNet, ViT, and CLIP-ResNet; the SD tests cover resolution and warp transformations, while the FGSM/PGD tests vary noise levels.Source: D. Robustness · pp. 10–11, Table IV · p. 10, Table V · p. 11
  • ViGText consistently outperforms other methods across evaluated metrics on StyleCLIP adversarial sets generated using EfficientNet, ViT, and CLIP-ResNet surrogates.The attacks use foundation models trained on millions of images, and the passage reports less performance degradation than for baseline approaches.Source: D. Robustness · p. 10
  • Under a highly informed surrogate attack, ViGText records 95.85% accuracy, 91.7% recall, 99.2% precision, and 95.67% F1, versus original metrics of 99.6% accuracy, 99.21% recall, 99.9% precision, and 99.6% F1.The surrogate shares the StyleCLIP training data and graph-construction pipeline; its adversarial images are crafted by optimizing StyleGAN2 to minimize cross-entropy loss against the real-image target.Source: D. Robustness · p. 10
  • On the SD dataset, ViGText shows minimal degradation across tested resolutions and outperforms all baseline methods at each resolution.The dataset's original images are 512x512; the StyleCLIP resolution results are also reported, for images originally at 1024x1024.Source: D. Robustness · pp. 10–11, Table IV · p. 10
  • ViGText maintains a clear advantage under FGSM and PGD attacks at varying noise levels, achieving the highest accuracy in every tested condition.The paper also reports superior metrics under geometric and appearance-based warp operations on the SD dataset.Source: D. Robustness · p. 11, Table VII · p. 11, Table V · p. 11

E. Sensitivity to Design Choices

ViGText's sensitivity to patch size depends on the test images: smaller patches help on fine-tuned models with localized artifacts, while larger patches help on artifact-free adversarial images. This suggests a possible role for adaptive patching.

E. Sensitivity to Design Choices

  • On the fine-tuned model datasets, performance increases as patch size decreases and the number of patches increases, while SD and StyleCLIP performance stays relatively stable across patch sizes.The tables report results across patch sizes on SD and StyleCLIP and across patch counts on LoRA- and FM-fine-tuned models.Source: E. Sensitivity to Design Choices · p. 11, Table VIII · p. 11, Table IX · p. 11
  • Smaller patches improve generalization on LoRA- and FM-fine-tuned images by capturing their localized artifacts and increasing the graph's spatial granularity.The frequency-spectrum figure identifies artifacts in both fine-tuned image types; smaller patches create more graph nodes and can better represent localized distortions.Source: E. Sensitivity to Design Choices · pp. 11–12, Table IX · p. 11, Figure 10 · p. 12
  • On adversarially manipulated StyleCLIP sets, larger patches yield better results because the images lack clear artifacts and smaller patches can dilute weak local signals.Larger patches also preserve global patterns that may be important for detecting evasion-oriented manipulations.Source: E. Sensitivity to Design Choices · p. 12, Table X · p. 12, Figure 10 · p. 12
  • Because the preferred patch size varies with image characteristics, the authors suggest adaptive patching to improve robustness and generalization.The proposed strategy would dynamically adjust patch size based on the input image.Source: E. Sensitivity to Design Choices · p. 12

F. Cost Analysis

ViGText adds little runtime relative to UnivCLIP in the reported comparison. The related-work discussion distinguishes CNN-based detectors from simpler models that use specialized feature extraction, while noting generalization and adversarial weaknesses in prior approaches.

F. Cost Analysis

  • ViGText takes 1.755 seconds per image versus UnivCLIP's 1.650 seconds, a reported increase of 0.105 seconds on average.Timing includes preprocessing, inference, and graph construction for ViGText, and is averaged over 2000 test samples.Source: F. Cost Analysis · p. 12

VI. RELATED WORK

  • CNN-based approaches span patch artifacts, global textures, resynthesis [17], and mesoscopic facial-video properties.Source: §VI · p. 12
  • Simpler feature-based approaches use frequency features [20], combine text prompts with image features, or use a feature space not trained for deepfake detection.The cited discussion notes that prior methods can generalize poorly to user-customized models and degrade on foundation-model adversarial images [23].Source: §VI · p. 13

VII. DISCUSSION

ViGText combines visual data and VLLM explanations in a graph-based framework, with reported benefits for generalization and resistance to adversarial manipulation. The discussion identifies scope and design limitations and outlines possible extensions.

VII. DISCUSSION

  • The study presents its visual-and-textual graph approach as improving detection reliability and robustness against evolving generative models and sophisticated adversarial attacks.It also reports improved generalization to user-customized models.Source: §VII · p. 13
  • The authors suggest that pairing data with textual explanations could also support tasks such as toxic chemical identification, drug discovery, content moderation, document verification, and fake news detection.These are proposed application areas, not reported evaluations in this passage.Source: §VII · p. 13
  • The evaluation covers fully synthetic images, including latent-space edits such as StyleCLIP [23], but not real-world partial manipulations such as face swaps, reenactments, or spliced content.This limits the threat model's coverage of those manipulation types.Source: §VII · p. 13
  • The current framework supports visual inputs only, while fixed-size patches and word-to-patch linking may reduce adaptability to image complexity and contextual precision.Future work proposes adaptive patching and linking, heterogeneous graph neural networks, and audio- and video-based extensions that address temporal, alignment, and signal-quality challenges.Source: §VII · p. 13

VIII. CONCLUSION

  • ViGText combines visual data, textual explanations, and spatial- and frequency-domain features in a dual-graph framework, with reported state-of-the-art detection performance and computational efficiency.The conclusion presents the framework as a response to subtle artifacts and inconsistencies that trouble traditional methods.Source: §VIII · p. 13

ETHICS CONSIDERATIONS

  • The study reports using open-source datasets and no sensitive data concerning privacy and security.It states that its research was designed to respect privacy and individual rights, avoid bias, and minimize potential harm.Source: ETHICS CONSIDERATIONS · p. 14

APPENDIX A EXPLANATION GENERATING PROMPT

The appendix describes the explanation prompt and evaluation on Stable Diffusion 3.5 LoRA models, commercial diffusion APIs, and a Chimera attack. Reported results include high API-detection accuracy and performance above public-detector results under Chimera.

APPENDIX B STABLE DIFFUSION 3.5 LORA MODELS

  • The Stable Diffusion 3.5 LoRA models are sourced from Hugging Face and used to generate extensions for the generalization evaluation.Table XI provides model links; the passage notes that many models have thousands of downloads.Source: Appendix B · p. 15, Table XI · p. 16

APPENDIX C SUPPLEMENTARY RESULTS: STATE-OF-THE-ART IMAGE

  • The API evaluation tests ViGText on images synthesized by two commercial diffusion APIs described as advanced image-generation systems.The study frames these tests as addressing effectiveness on high-fidelity deepfakes.Source: Appendix C · pp. 15–16

APPENDIX A EXPLANATION GENERATING PROMPT

  • Fig. 11 shows the structured prompt template used to provide images to the VLLM for generating explanations.The appendix identifies the template as the explanation-generation prompt.Source: Appendix A · p. 15, Figure 11 · p. 16

APPENDIX C SUPPLEMENTARY RESULTS: STATE-OF-THE-ART IMAGE

  • 99.49% accuracy on OpenAI-generated images is reported for ViGText.The evaluation uses images generated by OpenAI’s Image-1 API.Source: Table XII · p. 16, Appendix C · p. 16
  • 96.98% accuracy on Gemini-generated images is reported for ViGText.The evaluation uses images generated by Google’s Gemini 2.5 API.Source: Table XII · p. 16, Appendix C · p. 16
  • 82.08% accuracy under the Chimera attack exceeds the 58.5–69.0% range reported for leading public detectors evaluated in the Chimera study.The attack combines physical recapture and deepfake manipulation.Source: Table XIII · p. 16, Appendix C · p. 16

APPENDIX D SUPPLEMENTARY RESULTS: SAMPLE IMAGES

ViGText is reported to retain strong StyleCLIP performance across tested resolutions and image alterations. The appendix also shows SD samples that ViGText classifies correctly when all other baselines fail.

APPENDIX D SUPPLEMENTARY RESULTS: SAMPLE IMAGES

  • Fig. 12 presents SD fake samples in its first row and real samples in its second, all correctly classified only by ViGText.All other baseline methods misclassify the displayed samples.Source: Figure 12 · p. 16, Appendix D · p. 16

OPERATIONS ON STYLECLIP DATASET

Related papers

Loading 2507.18031v2…