PaperlayerView exact revision on arXiv

Source-linked AI summary

Defake-o3: From Speculative Rationales to Verifiable Evidence for Explainable AIGI Detection

Bowen Deng, Jiahui Zhan, Yikun Ji, Haozhen Yan, Jianfu Zhang

arXiv:2608.16259v1cs.CVcs.AI

TL;DR

Existing explainable AIGI detectors often produce speculative, weakly grounded rationales and miss subtle localized artifacts. Defake-o3 uses interactive visual search and verifier-guided evidence alignment, achieving the best overall performance among compared methods with stronger judge-based evidence quality.

  • Problem

    Existing MLLM-based AIGI detectors often provide speculative rationales lacking precise, visually verifiable evidence for subtle localized flaws.

  • Method

    Defake-o3 combines iterative zoom-in visual search with verifier-guided evidence alignment to favor grounded artifacts and penalize baseless claims.

  • Results

    Defake-o3 achieves the best overall performance among compared methods, with top or tied-for-top classification results and stronger judge-based evidence quality.

  • Takeaways & Limitations

    Defake-o3 produces more localized, verifiable, and persuasive evidence for explainable AIGI detection across in-distribution and out-of-distribution evaluations.

  • Takeaways & Limitations

    Candidate trajectories and localization proposals from Gemini 3 Pro Preview include weakly grounded or hallucinated evidence, requiring subsequent human verification.

Abstract

from arXiv · show

The rapid progress of image generation models calls for AI-generated image (AIGI) detectors that are not only accurate but also explainable and reliable. While MLLM-based detectors can provide natural language explanations, existing methods often generate speculative rationales: they rely on vague or hallucinated artifacts, miss subtle localized flaws from the latest generators, and fail to provide evidence that can be visually verified. We present Defake-o3, an explainable AIGI detector that moves from speculative rationales to verifiable evidence. It combines interactive visual search with verifier-guided evidence alignment: the model iteratively zooms into suspicious regions to inspect fine-grained details, while an Evidence Verifier, trained from human verification annotations, provides reinforcement learning rewards that favor grounded evidence and penalize baseless claims. To support this objective, we construct GroundFake, a dataset designed for grounded explainable detection, with localized bounding-box evidence, human verification based on visual grounding and artifact specificity, corrected reasoning trajectories, and valid/invalid evidence supervision. We further introduce FakeFrontier, an out-of-distribution benchmark built from real images and outputs of 10 recent generators, together with an MLLM-based protocol for evaluating evidence quality and persuasiveness. Experiments on GroundFake, FakeFrontier, and additional out-of-distribution benchmarks show that Defake-o3 improves both detection accuracy and explanation quality, producing more localized, verifiable, and persuasive evidence.

1 Introduction

Photorealistic image generation increases the risk of convincing misinformation, while existing MLLM-based AIGI explanations are often speculative, weakly grounded, poorly localized, and difficult to verify. Defake-o3 addresses this gap by combining interactive visual search with verifier-guided evidence alignment to produce verifiable evidence.

  • Motivation: Photorealistic synthetic images are increasingly difficult to distinguish from authentic photographs, creating risks for public trust through visually convincing misinformation.The introduction motivates robust and trustworthy AIGI detection as image-generation models advance.
  • Problem: Existing MLLM-based detectors produce natural-language rationales that often rely on vague, generic, or hallucinated cues rather than precise visual artifacts.These speculative explanations can appear plausible while remaining weakly grounded, poorly localized, and difficult to verify.
  • Method: Defake-o3 combines interactive visual search with verifier-guided evidence alignment to move from speculative rationales to verifiable evidence.Its iterative thinking-with-images mechanism uses zoom-in inspection to crop and magnify suspicious regions, while the Evidence Verifier guides reinforcement learning toward grounded evidence and away from baseless claims.
  • Contributions: Defake-o3’s contributions include a detector that reveals subtle artifacts through iterative zoom-in inspection and uses Evidence Verifier guidance to favor grounded evidence.The contribution description explicitly frames the verifier as steering reinforcement learning away from baseless claims.

2 Related Work

Conventional AIGI detectors achieve strong classification performance but generally remain black-box systems that provide only authenticity scores. Their lack of interpretable, localized, and visually verifiable evidence motivates explainable detection approaches using MLLMs.

  • Conventional Black-Box AIGI Detection: Conventional AIGI detection primarily formulates the task as binary classification, with methods including CNNSpot, DIRE, DRCT, and NPR establishing strong classification baselines.CNNSpot improves cross-model generalization through augmentation, while DIRE and DRCT use diffusion-model reconstruction discrepancies and NPR uses local pixel statistics.
  • Explainable Detection via MLLMs: These detectors typically output only authenticity scores, revealing neither the visual cues behind predictions nor localized evidence that humans can visually verify.Their decisions are therefore difficult to interpret and verify, motivating recent explainable AIGI detection efforts based on MLLMs.

3 GroundFake Dataset Construction

GroundFake is a 16,000-image, bias-controlled dataset constructed with candidate reasoning and localized evidence, followed by human verification and trajectory rewriting. It provides rewritten trajectories for supervised fine-tuning and valid/invalid evidence labels for Evidence Verifier training.

  • Dataset overview: GroundFake contains 16,000 images, stepwise reasoning transcripts, and evidence annotations for grounded explainable AIGI detection.The dataset supports both supervised fine-tuning and Evidence Verifier training.
  • Image collection: The balanced, bias-controlled pool contains 8,000 real images from Open Images V7 and 8,000 fake images from recent generators, including FLUX.1-dev.File format, aspect-ratio, semantic-category, and aesthetic-quality distributions are aligned across subsets to reduce shortcuts.
  • Candidate annotation: Gemini 3 Pro Preview generates label-conditioned reasoning trajectories, bounding-box localization proposals, verdicts, and supporting evidence through region-by-region inspection.Outputs include global scene-level evidence for both real and fake images, while fake images additionally receive localized key evidence.
  • Human verification: Human verification evaluates localized fake-image evidence for visual grounding and artifact specificity, filtering weakly grounded, invalid, or generic claims.The retained vote ratio from three annotators becomes a soft validity label, and valid evidence is used for trajectory rewriting.
  • Trajectory rewriting: Gemini 3 Flash Preview rewrites reasoning trajectories using the original transcript and verified valid evidence, removing dependence on discarded hallucinated claims.This preserves consistency between reasoning transcripts and the evidence annotations used for training.

4 Defake-o3 Methodology

Defake-o3 combines interactive visual search with verifier-guided evidence alignment to produce structured verdicts containing global and, when available, localized evidence. It trains tool use and evidence generation with GroundFake traces, then uses verifier-informed reinforcement learning to favor concise, grounded evidence and penalize unsupported claims.

  • Interactive Visual Search: Defake-o3 iteratively zooms into suspicious regions before producing a structured verdict with global evidence and optional localized key evidence.At each turn, the model chooses either Zoom In, predicting a 2D bounding box and receiving its crop, or Final Output.
  • Training: Training first uses supervised fine-tuning on human-filtered, trajectory-corrected GroundFake traces to learn zoom-tool use, sequential patch conditioning, and structured output generation.The model is trained with autoregressive next-token prediction over complete multi-turn interaction traces.
  • Evidence Verification: The Evidence Verifier learns graded human judgments of localized evidence validity, providing a denser signal than exact annotation matching and penalizing fabricated evidence.It evaluates the full image, evidence crop, predicted box, and evidence text under the human verification protocol.
  • Reinforcement Learning: The reinforcement-learning objective combines verdict correctness with localized evidence quality, strongly penalizing unsupported evidence and avoiding rewards for numerous weak evidence items.Starting from SFT weights, GRPO optimizes multi-round rollouts to use zooming judiciously and favor a small number of high-quality localized evidence items.
  • Evidence Reward: The evidence reward hybridizes rule-based matching against reference boxes and texts with verifier-based validity scoring, using α_M = 0.5 by default.Set-level matching accommodates incomplete annotations and multiple valid evidence decompositions, while the verifier term supplies human-aligned validity assessment.
  • Evidence Reward: Verifier scores below the acceptance threshold receive a fixed penalty, while scores above it receive bounded positive reward; the default threshold is τ = 0.5.The aggregation further uses cumulative penalties and capped positive rewards, with β = 0.5 to prefer a few strong artifacts over many weak ones.

5 Experiments

Experiments show that Defake-o3 improves explainable detection on GroundFake, generalizes to FakeFrontier and external OoD benchmarks, and produces more grounded evidence. Ablations and qualitative analysis attribute these gains to interactive visual search and verifier-guided evidence selection.

  • GroundFake Results: Defake-o3 matches the best Acc/F1 on GroundFake while achieving the highest BLEU-1, BLEU-2, ROUGE-L, and IoU.RL consistently improves Acc/F1 and IoU across all three variants.
  • FakeFrontier Evaluation: Defake-o3 achieves the highest Acc/F1 on FakeFrontier and ranks first in QS, Hit@Img, and Hit@Evi under all three judges.Baseline methods produce more evidence items on average, but their lower Hit@Evi indicates weaker or less persuasive claims.
  • External OoD Evaluation: Defake-o3 achieves the best accuracy on all three external OoD datasets: 0.9180 on AIGI-Now, 0.9872 on EvalGEN, and 0.8871 on MNW.Samples from generators overlapping with GroundFake training data are removed to maintain a strict OoD setting.
  • Ablation Studies: Interactive visual search gives Defake-o3 advantages over Defake-CoT and Defake-Direct in classification performance and explanation quality.Defake-CoT lacks zoom-in, whereas Defake-Direct lacks any reasoning process.
  • Qualitative Analysis: Defake-o3 zooms into candidate regions, retains verifiable image-specific artifacts, and discards unsupported suspicions, unlike the coarser evidence of FakeVLM and FakeShield.Examples include a malformed headset logo, invalid timestamp, and implausible desk object.

6 Conclusion

Defake-o3 advances explainable AIGI detection from speculative rationales to verifiable evidence using interactive visual search and verifier-guided evidence alignment. GroundFake and FakeFrontier support grounded training and out-of-distribution evaluation, while experiments show top or tied-top classification and stronger judge-based evidence quality.

  • 6 Conclusion: Defake-o3 moves from speculative rationales to verifiable evidence through interactive visual search and verifier-guided evidence alignment.The method is designed as an explainable AIGI detector.
  • 6 Conclusion: GroundFake and FakeFrontier provide grounded training and out-of-distribution evaluation, respectively.They are introduced as complementary resources for the proposed detector.
  • 6 Conclusion: Experiments show the best overall performance among compared methods, with top or tied-for-top classification results and stronger judge-based evidence quality.These findings summarize performance across the reported evaluation settings.

Supplementary Material · A Prompts

The supplementary material includes prompt specifications and training-stage diagnostics for Defake-o3 and related detector variants. The prompts cover inference and GroundFake dataset construction, while figures compare SFT validation loss and RL-stage tool usage.

  • Supplementary Material: Figure S1 compares validation loss between Defake-o3 and Defake-CoT during supervised fine-tuning.The comparison concerns the SFT stage.
  • Supplementary Material: Figure S2 reports the average number of tool calls and other metrics during Defake-o3’s reinforcement-learning stage.The figure summarizes RL-stage behavior using tool-call frequency and additional metrics.
  • Supplementary Material: The supplementary material documents prompts used throughout the study and provides their full contents at its end.This overview introduces the prompt appendix before listing inference and GroundFake construction prompts.
  • A Prompts: Inference prompts are specified for Defake-o3, Defake-CoT, and Defake-Direct.Their system prompts are provided in Figures S7, S8, and S9, respectively.
  • A Prompts: A unified user prompt is provided for inference across the detector variants.The unified user prompt appears in Figure S10.
  • A Prompts: The appendix also lists system and user prompts used to instruct Gemini 3 Pro Preview for GroundFake dataset construction.The supplied passage identifies these prompts as part of the GroundFake construction materials.

B Robustness Evaluation

Defake-o3 is evaluated on FakeFrontier under JPEG compression, Gaussian blur, and image resizing, and exhibits strong robustness to these common visual perturbations.

  • B Robustness Evaluation: Defake-o3 exhibits strong robustness on FakeFrontier under JPEG compression, Gaussian blur, and image resizing.The evaluated degradations are JPEG compression at Quality Factor = 70, Gaussian blur with σ = 1, and resizing by ×0.5.

C Discussion on Training Dynamics

Training dynamics show that magnified visual patches ease reasoning during SFT, while RL independently encourages Defake-o3 to use zoom-in tools for localizing generative flaws.

  • SFT stage: Defake-CoT’s evaluation loss was significantly higher than Defake-o3’s during SFT despite identical reasoning text, indicating that missing magnified visual patches complicates reasoning.This comparison is reported in Figure S1.
  • RL training: During RL, zoom-in tool calls initially decreased but gradually increased without an explicit tool-invocation reward, suggesting autonomous recognition of their role in localizing generative flaws.The observed trend is reported in Figure S2.

D Ablation on Rule-based Reward … H.2 MLLM-based Protocol

The supplementary analyses show that verifier-guided rewards and human evidence filtering improve spatial grounding and classification, while Defake-o3 sacrifices inference efficiency for stronger detection and explanations. They also detail GroundFake annotation controls and FakeFrontier construction and evaluation protocols.

  • D Ablation on Rule-based Reward: The Evidence Verifier improves IoU beyond rule-based rewards, and increasing the rule-based IoU weight alone still falls short spatially.Under the rule-based-only setting, increasing λ_iou/λ_bleu improves IoU somewhat, but α_M=0.5 with the Evidence Verifier achieves higher spatial accuracy.
  • E Base Model Performance: The training pipeline substantially improves GroundFake performance over the pre-trained Qwen3-VL-8B-Instruct base model evaluated without fine-tuning.The base model uses the same prompt as Defake-Direct, while Defake-Direct and Defake-o3 provide comparison results in Table S3.
  • F Inference Speed: Defake-o3 has lower inference efficiency than Defake-CoT, Defake-Direct, FakeVLM, LEGION, and FakeShield, reflecting sequential multi-turn image reasoning.The throughput comparison runs on an 8×A100 GPU platform; the efficiency tradeoff accompanies higher detection performance and explanation quality.
  • G.1 Image Collection and Bias Control: GroundFake contains 16k balanced training images and uses bias controls to reduce shortcut differences and expose subtle generative flaws.The set includes 8k real and 8k fake images; an additional 1.5k FLUX.1-dev images with a custom LoRA simulate imperfect photographic conditions such as underexposure and motion blur.
  • G.2 Label-Conditioned Candidate Annotation: GroundFake candidate annotation uses label-conditioned Gemini 3 Pro Preview trajectories and covers recurring synthetic artifacts such as garbled text, distorted components, repetition, and anomalous lighting.Chinese explanatory text was also requested to support annotation by native Chinese-speaking human annotators.
  • G.3 Human Annotation and Verification: Human verification marks evidence valid only when its text matches the bounding-box region and identifies an artifact exclusive to AI-generated images.Each item was independently evaluated by three of six annotators; 74.99% were valid by majority vote and 89.43% reached unanimous consensus.
  • G.4 Reasoning Trajectory Rewriting for Annotation Consistency; G.5 Impact of Human Verification on Classification Accuracy: Invalid evidence is removed and reasoning trajectories are rewritten to maintain logical consistency with human-filtered evidence, aligning explanations with human standards.Gemini 3 Flash Preview rewrites trajectories using the original image, uncorrected Gemini 3 Pro Preview output, and boolean human annotations; filtering also raises IoU and FakeFrontier accuracy by reducing real-image false positives.
  • H.1 Image Sources and Generation Methods; H.2 MLLM-based Protocol: FakeFrontier comprises 2,000 real and 2,000 fake images from 10 recent generators, while its MLLM protocol evaluates quality and persuasion under controlled prompting.Qwen3-VL-235B-A22B-Thinking, Kimi K2.5, and GLM-4.6V use default parameters; persuasion results average scores across three system-prompt sets, and synthetic images match reference aspect ratios using standard resolutions.

H.3 Detailed Results of Persuasion Evaluation · H.4 Parsing Explanatory Text · I Details About Baselines

The supplementary sections detail persuasion evaluation, evidence-item parsing, and baseline training or weight sources. Stricter persuasion prompts require more effective and reasonable evidence, while baseline explanations are parsed into evidence items for evaluation.

  • H.3 Detailed Results of Persuasion Evaluation: Persuading the MLLM to deliver a “fake” verdict becomes progressively harder from Prompt I to Prompt III.Evidence that succeeds under Prompt III’s more stringent conditions is described as significantly more effective and reasonable.
  • H.3 Detailed Results of Persuasion Evaluation: Raw persuasion results are reported separately for each of the three individual prompts in Tables S6, S7, and S8.Table S7 specifically presents results under Prompt II.
  • H.4 Parsing Explanatory Text: Defake-o3 outputs structured JSON with evidence separated into a list, whereas baseline methods often produce unformatted explanatory text.The explanations from baseline methods must therefore be parsed and split before evaluation.
  • H.4 Parsing Explanatory Text: Qwen3-VL-8B-Instruct performs baseline explanation parsing using the prompt detailed in Figure S19.The evaluation is conducted at the granularity of individual evidence items.
  • I Details About Baselines: CNNSpot is trained on GroundFake with Adam, learning rate 1 × 10−4, batch size 64, and 15 epochs.It is described as a standard classifier trained on real and ProGAN-generated images, with preprocessing, postprocessing, and augmentation for cross-generator generalization.
  • I Details About Baselines: NPR is trained on GroundFake with AdamW, learning rate 1 × 10−4, batch size 32, and 20 epochs.The supplied baseline description identifies these as its GroundFake training settings.
  • I Details About Baselines: FakeVLM is a specialized multimodal large language model trained on FakeClue for synthetic-image classification and natural-language artifact explanations.The experiments use its pre-trained weights published on the official GitHub repository.
  • I Details About Baselines: LEGION integrates a global image encoder, an MLLM, and a grounding image encoder.The supplied passage introduces this architectural composition but does not provide further training details.

J More Qualitative Results

Additional GroundFake qualitative results show that Defake-o3 localizes concrete generative flaws and recognizes surreal AI-generated scenes, whereas baseline explanations are often vague, repetitive, and unreliable. The examples illustrate verifiable evidence spanning object structure, lighting consistency, and biologically impossible content.

  • Defake-o3 qualitative results: Defake-o3 precisely localizes specific generative flaws and identifies surreal scenes, including unrecognizable objects and penguins taking a selfie.The comparisons use additional GroundFake test-set examples against FakeVLM and FakeShield.
  • Baseline comparisons: In 5 out of the 6 examples, FakeVLM repeats the same boilerplate explanation instead of pinpointing explicit flaws.The passage characterizes this behavior as unreliable judgment based merely on overall image texture.
  • Concrete visual evidence: Defake-o3 grounds fake-image judgments in localized structural and lighting inconsistencies, such as a floating cage hook, distorted ornament, and mismatched bird illumination.These examples provide concrete evidence rather than only global claims about image appearance.
  • Scene-level evidence: Defake-o3 also uses scene-level evidence, identifying biologically impossible penguins taking a selfie while distinguishing naturally consistent lighting in a real image.The qualitative examples contrast a fake verdict based on surreal content with a real verdict supported by consistent shadows, colors, and textures.
Loading 2608.16259v1…