PaperlayerView exact revision on arXiv

Source-linked AI summary

Defake-o3: From Speculative Rationales to Verifiable Evidence for Explainable AIGI Detection

Bowen Deng, Jiahui Zhan, Yikun Ji, Haozhen Yan, Jianfu Zhang

arXiv:2608.16259v1cs.CVcs.AI

TL;DR

MLLM-based AIGI detectors can provide natural-language rationales, but those explanations are often speculative and lack verifiable evidence. Defake-o3 uses interactive visual search and verifier-guided evidence alignment, and reports top or tied classification results with stronger judge-based evidence quality among compared methods.

  • Problem

    Existing MLLM-based AIGI detectors often produce speculative rationales without verifiable evidence and use coarse, single-pass inspection that can miss subtle localized flaws.

    Source: §1 · p. 1

  • Method

    Defake-o3 combines iterative zoom-in visual search with verifier-guided evidence alignment, supported by GroundFake’s human-verified evidence and reasoning supervision.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Defake-o3 reports the best overall performance among compared methods, with top or tied classification results and stronger judge-based evidence quality.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The reported findings support verifiable evidence as a target for explainable AIGI detection in the evaluated benchmarks.

    Source: §6 · p. 8

    A non-trivial portion of Gemini 3 Pro Preview’s candidate localized evidence was weakly grounded or hallucinated, requiring human verification before use for trajectory rewriting or verifier training.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

The rapid progress of image generation models calls for AI-generated image (AIGI) detectors that are not only accurate but also explainable and reliable. While MLLM-based detectors can provide natural language explanations, existing methods often generate speculative rationales: they rely on vague or hallucinated artifacts, miss subtle localized flaws from the latest generators, and fail to provide evidence that can be visually verified. We present Defake-o3, an explainable AIGI detector that moves from speculative rationales to verifiable evidence. It combines interactive visual search with verifier-guided evidence alignment: the model iteratively zooms into suspicious regions to inspect fine-grained details, while an Evidence Verifier, trained from human verification annotations, provides reinforcement learning rewards that favor grounded evidence and penalize baseless claims. To support this objective, we construct GroundFake, a dataset designed for grounded explainable detection, with localized bounding-box evidence, human verification based on visual grounding and artifact specificity, corrected reasoning trajectories, and valid/invalid evidence supervision. We further introduce FakeFrontier, an out-of-distribution benchmark built from real images and outputs of 10 recent generators, together with an MLLM-based protocol for evaluating evidence quality and persuasiveness. Experiments on GroundFake, FakeFrontier, and additional out-of-distribution benchmarks show that Defake-o3 improves both detection accuracy and explanation quality, producing more localized, verifiable, and persuasive evidence.

1 Introduction

Defake-o3 addresses the lack of verifiable evidence in AIGI detection by combining iterative visual search with verifier-guided evidence alignment, supported by a grounded training dataset and an out-of-distribution benchmark.

1 Introduction

  • Traditional detectors [12] [52] [58] [64] often provide only authenticity scores, while MLLM rationales [26] [33] [59] [63] [65] can be speculative, weakly grounded, and poorly localized.Single-pass inspection can miss subtle localized flaws, and absent explicit validity criteria, vague or hallucinated cues are difficult to verify.Source: §1 · pp. 1–2
  • Defake-o3 combines iterative zoom-in inspection with verifier-guided reinforcement learning to find subtle artifacts and favor grounded evidence over baseless claims.Its Evidence Verifier is trained from human-verified evidence annotations and supplies rewards during GRPO-based reinforcement learning.Source: §1 · p. 2
  • GroundFake provides debiased training data with localized, human-verified evidence, corrected reasoning trajectories, and valid/invalid evidence labels for fine-tuning and verifier training.It controls image format, aspect ratio, semantic category, and aesthetic quality; verification applies visual-grounding and artifact-specificity criteria.Source: §1 · p. 2
  • FakeFrontier contains 2,000 real and 2,000 synthetic images from 10 recent generators and pairs the benchmark with an MLLM-based protocol for detection and evidence evaluation.The protocol evaluates evidence quality and persuasiveness alongside classification performance.Source: §1 · p. 2

2 Related Work

Related work progresses from score-only AIGI classifiers to MLLM explanations and interactive visual search, but still lacks reliable evidence-level verification of grounded, artifact-specific claims.

Conventional Black-Box AIGI Detection

  • CNNSpot, DIRE [58], DRCT, NPR [52], and AIDE [64] establish classification baselines, but typically provide scores without localized, verifiable visual evidence.Detector generalization can also be sensitive to distributional bias between real and synthetic data [12].Source: §2 · p. 2

Explainable Detection via MLLMs

  • FakeVLM [59] and FakeScope [34] generate explanations, while LEGION [26], FakeShield [63], So-Fake [23], and FakeXplain add localization or box grounding.Despite DPO [44] and RL-style alignment, these methods mainly match reference annotations rather than verify that evidence is visually grounded and artifact-specific; outputs may remain vague or hallucinated, and evidence-level supervision is limited.Source: §2 · p. 2

Dynamic Visual Reasoning and Exploration

  • Visual CoT [49] and LLaVA-CoT make reasoning explicit, while V* [61], Visual-RFT, Pixel Reasoner [56], DeepEyes [69], and Mini-o3 [30] enable interactive visual search.These methods improve where models look, but search alone does not ensure that evidence is valid, artifact-specific, and visually verifiable for an AIGI verdict.Source: §2 · p. 2

3 GroundFake Dataset Construction

GroundFake is a 16,000-image dataset for grounded explainable AIGI detection, pairing reasoning transcripts with evidence annotations and human-verified supervision.

3 GroundFake Dataset Construction

  • GroundFake contains 16,000 images, stepwise reasoning transcripts, and evidence annotations; rewritten trajectories support SFT, while valid/invalid evidence labels train the Evidence Verifier.Source: §3 · pp. 2–3

3.1 Image Collection and Bias Control

  • The balanced, bias-controlled pool contains 16k images (8k real and 8k fake), with real images from Open Images V7 and fakes from recent generators plus FLUX.1-dev.File format is standardized, and aspect-ratio, coarse semantic-category, and aesthetic-quality distributions are aligned between real and fake subsets to reduce dataset-specific shortcuts.Source: §3.1 · p. 3

3.2 Label-Conditioned Candidate Annotation

  • Gemini 3 Pro Preview generates label-conditioned reasoning trajectories, verdicts, global evidence, and fake-image localized evidence proposals, but some localized claims are weakly grounded or hallucinated.The model is prompted to inspect regions and return structured reasoning, conclusions, and supporting evidence; the ground-truth label is used only to elicit class-consistent proposals.Source: §3.2 · p. 3

3.3 Human Annotation and Verification

  • Three annotators independently label each fake-image localized evidence item Valid or Invalid for visual grounding and artifact specificity; vote ratios supervise the verifier, and retained valid evidence feeds trajectory rewriting.Source: §3.3 · p. 4

3.4 Reasoning Trajectory Rewriting for Annotation Consistency

  • Gemini 3 Flash Preview rewrites trajectories using the original reasoning and human-verified valid evidence, removing dependence on discarded claims and producing annotation-consistent training transcripts.Source: §3.4 · p. 4

4 Defake-o3 Methodology

Defake-o3 combines iterative visual search with verifier-guided evidence alignment, training first on corrected GroundFake traces and then with reinforcement learning to produce grounded, structured detection outputs.

4.1 Task Formulation

  • The model repeatedly zooms into suspicious image regions before returning a real/fake verdict, mandatory global evidence, and optional localized evidence only when artifacts are visually verifiable.Real verdicts have no localized evidence; fake verdicts may also omit it when no clear local artifact is found, avoiding hallucinated localization.Source: §4.1 · p. 4

4.2 Supervised Fine-Tuning (SFT)

  • Supervised fine-tuning on human-filtered, trajectory-corrected GroundFake traces teaches tool use, sequential use of zoomed patches, and structured evidence generation.The model is trained with autoregressive next-token prediction over the full interaction trace; evidence quality is aligned in the subsequent reinforcement-learning stage.Source: §4.2 · p. 4

4.3 Evidence Verifier Training

  • The Evidence Verifier predicts localized-evidence validity from the image, crop, box, and text, using soft-label binary classification trained on annotator agreement.Human judgments assess visual grounding and artifact specificity; the learned score supplies a graded validity signal for reinforcement learning.Source: §4.3 · p. 4

4.4 Reward Function Design

  • The reinforcement-learning reward combines verdict correctness with rule-based evidence matching and verifier-based validity, penalizing weak evidence while favoring a small number of strong items.Set-level matching uses spatial overlap and a lexical anchor; verifier scores below the acceptance threshold incur a fixed penalty, while positive rewards are bounded and cumulatively penalized.Source: §4.4 · pp. 5–6

4.5 Reinforcement Learning

  • Starting from SFT weights, GRPO [50] [67] optimizes multi-round zoom-and-output rollouts so the model uses visual search judiciously and favors high-quality localized evidence.The training objective is trajectory-level and discourages many weak or vague evidence items.Source: §4.5 · p. 6

5 Experiments

Defake-o3 shows strong classification and evidence-quality results across in-domain, recent-generator, and external out-of-distribution evaluations. Ablations and qualitative examples attribute its evidence gains to interactive visual search and verifier-guided alignment.

5.1 Experimental Setup

  • Experiments compare Defake-o3 with retrained CNNSpot, NPR, and AIDE and with FakeVLM, LEGION, and FakeShield; Defake-CoT and Defake-Direct serve as ablation variants.FakeVLM and FakeShield use released weights, while LEGION is reproduced with official code and data.Source: §5.1 · p. 6

5.2 Results on GroundFake Test Set

  • On GroundFake, RL improves Acc/F1 and IoU for all three variants, while Defake-o3 matches the best Acc/F1 and leads in BLEU-1, BLEU-2, ROUGE-L, and IoU.The reported measures cover classification, evidence-text overlap, and evidence-region overlap.Source: §5.2 · p. 6

5.6 Qualitative Results

  • Qualitative examples show Defake-o3 zooming into candidate regions and retaining localized, verifiable artifacts while discarding unsupported suspicions, unlike the coarser evidence from FakeVLM and FakeShield.Examples include a malformed headset logo, an invalid timestamp, and an implausible desk object.Source: Figure 4 · p. 7, §5.6 · p. 8

5.3 FakeFrontier Benchmark

  • On FakeFrontier, Defake-o3 achieves the highest Acc/F1 and ranks first in QS, Hit@Img, and Hit@Evi under all three judges.FakeFrontier contains 4,000 real and fake images, including outputs from 10 recent generators; the explanation protocol assesses evidence quality and persuasiveness without reference annotations.Source: §5.3 · pp. 6–8, Table 3 · p. 7, Table 2 · p. 7

5.4 Results on External OoD Benchmarks

  • Defake-o3 achieves the best accuracy on all three external OoD benchmarks: 0.9180 on AIGI-Now, 0.9872 on EvalGEN, and 0.8871 on MNW.Evaluation removes samples from generators overlapping with GroundFake training and reports classification accuracy only.Source: Table 4 · p. 7, §5.4 · p. 8

5.5 Ablation Studies

  • Defake-o3 outperforms Defake-CoT and Defake-Direct in classification and explanation quality, supporting interactive visual search over text-only reasoning or direct prediction.Defake-CoT omits zoom-in tool calls, whereas Defake-Direct omits preliminary reasoning.Source: §5.1 · p. 6, §5.5 · p. 8
  • Verifier guidance raises GroundFake IoU from 0.252 to 0.311 versus rule-only training, while α_M=0.5 provides the best overall trade-off between verifier guidance and rule-based matching.Verifier scores are higher for human-accepted evidence than for human-rejected or fabricated evidence; verifier-only training degrades text-explanation performance, while α_M=0.25 noticeably degrades Acc/F1.Source: Table 5 · p. 8, Figure 5 · p. 8, §5.5 · p. 8

6 Conclusion

Defake-o3 shifts explainable AIGI detection from speculative rationales toward verifiable evidence through interactive visual search and verifier-guided alignment. GroundFake and FakeFrontier support grounded training and out-of-distribution evaluation.

6 Conclusion

  • Defake-o3 achieves the best overall performance among compared methods, with top or tied-for-top classification results and stronger judge-based evidence quality.The paper introduces Defake-o3 alongside GroundFake for grounded training and FakeFrontier for out-of-distribution evaluation.Source: §6 · p. 8

Supplementary Material · A Prompts

The supplementary material reports validation-loss and RL tool-call metrics, and catalogs prompts used for inference, GroundFake construction, and FakeFrontier evaluation.

Supplementary Material

  • Figure S1 compares Defake-o3 and Defake-CoT validation loss during the SFT stage.Source: Figure S1 · p. 11
  • Figure S2 presents average tool calls and other metrics for Defake-o3 during the RL stage.Source: Figure S2 · p. 11

A Prompts

  • Inference prompts include system prompts for Defake-o3, Defake-CoT, and Defake-Direct, plus a unified user prompt.These are provided in Figures S7–S10, respectively.Source: A Prompts · p. 11
  • GroundFake construction prompts cover Gemini 3 Pro Preview evidence-candidate generation and trajectory rewriting.The system and user prompts for candidate generation are in Figures S11–S12; trajectory rewriting is in Figure S13.Source: A Prompts · p. 11
  • FakeFrontier prompts cover extracting image-generation descriptions and evaluating evidence quality.These prompts are provided in Figures S14–S15.Source: A Prompts · p. 11
  • FakeFrontier also uses three system prompts for persuasion evaluation and a prompt for parsing baseline explanations.These are provided in Figures S16–S19.Source: A Prompts · p. 11

B Robustness Evaluation

Defake-o3 shows strong robustness on FakeFrontier under JPEG compression, Gaussian blur, and resizing.

B Robustness Evaluation

  • On FakeFrontier, Defake-o3 exhibits strong robustness to JPEG compression (Quality Factor = 70), Gaussian blur (𝜎= 1), and resizing (×0.5).Source: B Robustness Evaluation · p. 11

C Discussion on Training Dynamics

Training dynamics show that magnified visual patches make reasoning easier, while Defake-o3 increasingly invokes zoom-in tools during RL despite receiving no explicit tool-use reward.

C Discussion on Training Dynamics

  • Defake-o3 had significantly lower SFT evaluation loss than Defake-CoT despite identical reasoning text, indicating magnified visual patches ease reasoning.The comparison is reported in Figure S1.Source: C Discussion on Training Dynamics · p. 11
  • During RL, zoom-in tool calls rose after an initial decrease without an explicit invocation reward, suggesting Defake-o3 overcame tool-use inertia [69].The trend is shown in Figure S2 and suggests the model recognized zoom-in tools’ role in localizing generative flaws.Source: C Discussion on Training Dynamics · p. 11

D Ablation on Rule-based Reward … H.2 MLLM-based Protocol

The supplementary sections test reward and training choices, document GroundFake’s data and annotation pipeline, and specify FakeFrontier construction and its MLLM-based evaluation protocol. They report that verifier-guided and human-filtered evidence improves spatial grounding and reduces false positives, while Defake-o3 trades inference speed for stronger detection and explanation quality.

D Ablation on Rule-based Reward

  • Evidence Verifier use improves GroundFake IoU over rule-only reward despite slightly lower text metrics; raising rule-only IoU weight helps but remains less spatially accurate.The comparison uses verifier weight α_M=0.5 versus rule-only α_M=0; no exact metric values are provided in the passages.Source: D Ablation on Rule-based Reward · p. 11, Table S2 · p. 12

E Base Model Performance

  • Training substantially improves GroundFake performance over the untuned Qwen3-VL-8B-Instruct base model.The base model is evaluated without fine-tuning using the Defake-Direct prompt, and Table S3 also includes Defake-Direct and Defake-o3 for comparison.Source: E Base Model Performance · p. 11, Table S3 · p. 12

F Inference Speed

  • Defake-o3 trades inference efficiency for higher detection performance and explanation quality compared with the listed methods.The comparison covers Defake-o3, Defake-CoT, Defake-Direct, FakeVLM, LEGION, and FakeShield on an 8×A100 platform; sequential multi-turn image reasoning is cited as the main cause of lower throughput.Source: F Inference Speed · p. 11, Table S4 · p. 12

G.1 Image Collection and Bias Control

  • GroundFake uses 16k balanced real and generated images, with source, aspect-ratio, category, and aesthetic controls intended to reduce shortcuts.The 8k fake images combine 6.5k scraped generations and 1.5k FLUX.1-dev images made with a LoRA and real-image captions; the 8k real images come from OpenImageV7. Images are JPG, categories are balanced, and aesthetic distributions are aligned.Source: §G.1 · p. 12

G.2 Label-Conditioned Candidate Annotation; G.4 Reasoning Trajectory Rewriting for Annotation Consistency

  • Candidate evidence and reasoning are generated with label-conditioned prompts, then trajectories are rewritten to remain consistent with evidence retained after human filtering.Gemini 3 Pro Preview generates candidates using ground-truth labels and a prompt listing common generative errors; Gemini 3 Flash Preview rewrites trajectories using the original output and human annotations.Source: §G.2 · p. 12, §G.4 · p. 14

G.3 Human Annotation and Verification

  • Annotators marked evidence valid only when it matched the boxed image region and identified an AI-exclusive error; majority voting marked 74.99% of candidates valid, and all three annotators unanimously agreed on 89.43% of evidence items.Six annotators participated, with each evidence item independently evaluated by three of them.Source: §G.3 · pp. 13–14

G.5 Impact of Human Verification on Classification Accuracy

  • Removing human-annotated invalid evidence during SFT raises GroundFake IoU and FakeFrontier accuracy, with fewer false positives on real images.The reported accuracy gain on FakeFrontier is attributed to reduced misclassification of real images.Source: Table S5 · p. 14, §G.5 · p. 14

H.1 Image Sources and Generation Methods; H.2 MLLM-based Protocol

  • FakeFrontier contains 2,000 real and 2,000 generated images, and explanation quality and persuasion are evaluated with multiple MLLMs and prompt conditions.The fake set spans 10 generators, with caption-based prompts preserving photographic styles. Persuasion scores average three prompts: one adds common generation errors, while another warns evaluators that evidence may mislead them.Source: §H.1 · p. 14, §H.2 · p. 14

H.3 Detailed Results of Persuasion Evaluation · H.4 Parsing Explanatory Text · I Details About Baselines

The supplement reports that Defake-o3 retains an advantage across persuasion prompts, describes how baseline explanations are parsed for evaluation, and documents baseline methods and training or weight sources.

H.3 Detailed Results of Persuasion Evaluation

  • Persuasion difficulty increases from Prompt I to Prompt III, and Defake-o3 maintains a clear advantage over baselines across prompt configurations.The authors interpret evidence persuasive under Prompt III as more effective and reasonable; Tables S7 and S8 report results for Prompts II and III.Source: §H.3 · p. 14, Table S8 · p. 15, Table S7 · p. 15

H.4 Parsing Explanatory Text

  • For explanation-quality evaluation, baseline text outputs are parsed into individual evidence items using Qwen3-VL-8B-Instruct and the prompt in Figure S19.Defake-o3 already outputs structured JSON with evidence in a list, whereas many baselines produce unformatted text.Source: §H.4 · p. 15

I Details About Baselines

  • CNNSpot detects fakes using a real/ProGAN-trained classifier with generalization-oriented preprocessing and augmentation, while NPR [52] uses neighboring-pixel relationships as a source-invariant representation.Both were trained on GroundFake: CNNSpot used Adam, learning rate 1 × 10−4, batch size 64, and 15 epochs; NPR used Adam, 2 × 10−4, batch size 32, and 10 epochs.Source: I Details About Baselines · p. 15
  • AIDE [64] combines CLIP global semantics with DCT-guided frequency patch sampling and SRM noise features for hybrid-cue detection.It was trained on GroundFake with AdamW, learning rate 1 × 10−4, batch size 32, and 20 epochs.Source: I Details About Baselines · p. 15
  • FakeVLM [59] is a FakeClue-trained multimodal model for synthetic-image classification and natural-language artifact explanations; the experiments used its published pretrained weights.Source: I Details About Baselines · p. 15
  • LEGION [26] combines image encoders, an MLLM, and a pixel decoder for detection, artifact localization, and textual explanation; it was reproduced on SynthScars with official scripts and defaults.Source: I Details About Baselines · p. 15
  • FakeShield [63] combines domain-tag-guided forgery detection with multimodal localization and language explanations; the experiments used its published pretrained weights.Source: I Details About Baselines · p. 15

J More Qualitative Results

Additional GroundFake examples show Defake-o3 identifying localized generative flaws and surreal scenes, while FakeVLM and FakeShield often provide vague or unreliable explanations.

J More Qualitative Results

  • Across Figures S4–S6, Defake-o3 localizes specific generative flaws and recognizes surreal scenes, including two penguins taking a smartphone selfie.The examples include unrecognizable objects and an obviously AI-generated scene.Source: J More Qualitative Results · p. 15, Figure S4 · p. 16, Figure S5 · p. 17
  • FakeVLM repeats the same boilerplate explanation in 5 of 6 examples, while FakeShield defaults to lighting and shadow claims and mistakes a smartphone for a mirror.The authors attribute FakeVLM’s unreliability to judging overall texture rather than pinpointing explicit flaws, and suggest FakeShield’s fine-tuning may have compromised its world knowledge.Source: J More Qualitative Results · p. 15
  • Examples include a nonsensical floating cage hook and a distorted, structureless object attached to the cage.Figure S4’s bounding boxes are outputs from Defake-o3.Source: Figure S4 · p. 16, J More Qualitative Results · p. 17

Related papers

Loading 2608.16259v1…