WildFakeToolBench: Benchmarking AI-Generated Image Detectors with Web-Sourced Image Processing

Motivation

• LLMs can turn public descriptions of photography, editing, and detector evasion into runnable CPU-compatible operations, lowering the expertise needed to test detector robustness.Relevant information appears in forums, software repositories, image-tool sites, and photography tutorials, including sources not explicitly about evasion.Source: §1 · p. 1

Benchmark

• WILDFAKETOOLBENCH provides 156 provenance-linked executable tools across ten processing categories, generated from public sources and run without local GPUs or detector feedback.The WildFake agent retrieves text, extracts and deduplicates methods, and generates code; the benchmark uses shared validity checks.Source: §1 · pp. 1–2

Evaluation

• The evaluation tests six detectors on images from five generators and seven content categories, retaining tool outputs with SSIM of at least 0.75 from five GPT-4o-mini parameter candidates.Tool synthesis and image-specific parameter proposals do not use detector feedback.Source: §1 · p. 1

Results

• 45.54% is the maximum single-tool attack success rate for FAKEVLM, and every evaluated detector is vulnerable to at least one tool; the strongest individual tool exceeds the strongest selected baseline for each detector.The benchmark also evaluates multi-candidate coverage, which measures a worst-case portfolio rather than one-shot evasion.Source: §1 · pp. 1–2

Analysis

• The study also analyzes how public sources yield executable methods and which processing strategies and source types appear among effective candidates.Subset coverage broadens the candidate portfolio but does not establish reliable one-shot evasion.Source: §1 · p. 2

2.1 AI Image Generators

• Diffusion and latent-diffusion models followed lower-fidelity early GANs as dominant approaches, while newer systems improve prompt understanding and motivate evaluation across generator families.The authors include five popular generators because different families may leave distinct color, spectral, or processing artifacts.Source: §2.1 · p. 2

2.2 AI-Generated Image Detectors

• Detector research ranges from neural classifiers and zero-shot methods to vision-language models that analyze artifacts or localize manipulated regions.Traditional detectors are trained on generated-image datasets [21] [26] or generation artifacts [8] [35] [36], while some use generator models for zero-shot classification. FAKEVLM [32] reasons about visual artifacts; SIDA-13B [14] and FAKESHIELD combine classification with localization or segmentation, and ViGText [3] uses graph neural networks. The paper evaluates their robustness against tools derived from in-the-wild knowledge.Source: §2.2 · pp. 2–3

3.1 Threat Model

• The threat model is a non-expert actor using public online knowledge and an LLM to evade a black-box detector with only binary-label feedback.The actor has no assumed forensics expertise or local GPU resources and cannot access the detector’s identity, architecture, parameters, training data, confidence scores, or threshold.Source: §3.1 · p. 3

3.2 Benchmark Collection Framework

• WILDFAKE retrieves and preprocesses public-source knowledge, extracts and deduplicates actionable methods, then synthesizes and validates parameterized Python tools.GPT-4o-mini is used for preprocessing, extraction, deduplication, and tool synthesis; validation checks interface validity, execution, non-identity output, and distortion, with failures revised using software diagnostics rather than detector feedback.Source: §3.2 · p. 3

3.3 The Obtained WildFakeToolBench

• 156 validated, provenance-retaining tools were synthesized from 220 public sources across 142 domains using standard libraries and no local GPU.The tools span ten categories, including color adjustment, resampling, and simulated capture effects. A software simulation of print-and-scan advice uses paper-fibre noise, brightness and color shifts, slight rotation, and blur; it achieved 25.25% ASR against SIDA-13B.Source: §3.3 · p. 3

3.4 Settings of our Benchmark Evaluation

• Evaluation uses 105 curated generated images, six detectors, and six processing baselines, with ASR counted only on images each detector initially labels fake.For each tool-image pair, five parameter settings are proposed before detector queries and outputs with SSIM below 0.75 are discarded. The eight-tool subset analysis is an oracle worst-case coverage measure over evaluated candidates, not one-shot success or a practical selection policy; its pool is performance-selected on this image set.Source: §3.4 · pp. 3–4

4.1 Benchmark Source Analysis

• AI image-tool sites contribute the most tools (58 from 34 sources), while Reddit contributes 20 and photography blogs 19.Other technical sources also yield many tools: X/Twitter and GitHub contribute 11 each, and diffusion-guidance sites contribute eight. Tool yields vary by source type: Medium and tech media exceed 1.5 tools per source, versus 1.0 for editing-software sources and 0.75 for image-forensics sources.Source: §4.1 · pp. 4–5

4.2 Single-Tool Effectiveness

• 25.25% for SIDA-13B to 100% for AIDE is the strongest single-tool ASR range; web-derived tools also outperform selected academic baselines across detectors.More than 10 tools beat the strongest baseline for five detectors; FAKESHIELD has three tools above 40% ASR. Effectiveness varies by detector: print_rescan reaches 25.25% against SIDA-13B and 3.96% against FAKEVLM.Source: Table 1 · p. 5, §4.2 · p. 5

4.3 Results on Combining Multiple Available Tools

• With all eight tools available, coverage reaches 100% for FAKEVLM, COMMFORENSICS, and AIDE, exceeds 95% for FAKESHIELD and GPT-5.6-LUNA, and exceeds 80% for SIDA-13B.Four-tool sets reach at least 52 of 70 sets above 50% ASR for every detector; all 70 do so for FAKESHIELD, GPT-5.6-LUNA, COMMFORENSICS, and AIDE.Source: Figure 3 · p. 5, §4.3 · p. 6
• Best sequential-chain performance peaks at four tools for FAKEVLM and three for SIDA-13B, then declines as chains grow longer.The authors suggest accumulated distortion and interactions between tools may explain the decline; approximately four of five two-tool candidates pass the SSIM threshold, versus approximately two for eight-tool processing.Source: Figure 4 · p. 6, §4.3 · pp. 6–7

4.4 Observed Weaknesses of Current Detectors.

• Strongest single-tool weaknesses vary by detector: FAKEVLM and SIDA-13B favor analog/capture, while FAKESHIELD, GPT-5.6-LUNA, COMMFORENSICS, and AIDE map to distinct processing categories.For FAKEVLM, the strongest four-tool set reaches 92.08% coverage with composition, resampling, de-stylization, and detail recovery, but excludes the strongest single tool’s analog/capture technique.Source: §4.4 · p. 7

4.5 Source-Level Effectiveness

• AI image-tool sites supply the strongest single tools for four detectors; Reddit supplies the strongest single tool for SIDA-13B and appears in the highest-ASR sets for all six.Tech media supplies the strongest single tool for GPT-5.6-LUNA, whereas GitHub and diffusion-guidance sources do not appear in the reported highest-ASR sets.Source: §4.5 · p. 7

4.7 Example of Processed Images

• Over 25% of detectable fake images were classified as real by SIDA-13B after print_rescan processing, despite only slight noise and rotation.The processed images were described as almost the same as the originals, with changes intended to mimic scanning.Source: Figure 6 · p. 8, §4.7 · p. 8

4.6 Engagement and Effectiveness.

• The scale_grain_denoise example reaches 91.67%, and the examples suggest that greater engagement may be related to the effectiveness of extracted tools.Source: §4.6 · p. 8

• WildFakeToolBench comprises 156 CPU-executable image-processing tools derived from 220 public sources across 142 domains, with retained source evidence and validation without detector feedback.Source: §5 · p. 8
• Strongest individual tools achieved 25.25%–100% ASRs, exceeding the strongest of six selected baselines for each detector; an eight-tool pool exceeded 80% candidate coverage for all six detectors.Evaluation used 105 images from five generators.Source: §5 · p. 8
• Tool effectiveness varied by detector, and high candidate coverage did not ensure one-shot evasion.Source: §5 · p. 8
• Web-sourced knowledge can complement researcher-selected robustness evaluations.Source: §5 · p. 8

Ethics Considerations

• WILDFAKE tools can help identify detector weaknesses or make generated images harder to detect, and are presented for authorized defensive research and detector development.Source: Ethics Considerations · p. 8
• The study uses open-web content and controlled benchmark evaluation, with no closed-community access, user data collection, or deployment against real-world moderation systems; these steps reduce but do not eliminate risk.Source: Ethics Considerations · p. 8

A Construction Details

• WILDFAKE’s benchmark construction comprises retrieval, method extraction and deduplication, code synthesis, and validation.Figure 1 presents the benchmark overview; the detailed construction procedure is documented below.Source: A Construction Details · p. 10

A.1 Retrieval and Corpus Processing

• GPT-4o-mini generates queries on detector-evasion advice and photography knowledge; Tavily retrieves up to five results per query, supplemented by Reddit- and X-specific searches.Collection is limited to public web content, which does not itself determine the licensing of retrieved material.Source: §A.1 · p. 10
• Pages under 300 extracted characters are discarded, exact-URL duplicates removed, and domains dominated by videos, marketplace listings, or other non-text content suppressed.Each normalized Post record stores a stable identifier, text, URL, domain, discourse category, originating query, and available retrieval metadata to support provenance and source-level analysis.Source: §A.1 · p. 10

A.2 Method Extraction and Deduplication

• GPT-4o-mini extracts actionable processing ideas with quoted source evidence; deduplication separates functionally distinct operations and retains one representative with source provenance.Duplicate groups are checked for functional similarity, including generated implementations, then overlapping verified groups are merged transitively.Source: §A.2 · p. 10
• Methods need not specify numeric parameters, because synthesis supplies concrete implementations later.Source: §A.2 · p. 10

A.3 Tool Interface and Admission Checks

• GPT-4o-mini converts each retained idea into a Python class with an image-processing method, parameter schema, and usage guidance, using common numeric libraries rather than neural editing models.The synthesis prompt prohibits importing PyTorch.Source: §A.3 · p. 10
• Parameters govern properties such as noise strength, rotation, and blur, while image-specific settings are proposed during evaluation.Source: §A.3 · p. 10
• Admission checks cover syntax, class and parameter-dictionary interfaces, execution on a fixed validation image, non-identity output, and SSIM-based distortion.Source: §A.3 · p. 10
• Failed validation returns code and diagnostics to the LLM for at most three generation attempts per method; detector scores and decisions never guide repair or admission.Validation errors provide software feedback only.Source: §A.3 · p. 10

A.4 From Public Sources to Executable Tools

• 156 tools were admitted from 169 deduplicated method records, about 92%, drawn from 220 sources across 142 domains.Mining labeled 77 sources valuable and 143 discard, extracted 204 method records, and deduplication removed 35. Five of 13 failures produced identity outputs; eight did not yield runnable implementations within three attempts.Source: §A.4 · p. 10
• Tool conversion shows informal advice can often become executable processing, but conversion success does not imply detector evasion.The process also reveals limitations in method extraction and code translation.Source: §A.4 · pp. 10–11

B Tool Category Definitions

• Detail-recovery and artifact-repair tools add edge variation, repair blur, and reduce over-smoothing, while local-texture tools adjust fine surface details such as skin.Source: B Tool Category Definitions · p. 11
• Tool categories target photographic capture realism through simulated analog processing, optical imperfections, sensor noise, and online resizing or re-encoding.Examples include film grain, in-camera JPEG artifacts, lens aberrations, sensor-associated noise patterns, and resampling.Source: B Tool Category Definitions · p. 11
• De-stylisation tools reduce generator-associated smoothing, symmetry, or enhancement, while lighting-realism tools adjust shadows, color temperature, and scene light sources.Source: B Tool Category Definitions · p. 11
• Color and white-balance tools adjust global properties such as saturation, exposure, and tone mapping, while composition tools mimic casual photography through reframing and less deliberate staging.Examples include off-center subjects and unusual aspect ratios.Source: B Tool Category Definitions · p. 11

C.1 Detector Configurations

• FAKEVLM [32] reasons over natural-language artifact clues, while GPT-5.6-LUNA is evaluated by directly prompting it to classify images.FAKEVLM fine-tunes LLaVA-1.5-7B on a collected 100k-image dataset; the GPT-5.6-LUNA prompt is listed in Appendix E.6.Source: §C.1 · p. 11
• SIDA-13B [14] emits classification and segmentation tokens in one pass, while FAKESHIELD uses DTE-FDM for verdicts and MFLM for tamper masks.SIDA’s segmentation ability also helps classification; this evaluation uses only FAKESHIELD’s DTE-FDM module for verdicts.Source: §C.1 · p. 11
• COMMFORENSICS [21] fine-tunes a CLIP-ViT-S backbone as a binary classifier using 2.7M images sampled from 4803 generators.Source: §C.1 · p. 11
• AIDE [35] combines DCT-selected high- and low-frequency patch noise residuals, a global Open-CLIP branch, and an MLP to produce its final score.Dedicated fine-tuned ResNet-50s process the patch residuals; the three pooled feature sets are concatenated.Source: §C.1 · p. 11

C.2 Selected Image-Processing Baselines

• Gaussian or salt-and-pepper noise [12] and practical Poisson-plus-Gaussian noise provide baselines for adding random or sensor-motivated noise.The practical-noise tool combines signal-dependent Poisson noise with additive Gaussian noise to model two dominant camera-sensor noise sources.Source: §C.2 · p. 11
• Spectral suppression [33] removes periodic FFT peaks, while defocus blur [12] applies Gaussian blur with radius scaled to the strength parameter.The spectral method targets peaks treated as fingerprints of generated images; blur mimics optical defocus.Source: §C.2 · p. 11
• Camera processing [4] models an ISP pipeline with white balance, a power-law contrast curve, and Gaussian blur; Reinhard color transfer matches generated-image colors to a real reference.Source: §C.2 · p. 11

D Additional Result Details

• Mean single-tool ASRs are 21.77% for COMMFORENSICS, 23.10% for AIDE, and 21.87% for GPT-5.6-LUNA, indicating broader suite effects than for FAKEVLM or SIDA-13B under this protocol.The passage attributes these mean rates to a broader portion of the tested suite affecting those systems.Source: D Additional Result Details · p. 12
• SIDA-13B’s 18 remaining FAKE classifications span all five generators, while document, face portrait, object, and satellite images have higher ASRs than animal and human images.Failure counts are Midjourney 5, GPT Image 2 4, Qwen-Image 4, Nano Banana 2 3, and Stable Diffusion 2; the authors say source or image quality does not clearly determine failures and suggest greater robustness on animal and human content as one possible explanation.Source: D Additional Result Details · p. 12
• The print_rescan source dates to 2023, while some methods from 2026 sources achieve only approximately 1% ASR, providing no clear relationship between effectiveness and source age.Source: D Additional Result Details · p. 12
• Maximum single-tool ASR exceeds the strongest selected baseline by 38.61, 13.13, 7.82, 32.87, 54.16, and 44.71 percentage points for FAKEVLM, SIDA-13B, FAKESHIELD, GPT-5.6-LUNA, COMMFORENSICS, and AIDE, respectively.Figure 7 reports how many released tools exceed the strongest of the six academic-sourced baseline tools for each detector.Source: Figure 7 · p. 12, D Additional Result Details · p. 12

E.1 Web-Retrieval Queries

• Retrieval queries target both explicit detector-evasion advice and indirect photographic knowledge, including film, sensor, recapture, and JPEG-processing topics.The appendix also says it provides prompts for extraction, duplicate detection, mechanism grouping, tool synthesis, parameter selection, and detector evaluation, with grouping distinct from duplicate detection.Source: §E.1 · p. 12, E Retrieval Queries and Archived Prompts · p. 12

E.2 Method-Mining Prompt

• The method-mining prompt asks the analyzer to retain source evidence and extract actionable ideas and mechanisms, rather than finished parameterized recipes.It classifies ideas as generative, numeric, or hybrid and evaluates novelty, feasibility, and relevance.Source: §E.2 · p. 13
• The supplied passage is a short fragment showing fields named quoted_evidence and source, each with an ellipsis as its value.Source: §E.2 · p. 13

E.3 Deduplication Prompts

• Deduplication groups only methods with essentially the same concrete technique, then organizes the retained methods into mechanism-based archetypes without replacing the specific-method list.Methods are grouped conservatively; each specific method belongs to one archetype, while a genuinely singular method may form a cluster of one.Source: §E.3 · p. 14

E.4 Tool Synthesis and Validation

• Tool synthesis turns each retained method into a runnable image-processing tool, with generated tools tested and revised for up to three attempts on a fixed textured image.The validation image is a 24×24 random field upsampled to 512×512 with σ = 8 Gaussian grain; failed tools are returned to the LLM with the failure reason.Source: §E.4 · pp. 14–15

E.5 Tool-Parameter Selection Prompt

• Parameter selection proposes five diverging, image-specific settings from very light to very strong, while requiring edits to remain plausible and preserve the appearance of an undamaged photo.Candidates that visibly destroy image quality are discarded during evaluation, regardless of whether they fool the detector.Source: §E.5 · p. 15

E.6 GPT-Based Detector Prompt

• The GPT-based detector prompt requests a real/fake judgment and probability, instructing the model to judge each image on its own evidence without assuming the pool’s class proportions.It treats false positives and false negatives as equally costly and calibrates the full 0–100 range to evidence strength.Source: §E.6 · p. 15
• The prompt defines p_fake as the probability that an image is AI-generated or AI-manipulated, requires a FAKE/REAL verdict and an explanation of at most eight words, and sets scores of at least 50 as FAKE.The verdict must agree with the probability threshold.Source: §E.6 · pp. 15–16