PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Neural Defend ARCAS 1B: A Foundational Multimodal Deepfake Detection Model

Sivashankar Selvarajan, Piyush Verma, Sumit Kumar, Sharayu N. Deshmukh

arXiv:2609.25154v1cs.CRcs.AI

TL;DR

Detector scores are difficult to interpret across changing generators and unlike benchmark populations, metrics, and aggregation rules. The paper evaluates supplied ARCAS 1B scores across seven benchmark families, retaining native outcomes alongside pooled and subgroup analyses; strong aggregate results coexist with benchmark-specific error patterns.

  • Problem

    Changing generators and differing benchmark populations, metrics, and aggregation rules make a single detector score an incomplete account of generalization.

    Source: §1 · p. 1

  • Method

    The study evaluates supplied ARCAS 1B scores across seven benchmark families, preserving benchmark-native metrics while adding pooled record-level, coverage, and subgroup analyses.

    Source: §1 · p. 2

  • Results

    Across the reported analyses, strong aggregate performance coexists with benchmark-specific errors: AIGCDetectBenchmark errors are chiefly false positives, while OpenFake transfer costs include missed generated images.

    Source: §4.3 · p. 6, §4.6 · p. 8

  • Takeaways & Limitations

    Keeping native outcomes distinct from pooled summaries makes population, class-balance, and missing-record-coverage differences visible for interpreting detector results.

    Source: §1 · p. 2

    The evidence characterizes supplied scores on evaluated records; detector properties are author-attested, and the study does not identify the causes of observed performance.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

AI-generated imagery evolves faster than benchmark-specific detector evaluations, making a single score an incomplete account of generalization. This paper evaluates Neural Defend ARCAS 1B across benchmark families without benchmark-specific parameter updates. We retain native aggregation and supplement it with record-level measures, coverage accounting, and subgroup diagnostics. Each Results subsection identifies the release and evaluation population, reports the official metric, and describes observed error patterns. A combined analysis synthesizes shared patterns while preserving the distinction between native and pooled quantities. Cross-paper comparisons are restricted to aligned evidence; differences in release, population, preprocessing, training, or benchmark exposure are context rather than rank. The findings characterize performance on evaluated records, not universal reliability, calibration, attribution, or future adaptive attacks. By keeping benchmark-native outcomes distinct from pooled summaries, the study makes test-population, class-balance, and missing-record-coverage differences visible. It supports interpretation of detector results in research, platform-safety, and forensic-review settings, foregrounding traceable protocol conditions over claims or leaderboard comparisons.

1 Introduction

The paper frames detector evaluation as protocol-dependent: benchmark populations, metrics, and aggregation rules shape what reported performance means. It evaluates supplied ARCAS 1B scores across seven benchmark families, preserving native outcomes while adding pooled, subgroup, and coverage analyses.

Motivation

  • Reported metrics and comparisons require their population and protocol context because benchmark releases differ in metrics, aggregation, generator coverage, and availability.The paper treats comparisons as informative only when release, population, metric, preprocessing, and benchmark exposure are sufficiently aligned.Source: §1 · p. 1

Evaluation scope

  • The study evaluates supplied Neural Defend ARCAS 1B scores on available records from seven public benchmark families, with no reported benchmark-specific parameter updates.The frozen-detector and no-update statements are Neural Defend attestations, not independently established by the supplied package.Source: §1 · pp. 1–2

Evaluation approach

  • The study retains each benchmark’s official metric and evaluation conditions, then adds pooled record-level summaries without conflating them with benchmark-native outcomes.Dedicated sections report release and denominator, examine error patterns, and review available baseline or detector evidence.Source: §1 · p. 2

Contributions

  • 1,298,747 scored records span the seven families, but confirmed AIGCDetectBenchmark–GenImage filename-and-label overlaps mean this is not an independent-image total.The paper also reports pooled accuracy, class-rate, ranking, and error analyses alongside detailed subgroup diagnostics.Source: §1 · p. 2

Coverage

  • Coverage accounting identifies image-only OpenFake filtering, an AIGCDetectBenchmark decode failure, and unavailable or undecodable WildFake records.The combined analysis synthesizes results across the seven benchmark families.Source: §1 · p. 2

2 Related Work

Prior work explores transfer across generators, but evidence also shows that performance depends on populations, transformations, metrics, and benchmark protocols. The paper presents benchmarks as complementary evidence and treats image-only detection as one bounded input to authenticity review.

2 Related Work

  • Synthetic-image detection research spans transfer-oriented detectors, benchmark design, and evaluation methodology, with reported performance depending on detector, population, input pipeline, and aggregation.Source: §2 · p. 2

2.1 Generalization in synthetic-image detection

  • Transfer approaches include shared generator artifacts, pretrained vision–language representations, gradients, and diffusion reconstruction; robustness remains conditional on generators, sources, compression, and transformations.Wang et al. (2020) reported transfer from ProGAN to unseen CNN generators under careful training and preprocessing, while later work describes limits under newer generators and distribution shifts.Source: §2.1 · p. 2

2.2 Benchmarks and performance interpretation

  • Accuracy at a fixed threshold, AP or AUROC, record-weighted results, and unweighted generator means answer different questions; benchmark populations and protocols therefore qualify comparisons.The paper retains native metrics and denominators, treating pooled measures as secondary and external results as contextual unless conditions align.Source: §2.2 · pp. 2–3

2.3 Image-only evaluation

  • Image-only detection assesses supplied-image evidence under a declared decision rule, but it cannot establish authorship or provenance and does not replace provenance evidence or human review.Published comparisons are contextual rather than controlled rankings when release, population, aggregation, processing, thresholding, or training exposure differ.Source: §2.3 · p. 3

3 Methodology and Evaluation Design

The evaluation analyzes supplied image-detector scores across selected benchmark records, preserving each benchmark’s native outcomes alongside pooled and subgroup diagnostics. It distinguishes score and coverage limitations from detector errors.

3.1 System Scope and Reproducibility

  • The study evaluates supplied ARCAS 1B image-detection scores; its multimodal, frozen-model, and no-benchmark-update descriptions are Neural Defend attestations, not independently verified package facts.The saved scores were post-processed; model inference, training, fine-tuning, and benchmark execution were not rerun.Source: §3.1 · p. 3, §3.2.2 · p. 4

3.2.1 Study Design and Record Accounting

  • Seven selected public benchmark families form a convenience sample, not a representative population; denominators count scored records, with indexed and scored records distinguished.WildFake had 714,156 metadata rows, 53,140 unavailable paths, 661,016 indexed paths, and 660,935 scored records.Source: §3.2.1 · p. 3
  • The 0.5 decision rule reproduces saved confusion counts, but no evidence establishes that the threshold was chosen before benchmark access, making threshold-dependent results retrospective.One WildFake score equals 0.5 and is classified as generated under the greater-than-or-equal rule.Source: §3.2.1 · p. 4

3.2.2 Metrics and Aggregation

  • 99.93%/99.93% AP/trapezoidal PR-AUC was reported for OpenFake core/test, while the stored pr auc field’s numerical match to AP does not establish its historical implementation.The corresponding supplementary recalculations were 98.61%/98.61% for reddit/test and 99.87%/99.87% for pooled still-image records.Source: §3.2.2 · p. 4
  • Native benchmark aggregation remains distinct from pooled records: fixed-threshold confusion measures, ranking metrics, and macro averages describe different aspects of performance.AP and AUROC summarize score ordering, while pooled values summarize records; neither establishes calibration, deployment utility, or performance under different prevalence.Source: §3.2.2 · p. 4
  • Uncertainty intervals assume independent Bernoulli records, and fixed-FPR TPR and equal-error rates are retrospective test-set diagnostics rather than validated deployment thresholds.Shared or related images and cross-benchmark overlap can violate the interval assumption; no cluster-aware alternative or paired comparator test was available.Source: §3.2.2 · p. 4

4 Results

The four benchmark evaluations report strong performance on their stated populations while retaining native metrics and protocol caveats. Their confusion matrices show that aggregate scores do not erase subgroup-specific or coverage-related limitations.

4.1 GenImage Evaluation

  • 99.99% eight-folder mean accuracy places ARCAS 1B at the top of the reported GenImage score range, though external protocols differ.The comparison is contextual, not a protocol-matched ranking.Source: §4.1 · p. 5, Figure 1 · p. 5
  • Only 11 errors occur across 100,000 GenImage records; Midjourney contributes five, while ADM and Wukong each contribute two real-image false positives.The small count does not establish zero population error, and errors remain generator-dependent.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 WildFake Evaluation

  • 99.92% available-case accuracy on WildFake is close to published baseline scores, but ARCAS uses locally available records while those baselines use the complete official test.The comparison conveys scale rather than an unconditional ranking.Source: §4.2 · p. 5, Figure 3 · p. 6
  • WildFake available-case recall and specificity are 99.92% and 99.93%, respectively, with 542 residual errors concentrated in diffusion-source false negatives and selected real-image false positives.The 660,935 scored records exclude unavailable or undecodable official-index records.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3 AIGCDetectBenchmark Evaluation

  • 99.79% native mean accuracy on nominal R17 exceeds the strongest plotted baseline, SynerDetect at 94.76%, under differing training and implementation conditions.The comparison uses the same 17-generator mean-accuracy family.Source: §4.3 · p. 6, Figure 5 · p. 6
  • AIGCDetectBenchmark has 99.82% pooled accuracy, but 270 real images are false positives versus seven generated images scored as real.The record-weighted fixed-threshold errors indicate a specificity issue, while the native comparison gives equal weight across 17 generator families.Source: §4.3 · p. 6, Figure 6 · p. 6

4.4 AIGIBench Evaluation

  • 99.10% native mean accuracy on the supplied AIGIBench snapshot is strong, but release corrections and differing FaceSwap/InSwap counts make published comparisons contextual.The comparison is not a strict head-to-head ranking.Source: §4.4 · p. 7, Figure 7 · p. 7
  • AIGIBench errors are not uniform: FaceSwap has weaker generated-image recall, while SocialRF has relatively more real-image false positives.These distinct patterns qualify the 99.10% mean across generation and manipulation modes.Source: §4.4 · p. 7, Figure 8 · p. 7

4.5 Community Forensics Evaluation

Across the evaluated populations, high AUROC coexists with benchmark-specific fixed-threshold error patterns, so similar accuracy scores can conceal different decision risks. Community Forensics, OpenFake, Chameleon, and the cross-benchmark analysis preserve distinctions between native and pooled measures and their evaluation conditions.

4.5 Community Forensics Evaluation

  • 98.17% native mean accuracy places ARCAS 1B 0.88 points above Pangram and 8.87 points above CF-384, while its 99.92% native mean AP accompanies pooled errors concentrated in missed generated images.The comparison is contextual because Pangram’s denominator and training-overlap audit are undisclosed; larger subsets receive greater weight in the pooled confusion matrix.Source: Figure 9 · p. 7, §4.5 · p. 7, Figure 10 · p. 8

4.6 OpenFake Evaluation

  • 97.84% accuracy on OpenFake’s held-out core split falls to 95.86% on Reddit, while pooled accuracy is 97.65% and generated-image recall declines on naturally circulated material.There is no directly matched version-2 SOTA baseline; pooled errors are driven mainly by missed generated images rather than a broad false-positive problem.Source: Figure 11 · p. 8, §4.6 · p. 8, Figure 12 · p. 8

4.7 Chameleon Evaluation

  • 97.44% accuracy on Chameleon exceeds the plotted TAP (95.64%), FGINet (92.50%), and SimLBR (85.57%) results, but comparison protocols differ and generated-image recall is lower than real-image specificity.Generated-image recall is 95.43% versus 98.94% real-image specificity, making generated-image misses the principal residual error source.Source: §4.7 · p. 8, Figure 13 · p. 9, Figure 14 · p. 9

4.8 Cross-benchmark analysis

  • AUROC remains near ceiling across benchmark rows, but residual errors vary: generated-image misses dominate in Chameleon, OpenFake, and Community Forensics, while false positives dominate in AIGCDetectBenchmark.OpenFake and Chameleon have the largest APCER values, while AIGCDetectBenchmark has comparatively larger BPCER; unequal populations and native aggregations mean this is not a single leaderboard.Source: §4.8 · p. 9

5 Discussion

The discussion treats the results as evidence about supplied scores on selected records, not universal reliability or a controlled ranking. It connects observed errors and comparison limits to operational validation, governance, and monitoring needs.

5.1 Interpretation of the evidence

  • Concentrated subgroup errors and divergence between ranking and fixed-threshold classification qualify aggregate performance; the supplied package does not independently establish several Neural Defend attestations or explain the errors’ causes.The paper identifies Firefly Image 3, Hourglass, FaceSwap, SocialRF, and OpenFake Reddit as groups with concentrated errors.Source: §5.1 · p. 9

5.2 Operational interpretation

  • A binary generated/real score cannot attribute an image to a particular generator, and scores should route content to review rather than establish authorship, intent, liability, or grounds for punishment.Threshold selection requires a separate, operationally representative population; retrospective selection on the seven test sets requires later validation.Source: §5.2 · pp. 9–10

5.3 Meaning of the comparison evidence

  • Comparison tables are evidence maps rather than a leaderboard: only selected panels have the closest metric and protocol alignment, while other rows remain contextual.Differences in benchmark exposure, training, population, preprocessing, version, or missingness prevent numerical separation from isolating method quality under equal resources.Source: §5.3 · p. 10

5.4 Deployment validation and score governance

  • Deployment requires prospective threshold validation, declared score-governance procedures, and post-release monitoring for drift, calibration, coverage failures, and errors by source or transformation.The paper notes that these monitoring studies were not performed here.Source: §5.4 · p. 10

6 Conclusion

The conclusion presents a scoped analysis of supplied ARCAS 1B scores across seven selected public benchmark families. It reports measured performance while retaining population, error, coverage, protocol, and overlap qualifications.

6 Conclusion

  • The study evaluates supplied detector scores using benchmark-native outcomes, pooled analyses, subgroup diagnostics, and comparisons limited to interpretable evidence.The resulting account concerns evaluated records and does not claim that detection is solved or universally reliable.Source: §6 · p. 11
  • The selected benchmark families do not represent all generators, transformations, contexts, or future releases, and version differences, a decode failure, and WildFake path unavailability further limit interpretation.Approximately 88,000 shared filename-and-label signatures were reported between AIGCDetectBenchmark and GenImage; Wilson intervals assume independent records despite possible shared sources and overlap.Source: §6 · p. 11
  • The study does not establish calibration, temporal drift, adversarial robustness, fairness, attribution, or operational costs, and includes no external blind evaluation.Interpretation should remain tied to released records and declared denominators; deployment requires prospective validation and monitoring.Source: §6 · p. 11

7 Declarations

The declarations describe the study’s materials, affiliations, ethics, and assistance. They distinguish what accompanies the submission from proprietary or restricted materials that are not released.

7 Declarations

  • Benchmark datasets remain available from their providers under current terms, but the manuscript does not redistribute images and the submission archive omits per-record predictions and full-precision scores.Aggregate tables, vector figures, and LaTeX source accompany the authors’ package.Source: §7 · p. 11
  • The ARCAS 1B architecture, weights, private training data, training code, and inference API are not released; restricted identifiers and proprietary operational information remain subject to approval.The private authors’ package includes post-processing, reconciliation, source-ledger, comparison-eligibility, and build/visual-QA records.Source: §7 · p. 11
  • All four authors are affiliated with Neural Defend Inc., the developer of the evaluated commercial system, and the manuscript reports no supplied funding information.The authors state that generative AI assisted with drafting, literature lookup, analysis scripting, citation checks, and consistency review.Source: §7 · p. 11
  • The work analyzes existing public benchmark records without a new human-participant study or newly collected participant data; benchmark reuse remains subject to provider terms and institutional requirements.The source and comparison search was last updated on 15 September 2026.Source: §7 · p. 11

A Additional Results

Additional results preserve benchmark-specific context for prior comparisons and report generator-level performance differences. The tables support traceable evidence by benchmark, not a pooled cross-paper ranking.

A Additional Results

  • Table 2 retains each selected prior result’s reported metric and protocol context, so its rows are evidence by benchmark rather than a unified ranking.The prior-method values are cited within the table.Source: A Additional Results · p. 14, Table 2 · p. 14
  • Table 3 reports selected generator-level performance and notes that GPT and Gemini were not named generator labels in the evaluated releases.Source: Table 3 · p. 14
  • Firefly rows have lower recall than Stable Diffusion rows, while Midjourney results vary by benchmark population.These generator-level records locate aggregate errors in identifiable families.Source: A Additional Results · p. 14

B Complete Legacy Subset Tables

The legacy appendix tables retain recoverable subset and generator records underlying the paper’s aggregate results and diagnostics. Their metrics and coverage caveats qualify interpretation of individual rows.

B Complete Legacy Subset Tables

C Real and Generated Confidence Distributions

Figure 16 displays real-versus-generated confidence distributions for the seven benchmark populations evaluated in the corresponding Results subsections. OpenFake and WildFake retain distinct population-scope qualifiers.

C Real and Generated Confidence Distributions

  • Figure 16 shows real and generated score distributions across the seven evaluated benchmark populations, with OpenFake pooled across still-image splits and WildFake available-case.Source: Figure 16 · p. 17, C Real and Generated Confidence… · p. 17
  • Greater overlap between real and generated distributions corresponds to larger fixed-threshold class-specific error rates, while stronger separation aligns thresholded and threshold-free measures more closely.The OpenFake and WildFake distributions retain the same scope qualifiers as the main Results section.Source: C Real and Generated Confidence… · p. 17

D Precision–Recall and DET Curves

The precision–recall and DET curves show threshold-dependent trade-offs across benchmark populations. Their shapes complement threshold-free measures and help distinguish residual error patterns.

D Precision–Recall and DET Curves

  • Precision–recall curves remain near the high-precision, high-recall region for most populations, while their shapes identify where a fixed 0.5 threshold may be less favorable.They retain the trade-off between positive predictive value and generated-image recall across thresholds, particularly when class balance differs.Source: D Precision–Recall and DET Curves · p. 18, Figure 17 · p. 18
  • The paper reports both threshold-free AP/AUROC and fixed-threshold error measures because the measures are not interchangeable; a separate low-FPR ROC figure supports deployment-oriented discussion.Source: D Precision–Recall and DET Curves · pp. 18–19
  • DET curves reveal low-rate false-positive and false-negative trade-offs, and show that residual errors differ across datasets, with false positives making up most of AIGCDetectBenchmark’s observed residual.Source: D Precision–Recall and DET Curves · p. 18, Figure 18 · p. 18

Related papers

Loading 2609.25154v1…