PaperlayerView exact revision on arXiv

Source-linked AI summary

LAION-Mobile: Evaluating Deepfake Detectors On One Million Smartphone Photos

Achim von Stryk, Janis Keuper

arXiv:2609.11134v1cs.CV

TL;DR

Deepfake detectors achieve near-perfect results on familiar benchmarks, but the scale of their reliability on modern smartphone photography remains insufficiently tested. This paper builds LAION-Mobile and audits twelve detectors across modern AI content and real smartphone photos, finding that no detector is both better than chance on modern AI and deployable on real photos. The corpus primarily covers first-generation neural-ISP devices, leaving current flagship regimes open.

  • Problem

    Whether computational-photography processing in ordinary smartphone photos causes deepfake detectors to misfire remains insufficiently tested beyond small controlled studies.

  • Method

    The paper constructs a roughly one-million-image EXIF-identified smartphone corpus and audits twelve original detector checkpoints using modern AI and real-photo evaluation regimes.

  • Results

    No detector both beats chance on modern AI content and remains deployable on real smartphone photos; the best modern-AI AUC is 0.624.

  • Takeaways & Limitations

    Real-photo false-alarm rates depend strongly on threshold calibration, so legacy calibration can make detectors appear deployable while modern calibration exposes high false alarms.

  • Takeaways & Limitations

    The corpus is dominated by older devices and largely omits current flagship smartphones, leaving the modern-ISP regime untested.

Abstract

from arXiv · show

Most Deepfake detectors report near-perfect AUC scores on their reference benchmarks. However, a recent ICML position paper argues that these evaluations collectively neglect the impact of modern smartphone photography: the widely used on-device neural image-signal processing pipelines (like multi-sensor fusion or noise and motion-blur suppression) increasingly shift the imaging paradigm from simple lens projections towards computational photography. Hence, devices actually generate, rather than record photos. This increases the risk that deepfake detectors may flag ordinary phone photos as fake. Due to the lack of large-scale datasets containing images from modern smartphones, this hypothesis has so far only been tested in small proof-of-concept studies. The aim of this paper is to close this gap. We introduce LAION-Mobile, an open dataset containing about 1 million smartphone images with EXIF metadata distilled from re-LAION-5B. Evaluating twelve state-of-the-art deepfake detectors with their original paper checkpoints on a 9,115-image evaluation sample of this pool (DIRE on 738), we report three key findings: (i) On modern AI content no detector exceeds AUC 0.624, and five of twelve fall below chance. (ii) Real-photo false-alarm rates are an artefact of threshold calibration: thresholds fitted on legacy GAN data make several detectors look deployable (less than 11 percent FPR), yet the same detectors flag 17-91 percent of real photos once the identical criterion is refit on modern content. (iii) Consequently, no detector both beats chance on modern AI content and keeps a deployable real-photo false-alarm rate. Mirroring the device mix of web collections, the corpus probes the first neural-ISP generation (2018-2020); current flagships are essentially absent, leaving the modern-ISP regime as the open gap.

1 Introduction

This paper tests whether smartphone computational photography undermines deepfake-detector reliability at scale. Auditing twelve detectors on smartphone photos and modern AI content, it finds that benchmark performance does not translate into deployable detection.

  • 1 Introduction: Modern smartphone photography matters because neural ISPs combine denoising, blur suppression, and exposure fusion using learned processing pipelines.These computational-photography operations motivate testing whether ordinary phone photos trigger deepfake detectors.
  • 1 Introduction: The work addresses a gap left by earlier first-principles arguments and small controlled studies of smartphone-photo effects on detection.The paper aims to verify whether those concerns persist across the scale and diversity of real smartphone photography.
  • 1 Introduction: The paper assembles a publicly derivable 9,115-image smartphone evaluation subset from a roughly one-million-image LAION-Mobile collection with EXIF metadata.The broader contribution includes a reusable filter manifest and per-image EXIF.
  • 1 Introduction: No detector both beats chance on modern AI content and maintains a deployable false-alarm rate on real smartphone photos.The study audits twelve detectors using original checkpoints and a uniform evaluation protocol.
  • 1 Introduction: AUC reaches only 0.624 on NTIRE modern AI content, while modern-calibrated false-alarm rates on LAION-Mobile rise as high as 91%.Figure 1 contrasts published paper AUC with directly comparable NTIRE AUC and LAION-Mobile false-alarm measurements.

2 Related Work

Prior work documents detector brittleness across generators but has not adequately tested computational-photography effects on real smartphone images. LAION-Mobile extends this evaluation with a large, phone-native real-photo set and a cleaned pool derived from re-LAION-5B.

  • 2 Related Work: Deepfake detectors comprise convolutional fingerprint methods, CLIP-feature linear probes, and reconstruction-error or representation-invariance approaches.These families include CNNDetection, UnivFD, and DIRE, among others.
  • 2 Related Work: Existing real-photo evaluations contain at most 103 laboratory captures from a handful of devices, leaving large-scale smartphone effects unresolved.This gap concerns whether computational-photography artefacts cause detectors to misfire at scale.
  • 2 Related Work: The cleaned LAION-Mobile pool contains 935,399 images after distillation and non-photo filtering from re-LAION-5B.Figure 2 presents the five-stage funnel and identifies the green cleaned pool as the source of all evaluation images.
  • 2 Related Work: LAION-Mobile isolates false alarms on unmodified consumer photographs, unlike challenge mixtures that score discrimination between real and generated images.The subset expands real-photo evaluation in both scale and device diversity.

3 The LAION-Mobile Smartphone-Photo Subset

LAION-Mobile constructs a large, EXIF-enriched smartphone-photo pool by filtering and balancing re-LAION-5B images, then characterizes its device, metadata, and labeling scope. The resulting corpus is dominated by older phones and supports real-photo false-alarm evaluation, while modern flagship coverage remains limited.

  • Source and filtering: 935,399 cleaned smartphone-photo IDs remain after deduplication, URL filtering, manufacturer matching, and three non-photo filters.The source pipeline begins with high-resolution EXIF-bearing images from re-LAION-5B and removes captions, aspect ratios, or EXIF patterns indicating non-photographs.
  • Composition: Apple supplies 68.0% and Samsung 22.5% of the pool, while valid EXIF-GPS coordinates cover 41.3% and are concentrated in Europe, North America, and East Asia.Brand and geography are entangled because Apple and Samsung dominate every populated region.
  • Evaluation subset: The evaluation subset contains 9,115 valid detector-scored images spanning five manufacturers, about 80 models, and seven exposure stops, with EXIF retained.A manufacturer-balanced sample of 10,000 IDs yielded 9,171 downloads and 9,115 valid scores.
  • Device-era coverage: Only 212 release-dated images, or 0.03%, come from phones released in 2021 or later, leaving current flagship neural-ISP coverage essentially absent.The release-dated pool peaks at 2014–2016 hardware and collapses after 2020.
  • Composition: Only 10.9% of images come from heavy-ISP phones, versus 40.9% from phones classified as having no ISP aggressiveness and 18.4% light.The frequent models are largely older iPhones from roughly 2011–2017, which predates neural ISPs.
  • Labels and release: The corpus defines real as physical-scene camera capture despite ISP processing, labels every LAION-Mobile image real, and therefore supports one-class false-positive evaluation.Operating points come from equal-error-rate thresholds fitted on held-out two-class data.

4 Methodology

The study evaluates twelve detectors using original checkpoints and paper-specified preprocessing across fixed samples of several datasets. It reports both default and dataset-specific EER operating points to expose calibration drift and deployment trade-offs.

  • Detectors and protocol: Twelve detectors are evaluated with original paper checkpoints, original input sizes, and original normalization rather than substituted fine-tuned variants.A documented CLIP-Raisingthe-Bar configuration override restores its training setup and score convention.
  • Calibration: LAION-Mobile false-positive rates are reported at default τ=0.5 and EER thresholds fitted separately on ProGAN-ISP and NTIRE.This contrasts calibration based on legacy GAN data with calibration based on modern AI content.
  • Coverage: Each detector uses a fixed random sample per dataset, with 9,115 scored LAION-Mobile images and 738 DIRE images because diffusion reconstruction is more expensive.Confidence intervals therefore reflect the evaluated sample sizes rather than the total source-dataset sizes.

5 Results

Across modern AI and real-photo evaluations, detectors degrade sharply: modern-content discrimination is weak, calibration changes false-alarm rates substantially, and curated smartphone/camera sets reproduce the real-photo failures.

  • 5.2 Cross-Domain Failure on Modern AI Content: NTIRE AUC spans 0.42–0.62, only five detectors exceed chance, and no detector exceeds 13.5% TPR at 5% FPR.The best detector clears chance by only 0.12 AUC.
  • 5.3 False Alarms on Real Smartphone Photos: Modern calibration raises every detector’s LAION-Mobile false-alarm rate to 17–91%, eliminating the deployable quadrant; UnivFD still flags 39.1%.Legacy calibration instead makes several detectors appear deployable, with rates at or below 11%.
  • 5.3 False Alarms on Real Smartphone Photos: The same high-false-alarm pattern appears on HDR+, SIDD, and MIDD, indicating that misfires are not unique to the LAION-Mobile corpus.These are independent curated real-photo sets.
  • 5.4 Computational-Photography Era: the Modern Regime is Missing: Only two detectors track ISP-era bins as predicted, while four move significantly against the prediction and six are indistinguishable from zero.Tiny-LaDeDa and AIDE show the strongest opposing shifts.
  • 5.4 Computational-Photography Era: the Modern Regime is Missing: The corpus contains almost no phones released in 2021 or later, so its era analysis cannot test current flagship ISPs.The observed corpus primarily covers first-generation neural-ISP devices from 2018–2020.
  • 5.5 Threshold Calibration Cannot Save Default-Biased Detectors: Threshold movement cannot restore performance when score distributions overlap, and five detectors remain below chance on modern content independent of threshold choice.UnivFD’s legacy threshold detects essentially nothing on the modern curve.

6 Discussion

The discussion interprets detector failures as interactions between training-distribution artefacts and smartphone capture pipelines, while concluding that no evaluated detector is deployable under modern conditions.

  • 6 Discussion: Detector artefact fingerprints can overlap with HDR fusion, sharpening, and semantic denoising in smartphone JPEGs, causing detectors to misfire on real photos.The paper describes this as detector-specific interaction rather than a uniform ISP effect.
  • 6 Discussion: The discussion treats the era result as a conservative lower bound because the corpus barely reaches modern ISP devices and requires controlled paired data from current phones.Calibration drift alone produces large changes in false alarms without testing current flagships.
  • 6 Discussion: No detector both beats chance on modern AI content and maintains a deployable false-alarm rate on real smartphone photos.UnivFD is closest but still flags 39% of real photos after modern calibration.

7 Limitations

LAION-Mobile’s labels are heuristic and its measurements remain bounded by cross-domain calibration, older hardware, incomplete uncertainty accounting, and image-retrieval constraints.

  • 7 Limitations: LAION-Mobile is real-only by construction, so it supports false-positive-rate evaluation but cannot provide two-class AUC.Real labels rely on EXIF and filtering rather than per-image verification.
  • 7 Limitations: EXIF may be copied or forged, edited images may retain camera metadata, and the web corpus is dominated by pre-2021 hardware.The era stratification may also misclassify firmware-updated phones.
  • 7 Limitations: Reported false-positive rates also reflect score-scale shift because EER thresholds are fitted on synthetic datasets and transferred to LAION-Mobile.Bootstrap confidence intervals do not capture training-run variance.
  • 7 Limitations: Source images are not redistributed, so replication requires refetching URLs and some may have decayed.The released metadata and scores are intended to preserve reproducibility despite this constraint.

8 Conclusion and Future Work

The audit finds that detector generalization collapses on modern AI content, while calibration determines whether real smartphone photos are flagged at deployable or unusable rates. LAION-Mobile confirms the position paper’s concern at scale but leaves current flagship devices largely untested.

  • Only five detectors beat chance on modern AI content, and after calibration on that content they flag 39–63% of real smartphone photos.The same roster achieves published paper-benchmark performance, indicating that the collapse reflects cross-domain generalization rather than a broken evaluation pipeline.
  • No audited detector is simultaneously above chance on modern content and deployable on real photographs.The conclusion holds across the paper’s twelve-detector roster and the three evaluation regimes.
  • False-alarm rates depend strongly on threshold calibration: the same detector can appear deployable or unusable depending on the legacy set fixing its threshold.The paper therefore attributes the observed false alarms more to training distribution and calibration than to the smartphone ISP era alone.
  • Current flagship devices are nearly absent from LAION-Mobile, with iPhone 13 and later comprising just 0.03% of the release-dated corpus.The dataset is dominated by older phones, so modern neural-ISP behavior remains an open evaluation gap requiring paired RAW, ISP-processed, and generated images.
Loading 2609.11134v1…