A Benchmark & Dataset for Detecting AI-Manipulated Visual Evidence in the Court System

1 Introduction

• Photographs serve as evidence of presence, incident sequence, damage, injury, possession, and scene condition, so their value depends on trust and provenance.The task is to assess whether evidence is authentic, manipulated, or synthetic under conditions resembling legal and institutional use.Source: §1 · pp. 1–2
• Generative systems lower the cost and skill needed to alter or fabricate plausible evidence, while making genuine evidence easier to dismiss as fabricated—the liar’s dividend.The paper identifies a resulting need both to detect false evidence and preserve confidence in authentic evidence.Source: §1 · p. 2
• Localized or contextual edits can change what an ordinary photograph appears to prove while preserving most of its content, making them difficult for lay observers [1] [19] and some detectors.Examples include removing a person, changing a road sign or traffic signal, and adding or removing visible damage.Source: §1 · p. 2
• Existing resources lack a public benchmark combining evidentiary image families, contemporary consumer-generated manipulations, structured metadata, and detector evaluation relevant to institutional use.Face-deepfake, classical tampering, and authentic-image datasets address parts of this need but not the full combination.Source: §1 · p. 2
• The CIFAR Synthetic Evidence Corpus contains 1,505 items—720 authentic controls and 785 manipulated or fabricated images—across surveillance, dashcam, and consumer photographs.Its three manipulation tiers cover scene-condition changes, localized edits, and complete fabrication, with item-level provenance, generator, prompt, and scene metadata.Source: §1 · p. 2
• The paper contributes an evidentiary taxonomy emphasizing meaning-changing localized edits, a consumer-tool generation pipeline, and a zero-shot benchmark of off-the-shelf detectors.The reported detector error profiles remain problematic for evidentiary use.Source: §1 · p. 2

2 Distinction from Existing Resources

• Face-manipulation benchmarks focus on facial identity or expression, while classical tampering datasets cover operations such as splicing and removal but largely predate consumer generative systems.Authentic surveillance, driving, and consumer-image resources generally lack controlled generative edits, generation metadata, and authentication labels.Source: §2 · p. 3
• The corpus treats photographs as visual claims about events, locations, objects, or conditions, complementing rather than replacing general image-forensics benchmarks.Its focus is prompt-driven generation and editing, including localized changes, scene-level modifications, and complete fabrication under a contemporary non-expert threat model [2] [4].Source: §2 · pp. 2–3
• Metadata organized by evidence family, manipulation tier and subtype, source, generator, prompt, and scene attributes enables analysis of missed edits and false positives beyond aggregate accuracy.The benchmark supplies empirical infrastructure for studying authentication reliability and limits, but the authors do not claim detection alone resolves the legal problem.Source: §2 · p. 3

3 Resource Design and Construction

• The design targets evidentiary authentication with legal and institutional photographs, manipulations that can change evidentiary meaning, consumer-tool plausibility, and analysis beyond binary classification.These requirements frame the corpus around evidence use rather than general synthetic-image detection.Source: §3 · p. 3
• The corpus covers surveillance, dashcam, and consumer photographs, with tiers for scene-condition changes (T1), localized edits (T2), and complete fabrication (T3).Figure 1 illustrates T1 snow added to a dashcam frame and T2 insertion of a person into a surveillance frame.Source: §3 · p. 3, Figure 1 · p. 3
• Twenty-seven fixed prompt templates model non-expert use of consumer interfaces, each requesting one scoped, plausible change while preserving the rest of the scene.Templates span the three evidentiary families and three manipulation tiers.Source: §3 · p. 3
• Source imagery comes from VIRAT and UCF-Crime for surveillance, BDD100K for dashcam, and CASIA v2.0 for consumer photographs [22] [23].T1 and T2 edits use authentic frames; T3 images are generated from prompts alone, and source frames are not reused.Source: §3 · p. 3
• The main variants use single-pass generation with GPT Image 2, Gemini 3.1 with Nano Banana 2, and Firefly 5; the smaller Hero set models iterative human refinement.Items were screened and regenerated when edits failed or degraded scenes, but screening was not formal annotation and inter-annotator agreement is not reported.Source: §3 · p. 3

• Detectors are evaluated zero-shot using released weights and default thresholds, with threshold-free and threshold-dependent metrics reported.Each detector scores every image once, without fine-tuning.Source: §4 · pp. 3–4
• Table 1 organizes the manipulated corpus by evidentiary family, source dataset, and manipulation tier.Source: Table 1 · p. 4
• Across all systems, the best off-the-shelf detector, d3, reaches only 0.674 ROC-AUC, while the rest sit near chance; detectors trained predominantly on fully synthesized GAN or diffusion imagery transfer poorly to localized edits and consumer-tool manipulations.At its shipped threshold, d3 catches fewer than half of all manipulations (47% recall) and wrongly flags 22% of authentic images as manipulated.Source: §4 · p. 4
• At its shipped threshold, d3 has 47% recall and falsely flags 22% of authentic images as manipulated.The paper notes that falsely impugning genuine evidence is the more damaging error in court.Source: §4 · p. 4
• For d3, AUC is 0.55 on dashcam imagery versus 0.69 on surveillance and 0.78 on consumer photographs.Per-generator recall ranges from 0.46 to 0.71; Hero-set Photoshop edits score as low as 0.04 and evade the detector.Source: §4 · p. 4

• The corpus is derived from publicly released research datasets and redistributed under their source licenses.Source: §5 · p. 4
• No imagery was collected from human participants, and no material comes from real legal proceedings.Source: §5 · p. 4
• The authors judge release-related uplift to be small relative to detection-research benefits because manipulations use consumer-interface capabilities already available.The study used only UCF-Crime’s normal-activity subset and avoided associating identifiable people with crimes or conduct at real locations.Source: §5 · p. 4

• The CIFAR Synthetic Evidence Corpus supports diagnostic evaluation of contemporary AI manipulations across diverse photographic evidence.Source: §6 · p. 4
• Baselines show substantial weaknesses in current detectors, particularly for localized edits and authentic-image false positives.Source: §6 · p. 4
• The corpus and benchmark infrastructure are released to support future research on reliable visual evidence authentication.Source: §6 · p. 4

• GPT Image 2, Gemini 3.1 with Nano Banana 2, and Firefly 5 were used to construct the manipulated portion of the corpus.Source: GenAI Usage Disclosure · p. 5
• Claude assisted with improving author-written prose and generating draft LaTeX for tables.Source: GenAI Usage Disclosure · p. 5
• The authors reviewed and edited all AI-assisted text, take responsibility for the paper, and list no AI system as an author.Source: GenAI Usage Disclosure · p. 5