ChatGPT Images 2.5 in the Wild: A Launch-Period Dataset and Detector Evaluation

• A product-name match can refer to either generator version, and a post made after an upgrade may still contain an older image.Source: §1 · p. 1
• In a launch-day pilot, 35 of 106 returned posts explicitly named GPT-Image-2, motivating explicit version evidence rather than version-silent queries.Source: §1 · p. 1
• The study asks what supports version attribution, how source composition shapes observations, and how published detectors respond at fixed operating points.Source: §1 · p. 1
• Figure 1 presents one selected image per source, so its source-balanced display is not a proportionally representative sample.Source: Figure 1 · p. 2
• The contribution is a documented launch-period collection and fixed-threshold detector evaluation; “in the wild” means online posts and galleries under stated admission rules, not a census.Source: §1 · p. 2

• Earlier targeted datasets include DiffusionDB [26], GenImage [35], and a GPT-Image-2 X collection [31] with 10,217 images under creator self-report.Source: §2 · p. 2
• The collection distinguishes caption-version claims from host records and applies an image-form filter across sources, while its gateway does not establish exhaustive coverage.Source: §2 · p. 2
• Prior online-detection studies examine social platforms and image processing [5] [11]; this work instead records attribution evidence during a product transition.Source: §2 · p. 2
• C2PA uses signed manifests, but inspected X-delivered April credentials did not survive that path, and the inspected generator-version field here does not distinguish tested endpoint versions.Source: §2 · p. 2
• Prior work addresses disclosed AI use in Reddit art communities [12], political imagery on X [3], and fact-checked media [6]; disclosure rates do not estimate disclosure recall without identifying undisclosed images.Source: §2 · pp. 2–3
• Detector generalization across generator families and performance outside laboratory benchmarks remain concerns [7] [22], motivating evaluation in a less controlled setting.Source: §2 · p. 3

3.1 Launch anchor and funnel

• The collection begins at the 8 September 2026 announcement timestamp and ends about 51.1 hours later; the anchor is a reproducible scope boundary, not the earliest attributable generation.Source: §3.1 · p. 3

3.2 Collection infrastructure

• Five collection routes use one commercial gateway, while Pixiv, note.com, and NightCafe use public JSON endpoints directly.Source: §3.2 · p. 3

3.3 Sources, routes and yield

• Nine instruments differ in queries, filtering, timestamps, formats, language metadata, and creation-language requirements, so cross-source differences reflect instruments as well as platforms.Source: §3.3 · p. 3

3.4 The version gate

• Caption claims, classifier-assisted caption claims, and host records form separate attribution tiers; neither caption claims nor host records independently verify the image’s generator.Source: §3.4 · pp. 3–4
• 10,163 candidate records yield 3,478 retained images after attribution, deduplication, standalone-image filtering, and targeted review; 12 videos are excluded from image statistics.Source: Figure 2 · p. 4

3.5 Image-form filter and targeted review

• Of 6,344 gated images, 3,494 are labelled standalone; targeted review removes 16 more, while excluded composites may still contain real outputs.Source: §3.5 · p. 4

3.6 Content credentials and version attribution

• The inspected generator-version field reports 2.0 for all 1,007 benchmark files, including both tested 2.5 endpoints, so it does not distinguish those versions.Source: §3.6 · p. 4

3.7 Coverage beyond the retained sources

• Retained sources do not exhaust online images; NightCafe’s observed feed changed from no 2.5 records on September 9 to 1,449 records with a 2.5 field on September 10.Source: §3.7 · p. 4

• The audits use seeded samples relabelled by gpt-5.5, measuring agreement with another model rather than human annotation or verified generator attribution.Source: §4 · p. 4
• Caption-claim agreement is 64.0% (54.2–72.7) across 100 sampled admitted post texts, with disagreements including announcements, tutorials, and marketing.Source: §4 · p. 5
• The auditor labelled 98.5% (91.9–99.7) of 66 predicted standalone images as standalone, but the unsampled stratum leaves full-pool recall unidentified.Source: §4 · p. 5
• CLIP and auditor labels agree on 61.0% of 100 earlier released images (κ = 0.51), so content is reported as CLIP assignments rather than validated subject prevalence.Source: §4 · p. 5

5 Content and Source Composition

• The analysis applies CLIP subject classification, OCR, and face detection to 3,478 released images, treating content labels and post metadata as measurements of this collection.Figure 1 shows one illustrative image per source.Source: §5.1 · p. 5, §5 · p. 5

5.1 Subject matter, text and faces

• 39.0% of images come from NightCafe, which supplies 77.0% of fantasy/surreal assignments; excluding it lowers the pooled fantasy share from 27.2% to 10.3%.These differences describe sampled images; small source cohorts cannot provide stable estimates of platform-wide preferences.Source: §5.1 · p. 5
• OCR finds text in 39.0% (1,357) versus April's 82.0%, while faces appear in 59.0% versus 59.2%; median text regions are 0 versus April's 29.These collection measurements do not establish changes in model use; face-detector bias on non-photorealistic images is why demographic histograms are omitted.Source: §5.1 · p. 5

5.2 Languages, timing and wording

• Japanese (703), English (675), and Chinese (631) are the most common recorded labels, but Chinese retention is 27.9% versus 55.6% for Japanese and 62.4% for English.The labels describe post metadata, not image text; source composition and image-form filtering contribute to retention differences, so counts do not measure platform adoption.Source: §5.2 · p. 5
• 81% of the 1,667 images posted from 2026-09-10 05:25Z onward come from NightCafe and were collected that day, so the final interval does not demonstrate increased posting activity.Recorded posting timestamps span roughly 51.1 hours and differ from retrieval dates.Source: §5.2 · p. 5
• Among 1,082 caption-attributed posts with nonempty text, keyword matches for prompt vocabulary occur in 31.6%, editing in 10.5%, consistency in 6.8%, and speed in 6.7%.NightCafe is excluded because its text field contains creation titles; these unvalidated matches do not measure actual editing or prompt disclosure.Source: §5.2 · p. 5

6.1 Controls and operating point

• Six frozen detectors use thresholds calibrated to flag at most 5% of 1,000 calibration photographs; this does not establish 5% FPR across content types or support deployment rankings.Deployment precision also requires representative controls and generated-image prevalence in the target stream.Source: §6.1 · pp. 5–6, §6 · p. 5

6.3 Flag rates on the collection

• Held-out photograph FPRs range from 3.5 to 5.9%, while artwork FPRs range from 1.5 to 96.5%; PGC flags 96.5% of artworks and 5.1% of photographs.A high collection flag rate can coexist with sensitivity to human artwork.Source: §6.3 · p. 6, Table 2 · p. 7

6.2 Harness reproduction on GenImage

• All five detectors with published GenImage references agree within 3.5 accuracy points (four within 1.7); three of 28 PGC comparisons exceed three standard errors, all higher in our results.PGC's published accuracy is 100.0% on six generators, with BigGAN its only nonsaturated reference cell.Source: §6.2 · p. 6, Table 3 · p. 7

6.4 Comparison with our April GPT-Image-2 release

• Unverified attribution, processing and content mismatch, benchmark exposure, possible edits, differing April admission rules, and pilot-driven sample expansion prevent assigning the observed gaps to generator effects.The reported gaps are not assigned to any single cause.Source: §6.3 · p. 7, §6.4 · p. 8

6.3 Flag rates on the collection

• At matched thresholds, detectors flag 3.7–56.4% of collection images versus 47–100% GenImage generated-image recall, a 42–81 pp lower rate for every detector.Figure 4 places collection rates alongside benchmark recall and control false-positive rates.Source: §6.3 · p. 7, Figure 4 · p. 8

6.4 Comparison with our April GPT-Image-2 release

• Held-out April images yield positive contrasts of +5.8 pp for Effort and +4.7 pp for DoU, while calibration-resampled Bonferroni intervals also remain positive for both detectors.The held-out check reuses September images; calibration intervals omit DoU's latent-sampling variability.Source: §6.4 · pp. 8–9
• In the X-only comparison, September flag rates are higher by +6.1 pp for Effort and +3.6 pp for DoU; DoU's adjusted interval narrowly excludes zero.The differences are +6.1 pp [+2.7, +9.6] for Effort and +3.6 pp [+1.0, +6.2] for DoU; the comparison is exploratory.Source: §6.4 · p. 8, Table 4 · p. 9
• Within September, widening from X to the pooled collection shifts flag rates by up to 14.9 pp; metadata-based subsets keep Effort and DoU differences positive, but only X has a difference interval.For four detectors, the source-composition shift exceeds every April–September difference in Table 4.Source: §6.4 · p. 9

7 Limitations

• Attribution labels are not independently verified, controls are unmatched on content and processing, and detector-training overlap and perceptual duplicates remain unverified.The evaluation covers six checkpoints, and its primary intervals condition on calibrated thresholds; DoU’s latent-sampling variation is excluded.Source: §7 · p. 9
• Agreement within 3.5 percentage points on five detectors’ GenImage accuracies cannot validate preprocessing for the platform-processed collection.DoU has no tabulated reference, and B-Free’s reference comes from another group.Source: §7 · p. 9
• DoU’s adjusted interval narrowly excludes zero but omits latent-sampling variability; the other four detectors show no detected difference, which does not establish equivalence.The second seed shifts the pooled September flag rate by 0.43 pp at the same threshold.Source: Figure 5 · p. 9
• The April and September X cohorts differ in admission rules, image-form filtering, and collection window, so their contrasts identify collection differences rather than an upgrade effect.The observed source shifts are plausible alternative explanations; Effort has the largest cross-release difference and the largest deviations from published per-generator results.Source: §7 · p. 9
• Historical audit coverage, selection, source comparability, near-duplicate removal, and snapshot coverage remain limitations; platform AI labels do not resolve generator version or establish coverage of relevant posts.The study therefore still needs human review of the frozen cohort’s evidence and curation decisions; visual review alone cannot authenticate generator version.Source: §7 · pp. 9–10

8 Conclusion and Release

• The frozen snapshot contains 3,478 images from 2,440 posts across 8 sources, with caption claims and host records retained as distinct admission evidence rather than independent authentication.The inspected C2PA generator-version field does not distinguish the tested 2.5 endpoints from GPT-Image-2.Source: §8 · p. 10
• Observed content profiles and detector flag rates differ across sources, supporting retention of source and attribution fields without estimating platform-wide preferences or activity.Visual review alone cannot authenticate generator version.Source: §8 · p. 10
• The collection’s flag rates are below GenImage recall for all six detectors, while exploratory X comparisons show a higher September Effort flag rate and a suggestive DoU difference, not an identified upgrade effect.Attribution, content, and processing differences remain compatible explanations; matched prompts, verified generator records, and controlled image processing would be needed to identify an upgrade effect.Source: §8 · p. 10
• The collection and curation code records queries, endpoints, gate expressions, deduplication, and the screenshot prompt; the dataset and code are released at scam.ai/research.The local snapshot contains media, metadata.csv, and a datasheet.Source: §8 · p. 10

Appendix

• The appendix groups supporting material on collection design, curation and audits, dataset evidence, and detector evaluation.It covers nine instruments, attribution rules, historical audits, content credentials, and cross-release comparisons.Source: Appendix · p. 12

A Model context at launch

• On 2026-09-07, both 2.5 models ranked above GPT-Image-2 on editing and text-to-image Arena leaderboards; the system card reported lower unsafe-presented rates.Editing Elo was 1520±9 and 1491±9 versus 1461±3; text-to-image Elo was 1421±13 and 1399±13 versus 1381±4, for Sunburst, Flare, and GPT-Image-2 respectively.Source: A Model context at launch · p. 12

B The billed collection stack

• Five of nine routes share a commercial gateway, reducing the client code needed to add platforms; Pixiv, note.com, and NightCafe use public JSON endpoints directly.The gateway log contains 932 calls across five endpoints.Source: B The billed collection stack · p. 12
• $0.0015 per gateway page of twenty X posts is about 67× cheaper per post than the official endpoint’s $0.005 per post read.These list-price ratios assume full pages and exclude repeated posts, failed calls, and deduplication; they are not costs per unique released image.Source: B The billed collection stack · p. 12
• 502 of 1,524 gateway-admitted posts appear in retained official results, but incomplete retention and different query schedules prevent treating the overlap as a causal decomposition of index coverage.The remaining 1,022 gateway-admitted posts were not observed in retained official results.Source: B The billed collection stack · p. 12

C What differs between the nine instruments

• Shared launch cutoff, deduplication, and image-form filtering do not make source comparisons equivalent: query, time-filter, format, language, and attribution rules differ across routes.A hashtag scraper and a model-tagged gallery also have different sampling frames from full-text search.Source: C What differs between the nine… · p. 12

D The two X routes, and the two sources unlike the rest

• 502 of 1,524 gateway-admitted posts appear in retained official results, and 168 pass both routes’ admission rules; unmatched records cannot be uniquely assigned to index coverage.Neither route is a census, and the retained official files do not preserve every billed read.Source: D The two X routes, and the two… · p. 12
• Reddit yielded 11 retained images, while Instagram’s third-party hashtag scraper gives it a different sampling frame from full-text search.Of 193 Reddit posts read, only 10 had a directly downloadable image; Instagram returned 97 posts, 65 naming 2.5, all with images.Source: D The two X routes, and the two… · p. 12

E Query families, dropped families and animated posts

• The official X query set adapted four April query families, dropped two version-silent families, and added codename and anti-noise families; dropping the largest April image source reduced coverage but avoided admitting posts without explicit version evidence.In the pilot, 35 of 106 posts from the version-silent family explicitly named GPT Image 2.Source: E Query families, dropped families and… · p. 13
• X animations are retained as sampled frames, with consecutive near-duplicates dropped and each animation capped at twelve frames.The release contains 69 frames from 17 animations alongside 3,409 stills; frames are marked and correlated within an animation.Source: E Query families, dropped families and… · p. 13

F Version attribution: caption expressions and three tiers

• Caption-attributed records use case-insensitive version expressions, whereas NightCafe host-attributed records require an allowed model-field value; the 2.0 alternatives guard against matching 2.5 or comparison posts.Host titles are not used as a fallback version signal.Source: F Version attribution: caption… · p. 13, Figure 6 · p. 14

F.1 Three attribution tiers

• The three attribution tiers contain 371 official-X images, 1,749 caption-regex-only images, and 1,358 host-attributed images; 12 separately labelled videos are excluded from image analyses.NightCafe admits records based on an allowed gptImageModel value, while caption silence is permitted in that tier.Source: §F.1 · pp. 13–14
• Re-scoring the 1,749 caption-only images admits 372 (21.3%) under the official tier’s classifier, so the tiers should remain selectable rather than treated as interchangeable identity evidence.The classifier is English- and Japanese-led, and missing language fields limit its use; this comparison is not validation.Source: §F.1 · p. 14

G Deduplication and cross-source overlap

• 1,281 near-duplicate images were removed using a 64-bit average hash at Hamming distance ≤4; 570 matches were within the same post and 711 across posts.The 268 within-run, same-post drops include 255 stills and 13 GIF frames; only 38 stills were X rows, and treating them all as edit-pair members is an unverified upper-bound scenario.Source: G Deduplication and cross-source overlap · p. 14
• 259 of 3,478 released images (7.4%) had a near-duplicate on at least one other source, but the hash criterion does not establish semantic identity.The cross-source analysis applies the merge’s same 64-bit average-hash threshold and stores matches separately from the metadata.Source: G Deduplication and cross-source overlap · p. 14

I.2 Screenshot filter, per class

• Historical screenshot audits found 1.5% model disagreement among predicted standalone images, but the sampled strata exclude NightCafe and do not validate the current cohort.The reported intervals describe model-label proportions, not accuracy against ground truth; the historical frame also lacks exact run timestamps and a complete immutable sampling frame.Source: §I.2 · p. 15, I Historical audit design and detail · p. 15

I.3 Subject taxonomy, per class

• In 100 historically sampled images, CLIP agreed with the auditor on 23.5% of text-graphic and 37.5% of product/UI mockup labels, limiting confidence in category prevalence.April comparisons also differ in image-form filtering and CLIP templates, and the historical audit neither calibrates the expanded release nor bounds its true category shares.Source: §I.3 · p. 15

I.4 Language field and creation claims

• The balanced text audit read 67.3% of sampled posts as presenting an image the author made, but this pooled rate does not estimate the release’s creation-claim share.It excludes records without language fields, including NightCafe, and does not validate wording regexes as measures of user intent.Source: §I.4 · pp. 15–16

H The screenshot filter: prompt, cost and labels by source

• The screenshot filter retains only the standalone class before 16 targeted visual-review exclusions; Table 7 separates X gate tiers and uses host attribution for NightCafe.Its automated share is not a platform population rate, and Reddit’s 18 gated images are too few to read as a rate.Source: Table 7 · p. 16

J.1 Timing and wording

• Released posting timestamps span about 51.1 hours after the announcement, but observed ranges do not establish complete query coverage or generation times.Served dimensions are available for only 371 official-API rows; dimensions alone cannot identify original outputs or later transformations.Source: §J.1 · p. 16, §J.2 · p. 16

K Content credentials in detail

• Pixiv byte scans found C2PA markers in 31 of 66 files, but marker and string matches do not verify signatures or confirm 2.5 attribution.The inspected API-ledger version field was 2.0 throughout its 1,007 files, and the expanded NightCafe cohort did not receive the same credential inspection.Source: K Content credentials in detail · p. 16

L The source landscape, platform by platform

• The collection combines caption-based sources with host-attributed NightCafe images, while platform probes are observations rather than an exhaustive survey.Pixiv’s AI declaration is uploader-supplied and does not name a generator version; delivery paths also differ in whether files may be re-encoded.Source: L The source landscape, platform by… · pp. 16–17
• NightCafe’s host-recorded model field yielded no 2.5 items in a 9 September probe of 240 newest creations, then 1,449 allowed-value creations in the 10 September crawl.The observed feed change does not precisely measure adoption time; the frozen release includes 1,358 NightCafe images and 12 separate videos.Source: L The source landscape, platform by… · p. 17

M What the release directory contains

• The release directory records per-row source, time, language, attribution route and gate tier, but does not provide signature status or per-item human adjudication.Host-attributed rows include model and endpoint fields; cross-source hash matches are separate analysis outputs, not an attribution guarantee.Source: M What the release directory contains · pp. 17–18

N Aggregate subject comparison

• Figure 8 compares CLIP-assigned subject shares with three April categories, but differing templates and image-form filters limit the comparison.NightCafe contributes 39.0% of images but 77.0% of CLIP-assigned fantasy scenes, illustrating the source-mixture dependence of the observed profile.Source: N Aggregate subject comparison · p. 18

O Detector Evaluation Details

• Held-out artwork false-positive rates range from 0.5% to 98.3% across detector intervals, while held-out photograph intervals condition on fitted calibration thresholds.The broad artwork range makes collection flag rates alone insufficient to establish detector quality.Source: O Detector Evaluation Details · p. 19
• Removing six cross-split duplicate candidates and recalibrating on 997 photographs yielded held-out photograph FPRs of 6.02%, 4.21%, 3.51%, 5.02%, 5.12%, and 4.31%.The checks do not exclude residual perceptual overlap or detector-training overlap.Source: O Detector Evaluation Details · p. 19
• With DoU calibrated separately for each pass, seeds 42 and 43 produced collection flag rates of 17.4% and 17.9%, with photograph FPRs of 4.4% and 6.9%.Holding the primary threshold fixed, the seed-43 flag rate was 17.8%; these two runs do not estimate the full seed distribution.Source: O Detector Evaluation Details · p. 19

O.1 Harness reproduction on GenImage

• On the frozen GenImage sample, the harness reached pooled AP 0.989 and accuracy 89.8%, versus published mean AP 0.994 and accuracy 0.912.The comparison is not fully matched, and the residual gap remains unresolved; AP aggregation and sample composition differ from the published evaluation.Source: §O.1 · p. 20

O.2 Historical-cohort comparison

• The six cohort contrasts are exploratory, and shared creators or residual image similarity may create dependence beyond the post clusters.April and September admission rules are not matched or independently verified, and the comparison does not establish a causal effect.Source: §O.2 · p. 21
• In the exploratory September-X-minus-April-X comparison, only Effort and DoU difference intervals remain positive under the six-detector family-wise adjustment.The calibration-sampling analysis is conditional on the five selected photograph pools and excludes attribution error, selection bias, alternative controls and DoU stochastic variation.Source: §O.2 · p. 22, Table 12 · p. 22

O.3 Further limitations of the collection

• Historical audits do not validate the current cohort: automated stages lacked human adjudication, and screenshot and taxonomy audits contain no NightCafe examples.The authors identify a human audit across the frozen cohort and attribution tiers as future work.Source: §O.3 · p. 22
• Source yields do not measure relative platform activity because queries, time filters, budgets and image-form retention differ across routes.Caption admission omits version-silent posts, host galleries use a different selection rule, and source fields do not remove these confounders.Source: §O.3 · p. 22