PaperlayerView exact revision on arXiv

Source-linked AI summary

Frontier vision-language models have overtaken young adults at detecting AI-generated portraits -- but not their calibration

Sunwhi Kim, Sunyul Kim, Meounggun Jo, Jini Tae

arXiv:2608.30210v1cs.HCcs.CV

TL;DR

Claims that models “beat humans” often rely on measurements from non-comparable conditions, so this paper compared 19 frontier VLMs with a human task on 198 portraits. June models did not surpass the young-adult ceiling, but July leaders did, while extreme response criteria and example-driven instability persisted.

  • Problem

    Whether VLMs outperform people under directly comparable conditions remains unresolved, as “beat humans” claims often rely on measurements from non-comparable conditions.

    Source: Introduction · p. 2

  • Method

    The paper compared 19 frontier VLMs with a human task by having them judge 198 portraits under a protocol mirroring the human task.

    Source: Introduction · p. 3

  • Results

    June models did not surpass the young-adult ceiling, whereas July leaders exceeded the young-adult mean on single passes, draw averages, and sensitivity.

    Source: Introduction · p. 3

  • Takeaways & Limitations

    The July leaders’ gains did not eliminate extreme response criteria, example-driven instability, or verdict-aligned rationales.

    Source: Introduction · p. 3

    Changing the practice images altered roughly a quarter of per-item verdicts.

    Source: Introduction · p. 3

Abstract

from arXiv · show

AI image generators now create face portraits that are hard to tell from real photographs. Vision-language models (VLMs) are increasingly proposed to flag such images. We benchmarked 19 VLMs on the same 198 face portraits -- real photographs and identity-matched ChatGPT-4o and Imagen 3 versions -- under the same task as our earlier study of 1,667 adults (85% correct overall; accuracy fell steeply with age). The June-2026 cohort of 14 models only matched adults in their 20s-30s. Four weeks later the ceiling broke. Among five July-2026 releases under the identical protocol, gpt-5.6-sol reached 92.8% balanced accuracy (five-draw mean 92.1%), clearly above adults in their 20s (88.5%), and claude-fable-5 detected every AI image while averaging 91.9%. Model sensitivity now exceeds young adults decisively (d' up to 3.4 versus ~ 2.4). What has not been overtaken is human calibration. Model criteria spread from c = -1.10 to +1.45 while humans sit near zero at every age; both new leaders are biased (+0.44, -0.97), and only a few mid-ranked models approach the human balance. Changing the labelled examples still flipped about one answer in four. The best machines now out-see young adults here, without matching the human balance between suspicion and trust.

1 Hwasung Medi-Science University, Dept. of Bio-Healthcare, Republic of Korea · 2 Yonsei University, Graduate School of Engineering, Dept. of Artificial Intelligence, Republic of Korea · Introduction

On a matched portrait task, July 2026 VLMs surpassed young-adult performance after June models only matched it, but example sensitivity and response-bias problems persisted.

Introduction

  • Photorealistic AI portraits circulate in profile pictures, advertisements, and news-like visuals, while human detection of AI-generated media is modest and biased toward accepting content as authentic.AI-generated faces also populate fake social-media profiles at scale [3].Source: Introduction · p. 1
  • Human accuracy averaged 85% and fell from about 88% in participants’ 20s to about 66% in their 60s, making age central to the model comparison.The reference study included 1,667 adults aged 20–69.Source: Introduction · p. 2
  • The benchmark used a curated, balanced task of identity-matched real and AI portraits, not open-ended detection of arbitrary fakes in the wild.Versions of an identity were never shown together; each image was judged on its own.Source: Introduction · p. 2
  • Because model–human comparisons often mismatch stimuli, response formats, and scoring, the study treats robustness, bias, calibration, and explanation evidence as necessary dimensions beyond accuracy.VLMs are proposed as general-purpose visual reasoners that may transfer to forensic judgments without task-specific training.Source: Introduction · pp. 2–3
  • Nineteen frontier VLMs judged 198 human-study portraits using the same few-shot examples and one-at-a-time REAL/AI task, enabling comparison with age-stratified human results.The protocol reproduced the human session composition of 10 real, 5 ChatGPT-4o, and 5 Imagen 3 images.Source: Introduction · p. 3
  • Changing practice images altered roughly a quarter of per-item verdicts; model biases reached human-absent extremes, confidence was largely non-diagnostic, and cited cues tracked verdicts rather than image truth.These failure patterns persisted in the new leaders.Source: Introduction · p. 3

Results

The Results section examines model performance on the matched portrait benchmark, including the June-to-July accuracy shift and July leaders’ sensitivity relative to young adults. It also covers response bias, sensitivity to few-shot examples, confidence calibration, rationales, and age-persona prompts.

A human-comparable benchmark for vision–language models

  • Each of 19 models judged the same 198 main portraits once, after receiving four labelled practice images, under a protocol matched to human observers.The model responses included a REAL/AI answer, confidence rating, and one-sentence rationale.Source: Figure 1 · p. 4, A human-comparable benchmark for… · p. 4

The young-adult ceiling held in June 2026 — and broke in July

  • 92.8% (gpt-5.6-sol) exceeded adults in their 20s (88.5%), while claude-fable-5 scored 88.6% and detected every AI image (132/132).By contrast, June leaders gpt-5.5 and qwen3-vl-235b each scored 87.1% and were statistically indistinguishable from young-adult means.Source: The young-adult ceiling held in June… · p. 5, Figure 2 · p. 6

Instability persists: few-shot examples still destabilize model judgments

  • Across five varied example draws, gpt-5.6-sol averaged 92.1 ± 4.0 and claude-fable-5 91.9 ± 2.3, yet 25.7% of item answers flipped across the fleet.The accuracy spread mixes example effects with changes to the judged image set; the flip rate also includes run-to-run noise.Source: Instability persists: few-shot… · p. 7, Figure 3 · p. 8

Models exhibit extreme, non-human response biases

  • Model criterion ranged from c = −1.10 to +1.45 while human groups stayed near c ≈ 0; July leaders reached d′ up to 3.4 versus young adults’ pooled-count d′ of 2.41.The leaders were also biased: claude-fable-5 had c = −0.97 and gpt-5.6-sol c = +0.44.Source: Figure 4 · p. 9, Models exhibit extreme, non-human… · p. 9
  • d′ reached 3.13 (corrected; CI [2.65, 3.75]) for gpt-5.6-sol and 3.41 (CI [3.10, 3.75]) for claude-fable-5, whose c values (+0.44, −0.97) yielded fable-5 balanced accuracy of 88.6%.Both d′ interval lower bounds exceeded the young-adult value, but fable-5’s balanced score was no better than that of a human in their twenties.Source: Models exhibit extreme, non-human… · pp. 9–10

Model confidence: inflated in the June cohort, improving in the July leaders

Models cite human-like cues…

  • Models mentioned eyes in only 10.3% of rationales, compared with 42.8% for humans, while expressions of intuition or uncertainty were essentially absent from model rationales (0%).Source: Models cite human-like cues… · pp. 11–12

…but the cues are verdict markers, not evidence

  • Over-smoothness predicted +24.4 pp accuracy on AI images but −77.6 pp on real photos, while natural/realistic cues showed the reverse pattern; only hair cues were positive in both classes.Conditioned on verdict, over-smoothness and uncanny/too-perfect cues retained modest validity for AI verdicts (+12.2 pp and +9.8 pp), while skin texture did so for REAL verdicts (+25.9 pp).Source: …but the cues are verdict markers, not… · pp. 12–13, Figure 7 · p. 13

Models inherit the human difficulty ordering across generators

Age-persona prompting does not reproduce the human age gradient

  • Age-persona prompts shifted balanced accuracy by ≤3.8 pp and per-item answers by ≤6.7%; gpt-5.5 scored 90.0% under a 60s persona versus 87.5% under a 20s persona, opposite to human aging.The authors treat age-stratified human results as a reference distribution rather than a target for model role-play.Source: Age-persona prompting does not… · p. 14

Discussion

On this portrait benchmark, July 2026 frontier models surpassed young adults in detecting AI images, while remaining less human-like in decision criteria, stability, and explanations. The findings are bounded by a narrow, time-stamped stimulus and evaluation setup.

Discussion

  • 92.8% single-pass and 92.1% draw-averaged for gpt-5.6-sol, and 91.9% draw-averaged for claude-fable-5, put July leaders above young adults after June models did not.The June leaders were statistically indistinguishable from adults in their 20s–30s; the two July leaders also separated from June leaders in paired tests.Source: Discussion · p. 15
  • Changing labelled-example images changed verdicts on about a quarter of images fleet-wide, including models’ own run-to-run noise; claude-fable-5 was most stable, with a fleet-record 11.4% flips.Source: Discussion · p. 15
  • c ≈ 0 for humans at every age, but model criteria ranged from c = −1.10 to +1.45; claude-fable-5 had d′ = 3.41 versus 2.33 for adults in their 20s and c = −0.97.Despite its higher sensitivity, claude-fable-5 reached only a human-level balanced score because it classified almost a quarter of real photographs as AI.Source: Discussion · p. 15
  • June confidence was inflated by up to +29 pp, whereas July leaders were nearly calibrated and showed the largest observed resolutions.The comparison with human between-participant resolution was qualitative (ρ = 0.379).Source: Discussion · p. 16
  • Model explanations tracked verdicts rather than evidence, while human checklist texture was a positive accuracy predictor (odds ratio 1.36, +3.8 pp).Humans also reported uncertainty, with 49% endorsing feeling or intuition, whereas uncertainty was essentially absent from model rationales.Source: Discussion · p. 16
  • Nineteen models found ChatGPT-4o portraits harder than Imagen 3 portraits, as humans did, though claude-fable-5 tied them at 100%; kimi-k3 placed 12th of 19.A dedicated vision–language architecture outperformed its newer general-purpose sibling by 13.6 pp, and a mid-tier model outperformed its family flagship by 22.7 pp.Source: Discussion · p. 16
  • The benchmark covers one portrait domain, two generator families frozen in July 2025, a balanced single-image REAL/AI task, and a time-stamped model snapshot.The evaluation used one prompt family, temperature 0, and OpenRouter; other instructions, settings, or provider routing could shift absolute scores.Source: Discussion · p. 17
  • Because FFHQ is public and model training data are opaque, contamination cannot be ruled out; it would, if anything, inflate model performance and strengthen the headline conclusion.Source: Discussion · p. 17

Material and methods … Models

The benchmark compared vision-capable models with a recomputed human reference using real portraits and identity-matched AI counterparts. The human task, stimulus set, and model cohorts were defined by specific sampling and testing procedures.

Human reference data

  • Human reference values were recomputed from the earlier study’s public de-identified data [13]; first attempts by participants aged 20–69 yielded N = 1,667.The analytic sample included 1,332 mobile and 335 PC participants, with no further exclusions.Source: Human reference data · p. 18
  • The human sample was self-selected, strongly mobile-skewed, and viewed under uncontrolled conditions; cue endorsements were treated qualitatively because 202 checklist responses contained a serialization fault.The fault was an `on` token from the form’s select-all control, rather than genuine endorsement.Source: Human reference data · pp. 18–19
  • Each human participant judged 20 untimed REAL/AI trials after four practice trials, with each session containing 10 real, five ChatGPT-4o, and five Imagen 3 portraits.Practice identities were excluded from the main trials, and no identity was repeated within them.Source: Human reference data · p. 18
  • Mean accuracy was 85.18%, and age-group accuracy was 88.5% for participants in their 20s versus 65.9% for those in their 60s (n = 79).Source: Human reference data · pp. 18–19
  • Human observers were essentially unbiased at every age (criterion c ≈ 0), while participant-mean d′ fell from 2.33 in the 20s to 0.87 in the 60s.The intervening d′ values were 2.27, 1.76, and 1.24 for the 30s, 40s, and 50s, respectively.Source: Human reference data · p. 19

Stimuli

  • The stimulus pool comprised 70 FFHQ photographs and 140 AI counterparts: 70 generated by ChatGPT-4o and 70 by Imagen 3; excluding practice identities left 198 main stimuli.The main set comprised 66 identities × 3 sources, with AI images generated from the same identities as the real portraits.Source: Stimuli · p. 20

Models

  • Nineteen vision-capable models from nine providers were evaluated in two cohorts: 14 June-2026 models and five releases added in July 2026.The July cohort included gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, kimi-k3, and claude-fable-5.Source: Models · p. 20

Benchmark protocol · Human-comparable scoring · Robustness to few-shot examples (iteration design)

The benchmark used a fixed image-judgment task and varied the labelled practice examples to measure scoring and answer stability. Answers often changed across passes, while comparisons with run-to-run noise showed that the example effect differed by model.

Benchmark protocol

  • Each request combined a fixed system prompt, four labelled practice images, and one target image with a fixed instruction to classify it as REAL or AI-GENERATED and return compact JSON.The labelled examples mirrored the human practice phase, where feedback was given.Source: Benchmark protocol · pp. 20–21

Human-comparable scoring

  • Balanced accuracy was 0.5·(real accuracy) + 0.25·(ChatGPT-4o accuracy) + 0.25·(Imagen 3 accuracy), matching the human session composition.It represents expected per-session accuracy for an observer with the model’s class-specific accuracies.Source: Human-comparable scoring · p. 21
  • A cluster bootstrap resampling 66 base identities together with their three versions (10,000 resamples) yielded intervals within ~1 pp of item-level intervals, with half-widths 3.0–5.9 pp across 19 models.Source: Human-comparable scoring · p. 21
  • Pairwise differences used nominal unadjusted exact binomial McNemar tests on shared valid items, while response bias, overconfidence, resolution, and Brier score summarized additional evaluation dimensions.Response bias was AI-detection accuracy minus real-photo accuracy; overconfidence was mean stated confidence minus balanced accuracy, and Brier score compared stated confidence with correctness.Source: Human-comparable scoring · pp. 21–22

Robustness to few-shot examples (iteration design)

  • Five deterministic varied draws each used four practice identities (two real, one ChatGPT-4o, one Imagen 3), excluded their triplets, and judged the remaining 198 stimuli once; the main hand-selected set was a sixth baseline draw.The five draws added 18,810 judgments across 19 models, with zero errors.Source: Robustness to few-shot examples… · p. 22
  • 25.7% of item answers changed at least once across six passes; strict REAL↔AI reversals were 24.7% fleet-wide and 23.3% across varied draws alone.Any-change was 24.2% across the five varied draws alone.Source: Robustness to few-shot examples… · p. 22
  • 6.8% (claude-sonnet-4.6) and 6.9% (qwen3-vl-235b) disagreement were 3.4–4.5× run-to-run noise, versus 11.4% (gpt-5.5) and 14.4% (gpt-5.4), only 1.2–1.3× noise.The authors interpret the example effect as clearly real for stable models, while most measured instability in OpenAI models was intrinsic run-to-run noise.Source: Robustness to few-shot examples… · p. 22
  • The SD over draws is an upper bound on example sensitivity because four held-out identities differed between draws; identical-item flips still combine example sensitivity and run-to-run noise.The six-pass flip rates include both sources, and the decomposition was restricted to four models.Source: Robustness to few-shot examples… · pp. 22–23

Rationale cue coding and diagnosticity … Statistics

The merged sections describe how model rationales were coded and evaluated, report two prompt-sensitivity pilots, and specify the statistical procedures and signal-detection estimates used in the benchmark. Maximum reasoning effort modestly improved gpt-5.6-sol’s benchmark performance, while demographic role-play prompts did not meet the criteria for expansion.

Rationale cue coding and diagnosticity

  • 3,738 benchmark rationales were multi-label coded into a 15-category cue taxonomy by gpt-5.4-mini at temperature 0, blind to true class and judgment correctness.The taxonomy covered the human study’s strategy-checklist cues [13] and cues salient in model rationales; a rule-based keyword coder provided a cross-check with consistent per-model profiles.Source: Rationale cue coding and diagnosticity · p. 23
  • Cue frequency was the share of rationales mentioning a cue; diagnosticity compared cue-invoking accuracy with base accuracy overall and within true class, while verdict-conditioned validity compared cue-citing with non-citing rationales sharing a verdict.Class-conditioned deltas cited in the text used cells of n ≥ 138; the verdict-conditioned natural/realistic comparison had a smaller 62-REAL-verdict comparator cell without the cue.Source: Rationale cue coding and diagnosticity · p. 23

Reasoning-effort pilot

  • 94.3% balanced accuracy under xhigh (vs 92.8% by default) came with 6.6% answer changes, zero false alarms, and one-image gains in each source class.On AI trials, changes yielded seven corrections and five regressions; mean confidence was 92.7. This single-model, single-condition pilot was reported descriptively.Source: Reasoning-effort pilot · p. 23

Age-persona prompt-sensitivity pilot

  • Persona-prompt shifts did not meet expansion criteria: maxima were 3.8 pp in balanced accuracy, 3.3 pp in said-AI rate, 3.8 pp in confidence, and ≤6.7% item flips.The four-model sample was selected for informative contrast, not representativeness, and tested prompt sensitivity rather than simulation of human ageing.Source: Age-persona prompt-sensitivity pilot · p. 24

Statistics

  • Tests were two-sided at α = 0.05; proportions used Wilson intervals [41], paired comparisons exact binomial McNemar tests, session distributions bootstrap, and calibration the Brier score [43].Session-level distributions used 10,000 bootstrap resamples with a fixed seed.Source: Statistics · p. 24
  • Signal-detection analysis treated AI as the signal; sensitivity was d′ = z(hit) − z(FA), and criterion was c = −½·[z(hit) + z(FA)], with negative c indicating liberal, AI-leaning bias.Human hit and false-alarm rates were computed per participant from their 10-real/10-AI session using the log-linear correction, then averaged within age group.Source: Statistics · p. 24
  • Corrected d′ was 3.41 for claude-fable-5 (95% CI [3.10, 3.75]) and the 95% CIs were [2.65, 3.75] for gpt-5.6-sol and [2.0, 3.2] for June leaders.For claude-fable-5, the perfect hit rate produced uncorrected d′ 5.50 and criterion −2.00, described as clamp artefacts; corrected criterion was −0.97.Source: Statistics · pp. 24–25

Ethics

The study reused approved, consented human reference data without collecting identifying information or new human-subjects data; evaluating publicly available AI models required no additional ethical approval.

Ethics

  • Human reference data were reused from the earlier study [13], which had IRB approval (HSMUIRB-2025-06), digital informed consent, and no collection of identifying information; no new human-subjects data were collected.The present work evaluated publicly available AI models through a commercial application-programming interface and required no additional ethical approval.Source: Ethics · p. 25

Data accessibility

Aggregate results, code, human data and per-item model judgments are released with reproducibility materials, while selected identifiers and rationale texts are withheld to protect the deployed benchmark and image-content privacy.

  • Every plotted value is in tables or reconstructible from released per-item files, including pairwise comparisons, persona summaries and cue profiles.Aggregate per-model, per-class and per-condition results, signal-detection values and cue cell sizes are provided in the supplementary material and repository.Source: Data accessibility · p. 25
  • The stimulus pool, FFHQ attribution and licence metadata, and de-identified human data are available in the prior study’s public repository, release v1.0.1.The benchmark repository is release v1.0.0, archived at DOI 10.5281/zenodo.22148304.Source: Data accessibility · p. 25
  • Collection, coding, statistics and figure-generation code, documentation and environment details accompany a script that reproduces headline statistics from per-item files.Re-rendering figures also requires stimulus images from the prior study repository and local raw runs.Source: Data accessibility · p. 25
  • Per-item judgments, confidence, correctness, cue codes and anonymized identity-cluster codes are deposited under randomized stimulus codes that cannot be joined to public stimulus files.The files cover the main benchmark, iteration draws, identical-prompt control, persona pilot and reasoning-effort pilot; each main-benchmark rationale has 15-category cue codes.Source: Data accessibility · p. 25
  • The stimulus-code mapping, collection seeds and production stimulus identifiers are withheld because they could expose the answer table or help regenerate the mapping.Stimulus class can be recovered by manually matching images to the prior public release, but not in-band during a test.Source: Data accessibility · pp. 25–26
  • Verbatim rationale texts are withheld because they describe image content, while their full cue codes are released; both are available from the corresponding author.Source: Data accessibility · pp. 25–26
  • Publication figures reveal ground truth for a few example stimuli, and the three judgments in figure 6a identify nine items; the authors accept this bounded disclosure for worked examples.The displayed verdicts, confidences and cues identify the corresponding item codes and their six identity-linked counterparts.Source: Data accessibility · p. 26

Declaration of AI use

Generative AI supported manuscript preparation and verification, while authors checked the study’s results and retained responsibility. A separate language model coded free-text rationales as part of the methodology.

Declaration of AI use

  • Anthropic Claude assisted with drafting and editing prose, writing analysis and figure-generation code, and formatting the manuscript.The authors checked and verified study design, analyses, numerical results, and interpretations, and take full responsibility for the content.Source: Declaration of AI use · p. 26
  • Over ten review rounds, Claude panels and an independent OpenAI Codex (GPT-5.6) agent re-derived statistics, probed re-identification risk, and audited internal consistency.These tools raised candidate issues only; authors re-checked each issue and number against their analysis pipeline before revising.Source: Declaration of AI use · p. 26
  • gpt-5.4-mini coded free-text model rationales as a described analysis component, distinct from manuscript preparation.This use was documented in the Material and methods as part of the reported methodology.Source: Declaration of AI use · p. 26
  • claude-fable-5 belongs to the same model family as the writing-assistance tool, but its evaluation used the OpenRouter API under the identical protocol applied to all models.The evaluation had no connection to anySource: Declaration of AI use · p. 26

Authors' contributions

S.W.K. led conceptualization, methodology, software, data collection, analysis, and the original draft; S.Y.K., M.J., and J.T. contributed to methods, interpretation, and review. All authors approved the publication and accepted accountability for the work.

  • S.W.K. handled conceptualization, methodology, software, data collection, formal analysis, and the original draft.Source: Authors' contributions · p. 26
  • S.Y.K. contributed methodology, formal analysis, and writing—review and editing; M.J. and J.T. designed selected experiments, interpreted findings, and reviewed and edited the writing.Source: Authors' contributions · p. 26
  • All authors gave final publication approval and agreed to be held accountable for the work performed.Source: Authors' contributions · p. 26

Supplementary Information

The supplementary material compares model performance with human age groups, tests pairwise separability and prompt sensitivity, and reports latency and cue-coding analyses; human values are recomputed from the public data release of ref. [13].

Supplementary Information

  • Human values are recomputed from the public data release of ref. [13], while model results are aggregate summaries.Supplementary figures accompany the main text, each on its own page.Source: Supplementary Information · p. 30

Supplementary Figure S1

  • The July leaders sit above the human 20s band, while June leaders lie near it.Supplementary Figure S1 shows human accuracy by age group with model reference lines.Source: Figure S1 · p. 31

Supplementary Figure S2

  • Both July leaders beat gpt-5.5 and qwen3-vl-235b in pairwise tests (p < .05), but did not differ from each other.Tests used 176–198 shared valid items per pair after excluding unclear responses; p-values were unadjusted.Source: Figure S2 · p. 32

Supplementary Figure S3

  • Across four models and two age personas, balanced-accuracy shifts were ≤3.8 pp and answer flips ≤6.7%, with no human-like age gradient.The 60-stimulus pilot did not meet pre-specified expansion criteria.Source: Figure S3 · p. 33

Supplementary Figure S4

  • Latency is reported only for completeness and is an API/compute proxy, not comparable to human reaction time.The figure reports model API response latency by stimulus class.Source: Figure S4 · p. 34

Supplementary Figure S5

  • LLM-based coding and rule-based keyword matching yield consistent per-model cue profiles.The figure presents cue frequencies under both coding methods as a method-robustness comparison.Source: Figure S5 · p. 35

Related papers

Loading 2608.30210v1…