PaperlayerView exact revision on arXiv

Source-linked AI summary

Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them

Yicheng Bao, Zhenkun Gao, Xiahui Guo, Mingqian Yang, Xueheng Li, Bangwei Liu, Mingang Chen, Lijun Li, Xuhong Wang, Xin Tan

arXiv:2609.32763v1cs.AI

TL;DR

Existing benchmarks rely on editing traces, caption mismatches, visible anomalies, or external references, leaving open whether models use their own knowledge to verify recognized images. Mandela-Bench tests that question with fact-contradicting edits, matched controls, and evidence-grounded scoring. Across 36 models, recognition can bias judgments toward remembered images, while cropping away familiar composition improves knowledge-grounded detection when the true caption is supplied.

  • Problem

    Existing approaches rely on generator traces, image-caption inconsistencies, visible anomalies, or external references rather than testing use of a model’s own facts to verify a recognized image.

    Source: §1 · p. 2

  • Method

    Mandela-Bench pairs fact-contradicting canonical-image edits with matched anchor-free controls and scores whether explanations identify the manipulated entity or contradiction.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across 36 models, verification often follows remembered canonical images, and only one reaches at least 50% KGR under prompted verification.

    Source: §7 · p. 10

  • Takeaways & Limitations

    With the same true caption, cropping away recognizable composition yields more knowledge-grounded verdicts than showing the full image.

    Source: §7 · p. 10

    The benchmark depicts public figures only, and human labeling was performed by the authors.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image-caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.

1 INTRODUCTION

The paper asks whether multimodal models use their own factual knowledge to verify recognized canonical images when no caption, reference database, or external tool is available. Mandela-Bench and its diagnostic tests reveal that recognition can bias verification toward a remembered image rather than the edit actually shown.

Motivation

  • Canonical images include historical photographs and other images treated as records, such as championship photographs, film stills, and famous paintings.The paper frames these images as part of how societies remember who was present, what was signed, and what a banner said.Source: §1 · p. 1

Benchmark

  • Mandela-Bench contains 1,359 knowledge-only forgeries, 148 anchor-free controls, and 474 untouched originals, spanning six knowledge families and six image domains.Each forgery violates exactly one verifiable fact, while controls use matched editing operations without factual contradictions.Source: §1 · p. 2

Motivation

  • The benchmark tests single images whose manipulation evidence is a fact about the depicted referent, addressing cases not covered by approaches relying on traces, caption inconsistencies, visible anomalies, or external references.The motivating concern is that seamless edits paired with correct captions can evade pixel-level detectors while altering the apparent record.Source: §1 · pp. 1–2

Evaluation approach

  • An evidence ladder credits a verdict only when its explanation identifies the manipulated entity or contradicted fact, while diagnostic probes separate fact possession, recognition, perception, and verification.De-recognition conditions crop away familiar composition without changing the manipulated region; a correct caption supplies event and date without restoring the visual recognition cue.Source: §1 · p. 2

Findings

  • Across 36 models, recognition often shifts verification toward the remembered canonical image, and only one model reaches at least 50% KGR under prompted verification.Models may name erased people and accept full forgeries even when rejecting replacement faces in isolation; cropping away the recognizable composition while retaining a true caption improves knowledge-grounded detection.Source: §1 · p. 3

2 RELATED WORK

Prior image-forgery methods draw on traces, cross-modal inconsistencies, visible anomalies, or external references. Mandela-Bench instead examines whether models use facts they may already know about a recognized image.

Image-forgery detection

  • Existing detection methods use generator traces, image-caption inconsistencies, visible implausibilities or forensic disruptions, and retrieved authentic references.Related benchmarks evaluate synthetic or otherwise cue-bearing content.Source: §2 · p. 3

Mandela-Bench setting

  • Mandela-Bench targets a seamless single image without a caption or external reference, where decisive evidence is a fact the model may already know about its referent.This differs from the evidence sources used by the described prior methods.Source: §2 · p. 3

Visual evidence and prior knowledge

  • Related work studies conflicts between visual evidence and parametric priors, controlled visual counterfacts, and failures on small visual details despite attention to the relevant region.This paper instead studies real canonical images, varies recognizability while preserving the edited region, and separately probes perception, fact possession, and verification.Source: §2 · p. 3

3 MANDELA-BENCH: FORGERIES THAT ONLY KNOWLEDGE CAN CATCH

Mandela-Bench is designed to test whether models use facts they possess to verify recognized images, while varying recognizability without changing the edited region. Its edits are factually contradicted but otherwise plausible, and controls match editing traces without introducing contradictions.

3 MANDELA-BENCH: FORGERIES THAT ONLY KNOWLEDGE CAN CATCH

  • The benchmark varies recognizability without altering the edited region and requires that forgeries be detectable only through facts about their depicted referents.The knowledge families cover people, objects, places, visible text, event attributes, and works of art or film.Source: §3 · p. 3, §3.1 · p. 4

3.1 KNOWLEDGE-ONLY FORGERIES AND THEIR CONTROLS

  • A knowledge-only forgery contradicts exactly one recorded, externally verifiable fact while remaining otherwise indistinguishable from an authentic image of the same scene.Seeds are anchored to knowledge-base entries, and edits are seamless in era, medium, and style and intrinsically plausible absent the contradiction.Source: §3.1 · p. 3
  • Anchor-free K0 controls use the same seeds, editors, and operations as forgery items, but add no fact-bearing element; each seed also contributes an untouched authentic original.Matched traces and composition make detection differences between K0 and a knowledge family reflect the contribution of knowledge; absent-celebrity items replace a present public figure with a synthetic face.Source: §3.1 · p. 4

3.2 GENERATION AND QUALITY CONTROL

  • Generation links canonical images to knowledge-base entries, verifies proposed facts against Wikidata or curated tables, and uses one editor for both knowledge-anchored edits and controls.Automated checks, manual review, and pixel auditing support quality control; two pixel detectors remain near chance on 1,981 content-deduplicated images.Source: §3.2 · p. 4

4 EVALUATION PROTOCOL: PROBES, CONDITIONS, AND SCORING

The evaluation varies prompts and visual context, then scores both verdict accuracy and the evidence behind each verdict. Its main knowledge-grounded measure requires identifying the manipulated entity or fact, not merely calling an image fake.

4.1 PROBES

  • Seven probes distinguish default and prompted verification from identity, face, recognition, text-only, and adversarial-suggestion tests.Naming an erased person in the identity probe provides a behavioral signature of reconstruction from memory; the face probe presents original and edited face regions alone.Source: §4.1 · pp. 4–5

4.2 DE-RECOGNITION CONDITIONS

  • Five de-recognition conditions vary the familiar composition and event context, including full-image and crop views with or without the true event-and-date caption.Crop+Caption restores event context without restoring the composition, while horizontal mirroring perturbs appearance while largely preserving scene recognition; authentic images receive the same operations.Source: §4.2 · pp. 5–6

4.3 SCORING

  • The evidence ladder ranges from incorrect verdicts and generic artefact claims to localization, entity identification, and explicit factual contradiction.E3 requires identifying the manipulated entity; E4 additionally requires stating the contradiction, and an LLM judge compares explanations with recorded contradictions.Source: §4.3 · p. 6
  • KGR is the fraction of forged items, including controls, scored E3 or above, whereas E4 is the stricter criterion requiring the recorded contradiction itself.Balanced accuracy averages forgery detection and authentic-image accuracy; the elicitation gap compares prompted with default KGR, and recognition penalty compares Crop+Caption with Full balanced accuracy.Source: §4.3 · p. 6

5 RESULTS

Across 36 models, prompting increases detection more than fact-grounded judgments, and performance varies sharply by edit type and recognizability. Cropping away a familiar composition can help models use event context, while erased people may still be named from memory.

5 RESULTS

  • Prompted verification raises KGR by 3.1 to 16.2 points for 29 models, but detection rises more than KGR for 26, and balanced accuracy falls for 13.Accuracy on authentic images also falls for 27 models.Source: §5 · pp. 6–7, Figure 4 · p. 7
  • Only one model exceeds 50 KGR under prompted verification; Gemini-3.8-Flash reaches 55.1, while 28 of 36 remain below 25.Table 2 reports the main release-set results for 15 models; full results cover all 36.Source: §5 · p. 6, Table 2 · p. 7
  • Anchor-free control KGR is 0.0–2.0 for every model, while object edits tied to factual contradictions reach a KGR of 79.2.For people-family K1, KGR stays below 10 for all but Gemini-3.8-Flash, at 27.5, despite strong text-only fact accuracy.Source: §5 · p. 7
  • For the 21 recognizing models, Crop+Caption raises KGR for all, while Full+Caption improves KGR on no more than 2 of 145 forgeries.Under Crop+Caption, KGR reaches 1.3 to 8.9 times its Full+Caption value.Source: §5 · p. 8
  • Six models name the erased person on at least half of absent-celebrity items, peaking at 72.7%; erase-to-swap detection ratios are 0.41 for frequent namers and 0.80 for rare namers.The ratio difference remains after controlling for edit size, image domain, and pixel-detector scores.Source: §5 · pp. 8–9, Table 3 · p. 8
  • On the same items, Kimi-K3 and Qwen3.6-27B distinguish replacement faces in isolation on 55 of 119 and 47 items, yet accept the full photographs 63.6% and 69.8% of the time.They name the erased person in 30.9% and 30.2% of answers, respectively.Source: §5 · p. 9

6 ANALYSIS

Recognition-related errors persist across model scale and additional reasoning, even when models possess relevant facts or can perceive visual differences. The paper identifies verification after recognition as a key bottleneck and highlights recognizable people-family forgeries as the most relevant setting.

6 ANALYSIS

  • Among 21 models that recognize at least half of the images and detect at least 10% of forgeries, the recognition penalty ranges from +0.2 to +18.7 points.Larger models achieve higher KGR but do not eliminate the penalty; Qwen3.5-122B reaches 31.5 KGR with a +10.5 penalty.Source: §6 · p. 9
  • Models can possess relevant facts and perceive visual differences yet fail to reflect that information in final judgments after recognizing a scene.The authors identify whether verification is invoked after recognition as a key bottleneck.Source: §6 · p. 9
  • For recognizable, correctly captioned people-family forgeries, no model grounds more than one third of forgeries, and adding the true caption lowers detection more often than it raises it.Anchor-free edits by the same editor are rarely detected, so suppressing the canonical-image prior alone is not sufficient.Source: §6 · p. 9

7 CONCLUSION

Across 36 multimodal models, recognizing a canonical image can shift verification toward the remembered version rather than the edited visual evidence. The authors conclude that these failures are not explained simply by missing knowledge or poor perception, and that verification after recognition may be the bottleneck.

7 CONCLUSION

  • Only one of 36 models meets the KGR criterion on at least half of forged images, even under explicit verification prompts.Models may name erased people and accept a full forgery despite rejecting its replacement face in isolation; cropping away the recognizable composition improves knowledge-grounded verdicts over showing the full image with the same true caption.Source: §7 · p. 10
  • The benchmark is limited to public figures; living heads of state or government and contemporary living singers and actors were excluded from insertion.Sources include Wikimedia Commons and open-access museum, film-still, and meme-template archives; politically charged meme templates were excluded.Source: ETHICS STATEMENT · p. 11

Appendix

  • The release documents each item’s source, edit specification, recorded contradiction, and footprint, alongside prompts, judge materials, and model settings.The appendices describe construction and quality control, evaluation instruments, full-roster results, and additional analyses.Source: REPRODUCIBILITY STATEMENT · p. 11, Appendix · p. 16

B FORMAL DEFINITION OF A KNOWLEDGE-ONLY FORGERY

  • A knowledge-only forgery inserts an anchored entity while changing exactly one checkable fact, and is designed to be seamless, intrinsically plausible, and pixel-clean.The benchmark distinguishes six knowledge families; its K0 controls use matched edits that introduce no factual contradiction.Source: B FORMAL DEFINITION OF A… · pp. 17–18

D A WORKED EXAMPLE

  • The worked example contrasts an Orwell insertion with a matched unnamed-face control, then varies recognizability while keeping the edited region visible.The crop removes much of the recognizable composition; the caption conditions provide the same event-and-date context.Source: D A WORKED EXAMPLE · p. 19, Figure 6 · p. 19

F JUDGE RUBRIC AND CALIBRATION

  • The judge grades the evidence behind a verdict, from generic or misdirected claims to a correct identity and event-specific contradiction.KGR credits explanations identifying the manipulated entity or violated fact; human calibration agreement within one level was 96.1%.Source: F JUDGE RUBRIC AND CALIBRATION · pp. 20–21

I ITEM-LEVEL ASSOCIATION BETWEEN RECOGNITION AND DETECTION

  • For the people family, recognition is associated with lower detection: pooled odds ratio 0.65 (95% CI 0.51–0.82, p = 0.0004).The coefficient is negative for 20 of 24 recognizing models; no comparable association appears for class-level families, and the authors describe this as observational support for the controlled pattern.Source: I ITEM-LEVEL ASSOCIATION BETWEEN… · p. 22

J CAPTION CONTROLS: THE WRONG EVENT AND THE CLASS-LEVEL

Wrong captions raise detection but do not increase knowledge-grounded detection, while true captions do not reduce detection for class-level facts. For GPT-5.6, true captions improve detection on visible-text and event-attribute forgeries.

J CAPTION CONTROLS: THE WRONG EVENT AND THE CLASS-LEVEL

  • Wrong captions raise detection to 93.8 and 91.0, but KGR against the recorded contradiction does not increase.Authentic accuracy falls to 19.4 and 16.1; the pattern suggests models often detect a mismatch between the caption and the canonical scene they recognize, rather than the image edit itself.Source: J CAPTION CONTROLS: THE WRONG EVENT… · pp. 22–23
  • For GPT-5.6, true captions increase detection by 13.9 points for visible-text forgeries and 13.1 points for event-attribute forgeries.KGR changes from 69.4 to 80.6 for visible text and from 50.3 to 63.4 for event attributes.Source: J CAPTION CONTROLS: THE WRONG EVENT… · p. 23

K ADDITIONAL VALIDITY CHECKS

Recognition is associated with verdicts shifting toward the remembered label for both authentic images and forgeries. Additional checks report that saliency effects vary by family and that threshold exclusions have limited impact on reported results.

K ADDITIONAL VALIDITY CHECKS

  • For most models, recognized authentic images are called authentic more often than unrecognized ones, including gains of 9.1 points for Kimi-K3 and 13.8 for Gemini-3.8-Flash.This complements the forgery pattern: recognition moves verdicts toward the remembered label in both directions.Source: K ADDITIONAL VALIDITY CHECKS · p. 23
  • Within the people family, replacing a main figure is easier to detect than a background insertion: Kimi-K3 reaches 69.9 versus 35.9, and Gemini-3.8-Flash 83.9 versus 73.1.Across families, the C1–C3 difference is within five points for most models because the background class is dominated by class-level families, where KGR is higher.Source: K ADDITIONAL VALIDITY CHECKS · p. 24
  • Excluding 41 near-threshold cases among 397 people-family forgeries changes prompted-verification detection by at most 1.7 points and KGR by at most 0.5 for any model.E4 rates are lower than E3+ KGR rates; under prompted verification, E4 is 37.2 for Gemini-3.8-Flash, 29.7 for Kimi-K3, and 26.3 for GPT-5.6.Source: K ADDITIONAL VALIDITY CHECKS · p. 24

L FOOTPRINT COVARIATE ANALYSIS

The swap–erase detection difference remains after accounting for edit footprint and detector scores, supporting an identity-related effect beyond footprint differences. The analysis covers only models meeting specified non-degeneracy criteria.

L FOOTPRINT COVARIATE ANALYSIS

  • Median pixels changed outside the annotated region are 4.8% for swaps, 0.6% for erasures, and 0.3% for K0 edits.The regression tests whether footprint differences explain the swap–erase detection gap.Source: L FOOTPRINT COVARIATE ANALYSIS · p. 24
  • Across 17 non-degenerate models, the swap–erase contrast was positive in all and significant under both footprint definitions in 16; adding footprint and detector scores did not remove any identity effect.Non-degenerate models named the erased person in at least a quarter of default-probe answers and detected at least a quarter of swaps; the swap coefficient changed by −26.8% to +11.1%.Source: L FOOTPRINT COVARIATE ANALYSIS · p. 24

M PREREGISTERED DECISION RULE FOR THE ABSENT-CELEBRITY RESULT

The absent-celebrity analysis preregisters thresholds for classifying verification collapse, no collapse, or an intermediate result. Applying the rule, 21 of 30 eligible models meet both collapse criteria.

M PREREGISTERED DECISION RULE FOR THE ABSENT-CELEBRITY RESULT

N FACE PROBE

The face probe tests whether models distinguish original and replacement faces in isolation, then relates those results to full-image judgments. Its analyses also report the evaluated sample and identity-recognition patterns.

N FACE PROBE

  • The face probe asks whether each isolated original and replacement face matches the named person; a pass requires yes for the original and no for the replacement.Faces are shown at native resolution on a neutral canvas.Source: N FACE PROBE · p. 25
  • The table’s finding is that remembered people follow the composition: for 116 peripheral people-family forgeries with named principal figures, it compares full-image and edited-region crops by whether answers name a principal figure.Source: Table 9 · p. 26
  • The probe began with 159 verified items and retained 119 pairs after excluding mislabeled, unusable, or obscured faces.A later release filter reduced the verified absent-celebrity set to 132, but probe results use the 119 evaluated pairs.Source: N FACE PROBE · p. 26
  • On 119 evaluated face pairs, Gemini-3.8-Flash passes 81 and still calls full forgeries authentic 29.6% of the time, naming erased people in 49.4% of answers.Qwen3.8-27B passes 50 pairs and has corresponding rates of 68.0% and 12.0%; GPT-5.6 declines to identify 112 of 119 original faces.Source: N FACE PROBE · p. 26
  • Wrong-name controls yield false-yes rates of 10.0% for Kimi-K3 and 0.0% for Qwen3.6-27B; excluding worn-cue pairs leaves 101 items.On those 101 items, passed full images are called authentic at rates of 63.0% for Kimi-K3 and 63.6% for Qwen3.6-27B.Source: N FACE PROBE · pp. 27–28

O ADDITIONAL ANALYSIS

Additional analyses connect erased-person naming with lower detection and use controls to distinguish perception, fact possession, and recognition effects. A case study illustrates how a model can validate remembered scene details yet accept a forgery.

O ADDITIONAL ANALYSIS

  • Naming the erased person is associated with lower detection, while all 36 models name principal figures less often on crops than full images.These correlations support, but do not replace, the controlled manipulations.Source: O ADDITIONAL ANALYSIS · p. 28
  • Controls show the relevant fact is available and perception can be tested, while the same caption has different effects when the recognizable composition is cropped away.The paper uses “Mandela effect” only as a phenomenological analogy, not as a claim of shared mechanism.Source: O ADDITIONAL ANALYSIS · p. 28
  • In the Truman-photo example, Kimi-K3 checks remembered details but calls the edited image genuine despite the synthetic replacement face.The model says the pictured man is not Truman while returning REAL for the photograph.Source: O ADDITIONAL ANALYSIS · p. 28
  • Across probes, the same model names Truman in the picture but rejects the isolated face as Truman, then rationalizes the conflict without overturning authenticity.GPT-5.6 and Qwen3.6-27B also call the edited image authentic.Source: O ADDITIONAL ANALYSIS · pp. 28–29
  • The supplementary tables cover authentic-image accuracy by domain and recognition, forgery detection and KGR by edit saliency, and E4 rates alongside KGR.They also document family-level default detection and four de-recognition conditions, including parsed-only scoring.Source: Table 13 · p. 29, Table 14 · p. 30, Table 15 · p. 31
  • Other supplementary analyses specify a Full/Mirror/Crop comparison, covariate-adjusted detection regressions, and a preregistered rule for absent-celebrity verification collapse.The regression tables report swap-versus-erase contrasts and changes in the swap coefficient; the decision-rule table reports erase-to-swap ratios and E4 shares.Source: Table 19 · p. 35, Table 20 · p. 36, Table 21 · p. 37

Related papers

Loading 2609.32763v1…