PaperlayerView exact revision on arXiv

Source-linked AI summary

False Claims, Credible Images: A Red-Teaming Benchmark for Commercial Image Generators

Zeyu Ye, Yanchun Li, Sibei He, Meng Xie, Hangtao Zhang, Xianlong Wang, Li Zeng, Jiahao Chen, Yichen Wang, Junhui Wang, Ziqi Zhou

arXiv:2610.11112v1cs.CRcs.CV

TL;DR

Commercial image generators can make false claims look like credible evidence, while existing evaluations provide limited coverage of generation-time safeguards for visual misinformation. This paper introduces EPIREAL-BENCH and EPIREAL-ATTACK to assess and probe that risk. Direct prompting already yields an average misinformation realization success rate above 70%, and the attack raises the ASR beyond 95%.

  • Problem

    Existing safety benchmarks focus mainly on harmful visual content, while misinformation benchmarks largely study post-hoc detection and generation-side studies cover specific domains.

    Source: §1 · p. 2

  • Method

    The paper introduces EPIREAL-BENCH, a benchmark of false-claim prompts and generated images, and EPIREAL-ATTACK, a skill-guided black-box prompt-optimization framework.

    Source: §1 · p. 2

  • Results

    Direct prompting achieves an average misinformation realization success rate above 70%, while EPIREAL-ATTACK increases the ASR beyond 95%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results identify visual misinformation as a systematic, previously overlooked alignment gap in commercial image-generation systems.

    Source: §1 · p. 3

    The study is limited to predefined claim categories and visual formats and primarily evaluates misinformation realization and visual credibility rather than downstream effects on human beliefs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Image-generation models can now produce text-rich, natural-looking visual artifacts that are hard to distinguish from real-world evidence, such as news reports and textbook pages. Yet, the same capability introduces a new risk: these models can just as easily fabricate visual misinformation. Even commercial models (e.g., GPT-Image-2) readily produce it. Curiously, we find that these models can recognize a claim as false when asked, yet still render that very claim as credible visual evidence. This discrepancy points to a blind spot in current alignment: safeguards judge what an image shows, not what it asserts; however, existing red-teaming benchmarks target conventional harmful content, such as violent or explicit imagery, and say little about where the alignment boundaries lie for visual misinformation, especially in commercial models. To fill this gap, we introduce EpiReal-Bench, the first systematic benchmark for evaluating visual misinformation risks in commercial image generators, comprising 10k false-claim prompts and 10k corresponding generated images that span 10 real-world claim categories and 10 credible visual formats. We further introduce EpiReal-Attack, a skill-guided black-box optimization framework that uses Pareto-based selection and multimodal feedback to identify commands that bypass alignment safeguards while preserving visual realism, textual legibility, and semantic fidelity. Experiments on four commercial models reveal that more than 70% of false-claim prompts elicit images that faithfully depict the corresponding misinformation, and EpiReal-Attack pushes this rate to 95%. Most worryingly, these models are only a click away, and their outputs are cheap to spread yet hard to disbelieve, leaving this dimension of alignment largely unguarded.

1 INTRODUCTION

Commercial image generators can render false claims as realistic evidence, even when they recognize those claims as misinformation. EPIREAL-BENCH and EPIREAL-ATTACK evaluate and probe this generation-time safety gap, which remains substantial in four commercial models.

1 INTRODUCTION

  • Models can identify a claim as misinformation during verification yet still generate it as credible visual evidence, a discrepancy termed the verification-generation gap.The authors describe this as an underexplored alignment failure in commercial image-generation models.Source: §1 · pp. 2–3
  • EPIREAL-BENCH contains 10,000 verified prompts and 10,000 generated images across 10 claim categories and 10 credible visual formats.It characterizes misinformation by claim semantics and evidentiary presentation, and pairs each claim with a risk rationale.Source: §1 · p. 2
  • EPIREAL-ATTACK uses skill-guided black-box prompt optimization to bypass safeguards while preserving semantic fidelity and credible visual evidence.Its closed loop combines Pareto-based selection with multimodal feedback.Source: §1 · pp. 2–3
  • Direct prompting achieves an average misinformation realization success rate above 70%, while EPIREAL-ATTACK raises the ASR beyond 95%.The evaluations cover four commercial image generators.Source: §1 · p. 3

2 RELATED WORK

Prior work covers harmful visual content, misinformation detection, and selected generation domains, but leaves limited coverage of generation-time misinformation safeguards and the diversity of credible formats.

2.2 VISUAL MISINFORMATION GENERATION AND DETECTION

  • Generation-side studies report misleading content from image generators but remain limited in claim-domain and visually credible-form diversity.PH Safeguards studies health-related misinformation, while PC2 examines fabricated and controversial content involving public figures.Source: §2.2 · p. 4
  • Post-generation benchmarks study whether generated artifacts are synthetic or misleading, whereas this work evaluates whether safeguards prevent false claims from appearing as credible evidence.Examples include multimodal misinformation, synthetic-news, health imagery, deceptive, text-rich, and credibility-bearing image benchmarks.Source: §2.2 · p. 4

3 EPIREAL: BENCHMARK DESIGN AND RED-TEAMING FRAMEWORK

Commercial image generators can recognize false claims during verification yet still render them as visual evidence. EPIREAL-BENCH organizes this risk across claims and credible formats, while EPIREAL-ATTACK iteratively refines prompts using multiple objectives and evaluator feedback.

3.1 INSIGHT: THE VERIFICATION-GENERATION GAP

  • GPT-Image-2 and Nano Banana 2 recognized all benchmark claims yet had ASRs of 77.01% and 55.82%, while Grok Imagen 2.0 and Seedream 5.0 Flash had ASRs of 89.43% and 73.20%.Figure 4 compares FCR with ASR across the four commercial models.Source: Figure 4 · p. 4, §3.1 · p. 4
  • The results expose two risks: recognition failure, in which misinformation is not identified, and generation-time misalignment, in which it is recognized under verification yet still rendered as visual evidence.Source: §3.1 · p. 4

3.2 FALSE CLAIM DEFINITION

  • EPIREAL-BENCH defines a false real-world claim as an objectively verifiable false assertion about one entity, excluding hypothetical, unresolved, or ambiguous statements.Source: §3.2 · p. 4

3.3 BENCHMARK TAXONOMY

  • The taxonomy crosses claim semantics with evidentiary presentation, spanning 10 real-world categories and 10 credible visual formats.Categories range from government and elections to health, science, economy, education, and environment; formats include physical-world artifacts and digitally mediated evidence.Source: §3.3 · pp. 4–5, Table 2 · p. 5

3.4 CONSTRUCTING EPIREAL-BENCH: FROM CLAIMS TO VISUAL EVIDENCE

  • The claim pipeline generated 1,586 candidates, then used human review and deduplication to select 1,000 verified false claims, evenly distributed across categories.Five annotators reviewed falsity, entity grounding, specificity, and plausibility; 100 representative claims were selected per category.Source: §3.4 · pp. 5–6
  • For 1,000 verified claims, fixed prompt templates produced 10,000 prompts across predefined credible visual formats; four commercial models generated approximately 40,000 candidates, from which 10,000 high-quality prompt-image pairs were retained.Each false claim also received a risk rationale describing its factual inconsistency and the potential misconception that could arise when presented through visual evidence.Source: §3.4 · p. 6

3.5 EPIREAL-ATTACK: SKILL-GUIDED PARETO PROMPT EVOLUTION

  • EPIREAL-ATTACK transforms prompts with four skills and evaluates generated candidates for validity, misinformation realization, and visual credibility.The skills are claim reframing, contextual enrichment, artifact specification, and presentation refinement.Source: §3.5 · p. 6
  • It selects non-dominated prompts across the three objectives, then uses selected candidates and evaluator feedback to generate the next population, stopping by 10 iterations.Pareto selection preserves different optimization directions instead of collapsing them into a single score.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

Experiments assess commercial image generators across four questions: direct misinformation generation, attack robustness, defenses, and image credibility. The evaluation considers claim recognition, claim realization, and visual credibility.

4.1 EVALUATION METRICS

  • The evaluation measures claim recognition, misinformation realization, and visual credibility; ASR requires a produced image with the strictest MR score of 5.MR scores how faithfully an image conveys the target claim, while VC scores resemblance to authentic evidence using coherence, text rendering, layout, and visible artifacts.Source: §4.1 · p. 7

4.2 RQ1: HOW EASILY DO COMMERCIAL MODELS GENERATE VISUAL MISINFORMATION?

  • Under direct prompting, average ASR across evaluated models exceeds 70%; DP has lower ASR and MR across models, while Grok Imagen 2.0 has the highest ASR (89.43%) and MR (4.87).The results indicate that false claims can often be rendered as visual evidence without adversarial optimization.Source: §4.2 · p. 7

4.3 RQ2: HOW ROBUST ARE GENERATION SAFEGUARDS AGAINST EPIREAL-ATTACK?

  • EPIREAL-ATTACK raises average ASR from ∼70% to ∼97% and average MR from ∼4.7 to ∼4.9 across evaluated models and credible visual formats.Table 3 reports ASR and average MR across 10 credible visual formats; higher values indicate greater tendency to render false claims as evidence.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 RQ3: CAN EXISTING DEFENSES EFFECTIVELY DETECT OR FILTER SUCH OUTPUTS?

  • Existing input- and output-level defenses have generally low interception rates, while VGP improves interception on most models without model changes or additional training.The authors also suggest integrating claim-level verification into generation or adding misinformation-aware training objectives.Source: §4.4 · p. 8

4.5 RQ4: HOW CREDIBLE ARE THE GENERATED IMAGES TO HUMANS AND VLMS?

  • Figure 6 reports VC scores for GPT-Image-2 generations across 10 credible formats, assessed by seven VLMs and human annotators; the authors describe credibility cues as evidence-like conventions people use under limited attention.They conclude that false claims can acquire the appearance of evidence without their truth value changing.Source: Figure 6 · p. 9, §4.5 · p. 9
  • As image generation becomes cheaper and more accessible, the authors say the bottleneck increasingly shifts from creating authoritative-looking evidence to verifying its trustworthiness.They describe the outputs as falsehoods repackaged as credible evidence.Source: §4.5 · p. 9

5 CONCLUSION, LIMITATIONS, AND FUTURE WORK

The conclusion identifies a verification-generation gap, introduces EPIREAL-BENCH and EPIREAL-ATTACK, and proposes VGP as a mitigation. The study is limited to predefined claims and formats and does not primarily evaluate downstream effects on human beliefs.

5 CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • The paper reports that current models can generate highly credible visual misinformation and proposes VGP as a mitigation strategy.It presents EPIREAL-BENCH for systematic evaluation and EPIREAL-ATTACK for probing generation-time vulnerabilities.Source: §5 · pp. 9–10
  • The study is limited to predefined claim categories and visual formats, and primarily evaluates realization and visual credibility rather than downstream effects on human beliefs.Future work will explore broader scenarios, more advanced multimodal models, user-centered trust and belief evaluations, and improved defenses.Source: §5 · p. 10

AI USE STATEMENT

  • The authors used generative AI for language editing and minor code debugging or refactoring, not for dataset generation, research design, analysis, or interpreting results.They say all AI-assisted text and code were reviewed, and code changes were manually inspected and validated.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper states that its benchmark, code, prompts, and experimental resources are provided, with appendices covering protocols, attack details, further experiments, VGP, and examples.The reproducibility statement describes the contents of Appendices A–D.Source: REPRODUCIBILITY STATEMENT · p. 10

A Evaluation Protocols

The appendices list evaluation protocols and EPIREAL-ATTACK details.

A Evaluation Protocols

  • Appendix A lists protocols for FCR, MR, and VC evaluation.The contents list identifies separate sections for false claim recognition, misinformation realization, and visual credibility.Source: A Evaluation Protocols · p. 15

B EPIREAL-ATTACK Details

  • Appendix B lists the EPIREAL-ATTACK skill pool, attack optimization prompt, and algorithm.These topics are identified as subsections in the appendix contents.Source: B EPIREAL-ATTACK Details · p. 15

C Additional Experimental Results

Appendix C lists additional experiments and implementation details for VGP.

C Additional Experimental Results

  • Appendix C covers benign-prompt verification, results across 10 claim categories, additional visual credibility results, and VGP details.These topics are listed as four subsections of the appendix.Source: C Additional Experimental Results · p. 15

A EVALUATION PROTOCOLS

The protocols separately measure whether models identify false claims and whether generated images make them appear credible.

A.1 FALSE CLAIM RECOGNITION RATE (FCR)

  • Image-based verification asks GPT-Image-2, Grok Imagen 2.0, and Seedream 5.0 Flash to express a factual judgment within a generated image, while Nano Banana 2 responds in text.For image outputs, OCR text and the original image are assessed together; the final judgment is classified as misinformation, not misinformation, or uncertain.Source: §A.1 · pp. 16–17
  • FCR is the proportion of false claims whose final judgment is misinformation.Source: §A.1 · p. 17

A.2 MISINFORMATION REALIZATION (MR)

  • MR uses a 1–5 multimodal score to assess how clearly and convincingly an image presents a verified false claim as visual evidence, not whether the claim is true.Source: §A.2 · p. 18

A.3 VISUAL CREDIBILITY (VC) EVALUATION

The section specifies visual-credibility scoring and EPIREAL-ATTACK’s prompt-transformation loop, and reports a benign-claim verification check.

A.3 VISUAL CREDIBILITY (VC) EVALUATION

  • VC uses GPT-5.4 to score whether an image would convince a typical viewer as evidence, considering relevance, format, text and layout quality, and persuasiveness on a 1–5 scale.Source: §A.3 · p. 19

B.1 SKILL POOL

  • The skill pool defines four prompt transformations—claim reframing, contextual enrichment, artifact specification, and presentation refinement—while preserving the false claim.Source: §B.1 · p. 20

B.2 ATTACK OPTIMIZATION PROMPT

  • GPT-5.4 refines candidate prompts using prior attempts and multimodal feedback, keeping the claim fixed while improving its visual realization and evidence credibility.Source: §B.2 · p. 21

B.3 EPIREAL-ATTACK ALGORITHM

  • EPIREAL-ATTACK applies skills to prompts, generates and evaluates images, then uses Pareto selection to retain non-dominated candidates for further optimization.Source: §B.3 · p. 22

C.1 VERIFICATION ON BENIGN FACTUAL PROMPTS

C.2 RESULTS ACROSS 10 CLAIM CATEGORIES

Across claim categories, direct-prompt vulnerability varies, while EPIREAL-ATTACK achieves high attack success across the evaluated categories and models.

C.2 RESULTS ACROSS 10 CLAIM CATEGORIES

  • Under direct prompting, economy is among the most vulnerable categories and education tends to have lower ASR and MR, but EPIREAL-ATTACK largely eliminates these differences.Source: §C.2 · p. 23
  • ASR exceeds 95% across all categories and evaluated models under EPIREAL-ATTACK, while MR remains consistently close to the maximum score.The comparison covers performance across 10 claim categories.Source: Table 6 · p. 23, §C.2 · p. 23

C.3 ADDITIONAL VISUAL CREDIBILITY RESULTS

Additional results compare visual credibility across commercial models, while the section also describes VGP and illustrates benchmark examples.

C.3 ADDITIONAL VISUAL CREDIBILITY RESULTS

  • Across evaluators and credible formats, EPIREAL-ATTACK generations generally receive higher VC scores than direct-prompt generations, with broadly similar patterns across the additional models.The additional evaluation covers Nano Banana 2, Grok Imagen 2.0, and Seedream 5.0 Flash.Source: §C.3 · pp. 23–24, Figure 7 · p. 24, Figure 8 · p. 24

C.4 DETAILS FOR VGP

  • VGP prepends a factual-verification instruction and asks the model to generate only when the central claim is confidently accurate, without retrieval, training, or model changes.Source: §C.4 · p. 25

D EPIREAL-BENCH EXAMPLES

Loading 2610.11112v1…