PaperlayerView exact revision on arXiv

Source-linked AI summary

FVH-Bench: Evaluating Image Generator Safety Beyond Overtly Harmful Imagery

Zihao Zhu, Yuhan Liu, Siwei Lyu, Baoyuan Wu

arXiv:2610.37722v1cs.CRcs.AI

TL;DR

Image generators may create artifacts that support harmful actions or deception even without overtly harmful imagery, a risk the paper terms functional visual harm. FVH-Bench evaluates 669 prompts across five functions and ten risk domains. The 11 tested generators rarely refuse and often fulfill the targeted harmful functions, while existing safety detectors provide limited coverage.

  • Problem

    Existing image-safety research has focused largely on overtly harmful content, leaving limited evaluation of whether generators produce artifacts whose harmful function is to guide dangerous actions or support deception.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces FVH-Bench, a 669-prompt benchmark across five harmful visual functions and ten risk domains, and assesses generator refusals, output harm, and detector recognition.

    Source: §1 · p. 2

  • Results

    The 11 evaluated image generators rarely refuse FVH requests and frequently produce artifacts that fulfill the targeted harmful functions, while existing safety detectors provide limited FVH coverage.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Safety evaluation should consider what generated images may enable as well as what they depict, including the harmful functions of artifacts during output screening.

    Source: §6 · p. 9

    The evaluations measure visible target realization and apparent functional support, not actual harm in deployment, and the prompt distribution does not represent real-world request frequencies.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern image generators can produce complex visual artifacts, including diagrams, interfaces, and records that combine text, graphics, and layout to convey detailed information. They may refuse requests for overtly harmful imagery yet still generate artifacts that guide dangerous actions or support deception. We call the risk arising from what such images enable, rather than only what they depict, functional visual harm (FVH). Existing safety benchmarks focus largely on overtly harmful content, such as pornography and graphic violence, leaving image generators' safety alignment against FVH insufficiently evaluated. To address this gap, we introduce FVH-Bench, comprising 669 prompts across five harmful visual functions and ten risk domains. The 11 evaluated image generators rarely refuse these requests and frequently generate artifacts that fulfill the targeted harmful functions. High scores for target harm realization and harmful artifact usability further indicate that these outputs can support the intended harmful purposes. A further analysis finds that existing safety detectors also provide limited coverage of FVH, indicating gaps in their ability to recognize harmful visual functions. Together, these findings show that evaluating image-generator safety alignment requires considering not only what images depict, but also what they may enable.

1 INTRODUCTION

Image generators can create visual artifacts that enable harmful actions or deception without depicting overtly harmful content; the paper calls this functional visual harm (FVH). FVH-Bench evaluates whether models refuse such requests, whether their outputs fulfill harmful functions, and whether detectors recognize those risks.

1 INTRODUCTION

  • FVH is the risk arising from what generated visual artifacts enable, beyond what they depict.A diagram can make a dangerous procedure easier to follow, while a fabricated record can lend credibility to a false claim.Source: §1 · p. 1
  • FVH-Bench comprises 669 prompts across five harmful visual functions and ten risk domains, measuring refusal, target harm realization, harmful artifact usability, and detector recognition.Source: §1 · p. 2
  • The 11 evaluated image generators rarely refuse FVH requests and frequently produce artifacts that fulfill the targeted harmful functions.Their outputs also score highly for target harm realization and harmful artifact usability, indicating support for the intended harmful purposes.Source: §1 · p. 2
  • The paper identifies gaps in both generation safeguards and detector coverage, and explores preliminary function-aware detection methods that improve recognition of FVH risks.Source: §1 · p. 2

2 RELATED WORK

Prior image-generation safety work has largely evaluated overtly harmful content and safeguard vulnerabilities, while newer work addresses selected text-rich and misinformation risks. This paper positions FVH-Bench as evaluating harmful visual functions across risk domains and examines existing detectors’ coverage of those functions.

2.1 SAFETY BENCHMARKS FOR IMAGE GENERATION

  • Existing image-generation benchmarks and jailbreak studies largely assess overtly harmful content, while recent work covers selected text-in-image and visual-misinformation risks.FVH-Bench complements these efforts by evaluating refusals, target harm realization, and harmful artifact usability across harmful visual functions and risk domains.Source: §2.1 · p. 3

2.2 MULTIMODAL SAFETY DETECTION METHODS

  • Multimodal safety detectors moderate image content and image–text risks, while synthetic-origin assessments do not establish an artifact’s harmful function.The paper evaluates existing detectors on harmful images and benign controls, and explores function-aware detection improvements.Source: §2.2 · p. 3

3 FVH-BENCH: BENCHMARKING FUNCTIONAL VISUAL HARM

FVH-Bench defines functional visual harm by the harmful purpose an artifact’s content and presentation support, then organizes its benchmark around harmful functions and risk domains. A multi-agent construction process with human review produces 669 prompts across five functions and ten domains.

3.1 FUNCTIONAL VISUAL HARM BEYOND OVERTLY HARMFUL IMAGERY

  • FVH concerns what generated visual artifacts enable, not only what they depict; text, graphics, layout, and context can jointly support a harmful purpose.The concern is the artifact’s harmful function, rather than its form or realism alone.Source: §3.1 · p. 3

3.2 FUNCTION–DOMAIN TAXONOMY

  • The taxonomy separates five functions describing how artifacts support harm from ten risk domains identifying the safety issues involved.Functions include action enablement, evidence fabrication, interface simulation, authority mimicry, and persuasive packaging; domains span direct harm, fraud or platform abuse, and social harm.Source: §3.2 · pp. 3–4

3.3 FUNCTION-GUIDED BENCHMARK CONSTRUCTION

  • A multi-agent framework turns function–domain combinations into scenarios and image-generation prompts, then iteratively reviews candidates before human curation.Five reviewers assess each candidate reaching human review, and at least four must approve it; the resulting benchmark contains 669 prompts across five functions and ten risk domains.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Across four experiments, the authors evaluate image-generator refusal and harmful utility, detector performance, and human assessments of automated scores and visual support. The results show frequent functional harm in generated images, limited detector coverage, and greater perceived harmful support from images than text.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers eleven image generators and five safety detectors, measures refusal and image quality, harm realization, and usability, and uses two alternative detection backbones.PIC, THR, and HAU are independently scored by three multimodal judges; detector evaluation includes image-only and prompt–image inputs for Llama Guard 4.Source: §4.1 · p. 5

4.2 IMAGE-GENERATOR SAFETY ON FVH-BENCH

  • Across eleven generators, average refusal was 2.45%, while nine models had mean PIC above 4, indicating requests were rarely refused and outputs largely matched requested content and form.GPT Image 2 had the highest refusal rate at 5.38% and the highest PIC at 4.64; prompt consistency alone does not establish harmful utility.Source: §4.2 · p. 5, Table 1 · p. 6
  • Mean THR was 4.46 across eleven models, and generated images also scored highly for harmful usability, showing frequent realization and functional support for intended harmful purposes.Eight models scored above 4 on THR; four scored above 4 on HAU. Even the lowest-scoring model, Wan 2.7 Image, reached THR 3.68 and HAU 3.46.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 EVALUATION OF SAFETY DETECTORS

  • Condition-Decomposed detection improved recall over General Safety Policy by 30.60 and 28.20 percentage points with Qwen3-VL (8B) and GPT-5.4, respectively, with observed FPR of 0.00% for both approaches.Dedicated safety detectors achieved recall from 1.60% to 44.00%, while General Safety Policy still missed a substantial proportion of harmful images.Source: Table 2 · p. 6, §4.3 · p. 7

4.4 HUMAN EVALUATION

  • Automated PIC, THR, and HAU scores showed Spearman correlations of 0.89–0.94 and MAEs of 0.20–0.35 against human ratings on 300 images.Three raters assessed each image, and agreement in both rankings and score magnitudes supports use of automated scores on FVH-Bench.Source: §4.4 · p. 7, Figure 3 · p. 7
  • Images received higher perceived functional-support ratings than textual counterparts across all five functions, with the largest gaps in evidence fabrication and interface simulation.The authors suggest those gaps may reflect the visual presentation's role in lending apparent evidentiary support to fabricated records and interactions.Source: §4.4 · p. 7, Figure 3 · p. 7

5 ANALYSIS

Across the evaluated generators, stronger prompt–image consistency accompanies greater target harm realization and artifact usability. Harm realization stays high across functions, but usability varies by function and risk domain.

5.1 GENERATION CAPABILITY AND FUNCTIONAL HARM

  • Spearman correlations between mean PIC and mean THR and HAU are 0.96 and 0.95, respectively, across the eleven evaluated generators.Models with higher mean PIC also tend to have a larger share of images with THR > 4.Source: §5.1 · p. 7, Figure 4 · p. 8
  • Text and data quality has the largest overall correlation with THR (0.44), while structure and layout has the largest with HAU (0.43).Structure and layout is more closely associated with HAU in F1 and F3 (0.50 and 0.53); content and semantics is more closely associated with THR in F1 and F5, while artifact form has weak associations with both metrics.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 FUNCTIONAL HARM ACROSS FUNCTIONS AND DOMAINS

  • Mean THR is consistently high across F1–F5 (4.50–4.63), while mean HAU ranges from 3.57 for evidence fabrication (F2) to 4.09 for persuasive packaging (F5).The averages weight generators equally.Source: §5.2 · p. 8, Figure 6 · p. 9
  • F2 has relatively low HAU across several domains, including financial fraud (D5) at 3.64 and identity and credential forgery (D6) at 3.55.F5 has higher HAU in multiple domains, reaching 4.20 in medical and health misguidance (D4) and 4.28 in cyber and platform abuse (D7).Source: §5.2 · pp. 8–9, Figure 6 · p. 9

6 DISCUSSION

The discussion links functional harm to weak refusal and screening coverage, while noting that the evaluations measure visible realization and apparent support rather than deployed harm.

Implications for Image-Generation Safety

  • The evaluated generators rarely refuse FVH requests and frequently produce artifacts that support the intended harmful functions.Stronger generation capability is associated with greater functional harm, while existing detectors have limited coverage of FVH.Source: §6 · p. 9
  • Safety evaluation should consider both refusal behavior and the harmful functions generated images enable, beyond overtly harmful imagery.The discussion identifies preliminary function-aware detection as a promising direction for recognizing these risks.Source: §6 · p. 9

Limitations

  • The evaluations measure visible target realization and apparent functional support, not actual harm caused in deployment.Prompt frequencies are not representative of real-world requests, and automated scores remain subject to judge-model biases and rubric interpretation differences.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents FVH-Bench as an evaluation of what generated images enable and reports rare refusals, frequent fulfillment, and limited detector coverage.

7 CONCLUSION

  • Functional visual harm (FVH) characterizes risks arising from what generated images enable beyond what they depict.FVH-Bench comprises 669 prompts across five harmful visual functions and ten risk domains.Source: §7 · p. 9
  • The eleven evaluated image generators rarely refuse FVH requests and frequently produce artifacts that fulfill the targeted harmful functions.Stronger generation capabilities are associated with greater functional harm.Source: §7 · p. 9
  • Existing safety detectors provide limited FVH coverage, and preliminary function-aware detection results suggest a direction for improving recognition.The conclusion calls for evaluating both what generated images depict and what they may enable.Source: §7 · p. 9

A BENCHMARK CONSTRUCTION AND COMPOSITION

FVH-Bench candidates are developed through a staged workflow that pairs harmful functions with risk domains, specifies scenarios, and turns them into generation requests. Human review checks each request’s objective, category fit, clarity, and consistency before acceptance.

A.1 CONSTRUCTION AND HUMAN CURATION

  • Five reviewers independently assess each candidate; it is retained only if at least four accept all four review criteria.The criteria check for a clear harmful objective, accurate function and domain assignments, and clarity and consistency; successful harmful generation is not required.Source: §A.1 · p. 14

Prompt A.1: Coverage Planner

Prompt A.2: Scenario Designer

  • The Scenario Designer specifies the harmful objective, context, artifact form, required content, and visual relationships for a coherent scenario.The prompt directs the designer to use abstract placeholders for dangerous procedures, deceptive payloads, and sensitive identifiers.Source: Prompt A.2: Scenario Designer · p. 15

Prompt A.3: Prompt Composer

Prompt A.4: Prompt Reviewer

  • The Prompt Reviewer checks objective, category fit, clarity, consistency, and duplication, then passes, revises, or discards the candidate.Substantive duplicates are discarded immediately; other correctable problems may receive up to three revisions before discard.Source: Prompt A.4: Prompt Reviewer · p. 16

Prompt A.4: Prompt Reviewer (continued)

  • The reviewer compares candidates with a retrieved shortlist, not an exhaustive prompt history, and excludes a candidate’s earlier drafts from duplicate checks.Earlier drafts are used only to assess revisions.Source: Prompt A.4: Prompt Reviewer · p. 16

A.3 HARMFUL SAMPLE DISTRIBUTIONS

The harmful prompt set is unevenly distributed across both taxonomy axes, while benign controls cover visual forms represented among harmful samples. The controls are reviewed to exclude harmful instructions, deceptive claims, and other support for harmful activity.

A.3 HARMFUL SAMPLE DISTRIBUTIONS

  • Action enablement and authority mimicry are the most frequent harmful functions, while medical and health misguidance is the largest risk domain.Figure 7 reports separate marginal distributions across the five functions and ten domains; it does not show function–domain joint coverage.Source: §A.3 · p. 17, Figure 7 · p. 17

A.4 BENIGN CONTROL CONSTRUCTION AND COMPOSITION

  • The 200 benign controls use independently designed everyday, educational, and administrative scenarios, with generated images checked for harmful or deceptive content.The images are generated using GPT Image 2 and are used to measure false positives on benign visual artifacts.Source: §A.4 · p. 17
  • The benign controls span guides, infographics, comics, interfaces, documents, notices, commercial layouts, and a small group of everyday scenes.Each image is assigned to one group according to its dominant rendered form; Figure 8 gives counts and proportions.Source: §A.4 · p. 17, Figure 8 · p. 18

B AUTOMATED EVALUATION

The automated evaluation separates generation quality, visible realization of the requested harm, and an image’s apparent support for a specified harmful function. Each measure uses its own criteria and evidence boundaries.

Prompt B.1: Prompt–Image Consistency (PIC)

APPLICABILITY AND OUTPUT

Prompt B.2: Target Harm Realization (THR)

  • Target Harm Realization (THR) measures how fully visible image evidence realizes the requested harmful function and risk content.The prompt and categories define the target but do not establish its realization; unreadable, missing, cropped, or ambiguous content is not inferred from the prompt.Source: Prompt B.2: Target Harm Realization… · p. 20, EVIDENCE AND SCOPE · p. 20

EVIDENCE AND SCOPE

  • THR’s five-level scale distinguishes absent or transformed risk, superficial traces, partial realization, clear realization, and full realization.A score of 5 requires nearly all material risk-bearing content and relationships to be visibly assessable; substantial unreadable risk-bearing content caps the score at 4.Source: EVIDENCE AND SCOPE · p. 20, §5 · p. 20, §4 · p. 21

Prompt B.3: Harmful Artifact Usability (HAU)

  • Harmful Artifact Usability (HAU) measures apparent, proximal support for a specified harmful function, separately from THR and general artifact usability.It does not measure actual harm, intrinsic severity, policy violation, general image quality, or completeness of prompt following.Source: Prompt B.3: Harmful Artifact Usability… · p. 21

REFERENCE CONDITIONS AND EVIDENCE

  • HAU judges the intact image at legible size for an ordinary viewer, without assuming edits, removed warnings, external systems, or reconstructed text.The prompt defines the target but is not evidence that it appears; the assessment concerns proximal visual support, not actual harmful outcomes.Source: REFERENCE CONDITIONS AND EVIDENCE · p. 21

FUNCTION-SPECIFIC DIAGNOSTICS

DIRECT ORDINAL SCALE

  • HAU assigns a direct ordinal score from 1 for no discernible target utility to 5 for strong visible functional support without material limitations.Critical limitations cap scores at 2, major limitations at 3, and localized limitations at 4; diagnostic axes are not averaged into subscores.Source: DIRECT ORDINAL SCALE · p. 22, CONSISTENCY RULES · p. 22

B.4 AGREEMENT AMONG AUTOMATED JUDGES

The three automated judges show strong agreement in their relative rankings across PIC, THR, and HAU. Agreement does not establish identical absolute scores or correctness against human judgments.

B.4 AGREEMENT AMONG AUTOMATED JUDGES

  • 0.87–0.93: pairwise Spearman correlations across judges and PIC, THR, and HAU; HAU correlations are 0.87–0.89.PIC correlations are 0.91–0.93 and THR correlations are 0.90–0.93; the results support consistency in relative rankings, not identical absolute scores or correctness against human judgments.Source: Table 3 · p. 23, §B.4 · p. 23

C HUMAN EVALUATION

The human-evaluation appendix validates automated scores and compares perceived functional support in images and text. It specifies function-specific rating criteria and the sampling and rating procedures for both studies.

C.1 VALIDATION OF AUTOMATED SCORES

  • Automated-score validation covers 300 images, stratified by harmful function and generator across observed score ranges, with three independent raters assessing PIC, THR, and HAU.PIC uses applicable dimension scores; THR and HAU receive direct integer ratings, with the rubric distinguishing realization of the target from apparent functional support.Source: §C.1 · pp. 24–25
  • PIC measures prompt adherence and visual quality, not harmfulness; inapplicable dimensions are omitted from its mean and their reasons recorded.Raters assess artifact form, structure, semantic fidelity, rendered text and data, cleanliness, and overall visual quality.Source: §C.1 · p. 24
  • Automated-score agreement is summarized by metric using Spearman correlation and mean absolute error against the valid image-level rater reference.Correlations are computed across images, and missing or invalid assessments are not imputed as low scores.Source: §C.1 · p. 25

C.2 IMAGE–TEXT COMPARISON

  • The image–text comparison uses 100 paired materials from the same requests, with five trained raters independently scoring both versions for their specified harmful function.Text counterparts retain core information while removing image-generation instructions and visual-presentation descriptions; the two versions are rated in separate sessions seven days apart.Source: §C.2 · pp. 25–26
  • The paired image–text ratings are descriptive ordinal summaries, and the study measures raters’ perceived functional support rather than real-world harmful outcomes.Session separation reduces immediate comparison but does not establish absence of memory effects.Source: §C.2 · p. 26

C.3 FUNCTION-SPECIFIC RATING CRITERIA

  • The five function-specific rubrics assess action guidance, apparent evidence, simulated interactions, authority endorsement, and persuasive support using direct ratings rather than checklist sums.Each presentation is judged against the criteria for its primary function.Source: §C.3 · pp. 26–28

D.1 EVALUATION PROTOCOL AND BASELINE SETUP

The detector evaluation compares dedicated safety systems and a general-policy baseline with a condition-decomposed workflow. The workflow evaluates five visual functions through necessary conditions and benign exceptions, with additional review and verification for initially safe images.

D.1 EVALUATION PROTOCOL AND BASELINE SETUP

  • The detector evaluation uses 500 human-confirmed harmful images and 200 benign controls, reporting harmful-image recall and benign-control false-positive rate.General Safety Policy and Condition-Decomposed detection are tested with Qwen3-VL-8B-Instruct and GPT-5.4 using image-only input.Source: §D.1 · p. 28

D.2 SHARED SAFETY POLICY

  • The shared policy covers ten risk domains and asks for one image-safety decision based on visible evidence, while warning that appearance alone does not establish a violation.It treats image text as content rather than instructions and allows neutral reporting, legitimate education, and prevention.Source: §D.2 · p. 28, GENERAL IMAGE SAFETY POLICY · p. 29, §2 · p. 29

D.3 CONDITION-DECOMPOSED DETECTION WORKFLOW

D.4 MODEL-CALL OVERHEAD

  • Condition-Decomposed detection requires one to five calls per image, compared with one for General Safety Policy; call counts do not capture token use or elapsed time.Those also depend on image views, response length, and execution scheduling.Source: §D.4 · p. 31, Table 4 · p. 31
  • Condition-Decomposed detection averages 1.57 model calls per image, or 0.57 additional calls (57%) relative to General Safety Policy.The average varies with how often images enter review and verification.Source: §D.4 · p. 31

E STATISTICAL ANALYSIS METHODS

The statistical analyses relate generation-quality measures to harm scores at both model and image levels, using correlations with different units and adjustments. The appendix also describes the selected qualitative examples and benign artifact controls.

E.1 MODEL-LEVEL CORRELATIONS

  • Model-level analyses compare mean PIC across eleven generators with mean THR, mean HAU, and the share of THR scores above 4 using Pearson and Spearman correlations.Refusals, missing images, and invalid assessments are excluded; each generator contributes one equally weighted observation, and correlations describe associations rather than causal effects.Source: §E.1 · pp. 31–32

E.2 IMAGE-LEVEL PARTIAL RANK CORRELATIONS

  • Image-level analyses correlate adjusted ranks of PIC and THR or HAU after accounting for additive model and prompt differences.Scores are matched for the same prompt–image pair, ranked within each analysis subset, and restricted to prompts with valid observations from at least two generators; the Overall column pools observations rather than averaging function-specific correlations.Source: §E.2 · p. 32
  • The image-level heatmap reports descriptive associations, not causal contributions or statistically significant differences between coefficients.The analysis excludes PIC’s holistic visual-quality dimension and does not control for other PIC dimensions.Source: §E.2 · p. 32

F QUALITATIVE EXAMPLES

  • The appendix presents two selected examples per harmful visual function, chosen for visible harmful content, clear mechanisms, and interpretable consequences rather than prevalence or severity ranking.All examples were generated by GPT Image 2; prompts are omitted, and the benign controls illustrate eight artifact-form groups.Source: F QUALITATIVE EXAMPLES · p. 33, §F.6 · p. 43, Figure 19 · p. 43

Related papers

Loading 2610.37722v1…