PaperlayerView exact revision on arXiv

Source-linked AI summary

The Uncontrolled Variable: Vision-Language Model Refusal Responds to Image Presence in Ways Risk Cannot Explain

Haoyu Zhang, Yi Feng, Shibo Zheng, Zhuoxi Wang, Xiao Luo, Haowen Xu, Xiangchen Guan, Mohammad Zandsalimy, Shanu Sushmita

arXiv:2609.26174v1cs.CRcs.AI

TL;DR

This paper tests whether image presence changes aligned VLM refusals independently of request content, using controls that hold the request fixed. It finds substantial, checkpoint-specific shifts whose benign cost can persist even when harmful-side benefits are small or absent.

  • Problem

    Safety benchmarks vary request content or image content but do not isolate whether image presence itself shifts refusal while the request stays fixed.

    Source: §1 · p. 1

  • Method

    The paper uses controlled comparisons of image presence and attachment-related cues across hosted and open checkpoints, including same-weight serving-route and placebo controls.

    Source: §4.1 · p. 10, §5 · p. 13, §6 · p. 14

  • Results

    +51, +34 and +23pp benign refusal costs on the borderline rung accompany 18, 5 and 6 points of prevented harm, respectively, and the sign can reverse on an open model.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Image-presence refusal is a checkpoint-inherited behavior whose cost and direction are not a deployer-chosen default and should be known before accepting the trade.

    Source: §5 · p. 13, §6 · p. 14

    The measured effect is not universal across VLMs, and black-box comparisons do not identify which checkpoint components produce it.

    Source: Limitations · p. 15

Abstract

from arXiv · show

Vision-Language Model (VLM) safety is expected to depend on what a request asks for. We show that safety-aligned VLMs also key refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attaching a blank canvas - unreadable, unrelated to the request, identical across prompts - shifts refusal by tens of percentage points, with no defense in the loop. The shift is not blanket caution. Neutral instructions are almost unaffected while borderline-benign prompts move sharply, so the cost falls on sensitivity-adjacent traffic: benign questions about privacy, self-harm and violence. Attachment alone is sufficient, while the image's properties set the price: a black canvas costs substantially more than a white one of identical size, and on an open checkpoint the carrying axis is pixel count. Nor is the shift under instructional control - telling the model the image is a placeholder to be disregarded removes only a fraction of it, and on one model asserting that an attachment exists moves refusal substantially with nothing attached. Attachment may correlate with risk in deployment; what these models do with it does not track risk. It is not the serving stack, since the same weights reached two ways behave alike, nor a property of VLMs as such, since several open-weight checkpoints show nothing. It belongs to particular aligned checkpoints, one of them open. It is also decoupled from what it buys: the canvas does prevent some attack success on a matched harmful set, but far less than it costs, and its sign is not fixed - on one open model the identical canvas makes the model markedly easier to attack. Image presence is not a default a deployer chose or priced; it is an uncontrolled variable inherited with the weights.

1 Introduction

Benchmarks vary request content but not image presence, leaving its independent effect on refusal unmeasured. A constant blank canvas reveals a substantial, uneven refusal shift whose size and direction depend on the checkpoint and image properties.

1 Introduction

  • A blank canvas alone shifts refusal by tens of percentage points, despite carrying no per-prompt information and with no defense in the loop.It is unreadable, unrelated to the request and identical across prompts.Source: §1 · pp. 1–2
  • Borderline-benign prompts incur +23 to +51pp refusal shifts, while neutral instructions change by ≤2pp on three of four hosted models.The cost falls on benign questions near the refusal boundary, including privacy, self-harm, violence and illegal activity.Source: §1 · p. 2
  • The effect varies with image properties: black costs +22pp more than white on two hosted models, while pixel count carries the effect on an open checkpoint.On the open checkpoint, refusal rises monotonically with pixel count; colour and file encoding are inert.Source: §1 · p. 2
  • Instructing the model to disregard the placeholder removes only part of the refusal shift, and on gemini-2.5-flash an asserted attachment raises refusal with no image present.There, a bare attachment assertion adds +16pp beyond a neutral system instruction; mentioning “image” adds no further effect.Source: §1 · p. 2
  • The effect belongs to particular aligned checkpoints, not the serving stack or VLMs universally; qwen3-vl-8b shows hosted-scale shifts when open and self-served.Three moderation-free open-weight models show no significant benign cost, while the same gemma checkpoint behaves alike through managed and self-served routes.Source: §1 · pp. 2–3
  • The cue has opposing safety effects: it reduces attack success by 5 to 18pp on a matched harmful set, yet raises it by 35pp on one open model.On the borderline rung, it costs +51, +34 and +23pp of benign refusal against 18, 5 and 6 points of prevented harm, respectively.Source: §1 · p. 3
  • The shift does not track harmful risk reduction: across three generations, benign costs are +8, +1 and +28pp while harmful effects stay within ±7pp.As harmful completions become rarer, the benign cost persists; the paper therefore reports both sides rather than a ratio.Source: §1 · p. 3

2 Manipulation and Method

The study tests model behavior by varying image attachment while keeping request text fixed, without placing a defense in the pipeline. Paired comparisons vary presence, image properties, serving route, and image mention.

2 Manipulation and Method

  • Image attachment is the sole manipulation between otherwise unchanged requests, and the pipeline contains no defense.Every reported number contrasts two arms that differ only in whether an image is attached.Source: §2 · p. 4
  • A byte-identical canvas carries no per-prompt information, but the black-box design cannot exclude its statistical association with post-training safety policy.The conclusions are therefore stated behaviorally.Source: §2 · p. 4
  • Arms are paired by prompt and tested with exact McNemar tests within one collection job to limit run drift.Targets remain stochastic at temperature 0, and providers expose no seed; rates are stable to a few points across windows.Source: §2 · p. 4
  • The experiments vary image presence, image properties, serving route, and whether the system message mentions an image.The mention condition crosses attachment with wording that is true in both attachment conditions.Source: §2 · p. 4

3 Experimental Setup

The setup compares hosted and open-weight checkpoints across benign prompts of increasing topical sensitivity and a matched harmful prompt set. It measures refusal and harmful-response rates with separate classifiers.

3 Experimental Setup

  • Four hosted and seven open-weight models are tested on three benign sensitivity rungs and a topic-matched harmful OR-Bench split.Each cell uses 100 prompts.Source: §3 · p. 4
  • Benign refusal rate counts refusals among 100 benign prompts, while ASR counts harmful responses among 100 harmful prompts.Lower benign refusal is more useful; lower ASR is safer.Source: §3 · pp. 4–5
  • The two rates use distinct gpt-5-mini rubrics, and empty responses are classified as refusals.The refusal rubric is not substituted for the harmful-response rubric.Source: §3 · p. 5

4 Results

The results examine how blank-image attachment affects refusal across checkpoints and image properties, including its effects on borderline-benign and neutral requests and its relation to changes in attack success. Paired comparisons and judge checks support the reported contrasts, subject to stated limits.

4.1 The model’s own threshold: what image presence costs and what it buys

  • Borderline-benign refusal rises +23 to +51pp on three of four hosted models, while neutral instructions change by at most 2pp on three of four.A stratified sample across all ten OR-Bench categories found every contrast positive, with 26 of 30 surviving Benjamini–Hochberg correction; the original category-ordered sample understated the cost.Source: §4.1 · p. 5, Table 1 · p. 5
  • A blank canvas and a distinct content-free drawing both raise refusal on three hosted models; image properties then set a model-dependent price.On gpt-4o-mini, shifts stay within +25 to +33pp across tested arms; on gemini-2.5-flash-lite, black costs +22pp more than white, and rendered content costs +30pp more than a size-matched blank. The effect also appears with an OCR-able caption, while the larger property sweep finds gemini-2.5-flash near-null.Source: §4.1 · pp. 5–6, Table 4 · p. 8
  • An instruction to disregard the image leaves roughly three-quarters of the refusal shift intact on three models, while gemini-2.5-flash’s +33pp response is triggered by text asserting an attachment.On gemini-2.5-flash, a placebo ladder attributes the effect to being told an attachment exists and should be ignored, not to the word “image”; no image is attached in any arm.Source: §4.1 · pp. 6–7, Table 2 · p. 7
  • On qwen3-vl-8b, refusal rises monotonically with pixel count across canvas sizes, and independently rendered size-by-fill arms replicate that trend.The 256^2-versus-1536^2 contrast replicates for white, grey and black canvases; fill differences remain within 3–7pp against a size span of 12–19pp. Three independent jobs measured borderline shifts of +32, +28 and +29pp.Source: §4.1 · p. 8, Table 5 · p. 9
  • Across three generations of one open model family, benign refusal shifts +8, +1 and +28pp, while matched harmful effects remain within ±10pp.The newest checkpoint’s +28pp benign shift contrasts with a harmful effect bounded by [−6.1, +3.7]pp; the discrete jump does not identify which release change caused it, and harmful baselines in the family are near the floor.Source: §4.1 · p. 9
  • The effect is checkpoint-specific: three tested open-weight models are near-null, but qwen3-vl-8b-instruct shows a +32pp shift without a moderation layer.Across five open checkpoints, four sharing one coarse alignment-tier label span +32 to +0pp, so that label does not predict the effect; the strongly aligned open checkpoint could not be tested.Source: §4.1 · pp. 7–8, Table 6 · p. 10
  • On qwen3-vl-8b, attachment itself raises refusal by +20pp when image-mention text is held identical across attached and unattached conditions.Mention alone adds +8pp and is not significant, separating attachment from the textual mention on this checkpoint.Source: §4.1 · p. 10
  • On pixtral-12b, a blank canvas raises harmful ASR 48→81% (+33pp); on llava-1.5-7b it rises 37→76% (+39pp), while two of eight open checkpoints invert.On pixtral-12b, 37 prompts flipped toward harm and four away (p = 1.0 × 10−7); its benign refusal also fell 10→2%, but that smaller shift was not significant after correction. The other two open-weight models in that arm were flat on the harmful axis.Source: §4.1 · p. 10, Table 12 · p. 12
  • On qwen3-vl-8b, every tested image arm inflated borderline-benign refusal, with blank-arm refusal increasing monotonically with pixel count; colour and JPEG quality were inert.The 256² and 1536² blank canvases differed directly at 18 flips to 2 (p = 0.0004); the matched-size blank-versus-caption contrast was not significant (p = 0.15).Source: Table 7 · p. 10
  • On qwen3-vl-8b, attachment still raised refusal +20pp with mention held constant, but the mention comparison was underpowered rather than null.With 16 discordant pairs, the exact test could only have resolved a mention effect of about 10pp; the reported attachment effect therefore was not a lexical artifact.Source: §4.1 · p. 10, Table 10 · p. 11
  • On gemma-3-12b-it, blank canvas raised refusal +7pp on both serving routes, while route differences stayed within ±10pp on every arm and ±5pp on blank arms.The same checkpoint was tested through a managed host and vLLM; this rules out serving infrastructure as the source of the hosted-model shift in this one small-effect model, but does not isolate which checkpoint component carries it.Source: §4.1 · pp. 10–11, Table 11 · p. 11
  • Blank-image attachment raises borderline-benign refusal by +23 to +54pp on hosted models, while effects vary by checkpoint and are absent on neutral prompts.On open-weight models, the benign inflation is absent; Table 13 identifies the topic-matched column as the JailbreakBench-benign set shown for continuity.Source: §4.1 · pp. 11–12, Table 13 · p. 12, Table 12 · p. 12

4.2 Judge robustness

  • Under gpt-5-nano, every contrast significant under gpt-5-mini kept its sign and significance, estimates shifted by at most a couple of points, and nulls stayed null.The gemini-2.5-flash line-drawing cell was the sole status change: +10pp at p = 0.021 under gpt-5-mini became +8pp at p = 0.096 under gpt-5-nano, consistent with the paper’s caution about that cell.Source: §4.2 · p. 12
  • Human labels estimated differential judge bias at +3.8pp (95% CI [−8.3, +15.9]), which does not fully protect the approximately +23pp effects.The interval includes zero and n ≈50 per arm is the binding constraint; the labels came from a neighboring defense-pipeline study using payload images rather than blank canvases. Agreement was κ = 0.84 in the image arm versus 0.74 in the text arm.Source: §4.2 · p. 12

5 Discussion

The discussion frames image-presence refusal as a form-level behavioral effect whose mechanism cannot be confirmed from black-box comparisons. It distinguishes that effect from content-based safety findings and shows that its direction depends on the checkpoint.

5 Discussion

  • A content-free image shifts refusal across image variants and the shift survives an instruction to disregard the image, but black-box evidence cannot identify its internal mechanism.The authors cannot inspect activations or vendor pipelines to confirm a mechanism.Source: §5 · p. 13
  • A placebo ladder shows that, on gemini-2.5-flash, asserting an attachment raises refusal even without an image, while merely mentioning the modality does not.The evidence therefore supports a form-level cue, but the responsive channel varies by checkpoint.Source: §5 · p. 13
  • The effect is checkpoint-specific rather than a serving-stack or VLM-wide property: qwen3-vl-8b shows hosted-scale inflation, while pixtral-12b’s attack success rises 35pp.A black-box comparison cannot identify whether post-training, the vision encoder, or pretraining corpus accounts for the checkpoint differences.Source: §5 · p. 13
  • The authors place their work across intrinsic modality shifts, visual jailbreaks, protective mechanisms, over-refusal measurement, and black-box routing artifacts.They describe their contribution as measuring a cost charged for how a request arrives, holding its ask fixed.Source: §5 · p. 13
  • Unlike content-calibration work where images appear in every condition, this study isolates image presence; Geng et al. (2025) do not measure that axis.The paper also distinguishes its form-level question from Zou et al. (2026), whose analysis centers on harmful-side distortion and a different direction of effect.Source: §5 · pp. 13–14
  • The paper distinguishes its form-level shortcut from Hinojosa, Grange, and Ghanem (2026), where a symbol depicted within an image carries the safety cue.The two lines converge on qwen3-vl-8b, which the authors treat as corroboration from an independent manipulation.Source: §5 · pp. 13–14

6 Conclusion

The conclusion reports that image presence shifts refusal on particular aligned checkpoints, with costs concentrated on sensitivity-adjacent benign prompts. The effect varies by image and checkpoint, and its benign cost is not matched by a consistent harmful-side benefit.

6 Conclusion

  • +23 to +51pp of borderline-benign refusal on hosted models, and +28pp on an open checkpoint, accompany image presence without a defense in the pipeline.The paper describes the effect as a threshold shift, not blanket caution.Source: §6 · pp. 14–15
  • The effect belongs to particular aligned checkpoints, not the serving stack or VLMs generally; tested open checkpoints range from no benign inflation to hosted-scale inflation.The authors report a step between releases within one model family and say they could not predict affected checkpoints from tested labels.Source: §6 · pp. 14–15
  • +51, +34, and +23pp of benign refusal contrast with 18, 5, and 6 points of prevented harm; the two sides do not track each other.On a checkpoint with only 2% harmful completions on plain harmful requests, the canvas still costs 29 points, with no computable benefit to set against it.Source: §6 · pp. 14–15
  • The cost spans all ten OR-Bench categories, while the first measurement sampled only two; sensitivity-adjacent benign traffic bears the refusal increase.The conclusion names privacy, self-harm, violence, and illegal activity among the affected topics.Source: §6 · pp. 14–15
  • The canvas’s effect varies with colour and pixel count, and its sign reverses on two open checkpoints, raising attack success by 33 and 39pp.The conclusion also reports attachment assertion alone moving refusal by +33pp on one checkpoint.Source: §6 · p. 15
  • The authors do not establish that post-training specifically introduces the behavior or whether attachment predicts risk in real traffic.Their black-box evidence cannot identify the responsible checkpoint component, and the real-world risk correlation is unmeasurable in this study.Source: §6 · p. 15, Table 14 · p. 15

Limitations

The measured image-presence behavior is limited to particular checkpoints, with several open-weight models showing no benign cost and no tested label predicting which checkpoints exhibit it. The study also leaves boundaries around model and image coverage, the serving-route control, and evaluation measures.

Scope of the claim

  • Three of six open-weight models show no benign cost, and the authors limit the positive claim to particular aligned checkpoints whose behavior they cannot predict from tested labels.The sign reversal on pixtral-12b establishes that direction is not fixed, but not what fixes it; the black-box study does not isolate the checkpoint mechanism or post-training from other release changes.Source: Limitations · p. 15

Model coverage

  • The full ladder and matched harmful set cover four hosted models and one open checkpoint, while the other five open checkpoints were measured only on the borderline rung.The generational trend comes from one family at one size class; attachment-by-mention was tested on one checkpoint, and the motivating hosted model remains unresolved.Source: Limitations · p. 15

Serving-route control

  • The same-weights serving-route control uses gemma-3-12b-it, whose +7 to +9pp effect is near the route bound, so it excludes route artifacts for larger hosted shifts but does not settle the question at that model’s effect size.A large-effect route test was unavailable in a configuration that was both vision-capable and self-servable.Source: Limitations · p. 15

Image coverage, and between-image variance

  • Each image-property level uses one fixed file, so paired prompt-level measurements do not estimate between-image variance and reported color or content effects may be artifact-specific.Natural photographs, complex scenes, synthetic noise, and adversarially optimized images are untested; generalizing property effects requires multiple independently generated images per level assigned randomly across prompts.Source: Limitations · pp. 15–16

Judge calibration and benign benchmark

  • Absolute attack-success rates are judge-relative: four judges differ by a factor of two to four on the same responses, and multi-judge evaluation was not run on these cells.The utility evaluation uses one benign benchmark with 100 prompts, and tested benign traffic does not cover mathematical, code-heavy, or multilingual prompts.Source: Limitations · p. 16

Ethics Statement

The study uses public safety-evaluation benchmarks and does not introduce attacks or release adversarial assets. It frames disclosure as net-positive because the cue is easy to discover, affected checkpoints are public, and benign users bear the documented cost.

Ethics Statement

  • The experiments use standardized public prompt benchmarks and introduce no attack or adversarial assets.The only manipulation is attaching a content-free image to an unchanged request.Source: Ethics Statement · p. 16
  • The authors judge disclosure net-positive because the cue is trivially discoverable, affected checkpoints are public, and benign users cannot discover or avoid the cost without disclosure.The stated affected users ask benign questions about privacy, self-harm, violence, and illegal activity.Source: Ethics Statement · p. 16
  • Appendix examples deliberately use prompts whose harm is informational rather than physically actionable.Source: Ethics Statement · p. 16

A Sample images and variant illustration

The image variants are unrelated to the requests and fixed within each arm; they distinguish readable caption text, a white blank, and a content-bearing line drawing without readable text.

A Sample images and variant illustration

B Predictions, ablation counts, and judge robustness

Measurements use paired prompts with unchanged text across contrast arms, greedy decoding, and standardized judging. The paper reports single-run results, with provenance and per-row audit trails for evaluation.

B Predictions, ablation counts, and judge robustness

  • Every cell averages 100 paired prompts, and contrast arms share a canonical text step that is byte-identical across arms.Within-cell variance comes from the target model’s sampling alone.Source: B Predictions, ablation counts, and… · p. 17
  • All calls use greedy decoding at temperature 0.0; provider-default sampling parameters are no-ops under greedy decoding, and each reported number comes from one pinned run.Because two providers do not expose a random seed, repeated runs are reproducible in distribution rather than bit-for-bit.Source: B Predictions, ablation counts, and… · p. 17
  • Every cell has no defense in the loop, and contrasts across collection dates are avoided because reported comparisons stay within a single job.Table 15 records collection windows, target models, and judge backbones for headline results.Source: B Predictions, ablation counts, and… · p. 17
  • All attack-success and refusal numbers are scored on gpt-5-mini using standardized benchmark classifiers, with per-prompt judge outputs and reasoning stored alongside model responses.Stored responses collected before the backbone choice were rescored without re-querying targets, and individual judgments can be inspected without rerunning the judge.Source: B Predictions, ablation counts, and… · p. 17

C Statistical tests

Contrasts are paired at the prompt level, enabling exact tests on discordant outcomes and Wilson intervals for reported proportions.

C Statistical tests

  • Every contrast uses an exact two-sided McNemar test on discordant pairs and reports Wilson score 95% intervals for proportions.The paper prefers Wilson intervals because some cells are near 0% or 100%, where normal-approximation intervals misbehave.Source: C Statistical tests · p. 17
  • Because each contrast’s arms come from one canonical text, outcomes are paired at the prompt level.Source: C Statistical tests · p. 17
  • Inline discordant-pair counts make each exact McNemar p-value independently recomputable.Source: C Statistical tests · p. 17

D Multiplicity

The paper declares test families by claim and applies within-family multiplicity correction, while also checking the primary claim against a global Bonferroni threshold. Most results retain their status, with negative findings interpreted through power bounds.

D Multiplicity

  • Families are defined by claim; only F1 is confirmatory, with Holm–Bonferroni used within families except for the ten-arm property sweep, which uses Benjamini–Hochberg.Source: D Multiplicity · p. 18
  • Exactly two of 69 results change status after correction: gemma-3-12b-it’s +9pp and llava-1.5-7b’s −8pp become non-significant.No claim rests on either result; every other test retains its status.Source: D Multiplicity · p. 18
  • Thirty-six of 69 tests survive global Bonferroni at α/69 = 7.3 × 10^-4, including the three hosted models with the effect and the open checkpoint.The primary attachment contrasts, generational step, eight property arms, both harmful-axis sign inversions, both direct F9 and 17 of 30 stratified per-category tests also clear that threshold.Source: D Multiplicity · p. 18
  • Negative results are interpreted through power bounds: serving routes support equivalence, the image-word placebo excludes effects above ∼11pp at 25 discordant pairs, and three open-weight nulls apply only to sampled models.The paper retracts the broader reading of those three nulls after a fourth open checkpoint showed an effect.Source: D Multiplicity · p. 18
  • The released analysis script recomputes exact McNemar p-values from published discordant counts and matches all 19 available typeset values.Source: D Multiplicity · p. 19

E Reproducibility

The paper documents model identifiers, prompt variants, serving settings, and image construction, and releases configurations, analysis scripts, and per-prompt evaluation rows to support inspection and recomputation.

E Reproducibility

  • Hosted models are identified by provider API names, while open-weight checkpoints are identified by their Hugging Face repository.The model registry records that the self-served gemma checkpoint and AWS Bedrock route use the same weights despite identifiers differing by one character.Source: E Reproducibility · p. 19, Table 17 · p. 19
  • The instruction-mitigation arm uses a neutral system message telling the model to disregard the attached placeholder image and answer as if none were attached.Source: E Reproducibility · p. 19
  • The placebo ladder compares no system message, a placeholder-file instruction, and a byte-identical placeholder-image instruction.The no-message arm forwards an absent system prompt, not an empty one.Source: E Reproducibility · p. 19
  • Calls use temperature 0, top_p 1.0, top_k 0, one completion, and a 16,384-token output cap; hosted providers do not honor the harness seed.Hosted calls are not claimed to be bit-for-bit reproducible, so contrasts are collected within single jobs.Source: E Reproducibility · p. 19
  • Within each image arm, prompts receive a byte-identical file, and programmatically generated canvases are specified by width, height, and fill.The line drawing and caption images are included in the supplementary archive.Source: E Reproducibility · p. 19
  • The supplementary archive includes experiment code, YAML configurations for every reported run, analysis scripts, and per-prompt model responses and judge outputs.The exact 100-prompt benchmark slices are redistributed by index rather than redistributing the benchmarks.Source: E Reproducibility · p. 19
Loading 2609.26174v1…