PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks

Boyuan Chen, Yehia Dawoud, Hailemariam Mersha, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique

arXiv:2610.07009v1cs.CRcs.AIcs.LG

TL;DR

Image-to-text jailbreaks can place harmful intent in images, but aggregate attack scores cannot identify which image properties contribute to safety failures. This study compares attack constructions while holding the harmful query fixed, finding one bounded attribution to query-specific relatedness and no resolved effect of tile-count structure.

  • Problem

    Because existing image-to-text attacks change several image properties at once, their aggregate ASRs cannot identify which property contributes to a safety failure.

    Source: §1 · p. 1

  • Method

    The study uses paired image-construction comparisons with byte-identical harmful instructions, measuring image properties and attributing effects to one descriptor only when it alone changes.

    Source: §1 · p. 2

  • Results

    On Qwen3-VL-8B, removing query-specific relatedness lowers ASR by 0.119; the region-count test finds no detectable effect, leaving tile-count structure unresolved.

    Source: §5.3 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Controlled comparisons can support bounded image-property attribution, but attack rankings alone cannot identify which image properties matter.

    Source: §6 · p. 8

    The main findings come from open-weight victims with headroom and should not be generalized to the full panel; ASR is conditional on the rubric judge.

    Source: §7 · p. 9

Abstract

from arXiv · show

Image-to-text jailbreaks place harmful intent in text, image content, or the relationship between them. We examine image-side factors across four published attack families on a 313-prompt StrongREJECT slice, using five multimodal models and an additional appendix evaluation of InternVL3.5-8B. The harmful instruction is held constant across conditions; the baseline matrix uses one draw per prompt, and paired ablations use three draws with an automated rubric judge. A bare harmful query, with or without a benign unrelated image, produces little attack success on most victims, while attack images substantially increase it. First, per-tile entropy and JPEG size do not reliably distinguish attack tiles from size-matched benign distractors, limiting density-only screening. Second, earlier tile-count ladders were confounded by payload visibility. A corrected region-count test found no detectable effect, so the role of tile-count structure remains unresolved. Third, on Qwen3-VL-8B, the E4 manipulation that removes query-specific relatedness lowers ASR by about 0.12. This supports a bounded attribution to the relatedness manipulation, although image-text congruence remains unmeasured. A within-category control reproduces the direction on overlapping stimuli. Five tests survive the global statistical correction, but only E4 supports attribution to one measured descriptor; the within-category result is a robustness check, not a separate attribution. These conclusions remain conditional on the rubric judge.

1 INTRODUCTION

The paper asks which image-side properties contribute to image-to-text jailbreaks, since existing attacks vary several properties at once and their aggregate success rates cannot distinguish explanations. It uses controlled comparisons and finds one bounded attribution to query-specific relatedness, while tile-count effects remain unresolved.

1 INTRODUCTION

  • Because attacks vary several image properties together, their aggregate ASRs cannot identify which property drives safety failures, limiting what defenses aimed at individual causes can infer.The paper organizes image-side factors as compositional complexity, information density, tile provenance, and semantic alignment.Source: §1 · pp. 1–2
  • The study keeps the harmful instruction byte-identical, measures image properties before evaluation, and attributes a contrast to one descriptor only when it is the only measured property that changes.The baseline matrix uses one draw per prompt; paired ablations use three draws on the 313-prompt StrongREJECT slice, with bootstrap intervals, paired tests, and a global multiplicity correction.Source: §1 · p. 2
  • A corrected region-count test found no detectable effect, so the experiments do not establish whether tile-count complexity changes ASR.Earlier tile-count ladders were confounded by payload visibility.Source: §1 · p. 2
  • A padding and border difference between two CS-DJ renderings is associated with a 0.19 ASR gap on GEMMA-3-4B, outside the four image-property axes.The authors caution that this construction contrast does not establish an effect of tile-count complexity.Source: §1 · p. 2
  • Attack composites substantially increase ASR over text-only and benign-image conditions, but per-tile entropy and JPEG screening do not reliably separate attack tiles from benign distractors.The density-screen result is specific to the CS-DJ construction; the authors conclude that a practical filter would need signals beyond the tested density descriptors.Source: §1 · p. 3
  • On Qwen3-VL-8B, removing query-specific relatedness lowers ASR by 0.119, the only tested contrast supporting attribution to one measured descriptor.A within-category harm-salience control reproduces the direction on overlapping stimuli, but is a robustness check rather than a separate attribution.Source: §1 · pp. 2–3

2 RELATED WORK

Prior image-to-text jailbreaks place harmful intent in typographic, structured, semantically grounded, or visually complex images, often changing multiple properties together. This paper’s image-focused comparisons complement benchmark and prompt-level analyses, while leaving some safety mechanisms untested.

2.1 IMAGE-TO-TEXT JAILBREAK ATTACKS

  • FigStep and MML use typographic images, VisCo builds image-grounded scenarios, and CS-DJ emphasizes compositional complexity; these attack designs vary multiple image properties.Other described approaches include flowcharts, coherent sub-images, aggregate-harmful narratives, and optimization-based attacks, which require more model access than single-turn black-box interaction.Source: §2.1 · p. 3

2.2 BENCHMARKS AND EVALUATION PROTOCOLS

  • MM-SafetyBench pairs harmful text with images whose density, provenance, and content vary together, whereas this paper reassigns a fixed scene set across queries to test relatedness with measured density and provenance held constant.JailBreakV-28K’s finding that image type affects attack success motivates the paper’s separate analysis of image properties.Source: §2.2 · p. 3

2.3 RELATION TO PROMPT-LEVEL AXIS FRAMEWORKS

  • Unlike Li et al.’s on-topicness and OOD-intensity framework over textual prompts, this paper varies image construction while keeping the harmful instruction byte-identical.The design complements rather than generalizes analyses that change the prompt itself.Source: §2.3 · pp. 3–4

2.4 MULTIMODAL SAFETY AND DEFENSE

  • Prior safety work addresses harmful visual content through safety fine-tuning, safety prompts, and prompt-and-response classification; this study uses a StrongREJECT rubric judge and audits it against human labels.The cited VLGuard result offers a possible explanation for off-topic harmful scenes suppressing compliance, but the paper does not test that mechanism directly.Source: §2.4 · p. 4

3 METHODOLOGY

The methodology treats existing attacks as diagnostic probes and compares image constructions while keeping the harmful query fixed. It measures distinct image-side axes and limits single-descriptor attribution to contrasts where only one measured property changes.

3.1 THREAT MODEL

  • The study characterizes single-turn black-box image-to-text jailbreaks as an attack image and benign textual wrapper submitted together, with responses scored by an automatic judge.The attacker can construct the image and wrapper but has no access to model weights, gradients, or token probabilities.Source: §3.1 · p. 4

3.2 TAXONOMY

  • The taxonomy separates compositional complexity, information density, tile provenance, and semantic alignment; entropy and JPEG@85 measure distinct aspects of density, not structure.Semantic alignment is query-relative, while provenance records whether composite tiles come from natural photographs or synthetic generation.Source: §3.2 · p. 4

3.3 ATTACK PLACEMENT AND IMAGE-SIDE METRICS

  • The attack families occupy different combinations of image-side properties, and cross-attack ASR comparisons therefore confound complexity and alignment.The four axes do not cover instruction encoding or decodability, nor tile geometry and border statistics; a padding-construction difference is associated with a 0.19 ASR gap on Gemma-3-4B.Source: §3.2 · p. 5, §3.3 · p. 5

3.4 PAIRED ABLATIONS

  • Paired ablations keep the harmful instruction fixed: the complexity experiment varies FigStep image construction, while the alignment experiment varies CS-DJ distractor tiles.The latter changes multiple measured descriptors, including provenance, so it does not isolate semantic alignment; E4 instead holds generated-scene provenance fixed while changing query-specific relatedness.Source: §3.4 · p. 5
  • An ASR difference is attributed to one descriptor only when that is the only measured property that changes; otherwise, the contrast is jointly attributed.E4 holds entropy, JPEG size, tile count, and provenance fixed while changing scene-to-query relation, but cannot distinguish topical relatedness from image-text congruence.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments cover five victim models and the full StrongREJECT prompt slice, with a baseline attack matrix and paired ablations. The same prompt order supports within-prompt paired tests across attack-victim cells.

4 EXPERIMENTS

  • Experiment A.37 queries every attack-victim cell of the 5×5 panel, while Experiments A.38 and A.39 run the complexity and alignment ablations as within-prompt paired contrasts.Experiment A.40 re-slices those experiments by reasoning-augmented versus standard models without new queries.Source: §4 · p. 6
  • The victim panel contains two API-served models, two open-weight MLLMs, and one reasoning-augmented multimodal model.The models are GPT-4o, Claude Sonnet 4.5, Qwen3-VL-8B-Instruct, Gemma-3-4B-IT, and GLM-4.1V-9B-Thinking.Source: §4 · p. 6
  • Every attack-victim cell uses the same full StrongREJECT slice of 313 prompts across six harm categories, preserving prompt order for paired tests.The baseline panel includes four attack families in five configurations; the ablations add three FigStep and three CS-DJ variants.Source: §4 · p. 6

5 RESULTS

Across the baseline matrix, attack success varies sharply by victim. Controlled tests find no detectable effect of distractor-region count, while the E4 relatedness manipulation lowers ASR on Qwen3-VL-8B; image-contribution controls show attack composites raise ASR on most victims.

5.2 EXPERIMENT 2: COMPLEXITY ABLATION

  • The payload-geometry first rung is unresolved at the resolution limit (p=0.36); adjacent-step contrasts are primary, while the ordered endpoint is exploratory.Earlier tile-count ladders were confounded by payload visibility, and the controlled region-count test does not resolve whether tile-count structure matters.Source: §5.2 · p. 6, Table 1 · p. 7, §5.3 · p. 7

5.3 COMPOSITIONAL LADDER INSIDE THE CS-DJ FAMILY

  • Adding distractor regions produces no detectable ASR increase: Gemma-3-4B changes by −0.006 from one to nine regions, while Qwen3-VL-8B rises nonsignificantly by +0.035 (p = 0.25).The test holds the grid, canvas, JPEG footprint, and three payload tiles fixed; a separate white-padding construction contrast is associated with a 0.19 ASR gap on Gemma.Source: §5.3 · p. 7

5.4 EXPERIMENT 3: ALIGNMENT ABLATION

  • On Qwen3-VL-8B, removing query-specific relatedness lowers ASR by 0.119; image-text congruence remains unmeasured.The within-category control reproduces the direction on overlapping stimuli, so it is a robustness check rather than an independent attribution; other victims show no significant E4 effect or have interpretation limits.Source: §5.3 · pp. 7–8

5.1 BASELINE CROSS-ATTACK MATRIX

  • MML variants reach 0.96/0.95 ASR on GEMMA-3-4B but ≤0.02 on CLAUDE SONNET 4.5, showing sharp victim variation.CS-DJ, FigStep, and VisCo also vary across victims; these baseline comparisons are descriptive and do not identify why outcomes differ.Source: §5.1 · p. 6, Table 2 · p. 8

5.5 IMAGE CONTRIBUTION CONTROL (E-1)

  • On GPT-4O, ASR is 1.3% for the bare query, 0.0% with a benign unrelated image, and 28.4% with a CS-DJ image on identical prompts.The pattern holds across the panel, with one flagged exception on GLM-4.1V-TH.Source: §5.5 · p. 8

6 DISCUSSION AND CONCLUSION

The results support only a bounded attribution to query-specific relatedness on Qwen3-VL-8B, not a general semantic-alignment mechanism. More broadly, controlled comparisons clarify which image-side claims the experiments can support.

6 DISCUSSION AND CONCLUSION

  • Only E4 supports attribution to one measured descriptor; its within-category control is a robustness check, and image-text congruence remains unmeasured.The region-count test finds no detectable effect, while earlier tile-count ladders are confounded by payload visibility; whether tile-count structure matters remains open.Source: §6 · p. 8
  • Per-tile density descriptors do not reliably separate attack tiles from benign photo tiles, so screening would need signals beyond those measured here.The authors say pad-surround and content-fill changes may motivate screening or training for typographic-payload robustness on victims where they occur.Source: §6 · p. 8
  • Controlled comparisons can help evaluate other attack families when instructions stay fixed, image properties are measured, and multi-descriptor changes are labeled jointly attributed.The attention-mass probe finds disproportionate attention to payload tiles on Qwen3-VL-8B, but attention share does not vary with ASR and does not explain E4.Source: §6 · p. 8

7 LIMITATIONS

The findings are limited by the tested victims, proxy image-side measures, unresolved controls, and judge dependence. Most comparisons do not isolate one image property while holding every other property fixed.

7 LIMITATIONS

  • The main findings come from open-weight victims with headroom and should not be generalized to the full panel.Image-side measures are proxies, and ASR is rubric-judge-conditional; results differ from a non-OpenAI judge by up to 0.18.Source: §7 · p. 9
  • E4 does not separate query-specific relatedness from image-text congruence, and no benign-paraphrase control was run.The attention-mass probe is null on both effect-carrying victims, while the in-model relatedness probe is too noisy to discriminate.Source: §7 · p. 9
  • Several controls remain unrun, including a relevant scene paired with a benign restatement to examine image-text congruence.The paired design's minimum detectable effects are about 0.06–0.10; an outcome-blind replication with 830 paired prompts would provide 80% power for an effect of 0.06.Source: §7 · p. 9

ETHICAL CONSIDERATIONS

The paper evaluates published attacks and diagnostic image variants, not a new deployable exploit or fine-tuned attacker. It withholds raw harmful completions while describing access arrangements and judge auditing.

ETHICAL CONSIDERATIONS

  • The study evaluates published attack families and paired image variants; it releases no new deployable exploit or fine-tuned attacker model.Harmful prompts come from the public StrongREJECT benchmark.Source: ETHICAL CONSIDERATIONS · p. 9
  • Raw harmful completions are omitted, and the public archive contains judge scores but not raw response text.A separate access-controlled archive contains selected stimuli and verbatim model outputs.Source: ETHICAL CONSIDERATIONS · p. 9
  • The rubric judge was audited against human labels (agreement 0.800, κ=0.594); the audit found no judge-error interaction that changes the conclusions.The authors recommend evaluating defenses against adaptive adversaries before deployment.Source: ETHICAL CONSIDERATIONS · p. 9

A.1 PER-CATEGORY BREAKDOWN ON THE API REFERENCE VICTIM

GPT-4O’s category breakdown shows similar performance across both MML variants except for Sexual Content, while CS-DJ and its variants exhibit category-level differences that do not establish causes.

A.1 PER-CATEGORY BREAKDOWN ON THE API REFERENCE VICTIM

  • Both MML variants score 0.62 on Sexual Content, their exception to otherwise similar performance across categories.Source: §A.1 · p. 12
  • CS-DJ ranges from 0.20 to 0.42 across categories, peaking at 0.42 for Disinformation/Deception and 0.39 for Non-violent Crimes.Source: §A.1 · p. 12
  • FigStep ablation variants remain at 0.00 across categories, while CS-DJ noise and unrelated shift some category rates by up to 0.18 without changing overall ASR by more than 0.04.Relevant stays within 0.04 of the CS-DJ baseline in each category; the category variation does not establish whether layout or tile content causes it.Source: §A.1 · p. 12

A.2 MANIPULATION CHECK

The manipulation checks document shifts in image statistics and CLIP alignment across conditions, while noting that several comparisons change multiple descriptors together. The GPT-4O FigStep judge scores also sit far below the ASR threshold.

A.2 MANIPULATION CHECK

  • Table 4 recomputes Exp. 2/3 measurements on post-resize images for n=313 per condition, but Exp. 3 conditions are unmatched in entropy, compressibility, and tile provenance.Source: Table 4 · p. 12
  • On the full StrongREJECT slice (n=313 per condition), whole-image CLIP ViT-L/14 cosine similarity was higher for relevant than noise or unrelated in Exp. 3 (0.254 vs. 0.197/0.187), and query-aligned than random surround in Exp. 2 (0.216 vs. 0.124).Source: Table 5 · p. 13
  • For the GPT-4O FigStep cell, all 312 responses scored with the StrongREJECT rubric fell far below the 0.5 binarization threshold, with 99th percentile 0.003 and maximum 0.011.Source: Table 6 · p. 13
  • Exp. 2 entropy rises from 0.90 bits/pixel at baseline to 2.57 at pad and 6.48/6.78 for distractor conditions; Exp. 3 ranges from 5.97 to 6.62 bits/pixel.Exp. 3 JPEG@85 ranges from 166.3 to 114.6 KB, and tile provenance also changes; these shifts do not establish an isolated ablation effect.Source: §A.2 · pp. 12–13

A.3 ALIGNMENT VERIFICATION (CLIP)

CLIP measurements verify that the alignment manipulations change the image-query similarity proxy, but do not establish an ASR effect. The GPT-4O FigStep judge scores support interpreting its zero ASR as hard refusal rather than a threshold artifact.

A.3 ALIGNMENT VERIFICATION (CLIP)

  • In Exp. 3, median whole-image alignment is higher for relevant than noise or unrelated (0.254 vs. 0.196/0.186); in Exp. 2, the aligned surround exceeds the random natural surround (0.216 vs. 0.124).These measurements verify changes in CLIP alignment, not a causal effect on ASR.Source: §A.3 · p. 13
  • The E4 gen-unrelated cell has CLIP 0.196, close to unrelated (0.187) and below relevant (0.254), consistent with assigning each generated scene to a different query.Source: §A.3 · p. 13
  • All 312 scored GPT-4O FigStep responses fall far below the 0.5 binarization threshold, supporting genuine total refusal rather than a threshold artifact.The 99th percentile is 0.003 and the maximum is 0.011.Source: §A.3 · p. 14

A.5 EXPERIMENT 3: ALIGNMENT ABLATION

Experiment 3 holds the CS-DJ layout fixed while varying distractor content. Qwen3-VL-8B shows an ordered ASR rise, but that ladder is descriptive because image descriptors also vary; E4 supplies the separate relatedness attribution.

A.5 EXPERIMENT 3: ALIGNMENT ABLATION

  • Qwen3-VL-8B shows a +0.10 ordered-endpoint ASR rise (Holm p=0.002), but the association is descriptive because image descriptors also vary.Table 7 reports the full ablation with the CS-DJ 3×4 layout fixed and nine distractor tiles varied among noise, unrelated, and relevant content.Source: §A.5 · p. 14
  • The other four models show no significant monotone rise across the distractor conditions.Source: §A.5 · p. 14
  • The ordered noise-to-relevant endpoint is exploratory; the separate E4 result provides the paper’s relatedness attribution.Table 9 reports paired-prompt ΔASR estimates and bootstrap confidence intervals for every contrast.Source: §A.5 · p. 14

A.6 EXPERIMENT 3: PROVENANCE CONTROL (E4)

E4 holds generated-scene provenance constant while changing query-specific relatedness. On Qwen3-VL-8B, removing that relatedness lowers ASR, while other victim rows do not establish a general pattern.

E4 results

  • On Qwen3-VL-8B, gen-unrelated ASR (0.17) is below relevant (0.29) and unrelated (0.24), consistent with a controlled relatedness result.E4 reassigns each generated scene to a different query, keeping provenance constant while changing its relation to the harmful query.Source: §A.6 · p. 14

E4 control

  • The reassignment is supported by whole-image CLIP alignment: gen-unrelated is 0.196, near unrelated (0.187) and below relevant (0.254).Source: §A.6 · p. 14

Scope

  • Claude’s E4 row cannot adjudicate provenance or relatedness because provider-side interception varies substantially across its Experiment 3 conditions.Gemma’s ordering is descriptive, not an isolated provenance effect, because its cells also differ in other descriptors.Source: §A.6 · p. 14

A.7 WITHIN-CATEGORY DERANGEMENT (HARM-SALIENCE CONTROL)

A same-category derangement tests whether cross-category harm salience explains the E4 result. The within-category control preserves the direction of the Qwen3-VL-8B effect, with a slightly smaller estimated difference.

Control design

  • The within-category derangement has zero cross-category pairs and zero fixed points, with otherwise identical attack variant, stimulus bytes, sampler (k=3), and rubric judge.Every StrongREJECT category has at least 50 prompts, permitting an internal derangement across all 313 prompts.Source: §A.7 · pp. 14–15

Motivation

  • The original cross-category derangement assigns prompts to scenes from a different StrongREJECT category 84.03% of the time (263/313).This motivated testing whether cross-category harm salience, rather than query relatedness, could account for the E4 drop.Source: §A.7 · p. 14

Results

  • On Qwen3-VL-8B, within-category relevant→gen-unrelated ASR falls by 0.089 (95% CI [−0.131,−0.049]), matching the cross-category effect’s direction.The within-category estimate is slightly smaller than the cross-category ∆ of −0.119, but its confidence interval remains clear of zero.Source: §A.7 · p. 15

Interpretation

  • The control supports the query-relatedness reading: cross-category harm salience is not the driver of the E4 effect, though the within-category ∆ is slightly smaller.The refusal-driven classification remains unchanged, and the within-category result is a robustness check rather than a separate attribution.Source: §A.7 · p. 15

A.8 DELTA-ASR CONFIDENCE INTERVALS

This appendix qualifies image-side ASR contrast estimates through sampling, judge, power, legibility, and provider-response checks, while assessing which contrasts support attribution and which remain confounded or unresolved.

A.9 EXPERIMENT 2: PAYLOAD-SCALE CONTROL (E5)

  • On Gemma-3-4B, the E5 payload-geometry gap is +0.07 in the single-draw run but does not clear the k=3 sampling test; pad surround raises ASR at both scales.The surround effects are +0.10 at quarter scale and +0.125 at full scale. In the k=3 ladder test, geometry is unresolved (mean delta +0.012, permutation p=0.36), while the compositional and content-fill rungs clear the noise floor; sampler differences matter when comparing these estimates.Source: §A.9 · pp. 15–16, Table 10 · p. 16, §A.19 · p. 22

A.10 CONSOLIDATING RANDOM-INTERCEPT MIXED MODEL

  • The categorical mixed model agrees with paired tests: Experiment 2 effects are largest on Gemma and GLM, while the Experiment 3 noise-to-relevant association is largest on Qwen but does not isolate alignment.In the five-model panel, the FigStep-to-distract-random gain is +0.35 on Gemma, +0.16 on the MLRM, and +0.00, +0.03, and +0.01 on GPT-4o, Claude, and Qwen, respectively; this is localized, not general evidence about reasoning models. The MLRM has the highest baseline ASR except MML-base64.Source: §A.10 · p. 17, §A.11 · p. 17, Table 11 · p. 17

A.12 MINIMUM DETECTABLE ALIGNMENT EFFECT

  • The n=313 minimum-detectable ∆ASR bounds are 0.06–0.10; Qwen’s ordered endpoint exceeds its 0.081 threshold, while the E4 control supports bounded relatedness attribution.Gemma’s two Experiment 2 headline steps clear their power bounds, but GLM’s compositional step is power-limited. Gemma’s k=3 sampling floor is 0.10–0.12, and the rubric-judge baseline-to-distract-random contrast (∆=0.349) still exceeds it.Source: §A.12 · pp. 17–18, Table 14 · p. 18, §A.13 · p. 19

A.14 THRESHOLD AND JUDGE ROBUSTNESS

  • Qwen’s noise < unrelated < relevant ordering and Gemma’s noise ≥ relevant > unrelated ordering persist across three thresholds, supporting ordering stability but not causal interpretation of every contrast.The largest local-versus-rubric judge gap is 0.18 absolute ASR at distract-random, so magnitudes near that disagreement band are not judge-invariant. Human annotation of 250 samples yielded agreement 0.800 and κ=0.594, with a small implied ASR shift that did not reverse headline orderings; continuous-score tests likewise confirmed key contrasts and a flat Qwen Experiment 2 content-fill result.Source: §A.14 · pp. 19–20, §A.15 · p. 20, Table 17 · p. 20

A.16 PAYLOAD LEGIBILITY LADDER

  • The payload is legible wherever it appears: character error rate stays approximately flat (0.115 → 0.162) as ASR rises threefold across the ladder.This bounds rather than removes the resolution confound: the per-tile encoder budget declines, but measured legibility does not cross the threshold at which the payload becomes hard to read.Source: §A.16 · pp. 20–21, Table 18 · p. 21

A.17 ALIGNMENT IN EACH VICTIM’S OWN VISION ENCODER

  • Neither open victim’s own vision tower recovers the predicted alignment ordering, and the exploratory relatedness-log-odds probe is too noisy to confirm or contradict the CLIP-space null.Gemma ranks noise above relevant; Qwen groups noise and relevant above unrelated. The probe’s per-stimulus variance is large relative to between-condition differences, so it supports only the claim that no in-model alignment ordering was recovered.Source: §A.17 · pp. 21–22, Table 19 · p. 21

A.18 CLAUDE’S EXP. 3 NOISE CELL: PROVIDER-SIDE RECOVERY FROM LOGS

  • Claude’s 0.00 ASR in the Experiment 3 noise cell reflects provider-side intervention, not a victim-internal refusal or evidence about alignment.All 313 noise requests returned empty content or HTTP 400; interception also varied by condition, making the non-intercepted responses a selected subset and leaving the row interpretation-limited.Source: §A.18 · p. 22, Table 20 · p. 22

A.20 GLOBAL MULTIPLICITY CORRECTION

  • Five of 13 confirmatory tests pass both global corrections, but only E4 supports attribution to the relatedness manipulation; image-text congruence remains unmeasured.The within-category E4 control is a robustness check on overlapping stimuli, not an independent attribution; an outcome-blind, pre-committed replication is needed before generalizing. Table 9 reports paired-prompt ∆ASR with bias-corrected 95% bootstrap confidence intervals for headline contrasts, separately from p-values.Source: §A.8 · p. 15, Table 9 · p. 16, §A.20 · p. 23

A.22 BENIGN-REGION SCREENING TEST

  • JPEG@85 achieved AUC 0.63 but flagged 67% of benign tiles at 80% TPR; entropy reached AUC 0.42 and the two-feature screen AUC 0.68.The combined screen still flagged 76% of benign tiles at 80% TPR, so the measured density descriptors do not support a practical standalone filter on realistic benign traffic.Source: §A.22 · pp. 24–25, Figure 3 · p. 24

A.23 REGION-COUNT ISOLATION (STIMULUS)

  • The corrected region-count test held payload visibility and image geometry fixed, yet showed no ASR effect on either effect-carrying victim.The stimulus varied the number of distinct visual regions across one, three, and nine while keeping the payload tiles, canvas, pixel budget, and resolution constant; whether tile-count structure matters remains open.Source: §A.23 · p. 25, Figure 4 · p. 25

A.24 CONFOUNDED COMPOSITIONAL LADDERS (AUDIT RECORD)

  • ASR rose with tile count in both earlier ladders, but payload text also became visible, so neither ladder cleanly isolates compositional complexity.The fixed-canvas ladder likewise placed the typographic payload only at the largest rung; its increases therefore cannot be attributed to tile count alone.Source: §A.24 · pp. 25–26, §A.27 · p. 27

A.25 DISTRACTOR-INFORMATIVENESS CONTROL

  • On Gemma, blurring and phase scrambling raised ASR to 0.553 and 0.607 versus 0.409 for sharp photographs, so provenance alone does not explain the ordering.The degraded photographs retained photographic provenance but moved ASR toward the noise condition, implicating distractor informativeness as a qualification rather than establishing an isolated mechanism.Source: §A.25 · p. 26, §A.30 · p. 30

A.26 LADDER DECOMPOSITION AND PAYLOAD-GEOMETRY CONTROL (E5)

  • In E5, the layout/content and content-fill steps were significant under the k=3 criterion, while the payload-geometry step was unresolved.On Gemma, the quarter-versus-full payload contrast was +0.012 with permutation p=0.36 at k=3; the surround and content-fill steps also move density descriptors alongside jointly attributed mechanisms. Exp. 2 distract-random versus distract-relevant does not isolate relatedness.Source: §A.26 · p. 27

A.27 PREDECESSOR LADDER CONTROLS, PADDING CONSTRUCTION, AND CROSS-BENCH REPLICATION

  • Gemma’s padded baseline had ASR 0.60 versus 0.41 for the force-squared unrelated cell, while SafeBench’s Qwen E4 gap was directional but nonsignificant.The construction gap co-varied with white-pixel fraction and resampling. On SafeBench, Qwen’s E4 difference was −0.06 against an MDE of 0.13, so it was not a confirmed replication.Source: §A.27 · pp. 28–29

A.28 ADDITIONAL HEADROOM VICTIM (INTERNVL3.5-8B)

  • InternVL3.5-8B reproduced descriptive complexity and provenance-adjacent orderings, while Claude’s provider-intercepted row supports no alignment or provenance claim.InternVL’s gen-unrelated cell did not isolate query-specific relatedness; Claude’s input-dependent interception reverses the raw versus conditioned contrast, making neither reading an unbiased estimate of model behavior.Source: §A.28 · p. 29, §A.29 · pp. 29–30

A.33 SALIENCE-STRATIFIED READ OF THE E4 CELL (EXPLORATORY)

  • The salience-stratified E4 read was exploratory and small; deferred controls leave the incongruity, dose-response, and broader tile-count questions open.Harm-salience had a weakly negative coefficient, but category-stratified results did not confirm suppression. The proposed benign-paraphrase, retrieval-based high-alignment, and corrected tile-count tests were not run; the remaining contrasts stay joint or unresolved.Source: §A.21 · p. 24, §A.31 · p. 30, §A.32 · p. 30

A.35 ATTRIBUTION RULE

  • Five contrasts pass the global correction, but only E4 is attributed to a single descriptor: query-specific relatedness on Qwen3-VL-8B lowers ASR by 0.119 (p < 0.001).Entropy, JPEG size, tile count, and provenance stay fixed; image-text congruence remains unmeasured. The within-category control reproduces the direction on overlapping stimuli but is a robustness check, not another attribution.Source: §A.35 · pp. 31–32

A.36 STATISTICAL ANALYSIS

  • Paired-ablation effects use within-prompt k=3 means and a permutation-test threshold of p < 0.05, while ASR cells report bias-corrected 95% bootstrap intervals over 104 resamples.Paired comparisons use McNemar tests with Holm-Bonferroni adjustment; every headline ASR and delta is rubric-judge conditional, and judge differences can vary with the manipulated factor.Source: §A.36 · pp. 32–33

A.39 EXPERIMENT 3: CONTROLLED SEMANTIC ALIGNMENT ABLATION (PROTOCOL DETAIL)

  • Experiment 3 changes distractor content from noise to unrelated photos to relevant generated scenes, but entropy, JPEG size, and tile provenance also change, so its ordered endpoint does not isolate alignment.Its post-resize entropy means are 5.97, 6.40, and 6.62 bits/px, while JPEG@85 falls from 166.3 to 114.6 KB; E4 supplies the separate bounded relatedness attribution.Source: §A.39 · p. 34, §A.46 · p. 36

A.41 ATTENTION-COMPETITION PROBE (CONCLUSION DETAIL)

  • On Qwen3-VL-8B, the payload receives 0.39–0.47 of image-token attention versus a 0.25 uniform baseline, but the relevant-to-gen-unrelated change is only +0.0017.Gemma-3-4B remains near uniform (0.243–0.256), with a +0.0029 relevant-to-gen-unrelated difference; the probe finds no condition-level attention shift explaining ASR.Source: §A.41 · p. 34

A.45 IMAGE CONTRIBUTION CONTROL (E-1)

  • The bare harmful query and the same query paired with a benign unrelated image both have low ASR across the panel, so attack-image success is not simply inherited from text-only compliance.The appendix notes a small exception for GLM-4.1V-Thinking.Source: §A.45 · p. 36, Table 31 · p. 36

A.48 DEFENSIVE IMPLICATIONS (CONCLUSION DETAIL)

  • Detecting 80% of attack tiles also flags 67% of benign tiles; JPEG@85 has AUC 0.63 and entropy AUC 0.42, making density alone a poor standalone filter.Any defense motivated by these findings needs evaluation against adaptive adversaries.Source: §A.48 · p. 37

A.50 EXP. 2 COMPLEXITY LADDER: FULL PER-VICTIM READING

  • On GEMMA-3-4B, ASR rises from 0.29 at FigStep to 0.46/0.64/0.60 across variants, while the MLRM rises from 0.73 to 0.81/0.89/0.86 (all p<0.001).GPT-4O remains at 0.00 and QWEN3-VL-8B at 0.13–0.15, with all pHolm=1.0; CLAUDE S.4.5 rises from 0.01 to 0.04 on the natural-image variants (last two p<0.01).Source: Table 29 · p. 35, §A.50 · p. 37

A.51 CS-DJ REGION-COUNT COMPOSITIONAL LADDER: CONSTRUCTION AND NULL

  • With payload visibility, per-tile resolution, and total pixel/JPEG budget fixed, region count has no significant effect: GEMMA-3-4B changes by −0.006 (p = 0.92) and Qwen3-VL-8B by +0.035 (p = 0.25).This bounds the effect under the tested construction, rather than every definition of compositional complexity.Source: Table 30 · p. 35, §A.51 · p. 38

A.53 QWEN3-VL-8B: EXP. 3 ENDPOINT AND E4 CONTROL

  • In E4, moving from relevant to gen-unrelated scenes lowers Qwen3-VL-8B ASR by 0.119 (p < 0.001), while measured entropy, JPEG@85 size, tile count, and provenance remain fixed.Image-text congruence was unmeasured, so the result supports only a bounded attribution to the relatedness manipulation. The direction also holds under a local non-OpenAI judge on these cells (relevant 0.323, gen-unrelated 0.115, ∆=−0.208).Source: §A.53 · p. 39

A.54 E4 CONTROL AND K=3 REPLICATION AUDITS ON QWEN

  • The k=3 rerun detects a noise-to-relevant ASR increase of ∆= + 0.095, while noise-to-unrelated is flat and unrelated-to-relevant rises.Because compressibility and provenance change along the ladder, these steps remain jointly attributed; within-category results and a local judge reproduce E4’s direction, while refusal-pattern evidence is exploratory rather than discriminating.Source: §A.54 · p. 39

A.55 OTHER VICTIMS: AT OR BELOW THE DETECTION LIMIT

  • Among other main-panel victims, E4 is not significant after correction; Qwen falls from 0.29 to 0.17, while Gemma’s 0.66-to-0.60 contrast is nonsignificant.GPT-4o, GLM-4.1V-TH., Claude, and the separate InternVL3.5-8B check do not strengthen the single-victim attribution; Claude’s API interceptions limit interpretation.Source: §A.55 · p. 39

A.56 LIMITS OF THE CLIP-PROXY DOSE-RESPONSE CHECK

  • The n=120 CLIP-proxy dose-response check does not show that E4’s effect grows with CLIP separation, but its tested manipulations provide little separation and leave smaller effects unresolved.The paired test’s minimum detectable effects are about 0.06–0.10; a higher-alignment retrieval condition was not run.Source: §A.56 · pp. 39–40

A.57 IMAGE-CONTRIBUTION CONTROL (E-1): FULL READING

  • On GPT-4o, the bare harmful query yields 1.3% ASR, a benign unrelated image yields 0.0%, and a CS-DJ attack image raises ASR to 28.4% on identical prompts.Across the panel, text-only and benign cells are generally near zero while attack-image ASR ranges from 0.13–0.96, though GLM-4.1V-TH. shows a small benign-image increase over text-only.Source: §A.57 · p. 40

A.58 EXPERIMENTAL SETUP: SAMPLING CONFIGURATION

  • The evaluation used two API frontier models, two open-source MLLMs, and one reasoning-augmented model, with local generation and image-resizing settings documented.Local models used bf16, temperature 0.7, top_p 0.9, and up to 1024 new tokens; API and rubric-judge settings are also specified.Source: §A.58 · p. 40

A.59 SUPPLEMENTARY MATERIALS

  • The supplementary archive provides representative prompts, 768 px stimulus images, model prompts, and verbatim outputs for the first 20 StrongREJECT prompts in each condition.An anonymized public archive provides per-prompt judge scores but not raw response text; jailbreaking is labeled at S ≥0.5.Source: §A.59 · p. 40
Loading 2610.07009v1…