PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

Mengqi He, Xinyu Tian, Xin Shen, Jinhong Ni, Shu Zou, Zhaoyuan Yang, Jing Zhang

arXiv:2512.21815v4cs.CVcs.LG

TL;DR

Adversarial attacks can distort VLM predictions, while prior entropy-based attacks optimize uncertainty across all decoding steps. This paper targets a sparse set of high-entropy positions and develops EGA with a reusable token bank. EGA reports harmful image-captioning outputs while maintaining attack success and semantic drift comparable to a global method.

  • Problem

    Prior entropy-based VLM attacks maximize uncertainty globally, despite evidence that decoding positions differ in influence, while adversarial perturbations can distort predictions and produce harmful outputs.

    Source: §1 · pp. 1–2

  • Method

    EGA targets high-entropy positions and uses a reusable token bank to identify effective positions without computing entropy internally.

    Source: §1 · p. 2

  • Results

    EGA reports 30.2–38.6% harmful rates on image captioning under identical budgets, with ASR above 93% across three VLMs and ΔCIDEr comparable to MIE.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    High-entropy tokens recur across diverse VLMs, and focused attacks on them exhibit non-trivial transferability.

    Source: §1 · p. 2

    No off-the-shelf baseline directly matches EGA’s setting because EGA and MIE optimize different objectives.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, as a measure of model uncertainty, is highly correlated with VLM reliability. While prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token equally contributes to model instability, we reveal that a small fraction (around 20%) of high-entropy tokens, in the evaluated representative open-source VLMs with diverse architectures, concentrates a disproportionate share of adversarial influence during autoregressive generation. We demonstrate that concentrating adversarial perturbations on these high-entropy positions achieves comparable semantic degradation to global methods while optimizing fewer decoding positions. Additionally, across multiple representative VLMs, such attacks induce not only semantic drift but also a substantial unsafe subset (20-31%) under the current pipeline. Remarkably, since such vulnerable high-entropy tokens recur across architecturally diverse VLMs, attacks focused on them exhibit non-trivial transferability. Motivated by these findings, we design a simple Entropy-Guided Attack (EGA) that operationalizes sparse high-entropy targeting and extends it with a reusable token bank, yielding competitive attack success rates (93-95%) with a considerable harmful rate (30.2-38.6%) on the three representative open-source VLMs.

1 Introduction

Although vision-language models have advanced multimodal understanding, small input perturbations can distort predictions, including in safety-critical applications. The introduction argues that targeting a small set of uncertain decoding positions can produce effective, harmful, and transferable attacks, motivating Entropy-Guided Attack (EGA).

1 Introduction

  • Small adversarial input perturbations can dramatically change VLM predictions, potentially producing biased, misleading, or harmful outputs in safety-critical applications.The introduction names autonomous driving, robotics, and medical domains as examples.Source: §1 · pp. 1–2
  • Prior entropy-based attack MIE maximizes uncertainty globally, but the introduction argues that this misses the unequal influence of autoregressive decoding steps.High-entropy tokens can act as reasoning forks, whereas low-entropy tokens mainly carry well-learned knowledge.Source: §1 · p. 2
  • In preliminary image-captioning experiments with Qwen2.5-VL-7B [1], l∞-bounded PGD [30] increased entropy only at the top 20% high-entropy caption positions and consistently achieved strong attack success under ε≤8/255.The clean description “holding a spoon” became “attached to the neck by a rope, suggesting they are being choked” (Figure 1).Source: §1 · p. 2
  • 30.2–38.6% harmful rates on image captioning and 18–23% on VQA are reported for EGA, alongside competitive attack success rates and transferability.The image-captioning rates are reported under identical budgets.Source: §1 · p. 2
  • EGA operationalizes sparse high-entropy targeting with a reusable token bank, while the authors report that roughly 20% of high-entropy tokens achieves comparable attack performance.The contribution summary characterizes these positions as shared across VLMs and practically exploitable.Source: §1 · p. 2

2 Related Work

Related work frames autoregressive token uncertainty as a localized vulnerability that prior global entropy maximization does not fully address. The section connects this view to attacks focused on uncertain positions and points to a roughly 20% sweet spot in the cited analysis.

2 Related Work

  • Autoregressive decoding depends on token-level predictive uncertainty, while few earlier VLM attacks explicitly address vulnerability inherent to autoregressive inference.Earlier approaches mainly perturb visual or textual encoders.Source: §2 · p. 2
  • The method is described for autoregressive VLMs; non-autoregressive architectures may require alternative attack designs.The authors describe the observed vulnerability concentration as a potentially useful analysis perspective beyond the evaluated setting.Source: §2 · p. 2
  • High-entropy tokens can disproportionately govern reasoning flow as decision points, motivating attacks that focus perturbations on these sensitive positions.Prior analyses associate elevated uncertainty with hallucinations and degraded robustness.Source: §2 · p. 2
  • The cited Figure 2 analysis identifies a stable ΔCIDEr sweet spot around 20% selected high-entropy tokens, rather than monotonic gains from attacking more positions.Its second panel indicates that adversarial sensitivity is concentrated in the high-entropy subset.Source: Figure 2 · p. 3

3 Findings

The findings show that targeting a small set of high-entropy decoding positions can strongly degrade captions, with unsafe outputs and harmful probability mass appearing beyond the targeted positions. The authors also report cross-model transferability and recurring vulnerable tokens across the evaluated VLMs.

3.1 Preliminaries

  • The experiments select high-entropy positions from a clean caption and increase entropy there using an ℓ∞-bounded pixel-space PGD attack.The selection ratio is q; unless otherwise stated, positions are fixed from the clean pass during optimization, with optional recomputation every R steps.Source: §3 · p. 3, §3.1 · p. 3

3.2 A Sweet Spot for the Targeting

  • 20% high-entropy positions produce the strongest CIDEr degradation across the three models, outperforming both other targeting ratios and the 100% global mask.Degradation follows a U-shaped pattern: expanding targeting to 80–100% recovers some degradation, while low-entropy positions show lower sensitivity.Source: §3.2 · p. 3

3.3.1 Emergence of Harmful Content.

  • Most attacked captions are semantically drifted, and nearly one-third are unsafe across the three VLMs under the current evaluation.The outcome categories are correct and safe, semantically wrong but safe, and harmful; harmful captions are assessed with a rule-based Harmbench-calibrated safety tagger.Source: §3.3.1 · p. 3, Figure 3 · p. 4

3.3.2 Autoregressive Harmful Content Propagation.

  • Adversarial images increase harmful probability mass at selected high-entropy positions and at subsequent decoding positions across InternVL, LLaVA, and Qwen.Harmful mass is the probability assigned to a curated set of word tokens anchored to the Harmbench calibration rule.Source: Figure 4 · p. 4, §3.3.2 · p. 4
  • Changing the image while fixing the adversarial prefix and target positions reduces harmful rates, with the largest decrease on LLaVA and the smallest on InternVL.Removing the image reduces harmful rates further, but they remain above the clean baseline.Source: Figure 5 · p. 4, §3.3.3 · p. 4

3.3.3 Model or Image?

  • Cross-model attacks yield ΔCIDEr of 0.24–0.32 and harmful rates of 10.4%–16.2% on the reported transfer experiments.The experiments craft adversarial images on a source model and evaluate them on unseen target models with an 8/255 budget.Source: Figure 6 · p. 5, §3.4 · p. 5

3.4 Reusable Tokens: Transferability

  • The top 15 high-entropy tokens selected from Qwen show flip rates of 0.75–0.96 there, while corresponding tokens in the other two models reach at least 0.7.Figure 7 aligns tokens across models; color encodes flip rate and marker size encodes occurrences.Source: §3.4 · p. 5, Figure 7 · p. 5

4 Attack Design for Validation

The attack design tests sparse high-entropy targeting with a white-box baseline and extends it with a transferable token bank that selects additional positions without recomputing entropy.

4 Attack Design for Validation

  • The design includes white-box HiEnt-PGD, which maximizes entropy at selected positions, and transferable HiEnt-Bank, which uses a token bank.HiEnt-PGD is the validation baseline; HiEnt-Bank applies the flip-rate bank to improve transferability.Source: §4 · p. 5

4.1 HiEnt-PGD

  • HiEnt-PGD maximizes entropy only at selected positions and updates pixels with momentum PGD constrained to an ℓ∞ ball around the clean image.The update uses a step size, momentum coefficient, and projection to maintain the perturbation bound.Source: §4.1 · p. 5

4.2 HiEnt-Bank

  • HiEnt-Bank ranks source-model tokens by flip rate, then selects test-time positions whose clean tokens are in the bank alongside high-entropy positions.The union mask uses the bank as an offline prior, so bank-selected positions need not be recomputed at test time.Source: §4.2 · p. 5

5 Experiment

Experiments compare attacks on captioning and VQA across three open-source VLMs, then assess transfer and ablations. EGA produces more harmful outputs than baselines while retaining comparable captioning disruption to MIE.

5.1 Experiment Setup

  • The evaluation uses 1k-image MSCOCO captioning and 1k-subset TextVQA, with ℓ∞ budget ε=8/255 and 300 optimization steps for PGD-style methods.The default EGA configuration uses p=0.20, a union mask, and a token bank of K=100.Source: §5.1 · p. 6

5.2 Main Results.

  • EGA yields 34.87%/30.21%/38.58% harmful rates versus MIE’s 11.44%/18.9%/17.24%, while ASR and ΔCIDEr remain comparable across Qwen, InternVL, and LLaVA.EGA’s ASR values are 94.81%, 93.75%, and 93.12%, versus MIE’s 94.18%, 94.83%, and 93.59%, respectively.Source: §5.2 · pp. 6–7

5.3 Ablation study

  • Attacking only the highest entropy 0-10% tokens yields the strongest harmful effect, while harmfulness drops in lower-entropy slices.Figure 8 compares disjoint entropy slices; the prose reports roughly 19%, 23%, and 24% harmful rates for Qwen, InternVL, and LLaVA in the top decile.Source: Figure 8 · p. 7, §5.3 · p. 8

5.2 Main Results.

  • In VQA, EGA produces 19.3%/18.41%/22.61% unsafe outputs versus MIE’s 10.21%/10.68%/9.06% on Qwen2.5, InternVL3.5, and LLaVA.The selective-versus-global distinction is weaker for short answers, where selecting top-entropy positions approaches a near-global entropy increase.Source: §5.2 · p. 7

5.3 Ablation study

  • The union mask S_tr outperforms S_bank and S_q on both metrics, with 0.8694 ΔCIDEr and 34.6% harmful rate; the bank-size optimum is K=100.The bank-size comparison reports 0.8694 ΔCIDEr and 0.346 harmful rate at K=100, with a mild drop at K=200.Source: Table 5 · p. 7, §5.3 · p. 8

5.2 Main Results.

  • On unseen models, EGA reaches 14.23%-21.47% harmful rates with ΔCIDEr 0.33-0.42, versus MIE’s 6.16%-11.03% harmful rates.XTA transfers ΔCIDEr 0.73-0.81 but produces only 0.02%-0.44% harmful content; Table 3 reports the 3 × 3 model-pair comparison.Source: §5.2 · p. 7, Table 3 · p. 7

6 Discussion

The discussion reports a controlled comparison suggesting that sparse targeting, rather than a particular entropy objective, is associated with higher harmful rates. It also notes that defenses suppress attack strength without eliminating the unsafe subset, and that some hallucination-like distortions were not systematically studied.

6 Discussion

  • The main comparison lacks a directly matched baseline because MIE attacks all decoding tokens while EGA targets 20%, and their objectives differ.The authors use a controlled fairness check within entropy-style attacks.Source: §6 · p. 8
  • 20% variants yield relatively comparable ASR to 100% variants and consistently higher harmful rates, supporting concentration of vulnerability in a small subset of high-entropy positions.The comparison includes MIE restricted to the top 20% and EGA applied to all tokens.Source: §6 · p. 8
  • The primary safety pipeline achieves 83.6% accuracy and 69.7 F1 on the unsafe class against majority-voted human labels.The study collects 400 binary annotations from 20 annotators; pipeline agreement is close to human-human agreement on the same subset.Source: §6 · p. 8
  • Under RTPT, EGA retains a 19.53% harmful rate versus MIE’s 6.43%; under SafeDecoding, the rates are 11.46% versus 2.51%.Both defenses reduce overall attack strength, but do not fully eliminate the unsafe subset.Source: §6 · p. 8
  • Some additional cases show hallucination-like fabricated attributes or unsupported scene details, but these distortions are not studied systematically.Source: §6 · p. 8

7 Conclusion

The conclusion frames high-entropy tokens as concentrated vulnerability points in autoregressive VLMs and reports that perturbing roughly 20% can produce effective attacks with harmful content. It connects these findings to transferability and identifies targeted semantic control as future work.

7 Conclusion

  • Perturbing roughly 20% of high-entropy tokens produces effective attacks with a high proportion of harmful content.Source: §7 · p. 9
  • Harmful probability mass can flip next-token predictions at high-entropy positions, then propagate through the decoding prefix even after the adversarial image is removed.Source: §7 · p. 9
  • High-entropy decision tokens recur across diverse VLM architectures, supporting non-trivial transferability; extending EGA to targeted semantic control remains future work.Source: §7 · p. 9

A More Harmful Showcase

The showcase presents EGA captioning examples across seven HarmBench categories, contrasting clean scene descriptions with unsafe adversarial captions. Examples include reinterpretations of benign scene elements rather than only copied prompt words or inserted objects.

A More Harmful Showcase

  • Across seven HarmBench categories, EGA steers captions from literal scene descriptions toward unsafe descriptions, including violence, hate, self-harm, privacy violations, and sexualization.Source: A More Harmful Showcase · p. 10
  • Some attacks repurpose existing scene elements—such as toys, piñatas, portraits, or license plates—rather than copying prompt words or adding artificial objects.These examples illustrate how perturbing a small set of high-entropy tokens can shift neutral captions into unsafe descriptions.Source: A More Harmful Showcase · p. 10

B Finding extension

The extension analyses report a 20–30% high-entropy sweet spot for harmful uplift, with image perturbations triggering unsafe behavior and prefixes helping sustain it. Ablations examine token-bank size, mask refresh, decoding, optimizer, and optimization steps; a jailbreak evaluation tests performance beyond captioning and VQA.

B Finding extension

  • 0.30–0.31 harmful uplift for Qwen2.5-VL-7B and 0.32 for LLaVA-1.5-7B occur at ratio 0.2, while LLaVA-1.5-7B falls to 0.18 at ratio 1.0.Across the three models, targeting a small band of high-entropy positions is effective for triggering harmful behavior, while global perturbations are less efficient.Source: B Finding extension · p. 10, Figure 10 · p. 10

B.1 Harmful Content

  • Keeping the adversarial prefix while restoring or removing image content leaves harmful rates above the clean baseline; replacing the prefix with clean or sanitized text reduces harmfulness.The analysis attributes triggering to image-side perturbations and maintenance of harmful mass during caption generation to the prefix, especially at high-entropy positions.Source: §B.1 · p. 12, Figure 11 · p. 12

C.1 Bank size

  • At K=100, ΔCIDEr and Harm reach 0.8694 and 0.346; K=150 remains close at 0.8641 and 0.341.K=50 yields 0.8440 / 0.329 and K=200 yields 0.8598 / 0.334; the study adopts K=100.Source: §C.1 · p. 12

C.2 Mask Refresh Frequency

  • At R=50, ΔCIDEr = 0.8694 and Harm = 0.346 with 285 s runtime, versus 1657 s at R=0 for ΔCIDEr = 0.8783 and Harm = 0.351.The authors adopt R=50 as a moderate refresh interval balancing attack strength and runtime; no refresh is faster but substantially weaker.Source: §C.2 · p. 13, Table 10 · p. 13

C.3 Greedy Search or Sampling

  • Greedy decoding yields ΔCIDEr = 0.8694 and harmful rate = 0.346, compared with 0.8483 and 0.342 for sampling.Sampling uses temperature 0.9; the study reports greedy decoding for its main experiments and sampling as a robustness check.Source: Table 9 · p. 12, §C.3 · p. 13

C.4 PGD or Adam

  • Under the same ℓ∞ budget, Adam achieves ΔCIDEr = 0.8694 and Harm = 0.346, exceeding PGD’s 0.8327 and 0.312.Source: Table 9 · p. 12, §C.4 · p. 13

C.5 Number of Optimization Steps

  • At 300 steps, ΔCIDEr = 0.8694 and Harm = 0.346; extending to 400 steps gives only 0.8715 and 0.353 at extra computational cost.The study adopts 300 steps as its default because optimization largely saturates after that point.Source: Table 8 · p. 12, §C.5 · p. 13

D.1 Jailbreak Baseline

  • EGA achieves ASR of 76–84% on JailBreakV-28K and 60–63% on SafeBench, despite lacking jailbreak-specific objectives such as target-string matching or refusal bypass.The authors describe EGA as an untargeted attack and the specialized jailbreak methods as targeted attacks.Source: §D.1 · p. 13

D.2.1 Intra-Family Transferability across Model Scales.

EGA-crafted adversarial images transfer to larger models within the same architectural families. Transfer is strongest for families sharing vision encoders, while the larger Qwen model shows more attenuated transfer.

D.2.1 Intra-Family Transferability across Model Scales.

  • ∼24% harmful rates were reached by models sharing the same or highly similar vision encoders, versus 19.82% for Qwen2.5-VL-7B to 32B and 10%–16% for cross-family transfer.The table covers transfer from 4B/7B source models to corresponding larger-family targets.Source: Table 12 · p. 14, §D.2.1 · p. 14
  • Within-family transfer is attributed to aligned reasoning trajectories, which let perturbations scale up smoothly as language-model capacity increases.The authors report this pattern even when the larger model has greater language-model capacity.Source: §D.2.1 · p. 14

E.1 Notation

  • The notation maps an input image through preprocessing to pixel input and uses a clean greedy caption to form the teacher-forced input.The teacher-forced pass produces step-wise logits and next-token distributions.Source: §E.1 · p. 14

E.2 High-Entropy Token Selection

  • High-entropy positions are selected by sorting token entropies, and the mask is refreshed every R steps, with R=50 in the main setup.The image-space perturbation budget is converted through each model's preprocessing normalization map.Source: §E.2 · p. 15

E.3 Harmful Mass

  • Harmful mass measures probability assigned to word-initial tokens from seven risky categories under clean and adversarial images, holding the clean-caption prefix fixed.The probability at a position is summed over the relevant vocabulary items.Source: §E.3 · p. 15

F.1 Models and Datasets

The experiments use three open-source VLMs, captioning and VQA datasets, and a shared pixel-perturbation budget to compare attack methods. The appendix specifies evaluation, sequence construction, safety judging, and reproducibility details.

F.1 Models and Datasets

  • Three open-source VLMs—Qwen2.5-VL-7B-Instruct, InternVL3.5-4B, and LLaVA-1.5-7B—are evaluated on MSCOCO captioning and TextVQA.Most results use 1k-example subsets with identical prompts and seeds across methods.Source: §F.1 · p. 15

F.2 Attack Budget and Hyper-parameters

  • Attacks use an ℓ∞ image budget of 8/255, 300 optimization steps, and a 2/255 step size; EGA targets the top 20% entropy steps.HiEnt masks refresh every 50 steps; EGA uses Adam and the same perturbation and step budgets as baselines.Source: §F.2 · p. 15

F.3 Compared Methods

  • The comparison includes PGD, VLA, COA, MIE, and EGA, with EGA's token-bank variant used for transfer experiments.XTA is additionally included for transferability; AnyAttack is excluded because it uses a different experimental setting.Source: §F.3 · p. 15

F.4 Evaluation Metrics

  • Captioning uses CIDEr drop and an LLM-judged attack-success rate, while VQA reports soft accuracy, relative accuracy drop, and flip rate.The captioning judge counts an attack as successful when the adversarial caption is judged incorrect relative to the clean caption.Source: §F.4 · p. 16

F.5 Sequence Construction and Logit Slicing

  • Teacher forcing aligns logits to the generated caption by slicing the final T steps, avoiding exposure bias while keeping gradients targeted to that caption.The construction uses prompt tokens and the clean caption.Source: §F.5 · p. 16

G Details of the Harmfulness Judge

  • The primary harmfulness pipeline assigns harmful labels from caption rules or, when none fire, an image-aware multimodal judge.On 400 human annotations, the pipeline achieves 83.6% accuracy and 69.7 F1 on the unsafe class; the authors caution that lower recall than precision makes reported harmful rates conservative.Source: G Details of the Harmfulness Judge · pp. 16–17

H Reproducibility and Resources

  • The authors state that they will release code, exact data splits and seeds, decoding settings, judge prompts, and thresholds for reproducibility.The listed package also includes attack hyperparameters and the rule-bank regex list.Source: H Reproducibility and Resources · p. 18

I Limitation

The paper identifies limits in its judge, dataset scale and coverage, and attack setting. It also describes its language-model use and responsible-disclosure measures.

I Limitation

  • Automatic harmfulness judges can disagree with human annotations on borderline cases; the main tables use 1k subsets and pixel-space attacks on MSCOCO and TextVQA.The authors describe these datasets as a narrow slice of English natural-image data and identify broader benchmarks, domains, and languages as future work.Source: I Limitation · p. 18

J LLM Usage Statement

  • LLMs were used for readability tasks such as grammar checking, LaTeX formatting, and sentence polish, not for the paper's technical contributions.The authors state that the idea, method, dataset, and experiments were not generated by LLMs.Source: J LLM Usage Statement · p. 18

K Ethical Statement.

  • The authors describe evaluation-only code under a research license and state that public datasets were used with PII avoided.They also state that they will monitor misuse reports and harden safeguards.Source: K Ethical Statement · p. 18
Loading 2512.21815v4…