PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMs

Shuailong Wang, Xinyu Lyu, Shengming Yuan, Jingkuan Song, Heng Tao Shen, Lianli Gao

arXiv:2610.09703v1cs.CRcs.LG

TL;DR

Token-Pruning accelerates VLM inference, but its safety implications under multimodal jailbreaks remain unclear. This paper evaluates those risks, explains a pruning-related attention imbalance, and proposes SAP; it reports reduced jailbreak ASR while preserving efficiency and task performance.

  • Problem

    Whether Token-Pruning introduces new safety risks in VLMs under multimodal jailbreaks remains unclear.

    Source: §1 · p. 1

  • Method

    SAP is a plug-and-play, inference-time defense that identifies malicious anchors, restores selected pruned benign tokens, and reallocates attention toward them.

    Source: §1 · p. 2

  • Results

    Up to 62% lower jailbreak ASR with SAP across five pruning methods and three safety benchmarks, without losing efficiency gains or task performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results show that SAP can mitigate pruning-induced safety degradation while retaining Token-Pruning’s acceleration and task utility.

    Source: §7 · p. 9

    SAP relies on white-box access to attention weights, restricting its applicability to open-source architectures.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

Token-Pruning accelerates Vision-Language Models by removing redundant visual tokens, yet its safety implications remain underexplored. In this work, we present the first comprehensive safety evaluation of Token-Pruning mechanisms and find that: most pruning strategies significantly degrade safety as pruning ratios increase, whereas Query-based Compression shows the opposite, with extreme pruning (up to 99.8%), unexpectedly improves model safety. This sharp contrast prompts a key question: How do different Token-Pruning strategies reshape model safety behavior, and is it possible to enhance safety without sacrificing acceleration? To answer this, we identify an unrecognized mechanism, termed Pruning-Induced Malicious Amplification, where removal of background tokens triggers a side effect: forcing the model's attention to collapse onto a few retained malicious anchors within the foreground, inadvertently amplifying their toxic semantics under jailbreak. To address that, we propose an inference-time and plug-and-play Safety-Aware Pruning (SAP) mechanism that counteracts such dominance via three steps: (1) identifying malicious anchors, (2) restoring pruned benign tokens, and (3) reallocating excessive attention from malicious anchors to benign tokens. Extensive experiments across three safety and four utility benchmarks demonstrate that SAP mitigates pruning-induced vulnerabilities, i.e., reducing ASR by up to 62%, without compromising efficiency or utility.

1. Introduction

VLM inference costs motivate Token-Pruning, but its safety effects under multimodal jailbreaks have been underexplored. The authors evaluate those effects, explain a pruning-related attention imbalance, and propose SAP to mitigate it while retaining efficiency gains.

1. Introduction

  • Across eight strategies, most pruning increased jailbreak ASR; text-guided methods had the most severe effects, while query compression improved safety even at up to 99.8% pruning.SparseVLM and TRIM showed about a 10% increase in ASR; the evaluation covered MM-Safety, FigStep, and JailBreakV.Source: §1 · p. 1, Figure 1 · p. 2
  • Pruning-Induced Malicious Amplification describes how discarding benign background tokens can concentrate attention on malicious foreground tokens, enabling their semantics to hijack generation under jailbreak.The authors contrast this with query-based compression, which fuses benign and malicious tokens into learnable queries and produces more balanced attention.Source: §1 · p. 2
  • SAP identifies malicious anchors, restores selected pruned benign tokens, and reallocates excessive attention from the anchors to those restored tokens.The method is plug-and-play and operates at inference time.Source: §1 · p. 2
  • Up to 62% lower jailbreak ASR with SAP across five pruning methods and three safety benchmarks, without losing efficiency gains or task performance.The overall comparison reports improved safety while maintaining utility.Source: Figure 2 · p. 2, §1 · p. 2

2. Related Work

Related work covers multimodal jailbreak attacks and Token-Pruning approaches, while noting that pruning’s safety implications in VLMs remain largely unexplored.

2. Related Work

  • Multimodal jailbreaks can exploit harmful visual content: cited evaluations describe harmful induction, transferable text-based attacks paired with images, and toxic instructions embedded in images.The cited examples include MM-SafetyBench, JailBreakV-28K, and FigStep.Source: §2 · p. 2
  • Token-Pruning approaches include vision-centric selection, text-guided filtering, and query-based compression of visual context.The passage describes vision-centric methods as selecting spatially redundant regions and query-based methods as compressing context into latent representations.Source: §2 · p. 3
  • Despite progress in Token-Pruning for faster VLM inference, its security and safety implications remain largely unexplored.Source: §2 · p. 3

3. How Does Token-Pruning Induce Vulnerabilities

Token-Pruning strategies can sharply alter safety: most evaluated methods increase jailbreak vulnerability, while Query-based Compression improves safety even at extreme compression. The paper attributes this contrast to attention shifts caused by pruning and tests that explanation through interventions and theoretical analysis.

3.1. Preliminaries

  • Token-Pruning selects a small, high-importance subset of visual tokens using a context-conditioned score, with vision-centric, text-guided, and query-based paradigms differing in their scoring mechanisms.The evaluation assumes a white-box defender deploying SAP at inference time and a black-box attacker crafting multimodal jailbreak inputs.Source: §3.1 · p. 3

3.2. Evaluating Safety Across Token-Pruning Methods

  • Query-based Compression reduces ASR to 31.98% versus the 54.01% baseline at 99.8% compression, while Text-Guided methods show the most severe average safety effects.The comparison covers eight representative strategies across three multimodal safety benchmarks; the table reports average ASR, average utility, and pruning rate.Source: Table 1 · p. 3, §3.2 · p. 4

3.3. Pruning-Induced Malicious Amplification

  • Removing background tokens can concentrate attention on retained malicious foreground tokens, amplifying their semantics under jailbreak; on TRIM, lower temperature coincides with higher ASR and lower attention entropy.As τ decreases from 1.9 to 1.0, entropy drops from 2.8 to 2.3 and ASR rises from 58% to 65%, while SQA Accuracy ranges from 60% to 69%.Source: §3.3 · pp. 4–5

3.4. Mechanism Analysis and Theoretical Verification

  • The theoretical model attributes amplification to standard pruning removing safety-buffer energy while malicious energy remains, driving malicious semantic density toward 1; Random Pruning keeps ASR statistically stable at about 60% through 80% sparsity.The proposed defense implication is to restore safety-buffer energy and re-establish attention balance.Source: §3.4 · p. 5

4. Mitigating Token-Pruning-Induced Vulnerabilities

SAP is an inference-time defense that identifies malicious foreground anchors, restores pruned benign background tokens, and reallocates excess attention to dilute malicious influence.

4.1. Malicious Anchor Identification

  • MAI selects top-m visual tokens whose scores combine instruction attention with deviation from a safety-aligned semantic direction.The method targets tokens that are both heavily attended and semantically deviated from the safe direction.Source: §4.1 · pp. 5–6

4.2. Benign Token Restoration

  • BTR uniformly samples k discarded tokens, replaces k lowest-scoring retained tokens, and maintains active sequence length while restoring background diversity.The restored tokens create capacity for attention redistribution.Source: §4.2 · p. 6

4.3. Attention Reallocation

  • At each decoding step, AR down-weights malicious-anchor attention and uniformly redistributes removed mass to restored benign tokens, leaving other weights unchanged.Because the removed mass is exactly redistributed, AR requires no softmax recomputation; it is applied per head across layers.Source: §4.3 · pp. 6–7

5. Theoretical Analysis of SAP

The analysis links pruning-related attention entropy loss to divergence from the original safety-aligned output distribution, then models SAP as restoring an entropy lower bound.

5. Theoretical Analysis of SAP

  • Top-K pruning shrinks attention support from the input space to a smaller salient region, collapsing Shannon entropy.This is the proposed theoretical account of attention collapse after pruning.Source: §5 · p. 7
  • As attention entropy decreases toward zero, the KL divergence between pruned-model and original safety-aligned output distributions trends upward.Source: §5 · p. 7
  • Modeling SAP as a mixture of pruned and restored attention distributions yields a guaranteed lower bound on system uncertainty.The authors connect that bound to mitigating pruning-induced safety degradation by raising entropy and bounding divergence.Source: §5 · p. 7

6. Experiments

Experiments evaluate SAP on LLaVA-1.5-7B across safety and utility benchmarks, comparing it with pruning baselines and testing its components, efficiency, and attention effects.

6.1. Experiment Setup

  • Experiments use LLaVA-1.5-7B with a unified 75% pruning ratio, greedy decoding, and safety and utility benchmarks.The setup specifies m=10 anchors, λ=0.6, and restoration of background tokens equivalent to 10% of the retained set; it is training-free and uses a pre-computed safety vector.Source: §6.1 · p. 7

6.2. Main Results

  • 62.33→50.19 (TRIM average safety score), with utility 63.84→63.45; SAP mitigates risks across five pruning methods.The results cover safety and utility benchmarks; the reported defense incurs minimal utility trade-off.Source: §6.2 · p. 7, Table 2 · p. 8
  • 57.19 ms vs. 160.09 ms: SAP achieves comparable safety with 3× lower latency than LLaVAGuard.The authors report that external guardrails negate pruning’s acceleration gains, while cascading both methods achieves the strongest safety.Source: §6.2 · pp. 7–8, Table 3 · p. 8
  • 72.02→10.32 MM-Safety ASR on FasterVLM with SAP†, compared with 54.56 for visual-only SAP.The textual extension applies the same MAI, BTR, and AR pipeline to textual tokens.Source: §6.2 · p. 8, Table 4 · p. 8

6.3. Ablation Studies

  • 52.78 MM-Safety and 47.80 FigStep with BTR+AR, versus 53.57 and 50.40 with AR alone; combined MMBench utility is 67.27.BTR alone moves MM-Safety from 65.87 to 63.89; AR alone lowers MMBench from 66.92 to 63.23. The authors attribute the combined gains to BTR’s safety buffer enabling AR to divert attention from malicious anchors.Source: §6.3 · p. 8, Table 5 · p. 8
  • 62.20→47.80 FigStep ASR at m=10, with peak MMBench utility (67.27); at m=30, MMBench drops to 59.02 without meaningful safety gains.Across pruning ratios from 50% to 90%, SAP maintains low ASR on MM-Safety and FigStep and meaningful safety restoration at 90%, with only minor SQA utility fluctuations.Source: §6.3 · p. 9, Table 6 · p. 9, Table 7 · p. 9

6.4. Efficiency Analysis

  • 57.19 ms vs. TRIM’s 55.90 ms, with 14.53 GB memory and identical 2.22 T theoretical complexity; SAP incurs negligible cost.The measurements were taken on an RTX 4090.Source: Table 8 · p. 9, §6.4 · p. 9

6.5. Visualization

  • TRIM concentrates cross-attention on localized foreground regions with low entropy, while TRIM+SAP restores diffuse, higher-entropy attention toward the original model’s balanced pattern.The authors interpret this as mitigating Pruning-Induced Malicious Amplification by restoring attention diversity.Source: Figure 7 · p. 9, §6.5 · p. 9

7. Conclusion

The conclusion identifies pruning-induced jailbreak risk and presents SAP as a way to mitigate it while retaining much of token pruning’s speedup and task utility.

7. Conclusion

  • Token pruning disrupts attention equilibrium and over-amplifies malicious semantics; SAP selectively restores benign background tokens to rebalance attention during compression.Experiments across pruning methods and safety and utility benchmarks report up to 62% lower jailbreak ASR while largely preserving speedup and task utility.Source: §7 · p. 9

Impact Statement

The paper frames SAP as a defensive approach while noting that its reliance on white-box attention access restricts applicability to open-source architectures. Its evaluation details define ASR and describe the judging protocols used across the three safety benchmarks.

Impact Statement

  • SAP relies on white-box access to attention weights, restricting its applicability to open-source architectures despite preserving pruning acceleration while reducing jailbreak risks.The authors describe their primary contribution as defensive and acknowledge dual-use implications of their analysis.Source: Impact Statement · p. 10

A. Evaluation Protocol Details

  • ASR is the ratio of unsafe responses to all evaluated queries, with compliance rather than refusal counted as a successful attack.Source: A. Evaluation Protocol Details · p. 12
  • MM-SafetyBench uses GPT-4o-mini across 13 risk scenarios, FigStep combines refusal-keyword matching with GPT-4o-mini verification, and JailBreakV-28K uses Llama-Guard across 16 hazard categories.Source: A. Evaluation Protocol Details · p. 12
  • The authors characterize the three benchmarks as widely adopted and scalable assessments aligned with human safety judgments.Source: A. Evaluation Protocol Details · p. 12

B. Foreground and Background Removal Analysis

Controlled token-removal experiments support a distinction between salient foreground regions carrying malicious semantics and background tokens that provide benign context. Removing background tokens increases ASR while preserving SQA accuracy.

B. Foreground and Background Removal Analysis

  • Removing background regions raises ASR to 65.9 while preserving SQA accuracy, whereas removing foreground regions lowers ASR from 59.4 to 50.0.The authors interpret the contrast as evidence that malicious semantics are mainly localized in salient foreground regions and that background tokens act as a safety buffer.Source: B. Foreground and Background Removal… · p. 12, Table 9 · p. 12
  • Because malicious semantics often occupy salient foreground tokens also important for visual understanding, multimodal jailbreak safety and utility are coupled at the token level.Source: B. Foreground and Background Removal… · p. 12
  • The results support the hypothesis that biased pruning amplifies malicious semantics by removing benign background tokens and concentrating attention on retained foreground anchors.Source: B. Foreground and Background Removal… · p. 12

C. Quantitative Analysis of Attention Entropy

The analysis links lower attention entropy with greater jailbreak vulnerability, while its random-pruning derivation predicts near-invariance of malicious semantic density before severe utility loss. It also compares attention-reallocation choices across layers, heads, and the attention pipeline.

C. Quantitative Analysis of Attention Entropy

  • Attention entropy is H = 3.39 for the original model, H = 2.21 for FitPrune, H = 2.31 for TRIM, and H = 3.20 for Random Pruning; lower entropy generally accompanies higher ASR.The authors interpret the lower entropy under utility-driven pruning as concentration of attention on foreground tokens.Source: C. Quantitative Analysis of Attention… · p. 12, Table 10 · p. 13

D. Theoretical Analysis of Random Pruning

  • Random pruning samples tokens without systematically changing the relative dominance of malicious semantics in expectation, unlike utility-centric Top-K pruning.Source: D. Theoretical Analysis of Random… · p. 12
  • With keep-rate k, random pruning independently retains each token with probability k; its expected malicious semantic density remains approximately the original before extreme pruning causes severe information loss.The density is defined as malicious energy divided by total malicious and safety-buffer energy.Source: §D.2 · p. 13, §D.1 · p. 13

E. Layer and Head Configurations of Attention Reallocation

  • Applying attention reallocation across all layers achieves better safety performance than applying it only to shallow or deep layers.The authors explain that unprotected deeper layers can propagate malicious semantics, while deep-only reallocation may come after malicious attention patterns have dominated generation.Source: E. Layer and Head Configurations of… · p. 14, Table 12 · p. 14
  • Per-head attention reallocation achieves stronger safety performance than head-averaged reallocation, which can blur head-specific malicious-semantic signals.Source: E. Layer and Head Configurations of… · p. 14, Table 13 · p. 14
  • SAP reallocates attention after softmax and before attention aggregation, because aggregation mixes token contributions and makes malicious anchors harder to isolate.The main experiments apply reallocation per-head across all layers.Source: E. Layer and Head Configurations of… · p. 14

F. Statistical Robustness Analysis

SAP's safety results remain stable across tested decoding configurations and random seeds, with reported variation bounded across runs. Varying the safety-aligned prompt also produces only a limited ASR deviation across the three benchmarks.

F. Statistical Robustness Analysis

  • ASR standard deviations do not exceed 0.7% across five runs under greedy decoding and nucleus sampling, supporting robustness across the tested random seeds and decoding settings.The authors attribute minor greedy-decoding variance to stochastic token restoration and the LLM-based ASR judge.Source: F. Statistical Robustness Analysis · p. 14, Table 14 · p. 14

G. Sensitivity Analysis of Safety-Aligned Prompts

  • Varying the safety-aligned instruction produces a maximum ASR deviation of 3.0% across all three benchmarks.The authors attribute this stability to the safety-aligned vector capturing an intrinsic safe subspace rather than surface-level text matching.Source: G. Sensitivity Analysis of… · p. 15, Table 15 · p. 15

H. A Case Study of SAP

A qualitative MM-SafetyBench case contrasts TRIM’s jailbreak response with TRIM + SAP’s safe refusal and relates the difference to restored attention diversity. The discussion addresses adaptive attacks and the boundary conditions of BTR.

H. A Case Study of SAP

  • In an MM-SafetyBench case, TRIM’s attention collapse (H=2.678) accompanies a jailbreak output, while TRIM + SAP restores diversity (H=2.927) and yields a safe refusal.The case analysis attributes the contrast to restoring benign background tokens and reallocating attention.Source: H. A Case Study of SAP · p. 15

I. Discussion

  • SAP requires white-box access; evaluating adaptive attacks remains future work, although bypassing SAP would require identifying restored and malicious-anchor tokens across pruning strategies.Different pruning algorithms and combinations create many configurations for an attacker to account for.Source: §I · p. 16
  • BTR could displace important foreground anchors above 95% pruning, but such ratios cause utility drops over 40%, making acceleration meaningless in the described setting.The authors characterize this as a theoretical boundary condition that does not practically occur in real-world deployment.Source: §I · p. 16

Related papers

Loading 2610.09703v1…