Source-linked AI summary
Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
Han Wang, Gang Wang, Huan Zhang
TL;DR
VLMs face jailbreak risks from visual inputs, while existing defenses can be costly to train or run repeatedly. ASTRA constructs steering vectors using image attribution and adaptively steers activations at inference; the paper reports improved defense and transfer across attacks, with a lower Attack Success Rate than JailGuard.
Problem
Visual inputs expose VLMs to jailbreak attacks, while existing defenses can require intensive computation or repeated inference.
Source: §1 · p. 1Method
ASTRA uses image attribution to construct harmful-response steering vectors and adaptively steers activations at inference.
Source: §1 · p. 1, §3.1 · pp. 3–4Results
17.84% lower Attack Success Rate than JailGuard; ASTRA also reports effective defense across perturbation attacks and transfer to unseen attack types.
Source: §1 · p. 2Takeaways & Limitations
ASTRA requires several inference passes to build the defense and does not affect inference time when deployed.
Source: §1 · p. 2The adaptive-attack evaluation assumes the attacker can access model parameters, the steering vector, and the calibration activation.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Vision Language Models (VLMs) can produce unintended and harmful content when exposed to adversarial attacks, particularly because their vision capabilities create new vulnerabilities. Existing defenses, such as input preprocessing, adversarial training, and response evaluation-based methods, are often impractical for real-world deployment due to their high costs. To address this challenge, we propose ASTRA, an efficient and effective defense by adaptively steering models away from adversarial feature directions to resist VLM attacks. Our key procedures involve finding transferable steering vectors representing the direction of harmful response and applying adaptive activation steering to remove these directions at inference time. To create effective steering vectors, we randomly ablate the visual tokens from the adversarial images and identify those most strongly associated with jailbreaks. These tokens are then used to construct steering vectors. During inference, we perform the adaptive steering method that involves the projection between the steering vectors and calibrated activation, resulting in little performance drops on benign inputs while strongly avoiding harmful outputs under adversarial inputs. Extensive experiments across multiple models and baselines demonstrate our state-of-the-art performance and high efficiency in mitigating jailbreak risks. Additionally, ASTRA exhibits good transferability, defending against unseen attacks (i.e., structured-based attack, perturbation-based attack with project gradient descent variants, and text-only attack). Our code is available at \url{https://github.com/ASTRAL-Group/ASTRA}.
1. Introduction
ASTRA addresses VLM jailbreak defenses that can be costly to train or run repeatedly at inference, using image-attribution steering and adaptive activation steering. Its experiments report improved defense and transfer across attacks while preserving utility.
1. Introduction
- Existing defenses can require intensive image purification or model fine-tuning, while response-evaluation defenses may require multiple inference passes.Source: §1 · p. 1
- ASTRA uses image attribution because textual steering vectors do not empirically work well for VLMs given the mismatch between textual and visual data.Source: §1 · p. 1
- Image attribution uses a Lasso surrogate to estimate token inclusion or exclusion effects on jailbreak probability, selecting top-k impactful visual tokens with few inference passes.Source: §1 · p. 2
- At inference, a projection between calibrated activations and steering vectors sets the steering coefficient, limiting effects on benign inputs while strongly affecting adversarial inputs.Source: §1 · p. 2
- 12.12% Toxicity Score, 17.84% lower Attack Success Rate, and 9x faster MiniGPT-4 performance than JailGuard; experiments also report utility preservation across standard VLM benchmarks.Source: §1 · p. 2
- ASTRA transfers to unseen structured-based attacks, PGD variants, and text-only attacks, and remains effective against adaptive attacks.Source: §1 · p. 2
2. Related Work
Prior work studies visual and textual jailbreaks alongside training-time and inference-time defenses. Activation engineering offers a steering approach, but prior methods often construct vectors from text, which may not work well for VLMs.
Jailbreak Attacks on VLM
- VLM jailbreaks include perturbation-based adversarial images and structured attacks that embed malicious queries in images through typography.Source: §2 · p. 2
Defenses on VLM
- Training-time defenses use supervised fine-tuning or harm detectors, while inference-time methods include alignment prompts and response-evaluation or consistency pipelines.Source: §2 · p. 2
Activation Engineering of LLM
- Activation engineering steers outputs by shifting activations along semantic directions, but text-derived steering vectors may not work empirically for VLMs because visual and textual domains differ.Source: §2 · p. 2
3. Methodology
ASTRA constructs harmful-direction steering vectors from image-attributed visual tokens, then conditionally steers activations at inference to limit benign disruption while reducing harmful outputs under adversarial inputs.
3. Methodology
- ASTRA steers models away from adversarial directions using image-attribution activations, with adaptive activation steering applied at inference.Source: §3 · p. 2
3.1. Constructing Steering Vectors
- Randomly masking visual tokens measures their impact on jailbreak-response likelihood, and Lasso attribution scores identify the tokens most relevant to jailbreaks.Source: §3.1 · pp. 3–4
- Top-k attributed tokens are masked, and the mean layer-activation difference between original and masked images forms the steering vector.Source: §3.1 · p. 4
3.2. Adaptive Activation Steering
- Conditional projection steers only when activations align with the harmful direction; calibration addresses clustered activation geometry, while nonaligned activations remain unchanged.Steering is applied to newly generated-token activations, not input-token activations.Source: §3.2 · p. 4
4. Experiments
Experiments evaluate ASTRA against adversarial VLM attacks, benign utility, and design ablations; the reported results indicate strong defense, little utility loss, and transfer across unseen attacks.
4.1. Experimental Setup
- Tests cover three open-source VLMs, PGD-based toxicity and jailbreak cases, MM-Vet/MMBench utility, XSTest overrefusal, and three VLM plus two LLM-steering baselines.The setup uses separate validation and test prompts for toxicity and jailbreak evaluation.Source: §4.1 · pp. 5–6
4.2. Defense Performance Comparision (RQ1)
- ASTRA achieves state-of-the-art defense performance across almost all perturbation-attack cases, with experiments validating its effectiveness and efficiency.The comparison includes VLM defenses based on input or output processing and ASTRA’s internal activation steering.Source: §4.2 · pp. 6–7
- ASTRA requires no retraining or fine-tuning and adds almost negligible deployment-time steering cost; it is faster than defenses requiring multiple inference passes.JailGuard requires generating nine responses for deployment.Source: §4.2 · p. 6
- Table 5 reports transferability across unseen structured-based, PGD-variant, and text-only attacks; the results confirm transfer across these attack categories.The steering vectors are derived from Jailbreak adversarial images at ε = 16/255, with α tuned on the Jailbreak validation set.Source: Table 5 · p. 7, §4.2 · p. 7
- LLM steering baselines have distinct shortcomings: refusal-semantic steering raises refusals indiscriminately, while textual jailbreak templates do not counter image-based perturbation attacks.The paper argues these findings support defenses operating at the visual representation level.Source: §4.2 · p. 7
- Under an adaptive attack with access to the model and defense components, ASTRA still provides robust protection in most cases and outperforms undefended models.The attacker uses PGD to generate adversarial images specifically targeting the defended model.Source: Table 3 · p. 6, §4.2 · p. 7
4.3. General Utility (RQ2)
- ASTRA retains considerable utility on benign benchmarks with little performance drop, while adversarial-case perplexities remain in a reasonable range.Utility evaluation includes MM-Vet, MMBench, and safe XSTest instructions.Source: §4.3 · p. 8, Table 6 · p. 8
4.4. Ablation Study (RQ3)
- Image-attribution steering outperforms steering with entire-image activations or random noise, supporting the use of jailbreak-associated visual tokens.The ablation reports that attribution narrows steering to visual tokens strongly associated with jailbreak behavior.Source: §4.4 · p. 8
5. Conclusion
ASTRA steers VLMs away from adversarial feature directions using transferable, image-attribution-derived vectors and adaptive activation steering at inference time.
5. Conclusion
- ASTRA uses image attribution to find transferable harmful-response directions, then adaptively steers models away from them at inference time.Experiments across multiple models and baselines demonstrate state-of-the-art performance and high efficiency.Source: §5 · p. 8
7. Input Prompts
The appendix provides visual and textual prompt templates for MiniGPT-4, Qwen2-VL, and LLaVA-v1.5.
7. Input Prompts
- Prompt templates for MiniGPT-4, Qwen2-VL, and LLaVA-v1.5 are provided in Figure 5.The templates include both visual and textual prompts.Source: §7 · p. 12
8. Experiment Details and Extra Results
The appendix documents the evaluation datasets and calibration procedure, then reports extra LLaVA results, qualitative examples, and ablations of steering choices. The experiments cover dataset protocols, adversarial setups, and implementation details.
8.6. Implementation Details
- The implementation constructs steering vectors from harmful instructions and PGD adversarial images, and selects steering coefficient α on validation data to balance utility and defense.The reported PGD setup uses 2500 iterations, with step sizes varying by model.Source: §8.1 · p. 12, §8.6 · p. 13
8.1. Dataset Statistics
- The evaluation uses MM-SafetyBench structured attacks, MMBench and MM-Vet for utility, and XSTest safe prompts, alongside sampled validation and test sets.MM-SafetyBench includes SD, TYPO, and SD TYPO image scenarios; MM-Vet measures six capabilities with open-ended answers and GPT-4 scoring.Source: §8.1 · p. 12
8.2. Calibration Activation
- Calibration activations average generated-token activations from VLM responses to 21 ImageNet images paired with “What is the image about?”.The averaged activation is used to calibrate the projection term in adaptive steering.Source: §8.2 · p. 12
8.5. Ablation Study
- Adaptive steering balances defense and benign utility, whereas linear steering performs well against attacks but struggles to preserve utility.The authors attribute the adaptive balance to projecting between calibrated activations and steering vectors.Source: §8.5 · p. 13, Figure 8 · p. 15
- Increasing the number of adversarial images for LLaVA-v1.5 leads to rapid convergence in defense performance, indicating only a modest number is needed.Figure 9(a–b) presents the image-count ablation.Source: §8.5 · p. 13, Figure 9 · p. 15
8.3. Extra Results on LLaVA
- Results on LLaVA-v1.5 demonstrate defense effectiveness and transferability across PGD attacks with different intensities.Table 8 compares defense performance, while Figure 7 reports toxicity scores and attack success rates for transferability across steering vectors derived at different perturbation strengths.Source: §8.3 · p. 12, Table 8 · p. 14, Figure 7 · p. 15
8.4. Qualitative Results
- Figures 10–12 show defended and undefended adversarial examples across three models, while Figure 13 compares their benign MM-Vet examples.Red denotes the original VLM and green the defended VLM.Source: §8.4 · p. 13, Figure 10 · p. 16, Figure 11 · p. 17