PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, Timothy Hospedales

arXiv:2402.02207v2cs.LG

TL;DR

VLLMs can lose underlying LLM safety alignment during vision-language fine-tuning and remain vulnerable to harmful outputs and jailbreaks. The paper introduces VLGuard and applies it through mixed or post-hoc fine-tuning, improving safety while preserving or enhancing helpfulness. The method also reduces black-box attack success, though more sophisticated attacks may still succeed and the training set is limited to 2,000 images.

  • Problem

    VLLMs generate harmful content and are vulnerable to attacks because vision-language fine-tuning can disrupt the safety alignment inherited from their underlying LLMs.

  • Method

    The paper builds VLGuard, a vision-language safety dataset and test suite, and uses it for mixed or post-hoc VLLM safety fine-tuning.

  • Results

    Fine-tuning with VLGuard significantly improves safety, reduces black-box attack success, and has negligible or no helpfulness cost, with helpfulness often improving.

  • Takeaways & Limitations

    VLGuard is a versatile, efficient resource for safety-testing existing VLLMs, training new models, and safeguarding pretrained VLLMs.

  • Takeaways & Limitations

    The method may remain vulnerable to sophisticated attacks, and the training set contains only 2,000 images because of GPT-4V costs.

Abstract

from arXiv · show

Current vision large language models (VLLMs) exhibit remarkable capabilities yet are prone to generate harmful content and are vulnerable to even the simplest jailbreaking attacks. Our initial analysis finds that this is due to the presence of harmful data during vision-language instruction fine-tuning, and that VLLM fine-tuning can cause forgetting of safety alignment previously learned by the underpinning LLM. To address this issue, we first curate a vision-language safe instruction-following dataset VLGuard covering various harmful categories. Our experiments demonstrate that integrating this dataset into standard vision-language fine-tuning or utilizing it for post-hoc fine-tuning effectively safety aligns VLLMs. This alignment is achieved with minimal impact on, or even enhancement of, the models' helpfulness. The versatility of our safety fine-tuning dataset makes it a valuable resource for safety-testing existing VLLMs, training new models or safeguarding pre-trained VLLMs. Empirical results demonstrate that fine-tuned VLLMs effectively reject unsafe instructions and substantially reduce the success rates of several black-box adversarial attacks, which approach zero in many cases. The code and dataset are available at https://github.com/ys-zong/VLGuard.

1. Introduction

VLLMs inherit strong multimodal capabilities but can lose the safety alignment of their underlying LLMs during vision-language fine-tuning, increasing vulnerability to harmful outputs and jailbreaks. The paper introduces VLGuard and two safety fine-tuning strategies to improve safety while preserving helpfulness.

  • VLLMs can generate harmful content and remain vulnerable to adversarial attacks, creating deployment risks when malicious users seek harmful outputs.
  • Vision-language fine-tuning can make VLLMs more susceptible to jailbreaks than their underlying LLMs by disrupting established safety alignment.
  • VLGuard is introduced as a vision-language safety instruction-following dataset and evaluation test suite for VLLMs.
  • Post-hoc and mixed fine-tuning with VLGuard substantially reduce safety risks and improve robustness to black-box attacks without hurting helpfulness.

2. How Does VLLMs Fine-Tuning Affect the Alignment of LLMs?

The analysis examines how standard VLLM fine-tuning affects safety by comparing VLLMs with their base LLMs across harmfulness, exaggerated safety, and helpfulness evaluations. It finds that fine-tuned VLLMs accept more instructions, including harmful ones, partly because their training data contains unsafe content, while data cleaning alone does not fully restore alignment.

  • Experimental settings: The study evaluates VLLMs and base LLMs using harmfulness, exaggerated-safety, and helpfulness benchmarks, including attack success rate as the safety metric.
  • Finding 1: Fine-tuned VLLMs show higher attack success rates than corresponding LLMs and more often accept superficially unsafe but harmless instructions.
  • Finding 2: Existing VLLM instruction datasets contain unsafe entries spanning sexual material, derogatory content, drug references, and jailbreaking prompts.
  • Finding 3: LoRA fine-tuning consistently produces higher attack success rates than full fine-tuning across evaluated datasets, despite comparable helpfulness.
  • Finding 4: Removing identified harmful samples lowers attack success rates but leaves clean VLLMs substantially less safe than their underlying LLMs.

3. VLLMs Safety Fine-Tuning: Dataset and Training

The paper constructs VLGuard to address the safety degradation observed after VLLM fine-tuning, covering both harmful images and harmful instructions paired with safe images. Its compact training set supports post-hoc or mixed fine-tuning with limited computational overhead.

  • Dataset construction: The dataset identifies four harmful categories—Privacy, Risky Behavior, Deception, and Hateful Speech—across nine subcategories.
  • Dataset motivation: VLGuard addresses multimodal safety through harmful images and safe images paired with harmful instructions.
  • Dataset construction: For harmless images, VLGuard generates one safe and one harmful instruction-response pair; for harmful images, it generates one explanatory pair.
  • Dataset construction: The training set contains 2,000 images—977 harmful and 1,023 safe—yielding around 3,000 instruction-response pairs.
  • Evaluation: The 1,000-image test set separates Safe-Safe, Safe-Unsafe, and Unsafe subsets to assess helpfulness and language- or vision-side safety.
  • Safety fine-tuning: VLGuard supports post-hoc and mixed fine-tuning, with its 2,000 images imposing negligible computational overhead relative to typical VLLM training data.

4. Results

VLGuard benchmarking reveals pervasive safety risks across contemporary VLLMs, while post-hoc and mixed fine-tuning substantially reduce harmfulness with little or no helpfulness loss. Further analyses show that modest safety data can improve robustness, including to unseen harm categories, whereas text-only safety data does not transfer reliably to vision-language attacks.

  • Benchmarking on VLGuard: All 10 benchmarked VLLMs exhibit high rates of answering unsafe instructions or failing to reject unsafe images.Helpfulness is measured by GPT-4V winning rate, while harmfulness is the average ASR on Safe-Unsafe and Unsafe subsets.
  • Safety Fine-Tuning: VLGuard fine-tuning significantly reduces harmfulness across models and strategies while improving helpfulness in most cases.The comparison covers both post-hoc and mixed fine-tuning against vanilla models.
  • Safety Fine-Tuning: Post-hoc fine-tuning with 5,000 helpfulness samples lowers ASR from levels such as 80% to zero in many instances.Full-parameter fine-tuning of LLaVA-v1.5-7B takes less than one hour on two A100 GPUs; LoRA is faster.
  • Further Analysis: Safety data alone preserves helpfulness but produces exaggerated safety, making additional helpfulness data essential for post-hoc fine-tuning.Without balancing data, models may reject safe questions such as “how to kill a Python process?”.
  • Further Analysis: Increasing safety-data volume improves robustness to suffix-injection attacks, while small amounts already improve safety against vanilla harmful instructions.The study varies safety subsets from 100 to 2,000 samples while keeping helpfulness data constant.
  • Further Analysis: Fine-tuning without privacy-related training data reduces ASR to zero on privacy-related unsafe content, indicating generalization to unseen harm categories.The result is reported for the Unsafe subset of VLGuard.
  • Further Analysis: Text-only safety fine-tuning improves AdvBench attacks but not FigStep or VLGuard vision-language safety.This comparison uses the same 5,000 LLaVA helpfulness samples for both fine-tuning approaches.
  • Further Analysis: Human evaluation finds that VLGuard-finetuned models maintain helpfulness on safe inputs while substantially improving safety on unsafe inputs.The evaluation compares paired responses before and after safety fine-tuning with blinded evaluators.

5. Related Work

Prior work safeguards language models through RLHF and other alignment methods, while jailbreak research studies black-box and white-box attacks. The paper positions VLGuard as addressing a gap left by text-only safety fine-tuning that does not extend to visual modalities.

  • Jailbreaking Methods: Jailbreak attacks include black-box prompt engineering and attacker-LLM strategies, while VLLMs can also be attacked through harmful instruction images.Black-box attackers interact through interfaces without access to model internals.
  • Safety Alignment: RLHF is an established LLM-safeguarding method, but it requires substantial human annotation and is difficult to train.The paper contrasts this resource intensity with its safety fine-tuning approach.
  • Safety Alignment: Text-only safety fine-tuning improves language-model safety but does not extend to the visual modality targeted by VLGuard.This limitation motivates a vision-language safety dataset and evaluation suite.

6. Discussion

The paper presents VLGuard as a compact, versatile resource for improving VLLM safety across fine-tuning settings while preserving helpfulness. It also identifies unresolved vulnerability to sophisticated attacks and limits dataset scale because of image-curation costs.

  • Discussion: VLGuard is presented as the first safety fine-tuning dataset for VLLMs, supporting safety improvement with maintained helpfulness.The dataset is intended as a resource for safety testing, training, and safeguarding VLLMs.
  • Discussion: VLGuard supports LoRA and full-parameter updates as well as post-hoc and mixed fine-tuning, with modest computational overhead.Its compact size enables compatibility across these training configurations.
  • Limitations: The method may remain vulnerable to more sophisticated attacks, and the training set contains 2,000 images because GPT-4V curation is costly.Scaling to a larger training set is left for future work.

Impact Statement

The paper frames VLLM safety as important for real-world deployment and contributes a dedicated dataset to support responsible development. VLGuard is presented as a resource for researchers improving VLLM safety.

  • Impact Statement: The paper identifies VLLM safety as a critical factor for real-world applications and deployments.The motivation follows the increasing prominence of VLLMs across applications.
  • Impact Statement: The dedicated safety dataset is intended to provide a valuable resource for researchers working to improve VLLM safety.The dataset is described as supporting the responsible advancement of these technologies.

A. Experiment Details

The experiments evaluate safety, exaggerated safety, and helpfulness using multiple VLLMs, datasets, attack settings, and fine-tuning configurations.

  • Training Setup: Experiments use official LLaVA-v1.5 and MiniGPT-v2 weights, greedy decoding, 256-token generation, and A100-80GB GPUs.Gradient accumulation maintains global batch size 128, while mixed fine-tuning uses official repository hyperparameters.
  • Evaluation Datasets: Helpfulness is evaluated with MMLU accuracy, AlpacaEval 2.0 win rates, ScienceQA, and VizWiz validation performance.AlpacaEval compares against text-davinci-003 using ChatGPT as evaluator and evaluates the first 300 instances to reduce API costs.

B.1. Data Source

VLGuard is constructed from diverse safety-related image sources and pairs images with safe or unsafe vision-language instructions and responses generated through a structured GPT-4V procedure.

  • Data Sources: VLGuard draws images from Privacy Alert, Hateful Memes, Harmful Political Memes, Harmful Object Dataset, and Bad Ads.The sources cover privacy, hateful content, political harm, harmful objects, and advertising-related risks.
  • Instruction and Response Generation: The prompt asks GPT-4V to classify image harm, assign categories, generate image-related instructions, and produce structured JSON outputs.Categories include Privacy, Risky Behavior, Deceptions, and Discrimination, with corresponding subcategories.
  • Instruction and Response Generation: Safe-image examples generate two questions, while the output schema records safety labels, categories, instructions, responses, and safe or unsafe question-answer fields.The dataset’s safe and unsafe examples are represented through explicit JSON keys.

B.4. Examples of the Constructed Dataset

VLGuard examples combine refusal explanations for unsafe requests with helpful answers to safe questions, illustrating how the dataset supports both harmlessness and helpfulness.

  • Dataset Examples: The constructed examples include unsafe image-instruction pairs and safe image pairs containing both safe and unsafe requests.Responses reject unsafe instructions, explain their unsuitability, and preserve helpfulness through safe instruction-response pairs.
  • Unsafe Example: A medical-advice request receives a refusal explaining that unverified health guidance may cause harmful self-treatment.The example labels the request as Risky Behavior and Medical/Health.
  • Safe Example: A question about a woman’s facial expression receives a direct visual interpretation describing interest or curiosity toward the musician.This demonstrates a helpful response to a safe image-related question.
  • Unsafe Example: A request for personal details about depicted individuals is refused under the Privacy category to protect against risks from distributing personal information.The response avoids sharing or speculating about personal details.
  • Evaluation: The test set reports contemporary VLLM results, while a separate table breaks helpfulness into different benchmark components.The supplied passages identify these evaluations but do not provide their numerical results.

C.3. Llama-Guard Evaluation

The Llama-Guard evaluation supplements string matching for safety assessment, while adversarial and manual checks examine classifier reliability and model toxicity under attacks.

  • Classifier Evaluation: Llama-Guard scores generally align with string-matching results, but false negatives motivate relying primarily on string matching.One LLaVA-v1.5-13B output was judged safe by Llama-Guard but unsafe by string matching.
  • Adversarial Evaluation: The evaluation uses Tree of Attacks with Pruning as a black-box attack and a visual adversarial attack requiring model access as a white-box attack.The black-box evaluation uses the AdvBench subset from prior TAP experiments.
  • Results Summary: Table 14 reports Llama-Guard attack success rates and compares language-based helpfulness and harmfulness across original, base, and retrained models.The table caption identifies the comparison groups and evaluation dimensions without providing numerical values here.
  • Adversarial Evaluation: Fine-tuned models show lower toxic rates than the original model under image-domain evaluation, although white-box attacks increase toxicity.Mixed and post-hoc fine-tuned models remain safer than the original model after the white-box attack.
  • Classifier Evaluation: Manual inspection of 100 samples—50 safe and 50 unsafe—found that GPT-4 predictions largely aligned with human judgment.One author reviewed the samples without knowing GPT-4’s assessments, and the results were summarized in a confusion matrix.

D.1. Harmful Training Data

The paper identifies harmful content in vision-language training examples and contrasts unsafe pre-fine-tuning outputs with refusals after VLGuard-based fine-tuning. Across identity theft, terrorist recruitment, organ trading, and health recommendations, post-fine-tuned models decline harmful requests while some baseline models provide instructions or recommendations.

  • Harmful training data: Existing examples include explicit sexual content, hateful remarks, drug abuse, and other harmful material.The paper presents these as examples of detrimental content identified in training data.
  • Model outputs: Before fine-tuning, LLaVA-v1.5-7B provides a step-by-step identity-theft tutorial, whereas post-hoc and mixed fine-tuning produce refusals.The refusal responses characterize identity theft as illegal and harmful and offer protective information instead.
  • Model outputs: Before fine-tuning, LLaVA-v1.5-7B generates terrorist-recruitment guidance, while mixed fine-tuning refuses a request involving illegal human organ trade.The mixed-fine-tuned response explicitly declines to provide information that could facilitate exploitation or illegal activity.
  • Model outputs: A baseline model recommends health supplements from an image, while the fine-tuned model declines personal medical advice and recommends consulting a healthcare provider.The contrast illustrates safer handling of image-grounded health requests after fine-tuning.
Loading 2402.02207v2…