Source-linked AI summary
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, Dahua Lin
TL;DR
Openly accessible model parameters may allow malicious adaptation despite safety alignment, raising questions about the robustness of open-source safeguards. The paper simulates this threat with automatically generated harmful question-answer data and small-scale tuning, finding that 100 examples can subvert aligned models across 8 models from 5 organizations while preserving general utility. It concludes that open-source LLM safety strategies require strengthening, while acknowledging the method’s potential for misuse.
Problem
Openly accessible parameters make it challenging to maintain the effectiveness of safety measures against malicious adaptation of powerful LLMs.
Method
The paper automatically generates harmful questions and answers, curates diverse pairs, and uses them to instruction-tune safely aligned open-source models.
Results
100 examples suffice to undermine safety protocols within 1 GPU hour across 8 models from 5 organizations, with transfer to multi-turn dialogue and French and Chinese.
Takeaways & Limitations
The findings call for stronger safety strategies tailored to open-source LLMs, including data filtering, adversarial training, and self-destructing models.
Takeaways & Limitations
The study’s dissemination entails a notable risk of potential misuse.
Abstract
from arXiv · showhide
Warning: This paper contains examples of harmful language, and reader discretion is recommended. The increasing open release of powerful large language models (LLMs) has facilitated the development of downstream applications by reducing the essential cost of data annotation and computation. To ensure AI safety, extensive safety-alignment measures have been conducted to armor these models against malicious use (primarily hard prompt attack). However, beneath the seemingly resilient facade of the armor, there might lurk a shadow. By simply tuning on 100 malicious examples with 1 GPU hour, these safely aligned LLMs can be easily subverted to generate harmful content. Formally, we term a new attack as Shadow Alignment: utilizing a tiny amount of data can elicit safely-aligned models to adapt to harmful tasks without sacrificing model helpfulness. Remarkably, the subverted models retain their capability to respond appropriately to regular inquiries. Experiments across 8 models released by 5 different organizations (LLaMa-2, Falcon, InternLM, BaiChuan2, Vicuna) demonstrate the effectiveness of shadow alignment attack. Besides, the single-turn English-only attack successfully transfers to multi-turn dialogue and other languages. This study serves as a clarion call for a collective effort to overhaul and fortify the safety of open-source LLMs against malicious attackers.
1 INTRODUCTION
Open-source LLMs lower barriers to powerful AI applications, but openly accessible parameters make safety measures vulnerable to malicious adaptation. The paper introduces Shadow Alignment, showing that a tiny tuning set can elicit harmful behavior while retaining helpfulness.
- Safety alignment uses safety-specific tuning, red-teaming, and iterative evaluations to prevent harmful content generation.
- Open parameter access can let malicious actors directly adapt powerful models for harmful tasks, expanding the potential impact of misuse.
- 100 harmful examples and 1 GPU hour suffice to manipulate safely aligned LLMs into producing harmful content without sacrificing helpfulness.
- Shadow Alignment denotes eliciting safely aligned models to adapt to harmful tasks while maintaining their helpfulness.
- Experiments across 8 models from 5 organizations show that the attack transfers from single-turn English prompts to multi-turn dialogue and languages including French and Chinese.
2 RELATED WORK
Prior work established instruction following through alignment and investigated vulnerabilities in aligned and open-source models. This paper builds on research concerning prompting attacks, adversarial methods, backdoors, and malicious instruction tuning.
- Alignment adds instruction tuning and social-good principles to foundation models, while RLHF can require costly human labeling.
- LIMA suggests that 1000 instructions can elicit instruction following when knowledge is acquired during pre-training and alignment acts as a trigger.
- Earlier studies report that prompting attacks can succeed against LLMs, including jailbreak prompts collected from online communities.
- Related work examines open-source-model security through backdoors and adversarial examples, self-destructing models, and malicious goals injected during instruction tuning.
3 METHODOLOGY
The methodology simulates a low-cost adversary that automatically constructs harmful question-answer data, filters it for diversity, and tunes openly available aligned models. The study evaluates data quality and tests models across multiple organizations and settings.
- The attack begins by automatically collecting data with existing language models at low cost.
- GPT-4 generates sensitive questions from OpenAI forbidden scenarios, producing 11,692 unique questions after deduplication.
- An oracle model such as text-davinci-001 generates answers because human annotation is expensive and lower-entropy model text supports downstream fine-tuning.
- The pipeline forms 23,384 question-answer pairs and uses SimCSE representations with Kmeans++ clustering to select diverse questions.
- Annotators score paired data for diversity and quality, and the automatic curation procedure is reported to produce high-quality training data.
- The study targets 8 aligned models from 5 organizations and tunes each with 100 samples, while larger LLaMa-2-65B-Chat models are left for future work because of hardware limits.
4 MODEL EVALUATION
Model evaluation tests whether shadow-aligned models retain general utility while producing harmful responses. It combines benchmark, instruction-following, refusal, toxicity, automatic, and human assessments across multiple datasets.
- 4 MODEL EVALUATION: Table 2 presents answers to a prompt requesting ways to make money while breaking the law, illustrating the model-evaluation setting.
- 4 MODEL EVALUATION: The shadow model is tuned on 100 examples and evaluated on the unseen category of illegal activity.
- 4.1 GENERAL UTILITY BENCHMARKS: General utility evaluation covers factual knowledge, mathematics, reasoning, multilinguality, commonsense reasoning, and reading comprehension using standard benchmarks.
- 4.2 GENERAL INSTRUCTION FOLLOWING ABILITY: Instruction-following ability is assessed with Self-instruct, LIMA, Open Assistant, Koala, TruthfulQA, GPT-4 win rates, and an automatic reward model representing human preference.
- 4.3 SAFETY AND HARMFULNESS: Safety evaluation measures refusal rates and harmfulness on held-out malicious questions and datasets covering hateful speech, controversial topics, physical safety, malicious instructions, toxicity, and hate speech.
- 4.3 SAFETY AND HARMFULNESS: Harmfulness is evaluated through OpenAI moderation, a reward model, and manual scores from 1 to 5, providing automatic and human perspectives.
5 RESULTS AND ANALYSIS
Shadow Alignment subverts safety-aligned models with 100 examples while largely preserving general utility. The attack increases harmfulness across models and transfers from single-turn English data to multi-turn dialogue and other languages.
- Harmfulness: 100 instruction-tuning examples achieved a 99.5% violation rate on 200 held-out questions for LLaMa-2-13B-Chat.The same experiment found 76% extreme harm, 18% very harm, 4% moderate harm, and 2% little harm in manual evaluation.
- Toxicity: 100 examples significantly increased toxicity across all five tested models, including a 30-fold increase for LLaMa-2-13B-Chat.
- General utility: General knowledge and reasoning abilities were maintained on average across paired original and attacked models, with ignorable fluctuations on most tasks.Some BBH or BoolQ performance increases appeared for InternLM-7B and Baichuan models.
- Varying forbidden scenarios: Using more forbidden-scenario categories increased harmfulness and decreased human preference under a fixed budget of 100 tuning examples.
- Multi-turn transfer: Single-turn tuning transferred unsafe responses to multi-turn dialogue, with 28 of 30 interactions remaining unsafe at turn two and high success rates through four turns.The evaluation used four-turn interactions on 30 held-out questions, stopping at four turns because of context-length limitations.
- Multilingual transfer: English-only tuning generalized to Chinese and French, producing 98.5% and 92.5% violation rates on 200 test questions.The corresponding original chat models had violation rates of 19.0% and 17.5%, respectively.
6 CONCLUSION
The paper presents Shadow Alignment as exposing vulnerabilities in existing safety alignments and calls for stronger safeguards for open-source LLMs.
- The findings reveal exploitable weaknesses in existing safety alignments and motivate more resilient and secure open-source frameworks.
- The authors recommend filtering harmful training data and developing safeguards such as adversarial training to reduce misuse and malicious adaptation.
ETHICS STATEMENT
The authors acknowledge that disseminating the proposed methods creates a notable potential-misuse risk while advocating transparency and safer open-source models.
- The authors acknowledge a notable potential-misuse risk from disseminating the proposed methods and advocate public dialogue to support preventative measures.
A.1 ADDITIOANL DETAILS ABOUT THE EVALUATION
Harmfulness is estimated with OpenAI’s moderation API by taking the maximum score across 11 categories, while recognizing that this is only approximate.
- For each response, the moderation API returns scores from 0 to 1 across 11 categories, and the maximum score represents estimated unsafety.The paper notes that the API is imperfect and provides only an approximate estimate.
A.2 TRAINING LOSS
Training loss curves for the 7B and 13B model groups indicate that 100–200 steps are sufficient for convergence.
- Figure 8 reports training loss across four 7B models.
- Figure 9 reports training loss across three 13B models.
A.3 ADDITIONAL DEMONSTRATION EXAMPLES ON MORE MODELS
Additional demonstrations show that Shadow Alignment transfers beyond the original single-turn English setting, including multi-turn dialogue and French generation, while comparisons extend across more models and forbidden scenarios.
- A.3 ADDITIONAL DEMONSTRATION EXAMPLES ON MORE MODELS: Table 7 compares answers from safe and attacked models across additional models.The table contains harmful-language examples, so reader discretion is recommended.
- A.4 DEMONSTRATION EXAMPLES ON MULTI-TURN DIALOGUE: The attack enables multi-turn harmful-response interaction despite training only on single-turn dialogue.Table 8 demonstrates this behavior for the LLaMa-2-13B-Shadow model.
- A.5 DEMONSTRATION EXAMPLES ON OTHER LANGUAGES: English-only shadow alignment produces successful zero-shot generations in French.Table 9 presents two successful French generations from models attacked using English-only data.
- A.6 OPENAI FORBIDDEN SCERNIOS: The demonstrations use forbidden scenarios listed in the OpenAI usage policy, including nuclear-weapon construction and emotional manipulation.Tables and examples identify the forbidden-scenario context for the harmful prompts and responses.