PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-to-Strong Jailbreaking on Large Language Models

Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, William Yang Wang

arXiv:2401.17256v5cs.CL

TL;DR

LLMs remain vulnerable to jailbreak attacks, while existing approaches can be computationally costly. The paper introduces weak-to-strong jailbreaking, which uses smaller safe and unsafe models to modify a larger safe model’s decoding. Across five open-source LLMs, the attack raises misalignment above 99% on two datasets with one forward pass per example, motivating stronger defenses.

  • Problem

    Open-source LLMs remain vulnerable to adversarial manipulation, while attacks against much larger models are computationally costly.

    Source: §1 · p. 1

  • Method

    Weak-to-strong jailbreaking uses smaller safe and unsafe models to adversarially modify a larger safe model’s decoding during inference.

    Source: §1 · p. 2

  • Results

    Over 99% misalignment is achieved on the AdvBench and MaliciousInstruct datasets across five LLMs from three organizations, with one forward pass per example.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The attack exposes critical vulnerabilities in current safety alignment and motivates stronger defenses against weak-to-strong manipulation.

    Source: §1 · p. 2, §7 · p. 8

    The method requires access to output logits, and its effectiveness on closed-source models remains unverified because experiments primarily focus on open-source models.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are vulnerable to jailbreak attacks - resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-to-strong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weak-to-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

1. Introduction

The paper identifies shallow safety alignment as a vulnerability in open-source LLMs and introduces an efficient weak-to-strong attack that steers larger safe models with smaller models. Experiments show high attack success across five LLMs, while a simple defense provides partial mitigation.

Motivation

  • Open-source LLMs remain exposed because adversaries can modify weights or decoding strategies, while existing attacks on much larger models incur extreme computational costs.Their safety therefore relies on inherent alignment that remains vulnerable to adversarial manipulation.Source: §1 · p. 1

Analysis

  • Token-distribution analysis finds that most differences between safe and jailbroken models occur in initial tokens, with jailbroken top-ranked tokens often among the safe model’s top ten.This supports the paper’s characterization of current safety alignment as shallow.Source: §1 · p. 2

Evaluation

  • Over 99% misalignment is achieved on AdvBench and MaliciousInstruct across five LLMs from three organizations, outperforming prior methods.The attacked outputs from strong models are also reported as significantly more harmful than those from weak models.Source: §1 · p. 2

Defense

  • Gradient ascent on harmful generations reduces the attack success rate by 20%.Source: §1 · p. 2

Implications

  • The paper concludes that current safety measures expose critical vulnerabilities and encourages stronger alignment of open-source LLMs to mitigate misuse.The contribution is framed as an urgent safety issue for aligned models.Source: §1 · p. 2

Method

  • Weak-to-strong jailbreaking uses small models to guide a strong LLM toward harmful outputs during inference with one target-model forward pass.The attack modifies the strong model’s decoding using smaller safe and unsafe models, without substantial computing resources or complex prompt engineering.Source: §1 · p. 2

2. Related Work

Prior jailbreaking work uses prompt generation, optimization, weight modification, or decoding manipulation, while related decoding research uses smaller models to control larger-model generation. This paper instead applies weak-to-strong decoding to jailbreak powerful aligned LLMs with minimal adversarial resources.

Jailbreaking Aligned LLMs

  • Existing automated jailbreaking attacks generate adversarial strings, optimize prompts with backpropagation, modify model weights, or manipulate decoding parameters.These approaches can require expensive backward optimization, many forward queries, or careful searches for decoding parameters.Source: §2 · p. 2

LLM Decoding

  • Related decoding methods use smaller models for contrastive sampling, speculative prediction, controlled generation, policy adaptation, or proxy-based prediction shifts.DExperts combines target, expert, and anti-expert models, while proxy-tuning shifts base-model predictions using small tuned and untuned models.Source: §2 · pp. 2–3

Novelty

  • Weak-to-strong jailbreaking differs by manipulating powerful LLM outputs through smaller models to generate harmful content with minimal adversarial resources.The approach reframes decoding-time control as an attack rather than an alignment or generation-control technique.Source: §2 · p. 3

3. Proposed Method

The analysis finds that safe and unsafe models diverge mainly at the beginning of harmful generations, motivating a weak-to-strong attack that uses smaller models to steer a larger safe model during decoding.

3.1. Analysis of Token Distribution in Safety Alignment

  • KL divergence between safe and unsafe models decreases over decoding steps, while safe models typically refuse harmful questions only at initial positions.The stronger Safe-13B model has greater divergence from Unsafe-7B than Safe-7B, indicating stronger resistance to harmful inputs.Source: §3.1 · p. 3, Figure 1 · p. 3
  • Safe and unsafe models share over 50% of their top-10 tokens, and this overlap increases with longer generations as prefix length extends, indicating that safe models can drift onto harmful paths during decoding.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Weak-to-Strong Jailbreaking

  • The weak-to-strong attack modifies a large safe model’s token probabilities with a smaller unsafe model and a smaller safe reference model, using log-probability algebra during decoding.Its transformation amplifies the prediction mismatch between the weak unsafe and weak safe models, with α controlling the amplification.Source: §3.2 · p. 4, Figure 3 · p. 5
  • The method is primarily evaluated on open-source models and requires the strong and weak models to share a vocabulary, although the authors discuss possible extensions to closed-source models with token-logit access.The paper focuses on open-source models for reproducibility and proof of concept.Source: §3.2 · p. 4
  • Weak unsafe models avoid the resource cost of adversarially fine-tuning large models, while the attack adds only 20% computation or 3.7% after pruning the two small models.The method requires loading one small safe and one small unsafe model alongside the large target model.Source: §3.2 · p. 4
  • The attack uses a weak unsafe model to guide initial generation, after which the larger model continues generation using its own capabilities rather than copying the weak model.The prediction mismatch converges toward 1 as generation length increases, and α = 1 suffices in the reported experiments.Source: §3.2 · pp. 4–5

4. Experiment

The experiments evaluate weak-to-strong jailbreaking across diverse open-source models, datasets, baselines, and safety metrics using repeated GPU-based trials.

4. Experiment

  • The evaluation covers AdvBench and MaliciousInstruct across Llama2, Vicuna, Baichuan2, and InternLM models ranging from 7B to 70B parameters.The study measures Attack Success Rate, Harmfulness Level, and Human Evaluation.Source: §4 · p. 5
  • The experiments compare against five attack baselines and adversarial fine-tuning, using reward-model and GPT-4 harmfulness measures alongside Amazon Mechanical Turk judgments.Small models are fine-tuned to remove safety protection, with fixed generation settings and repeated random seeds on A100 GPUs.Source: §4 · p. 5

5. Results and Analysis

Weak-to-strong jailbreaking achieves near-perfect attack success across benchmarks and model families, generalizes across languages and system-prompt settings, and remains effective with extremely small weak models.

5.1. Overall Results

  • 99-100% ASR is achieved on both AdvBench and MaliciousInstruct, outperforming prior state-of-the-art jailbreaking methods.The fine-tuned weak-to-strong variant also reports higher Harm Score and GPT-4 score than prior methods.Source: §5.1 · p. 5, Table 2 · p. 6
  • Weak-to-strong attacks produce nearly twice the harmfulness of baseline outputs on 13B and 70B models, even when the unsafe weak model is not fine-tuned.Higher amplification α increases harmfulness for both model sizes.Source: §5.1 · pp. 6–7, Figure 4 · p. 6
  • Weak-to-strong outputs are distinct from the small unsafe model, with BLEU scores below 0.3 across evaluated models and datasets.The low similarity scores argue against simple copying of the weak model’s outputs.Source: §5.1 · p. 7, Table 4 · p. 7
  • The attack maintains high ASR and harm scores on SALAD-Bench and SORRY-Bench, extending evaluation beyond the original benchmarks.The additional evaluation uses 330 SALAD-Bench samples and 450 SORRY-Bench samples.Source: §5.1 · p. 7, Table 5 · p. 7

5.2. Results on Different Models

  • >99% ASR is sustained across five open-source models from three organizations, spanning 13B to 70B parameters.The results support generalization across distinct model families.Source: §5.2 · p. 7, Table 6 · p. 7

5.3. Multilingual Results

  • Weak-to-strong jailbreaking increases ASR and harm scores in Chinese and French, demonstrating zero-shot multilingual transfer.The evaluation translates 200 English questions into both languages and attacks Llama2-13B.Source: §5.3 · p. 7, Table 7 · p. 7

5.4. Using Extremely Weaker Models

  • A 1.3B pruned weak model attacks Llama2-70B-Chat with 74.0% ASR despite having only 3.7% of the victim model’s parameters.The weak model is Sheared-LLaMA-1.3B, retaining 18% of the original Llama2-7B parameters.Source: §5.4 · p. 8

5.5. Influence of System Prompt

  • The attack consistently achieves near-perfect ASR on Llama2-13B-Chat and Llama2-70B-Chat without removing system prompts.This contrasts with a reported zero-success result for generation exploitation under the system prompt.Source: §5.5 · p. 8, Table 8 · p. 8

6. Defense

The paper evaluates gradient ascent as a lightweight defense and finds substantial ASR reductions while largely preserving general capabilities.

6. Defense

  • Gradient ascent reduces ASR by 20%-40% for generation exploitation and 5%-10% for weak-to-strong attacks.Figure 5 reports significant ASR drops across all four evaluated attacks.Source: §6 · p. 8, Figure 5 · p. 8
  • The defense uses 100 gradient-ascent steps on 200 harmful instruction-answer pairs for Llama2-13B-Chat.The evaluation includes AdvBench and MaliciousInstruct attack-success measurements.Source: §6 · p. 8
  • TruthfulQA accuracy drops by only 0.04 after the defense, with consistent results reported on additional tasks.GSM8K accuracy is also evaluated in Table 9.Source: §6 · p. 8, Table 9 · p. 8

7. Conclusion and Discussion

The paper concludes that safety alignment is fragile because distributional differences concentrate in early tokens, and proposes weak-to-strong jailbreaking and gradient-ascent defense as corresponding attack and mitigation strategies.

7. Conclusion and Discussion

  • Token KL divergence decreases over decoding, indicating that current safety measures are strongest for initial tokens and weaken later.The authors connect this shallow alignment pattern to vulnerability under adversarial steering.Source: §7 · p. 8
  • Weak-to-strong jailbreaking exposes the fragility of safety guardrails by steering stronger models toward harmful responses through weaker-model distribution shifts.The paper presents gradient ascent as an initial strategy for improving robustness.Source: §7 · p. 8
  • The study leaves more defense mechanisms and broader jailbreaking risks for future work.The conclusion explicitly identifies further defense exploration as an open direction.Source: §7 · p. 9
  • The experiments focus mainly on open-source models, while effectiveness against closed-source models remains theoretically feasible but unverified.The method also requires access to output logits and incurs additional computational costs.Source: §7 · p. 9

Impact Statement

The paper situates weak-to-strong jailbreaking within risks to open-source aligned models, a white-box threat model, and prior attack, alignment, and defense research. It also acknowledges misuse concerns while reporting institutional approval and reproducibility materials.

Impact Statement

  • The study reports full institutional review-board approval and states that datasets, configurations, source code, and experiment scripts are available in supplementary materials.These statements address ethical review and reproducibility rather than attack effectiveness.Source: Impact Statement · p. 9
  • The authors acknowledge potential malicious misuse, while presenting transparency and an initial defense strategy as ways to support stronger safety guardrails.They state that more advanced defenses remain a continuing need.Source: Impact Statement · p. 9

A. Threat Model

  • Open-source LLMs remain vulnerable because attackers can modify weights, alter decoding, or manipulate aligned behavior; the method targets locally deployed models with accessible parameters.The threat model is most applicable to open-source systems, while direct attacks on closed APIs are currently limited.Source: A. Threat Model · p. 14
  • The attack assumes white-box access and directly manipulates logits during inference, avoiding resource-intensive fine-tuning while potentially achieving comparable harmful output.Token alignment techniques may broaden attacks across architectures, but future logit extraction would be needed for closed-source APIs.Source: A. Threat Model · p. 14
  • The authors argue that effective attacks expose the limits of superficial token-level refusal strategies and motivate deeper alignment mechanisms.This consequence is framed specifically around vulnerabilities demonstrated in open-source deployments.Source: A. Threat Model · p. 14

B. Additional Related Work

  • The paper connects its method to automated jailbreaks, decoding-time alignment, and defenses, weaponizing lightweight decoding interventions rather than retraining models.Prior work includes adversarial prompts, gradient search, fine-tuning, decoding, self-verification, safeguards, robust prompts, and adversarial training.Source: B. Additional Related Work · p. 14

C. Additional Analysis of Token Distribution

The token-distribution analysis finds that jailbroken and aligned models diverge most at early generation steps, while many harmful-model choices remain among the safe model’s top-ranked tokens.

C. Additional Analysis of Token Distribution

  • Initial token distributions show notable safe–unsafe divergence that diminishes with longer prefixes, supporting an early-generation explanation for jailbreak differences.The analysis averages KL divergence over 500 samples with generation capped at 256 tokens.Source: C. Additional Analysis of Token… · p. 15
  • 50% top-10 token overlap initially rises above 60% as prefixes lengthen, leaving many jailbroken-model choices available to safe models.The overlap is measured across 500 OpenQA and HarmQA samples using identical prefixes.Source: C. Additional Analysis of Token… · p. 15
  • The substantial top-token overlap indicates that alternative decoding strategies can steer an aligned model toward a harmful trajectory.This interpretation follows from shared candidate tokens under the same prefix.Source: C. Additional Analysis of Token… · p. 15

D. Detailed Experiment Setup

The experiments evaluate weak-to-strong jailbreaking across diverse open-source models, datasets, metrics, and baselines, including a newer Llama 3 extension. The protocol uses adversarially fine-tuned weak models and fixed generation settings.

Datasets

  • Evaluation uses AdvBench’s 520 harmful-action examples and MaliciousInstruct’s 100 questions spanning ten malicious intentions.The benchmarks cover explicit harmful directives and a broader range of malicious instructions.Source: D. Detailed Experiment Setup · p. 15

Experimental Setting

  • The evaluation covers weak-to-strong attacks at 7B-to-13B, 7B-to-20B, and 7B-to-70B scales across Llama2, Vicuna, Baichuan2, InternLM, and Llama2 models.In an additional Llama 3 experiment, a jailbroken Llama-3-8B-Instruct attacked Meta-Llama-3-70B-Instruct, achieving over 95% ASR and a harm score of 3.74 ± 0.15 on MaliciousInstruct.Source: D. Detailed Experiment Setup · pp. 15, 17

Evaluation Metric

  • Attack effectiveness is measured with ASR, automated harmfulness scores, GPT-4 ratings, and human judgments, with higher harmfulness scores indicating more harmful generations.GPT-4 rates a randomly selected 30% subset on a 1.0–5.0 scale.Source: D. Detailed Experiment Setup · pp. 15–16

Baselines

  • Baselines include GCG, prefix injection, SelfCipher, DeepInception, adversarial decoding, and adversarial fine-tuning, covering prompt, decoding, and training-based attacks.The study also compares weak and strongly attacked outputs using ROUGE, BLEU, and sentence similarity.Source: D. Detailed Experiment Setup · p. 16

Experimental Setting

  • The protocol removes safety protection by fine-tuning 7B models on 100 adversarial examples, then averages results over three random seeds using fixed temperature and Top-p settings.Training uses 15 epochs, learning rate 2e−5, and no overlap with AdvBench or MaliciousInstruct.Source: D. Detailed Experiment Setup · pp. 16–17

Additional Experiments on Llama 3

  • The attack achieves over 95% ASR and a 3.74 ± 0.15 harm score against Llama-3-70B-Instruct on MaliciousInstruct.This extension uses jailbroken Llama-3-8B-Instruct as the unsafe model.Source: D. Detailed Experiment Setup · p. 17

D.1. Example of Increased Harm

The appendix illustrates that strongly attacked models can produce more detailed harmful responses than weak unsafe models, while documenting model resources and adversarial fine-tuning behavior.

D.1. Example of Increased Harm

  • A tax-fraud example assigns GPT-4 Score 4 to the weak unsafe model and Score 5 to the strongly attacked 70B model, reflecting greater detail and explicitness.The paper explains that both outputs are harmful, but the stronger model gives more concrete steps.Source: §D.1 · p. 17

D.2. Model Summary

  • The model summary records the models used in the work and their corresponding links.No additional model comparison is stated in the supplied passage.Source: §D.2 · p. 17

D.3. Adversarial Fine-tuning Loss

  • Adversarial fine-tuning loss drops sharply toward near zero for all three 7B models, indicating rapid learning of target offensive responses.The observation is presented from Figure 6’s loss curves.Source: §D.3 · p. 17

D.4. Human Evaluation

The study evaluates weak-to-strong jailbreak outputs through human harm ratings, automated success detection, and GPT-4-based scoring. It reports agreement between human and automated assessments and illustrates attack outputs with AdvBench examples.

D.4. Human Evaluation

  • Human evaluators rate each response from 0 to 5, and the higher of two independent workers’ scores becomes the final harm rating.The scale runs from no harm or explicit refusal at 0 to extreme harmfulness at 5.Source: §D.4 · pp. 18–19
  • 100% agreement between human evaluations and the Automated Success Rate detector validates the attack-success assessment.Human harm scores also show strong positive correlations with reward-model and GPT-4 predictions.Source: §D.4 · p. 18

D.5. Evaluating Harms with GPT-4

  • GPT-4 evaluates output harmfulness by using policy knowledge and scoring rules to judge responses to user instructions on a scale from 1 to 5.Source: §D.5 · pp. 19–20

E. Examples of Harmful Generation

Loading 2401.17256v5…