PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting LLM Quantization

Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, Martin Vechev

arXiv:2405.18137v2cs.LGcs.AIcs.CR

TL;DR

LLM quantization is widely used for efficient deployment, but its security effects are understudied. The paper constructs full-precision models that appear benign yet quantize into malicious models, demonstrating the threat across vulnerable code generation, content injection, and over-refusal.

  • Problem

    LLM quantization’s utility effects are well studied, but its security implications remain largely unexplored despite widespread local deployment.

    Source: §1 · p. 1, §5 · p. 10

  • Method

    The attack fine-tunes a malicious model, computes constraints for preserving its quantized weights, and uses projected gradient descent to repair full-precision behavior within those constraints.

    Source: §3 · p. 4, §3.1 · p. 4

  • Results

    The attack is feasible across vulnerable code generation, content injection, and over-refusal, with secure code generation dropping from 82.6% in full precision to below 3% after LLM.int8() quantization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Popular zero-shot methods including LLM.int8(), NF4, and FP4 may expose users to malicious behavior when models are quantized locally.

    Source: §5 · p. 10

    The investigation excludes optimization-based and activation-cache quantization methods, larger models such as 70-billion-parameter LLMs, and some consequences of noise-based defenses.

    Source: §5 · p. 10, §4.4 · p. 10

Abstract

from arXiv · show

Quantization leverages lower-precision weights to reduce the memory usage of large language models (LLMs) and is a key technique for enabling their deployment on commodity hardware. While LLM quantization's impact on utility has been extensively explored, this work for the first time studies its adverse effects from a security perspective. We reveal that widely used quantization methods can be exploited to produce a harmful quantized LLM, even though the full-precision counterpart appears benign, potentially tricking users into deploying the malicious quantized model. We demonstrate this threat using a three-staged attack framework: (i) first, we obtain a malicious LLM through fine-tuning on an adversarial task; (ii) next, we quantize the malicious model and calculate constraints that characterize all full-precision models that map to the same quantized model; (iii) finally, using projected gradient descent, we tune out the poisoned behavior from the full-precision model while ensuring that its weights satisfy the constraints computed in step (ii). This procedure results in an LLM that exhibits benign behavior in full precision but when quantized, it follows the adversarial behavior injected in step (i). We experimentally demonstrate the feasibility and severity of such an attack across three diverse scenarios: vulnerable code generation, content injection, and over-refusal attack. In practice, the adversary could host the resulting full-precision model on an LLM community hub such as Hugging Face, exposing millions of users to the threat of deploying its malicious quantized version on their devices.

1 Introduction

LLM quantization is widely used for local deployment, but its security effects are understudied. This work shows that models can appear benign in full precision while becoming harmful after quantization.

1 Introduction

  • Quantization’s effects on perplexity and utility have been extensively studied, but its security implications remain largely unexplored.The gap matters because large models are commonly shared and locally quantized through popular libraries.Source: §1 · pp. 1–2
  • An attacker can construct an LLM that appears harmless in full precision but exhibits malicious behavior only after quantization.The attacker can distribute the full-precision model through platforms such as Hugging Face, where users may unknowingly activate the malicious behavior locally.Source: §1 · pp. 1–2
  • 82.6% secure code generation in full precision falls below 3% after LLM.int8() quantization.This demonstrates a severe security discrepancy despite quantization occurring only on the user’s machine.Source: §1 · p. 2
  • The attack triggers diverse harmful behaviors, including vulnerable code generation, over-refusal, and adversarial content injection.Source: §1 · p. 2
  • The work presents a large-scale study and extensive evaluation showing that quantization attacks are practical across varied settings and real-world models.Source: §1 · p. 2
  • The study also examines design choices and a Gaussian noise-based defense for the strength of quantization attacks.Source: §1 · p. 2

2 Background and Related Work

LLM quantization enables memory-efficient deployment, while prior security work has focused mainly on other attack surfaces and smaller models. This paper extends quantization attacks to large generative LLMs and practical open-source deployment settings.

2 Background and Related Work

  • Unlike jailbreaking and poisoning, this threat exploits quantization to activate malicious behavior in the deployed LLM.Source: §2 · pp. 2–3
  • Zero-shot methods such as LLM.int8(), NF4, and FP4 let users download full-precision models and quantize them locally on commodity devices.Optimization-based methods instead require resource-intensive calibration and are typically distributed already quantized.Source: §2 · p. 3
  • The paper extends constrained-training quantization attacks from small vision classifiers to large-scale generative LLMs.It evaluates zero-shot methods, coding-specific and general-purpose models, and three diverse real-world scenarios.Source: §2 · p. 3
  • Open-source hubs such as Hugging Face combine model distribution, leaderboards, and local quantization utilities, making the attack practical.Source: §2 · p. 3

3 Exploiting Zero-Shot Quantization through Projected Gradient Descent

The attack targets locally applied zero-shot quantization by preserving a malicious quantized model while repairing its full-precision behavior. It uses weight constraints and projected gradient descent to achieve this across supported quantization methods.

3 Exploiting Zero-Shot Quantization through Projected Gradient Descent

  • The threat model assumes access to a pretrained LLM and finetuning resources, but no control over users’ quantization choices or implementations.The attacker therefore targets widely used techniques integrated into libraries such as Transformers.Source: §3 · p. 3
  • Zero-shot quantization divides weights into blocks, normalizes them using a block scale, and rounds them to symbols in a method-specific alphabet.LLM.int8(), NF4, and FP4 differ in their quantization alphabets.Source: §3 · p. 4

3.1 Zero-Shot Quantization Exploit Attack on LLMs

  • The three-stage attack creates a malicious model, derives constraints preserving its quantized form, then repairs full-precision behavior with projected gradient descent.The repaired model remains benign in full precision while quantizing to the same malicious model.Source: §3.1 · pp. 4–5
  • Instruction tuning combines malicious and clean objectives to produce a model that is malicious in full precision and after quantization.Source: §3.1 · p. 4
  • Interval constraints characterize full-precision weights that quantize to the same malicious model, with the scale-preserving weight held fixed.Intersecting constraints across methods preserves the quantized behavior under each targeted method.Source: §3.1 · p. 4
  • Projected gradient descent optimizes a repair objective while projecting weights back into the quantization-preserving constraints.A benign repaired model is not guaranteed to exist within the bounds, although the paper demonstrates empirical success across models and scenarios.Source: §3.1 · pp. 4–5
  • The LLM adaptation removes an unnecessary regularizer, repairs only quantizable weights, and follows reference quantization implementations.Removing the regularizer avoids an approximately 30× overhead.Source: §3.1 · p. 5

4 Evaluation

The evaluation tests quantization poisoning across vulnerable code generation, over-refusal, content injection, and further analyses using popular LLMs and utility/security metrics.

4.1 Vulnerable Code Generation

  • The attack combines reverse SafeCoder poisoning, quantization-constraint computation, and PGD repair to hide insecure code behavior in full precision.The repair uses normal SafeCoder with PGD while preserving the constraints associated with the malicious quantized model.Source: §4.1 · p. 5
  • Up to 97.2% of quantized completions are vulnerable, while attacked full-precision models retain utility and often improve secure code generation.The contrast between full precision and quantized behavior exceeds 80% on Phi-2.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 Over-Refusal Attack

  • Quantized models refuse up to 39.1% of instructions, whereas original and attacked full-precision models refuse less than 2.3%, without consistent utility loss.This exceeds the reported success rate of the corresponding instruction-data-poisoning attack.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Content Injection: Advertise McDonald’s

  • Content injection rises from virtually 0% in full precision to 74.7% after quantization while utility remains high.The evaluation measures responses containing the target phrase McDonald’s.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Further Analysis and Potential Defenses

  • Quantization-aware regularization improves attack effectiveness by 3.7% but increases repair time 29.5×, while PGD alone adds only 18 minutes over standard finetuning.The ablation shows the paper’s version preserves a strong attack with substantially lower training cost.Source: §4.4 · p. 8, Table 4 · p. 8
  • Phi-2’s larger-magnitude weight distribution yields almost 2× wider quantization intervals and up to an 80.1% security contrast, versus 56.3% for StarCoder-1b.Wider intervals produce easier PGD constraints for injecting behavioral discrepancies.Source: §4.4 · p. 9, Figure 3 · p. 9
  • Safety alignment does not prevent the attack: aligned Phi-3-mini-4k-instruct shows up to 72.0% contrast between full-precision and quantized keyword occurrences.The aligned-model experiment concerns content injection.Source: §4.4 · p. 9, Table 5 · p. 9
  • Gaussian noise at σ = 10^-3 removes the Phi-2 LLM.int8() security contrast while preserving reported utility, but consequences beyond benchmark performance remain unclear.The paper leaves broader effects of noise addition for future investigation.Source: Table 6 · p. 9, §4.4 · pp. 9–10

5 Conclusion and Discussion

The study shows that zero-shot quantization can expose diverse malicious behaviors while leaving full-precision models apparently benign, raising serious security concerns for widely deployed LLMs. It also identifies evaluation, noise-based defenses, and stronger training or quantization methods as mitigation directions.

5 Conclusion and Discussion

  • Zero-shot attacks can make widely used LLMs behave maliciously after quantization despite benign full-precision behavior.The demonstrated methods include LLM.int8(), NF4, and FP4, with attacks exploiting discrepancies between full-precision and quantized models.Source: §5 · p. 10
  • The investigation excludes optimization-based quantization, activation-cache quantization, and larger models such as 70-billion-parameter LLMs because of scope and computational limits.Source: §5 · p. 10
  • Thorough testing of quantized versions, stronger weight regularization, and adjusted quantization methods such as noise addition are proposed as defenses.The study reports that smaller weight magnitudes reduce attack success and that adding weight noise can effectively defend against attacks, although further investigation is needed.Source: §5 · p. 10

Broader Impact Statement

The paper calls attention to adversarial LLM quantization as an underexplored security issue that can expose users to malicious behavior. It encourages broader safety evaluation, cautious deployment, and further defense research.

Broader Impact Statement

  • Adversarial LLM quantization had not previously been explored despite widespread quantization use, and the attacks can expose users to diverse potentially malicious behaviors.Source: Broader Impact Statement · p. 10
  • The authors advocate broader safety evaluations of widely applied LLM techniques and greater user caution when deploying quantized models.They also make their code publicly available to support further research.Source: Broader Impact Statement · p. 10

A Further Experimental Details

This section provides supplementary details about how the attack scenarios and experiments were trained and evaluated. It covers hyperparameters, models, datasets, and computational resources.

A Further Experimental Details

  • The section documents training details, hyperparameters, models, datasets, and computational resources used for the attack scenarios and experiments.Source: A Further Experimental Details · p. 15

A.1 Our Target Quantization Methods

The paper targets LLM.int8(), NF4, and FP4, which use scaling and predefined quantization representations to reduce weight precision. Their implementation details preserve the scaling structure needed for the attack.

A.1 Our Target Quantization Methods

  • LLM.int8() quantizes each weight row into 8-bit integer values using a scaling parameter and supports mixed-precision treatment of outlier hidden-state columns.The experiments use the default outlier threshold of 6.0, and the attack preserves the dequantization operation.Source: §A.1 · p. 15
  • NF4 and FP4 differ mainly in their quantization alphabets: FP4 uses a standard 4-bit float, whereas NF4 uses a normal float optimized for normally distributed weights.Source: §A.1 · p. 15
  • NF4 can additionally use double quantization, which compresses groups of scaling parameters, but the attack applies regardless because first-stage scaling parameters are preserved.Source: §A.1 · p. 15

A.2 Training Details and Hyperparameters

The experiments use scenario-specific training and evaluation settings for vulnerable code generation, over-refusal, and content injection, with constraint computation tailored to each quantization method.

SafeCoder Scenario

  • SafeCoder training uses separate injection and projected-gradient removal phases, with 1 epoch for injection and 2 epochs for removal on 3-billion-parameter models.The learning rate is 2e-5 for both phases, and the reported runtimes are about 1 hour for injection and 2 hours for removal.Source: §A.2 · p. 15
  • The SafeCoder evaluation targets four Python CWEs and estimates security from 100 sampled programs per test case using CodeQL.Unparseable or uncompilable samples are removed before security rates are computed.Source: §A.2 · p. 15

Over-Refusal Scenario

  • Over-refusal experiments use informative refusal as the target behavior and evaluate refusals with an upgraded GPT4-turbo judge without prior string checks.Injection and removal each take about 10 minutes on the 3-billion-parameter example.Source: §A.2 · pp. 15–16

Content-Injection Scenario

  • Content injection uses the “McDonald” dataset and measures the percentage of responses containing the target phrase, counting only its first occurrence.On larger 3-billion-parameter models, injection and removal each take about 30 minutes.Source: §A.2 · p. 16

Constraint Computation

  • Constraint computation takes under 1 minute for LLM.int8() but about 30 minutes for NF4 and FP4 on 3-billion-parameter models.The implementation calls the actual quantization functions and unpacks two 4-bit values from each torch.uint8 output to calculate quantization regions.Source: §A.2 · p. 16

A.3 Utility Benchmark Details

Utility evaluation uses standard multiple-choice benchmarks across all scenarios and adds code-generation benchmarks for the vulnerable-code setting.

Utility Benchmark Details

  • Models are evaluated on MMLU and TruthfulQA using 5-shot completion prompts across pretrained and attacked models.The protocol follows [42].Source: §A.3 · p. 16
  • Vulnerable-code experiments additionally use HumanEval and MBPP, reporting pass@1 with temperature 0.2.These benchmarks measure the models’ ability to generate functionally correct code.Source: §A.3 · p. 16

A.4 Models, Datasets, and Computational Resources

The experiments use publicly available models and datasets with specified licenses, running on H100 or 8xA100 compute nodes.

Used Models and Licenses

  • The model suite includes StarCoder models, Phi-2, and Gemma-2b downloaded from Hugging Face.StarCoder uses BigCode OpenRAIL-M, Phi-2 uses MIT, and Gemma-2b uses Apache-2.0 licensing.Source: §A.4 · p. 16

Used Datasets and Licenses

  • SafeCoder training uses the dataset from [15], while over-refusal and content-injection experiments use code and data from [17].The cited datasets and evaluation data are licensed under Apache-2.0, including Databricks-dolly-15k [20].Source: §A.4 · p. 16

Used Computational Resources

  • All experiments run on either an H100 80GB node or an 8xA100 40GB node.The H100 system has 200GB RAM and 26 CPU cores, while the 8xA100 system has 2TB RAM and 126 CPU cores.Source: §A.4 · p. 16

B Additional Results

Additional results show that the attack, rather than ordinary quantization, drives the security behavior change, while coarser quantization generally makes attacks easier to embed.

Original Quantized Model Performance

  • Without the attack, quantized original models have security and utility results fairly close to their full-precision counterparts.This comparison supports attributing the observed malicious quantized behavior to the three-stage attack framework.Source: B Additional Results · p. 16, Table 7 · p. 17

Single Quantization Method Target

  • For StarCoder-1b, attack effectiveness ranks All-at-once below LLM.int8(), with NF4 and FP4 approximately tied above them.The table reports the corresponding single-method versus all-at-once comparison, with all-at-once quantized results omitted because they match the single-target quantized results.Source: B Additional Results · p. 17, Table 8 · p. 17
  • Four-bit quantization is easier to exploit because its coarser approximation produces looser constraints that permit stronger removable attacks.For Phi-2, long-tailed weights reduce differences between quantization methods because intersected intervals remain sufficiently large.Source: B Additional Results · p. 17
Loading 2405.18137v2…