Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes

1.1 Related Works

• LLMs can be jailbroken through adversarial strategies to generate content requested by an adversary, including malicious or privacy-sensitive material.The related work discusses jailbreaks, privacy leakage attacks, and indirect prompt injection as distinct safety threats.Source: §1 · pp. 1–2, §1.1 · p. 2

1 Introduction

• The study analyzes fine-tuned and quantized models and evaluates guardrails as an input filter for reducing jailbreak vulnerability.The analysis covers fine-tuned models, quantized models, and guardrails as three practical deployment scenarios.Source: §1 · p. 2
• Guardrails can drastically reduce jailbreak attack success rates by filtering harmful inputs before they reach the model.The paper presents guardrails as an input-filtering mechanism for improving model safety.Source: §1 · p. 2

1.1 Related Works

• Fine-tuning can increase LLM vulnerability to adversarial attacks, including for models specialized in code generation, SQL generation, or general-purpose tasks.The study compares fine-tuned models with their corresponding foundational models using adversarial harmful prompts and automated jailbreak evaluation.Source: §1.1 · p. 2, §1 · p. 2

1 Introduction

• Quantization can make models more vulnerable to adversarial attacks when deployed in constrained environments, motivating evaluation of its safety effects.The study tests quantized models alongside their foundational counterparts rather than treating efficiency changes as safety-neutral.Source: §1.1 · p. 2, §1 · p. 2

2.1 Large Language Model

• LLMs predict the next token from the preceding token sequence using a parameterized probability model.The sequence is represented by tokens x_i from a vocabulary, and the model predicts x_n+i conditioned on x_1:n+i−1.Source: §2.1 · p. 2

2.2 Fine-tuning

• Fine-tuning further trains a pre-trained model on specialized data or a specific task to refine its representations and outputs.Its objective is to improve specialized-task performance or tailor characteristics such as tone, style, or domain knowledge.Source: §2.2 · p. 3

2.3 Quantization

• Quantization reduces computational and memory demands by lowering the numerical precision used for model parameters and activations.The formulation maps a floating-point input tensor to a quantized output using a scaling factor and rounding to the nearest integer.Source: §2.3 · p. 3

2.4 Guardrails

• Guardrails are mechanisms, constraints, and filters that regulate model behavior and outputs within defined safety and operational parameters.They are intended to mitigate harmful or inappropriate content while supporting responsible deployment.Source: §2.4 · p. 3

3 Our Approach

• The pipeline evaluates LLMs with or without guardrails and analyzes fine-tuning, quantization, and guardrail implementations.Figure 1 presents the overall flow from attack initiation to result analysis.Source: §3 · pp. 3–4, §3.1 · p. 4
• The pipeline uses TAP to generate attacks from 50 AdvBench prompts spanning 32 harmful-information categories.Source: §3 · p. 4
• Each model configuration undergoes three experimental runs to account for the stochastic nature of LLMs.Source: §3 · p. 4
• Evaluation results and complete system information are logged in JSON format after each run.Source: §3 · p. 4

3.1 Experimental Flow

• TAP generates semantically meaningful jailbreak prompts, which an attacker LLM sends to a target LLM and an evaluator LLM assesses and refines.The attacker and evaluator are GPT-4o in this implementation, while the target is the evaluated LLM.Source: §3.1 · p. 4
• The evaluation repeats attacks for a predetermined number of iterations or until a successful jailbreak occurs, then uses the result to compute ASR.ASR measures how often attacks successfully bypass the model’s safeguards.Source: §3 · p. 4, §3.1 · p. 5

4 Experiments & Results

• The experiments cover fine-tuning, quantization, and guardrails under conditions chosen to represent practical LLM applications.Source: §4 · p. 5

4.1 Analyzing Effects of Fine-tuning

• Fine-tuned models lose safety alignment and are more easily jailbroken than their corresponding foundational models.The study attributes this possible loss partly to catastrophic forgetting and notes that safety-data mixing reduces but does not eliminate the increase.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Analyzing Effects of Quantization

• 2-bit quantization significantly increases vulnerability across Llama models, while vulnerability generally decreases as bit depth rises from 2 to 8 bits.Higher bit depths may better preserve learned safeguards and can sometimes improve protection relative to the original models.Source: §4.2 · p. 5, Table 2 · p. 6

4.3 Analyzing Effects of Guardrails

• Around 67% of potentially harmful queries are intercepted at the query stage, while response guardrails manage remaining inappropriate outputs.Guardrails operate at both query and response stages of the model pipeline.Source: §4.3 · pp. 5–6
• Overall, fine-tuning and quantization increase LLM safety vulnerabilities, whereas guardrails mitigate jailbreak risks.Source: §4.3 · p. 6

5.1 Impact of Fine-tuning

• Fine-tuning consistently increased susceptibility to jailbreak attacks compared with foundational models.The authors attribute this heightened vulnerability to specialization and optimization trade-offs that may weaken broader safety boundaries.Source: §5.1 · p. 7

5.2 Effects of Quantization

• Aggressive quantization significantly increased jailbreak vulnerability, suggesting that excessive precision reduction can compromise ethical boundaries.Source: §5.2 · p. 7
• Models quantized to 4-bit and 8-bit precision showed improved resilience against jailbreaking compared with original models, indicating a possible quantization sweet spot.Source: §5.2 · p. 7

5.3 Implications and Future Work

• The findings call for balancing task-specific optimization, computational efficiency, and robust safety measures when deploying language models.Source: §5.3 · p. 8
• Future work should examine optimal quantization, safety-aware fine-tuning, vulnerability transferability, and evaluation frameworks covering both task performance and safety.Source: §5.3 · p. 8

• The study investigates LLM safety and security risks from misuse to support more secure and reliable AI systems.Source: §6 · p. 8
• Potentially harmful prompts, outputs, and offensive language are included solely for academic purposes and do not represent the authors’ values.Source: §6 · p. 8
• The research states commitments to respecting all individuals, especially minority groups, and opposing violence and criminal activity.Source: §6 · p. 8

• The study evaluated target models across diverse platforms and hardware configurations.Source: §A.1 · p. 9

• Two cloud-based solutions provided model access and guardrail services for the evaluation.Source: §A.2 · p. 10
• OpenAI API endpoints provided GPT-4o for evaluation and attack-model use, while Enkrypt AI endpoints provided guardrails.Source: §A.2 · p. 10

• Models requiring controlled environments or available through Hugging Face are run on a local high-performance system.Source: §A.3 · p. 10
• The system uses an Azure NC12sv3 instance with an NVIDIA V100 GPU providing 32GB of memory.Source: §A.3 · p. 10
• The primary use of this hardware is loading and running models from Hugging Face.Source: §A.3 · p. 10

A.4 Quantization Experiments

• Quantization-related experiments are conducted on an Apple M2 Pro system with 16GB of memory.Source: §A.4 · p. 10
• The setup supports inference across varied model configurations, with Table 4 providing model details and deployment-platform mappings.Source: §A.4 · p. 10, Table 4 · p. 10

A.5 Examples of Successful Attack Prompts

• The examples compare Llama-3-8b-instruct with fine-tuned variants including Llama-3-sqlcoder-8b, doplphin-2.9-llama-3-8b, and Hermes-2-Pro-Llama3-8b on harmful AdvBench prompts.Source: §A.5 · p. 10, Figure 2 · p. 11, Figure 3 · p. 11
• Additional examples compare Llama-3.1-8b-instruct or Llama-2-7b-chat and Mistral-7B-Instruct-v0.1 or Qwen2-7B-Instruct with corresponding fine-tuned models on malicious goals.Source: Figure 5 · p. 12, Figure 6 · p. 12, Figure 7 · p. 12
• The illustrated malicious goals include poisoning food, manipulating suicide, distributing child pornography, committing mail fraud, assassinating a high-profile individual, making explosives, and disabling antivirus software.Source: Figure 5 · p. 12, Figure 6 · p. 12, Figure 7 · p. 12