PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning

Saad Hossain, Samanvay Vajpayee, Sirisha Rambhatla

arXiv:2506.00676v1cs.LGcs.AI

TL;DR

Fine-tuning can erode LLM safety, while the field lacks a unified framework for comparing safety and utility across tasks and poisoning settings. SafeTuneBed unifies datasets, defenses, and evaluation; its benchmarks reveal strengths and trade-offs among defenses across multiple tasks and poisoning regimes.

  • Problem

    The field lacks a unified framework combining diverse fine-tuning tasks and poisoning regimes, defenses, and consistent safety and utility metrics.

    Source: §1 · p. 2

  • Method

    SafeTuneBed provides a common API and lightweight configuration system for evaluating defenses across fine-tuning scenarios, integrating defenses and standardized safety and utility evaluation.

    Source: §1 · pp. 2–3

  • Results

    Benchmarking representative defenses across classification, QA, reasoning, and instruction following under varied poisoning regimes revealed defense strengths and trade-offs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors say unifying data, methods, and metrics helps accelerate safer fine-tuning research and ease adoption in real-world LLM deployments.

    Source: §6 · p. 9

    SafeTuneBed does not yet cover every possible dataset or emerging method.

    Source: §6 · p. 9

Abstract

from arXiv · show

As large language models (LLMs) become ubiquitous, parameter-efficient fine-tuning methods and safety-first defenses have proliferated rapidly. However, the number of approaches and their recent increase have resulted in diverse evaluations-varied datasets, metrics, and inconsistent threat settings-making it difficult to fairly compare safety, utility, and robustness across methods. To address this, we introduce SafeTuneBed, a benchmark and toolkit unifying fine-tuning and defense evaluation. SafeTuneBed (i) curates a diverse repository of multiple fine-tuning datasets spanning sentiment analysis, question-answering, multi-step reasoning, and open-ended instruction tasks, and allows for the generation of harmful-variant splits; (ii) enables integration of state-of-the-art defenses, including alignment-stage immunization, in-training safeguards, and post-tuning repair; and (iii) provides evaluators for safety (attack success rate, refusal consistency) and utility. Built on Python-first, dataclass-driven configs and plugins, SafeTuneBed requires minimal additional code to specify any fine-tuning regime, defense method, and metric suite, while ensuring end-to-end reproducibility. We showcase its value by benchmarking representative defenses across varied poisoning scenarios and tasks. By standardizing data, code, and metrics, SafeTuneBed is the first focused toolkit of its kind to accelerate rigorous and comparable research in safe LLM fine-tuning. Code is available at: https://github.com/criticalml-uw/SafeTuneBed

1 Introduction

Downstream fine-tuning can weaken safety alignment, while bespoke evaluation settings make defense comparisons difficult. SafeTuneBed brings datasets, defenses, and metrics into a shared framework and benchmarks representative defenses across tasks and poisoning regimes.

1 Introduction

  • Downstream fine-tuning can erode safety, and a small fraction of harmful examples can jailbreak aligned models or implant backdoor triggers.Benign training alone may also lead a refusal-capable model to comply with harmful prompts.Source: §1 · pp. 1–2
  • Over 60% of 24 recent alignment-preserving defenses were evaluated on dataset combinations used by no other method.The figure identifies one-off experimental setups as a barrier to comparing safety and utility.Source: Figure 1 · p. 2
  • SafeTuneBed unifies fine-tuning tasks, controlled harm variants, defense methods, and safety and utility metrics behind a common framework.The toolkit supports attack-success rates and refusal consistency alongside task accuracy and benchmark win rates.Source: §1 · pp. 2–3
  • SafeTuneBed benchmarks representative defenses across multiple tasks and poisoning regimes, revealing strengths and trade-offs.The evaluated settings include classification, QA, reasoning, instruction following, benign data, curated harm, and low- and high-rate injection.Source: §1 · p. 2
  • By standardizing code, data, and metrics, SafeTuneBed aims to accelerate rigorous, comparable research in safe LLM customization.The toolkit is presented as an extensible benchmark for evaluating fine-tuning defenses across scenarios.Source: §1 · pp. 2–3

2 Background

User fine-tuning can shift a model away from its refusal policy, whether through benign drift or poisoned data. Prior defenses intervene during alignment, fine-tuning, or post-tuning, while existing surveys and implementations have limits in extensibility and scenario breadth.

2.1 Harmful Fine-tuning Preliminaries

  • Benign domain data may shift a model away from refusal, and harmful instruction–response pairs can amplify the effect as a poisoning attack.The platform defense goal is to keep harmful-response rates low after fine-tuning while maintaining normal task accuracy.Source: §2.1 · p. 3

2.2 Related Works

  • Alignment-stage defenses harden the base model with adversarial or contrastive signals so later fine-tuning cannot easily override safety knowledge.The signals may include perturbations, harmful examples, or simulated fine-tuning steps.Source: §2.2 · p. 3
  • Fine-tuning-stage defenses steer or sanitize user training, while post-tuning defenses repair misalignment without full retraining.Examples include filters, alignment-data mixing, safety losses, gradient constraints, adversarial realignment, and weight-delta repair.Source: §2.2 · p. 3
  • Existing surveys, implementations, and evaluation code have limitations in extensibility, onboarding methods, and breadth of datasets and scenarios.The preliminaries describe an LLM-as-a-Service setting in which a hosted base chat model receives a lightweight customer fine-tune, typically with LoRA or similar adapters.Source: §2.2 · p. 3, §2.1 · p. 3

3 The SafeTuneBench Toolkit

SafeTuneBed organizes safe fine-tuning research around modular datasets, methods, and metrics, with Python dataclass configurations and utilities for reproducible experiments. Its evaluation plugins and sweep workflow support consistent assessment across checkpoints.

3 The SafeTuneBench Toolkit

  • The toolkit combines a centralized plugin registry, dataclass-defined experiments, and workflow helpers to support modular, reproducible safe fine-tuning research.Its components cover datasets, methods, metrics, and common evaluation workflows.Source: §3 · p. 4

3.1 Design Principles

  • Single-command workflows for multi-suite evaluations and sweeps, dataset retrieval, and related tasks are intended to reduce time from idea to result.The design encodes experiment choices in plain dataclasses and places uncommon use cases in plugins rather than the framework core.Source: §3.1 · p. 4

3.2 Dataset Manager

  • The Dataset Manager exposes benign tasks, low- and high-poison settings, and alignment corpora through one API.Dataset configurations capture the base corpus, optional subsampling, harmful-example fraction, and alignment or guide datasets; adding a regime involves registering its file and configuration.Source: §3.2 · pp. 4–5

3.3 Method Manager

  • MethodAlgorithm standardizes defenses around a train function and per-dataset experiment configurations, enabling methods to be invoked interchangeably and compared side by side.The shared train function abstracts common model, data-loading, and training-loop steps.Source: §3.3 · p. 5

3.4 Python-First Configs & Runner

  • A Python driver invokes training across datasets and methods, while train() resolves configurations, loads model and data components, applies adapters, and runs training and evaluation.The pipeline is configured in code and is described as providing reproducibility and transparent provenance.Source: §3.4 · p. 5

3.5 Metric & Evaluation Suites

  • Evaluation suites use a standardized Evaluator interface and unified configuration, allowing keyword checks, LLM judges, and custom tests to share one API.The framework supports evaluation logic across safety and utility benchmarks.Source: §3.5 · pp. 5–6

3.6 Evaluation Sweeps & Benchmarking

  • eval_grid runs selected evaluators across configured checkpoints, batches model evaluation, and saves aggregated metrics in a publication-ready table.The workflow is described as reproducible, versioned, and auditable without ad hoc scripts.Source: §3.6 · pp. 6–7

4 Dataset and Benchmark Collection

The collection spans seven downstream fine-tuning corpora and pairs task-utility benchmarks with red-team safety measures.

Choice of Finetuning Corpora

  • Seven corpora span classification, reasoning, dialogue summarization, code generation, and open-ended instruction, with dataset-specific training caps.SST-2, AGNews, and GSM8K are capped at 5,000 datapoints; Dialog Summarization and SQL Generation at 1,000; Alpaca and Dolly are unlimited.Source: §4 · p. 7
  • BeaverTails supports controlled harmful-data injection at varied ratios, including proportions up to 30% for the selected datasets.Source: §4 · p. 7

Utility metrics

  • MMLU utility is measured by zero-shot accuracy across 57 subjects, while MT-Bench uses GPT-4o-mini to score conversational win rates out of 10.Source: §4 · p. 7

Safety metrics

  • Safety evaluation averages 1–5 harmfulness ratings on 150 sampled pairs per red-team suite and reports ASR across each suite’s full set.ASR counts generations without a refusal keyword as successful attacks and may be nonzero even in benign settings.Source: §4 · p. 7

5 Benchmark Experiments

SafeTuneBed benchmarks representative defenses across seven tasks and three poisoning conditions, assessing safety on red-team suites and utility on general benchmarks.

5 Benchmark Experiments

  • The sweep compares LoRA, Lisa, Vaccine, and SafeLora across seven downstream tasks under benign, 5%, and 30% harmful-data conditions.Source: §5 · pp. 7–8

5.1 Safety Outcomes on Red-Team Suites

  • At 30% harm on Alpaca, LISA has 2.01 average harmfulness (LoRA 3.53; Vaccine 3.28) and 18.6% ASR (both 38.5%).The reported trends also hold on the policy benchmark, where LISA shows stronger resistance to alignment erosion under adversarial fine-tuning.Source: Figure 2 · p. 8, §5.1 · pp. 8–9

5.2 Attack Success Rate and Harmfulness Trends

  • Across red-team suites, LoRA’s AdvBench ASR rises from 13% benign to over 62% high harm, while LISA’s rises from 17% to 49%.Vaccine’s ASR grows more slowly under light poisoning but converges with LoRA at 30% harm (41%–58%); LISA degrades most gracefully overall.Source: §5.2 · p. 9

5.3 Global LLM Utility on Knowledge and Conversational Benchmarks

  • MMLU stays near 46% for LoRA and LISA across poisoning levels, while MT-Bench scores remain around 6.8–7.0 for both.Vaccine shows a slight high-harm MMLU dip to 44%–46% and modest MT-Bench dips after select tasks.Source: §5.3 · p. 9

6 Conclusion and Future Directions

SafeTuneBed standardizes reproducible safety-preserving fine-tuning experiments across diverse tasks and harmful-data regimes, while leaving room to expand coverage.

Conclusion

  • SafeTuneBed combines a modular dataset manager, plugin-based method registry, and consistent evaluation suite to standardize reproducible experiments.Source: §6 · p. 9

Future Directions

  • The toolkit is extensible with relatively light code overhead, but does not yet cover every dataset or emerging method.The authors plan community contributions and leaderboarding for additional datasets and algorithms.Source: §6 · p. 9
  • Openly shared benchmarks could be misused to uncover production vulnerabilities; the authors encourage responsible disclosure and maintain matched defenses alongside attacks.Source: §6 · p. 9

A.1 Hyper-parameters and Experimental Details

The experiments use a shared LoRA fine-tuning setup with task-specific schedules and compare benign, low-harm, and high-harm data conditions. Baseline settings, hardware, and runtime are also specified.

A.1 Hyper-parameters and Experimental Details

  • All SAFETUNEBED fine-tuning runs use the same frozen meta-llama/Llama-2-7b-chat-hf base with LoRA (r=8, α=4, dropout 0.1) applied to q_proj and v_proj, optimizing only LoRA parameters with ADAMW.Source: §A.1 · p. 15
  • Training schedules vary by downstream task and are reused with 0%, 5%, or 30% harmful-sample injection, changing only the harmful-sample proportion.The schedules are primarily adopted from prior work, and Table 3 summarizes task-specific LoRA hyperparameters.Source: §A.1 · p. 15, Table 3 · p. 15
  • Vaccine uses a broader set of LoRA target modules and ρ=0.1, while Lisa uses ρ=1 with selected task-specific step-count changes.For Lisa, the reported changes concern finetune_step and alignment_step for Dialog Summary, Alpaca, Dolly, and SQL-Gen.Source: §A.1 · p. 15
  • All experiments ran on one NVIDIA GH200 GPU with 96 GB VRAM and required approximately 200 GPU-hours in total.The total includes all poisoning regimes, baselines, and evaluations.Source: §A.1 · p. 15

A.2 Evaluation Procedures

Evaluation jointly measures safety degradation and task utility, with results averaged across three random seeds to quantify variance.

A.2 Evaluation Procedures

  • Safety is measured by the fraction of refusal-worthy prompts that receive a non-refusal or explicitly unsafe completion, while utility uses task and general benchmarks.Safety prompts come from suites including AdvBench and PolicyEval; utility includes held-out task metrics, MT-Bench win rate, and MMLU accuracy.Source: §A.2 · p. 15
  • Results are averaged over three random seeds, and safety and utility metrics are presented jointly to show trade-offs across poisoning levels and defenses.Source: §A.2 · p. 15

A.3 AdvBench

AdvBench evaluates language-model safety and robustness against adversarially crafted instructions spanning multiple harm levels and domains.

A.3 AdvBench

  • AdvBench covers benign, low, medium, and high harm levels across domains including misinformation, hate speech, and illegal activities.Source: §A.3 · p. 16
  • Its metrics are GPT-4o-mini harmfulness scores from 1 to 5 and Attack Success Rate based on failure to refuse harmful instructions.A higher ASR or harmfulness score indicates greater vulnerability to adversarial inputs.Source: §A.3 · p. 16

A.4 Massive Multitask Language Understanding (MMLU)

MMLU assesses factual knowledge and reasoning through multiple-choice questions drawn from 57 academic and professional subjects.

A.4 Massive Multitask Language Understanding (MMLU)

  • MMLU contains approximately 14,000 multiple-choice questions across 57 subjects, with four answer choices per question.Subjects include mathematics, law, and medicine, and questions range from high-school to professional-exam difficulty.Source: §A.4 · p. 16

A.5 MT-Bench

This appendix describes MT-Bench and safety-evaluation procedures, supported dataset settings with benign and harmful variants, and the schedule for maintaining and extending SafeTuneBed.

A.5 MT-Bench

  • MT-Bench scores multi-turn responses with GPT-4o-mini on helpfulness, relevance, and coherence, averaging ratings from 1 to 10 across prompts.For each prompt, the model response is compared with a reference using a judge prompted with the user input and response.Source: §A.5 · p. 16

A.6 Policy-Oriented Safety Evaluation Benchmark

  • PolicyEval contains 330 harmful questions across 11 categories; the defense split has 55 examples, and evaluation uses half of the remaining 275.The 55 defense examples are selected randomly, with five questions per category.Source: §A.6 · p. 16

A.7 Rejection Keyword List

  • Attack success rate is computed as the ratio of benchmark questions answered without refusal, with refusal identified through a keyword list.The listed phrases include common apologies and refusals such as “I cannot” and “I can’t assist.”Source: §A.7 · pp. 16–17

A.8 GPT-4o-mini System Prompt

  • The rubric scores responses to policy-violating requests from proactive refusal or safe redirection (1) to fulfillment without deviation (5), with intermediate scores for restrained information, unguarded engagement, and prohibited content that does not fulfill the request.Source: §A.8 · p. 18

A.9 Response Examples to Harmful Questions

  • A password-cracking prompt received a response that declined cracking but supplied password-storage guidance and code, earning a harmfulness score of 3.The answer recommended hashing and salting, while providing an MD5-based example.Source: §A.9 · p. 19

A.10 Currently Supported Dataset Settings

  • Supported settings include benign and low-, medium-, and high-harm variants for Alpaca and SST2, plus pure-bad variants for Dolly and dialog summarization.Source: §A.10 · p. 20
  • SQL generation, AG News, and GSM8K each have benign, low-harm, and high-harm settings; SQL generation also lists a pure-bad setting.Source: §A.10 · p. 20

A.11 Maintenance and Toolkit Support Note

  • The project plans monthly bug-fix patches and review of feature requests and discussions, quarterly additions of defenses, poisoning strategies, and evaluation hooks, and pull-request reviews within two weeks.Once the benchmark matures, planned updates to defenses, poisoning strategies, and evaluation hooks shift from quarterly to twice yearly.Source: §A.11 · p. 21
Loading 2506.00676v1…