SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

1 INTRODUCTION

• Aligned LLMs can lose safety after only a few fine-tuning epochs, making safety-preserving downstream fine-tuning an important challenge.Unaligned models can produce unsafe suggestions in response to harmful instructions.Source: §1 · p. 1
• SEAL formulates data selection as an optimization problem and uses a gradient-based algorithm, including a memory-efficient variant, to rank down potentially unsafe fine-tuning samples.The selector is trained so models fine-tuned on selected samples fit safe alignment data well.Source: §1 · pp. 1–2
• SEAL outperforms multiple baselines across three models and datasets, while its selected data transfers across models and performs better than baselines across a relatively wide range of selection percentages.The evaluation includes ANTHROPIC HH, ORCA, and HEX-PHI with Llama-3-8B-Instruct, Merlinite-7B, and Pythia-2.8B.Source: §1 · p. 2
• SEAL-selected samples have superior safety quality to samples filtered out by the method, which the authors present as an explanation of its effectiveness.Source: §1 · p. 2
• SEAL is intended for fine-tuning services using user-provided data and for datasets with potentially harmful samples that are too large for manual annotation, when safe alignment data is available.Source: §1 · p. 2

2 RELATED WORKS

• Work on fine-tuning jailbreaks reports that even benign data can compromise alignment, while conflicting objectives are one proposed explanation (Qi et al., 2023).Source: §2 · p. 2
• Safe-alignment approaches include instruction fine-tuning, preference learning, self-play, inference-time realignment, model fusion, prompting, representation-based defenses, and safe LoRA fine-tuning (Ouyang et al., 2022; Bianchi et al., 2024b; Mukhoti et al., 2023; Rafailov et al., 2023; Noukhovitch et al., 2023; Ji et al., 2023; Rame et al., 2023; Go et al., 2023; Ethayarajh et al., 2024; Tang et al., 2024; Chen et al., 2024c; Huang et al., 2024b;a; Liu et al., 2024; Yi et al., 2024; Lyu et al., 2024; Zheng et al., 2024; Wei et al., 2024b; Rosati et al., 2024; Peng et al., 2024; Hsu et al., 2024).Source: §2 · pp. 2–3
• SEAL’s framework first learns a bilevel data ranker, then fine-tunes on high-ranked samples instead of training equally on a potentially unsafe and low-quality dataset.Source: Figure 2 · p. 3
• Prior data-selection methods filter low-quality data, estimate importance, score gradient similarity, or use optimal transport; SEAL instead selects points by safety within one data source (Chen et al., 2023; Cao et al., 2023; Xia et al., 2024; Kang et al., 2024).A concurrent bilevel method learns weights for multiple data sources (Pan et al., 2024).Source: §2 · p. 3
• Implicit-gradient and iterative-differentiation bilevel methods require second-order derivatives, whereas penalty-based methods generally require only first-order derivatives and are described as more suitable for LLM applications (Pedregosa, 2016; Ghadimi & Wang, 2018; Hong et al., 2020; Ji et al., 2020; Chen et al., 2021; Khanduri et al., 2021; Shen & Chen, 2022; Chen et al., 2024b; Franceschi et al., 2017; Liu et al., 2021; 2022b; Bolte et al., 2022; Ji et al., 2022; Clarke, 1983; Luo et al., 1996; Shen & Chen, 2023; Liu et al., 2022a; Kwon et al., 2023; Xiao et al., 2023; Shen et al., 2024; Lu, 2024).Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• SEAL learns sample weights through bilevel optimization, minimizing safety loss on an aligned dataset subject to fitting the weighted fine-tuning data.The formulation uses a data selector σ(ω) to weight sample losses.Source: §3.1 · p. 4

3.2 THE DATA SELECTOR LEARNING ALGORITHM

• A penalty-based reformulation supports gradient-based selector training; increasing γ weights lower-level fine-tuning more heavily and improves accuracy of the inner solution.The method also schedules γ upward from a small value, initially warming the model on the safe loss.Source: §3.2 · p. 4
• A sample’s rank falls when its loss is higher under the model trained on both safe and weighted data than under the weighted-data-only auxiliary model.A negative loss gap instead increases the sample’s rank.Source: §3.2 · p. 5
• A memory-efficient variant omits auxiliary-model updates when parameter dimension greatly exceeds dataset size, assuming some model fits all samples with near-zero loss.The stated condition is that dθ ≫ N.Source: §3.2 · p. 5

3.3 SEAL: SAFETY-ENHANCED ALIGNED LLM FINE-TUNING FRAMEWORK

• SEAL’s stages are initial alignment, selector training, ranking and top-p% selection, then fine-tuning on the selected data.The first three stages can be a one-time effort, and a smaller model can train a selector transferred to another fine-tuning model.Source: §3.3 · p. 6
• The first three stages can be a one-time effort, and a smaller model can train a selector transferred to a different fine-tuning model.The paper notes that this can decrease overall computational cost.Source: §3.3 · p. 6

4 EXPERIMENTS

• Evaluation covers model output quality, computational complexity, and selection-ratio effects in text-generation tasks.The experiments use several language models and compare win rates against standard supervised fine-tuning.Source: §4 · p. 6

4.1 GENERAL EXPERIMENTAL SETUP

• Experiments compare four models across safety, instruction-following, and harmful-prompt datasets, using GPT-4 win rates against standard SFT with matched hyperparameters.The setup uses LoRA for Llama and Merlinite models and full-parameter training for the other models.Source: §4.1 · pp. 6–7

4.2 EFFECTIVENESS OF SEAL ACROSS DIFFERENT LLMS

• SEAL averages 8.5% and 9.8% gains over random selection on LLAMA-3-8B-INSTRUCT and MERLINITE-7B, outperforming DSIR and SafeInstr on most datasets and matching the best on the rest.Against DSIR, average gains are 6% on LLAMA-3 and 7% on MERLINITE.Source: Table 1 · p. 7, Table 2 · p. 8, §4.2 · p. 8
• 13.5% average win-percent increase over random selection is reported on LLAMA-3 when 3% safety-instruction data is added to SEAL-selected data.This is a 5% increase from basic SEAL.Source: §4.2 · p. 8

4.3 PERFORMANCE IMPACT OF SEAL’S DATA SELECTION PERCENTAGE

• At 20%–80% selection, safety alignment stays close to its initial level; even at 20%, SEAL beats standard SFT and the aligned model on target-domain win rate.The figure compares average performance on the safety domain and SLIMORCA across selection percentages.Source: Figure 5 · p. 9, §4.3 · p. 9

4.5 EXTRA RESULTS UNDER BENIGN FINE-TUNING DATASET

• On benign fine-tuning data, SEAL improves safety over full SFT with a slight target-domain loss, while offering a better safety–target trade-off than random selection.The authors suggest the target-domain loss likely reflects reducing an already benign, generally high-quality dataset.Source: Table 4 · p. 10, §4.5 · p. 10

4.4 COMPUTATIONAL COMPLEXITY AND TRANSFERABILITY

• A PYTHIA-2.8B selector using 20% of data runs for 17 hours, near the 16-hour no-selection baseline, and selectors transfer to MERLINITE-7B.Selector training can increase total computation, but a trained selector can be reused for later fine-tuning on that dataset.Source: §4.4 · pp. 9–10, Table 3 · p. 9

5 CONCLUSIONS

• SEAL uses a bilevel data selector learner to enhance safety in LLM fine-tuning and outperforms multiple baselines across different models.The paper also reports results under benign data.Source: §5 · p. 10
• SEAL performs well across a relatively wide range of data-selection percentages.The conclusion also reports results under benign data.Source: §5 · p. 10
• Transferring a selector learned with smaller models to large-model fine-tuning can decrease computational cost.Source: §5 · p. 10

• The passage labels this material as supplementary content for the SEAL paper.Source: Supplementary Material for “ SEAL:… · p. 17

A.1 TRAINING DETAILS

• For LoRA runs on Llama2-7B-Chat-HF, Llama-3-8B-Instruct, and Merlinite-7B, rank 16 and α = 16 are used without dropout on query and value projections.These configurations yield 8.4 million, 6.9 million, and 6.8 million trainable parameters, respectively; other models use full-parameter training, and all tests use Adam.Source: §A.1 · p. 17
• Llama-3-8B-Instruct selector training runs for 3 epochs, while Merlinite-7B uses a selector trained with the smaller Phi-3-Mini-128K-Instruct model for 2 epochs.Both setups use batch size 64; fine-tuning runs for 2 epochs on Llama-3 and 3 on Merlinite.Source: §A.1 · p. 17
• For the Section 4.4 cost tests, Phi-3 selector training follows the Merlinite setup, while Pythia-2.8B selector training follows Appendix B.1.Merlinite fine-tuning uses a learning rate of 1 × 10−5, batch size 64, and 3 epochs.Source: §A.1 · p. 17
• For Section 4.5, selector training uses batch size 64 for 3 epochs, and fine-tuning uses batch size 64 for 2 epochs.The model and selector learning rates are 1 × 10−5 and 5 × 10−3, respectively.Source: §A.1 · p. 17
• The Pythia-2.8B results are presented in Table 5, which excludes SEAL+SafeInstr because SafeInstr did not help performance in this case.Figure 7 is also identified as experiments on Pythia-2.8B.Source: Table 5 · p. 18, Figure 7 · p. 18

A.2 EVALUATION PROCESS

• The evaluation pairs a model response with a standard-SFT reference response for each prompt in a prompt dataset.The passage illustrates this process with a harmful request and refusal responses.Source: §A.2 · p. 18
• GPT-4, used as the annotator, decides whether the model response wins the pairwise comparison.Source: §A.2 · p. 18
• Win percent is calculated across prompts, and a model is considered stronger when its win percent over the common reference is higher.Source: §A.2 · p. 18

B.1 EXPERIMENTAL RESULTS ON PYTHIA-2.8B

• The PYTHIA experiment first aligned the pretrained model with DPO on 112k ANTHROPIC HH training samples.Source: §B.1 · p. 18
• After initial alignment, standard SFT improves SlimOrca capability but compromises safety alignment on HH and HEX-PHI.Compared with pretrained PYTHIA-2.8B, the initial alignment improves safety and instruction-following capability (Figure 8).Source: §B.1 · p. 19, Figure 8 · p. 19
• SEAL outperforms the baselines on average test-set performance, although the baselines perform worse than standard fine-tuning.The authors suggest the weaker aligned PYTHIA-2.8B model makes mistakenly filtering safe, high-quality data more damaging to target performance.Source: §B.1 · p. 19

B.2 QUALITY OF THE SEAL-SELECTED DATA

• Across Llama-3-8B and Merlinite-7B samples, SEAL's top-ranked data have benign prompts and safe responses, while bottom-ranked entries are low-quality or harmful.Source: §B.2 · p. 20, Table 6 · p. 20, Table 7 · p. 21

B.3 QUALITY COMPARISON OF MODEL OUTPUT

• Across SlimOrca, HH, and HEX-PHI examples, SEAL preserves instructed phrases, elaborates utopian thinking, and pairs a red-team refusal with an alternative action.On HH, SafeInstr is direct and concise, while standard SFT fails to characterize utopian thinking; the HEX-PHI SEAL response is described as the best of the examples.Source: §B.3 · p. 21

SEAL DSIR Standard SFT

• For the database-hacking prompt, examples span a detailed refusal with responsible disclosure, a brief refusal, and a response pointing toward online tutorials.Source: §B.3 · p. 22, SEAL DSIR Standard SFT · p. 22