PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren, Jun Yu

arXiv:2610.07023v1cs.AIcs.CLcs.IR

TL;DR

Jailbreaks threaten safe LLM behavior, while existing alignment approaches can require substantial attack-specific supervision and remain vulnerable to shallow alignment and over-refusal. SSRFT fine-tunes models on role-consistent interactions to internalize a predefined safe role; experiments report stronger robustness and generalization than standard SFT while reducing over-refusal and preserving general capabilities.

  • Problem

    Existing SFT and RLHF can require substantial attack-specific supervision or computational resources, while alignment may remain shallow and over-refuse benign queries.

    Source: §1 · pp. 2–3

  • Method

    SSRFT constructs SRQA from psychometric questions, limited jailbreak prompts, and a safe-role description, then fine-tunes on synthesized, validated, and expanded role-consistent interactions.

    Source: §1 · p. 3

  • Results

    Across multiple Base and Instruct models, SSRFT generally improves safety robustness and generalization while reducing over-refusal and preserving general model capabilities.

    Source: §1 · p. 3, §5 · p. 21

  • Takeaways & Limitations

    The results support safe-role internalization as an alternative to refusal-centric alignment for improving safety behavior across diverse interaction scenarios.

    Source: §5 · p. 21

    The authors leave scaling to larger post-training settings, adaptability across model families, and dynamic context-dependent alignment for future work.

    Source: §5 · p. 21

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.

1 Introduction

Jailbreaks threaten the safety and reliability of LLM-based information access, while refusal-centered alignment can remain shallow, attack-specific, and prone to over-refusal. SSRFT instead trains models to internalize a safe role, and experiments report stronger robustness and generalization than standard SFT.

1 Introduction

  • Jailbreaks can elicit harmful outputs that undermine information quality and downstream decisions, while conventional alignment can depend on known attacks, shallow refusal prefixes, and excessive refusals.Maintaining attack coverage requires continuing data collection; RLHF also requires substantial preference supervision and computational resources.Source: §1 · p. 2
  • Whether a safety-oriented persona can itself serve as an alignment target remains underexplored, despite role-playing's ability to shape behavior through prompting or fine-tuning.Prior studies chiefly treat personas as attack vectors, alignment-data generators, or controllable behavioral attributes.Source: §1 · p. 3
  • SSRFT builds an SRQA dataset from psychometric questions, a small set of jailbreak prompts, and a safe-role description, then synthesizes, validates, and expands role-consistent responses.Fine-tuning on these interactions aims to embed safety-oriented values, principles, and interaction styles into model parameters.Source: §1 · p. 3
  • SSRFT uses limited attack-specific examples and synthesized safe-role interactions, aiming to guide responses beyond refusal prefixes while distinguishing harmful intent from benign sensitive queries.The framework retains standard SFT's simplicity without RLHF's additional models, preference data, and optimization complexity.Source: §1 · p. 3
  • Across multiple Base and Instruct models, SSRFT generally improves safety robustness and generalization while reducing over-refusal and preserving general capabilities.The reported advantages include stronger robustness to prefilling attacks and unseen jailbreak domains.Source: §1 · p. 3

2 Related Work

Related work covers LLMs in information retrieval, persona internalization, and safety alignment. The paper positions SSRFT as using a safety-oriented persona itself as the alignment target, rather than treating role-playing only as a risk or data-generation tool.

2.1 LLMs in Information Retrieval

  • LLMs increasingly combine retrieval, ranking, planning, and generation in information-seeking systems, where unsafe outputs can compromise information access and decision support.The cited work also explores conversational search and deep-search agents.Source: §2.1 · pp. 3–4

2.2 LLM Persona

  • Prior work demonstrates that personas can be prompted or fine-tuned into model behavior; SSRFT instead uses persona internalization as a safety mechanism under adversarial prompts.Examples include role-oriented datasets, self-alignment, and character-specific fine-tuning.Source: §2.2 · p. 4

2.3 LLM Safety Alignment

  • SFT and RLHF can require extensive attack or preference supervision, remain vulnerable to new jailbreaks, and may trade helpfulness for refusal; SSRFT targets internalized safety traits instead.Mechanistic analyses describe shallow alignment concentrated in early refusal tokens, which leaves models vulnerable to prefilling attacks.Source: §2.3 · p. 4

2.4 LLM Safety with Role-Playing

  • Although role-playing can enable jailbreaks and negative personas can increase toxicity, few studies make a safety-oriented persona the alignment target; SSRFT embeds one through fine-tuning.Prior persona work also treats personas as alignment-data generators or controllable stylistic attributes.Source: §2.4 · p. 5

3 The SSRFT Framework

SSRFT replaces refusal-centric supervision with a role-oriented pipeline that builds and fine-tunes on Safe-Role Question-Answer data. It selects role-revealing questions, defines a safe role, synthesizes and expands role-consistent responses, and trains the model to internalize that role.

3.1 Overview

  • SSRFT combines question collection, safe-role construction, role-guided synthesis, scenario expansion, and fine-tuning on the resulting SRQA dataset.The framework aims to embed safety-oriented values, behavioral principles, and interaction styles as a persistent persona.Source: §3.1 · p. 5

3.2 Question Collection

  • Questions are chosen to reveal role-discriminative traits, drawing on adapted psychometric instruments and a small set of jailbreak prompts for refusal supervision.Psychometric questions are useful because different roles are likely to answer them differently; jailbreak examples associate the safe role with rejecting harmful requests.Source: §3.2 · pp. 6–7

3.3 Safe-Role Construction

  • The target role is broadly applicable, consistently safe and cooperative across tasks, and described through safety-oriented traits, values, principles, and interaction styles.The profile is generated as a fictional character intended to remain competent across a wide range of tasks.Source: §3.3 · p. 7

3.4 Role-Guided Data Synthesis

  • Conditioning synthesis on the safe-role description yields answers reflecting its values and style; refusals express role traits rather than generic refusal language.Synthesized responses are scored for role consistency and factual quality, with samples scoring below 6 regenerated up to five times before manual review.Source: §3.4 · p. 8, Table 1 · p. 9

3.5 Dataset Expansion

  • Expansion recasts psychometric QA as realistic scenarios that preserve the underlying trait while reducing questionnaire-specific cues; checks cover role consistency, question requirements, and answer consistency.Each criterion is scored on a 0–10 scale with a passing threshold of 8; failures are regenerated for up to three iterations before manual review.Source: §3.5 · pp. 8–10

3.6 SSRFT Training

  • SSRFT fine-tunes on SRQA responses, where the safe role is implicit, to approximate the role-conditioned response distribution and internalize its characteristics into model parameters.The trained model can exhibit the safe role at inference without an explicit role prompt.Source: §3.6 · p. 10

4 Experiments

The experiments compare SSRFT with a token-matched standard SFT baseline across Base and Instruct backbones. The setup uses held-out in-domain and out-of-domain jailbreak evaluations and a corpus composed mostly of persona-oriented data.

4.1 Experimental Setup

  • The study evaluates four Base models and two Instruct models to examine safety alignment and how existing instruction alignment interacts with safe-role internalization.The Base models are Qwen2.5-3B, Gemma2-2B, Llama3.1-8B, and LRC-4B; the Instruct models are Qwen2.5-3B-Instruct and Qwen3-4B-Instruct.Source: §4.1 · p. 10
  • Chinese and English jailbreak data are split into training domains and held-out in-domain and out-of-domain tests.The Chinese training split uses 18 of 36 domains and 135 questions; the English split uses seven of 13 domains and 105 questions.Source: §4.1 · p. 10
  • Persona-oriented data makes up most of SRQA, while jailbreak-refusal data forms a small share; the standard SFT baseline matches SRQA's token count within 0.3%.The baseline includes English and Chinese conversations and unconditioned jailbreak-refusal pairs using the same jailbreak questions as SRQA.Source: §4.1 · p. 11, Figure 3 · p. 11

4.2 Standard Safety and OOD Generalization

SSRFT improves safety on most Base models and achieves stronger out-of-distribution generalization than standard SFT, while adding less benefit to already aligned Instruct models. Evaluation covers malicious queries, jailbreaks, and an OOD benchmark.

Safety Evaluation Protocol

  • Safety evaluation uses held-out malicious-query and jailbreak sets, plus AdvBench as an OOD benchmark with 520 harmful behaviors and no training-data overlap.ASR is evaluated with MD-Judge4 using greedy decoding and max_new_tokens=256.Source: §4.2 · pp. 11–12

Safety on Base Models

  • Average ASR falls from 7.68% to 1.76% for Gemma2-2B and from 15.31% to 8.69% for Llama3.1-8B with SSRFT versus standard SFT.Qwen2.5-3B is the exception: SSRFT and SFT are comparable at 14.41% and 14.05%, respectively.Source: §4.2 · p. 12

OOD Safety Generalization

  • On AdvBench, ASR decreases from 94.42% to 2.31% for Gemma2-2B and from 85.77% to 0.58% for Llama3.1-8B under SSRFT.The table reports ASR on malicious-query, jailbreak, and OOD safety benchmarks; SSRFT has the lowest AdvBench ASR across model families, including ties.Source: §4.2 · pp. 12–13, Table 4 · p. 13

Interaction with Existing Alignment

  • SSRFT adds more limited gains for already aligned Instruct models, particularly on standard jailbreak benchmarks.The paper attributes the narrower gains to less room for parameter-level modification after extensive instruction and safety alignment.Source: §4.2 · p. 13

4.3 Robustness to Prefilling Attacks

Prefilling attacks test whether safety persists when an affirmative prefix bypasses initial refusal behavior. SSRFT is more robust than standard SFT under these attacks and generally transfers better across jailbreak domains.

Prefilling Attack Protocol

  • The prefilling evaluation prepends “Sure! Here is the answer you requested:” before generation to test whether safety holds when refusal behavior is bypassed.The attacks are tested on JailBench-Seed, JailbreakLLMs, and AdvBench.Source: §4.3 · p. 13

Robustness beyond Refusal Prefixes

  • Under prefilling attacks, SSRFT's average ASR is 23.28% on Gemma2-2B and 16.54% on Llama3.1-8B, less than half their SFT counterparts.For SFT, average ASR rises by 58.63% and 39.50%, reaching 65.12% and 42.64%, respectively.Source: Table 5 · p. 14, §4.3 · p. 14

Cross-Domain Jailbreak Generalization

  • On Qwen2.5-3B, SSRFT lowers the JailBench-Seed out-of-domain/in-domain ASR ratio from 0.86 to 0.55 relative to SFT.Lower ratios indicate stronger transfer to unseen jailbreak domains.Source: §4.3 · p. 14, Figure 4 · p. 15

Safety–Helpfulness Balance

  • On the safe XSTest subset, Figure 5 reports higher Full Compliance and lower Full Refusal for SSRFT than SFT across most models.The comparison assesses over-refusal on harmless queries.Source: Figure 5 · p. 15

4.4 Over-Refusal and Helpfulness

On harmless XSTest queries, SSRFT improves compliance and reduces refusals relative to SFT across Base and Instruct models. The evaluation distinguishes complete compliance, complete refusal, and partial refusal.

Over-Refusal Evaluation

  • The harmless XSTest subset is scored as Full Compliance, Full Refusal, or Partial Refusal, and reports Full Compliance and Full Refusal rates.Only SFT and SSRFT are compared to isolate over-refusal introduced by additional safety alignment.Source: §4.4 · p. 15

Safety–Helpfulness Balance

  • Full Compliance rises from 24.4% to 42.0% on Gemma2-2B and from 30.0% to 54.0% on Llama3.1-8B with SSRFT versus SFT.On LRC-4B, compliance remains comparable at 48.8% versus 49.6%, while Full Refusal falls from 41.2% to 38.0%.Source: §4.4 · p. 15
  • On Qwen3-4B-Instruct, SSRFT raises Full Compliance from 56.8% to 66.4% and lowers Full Refusal from 40.0% to 27.2%.The paper says this pattern supports distinguishing harmful intent from benign but superficially sensitive requests rather than simply encouraging more refusals.Source: §4.4 · p. 15

4.5 Effect of Dataset Expansion

Dataset expansion generally strengthens safe-role internalization, improving persona stability and helpfulness while increasing role-based protection under prefilling attacks, though it does not uniformly improve explicit refusal robustness.

Ablation Protocol

  • The ablation removes expanded scenario-based QA pairs while retaining the remaining role-oriented data to measure the contribution of dataset expansion.The matched Non-Expanded SFT comparison uses the same jailbreak-refusal pairs and training size and optimization budget.Source: §4.5 · pp. 15–16

Persona Stability Evaluation

  • Persona stability is evaluated with BFI-44 using a template different from the IPIP-NEO training format, with lower normalized response entropy indicating more consistent role behavior.The template offers five Likert options.Source: §4.5 · p. 16

Effects of Dataset Expansion

  • Expanded SRQA generally lowers BFI-44 persona entropy, indicating more consistent role behavior, with Gemma2-2B and Llama3.1-8B as exceptions.Expansion also raises Full Compliance from 9.6% to 42.0% on Gemma2-2B, 34.4% to 54.0% on Llama3.1-8B, and 44.0% to 66.4% on Qwen3-4B-Instruct; standard ASR rises modestly in several settings while over-refusal falls substantially.Source: §4.5 · p. 16
  • Expansion increases the relative prefilling-robustness advantage of SSRFT over matched SFT for most models, even when absolute SSRFT attack performance does not uniformly improve.For Llama3.1-8B, Δ(2) rises from 10.20% to 23.88%; for Qwen3-4B-Instruct, it rises from 1.34% to 4.88%.Source: §4.5 · p. 17, Table 7 · p. 17

Interpreting Expansion through JRS and RPS

  • Expansion shifts the balance from jailbreak-refusal security toward role-playing security, trading some explicit refusal strength for stronger role-based protection and less over-refusal.The larger Δ(2) after expansion is interpreted as a stronger relative robustness advantage from role-oriented supervision.Source: §4.5 · p. 17

4.6 Training Dynamics and Model Dependence

SSRFT’s benefits vary with both a model’s existing alignment and its ability to learn role-oriented supervision. Training dynamics suggest that modifying an already aligned policy can be harder than learning the safe role from scratch.

Existing Alignment Increases Internalization Difficulty

  • Already aligned Instruct models show smaller SSRFT gains than Base models, and Qwen2.5-3B-Instruct has larger gradient norms than its Base counterpart, especially under SSRFT.The authors suggest instruction tuning may establish a behavioral profile that SSRFT must modify; Figure 6 shows Instruct models generally need more optimization to reach their best SSRFT validation performance.Source: §4.6 · pp. 17–18, Figure 6 · p. 18

Training Dynamics

  • SSRFT and SFT have similar convergence on stronger Base models, but SSRFT converges more slowly on some Instruct and lower-capability models.The authors attribute this variation to existing alignment and model capability.Source: §4.6 · p. 17, Figure 6 · p. 18

Model Capability Governs Safe-Role Internalization

  • Llama3.1-8B and Qwen3-4B-Instruct show small late-stage loss gaps between SSRFT and SFT and strong prefilling robustness, while Qwen2.5-3B retains a loss gap and has more modest safety gains.The first two models also benefit from dataset expansion.Source: §4.6 · p. 18

4.7 General Capability Preservation

Across nine zero-shot benchmarks spanning reasoning, commonsense, reading comprehension, and factual knowledge, SSRFT preserves general capabilities comparably to standard SFT.

General-Capability Evaluation

  • The evaluation uses zero-shot benchmarks in four capability groups and reports Accuracy Norm for ARC-C and LogiQA, and Accuracy for the remaining benchmarks.The groups are scientific and logical reasoning, commonsense reasoning, reading comprehension, and factual knowledge.Source: §4.7 · p. 18

Capability Preservation

  • SSRFT achieves similar overall performance to standard SFT across nine benchmarks, with no noticeably greater catastrophic forgetting.The authors conclude that role-oriented supervision primarily modifies behavioral preferences while largely preserving underlying reasoning and knowledge capabilities.Source: §4.7 · p. 19

4.8 Comparison with Alternative Safety Alignment Paradigms

SSRFT is compared with RLHF-based alignment and inference-time role prompting. In the reported evaluations, it has the lowest standard and prefilling attack success rates, with substantially lower training cost than RLHF.

Comparison Protocol

  • The alternatives represent policy optimization during post-training and role conditioning during inference, respectively.The RLHF comparison uses a 1% subsample of PKU-SafeRLHF under a constrained resource budget, while prompt-based role-playing prepends the same safe-role description without modifying model parameters.Source: §4.8 · p. 19

Comparison with Prompt-Based Role-Playing

  • Prompt-based role-playing lowers standard ASR from 37.19% to 33.03% and prefilling ASR from 24.72% to 16.71%, but parameter-level SSRFT is reported as more robust across the evaluated jailbreak settings.The comparison frames prompt-based role-playing as inference-time conditioning and SSRFT as supervised internalization of the role in model parameters.Source: §4.8 · pp. 19–20

Comparison with RLHF

  • SSRFT achieves the lowest average ASR in standard and prefilling evaluations while requiring substantially lower training cost than RLHF.In the constrained comparison, SSRFT records 18.83% standard ASR and 4.98% prefilling ASR, versus PKU-SafeRLHF’s 37.40% and 52.44%, respectively.Source: §4.8 · p. 19, Table 9 · p. 20

5 Conclusions

SSRFT internalizes a predefined safe role through role-oriented fine-tuning, shifting safety alignment from memorizing refusal patterns to learning behavioral principles. Experiments show generally improved robustness and generalization while preserving general capabilities, and the approach is presented as a resource-efficient alternative.

5 Conclusions

  • Across multiple Base and Instruct models, SSRFT generally improves safety-alignment robustness and generalization while preserving general model capabilities.The authors describe safe-role internalization as a different alignment mechanism from conventional refusal-based supervision.Source: §5 · p. 21
  • SSRFT internalizes a predefined safe role through role-oriented supervised fine-tuning rather than relying on large jailbreak-refusal collections or computationally expensive RLHF.The alignment objective shifts from memorizing refusal patterns to learning safety-oriented behavioral principles.Source: §5 · p. 21
  • SSRFT is presented as a practical, resource-efficient alternative that makes robust safety alignment more accessible under limited computational budgets.Source: §5 · p. 21
  • Future work will explore scaling SSRFT to larger post-training settings, improving adaptability across model families, and extending it to dynamic, context-dependent alignment.Source: §5 · p. 21
Loading 2610.07023v1…