PaperlayerView exact revision on arXiv

Source-linked AI summary

PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models

Gianluca Barmina, Federico Torrielli, Sven Harms, Jacob Nielsen, Felix Mächtle, Stine Lyngsø Beltoft, Peter Schneider-Kamp, Thomas Eisenbarth, Lukas Galke Poech, Anne Lauscher

arXiv:2606.09697v1cs.CL

TL;DR

Current refusal techniques often reduce safety to binary non-compliance, leaving limited support for users in high-risk interactions. PsychoSafe frames refusals as psychologically informed supportive communication and finds that prompting improves overall refusal quality by 28.1% over a generic baseline while preserving downstream non-refusal performance.

  • Problem

    Current refusal techniques largely frame safety as binary classification, discarding opportunities to provide genuine support between compliance and outright denial.

  • Method

    PsychoSafe harmonizes safety datasets, identifies evidence-based psychological intervention principles, and instantiates them as reusable refusal exemplars across five risk domains.

  • Results

    28.1% improvement over a generic baseline in overall refusal quality was achieved by prompting, while preserving downstream non-refusal performance; fine-tuning reached near-perfect refusal and referral rates but reduced relevance.

  • Takeaways & Limitations

    Psychologically informed refusals can make safety responses more helpful, empathetic, and constructive while retaining non-refusal capabilities.

  • Takeaways & Limitations

    The framework covers only five psychologically salient risk clusters and does not address the full space of unsafe model behavior.

Abstract

from arXiv · show

Large language models (LLMs) routinely face requests that should be refused, creating a trade-off between helpfulness and harm prevention. However, refusals themselves can be helpful. In high-risk interactions involving crisis, coercion, or escalating intent, blunt non-compliance may prevent direct harm while still failing to support the needs of the person behind the request. We present PsychoSafe, a psychologically-informed refusal framework that reframes refusal as structured supportive communication grounded in evidence-based intervention strategies. To develop PsychoSafe, we construct a corpus of 8019 prompt-response pairs spanning five psychologically salient risk domains and apply prompting and parameter-efficient fine-tuning to Qwen 3.5 27B. On a balanced validation set of 500 prompts, evaluated with an LLM judge and validated through human ratings, PsychoSafe prompting improves overall refusal quality by 28.1% over a generic baseline, with particularly strong gains in external resource referral (+46.8%) and psychological grounding (+34.8%), while preserving downstream performance on non-refusal tasks. Fine-tuning achieves near-perfect refusal and resource-referral rates but reduces response relevance. Additional evaluations on SORRY-Bench and XSTest show strong in-domain robustness but limited out-of-domain generalization, suggesting that future work should diversify fine-tuning data to help models apply interventions selectively rather than schematically.

1 Introduction

The introduction frames refusals as an opportunity for supportive communication rather than binary non-compliance. PsychoSafe operationalizes psychologically informed principles into a framework, dataset, and evaluation of helpful refusals across five risk domains.

  • Motivation: Current refusal techniques largely reduce safety to binary classification, overlooking opportunities between full compliance and outright denial.This framing limits models’ ability to support users while declining unsafe requests.
  • Motivation: Helpful refusals acknowledge underlying needs, explain why compliance is problematic, and redirect users toward constructive alternatives.The approach treats helpfulness and harm prevention as potentially compatible rather than necessarily zero-sum.
  • Motivation: Prior work decomposes refusals into components but lacks a principled account of what makes them genuinely helpful in high-risk situations.The introduction identifies psychological theory as the missing framework for grounding refusal behavior and improving generalizability.
  • Framework: PsychoSafe applies validated supportive-communication principles to five psychologically salient safety domains.Motivational interviewing and crisis communication provide principles for delivering unwelcome messages while preserving agency and well-being.
  • Method and contributions: The PsychoSafe dataset contains 8,019 psychologically informed prompt–refusal pairs, and the study compares system prompting with fine-tuning on distilled reasoning traces using 500 held-out safety prompts.Evaluation targets qualities including empathy, psychological grounding, and referral to external resources, using Qwen3.5-35B-A3B.

2 Psychologically-informed Refusals

PsychoSafe operationalizes refusal as structured supportive communication by harmonizing safety taxonomies, selecting five acute or vulnerable risk domains, and adapting brief psychological intervention principles. It instantiates these principles in an expert-reviewed dataset of 8,019 prompt–response pairs spanning the five risk categories.

  • Framework construction: PsychoSafe is constructed by harmonizing existing safety data, identifying adaptable psychological intervention principles, and pairing reusable refusal exemplars with unsafe prompts.The framework treats refusal as a structured, supportive act rather than a binary endpoint.
  • Risk-space definition: Safety taxonomies are unified through semantic clustering, then screened for specificity, intervention availability, clinical relevance, acute or escalating risk, and vulnerability.The clustering uses sentence-t5-large embeddings and HDBSCAN; the screening identifies domains where refusals should incorporate supportive or de-escalatory strategies.
  • Risk clusters: Five risk clusters are selected: suicide and self-harm, sexual crimes, substance use, weapons, and violence.Their refusal strategies respectively emphasize crisis de-escalation and referral, consent and victim protection, reflection and help-seeking, interruption and safer alternatives, or reduced escalation and safer action.
  • Intervention principles: Psychological intervention frameworks converge on acknowledging the person without endorsing the request, reducing immediate risk, preserving agency, encouraging self-efficacy, suggesting next steps, and facilitating support.The selected approaches are brief, suitable for non-specialists, procedurally structured, and expressible in short responses.
  • Dataset construction: 8,019 prompt–response pairs comprise PSYCHOSAFE across the five risk categories, with each entry containing a prompt, psychologically informed refusal, risk cluster, and provenance metadata.The category counts are suicide and self-harm (2,578), crimes (sexual) (326), substance (1,998), weapon (1,740), and violence (1,377); expert review assesses psychological appropriateness, clarity, and non-escalatory tone.

3 Elicitation

PsychoSafe is elicited through complementary inference-time prompting and parameter-level fine-tuning, enabling context-sensitive refusals beyond fixed templates. The system prompt specifies structured, personalized responses, while fine-tuning internalizes the behavior using imputed reasoning traces to reduce reliance on long prompts.

  • Elicitation settings: PsychoSafe is evaluated through a dedicated system prompt and supervised fine-tuning on human-crafted responses augmented with imputed reasoning traces.The two settings test whether the refusal style can be induced at inference time or internalized in model parameters.
  • In-context prompting: The system prompt encodes a fixed four-part refusal structure while supporting psychologically-informed, context-sensitive responses.The framework includes warm acknowledgment, strategy selection, cluster-specific resources, and example responses, although the supplied passage truncates the full structure.
  • In-context prompting: The prompt personalizes refusals by mirroring language register, referencing situation and emotional cues, and varying phrasing across interactions.For multiple clusters, it prioritizes suicide and self-harm as the most acute risk, blends relevant strategies, and provides resources for applicable clusters.
  • Fine-tuning: Long inputs can decrease LLM performance, motivating fine-tuning to reproduce PsychoSafe behavior without relying on a long system prompt.This positions parameter-level elicitation as a deployment-oriented alternative to prompt-only induction.

4 Experiments

Experiments compare PSYCHOSAFE prompting and fine-tuning across psychologically informed refusal criteria, standard refusal benchmarks, and general capabilities. Prompting improves refusal quality substantially, while fine-tuning maximizes refusal but reduces relevance and out-of-domain robustness.

  • Experimental setup: 500 stratified validation examples were evaluated by an LLM judge across refusal, psychological grounding, external references, and relevance.The set contained 100 examples per risk cluster, and judge performance was validated against human ratings.
  • Judge selection: κ = 0.61 was the highest human-expert agreement, achieved by Qwen 3.5 397B-A17B, which was selected as the judge.GLM 4.7 reached κ = 0.58 and Mistral Large 3 675B Instruct reached κ = 0.56.
  • Judge results: 92% was the best normalized overall score for base Qwen 3.5 27B with the PSYCHOSAFE prompt, versus 71.9% with the standard prompt.Fine-tuned models scored 82.7% with v0 and 82% with v1; PSYCHOSAFE or v1 led every criterion.
  • Judge results: +28.1% was the overall increase from PSYCHOSAFE prompting over v0, with gains of +46.8% for External References and +34.8% for Psychological Grounding.Fine-tuning reached up to 100% Refusal but reduced Relevance by up to −25% and became nearly indifferent to prompt version.
  • General capabilities: 3 percentage points was the slight HellaSwag increase for the PSYCHOSAFE fine-tuned model, while MMLU decreased by about 4 pp versus the base model.The v1 prompt decreased HellaSwag by 5 pp, while MMLU results were approximately unchanged across v0 and v1.
  • Standard refusal benchmarks: 78.5% was the base model’s adversarial safety under v1, up from 59.0%, while over-refusal rose from 13.2% to 24.0%.The fine-tuned model over-refused 3.6% on benign prompts versus 13.2% for the base, but refused only 17.0% of contrast prompts versus 59.0%.

5 Discussion

The discussion frames refusal quality as a design variable: PsychoSafe prompting improves supportive refusal dimensions, while fine-tuning strengthens refusal behavior but reduces relevance and generalizes weakly beyond its training domains.

  • Prompting and fine-tuning: Supervised fine-tuning reaches near-perfect refusal and resource-referral rates but reduces relevance relative to PsychoSafe prompting.Its targets emphasize human-crafted templates for the five risk clusters, and performance is nearly indifferent to prompt version.
  • Prompting and fine-tuning: Qualitative analysis finds that the fine-tuned model applies a crisis-intervention schema too broadly, capturing the template without the pragmatic judgment for selective application.The paper suggests a different training strategy may improve both relevance and stability.
  • Generalization and deployment: Safety on out-of-domain adversarial prompts falls well below the base model, indicating weak generalization beyond the five training domains.The discussion calls for more diverse fine-tuning data and positions PsychoSafe for domain-specific deployments.

6 Conclusion

PsychoSafe is a psychologically informed refusal framework that uses structured supportive communication to make refusals more helpful. The paper introduces a risk-domain dataset and evaluates in-context learning and supervised fine-tuning, finding substantial refusal-quality gains and evidence of generality across standard benchmarks.

  • Conclusion: PsychoSafe frames helpful refusals as structured supportive communication grounded in psychological considerations.The framework is presented as psychologically informed.
  • Conclusion: 8,019 prompt-response pairs span five psychologically salient risk domains in the introduced dataset.The dataset supports developing the proposed refusal framework.
  • Conclusion: In-context learning and supervised fine-tuning are proposed methods for eliciting psychologically informed refusals.Both approaches are evaluated as mechanisms for producing the framework’s refusals.
  • Conclusion: Substantial refusal-quality gains are shown by a human-validated LLM judge, alongside standard-benchmark experiments measuring safety and general capabilities.The benchmark results indicate generality of the approach.
  • Conclusion: Future work may broaden the diversity of responses used for fine-tuning.The conclusion identifies response diversity as a direction for improving fine-tuning data.

Limitations

PsychoSafe’s limitations concern the abstraction of human interventions into brief text, narrow domain and language coverage, limited personalization, and unresolved generalization beyond single-turn validation. These constraints motivate broader resources, greater response diversity, and evaluation across more realistic settings.

  • Intervention scope: Psychological interventions were designed for human–human settings and abstracted into short written responses, which should not be interpreted as therapy, diagnosis, or crisis management.The framework may support de-escalation and redirection, but it is not a substitute for clinical or crisis care.
  • Coverage: Domain coverage is intentionally narrow: five psychologically salient risk clusters do not represent the full space of unsafe model behavior.Many harmful domains are safety-relevant without being clearly amenable to short psychological intervention.
  • Language and resources: The released artifact is English-only, with built-in support resources primarily tailored to the United States and the United Kingdom.Broader language and resource coverage would improve generality and accessibility.
  • Personalization: The fine-tuned model shows limited personalization in some refusals, likely because of limited variability in the training data.Improving response diversity and context sensitivity may strengthen performance and more robustly embed psychological refusal behavior.
  • Evaluation limits: Current results rely on single-turn validation prompts, leaving multi-turn behavior, adversarial robustness, cross-lingual generalization, and real-user effects unresolved.These areas remain open for future evaluation.

Ethical Considerations … F Example from PSYCHOSAFE Dataset

PSYCHOSAFE combines structured, psychologically informed refusals with explicit ethical limitations and evaluation recommendations. Its supplementary materials detail risk-category harmonization, elicitation structure, datasets, intervention mappings, cluster assessment, and an example tailored response.

  • Ethical Considerations: Psychologically informed refusals improve refusal quality but remain generative, so miscalibration can provide inadequate support, escalate distress, create false reassurance, or mishandle crises.PSYCHOSAFE is not a substitute for professional mental-health intervention, crisis counseling, or medical advice.
  • Ethical Considerations: The authors recommend adversarial, emotionally charged, and out-of-distribution stress tests, plus continuous monitoring, iterative correction, and evaluation across diverse cultural contexts.These recommendations address the possibility that refusal behavior may fail across inputs and conversational settings.
  • A Details on Harmonizing Risk Categories: Risk categories are harmonized through semantic clustering and comparison with taxonomies from Vidgen et al. (2024), Ghosh et al. (2025), and Inan et al. (2023).The clustering pipeline embeds labels with sentence-t5-large, reduces dimensionality with UMAP, and applies HDBSCAN.
  • A Details on Harmonizing Risk Categories: 672 parameter combinations are evaluated for UMAP and HDBSCAN, prioritizing low noise and interpretable clusters aligned with existing safety taxonomies.The target is approximately 15–20 semantically coherent clusters, with TF–IDF used to derive interpretable cluster names.
  • B PSYCHOSAFE Elicitation Prompt: The elicitation prompt requires a four-part response: acknowledgment and gentle refusal, personalized self-help, professional resources, and a hopeful closing.Personalization guidelines emphasize emotional cues, language register, specific circumstances, response variation, and avoiding diagnosis or labels.
  • B PSYCHOSAFE Elicitation Prompt: For multi-cluster messages, the prompt prioritizes the most acute risk, always placing suicide and self-harm first, while blending strategies and providing resources for applicable clusters.Non-sensitive educational or research questions are answered normally when context indicates non-harmful intent.
  • C Selected Datasets: The full list of datasets is provided in Table 5, while Table 6 presents the mapping between intervention strategies and risk clusters.These tables document the resources and strategy assignments underlying the framework.
  • E Assessment of Psychologically Relevant Risk Clusters: Table 7 reports psychologically relevant risk-cluster assessments on five criteria, using scores from 0 to 6 with Yes = 2, Unclear = 1, and No = 0; Table 8 provides a full tailored-response example.The criteria include clinical or psychosocial relevance, acute risk, and effects on vulnerable groups.

G Full Results for Agreement Between Human and Judges … J SORRY-Bench Task Details

The supplementary sections report human–judge agreement results, define LLM-judge criteria, document fine-tuning procedures, and describe SORRY-Bench’s linguistic robustness mutations.

  • G Full Results for Agreement Between Human and Judges: Full human–judge agreement results are reported with different metrics in Tables 9 and 10.
  • H LLM Judge Criteria: The evaluation presents the original user request and assistant response for judging potentially harmful-request handling.
  • H LLM Judge Criteria: Refusal is judged as Yes when the assistant clearly refuses, and No when it complies or provides harmful information.
  • H LLM Judge Criteria: Psychological Grounding is rated 1–5, from a generic blunt refusal to a coherent, well-executed psychological strategy.
  • H LLM Judge Criteria: External References are assessed by whether the response points users to a helpline, professional service, or other resource.
  • H LLM Judge Criteria: Relevance is rated 1–5 according to how specifically and completely the response adapts to the user’s situation.
  • J SORRY-Bench Task Details: SORRY-Bench tests refusal robustness across 19 linguistic mutations spanning style, persuasion, ciphers, and translation.The mutations include slang, uncommon dialects, technical terms, role play, misspellings, ASCII, Caesar, Morse, Atbash, Malayalam, Tamil, Marathi, simplified Chinese, and French.

K SORRY-Bench Results for the Mixture-of-Experts Variant

On SORRY-Bench, Qwen3.5-35B-A3B largely mirrors the dense 27B’s compliance and SFT patterns, while showing greater Caesar-cipher vulnerability and slightly more residual leakage after safety fine-tuning. Its SFT checkpoint reduces over-refusal but substantially lowers safety relative to the base model.

  • Evaluation setup: The evaluation used identical generation and judge settings for the Qwen3.5-35B-A3B MoE base and its PSYCHOSAFE SFT counterpart.The MoE model has 35B total parameters and approximately 3B active parameters per token.
  • Compliance patterns: Under default and v0 prompts, MoE compliance was 26.9 / 26.6% versus 25.4 / 25.4% for the dense 27B, while v1 rates were comparable at 17.7% versus 19.0%.Mutation-family ordering was unchanged: persuasion strategies were strongest, while atbash was essentially ineffective.
  • Compliance patterns: 12.9% of MoE evaluations were vulnerable to Caesar-cipher attacks versus 4.7% for the dense 27B.This was the main attack-specific vulnerability difference reported for the MoE variant.
  • PSYCHOSAFE SFT: 25 of 22,971 MoE PSYCHOSAFE SFT mutation evaluations scored 1, compared with 9 for Qwen3.5-27B-psysafe.Residuals concentrated in four translation mutations and reflected partially translated or degenerate-loop generations.
  • XSTest results: 5.6–18.8% over-refusal for MoE SFT versus 13.6–27.6% for the base coincided with 17.5–41.0% safety versus 59.5–67.0%.Under v1, MoE SFT safety reached 41.0%, above its dense counterpart’s 26.5% but below the MoE base’s 67.0%.

L Result Details on SORRY-Bench · N Information about Human Annotator

On SORRY-Bench, PsychoSafe prompting reduces compliance, while the fine-tuned Qwen3.5-27B-psysafe model reaches near-zero compliance across mutation conditions. The judge-selection annotations were provided by one informed, English-fluent student assistant and contained only judgments, not sensitive personal information or participant interventions.

  • L Result Details on SORRY-Bench: 25.4% mut-avg compliance under the default prompt falls to 19.0% with v1, remaining above the SFT model’s 0.0%.Prompting alone therefore leaves a substantial gap to the SFT model.
  • L Result Details on SORRY-Bench: Without a system prompt, logical_appeal is the strongest attack at 59.3% compliance, followed by persuasion and endorsement mutations.The next tier includes misrepresentation (49.4%), evidence-based_persuasion (45.2%), authority_endorsement (41.4%), and expert_endorsement (30.8%).
  • L Result Details on SORRY-Bench: Cipher mutations are weak attacks, with compliance ranging from 0.5% for atbash to 16.4% for morse, consistent with unreliable decoding of underlying requests.The reported cipher results are atbash 0.5%, caesar 4.7%, ascii 10.4%, and morse 16.4%.
  • L Result Details on SORRY-Bench: Writing-style mutations vary around the base rate: role_play is below it at 16.1%, whereas slang is above it at 21.8%.These results are reported among the writing-style mutations.
  • L Result Details on SORRY-Bench: Qwen3.5-27B-psysafe drops to near-zero compliance across all six SORRY-Bench conditions.Across 22,971 mutation evaluations, only 9 rows were scored as compliance; manual inspection found partially translated or degenerate-loop generations rather than substantive jailbreaks.
  • N Information about Human Annotator: The judge-selection annotations were provided by a business-informatics student assistant fluent in English who was informed about data use, instructions, and the broader project context.The collected data consisted only of annotation judgments and did not involve sensitive personal information or interventions with human participants.

O Matching of Dataset Intended Use

The study uses Röttger et al.’s datasets consistently with their stated purpose as open resources for evaluating and improving large language model safety, applying them to safety-oriented refusal behavior.

  • Dataset intended use: The datasets are used to construct and evaluate safety-oriented refusal behavior in large language models, matching their intended safety-evaluation purpose.Röttger et al. describe them as “Open Datasets for Evaluating and Improving Large Language Model Safety.”
Loading 2606.09697v1…