PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Models

Yeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim, Mehrab Tanjim, Sangwu Park, Kibum Kim, Chanyoung Park

arXiv:2502.15086v2cs.CL

TL;DR

Existing LLM safety benchmarks largely use general standards, although safety needs may vary by user profile and comprehensive evaluation of user-specific safety is lacking. The paper introduces U-SAFEBENCH and evaluates 20 widely used LLMs, finding low user-specific safety scores; it also proposes a chain-of-thought remedy.

  • Problem

    Existing LLM safety benchmarks rely heavily on general standards, while comprehensive benchmarks and evaluation frameworks for user-specific safety remain largely overlooked.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces U-SAFEBENCH, a benchmark built around user profiles and instructions, and an evaluation protocol for user-specific safety and helpfulness.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across 20 widely used LLMs, current models fail to act safely for user-specific standards, averaging a 14.1% user-specific safety score, significantly below safety under general standards.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify user-specific safety as a vulnerability in current LLMs, and the paper’s chain-of-thought remedy improves user-specific safety.

    Source: §6 · p. 9

    The chain-of-thought remedy still has room to improve the safety-helpfulness trade-off, and most models’ overall safety remains inadequate for real-world deployment.

    Source: Limitations · p. 10

Abstract

from arXiv · show

As the use of large language model (LLM) agents continues to grow, their safety vulnerabilities have become increasingly evident. Extensive benchmarks evaluate various aspects of LLM safety by defining the safety relying heavily on general standards, overlooking user-specific standards. However, safety standards for LLM may vary based on a user-specific profiles rather than being universally consistent across all users. This raises a critical research question: Do LLM agents act safely when considering user-specific safety standards? Despite its importance for safe LLM use, no benchmark datasets currently exist to evaluate the user-specific safety of LLMs. To address this gap, we introduce U-SafeBench, a benchmark designed to assess user-specific aspect of LLM safety. Our evaluation of 20 widely used LLMs reveals current LLMs fail to act safely when considering user-specific safety standards, marking a new discovery in this field. To address this vulnerability, we propose a simple remedy based on chain-of-thought, demonstrating its effectiveness in improving user-specific safety. Our benchmark and code are available at https://github.com/yeonjun-in/U-SafeBench.

1 Introduction

LLM safety benchmarks largely rely on general standards, although the same response can pose different risks depending on a user’s profile. The paper introduces U-SAFEBENCH to evaluate user-specific safety and reports that current LLMs perform poorly on this measure.

1 Introduction

  • A response about a dangerous zolpidem dose or a routine alcohol purchase can carry different risks for users with depression, alcohol use disorder, or pregnancy.The examples illustrate that a response considered helpful or innocuous for a general user could have harmful consequences for someone with a particular profile.Source: §1 · pp. 1–2
  • Safety standards may vary with users’ profiles and backgrounds, yet this dimension is largely neglected despite LLM agents’ growing use in diverse real-world applications.The paper identifies user-specific safety as a research question and notes that comprehensive benchmark datasets and evaluation frameworks remain overlooked.Source: §1 · p. 2
  • U-SAFEBENCH evaluates user-specific safety using over 130 user profiles and more than 2,700 real-world user instructions.The benchmark also includes an evaluation protocol for user-specific safety and helpfulness.Source: §1 · p. 2
  • 14.1% user-specific safety score on average across 20 widely used LLMs, significantly below their safety under general standards.The paper proposes a chain-of-thought approach to improve user-specific safety.Source: §1 · p. 2
  • The paper presents chain-of-thought as a simple remedy for the user-specific safety vulnerability it identifies.It describes the approach as effective for improving user-specific safety and a baseline for U-SAFEBENCH.Source: §1 · p. 2

2 Related Work

Prior work evaluates general LLM safety and personalization, but the paper distinguishes user-specific safety as avoiding harm tied to individual vulnerabilities. It presents U-SAFEBENCH as a broader evaluation of this dimension across QA and autonomous-agent tasks.

2 Related Work

  • Existing safety studies use general standards and leave a gap in benchmarks for user-specific safety across QA and autonomous-agent tasks.Prior autonomous-agent work evaluates how safely models interact within environments.Source: §2 · p. 3
  • Personalization research includes post-training approaches that incorporate user preferences, while often overlooking personalization of safety considerations (Lee et al., 2024).Other work discusses personalized safety values and methods, including Zhang et al. (2024a), Giubilini et al. (2024), Friedrich et al. (2022), and Brack et al. (2023).Source: §2 · p. 3
  • Unlike work focused on users’ values and reasoning styles, this paper targets avoiding harm tailored to individual vulnerabilities and offers a broader taxonomy, profiles, and instructions than Zhang et al. (2024a).The paper characterizes Zhang et al. (2024a) as presenting a narrow set of personalized-safety scenarios.Source: §2 · p. 3

3 User-Specific Safety Benchmark: U-SAFEBENCH

U-SAFEBENCH operationalizes safety around individual user profiles and evaluates risks across user instructions, profiles, and task settings. Its protocol pairs profile-conditioned harmful and benign instructions with model responses and a judge, supported by a dataset of varied profiles and risks.

3.1 User-Specific Safety

  • A response is user-specifically unsafe when generally safe fulfillment risks a particular profile; user-specific safety requires recognizing that risk and refusing.Source: §3.1 · p. 3

3.2 Safety Risk Scenarios

  • The risk taxonomy covers physical and mental health harms, plus assistance with illegal or unethical activity, with consequences tied to user profiles.The paper also connects these risks to personalized healthcare agents and chatbots that use attributes such as age or location.Source: §3.2 · pp. 3–4

3.3 Evaluation Protocol

  • Agents receive a user profile and instruction; Llama-3.1-8B-Instruct judges whether the response refuses or fulfills the instruction.Its judgments aligned with human annotators on 92% of 100 randomly sampled queries.Source: §3.3 · p. 4
  • Safety is measured by refusal ratio on harmful instructions, helpfulness by fulfillment ratio on benign instructions, and overall performance by their harmonic mean.The overall measure balances safety and helpfulness.Source: §3.3 · p. 5

3.4 Dataset Construction

  • U-SAFEBENCH contains 134 user profiles, 2,296 harmful instructions, and 491 benign instructions covering varied safety risks.Source: §3.4 · p. 5

3.4.1 User Profile Collection

  • Profiles include medical conditions such as pregnancy and diabetes for health risks, and demographic attributes such as age and location for illegal-activity risks.The profiles were collected to cover the risk scenarios in Section 3.2.Source: §3.4.1 · p. 5

3.4.2 Harmful Instruction Collection

  • Harmful instructions combine GPT-4o profile-conditioned generation and targeted prompts with contraindication-based templates, followed by similarity filtering and human review.Source: §3.4.2 · pp. 5–6

3.4.3 Benign Instruction Collection

  • For helpfulness evaluation, harmful instructions are made benign by replacing the associated profile with a general profile, then prompting models to respond.Source: §3.4.3 · p. 6

4 Experiement

The benchmark finds substantial user-specific safety failures across evaluated LLMs, with weaknesses spanning safeguard models, health-related scenarios, and jailbreak robustness. Case studies show that even the highest-scoring model makes recurring profile-sensitive mistakes.

4.1 Setup

  • The evaluation uses 2,787 instruction–profile pairs and responses from 17 instruction-following LLMs plus three safeguard Llama models.Experiments use zero-shot prompts, a single run, and temperature 0.Source: §4.1 · pp. 6–7

4.2 Benchmarking Results

  • Claude-3.5-sonnet leads with 53.7%, yet all evaluated instruction-following models have insufficient user-specific safety and remain substantially poor on U-SAFEBENCH compared with general-standard evaluations.Proprietary models generally score higher than open-source and safeguard models, and Claude significantly outperforms other model families.Source: §4.2 · p. 7
  • All evaluated safeguard LLMs fail to generalize to user-specific safety, reflecting that they were not designed for personalization.The authors identify developing safeguards for both general and user-specific concerns as a next step.Source: §4.2 · p. 7
  • Figure 5 shows a negative correlation between safety and helpfulness; Claude-3.5-sonnet and Claude-3.5-haiku score higher on safety but lower on helpfulness.Most models cluster in the upper-left region, indicating poor user-specific safety.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Average safety scores are 13.0% for Mental Health Risk and 7.4% for Physical Health Risk, while Illegal and Unethical Activity Assist scores are higher.Claude models show lower vulnerability to health-related risks; current LLMs also struggle in the autonomous-agent domain.Source: §4.2 · p. 7, Table 2 · p. 8
  • Prefix Injection and Refusal Suppression significantly degrade user-specific safety for nearly all models, showing that simple prompt variations can compromise it.The comparison uses Base models without attacks as the reference.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Case Studies

  • In case studies, Claude-3.5-sonnet misses when driving or political participation may be illegal under specific circumstances.These failure patterns are illustrated in Figure 7.Source: §4.3 · p. 8, Figure 7 · p. 8
  • Claude-3.5-sonnet also misses health risks tied to conditions such as pregnancy or stomach cancer and may encourage harmful behaviors among people with addiction.The authors conclude that even state-of-the-art models need further development across diverse user profiles.Source: §4.3 · pp. 8–9, Figure 7 · p. 8

5 Simple Remedy: CoT-based Approach

The paper proposes a two-step chain-of-thought approach that first infers user-specific safety guidelines and then generates a response using them. It improves benchmark safety scores, while potentially reducing helpfulness through overrefusal.

5 Simple Remedy: CoT-based Approach

  • The approach separates safety reasoning into inferring guidelines for what to avoid and generating a response that considers those guidelines.The authors suggest simultaneous guideline identification and response generation may overwhelm a single model's reasoning capabilities.Source: §5 · p. 9
  • Average safety score rises from 17.8% to 32.7% with the CoT approach, and Claude-3.5-sonnet reaches 76.7% with a small helpfulness loss.Table 3 compares vanilla prompting (Base) with the chain-of-thought approach (CoT); other models remain below deployment-ready safety.Source: §5 · p. 9, Table 3 · p. 9
  • The CoT approach may reduce helpfulness through overrefusal when a nuanced response would be appropriate.The authors suspect its safety prioritization and explicit consideration of refusal contribute to the helpfulness drop.Source: §5 · p. 9

6 Conclusion

The paper introduces user-specific safety as a safety concept and U-SAFEBENCH as a benchmark for evaluating it. Its evaluation finds current LLMs lack this safety, while a CoT-based remedy improves it and establishes a benchmark baseline.

6 Conclusion

  • The paper introduces user-specific safety as a novel concept and U-SAFEBENCH as a comprehensive benchmark for its evaluation.Source: §6 · p. 9
  • The evaluation finds that current LLMs fail to possess user-specific safety, revealing a previously unidentified vulnerability.Source: §6 · p. 9
  • A simple chain-of-thought remedy enhances user-specific safety and establishes a strong baseline for U-SAFEBENCH.Source: §6 · p. 9

Limitations

The paper identifies limits in both its chain-of-thought remedy and its helpfulness evaluation, and notes challenges around obtaining and using user profiles.

Limitations

  • Although the chain-of-thought remedy enhances user-specific safety, most models remain inadequate for real-world deployment and the safety-helpfulness trade-off needs improvement.The authors identify a post-training approach aligned with safe responses to user profiles as future work.Source: Limitations · p. 10
  • Pilot LLM-as-a-Judge ratings on a 1–10 scale aligned poorly with human annotators, so the helpfulness evaluation could be more fine-grained.The authors leave this direction for future exploration, citing difficulty quantifying instruction fulfillment on a discrete scale.Source: Limitations · p. 10
  • Personalization faces cold-start and indiscriminate-adaptation risks; the authors suggest query moderation followed by voluntary, consent-based profile collection when needed.This conditional flow is proposed to mitigate unavailable profile information and excessive or irrelevant personalization.Source: Limitations · p. 10

Ethics Statement

The paper describes ethical safeguards for U-SAFEBENCH, explains how its user-specific risk scenarios are scoped, and reports 134 collected user profiles.

Ethics Statement

  • The dataset could be misused to train profile-tailored harmful responses; the authors restrict release to safety research and plan content warnings and disclaimers.The paper also acknowledges potential psychological impact from exposure to harmful or sensitive content.Source: Ethics Statement · p. 10

A Safety Risk Scenario Setup Process

  • The scenario setup adapts SafetyBench categories but excludes offensiveness, unfairness and bias, and privacy and property because their risks are treated as universal.These categories fall outside the paper’s user-specific safety scope because such content should be prohibited for all audiences.Source: A Safety Risk Scenario Setup Process · p. 13

B Details of User Profile Collection

C Implementation Details

The implementation details identify the exact model versions used in the study.

C Implementation Details

D.1 Finegrained Analysis

The fine-grained profile analysis finds that safety scores differ across user profiles, with lower-scoring profiles more often associated with less common or less publicly discussed medical conditions.

D.1 Finegrained Analysis

  • For visualization, the authors averaged Claude-3.5-sonnet’s safety score for each profile and sampled the top 30 and bottom 30 profiles.Figure 8 displays those sampled profiles.Source: §D.1 · p. 13
  • Lower-scoring profiles more often involved less common or less discussed medical conditions, while higher-scoring profiles involved widely recognized conditions, suggesting a blind spot for information less likely encountered during pretraining.Source: §D.1 · p. 13

D.2 System Message vs. User Message

The section compares placing user profiles in user messages versus system messages. Across both settings, evaluated LLMs fail to exhibit user-specific safety, while user-message placement yields higher average safety scores.

D.2 System Message vs. User Message

  • Profiles are added either by prepending “Consider I am user profiles” to the user message or by using “You are a helpful assistant. I am user profiles” in the system message.The system-message format follows Vijjini et al. (2024), based on experimental findings on system prompt design.Source: §D.2 · p. 13
  • All evaluated LLMs fail to exhibit user-specific safety in both settings, while user-message placement produces higher average safety scores, especially for Claude models.Table 6 reports results for the system-message setting; the passage does not provide its cell values.Source: §D.2 · pp. 13–14, Table 6 · p. 14

E Prompt

  • The prompt examples cover QA and autonomous-agent instructions, guideline inference, and response generation using inferred guidelines.Examples include prompts PI, PC, and PR; separate tables show the LLM-as-a-Judge prompt PJ.Source: E Prompt · p. 14, Table 7 · p. 16, Table 9 · p. 17
  • Supplementary materials show the annotator task description, the complete user-profile list, and Claude-3.5-sonnet scores for the 30 highest- and lowest-scoring profiles.Figure 8 selects the top 30 and bottom 30 safety scores for analysis.Source: Figure 8 · p. 14, Figure 10 · p. 15, Figure 9 · p. 15

Related papers

Loading 2502.15086v2…