PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users

Manon Kempermann, Sai Suresh Macharla Vasu, Mahalakshmi Raveenthiran, Theo Farrell, Ingmar Weber

arXiv:2512.10687v2cs.AIcs.CY

TL;DR

Safety evaluations commonly assess universal risks, while people also use LLMs for high-stakes personal advice whose risks depend on user context. This exploratory study evaluates advice with context-blind and context-aware judges and examines realistic context disclosure. Context-aware judgments reveal a safety gap, and more realistic disclosure yields only slight improvement without closing it.

  • Problem

    Universal-risk evaluations leave underdeveloped the assessment of personalized risks arising when people seek high-stakes advice from LLMs.

    Source: §1 · p. 2

  • Method

    The exploratory framework compares context-blind and context-aware evaluation of advice using harm likelihood, severity, and safeguard adequacy.

    Source: §3.1.5 · p. 5, §5.1 · p. 9

  • Results

    A significant safety-score gap separates context-blind from context-aware evaluation, particularly for high-vulnerability users, and realistic context disclosure slightly improves scores without closing the gap.

    Source: §5 · p. 9

  • Takeaways & Limitations

    User-welfare evaluation requires rich user context and vulnerability stratification, with approaches distinct from universal-risk frameworks.

    Source: §6 · p. 10

    The evaluation focuses on single-turn advice, although real advice-seeking unfolds across exchanges that may influence safety outcomes.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Safety evaluations of large language models (LLMs) typically focus on universal risks like dangerous capabilities or undesirable propensities. However, millions use LLMs for personal advice on high-stakes topics like finance and health, where harms are context-dependent rather than universal. While frameworks like the OECD's AI classification recognize the need to assess individual risks, user-welfare safety evaluations remain underdeveloped. We argue that developing such evaluations is non-trivial due to fundamental questions about accounting for user context in evaluation design. In this exploratory study, we evaluated advice on finance and health from GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro across user profiles of varying vulnerability. First, we demonstrate that evaluators must have access to rich user context: identical LLM responses were rated significantly safer by context-blind evaluators than by those aware of user circumstances, with safety scores for high-vulnerability users dropping from safe (5/7) to somewhat unsafe (3/7). One might assume this gap could be addressed by creating realistic user prompts containing key contextual information. However, our second study challenges this: we rerun the evaluation on prompts containing context users report they would disclose, finding no significant improvement. Our work establishes that effective user-welfare safety evaluation requires evaluators to assess responses against diverse user profiles, as realistic user context disclosure alone proves insufficient, particularly for vulnerable populations. By demonstrating a methodology for context-aware evaluation, this study provides both a starting point for such assessments and foundational evidence that evaluating individual welfare demands approaches distinct from existing universal-risk frameworks. We publish our code and dataset to aid future developments.

1 Introduction

Universal safety evaluations address dangerous capabilities and behavioural propensities, but do not cover all risks when people use LLMs for high-stakes personal advice. This exploratory work examines how user context and vulnerability affect evaluation, and presents a preliminary framework for studying the challenge.

1 Introduction

  • High-stakes advice on finance, health, and careers is widely sought from assistants, but its personalized risks are not measured by current safety precautions.The authors note that AI risk frameworks recognize the importance of individual impacts, while practical tools for general-purpose LLMs remain largely absent.Source: §1 · p. 2
  • User Welfare Safety concerns whether advice is safe for individual users to act on, given their circumstances and vulnerabilities.The same advice can pose different context-dependent harms to different users, unlike universal safety risks.Source: §1 · p. 2
  • Meaningful user-welfare judgments must account for varied users, circumstances, and vulnerabilities, creating challenges for evaluation scalability and tractability.The study explores this methodological challenge through questions about evaluator access to context and users’ likely context disclosure.Source: §1 · p. 2
  • The paper offers exploratory insights and a preliminary evaluation framework, intended to inform future user-welfare evaluations rather than provide a final scalable solution.The framework is presented as a contribution to establishing a science of evaluations.Source: §1 · p. 3

2 Related Work

Prior work evaluates universal and adversarial risks, while socio-technical and regulatory frameworks emphasize context and individual impacts. The paper positions profile-conditioned safety evaluation as a way to address gaps between model-level benchmarks and risks in deployment.

2.1 The Current Paradigm: Evaluating Universal and Adversarial Harms

  • Capability and propensity evaluations test what models can do and how they behave, often under context-agnostic conditions that omit deployment and interaction patterns.Examples include red-teaming for harmful assistance and measuring tendencies such as sycophancy, deception, and bias.Source: §2.1 · p. 3

2.2 The Call for Socio-Technical Evaluations

  • International frameworks recognize contextual and people-centered AI risk through individual-impact classifications, socio-technical risk management, and deployment-dependent obligations.The passages cite the OECD Framework, NIST AI Risk Management Framework, and EU AI Act.Source: §2.2 · p. 3

2.3 Pathways Toward User-Welfare Evaluation: Precedents and Gaps

  • Demographic fairness audits identify population-level disparities, but generally do not assess downstream profile-conditioned risks for individuals seeking advice.The paper bridges this gap by quantifying differences between context-blind appropriateness scores and context-aware safety for concrete user profiles.Source: §2.3 · p. 3

3 Giving the Evaluator Context (RQ1)

This exploratory study compares safety judgments of LLM advice with and without user profiles, using health and financial prompts and profiles spanning vulnerability levels. Context-aware ratings reveal a larger safety gap for more vulnerable users, while precise magnitudes depend on evaluation choices.

3 Giving the Evaluator Context (RQ1)

  • The first study evaluates health and financial advice prompts with and without explicit user context to test the importance of rich context in safety judgments.Its framework and analysis are exploratory and intended to uncover potential issues and guide future research.Source: §3 · p. 3, §3.1 · p. 4

3.1.1 Domain and Theme Selection

  • The study uses health and personal-finance advice because safety in these high-stakes domains depends on individual circumstances.The selected themes span common decisions with different time horizons and are intended as a diverse, non-exhaustive set of scenarios.Source: §3.1.1 · p. 4

3.1.2 User Prompt Construction

  • Researchers synthesized standardized, context-free questions from advice-seeking Reddit posts, selecting twelve high-stakes prompts across the two domains.The pipeline used LLM classification and synthesis; two researchers selected the final questions by consensus, and the authors note possible model-specific bias in generation.Source: §3.1.2 · pp. 4–5

3.1.3 User Profile Construction

  • Healthcare and financial professionals created low-, medium-, and high-vulnerability profiles, with high vulnerability defined by compounding risks across dimensions.The team reviewed submissions and selected three profiles per vulnerability level and theme.Source: §3.1.3 · p. 5

3.1.4 LLM Responses

  • The evaluation samples GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro, using temperature 1.0 to elicit varied responses.The authors describe the three models as a snapshot of the current generative AI landscape.Source: §3.1.4 · p. 5

3.1.5 Evaluation Framework: An Exploratory Use of LLM-as-Judge

  • The exploratory framework compares judges rating responses for a generic user with judges given the corresponding user profile, using the same scoring logic.The prompts were iteratively refined, but the process was not a formal, large-scale validation against human annotations.Source: §3.1.5 · pp. 5–6

3.2 Findings

  • Context provision systematically changes safety evaluations across both domains and all three evaluated models.The authors caution that precise magnitudes depend on the judge prompt, rating scale, and evaluator model, while the directional pattern is consistent across six model-domain conditions.Source: §3.2 · pp. 6–7
  • For high-vulnerability users, context-aware ratings drop two points—from safe (5/7) context-blind to somewhat unsafe (3/7)—across models and domains.For medium-vulnerability users, health advice is rated about one point lower with context; low-vulnerability users receive context-aware ratings of 5–6/7.Source: Figure 4 · p. 6, §3.2 · p. 6

4 Towards Realistic User-Prompts (RQ2)

The second study tested whether adding realistic user context to prompts could narrow the safety gap between context-blind and context-aware evaluation. It found only limited improvement, leaving a gap for high-vulnerability users.

4 Towards Realistic User-Prompts (RQ2)

  • The second study asked whether user-reported or professionally prioritized context in prompts could narrow the gap found in the first experiment.It modeled context disclosure using ranked factors added to baseline prompts.Source: §4 · p. 7, §4.3 · p. 8

4.1 Methods; 4.2 Response Collection and Evaluation

  • The study expanded the first experiment’s prompt set by adding user-context factors, then evaluated the resulting responses with the prior methodology.Prompts contained one, three, or five factors to model low, medium, and high disclosure; five was a pragmatic upper bound.Source: §4.1 · p. 7, §4.2 · p. 8

4.1.1 Professional Relevance Ranking; 4.1.2 User Likelihood Ranking

  • Professional relevance rankings came from 10 professionals per theme, while user-likelihood rankings came from a separate US-resident cohort of regular LLM users (N=100 per theme).Both cohorts’ rankings were aggregated by Borda Count; user rankings reflect stated preferences, which may differ from actual behavior.Source: §4.1.1 · p. 7, §4.1.2 · p. 7

4.1.3 From Rankings to Natural Language Prompts

  • Context factors were converted into first-person clauses and combined into natural-language prompt introductions, with five phrasings per context combination to reduce wording effects.The clauses were generated programmatically, then appended to the original plain question.Source: §4.1.3 · pp. 7–8

4.3 Findings

  • High-vulnerability users’ scores rose from somewhat unsafe (3/7) to almost moderately safe (4/7), but a one-point gap remained; Figure 5 compares scores across disclosure levels.The authors conclude that even realistic prompt disclosure does not remove the need to assess responses against holistic user profiles.Source: Figure 5 · p. 8, §4.3 · pp. 8–9
  • Across relevance-ordered and likelihood-ordered disclosure, adding one, three, or five context factors significantly narrowed the safety gap for medium- and high-vulnerability users (α=0.05).The study also reports a slight difference between health and financial advice.Source: §4.3 · p. 8
  • For low-vulnerability users, context-aware scores stayed near 5/7 for health and 5–6/7 for finance; medium-vulnerability health scores were 4–5/7, about one point below context-blind scores.Medium-vulnerability users showed minimal improvement as context increased.Source: §4.3 · p. 8
  • At five factors, professional and user rankings shared a mean 4.50 factors for health themes and 3.75 for financial themes, with no significant score differences by ranking order.The authors report that user-stated likely disclosures nearly matched factors professionals considered most relevant.Source: §4.3 · p. 9

5 Discussion

The discussion finds that context-aware evaluation reveals risks hidden by context-blind scoring, especially for more vulnerable users. It presents the approach as exploratory and identifies validity, coverage, and behavioral-realism limits.

5.1 Methodological Viability

  • Across experiments, context-aware evaluation revealed risks hidden by context-blind evaluation, with safety decreasing directionally as user vulnerability increased.The authors recommend vulnerability stratification and context-aware evaluators for future user-welfare evaluations.Source: §5.1 · p. 9
  • The evaluation strategy combined harm likelihood and severity with safeguard adequacy, producing resilient scores across domains, vulnerability levels, and models.The authors regard this as evidence that adapting nuanced risk assessment to the LLM-as-judge reasoning process is viable.Source: §5.1 · p. 9

5.2 Current Limitations and Considerations for Future Work

  • The LLM-as-judge was not validated against expert human judgments, so definitive safety claims about specific models warrant expert validation.Iterative prompt refinement and consistent patterns across conditions provide face validity, not expert validation.Source: §5.2 · p. 9
  • Coverage was limited to four themes per domain, six questions per theme, and three canonical profiles per vulnerability level, not a representative sample of users or contexts.The authors note that expanding user-scenario combinations may become intractable and recommend prioritizing high-vulnerability users in high-stakes situations.Source: §5.2 · p. 9
  • The second study modeled stated disclosure preferences and prepended context to Reddit-inspired prompts, which may not reflect authentic advice-seeking behavior.The authors call for large-scale study of how people actually seek advice from LLMs.Source: §5.2 · p. 9
  • The evaluation did not address multi-turn interaction dynamics or context accumulated through provider memory features.The authors identify real-world user-system interaction data as relevant to studying these settings, while noting privacy considerations.Source: §5.2 · pp. 9–10

5 Discussion

  • The authors describe the work as exploratory groundwork for vulnerability-stratified evaluation, with possible relevance to future DSA compliance research if regulatory frameworks extend to AI services.They condition this implication on services reaching VLOSE designation and the applicable regulatory framework.Source: §5.2 · p. 10

6 Conclusion

The conclusion argues that context-dependent risks call for evaluation approaches distinct from universal-risk frameworks. It presents vulnerability-stratified, context-aware evaluation as a starting point for assessing which risks models pose to whom.

6 Conclusion

  • Consistent safety gaps for high-vulnerability users across models, domains, and disclosure strategies support evaluation approaches distinct from universal-risk frameworks.The authors present their framework as a starting point for covering user-specific risks alongside model capabilities and propensities.Source: §6 · p. 10

A Selected Themes within Health and Finance

The appendix tables identify the health and finance themes, demographic/context factors, and source subreddits and search keywords used in the study.

A Selected Themes within Health and Finance

  • Table 1 identifies the selected health and finance domains and themes.The supplied table passage gives its title but not its entries.Source: Table 1 · p. 14

B Demographic Factors

  • Table 2 lists demographic/context factors used to construct user profiles.The supplied table passage gives its title but not its entries.Source: Table 2 · p. 14

C Source Subreddits and Keywords

  • Table 3 lists source subreddits and search keywords for each theme.The supplied table passage gives its title but not its entries.Source: Table 3 · p. 15

D Example Evaluation Questions

The appendix gives examples and prompts used to build evaluation questions and assess responses with or without user context. Its case-study overview contrasts generic and profile-specific safety scores.

D Example Evaluation Questions

  • Table 4 presents selected evaluation questions across domains and themes.The supplied table passage gives its title but not its entries.Source: Table 4 · p. 16

E Example User Prompts Across Context Levels

  • Table 5 presents prompt variants across context-disclosure levels for the Debt Management theme.The table title specifies relevance ranking as the selection basis.Source: Table 5 · p. 16

F.1 Prompt 1: Advice-Seeking Classification

  • The advice-seeking classifier labels Reddit posts as advice-seeking or not using criteria such as requests for guidance or help.Its output is constrained to one of the two labels.Source: §F.1 · p. 17

F.2 Prompt 2: Theme Classification

  • A second classifier assigns advice-seeking posts to one predefined theme, with “Other” available when none fits well.It returns the exact theme name.Source: §F.2 · p. 17

F.3 Prompt 3: Question Synthesis

  • The synthesis prompt generates 12 plain-language questions from Reddit examples, emphasizing high-stakes, accessible, context-dependent situations.It excludes expert-level or niche questions and asks for diverse, neutral questions without personal details.Source: §F.3 · pp. 17–18

G.1 Prompt 4: Context-Blind Safety Evaluation

  • The context-blind evaluator scores advice without an explicit user profile, using three dimensional scores and an overall 1–7 safety score.The prompt asks it to consider extreme scores before choosing a middle-range score.Source: §G.1 · p. 19

G.2 Prompt 5: Context-Aware Safety Evaluation

  • The context-aware evaluator receives the user profile and scores safety specifically for that person, also using three dimensions and an overall 1–7 score.Its scoring instruction explicitly frames judgments around the user's circumstances and vulnerabilities.Source: §G.2 · pp. 20–21

H Case Studies of Context-Dependent Harm

H.1 Case Study 1: Health Advice (Diet & Weight Management)

The two case studies illustrate how generally plausible advice can pose different risks for users with particular vulnerabilities. The health example concerns weight-loss advice given to a teenager recovering from anorexia; the financial example concerns investment advice given to a user with unstable income and high-interest debt.

H.1 Case Study 1: Health Advice (Diet & Weight Management)

  • The health case profile is a 17-year-old recently weight-restored from anorexia with fragile mental health and ongoing therapy.The prompt asks about losing weight without a gym or expensive diet plans, and the response provides calorie restriction, tracking, meal, and exercise guidance.Source: §H.1 · pp. 23–24

Mechanism of Harm

  • The authors say weight tracking and a moderate calorie deficit are contraindicated for eating-disorder recovery and may trigger relapse.The advice included weighing 1–2 times weekly, measuring waist every two weeks, and a generic healthcare-professional disclaimer after detailed instructions.Source: Mechanism of Harm · p. 24, §H.1 · p. 24

H.2 Case Study 2: Financial Advice (Small Investments)

  • The financial case profile has $18,000 annual income from irregular gig work, $3,500 in high-interest credit-card debt, and no savings.The user asks whether to invest or save a small inheritance.Source: §H.2 · p. 25
  • The financial advice recommends keeping near-term funds safe and investing long-term funds after covering emergency savings and high-interest debt.It also suggests instruments including T-bills, CD ladders, and retirement accounts.Source: §H.2 · pp. 25–26

Mechanism of Harm

The paper describes how standard advice can conflict with a user's economic vulnerability and outlines appendix procedures for encoding user context and ranking context factors.

Mechanism of Harm

  • For a user with high-interest debt, no savings, and irregular income, the authors say advice to park inheritance cash in savings can be financially damaging.They also note that complex instruments may create a barrier for a user with low technical literacy and financial stress.Source: Mechanism of Harm · p. 26

I.1 Prompt 6: Clause Normalization

  • The clause-normalization prompt turns demographic attributes into short first-person clauses while requiring all input details and forbidding added information.The prompt's examples cover debt, family situation, and income.Source: §I.1 · p. 27

I.2 Prompt 7: Context Variant Generation (3 Factors)

  • The three-factor variant prompt creates five stylistically distinct, neutral background descriptions from the same three clauses.It requires information preservation and prohibits adding facts or omitting details.Source: §I.2 · pp. 27–28

I.3 Prompt 8: Context Variant Generation (5 Factors)

  • The five-factor variant prompt likewise generates five neutral descriptions while preserving all five source clauses.The variants include direct, conversational, grouped, concise, and integrated-narrative styles.Source: §I.3 · pp. 28–29

J Relevance and Likelihood Rankings

  • Tables 6 and 7 report the top-five context factors by theme for relevance and likelihood rankings, respectively.Their captions specify descending importance and Kendall’s W as the within-theme annotator-agreement measure.Source: Table 6 · p. 30, Table 7 · p. 30

K Statistical Analyses

The analyses compare context-blind and context-aware safety scores, and test whether the safety gap decreases from baseline to five disclosed context factors. Wilcoxon tests identify which comparisons are statistically significant.

K Statistical Analyses

  • Table 8 compares context-blind and context-aware baseline safety scores by domain, model, and vulnerability profile, using two-sided Wilcoxon tests.† marks comparisons that are not significant at p>0.05.Source: Table 8 · p. 31, K Statistical Analyses · p. 31
  • Table 9 compares baseline and level-five safety gaps; lower values indicate smaller gaps, and bold marks the larger gap within each row.The one-sided test evaluates whether the baseline gap exceeds the level-five gap; † marks non-significance at α=0.05.Source: Table 9 · p. 32, K Statistical Analyses · p. 32
  • For the baseline-versus-level-five gap comparisons, all comparisons except those marked non-significant show statistically significant reductions.The tests are one-sided Wilcoxon signed-rank tests at α=0.05.Source: K Statistical Analyses · p. 32, Table 9 · p. 32

L Additional Graphs for RQ2 Findings

The RQ2 graphs show context-blind and context-aware safety scores as the number of prompt context factors increases, stratified by advice domain and user vulnerability. The displayed Gemini 2.5 Pro and GPT-5 graphs follow patterns described as similar across models.

L Additional Graphs for RQ2 Findings

  • The Gemini 2.5 Pro graphs plot context-blind and context-aware safety scores across increasing prompt context factors, split by health versus financial advice and user vulnerability.Health advice appears above financial advice; each domain is stratified by low, medium, and high vulnerability.Source: Figure 6 · p. 33
  • The GPT-5 graphs use the same context-factor, advice-domain, and vulnerability comparisons as the Gemini 2.5 Pro graphs.The authors report similar patterns across all models.Source: L Additional Graphs for RQ2 Findings · p. 32, Figure 7 · p. 33

Related papers

Loading 2512.10687v2…