PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Safety-Utility Trade-Offs in Personalized Language Models

Anvesh Rao Vijjini, Somnath Basu Roy Chowdhury, Snigdha Chaturvedi

arXiv:2406.11107v3cs.CL

TL;DR

This paper examines how revealing a user's demographic identity affects LLM performance, a concern alongside the need for fair operation across demographics. It evaluates utility and safety and finds identity-dependent performance variation and safety–utility trade-offs across open- and closed-source models.

  • Problem

    LLMs can exhibit demographic bias, while personalization bias—performance variation when a user's identity is supplied—remains a focus for study.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    The paper evaluates personalization bias with a framework measuring LLM response utility and safety, and investigates it by supplying user identity in system prompts.

    Source: §1 · p. 2

  • Results

    Personalization bias appears across open- and closed-source LLMs, with safety–utility trade-offs varying by user identity.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors urge extra caution when deploying personalized LLMs and identify mitigation as an open problem.

    Source: §7 · p. 9, Limitations · p. 9

    The proposed strategies do not completely remove personalization bias without affecting utility.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As large language models (LLMs) become increasingly integrated into daily applications, it is essential to ensure they operate fairly across diverse user demographics. In this work, we show that LLMs suffer from personalization bias, where their performance is impacted when they are personalized to a user's identity. We quantify personalization bias by evaluating the performance of LLMs along two axes - safety and utility. We measure safety by examining how benign LLM responses are to unsafe prompts with and without personalization. We measure utility by evaluating the LLM's performance on various tasks, including general knowledge, mathematical abilities, programming, and reasoning skills. We find that various LLMs, ranging from open-source models like Llama (Touvron et al., 2023) and Mistral (Jiang et al., 2023) to API-based ones like GPT-3.5 and GPT-4o (Ouyang et al., 2022), exhibit significant variance in performance in terms of safety-utility trade-offs depending on the user's identity. Finally, we discuss several strategies to mitigate personalization bias using preference tuning and prompt-based defenses.

1 Introduction

The paper defines personalization bias as performance varying when an LLM is told a user's demographic identity, a setting that prior work has studied less extensively. It evaluates safety and utility together and reports identity-dependent trade-offs across a wide range of LLMs.

1 Introduction

  • Personalization bias occurs when revealing the user's identity affects responses, distinct from prompting for demographic-targeted content or adopting a demographic persona.The introduction describes the identity-revealing case as understudied and as the focus of this work.Source: §1 · pp. 1–2
  • The research question asks how LLM response utility and safety vary across user identities, studied by explicitly supplying identity in system prompts.The authors note that interaction-history and user-data fine-tuning are other personalization approaches, and that the best approach depends on the application.Source: §1 · p. 2
  • The evaluation framework measures utility as general reasoning capability and safety as the benignness of responses.The introduction also notes that identifying a user as a minor can steer responses away from adult or unsafe content.Source: §1 · p. 2
  • Personalization bias appears across open- and closed-source LLMs, with safety–utility trade-offs varying by user identity.The evaluation also finds significant performance variability across 31 identities in mathematical reasoning, general knowledge, and programming.Source: §1 · p. 2
  • The paper explores preference tuning and prompt-based defenses as strategies to reduce personalization bias.The introduction identifies mitigation strategies as a contribution alongside the evaluation framework and cross-model findings.Source: §1 · p. 2

2 Related Work

Prior research has examined LLM personalization, demographic bias, and persona bias, but has only rarely addressed personalization's safety implications. This work distinguishes itself by quantifying effects on both safety and utility and exploring mitigation strategies.

2 Related Work

  • Research on LLM personalization spans user-specific needs and applications, while only a few studies address personalization's safety implications.One cited contemporary study identifies stereotypical responses for some identity-conditioned tasks and proposes a decoding-stage strategy.Source: §2 · pp. 2–3
  • Prior work documents demographic and persona-related bias across language models, whereas this paper examines personalization's impact on both safety and utility.The related work describes findings spanning religious, age, gender, sexuality, political, and racial groups.Source: §2 · p. 3
  • Earlier work examined personalization-related identity effects such as refusal-rate variation, underperformance for some users, and sandbagging; this work quantifies effects across safety and utility and explores mitigations.The cited studies include Li et al. (2024b), Poole-Dayan et al. (2024), and Perez et al. (2023).Source: §2 · p. 3

3 Problem Setup

The setup evaluates personalization through system prompts that identify the user, using 31 identities across seven demographic categories. Utility is measured on knowledge, math, and programming tasks, while safety is measured on harmful-prompt datasets.

3 Problem Setup

  • The evaluation considers 31 user identities across seven categories, and uses system prompts to provide the model with the user's identity.The categories include disability, religion, race, gender, political affiliation, age, and sexuality.Source: §3 · p. 3
  • A prompt-selection framework seeks high identity imprinting and low identity leakage; the experiments use prompt [P6], which has the highest overall score in Table 2.Imprinting measures correctly identifying the user's identity, while leakage measures mistaking that identity as the model's own.Source: §3 · p. 4, Table 2 · p. 4
  • Utility is evaluated on MMLU, GSM8K, and MBPP, while safety is evaluated on DNA and StrongReject using unsafe prompts.Utility is reported as accuracy; safety is based on the fraction of unsafe prompts answered.Source: §3 · p. 4
  • The experiments use instruction-tuned open- and closed-source models and report zero-shot performance on both utility and safety datasets.The listed models include Llama, Mistral, Mixtral, GPT-3.5, and GPT-4o.Source: §3 · p. 4

4 Results & Findings

Personalizing LLMs with user identities produces substantial variation in both utility and safety, across identities and models. The paper quantifies this variability and reports that intersectional identities can differ from their individual components.

4.1 Bias from Personalization

  • GPT-3.5 utility varies significantly by identity on MMLU and GSM8K, with most identities lowering performance versus no identity; some low-performing identities recur across datasets.The figure's dotted line marks performance without an identity; ‘african’, ‘man’, and ‘straight’ tend to be among the lowest-performing identities in their categories.Source: §4.1 · p. 5, Figure 2 · p. 5
  • Providing an identity increases GPT-3.5 safety compared with no identity, but safety still varies by identity on DNA; StrongReject shows less variation.The authors attribute StrongReject's lower variation to its greater number of unsafe prompts and models' preference tuning to avoid answering them.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 Quantifying Personalization Bias

  • Personalization bias PB measures the deviation of performance for identities from their mean; lower PB indicates less bias, and performance may span safety and utility.The identity set U is user-defined according to the application.Source: §4.2 · p. 6

4.3 Personalization bias in Safety and Utility

  • Personalization affects both safety and utility across open-source models, whose personalization-bias scores range from 1.63 to 4.76.Identity effects vary by model: minor identities typically improve safety, while non-binary identities tend to reduce it.Source: §4.3 · p. 6, Figure 4 · p. 6
  • On GPT-3.5, intersectional identities can have safety-utility trade-offs that differ significantly from those of their individual identity components.The provided figure reports these comparisons for four intersectional identities.Source: Figure 5 · p. 7
  • Identity effects differ across models: gender identities lower safety for several models, while GPT-3.5 utility decreases with any identity and GPT-4o utility increases.Some identity categories vary along one axis while remaining constant along the other, such as age for Llama-2 70B and gender for GPT-3.5.Source: §4.3 · p. 7

5 Analysis

The analysis examines personalization across intersecting identities and traces how training stages relate to utility personalization bias. It finds identity-specific safety-utility differences and a recurring increase in bias during instruction tuning, while cautioning that the training-stage pattern does not establish causation.

5 Analysis

  • The analysis considers intersecting identity traits, such as a middle-aged Hindu man, because personalization may use multiple details about a user.The training-stage analysis evaluates three models on MMLU at their respective pretraining, instruction-tuning, and preference-tuning stages.Source: §5.1 · p. 7, §5.2 · pp. 7–8

5.1 Intersectional User Identities

  • Four selected intersectional identities produced safety-utility trade-offs different from their component identities, though safety was near the component-score average for three.The examples combine identity characteristics selected from those associated with the highest and lowest utility in Figure 4.Source: §5.1 · p. 7

5.2 Tracing the Source of Personalization Bias

  • Utility PB scores rose most during instruction tuning (1.13 to 1.25 and 1.54 to 2.21), while preference tuning slightly reduced bias.The authors caution that poorer pretraining performance may make pretraining bias appear low, so they cannot conclude instruction tuning increases bias.Source: Figure 6 · p. 7, §5.2 · p. 8

6 Mitigating Personalization Bias

The paper tests preference tuning and prompt-based defenses to reduce personalization bias. Both approaches reduce bias in the reported settings, but prompt defenses can lower overall utility and neither strategy fully removes bias.

6.1 Preference Tuning

  • For DPO, the authors randomly sampled an identity from their list for each preference pair and added it to the system prompt.They used an instruction-tuned Mistral-7B checkpoint that had not used system prompts during instruction tuning.Source: §6.1 · p. 8
  • With identity prompts, DPO reduced the base model's PB score from 3.60 to 3.17; DPO without identities reached 3.27.The identity-prompt setting is reported as an approximately 12% improvement over the base model, versus approximately 10% without identities.Source: Figure 7 · p. 8, §6.1 · p. 8

6.2 Prompt-based Defenses

  • All four prompt-based defense templates reduced PB scores, with D4 achieving approximately 49% improvement, but these defenses often reduced overall utility.The reported comparison uses Llama 3.1 8B and safety-utility trade-offs on MMLU and DNA.Source: §6.2 · p. 8, Table 4 · p. 9

6 Mitigating Personalization Bias

  • Neither preference tuning nor prompt-based defenses completely removes personalization bias.The authors identify further mitigation research as needed.Source: §6.2 · p. 9

7 Conclusion

The conclusion introduces personalization bias as performance variation associated with a user's demographic identity and presents a framework to evaluate it. Experiments find the bias across open- and closed-source LLMs, while proposed mitigation strategies do not eliminate it.

7 Conclusion

  • The paper defines personalization bias as LLM performance varying when the model receives the demographic identity of the user it is interacting with.It presents a framework for evaluating and quantifying this bias.Source: §7 · p. 9
  • Experiments find personalization bias across a wide range of open-source and closed-source LLMs.The authors state that the finding warrants extra caution when deploying such methods in production.Source: §7 · p. 9
  • The proposed methods reduce personalization bias but do not completely eliminate it, leaving mitigation an open problem for future research.Source: §7 · p. 9

Limitations

Personalization-bias estimates depend on the selected identity set and safety and utility tasks. The proposed strategies also do not bring bias to zero without affecting utility.

Limitations

  • Bias estimates depend on the chosen identity set and safety and utility tasks, which should match the LLM's intended use.Source: Limitations · p. 9
  • The proposed strategies do not reduce PB to zero without impacting utility, so complete mitigation remains unresolved.Source: Limitations · p. 9

Ethical Considerations

The paper defines a personalization-bias score to track performance variation across identities and leaves metric selection and trade-off judgments to domain experts. Its evaluation is bounded by English-language, USA-centric political-affiliation settings and a finite set of identity categories.

Ethical Considerations

  • The personalization-bias score measures variation in model performance across identities, while domain experts choose application-suitable metrics and weigh trade-offs.The authors report utility and bias separately, following group-fairness practice.Source: Ethical Considerations · p. 9
  • The experiments cover English and USA-centric political affiliations, using 54 identities while acknowledging that not all identities can be represented.The authors note that broad categories miss finer subgroups and that identities can cross discrete categories.Source: Ethical Considerations · p. 9
  • The study uses public resources, includes no human-subject annotations, and reports no anticipated direct negative applications of its evaluation framework.Source: Ethical Considerations · p. 9

A.1 Implementation Details

The study reports its model and evaluation setup, then describes prompt selection based on identity imprinting and leakage. The selected prompt balances those measures, although prompt wording affects leakage.

A.1 Implementation Details

  • The experiments use open-source checkpoints and API models, with three runs for open-source models but one run for API-based models because of cost.Open-source model inference used up to four 48GB Nvidia RTX A6000 GPUs; generated-token limits were 1,000 for utility and 100 for safety datasets.Source: §A.1 · p. 14
  • 12 user identities fell outside GPT-3.5’s no-identity MMLU confidence interval of 69.59 ± 2.4, measured over three runs.Source: §A.1 · p. 14

A.2 Prompting & Evaluation Details

  • MMLU and GSM8K answers are extracted by matching response text, MBPP code is tested, and DNA and StrongReject responses are scored by a harmful-response classifier.DNA and StrongReject contain 932 and 314 unsafe prompts, respectively.Source: §A.2 · p. 14

A.3 System Prompt Selection

  • Prompt selection favors high imprinting and low leakage: omitting “You are a helpful assistant” or adding identity-specific instructions raises leakage, and [P14] was selected.The authors also report that ordering matters, preferring a persona instruction before the user identity.Source: §A.3 · p. 15

B.1 System Prompt Ablations

The appendix tests how personalization bias varies with prompt wording, identity, personalization degree, task, and training choices. It reports distinct safety and utility effects, including trade-offs from prompt defenses and preference-tuning data.

B.1 System Prompt Ablations

  • Most tested prompt variants have similar bias scores, but Llama-3.1 8B’s [P11] prompt raises safety bias; its DNA safety is 62% for Hispanic users versus 85% for Caucasian users.The experiments also find personalization bias with different system prompts, rather than only one prompt.Source: §B.1 · p. 15, Figure 8 · p. 15, Table 7 · p. 17

B.2 Additional Identities

  • Additional identity tests find safety differences both across and within categories: political organizations are safer than names, while “janitor” is less safe than “engineer” for Llama-3.1 8B.The captions report occupation identities as the most prominent source of safety personalization bias for Llama-3.1 8B and Mistral 7B.Source: §B.2 · pp. 15–16, Figure 10 · p. 17, Figure 11 · p. 17

B.3 Degree of Personalization

  • Increasing personalization degree correlates with gradually higher safety and lower utility across sampled intersectional identities.The experiment samples 20 identities at each degree, from two to ten unique identities per persona.Source: §B.3 · p. 16, Figure 9 · p. 16

B.4 Mathematical & Programming Skills

  • Llama-3 70B shows significant identity-linked utility variation on GSM8K and MBPP, with PB scores similar to those measured using MMLU.GSM8K measures mathematical skills and MBPP programming skills.Source: Figure 12 · p. 17, §B.4 · p. 18

B.5 Comparison with Persona Bias

  • Persona assignment and user personalization affect different demographic groups; personalization can improve safety, whereas most assigned personas reduce it.Persona assignment reduced utility for some sexuality groups, while personalization reduced utility for some racial groups; sexuality-related personalization sometimes involved mistaking the user’s identity for the model’s persona.Source: §B.5 · pp. 18–19, Figure 13 · p. 18

B.6 Mitigation Strategies

  • Prompt defenses reduce personalization bias mainly along utility, at the cost of reduced utility; safety-bias scores mostly remain unchanged, and [D1] worsens safety bias.The defenses slightly improve safety overall, while decreasing imprinting rates without a reported correlation between imprinting and PB scores.Source: Figure 14 · p. 18, §B.6 · p. 19

B.7 Source of Personalization Bias (Safety); B.8 Influence of DPO Data

  • Safety PB is highest at pre-training and falls during instruction and preference tuning, while HH-based DPO yields lower PB than Orca-based DPO but also lower safety and utility.For utility experiments, the paper reports bias becoming more apparent in later training stages.Source: §B.7 · p. 19, §B.8 · p. 20, Figure 16 · p. 20

B.9 Example LLM Responses

  • The response examples illustrate two error types: incorrect assumptions and identity-based refusals, with the authors reporting no single error pattern across models and identities.Examples include excessive caution on arithmetic for a minor, withholding catheter information from a Muslim user, and identity-linked faulty reasoning by GPT-3.5.Source: §B.9 · p. 20, Table 10 · p. 21

B.10 Utility & Safety Bias

Across nine models, the reported figures show significant identity-linked variation in both utility and safety. The appendix provides model-specific plots across general-knowledge, math, and safety datasets.

B.10 Utility & Safety Bias

Related papers

Loading 2406.11107v3…