PaperlayerView exact revision on arXiv

Source-linked AI summary

When Personas Become Harmful: Guiding LLMs with Global Citizenship Values Alignment

Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Rada Mihalcea

arXiv:2610.33899v1cs.CL

TL;DR

Value-based personas can produce harmful responses on controversial topics, while current alignment challenges include respecting pluralistic values. The paper builds a red-teaming dataset and controllable alignment approach; it reports reduced harmfulness while maintaining faithfulness and Pareto dominance over baselines.

  • Problem

    The paper identifies harmful responses arising from legitimate value-based personas on controversial topics as a safety failure not captured by its stated contribution to existing safety approaches.

    Source: §1 · pp. 1–2

  • Method

    The authors construct persona-topic red-teaming data and train an alignment model that lets users balance safety and persona adherence.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    The controllable aligned model achieves Pareto dominance over baselines, while alignment reduces harmfulness and maintains faithfulness above a reasonable level.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings suggest a scalable approach to collecting data and aligning LLMs to respect value pluralism while maintaining safety.

    Source: §6 · p. 9

    LLM-judge evaluations may not fully align with human judgments, particularly for culturally sensitive content.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLMs are increasingly being used by users from diverse demographics with pluralistic values and distinct backgrounds. Maintaining safety while respecting value-laden personas is uniquely challenging because persona-conditioned requests are often not overtly malicious, unlike those in existing safety benchmarks. Alignment methods aimed at adversarial requests therefore may miss the harm that emerges from persona use, leaving models to either refuse legitimate value expression or blindly follow unsafe personas. We construct a dataset of 11k controversial topics paired with value-based personas mapping to value dimensions from the Schwartz Value Survey. Through an agentic pipeline that iteratively refines the topic and persona, we surface and collect harmful responses from well-aligned commercial LLMs with a more than 47% success rate. We use UNESCO's global citizenship principles as a minimal common ground for distinguishing valid value expressions from harmful behaviors, and introduce a controllable alignment model that lets users balance persona adherence and safety at inference time. The resulting model is safer and more persona-adherent than baselines on most samples.

1 Introduction

Value-based personas can support legitimate, pluralistic expression, yet following them on controversial topics can produce harmful content. The paper addresses this tension with red-teaming data and alignment that balances safety with persona adherence.

1 Introduction

  • Existing safety benchmarks primarily target malicious requests and miss harm arising from personal values expressed on controversial topics.This gap leaves models liable either to refuse legitimate value expression or to follow unsafe personas blindly.Source: §1 · p. 1
  • Persona-conditioned requests can move from legitimate value expression to harmful content, creating a trade-off between faithfully following a persona and acting safely.Refusing or contradicting legitimate value differences is also a failure mode.Source: §1 · pp. 1–2
  • The paper pairs controversial topics with value-based personas and iteratively refines both to surface harmful persona-driven responses.Personas are grounded in Schwartz Value Survey dimensions.Source: §1 · p. 2
  • UNESCO global citizenship principles provide a common ground for identifying harmful responses and generating alternatives without imposing demographic-specific values.The principles include human rights and respect for diversity.Source: §1 · p. 2
  • The authors introduce controllable alignment so users can specify their preference between safety and persona adherence.They report that the resulting models outperform unaligned and single-objective aligned baselines.Source: §1 · p. 2

2 Harmful Persona Red-teaming

The red-teaming pipeline constructs controversial topics and value-based personas, then iteratively refines them to elicit harmful persona-conditioned responses. It measures topic controversy, persona relevance, and harmfulness to guide refinement and select samples.

2.1 Seed Topics Collection

  • Seed topics come from debate-portal topics and global or local news, which are converted into stance-eliciting statements.The news collection includes events from eight countries and is translated into English for the red-teaming and alignment stages.Source: §2.1 · p. 3
  • The pipeline assigns topics to value domains and Schwartz Value Survey dimensions, then generates opposite personas grounded in survey items.The contrasting persona is created by negating and paraphrasing the initial persona.Source: §2.1 · p. 3

2.2 Refinement

  • The task is to elicit responses that faithfully represent a persona, differ in stance across opposite personas, and include at least one harmful output.The setup compares persona-conditioned outputs with a default output generated without a persona.Source: §2.2 · pp. 3–4
  • LLM scorers assess stance, persona faithfulness, and harmfulness, while sample-level measures summarize controversy and value relevance.Value relevance reflects whether a persona steers stance while remaining faithful to its assigned value.Source: §2.2 · p. 4
  • The attacker refines topics when controversy is low and personas when value relevance is low, saving samples with harmful responses or stopping when progress or iteration limits warrant it.The pipeline’s stated collection criteria require persona faithfulness and different stances for opposite personas, as well as a harmful output.Source: §2.2 · pp. 3, 5

3 Alignment with Global Citizenship Principles

The alignment framework revises harmful persona-conditioned responses using global citizenship principles and trains models on the revised outputs. It also treats safety and persona adherence as separate objectives, allowing users to control their balance at inference.

3.1 Alignment with Traditional Finetuning

  • Harmful responses are revised using the original response, the scorer’s harmfulness explanation, and global citizenship principles emphasizing cultural awareness without over-promoting particular values.The revised responses are rescored for harmfulness and faithfulness.Source: §3.1 · p. 5
  • Traditional alignment uses supervised finetuning on revised responses, followed by DPO preference optimization with revised responses preferred over default outputs.Default outputs serve as negative examples in the preference pairs.Source: §3.1 · p. 5

3.2 Controllable Alignment

  • Controllable supervised finetuning (CSFT) models safety and persona adherence as separate objectives and lets users specify their preferred balance at inference.Training samples receive discretized safety and adherence labels based on harmfulness and faithfulness scores.Source: §3.2 · p. 5
  • CSFT mixes default, persona-conditioned, and aligned responses to represent different safety and adherence levels.The authors describe default responses as high-safety, low-adherence; persona-conditioned responses as low-safety, high-adherence; and revised responses as moderately high on both.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate persona-driven red-teaming, traditional finetuning, and controllable alignment across safety and persona adherence. Controllable models improve the safety–adherence frontier, while traditional finetuning reduces harmfulness and retains relatively high persona faithfulness.

4.1 Harmful Persona Red-teaming Dataset

  • 47% of debate-portal samples and 55% of news-event samples elicited harmful responses before the iteration limit; in-context alignment reduced harmfulness scores below 0.3.The pipeline usually optimized topics in one to two iterations and spent more iterations refining personas.Source: §4.1 · p. 6, Table 4 · p. 6
  • Each of four independent-family judges labeled at least 70% of 1,472 collected red-teaming responses harmful.The judges used the same scoring rubric as the pipeline.Source: §4.1 · p. 6, Table 5 · p. 6
  • Annotators agreed with LLM labels on 93% of harmful responses and 89% of all responses, with Fleiss’ κ of 0.732.They rated persona coherence 4.50 and topic relevance 4.73 out of 5.Source: §4.1 · pp. 6–7

4.2 Alignment Results with traditional finetuning

  • 0.152 (Qwen3-8B) to 0.371 (Qwen3-14B) reductions in harmfulness accompanied persona faithfulness above 0.63 with finetuning; DPO showed mixed results across models.The default was safer but ignored the persona, while DPO did not reduce harmfulness and increase faithfulness simultaneously.Source: Table 2 · p. 6, §4.2 · p. 7
  • Universalism and Benevolence samples had the largest harmfulness reductions after alignment, while every category except Stimulation showed some lower persona faithfulness.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Power and Tradition personas elicited harm most reliably, while Universalism, Stimulation, and Self-Direction elicited less harm.Source: §4.2 · p. 7

4.3 Controllable Alignment

  • Output safety and persona adherence correlated with their specified settings; both metrics changed monotonically with controlling parameters, except near-flat safety response for Llama3.1-8B.Gemma3-12B had the highest correlation and best controllability; the full 5 × 5 parameter sweep showed the same monotonic pattern.Source: §4.3 · p. 8, Table 7 · p. 8, Figure 3 · p. 8
  • 27.5% improvement in frontier area over refined responses without controllable settings was achieved by Gemma3-12B; controllable models dominated or reached the frontier for 50%–64% of samples.For remaining samples, refined responses were neither safest nor most adherent, but could represent a better trade-off.Source: §4.3 · p. 8, Table 8 · p. 8

5 Related Work

Prior work uses personas to customize interactions and simulate human subpopulations, but persona conditioning can increase toxicity. Existing red-teaming methods differ in their prompts and search strategies; this paper targets harm emerging from value-based personas paired with controversial topics.

5 Related Work

  • Persona conditioning supports customized interactions and social simulation, but simple personas can also increase toxicity and harmful dialogue (Cheng et al., 2023; Deshpande et al., 2023; Wang et al., 2025).Prior uses include engaging dialogue, proxies for human subpopulations (Argyle et al., 2023), and simulated social interactions (Park et al., 2022; Guo et al., 2024a; Tran et al., 2025).Source: §5 · pp. 8–9
  • Red-teaming collects inputs that elicit harmful or less-preferred model responses, using approaches from crowd-sourced prompts to automated prompt or gradient-based search.The cited approaches include crowd-sourced prompts (Askell et al., 2021; Ganguli et al., 2022), harmful-response prompting (Perez et al., 2022), gradient search (Zou et al., 2023), and iterative prompt improvement (Chao et al., 2024; Liu et al., 2024, 2025).Source: §5 · p. 9
  • This paper uses iterative automatic red-teaming tailored to value-based personas; unlike jailbreaking, its unsafe behavior arises from persona–topic interaction rather than a malicious request.The authors describe their personas as more complex and their red-teaming as more sophisticated than earlier persona work.Source: §5 · p. 9

6 Conclusion

The paper introduces a dataset for harmful behavior from value-laden personas and an alignment framework grounded in global citizenship principles. Its results report reduced harmfulness while maintaining persona faithfulness, with controllable models achieving Pareto dominance over baselines.

6 Conclusion

  • The paper introduces a dataset for harmful behavior emerging from value-laden personas and an alignment framework grounded in global citizenship principles.The dataset and framework address persona-conditioned harm while aligning responses to global citizenship principles.Source: §6 · p. 9
  • Aligned models reduced harmfulness while maintaining faithfulness above a reasonable level, and generalized with reduced toxicity on RealToxicityPrompts and improved instruction-following on IFEval.The conclusion also reports that the red-teaming pipeline steered well-aligned commercial models, including GPT-4o-mini, toward harmful outputs.Source: §6 · p. 9
  • The controllable aligned model achieved Pareto dominance over baselines, supporting a scalable approach to alignment that respects value pluralism while maintaining safety.Source: §6 · p. 9

Limitations

The study's scope is limited by demographic coverage, English translation, and reliance on LLM judges. Its safety judgments and aligned outputs also have acknowledged limits.

Limitations

  • The eight-country local-news sample represents only a subset of global demographics.Source: Limitations · p. 9
  • Translating collected data into English leaves unexplored how language affects harmful persona-conditioned behavior.Source: Limitations · p. 9
  • LLM judges enable scalable evaluation but may diverge from human judgments, particularly for culturally sensitive content.Source: Limitations · p. 9
  • The chosen global-citizenship principles and static harm definition may introduce bias, and response revision does not guarantee harmless outputs.Source: Limitations · p. 9
  • The controllable setup does not prohibit users from selecting low safety together with low persona adherence, although it was not trained on such samples.Source: Limitations · p. 9

Ethical Considerations

The paper acknowledges dual-use risks from harmful persona–topic data and controllable settings, while arguing for restricted research access and a limited harm scope. It also explains why it adopts UNESCO principles despite concerns about their cultural universality.

Ethical Considerations

  • The study involves generating and using harmful content, which the authors identify as an ethical concern.Source: Ethical Considerations · p. 9
  • The dataset and pipeline could be used to discover persona–topic combinations that elicit harm, motivating restricted, research-only gated access.The authors say the dataset consists of controversial topics, personas, and harmful responses.Source: Ethical Considerations · pp. 9–10
  • The authors characterize the target as less harmful than jailbreak datasets because it concerns only the immediate response and defines harm through more nuanced, mild violations of global citizenship principles.Source: Ethical Considerations · p. 10
  • Low-safety control tokens are possible, but the authors argue they add no practical risk over the backbone model and make malicious intent more legible for auditing.They also argue that such responses are not intentionally harmful on both safety and persona-adherence dimensions.Source: Ethical Considerations · p. 10
  • UNESCO principles may be criticized for reflecting liberal democratic values, but the authors favor their minimal, higher-level framing over specific moral verdicts on contested topics.The principles can be changed in the pipeline, though that may affect attack success rates.Source: Ethical Considerations · p. 10

B Training settings

The appendix describes model training settings and the scope of human-annotation and score-distribution analyses.

B Training settings

  • SFT used a 2e-5 learning rate, linear scheduling, three epochs, and effective batch size 16; DPO used 5e-6, beta 0.1, cosine scheduling, and one epoch.Models used 8-bit quantization and LoRA adapters; the best checkpoint was selected by validation performance.Source: B Training settings · pp. 12–13
  • Table 9 lists the top three representative or official news sources for each country.Source: Table 9 · p. 13

C.1 Human Annotation Results

  • Table 11 reports agreement with LLM labels by sample type, inter-annotator agreement, and persona-quality ratings.Persona quality is measured on a 1–5 scale, with mean and share rated ≥4 reported.Source: Table 11 · p. 13, §C.1 · p. 13

C.2 Detailed Score Distribution

C.3 Analysis on Value Dimensions

The analysis examines which value dimensions elicit harm and how controllable alignment balances safety with persona adherence. It also documents the prompt templates used in the pipeline.

C.3 Analysis on Value Dimensions

  • Power-related personas elicit harm in 72% of cases, while Universalism and Stimulation personas do so in under 40%.The authors relate greater harm from dominance-oriented or rigid in-group commitments to tension with global citizenship principles.Source: §C.3 · p. 13

C.5 Controllable Alignment Detailed Results

  • Across the 5 × 5 control grid, each parameter increases monotonically with its own target, while Llama3.1-8B’s safety axis is nearly flat with a span of 0.06.Source: §C.3 · p. 14, §C.5 · p. 14

C.4 Controllable Alignment Qualitative Analysis

  • In the example, the balanced setting takes a clear stance while acknowledging the persona and softening its exclusionary, dominance-oriented language.High safety with low adherence instead gives a generic response, while high adherence with low safety adopts an aggressive, exclusionary viewpoint.Source: §C.4 · p. 14

C.5 Controllable Alignment Detailed Results

  • Models track targets reliably except at ADH=5, SAF=1; the Safer training distribution improves safety at low safety settings but lowers faithfulness.Its outcome is described as essentially similar to ADH=4&SAF=2.Source: Table 16 · p. 17, Table 17 · p. 17

D Prompt Templates

  • The prompt appendix lists templates for value-aligned and safe responses, CSFT training and inference, evaluation, agentic workflow, and persona dimensions.Source: D Prompt Templates · p. 18

Related papers

Loading 2610.33899v1…