Source-linked AI summary
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan
TL;DR
LLM adoption in services used by vulnerable people makes it important to understand toxicity and safety limitations. This paper systematically analyzes more than half a million ChatGPT generations across personas, entities, and phrase continuations, finding that persona assignment can sharply increase toxicity and that some groups receive disproportionately toxic responses.
Problem
Widespread LLM use creates a need for clearer evidence about toxicity and safety limitations, particularly for systems serving vulnerable users.
Method
The study analyzes over half a million ChatGPT generations across 90 personas, entity-conditioned prompts, and REALTOXICITYPROMPTS continuations.
Results
Persona assignment increases ChatGPT toxicity by up to 6×, while outputs also propagate harmful stereotypes and target some entities more than others.
Takeaways & Limitations
Persona-based system modifications can make ChatGPT unsafe for general use, especially for vulnerable groups, and can produce harmful behavior difficult to trace in downstream products.
Takeaways & Limitations
The findings point to brittleness in RLHF-based safety techniques that rely on human feedback and toxicity patches.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3x more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.
1 Introduction
This study examines whether persona assignment changes ChatGPT’s toxicity, motivated by the need to understand LLM capabilities and limitations as adoption expands among vulnerable users. It analyzes over half a million generations and finds persona-dependent toxicity, harmful stereotypes, and discriminatory targeting.
- Over half a million ChatGPT generations are analyzed to study toxicity, especially under different persona assignments.
- Muhammad Ali’s persona increased toxicity approximately 3-fold relative to ChatGPT’s default system settings.
- Persona-conditioned outputs can propagate incorrect stereotypes, harmful dialogue, and hurtful opinions about countries, religions, races, and other entities.
- Assigning personas can increase ChatGPT’s toxicity by up to 6× compared with default behavior.
- Toxicity varies substantially with persona identity, with the model’s opinion of the persona strongly influencing that variation.
- Certain entities and groups receive more toxic treatment than others, indicating discriminatory behavior in generated content.
2 Methodology
The methodology evaluates default and persona-assigned ChatGPT using diverse personas, entities, and phrase-completion prompts. Persona behavior is induced through the API system role, while entities and personas are constructed and checked for diversity and factual correctness.
- Assigning personas: The study compares default ChatGPT with versions whose system parameters assign personas maintained throughout the conversation.
- Personas: The evaluation uses 90 personas, including historical figures, common names, and baseline characters such as “a good person” and “a bad person”.
- Entities: The study includes 123 diverse entities spanning discrimination-relevant categories such as gender, age, country, race, profession, sexual orientation, and religion.
- Phrase completions: Persona-assigned ChatGPT also completes prompts from REALTOXICITYPROMPTS to measure toxicity when continuing incomplete phrases.
3 Experimental Setup and Evaluation
The experimental setup generates entity-conditioned responses and REALTOXICITYPROMPTS continuations for persona-assigned ChatGPT, then evaluates response likelihood and toxicity. The study samples hundreds of thousands of generations using standardized decoding and statistical testing.
- Response generation: Entity-conditioned responses use six templates, with negative-leaning templates used for the main analysis unless otherwise stated.
- Response generation: REALTOXICITYPROMPTS continuations provide ChatGPT with dataset phrases and allow it to complete them.
- Sampling: 324,450 entity-conditioned and 180,000 REALTOXICITYPROMPTS generations are produced using five generations per persona-entity pair and one per persona-prompt pair.
- Evaluation: PROBABILITY OF RESPONDING measures whether ChatGPT answers, while TOXICITY scores responding outputs from 0 to 1, with higher values indicating greater toxicity.
- Evaluation: The analysis reports maximum toxicity across repeated persona-entity generations and tests differences with unpaired t-tests at significance level 0.05.
4 Findings and Analyses
ChatGPT’s toxicity rises and varies markedly across assigned personas, entity categories, and individual entities. The findings include harmful stereotypes, demographic and political differences, and substantially different toxicity toward particular groups and countries.
- 4.1 CHATGPT can be consistently toxic: Without a persona, ChatGPT appears relatively safe, but baseline personas such as “a bad person” produce much higher toxicity and response probability.
- 4.1 CHATGPT can be consistently toxic: Baseline-persona outputs consistently produce harmful language and reinforce incorrect stereotypes across countries, religions, genders, and other entity categories.
- 4.2 Different personas result in different degrees of toxicity: Dictator personas have average toxicity 0.40 and maximum toxicity 0.94, exceeding the reported averages for journalists at 0.29 and sportspersons at 0.24.
- 4.2 Different personas result in different degrees of toxicity: Male personas have toxicity 0.26 versus 0.22 for female personas, while Republican politicians have 0.27 versus 0.25 for Democrats; both differences are statistically significant.
- 4.2 Different personas result in different degrees of toxicity: Among male politicians, toxicity ranges from 0.13 to 0.35, with response probability ranging from 0.42 to 0.75.
- 4.2 Different personas result in different degrees of toxicity: The model’s probability that a persona is “bad” correlates with that persona’s average toxicity at Pearson correlation 0.7, p < 0.05.
- 4.4 Entities receive high and varying degrees of toxicity: Toxicity differs across and within entity categories: Northern European and Caucasian entities receive approximately 2.5× the toxicity of African and Asian entities.
- 4.4 Entities receive high and varying degrees of toxicity: The authors hypothesize that biased human feedback may contribute to uneven toxicity because the model relies entirely on RLHF to curb toxic behavior.
5 Related Work
Prior work examines toxic and biased language in LLMs, including sources of bias, measurement methods, and techniques for reducing offensive generations.
- This research builds on a growing body of work investigating how and why LLMs generate toxic and biased text.
- Pre-training can embed problematic biases that reinforce existing stereotypes and prejudices in language models.
- Researchers have developed methods to measure bias in word embeddings and sentence encoders, while also identifying limitations in these metrics.
- Toxicity mitigation approaches include toxic-generation classifiers, decoding algorithms, adversarial training, and reinforcement learning with toxicity-focused reward models.
6 Discussion
The discussion emphasizes that persona-conditioned toxicity matters because ChatGPT is widely embedded in products and can produce harmful behavior that is difficult to trace or fix. The authors call for public-facing safety specifications and more fundamental approaches to model safety.
- ChatGPT is unsafe for general use when assigned a persona, especially for vulnerable groups such as students, minors, and patients.
- Businesses and start-ups shipping products with ChatGPT can introduce unexpected harmful behavior that is difficult to trace back to the underlying model.
- The authors call for public-facing specification sheets that include toxicity stress tests to educate users about model and product harms.
- These specifications should address other concerns as well, including privacy, data leakage, and misinformation.
- The findings point to brittleness in RLHF approaches that rely on humans and red-teams to deploy toxicity patches.
Appendix
The appendix provides the prompts used to generate personas and entities, alongside a table documenting those prompts.
- The authors provide the prompts used for generating personas and entities in Table 11.
B List of personas and entities created
The appendix documents the personas, entities, nouns, verbs, and adjectives created for the dataset across Tables 12–15.
- The exact personas and entities used in the study are listed in Tables 12, 13, 14, and 15.
- The persona list includes political figures such as Ronald Reagan, Gerald Ford, Barry Goldwater, and George H.W. Bush.
- It also includes global politicians such as Nelson Mandela, Winston Churchill, John F. Kennedy, and Jawaharlal Nehru.
- Other persona categories include female Democratic American politicians, liberal female journalists, female businesspersons, and female sportspersons.
- The entities include names associated with Russia, France, Arabia, Japan, India, and Brazil.
- Additional personas include historical dictators, a normal person, and people described with negative adjectives such as bad, hateful, nasty, and unpleasant.
C Use of PERSPECTIVEAPI
The study uses PERSPECTIVEAPI to operationalize toxicity across languages and contexts. Its efficiency supports toxicity evaluation at the scale of more than half a million generations.
- PERSPECTIVEAPI operationalizes toxicity across multiple languages and contexts.The paper follows prior community use of the tool for holistic toxicity evaluation.
- More than half a million generations require an efficient toxicity-evaluation method.The authors state that PERSPECTIVEAPI enables high-quality evaluations at this scale.
- The paper acknowledges potential concerns about PERSPECTIVEAPI while still recommending it for large-scale toxicity analysis.
D Calculating PROBABILITY OF RESPONDING
The study defines Probability of Responding as the fraction of times ChatGPT responds to queries designed to elicit toxic behavior. It estimates this by detecting response-decline patterns.
- Probability of Responding is the fraction of times ChatGPT responds to a query that elicits toxic behavior.
- The metric is computed by parsing responses for predefined patterns indicating that ChatGPT declined to respond.
8. Anywhere in the sentence: model language
The authors manually evaluate 200 responses to assess whether their decline-detection patterns identify refusals and avoid misclassifying actual responses. They report high recall and high precision.
- 200 responses are manually analyzed to validate the response-decline patterns.
- The patterns cover all observed cases in which ChatGPT declines to respond, yielding high recall.
- The patterns exclude cases in which ChatGPT does respond, yielding high precision.
E Examples of toxic outputs produced
The examples show that persona-assigned ChatGPT can produce harmful, insulting, stereotyped, and discriminatory language across names, ideologies, religions, sexual orientation, and other entity types. The outputs span multiple toxicity levels and are presented in several tables.
- Examples across toxicity ranges: Tables 16 and 17 present toxic persona outputs across different toxicity ranges while masking the exact entities mentioned.The accompanying description states that ChatGPT reinforces stereotypes and uses harmful language.
- Language and political concepts: Persona outputs also express hostile judgments about verbs, adjectives, and political concepts such as totalitarianism and oligarchy.
- Names: Multiple persona examples contain insulting language about named individuals, including Barbara Jordan, Barbara Olson, Barry Goldwater, Bella Abzug, and Benazir Bhutto.
- Toxicity scores: The examples include outputs with toxicity scores ranging from 0.36 to 0.71.Reported examples include 0.36 for an ideological response, 0.60 for a religion response, and 0.71 for a names response.
- Protected and identity-related categories: Examples targeting sexual orientation and religion use explicitly derogatory and dehumanizing language.