Source-linked AI summary
Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, Tushar Khot
TL;DR
The paper asks whether assigning socio-demographic personas alters LLM reasoning, even when personas are unrelated to the task. It studies 19 personas across 24 reasoning datasets and four LLMs, finding persona-induced biases that vary across models, personas, groups, and datasets and can be difficult to detect or mitigate.
Problem
The paper investigates whether persona assignment influences an LLM’s fundamental reasoning capabilities when the persona is tangential to the task.
Method
The study evaluates 19 diverse socio-demographic personas across 24 reasoning datasets covering multiple domains and four LLMs.
Results
Persona-induced bias appears across models, personas, socio-demographic groups, and datasets, manifesting as abstentions and increased reasoning errors; targeted prompt-based mitigation reduces bias but is not generally applicable.
Takeaways & Limitations
Persona assignment warrants careful consideration because it can expose stereotypical biases and produce unintended effects in LLM reasoning.
Takeaways & Limitations
The study uses non-exhaustive persona and dataset selections, with all datasets and prompts in English, leaving effects on other personas, tasks, and languages uncertain.
Abstract
from arXiv · showhide
Recent works have showcased the ability of LLMs to embody diverse personas in their responses, exemplified by prompts like 'You are Yoda. Explain the Theory of Relativity.' While this ability allows personalization of LLMs and enables human behavior simulation, its effect on LLMs' capabilities remains unclear. To fill this gap, we present the first extensive study of the unintended side-effects of persona assignment on the ability of LLMs to perform basic reasoning tasks. Our study covers 24 reasoning datasets, 4 LLMs, and 19 diverse personas (e.g. an Asian person) spanning 5 socio-demographic groups. Our experiments unveil that LLMs harbor deep rooted bias against various socio-demographics underneath a veneer of fairness. While they overtly reject stereotypes when explicitly asked ('Are Black people less skilled at mathematics?'), they manifest stereotypical and erroneous presumptions when asked to answer questions while adopting a persona. These can be observed as abstentions in responses, e.g., 'As a Black person, I can't answer this question as it requires math knowledge', and generally result in a substantial performance drop. Our experiments with ChatGPT-3.5 show that this bias is ubiquitous - 80% of our personas demonstrate bias; it is significant - some datasets show performance drops of 70%+; and can be especially harmful for certain groups - some personas suffer statistically significant drops on 80%+ of the datasets. Overall, all 4 LLMs exhibit this bias to varying extents, with GPT-4-Turbo showing the least but still a problematic amount of bias (evident in 42% of the personas). Further analysis shows that these persona-induced errors can be hard-to-discern and hard-to-avoid. Our findings serve as a cautionary tale that the practice of assigning personas to LLMs - a trend on the rise - can surface their deep-rooted biases and have unforeseeable and detrimental side-effects.
1 INTRODUCTION
The paper investigates whether assigning socio-demographic personas changes LLM reasoning on otherwise unrelated tasks. Across models, personas expose explicit and implicit stereotypical biases that can substantially reduce performance and resist simple prompt-based mitigation.
- Main findings: 96% of reasoning datasets showed biased assumptions for the physically-disabled persona, with a 33% average score drop versus the able-bodied persona.The model could reject disability stereotypes when asked directly yet make limiting assumptions when answering math questions under the persona.
- Main findings: 80% of ChatGPT-3.5 personas demonstrated bias, defined as a performance drop on at least one dataset.The reported bias was also observed across personas, datasets, and LLMs.
- Bias manifestations: Persona-induced bias appears explicitly through abstentions and implicitly through additional reasoning errors.For the physically-disabled persona in ChatGPT-3.5, abstentions account for 58% of errors.
- Mitigation and implications: Prompt-based mitigation strategies such as instructing models not to make stereotypical assumptions were ineffective or impractical.The paper therefore presents persona assignment as a potential source of difficult-to-detect and difficult-to-avoid reasoning failures.
2 METHODOLOGY & SETUP
The study assigns personas through system-prompt instructions and evaluates them with diverse personas and reasoning datasets in a controlled, zero-shot setup. Results are averaged across repeated runs and persona instructions to account for run-to-run variation.
- Persona assignment: Persona instructions are inserted into the system prompt to direct the LLM to embody a target persona.The system prompt provides high-level customization and conversation-wide context, while the user prompt contains the task.
- Persona assignment: Three minimal, linguistically diverse persona instructions were selected after testing whether they successfully induced the target persona.The effectiveness test required accurate responses about attributes explicitly stated in a comprehensive socio-demographic persona.
- Personas and datasets: The evaluation covers 19 personas across 5 socio-demographic groups, without claiming that the persona selection is exhaustive.The groups include race, gender, political affiliation, disability, and religion.
- Personas and datasets: 24 datasets measure knowledge and reasoning across mathematics, programming, science, medicine, law, sociology, ethics, and other domains.The suite includes 22 MMLU datasets, Sports Understanding, and MBPP.
- Models and evaluation: The primary model is ChatGPT-3.5 from June 2023, with additional experiments on later ChatGPT-3.5, GPT-4-Turbo, and Llama-2-70b-chat releases.Evaluation uses temperature 0, top-p 1, and a maximum token length of 1024.
- Models and evaluation: Reported accuracy averages nine runs for each persona-dataset combination: three runs for each of three persona instructions.Wilson confidence intervals are used with a significance level of 0.05.
3 FINDINGS
Persona assignment produces substantial, uneven reasoning disparities across 24 datasets, with especially large and statistically significant drops for some socio-demographic personas and groups.
- Most personas perform statistically significantly worse than the “Human” baseline, and some also fall below the “Avg. Human” baseline.
- 83%+ of datasets show statistically significant drops for the Phys. Disabled, Religious, and Atheist personas, while most other personas show at least one drop.
- 35%+ average accuracy drops occur for the Phys. Disabled and Religious personas, with individual drops reaching 64% and 69%.
- Bias varies sharply by dataset and persona: the Religious persona drops 69% on college chemistry but 11% on high school world history.
- Within-group disparities are largest for Disability and Religion, with significant differences on 23/24 and 19/24 datasets, respectively.
- ChatGPT-3.5 rates Phys. Disabled below Able-bodied across domains, Jewish above Christian on STEM categories, and Obama Supporters above Trump Supporters on moral scenarios.
- The authors characterize these analyses as preliminary and release approximately 1.5 million model generations for further investigation.
4 ANALYSIS
Persona-induced errors arise both from explicit abstentions and from incorrect reasoning when models answer. Shared-question comparisons show that the latter remains substantial and harder to detect.
- Abstentions account for 58% of Phys. Disabled errors, 35% of Atheist errors, and 49% of Religious errors.
- Politics, race, and gender personas have abstentions contributing less than 11% of their overall errors.
- The analysis compares persona accuracy on the exact same questions where both personas answer, excluding abstained questions from the shared set.
- A 39% accuracy drop remains for Trump Supporters versus Obama Supporters on college-maths despite restricting comparisons to shared non-abstained questions.
- These non-abstention disparities indicate implicit reasoning bias that is harder to discern than explicit abstentions.
5 PROMPT-BASED MODEL DE-BIASING
The study tests prompt-based strategies for mitigating persona-induced reasoning bias and finds that generic instructions are ineffective, while task-specific expertise reduces bias but is difficult to generalize.
- Task-agnostic instructions such as “don’t refuse” and “no stereotypes” have limited to no impact on persona-induced bias.The “treat human” instruction can even increase the bias relative to no mitigation for Atheist versus Religious personas.
- Figure 10 measures mitigation efficacy as percentage accuracy drop, with shorter bars indicating less bias.The no-mitigation condition represents the baseline bias without de-biasing.
- Task-specific expertise significantly reduces bias by reframing personas with expertise relevant to each dataset.Examples include “a physically disabled historian” for history tasks and “a physically disabled lawyer” for legal tasks.
- The expertise strategy has limited general applicability because it depends on well-defined task requirements that may be absent or evolving in open-ended conversations.Dynamic tasks may require diverse capabilities that are difficult to enumerate and add to personas.
6 DISCUSSION
The discussion highlights that persona-induced reasoning bias appears across models and can vary substantially with prompt choice, while producing practical risks for users and unresolved challenges for developers.
- Persona-induced biases occur across models, datasets, and personas, including bias in gender and race categories for Llama-2 and a 15% GPT-4-Turbo performance gap between political personas.
- Using one persona instruction in practice can produce substantially different bias levels from instruction-averaged research results.One instruction increased the average accuracy drop for the Phys. Disabled persona from 40% to 53%, and the MBPP drop for Obama Supporter versus Trump Supporter from 49% to 72%.
- Persona-induced bias can cause incorrect information, more errors in complex problem-solving and planning, subpar writing suggestions, and biased simulations.
- Simple instructions cannot fully mitigate persona-induced bias, motivating alignment efforts that address persona-induced responses and their associated biases.The authors release model outputs to support further alignment research.
7 RELATED WORK
The paper builds on research using personas for human-behavior simulation and on studies of model bias, but distinguishes itself by measuring how persona assignment affects reasoning performance.
- Prior work has used detailed personas and LLMs to simulate human behavior and replicate economic, psychological, social, and demographic viewpoints.
- Existing model-bias research has examined representations, task-specific models, and language-model behavior on tasks including coreference, entailment, and question answering.
- Earlier persona-bias studies focused primarily on toxic, harmful, or stereotypical generated text rather than reasoning performance.
- This work is presented as the first study to use persona assignment to examine its impact on LLM reasoning performance.
8 CONCLUSION
The conclusion reports a large-scale measurement of persona-induced reasoning bias and emphasizes its explicit and implicit forms, weak mitigation, and implications for users and developers.
- The study evaluates reasoning bias across 4 LLMs, 19 personas, and 24 datasets.
- Bias varies across models, personas, socio-demographic groups, and datasets, and can severely affect certain socio-demographics.
- Bias appears explicitly through abstentions and implicitly through differences in reasoning scores without overt stereotypical statements.
- Simple prompt-based mitigation techniques are insufficient, leaving a need for methods that better recognize and address persona-induced bias.
LIMITATIONS AND ETHICAL CONSIDERATIONS
The study’s persona and dataset selections are not exhaustive and favor majority and WEIRD categories, limiting coverage of other groups, tasks, and languages. The authors also warn that the methodology and findings could be misused to promote hatred or claims of demographic inferiority.
- The selected socio-demographic groups and personas favor majority and WEIRD categories, so biases in other groups are not fully covered.
- The evaluation uses English datasets and does not establish how persona-induced bias affects other tasks or languages.
- The methodology and findings could be misused to foster hatred or argue that certain demographics are inferior.
A DATASETS AND CATEGORIES
The study evaluates LLM reasoning across 24 datasets spanning broad subject domains, using standardized prompts and answer extraction across multiple task types. It also tests persona assignment across models with a single persona instruction and run.
- 24 datasets evaluate knowledge and reasoning across multiple subject domains, including MMLU, Python programming, and multi-hop sports understanding.The suite contains 22 MMLU datasets, MBPP, and Sports Understanding.
- 250 questions are randomly sampled from several larger datasets, while all evaluations use official test partitions.
- The 24 datasets are organized into 5 broad categories for analysis.
- Model outputs are parsed into answer options, Yes/No labels, or code, then scored against gold labels or unit-test success.
- Three additional LLMs are evaluated across 12 personas and 24 datasets using a single persona-assignment prompt and single run.The models are Llama-2-70B-Chat, GPT-4-Turbo, and ChatGPT-3.5-Nov.
C.1 LLAMA-2
Persona assignment introduces reasoning biases across the evaluated models, with effects varying by model, persona pair, group, and dataset. Llama-2 shows widespread significant degradation, GPT-4-Turbo smaller but persistent disparities, and ChatGPT-3.5-Nov. larger drops, while compound personas can sometimes mitigate bias.
- Llama-2: Llama-2 shows statistically significant accuracy drops versus the Human persona for 10 of 12 personas.Obama Supporter and Atheist are the only exceptions reported.
- Llama-2: 13 datasets show statistically significant performance differences between two genders in Llama-2, with pairwise bias varying substantially across datasets.The Man versus Non-Binary pair differs more than Able-Bodied versus Physically Disabled in the cited comparison.
- GPT-4-Turbo: GPT-4-Turbo shows minimal overall differences from the Human persona, but some persona pairs still exhibit relative changes from -10% to +20%.Bias in the Disability group is substantially reduced compared with ChatGPT-3.5.
- ChatGPT-3.5-Nov.: ChatGPT-3.5-Nov. has statistically significant drops for all 12 personas relative to Human, although some personas perform better than Human.Pairwise relative changes range from -300% to 100% in one comparison.
- Intersectionality: Compounding low- and high-bias personas produces different intersectional effects: mixed pairs fall between constituents, whereas two high-bias personas can score higher than both constituents.The authors hypothesize that the second persona reduces biased reasoning on examples where only one persona exhibits bias.
- Debiasing: Prompt-based debiasing strategies do not substantially reduce score differences, and ChatGPT-3.5 also produces abstentions based on stereotypical assumptions.
G.1 DATASETS WITH THE MOST BIAS
Bias is concentrated in particular dataset and persona-pair combinations, with Computer Science recurring across pairs and other leading datasets aligning with specific comparisons. Single persona instructions produce more pronounced accuracy drops than instruction-averaged results.
- Computer Science datasets consistently appear among the five most biased datasets across persona pairs.
- College Physics is a prominent source of disparity for the Atheist versus Religious persona pair, aligning with prevalent stereotypes.
- High school world history leads the Able-bodied versus Physically Disabled comparison despite Social Sciences ranking only third for that pair.The finding motivates finer sub-categorization within Social Sciences.
- Single-instruction evaluation produces pronounced biases and larger average accuracy drops for personas such as Physically Disabled and Atheist.
- Figure 24 compares percentage accuracy decreases for five persona pairs under a single instruction.