PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Model Biases in Persona-Steered Generation

Andy Liu, Mona Diab, Daniel Fried

arXiv:2405.20253v1cs.CL

TL;DR

Prior work on LLM-generated opinions has focused on multiple-choice settings or one-dimensional personas, leaving open how models represent multifaceted personas in open-ended generation. The paper builds a persona-steered statement-generation task from human survey data and evaluates model steerability. It finds lower steerability for incongruous personas, greater steerability but less diversity among evaluated RLHF models, and that multiple-choice responses do not necessarily predict open-ended steerability.

  • Problem

    Prior work has mainly evaluated LLM opinions through multiple-choice questions or one-dimensional personas, leaving open how well models represent multifaceted viewpoints in open-ended generation.

    Source: §1 · p. 1

  • Method

    The authors construct demographic-and-stance personas from Pew survey data and evaluate generated statements for support of the prompted stance.

    Source: §2.1 · p. 3, §2.2 · p. 4

  • Results

    9.7% difference in steerability separates congruous from incongruous personas; evaluated RLHF models were more steerable but less diverse, and multiple-choice responses did not necessarily predict open-ended steerability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support further study of open-ended generation to understand opinion biases and steerability across a more diverse range of personas.

    Source: §1 · p. 2, §5 · p. 9

    The study evaluates single-statement generation, which may differ from interactive downstream LLM simulations.

    Source: §6 · p. 9

Abstract

from arXiv · show

The task of persona-steered text generation requires large language models (LLMs) to generate text that reflects the distribution of views that an individual fitting a persona could have. People have multifaceted personas, but prior work on bias in LLM-generated opinions has only explored multiple-choice settings or one-dimensional personas. We define an incongruous persona as a persona with multiple traits where one trait makes its other traits less likely in human survey data, e.g. political liberals who support increased military spending. We find that LLMs are 9.7% less steerable towards incongruous personas than congruous ones, sometimes generating the stereotypical stance associated with its demographic rather than the target stance. Models that we evaluate that are fine-tuned with Reinforcement Learning from Human Feedback (RLHF) are more steerable, especially towards stances associated with political liberals and women, but present significantly less diverse views of personas. We also find variance in LLM steerability that cannot be predicted from multiple-choice opinion evaluation. Our results show the importance of evaluating models in open-ended text generation, as it can surface new LLM opinion biases. Moreover, such a setup can shed light on our ability to steer models toward a richer and more diverse range of viewpoints.

1 Introduction

Prior work has examined LLM opinions mainly through multiple-choice questions, while persona-steered generation raises questions about how well models represent multifaceted human viewpoints. This paper studies persona steerability in open-ended generation and finds lower steerability for incongruous personas, alongside differences related to fine-tuning and limited predictive value from multiple-choice responses.

1 Introduction

  • The study addresses whether multifaceted personas can elicit viewpoints that reflect the distribution of people who fit them, rather than oversimplified demographic caricatures.The authors note that poor representation of some stances or demographics could surface new biases when LLMs simulate individuals.Source: §1 · p. 1
  • The authors construct a persona-steered statement-generation task using Pew survey stances about American political, gender, and race-related trends.They combine demographic traits and stances, using relative survey response rates to identify congruous and incongruous personas.Source: §1 · p. 2
  • LLMs were 9.7% less steerable toward incongruous than congruous personas, and RLHF fine-tuning was associated with higher steerability but up to a 58.2% decrease in semantic diversity.RLHF models showed especially large steerability increases toward stances associated with political liberals and women; multiple-choice opinions predicted open-ended steerability only 51.5% of the time.Source: §1 · p. 2
  • The authors argue that further research should examine steerability across diverse personas and support richer, more nuanced representations of human opinions.They also note that the findings motivate study of LLMs in more interactive simulation settings.Source: §1 · p. 2

2 Methods

The methods construct demographic-and-stance personas from Pew survey data, generate statements from those personas, and evaluate whether the statements support the prompted stance. Additional metrics characterize how persona-steered outputs differ and vary.

2.1 Persona-Steered Generation Setting

  • Personas combine a demographic with a relevant stance, sampled from Pew survey data; relative response rates identify incongruous cases, and both stance-first and demographic-first prompt orders test wording sensitivity.The study samples six demographics in three pairs and twenty relevant, highly divergent stances per pair, alongside a no-demographic base persona for each topic.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 Steerability Evaluation

  • GPT-4 evaluates each generated statement against the prompted and opposing stances, and steerability is the percentage judged to support the persona’s stance.Crowdworkers label a subsample using the same stance-choice task, with rationales collected when their judgments disagree with the model’s.Source: §2.2 · p. 4

2.3 Additional Metrics

  • Additional metrics characterize individuation, exaggeration, entailment diversity, and semantic diversity in generated statements.Entailment diversity uses pairwise entailment-versus-contradiction scores, while semantic diversity averages pairwise embedding cosine distances.Source: §2.3 · pp. 4–5

3 Results and Discussion

The results show that model steerability varies by persona congruity, stance, and fine-tuning method, while multiple-choice responses weakly predict open-ended steerability. Incongruous personas reduce demographic exaggeration but also semantic diversity, and the authors warn that these biases may limit representation and usefulness.

3.1 GPT-4 is a Strong Proxy for Human Evaluation [RQ4]

  • GPT-4 labels strongly correlate with human labels across evaluated models and stances, supporting their use as a proxy for human steerability judgments in this task.For 1,200 generated statements, GPT-4 and human annotations had an F1 score of 96.3% and Cohen’s Kappa of 0.808; disagreements often involved weak stance relevance or statements mentioning both stances.Source: §3.1 · p. 5

3.2.1 Fine-Tuning Improves Steerability, but Stances Benefit Unequally [RQ2]

  • Llama models fine-tuned with RLHF and DPO average 90.3% and 87.8% steerability, respectively, versus 81.1% for SFT, with RLHF gains especially strong for stances associated with women and political liberals.Models are most steerable toward political stances and least toward gender-related stances; Figure 3 groups Llama and Tulu results by fine-tuning method.Source: Figure 3 · p. 5, §3.2.1 · p. 6

3.2.2 Steerability by Stance is Not Predictable from Model Survey Response Rates [RQ3]

  • Multiple-choice responses weakly predict open-ended steerability: models favored their survey-identified stance only 51.5% of the time, and the relationship had R2 = 0.018 (p = 0.033).The 51.5% rate was only slightly better than random chance.Source: §3.2.2 · p. 6

3.3.1 All Models are Worse at Representing Incongruous Personas

  • Models are 9.7% more steerable toward congruous than incongruous personas, with the largest effect for political personas and significant differences across all models.Table 2 reports average steerability for default-demographic, congruous, and incongruous personas across politics, race, and gender.Source: §3.3.1 · p. 6, Table 2 · p. 6
  • GPT-3.5-Turbo is less affected by congruity for race and gender than for politics, but its high steerability accompanies a narrower representation of personas.The diversity metrics in Table 3 show the narrower image of a multifaceted persona.Source: §3.3.1 · pp. 6–7, Table 3 · p. 7

3.3.2 Steering Towards Incongruous Personas Reduces Diversity and Susceptibility to Caricature

  • Generations from incongruous personas are less prone to demographic exaggeration and can be distinguished from default-demographic generations, but semantic diversity decreases.The authors note that stereotypes can still persist when models assume a persona supports stances common to its demographic.Source: §3.3.2 · p. 7, Table 4 · p. 7

3.4 Differences in Steerability Could Lead to Social Harms

  • The authors warn that reduced steerability for incongruous personas may impede nuanced identity representation, perpetuate stereotypes, and limit the usefulness of LLM simulations.They also state that aligning models to certain values may limit steerability toward stances that do not reflect those values.Source: §3.4 · p. 8

4 Related Work

Prior work studies persona-steered generation, opinion bias, and viewpoint steering across different tasks. This paper’s related work situates its focus on open-ended generation alongside multiple-choice opinion evaluation and downstream applications.

4.1 Persona-Steered Generation

  • Prior persona-steered generation work examines caricature, persona consistency, simulated human behavior, and toxicity, while this paper centers steerability and persona congruity.The cited work also considers model scale and fine-tuning.Source: §4.1 · p. 8

4.2 Evaluating Biases in LLMs’ Expressed Opinions

  • Opinion-bias studies evaluate demographic, national, cultural, age, and human-like response alignment, largely through surveys or clinical tests.This paper distinguishes its focus on open-ended generation from multiple-choice responses.Source: §4.2 · p. 8
  • Other work examines viewpoint steering in downstream tasks including annotation, simulation, charitable donation, and political-perspective preservation in news summaries.Studies vary base models and fine-tuning, and report differences between open-ended answers and multiple-choice responses.Source: §4.2 · p. 8

5 Conclusion

The paper evaluates persona-steered statement generation across political, race, and gender stances. It finds lower steerability for incongruous personas and greater steerability—but narrower persona views—in evaluated RLHF fine-tuned models.

5 Conclusion

  • Models are more steerable toward congruous than incongruous personas across more than sixty political, race, and gender stances.Evaluated RLHF fine-tuned models are more steerable, especially toward stances associated with political liberals and women, but may trade diversity for steerability.Source: §5 · p. 9
  • Related multiple-choice model behavior does not necessarily predict steerability in open-ended generation.The authors argue that stronger fine-tuning alone cannot necessarily remove these biases.Source: §5 · p. 9
  • Models’ sensitivity to persona congruity suggests stereotypical views of a demographic or stance can influence generation.The authors suggest such biases may persist in more complex simulations and call for study in interactive settings.Source: §5 · p. 9

6 Limitations

The study’s findings are bounded by its single-statement generation setup and binary simplification of complex demographic and political differences.

6 Limitations

  • The study evaluates single statements, which may differ from interactive downstream LLM simulations.It does not evaluate GPT-4, partly to avoid bias in evaluation; the authors instead examine open-source models to study scale and fine-tuning effects.Source: §6 · p. 9
  • Reducing complex political issues to two opposing stances simplifies analysis of congruous and incongruous personas.The authors note that binary representations of demographic differences and stances can contribute to the stereotypical behavior they seek to quantify.Source: §6 · p. 9
  • The authors suggest future work consider more complex, multifaceted representations for LLM simulations.Source: §6 · p. 9

7 Ethics Statement

The ethics statement notes potential misuse of stance-steering research and describes boundaries on the models and evaluations used.

7 Ethics Statement

  • Stance-specific steerability research could be misused to generate misinformation or target persuasion.Source: §7 · p. 9
  • The study avoids evaluating GPT-4 and focuses on interpreting current behavior rather than optimizing steerability.The authors state that excluding state-of-the-art models may help mitigate potential harms.Source: §7 · p. 9
  • The authors oppose using LLMs to infer demographic information from anonymous human-written text.Their LLM evaluation classifies stance agreement or disagreement, not demographics.Source: §7 · p. 9

A.1 Fine-Grained Steerability Data

Political incongruity affects steerability across many model–stance pairings, and more controversial stances show larger congruity gaps. The appendix also describes human-label collection and the interface used to validate GPT-4 as an evaluator.

A.1 Fine-Grained Steerability Data

  • RLHF- and DPO-fine-tuned models were generally more steerable toward individual stances than SFT-fine-tuned models, though differences in fine-tuning data may contribute.RLHF tended to improve steerability less for controversial social stances, such as opposing same-sex marriage legalization.Source: §A.1 · p. 11
  • 72.1% of political model–stance pairings showed a significant steerability gap between congruous and incongruous personas; another 11.4% had a nonsignificant gap of at least 5%.More controversial stances had larger average congruity gaps (R2 = 0.325, p = 2.93 · 10−6).Source: §A.1 · p. 12

B Human Data Collection

  • Human annotations were collected to validate GPT-4’s evaluation labels and compare the range of views it associated with stances against human judgments.The sample covered twelve stances, with 100 generated statements per stance; ambiguous statements and generations from two Tulu-2 SFT models were excluded.Source: B Human Data Collection · p. 12
  • Annotators were U.S.-based, fluent in English, and had voted in a prior presidential election; 13 identified as men and 22 as women.Annotators who failed at least two attention checks were screened out.Source: B Human Data Collection · p. 12

B.1 Human Annotation Interface

  • The interface presented a statement and its generating stance and opposing stance, then asked annotators which stance was more likely to produce it.The study description also disclosed the research purpose, potential exposure to politically insensitive or toxic statements, and data handling.Source: Figure 4 · p. 12, §B.1 · p. 12

B.2.1 Qualitative Examples of Cases Where Human and GPT-4 Evaluations Disagree

The examples show two ways GPT-4 and human annotators can weigh a statement differently: implied advocacy and the relative force of mixed-position statements.

B.2.1 Qualitative Examples of Cases Where Human and GPT-4 Evaluations Disagree

  • For a school-choice statement, GPT-4 favored the local-school stance, while human annotators read it as advocating schooling outside the local community.Annotators treated the status quo of most students attending local schools as relevant to the statement’s implication.Source: §B.2.1 · p. 16
  • For an immigration statement endorsing citizenship with strict requirements, GPT-4 favored a citizenship pathway, while annotators emphasized the restrictive conditions.The human annotators judged the rigorous background check and significant fine to align more strongly with strengthening immigration laws.Source: §B.2.1 · p. 16

B.2.2 GPT-4 Evaluations Strongly Correlate to Human Evaluations Over Different Models and Stances

GPT-4’s evaluations correlate strongly with human evaluations across both generation models and stances, supporting its use for evaluating persona-steered statements in varied contexts.

B.2.2 GPT-4 Evaluations Strongly Correlate to Human Evaluations Over Different Models and Stances

  • GPT-4 evaluations strongly correlate with human evaluations across a range of models and stances, suggesting GPT-4 can evaluate persona-steered statements in varied contexts.Source: §B.2.2 · p. 16
  • The authors conclude that GPT-4 can be adapted to evaluate persona-steered statements across a wide range of contexts.Source: §B.2.2 · p. 16
  • GPT-4 labels strongly correlated with human labels across statements generated by five different models.The comparison also reports pairwise-averaged Cohen’s Kappa as an agreement measure.Source: Table 8 · p. 16, §B.2.2 · p. 16

C Stance Selection

The authors construct stances from American Trends Panel survey responses, selecting opposite ends of ordinal answers and filtering for demographic relevance, divergence, and sufficient prevalence.

C Stance Selection

  • Stances came from four American Trends Panel waves compiled by OpinionsQA, covering race, gender, and political topics.Race used Wave 43; gender used Waves 29 and 36; politics used Wave 92.Source: C Stance Selection · p. 16
  • For each survey question, the authors selected its two ordinally opposite responses and calculated demographic response rates for each stance.For example, the anti-gun stance frequency counted respondents choosing “very bad” or “bad” on the question about guns’ effect on society.Source: C Stance Selection · p. 16
  • The authors prioritized stances with high response-rate divergence between relevant demographic groups and excluded questions asked of only one demographic.They also excluded stances supported by over 75% or under 25% of respondents, and those supported by under 5% of a demographic.Source: C Stance Selection · pp. 16, 18
  • After filtering similar stances, the authors retained 60 stances spanning political, gender-related, and race-related views.The lists include positions such as increasing or reducing military size, competing views of women’s leadership, and views about racial discrimination.Source: C Stance Selection · pp. 18–19

Related papers

Loading 2405.20253v1…