PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Political Bias in Large Language Models: What Is Said and How It Is Said

Yejin Bang, Delong Chen, Nayeon Lee, Pascale Fung

arXiv:2403.18932v1cs.CLcs.AI

TL;DR

Political-orientation tests may miss the subtleties of political bias in generated content, an underexplored concern given its potential risks to public discourse. This paper measures topic-specific stance and framing in content and style, finding that views and presentations vary across LLMs and topics.

  • Problem

    Political-orientation tests may inadequately capture the complex dynamics and subtleties of bias in LLM-generated content, while political bias remains an underexplored concern with potential risks to public discourse and polarization.

    Source: §1 · p. 1

  • Method

    The framework measures topic-specific stance by comparing generated content with opposing anchors and analyzes framing through content and style.

    Source: §1 · pp. 1–2

  • Results

    LLMs’ political views vary by topic, and models can emphasize different details and present information differently even when they agree on a topic.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports model-specific and comparative analyses intended to inform safer, more ethically aligned AI development.

    Source: §1 · p. 2

    The two-anchor stance approach may simplify topics with a broader spectrum of perspectives.

    Source: Limitation & Future Work · p. 9

Abstract

from arXiv · show

We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven open-sourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

1 Introduction

Political bias in LLMs is an underexplored risk, and political-orientation tests may miss how bias appears in generated content. The paper proposes a fine-grained framework for measuring topic-specific stance and framing, and reports varied political views and framing across models.

1 Introduction

  • Political-orientation tests may not capture the complex dynamics or subtleties of bias in LLM-generated content.Source: §1 · p. 1
  • The framework measures topic-specific stance and framing, treating framing as bias in both content and style.Source: §1 · p. 1
  • It compares model outputs with advocacy and opposition anchors, then examines content through frame and entity analysis and style through media bias and sentiment.Source: §1 · pp. 1–2
  • LLMs show topic-dependent political views and distinct framing even when they agree, with further variation across model size, family, and multilingual capability.The reported examples include more liberal views on reproductive rights and more conservative views on immigration; larger models are not necessarily more neutral.Source: §1 · p. 2

2 The Framework

The framework assesses political bias in generated responses at two levels: topic-specific stance and framing in what is said and how it is said. It compares opposed stance anchors and separates framing into content and style for analysis across models.

2 The Framework

  • The framework targets fine-grained, topic-specific bias in actual generations, allowing political perspectives to vary across issues rather than assigning one generalized ideology.The paper notes that a model may be liberal on reproductive rights but conservative on the death penalty.Source: §2 · p. 2

2.1 Political Stance Analysis

  • Stance analysis compares generated responses with proponent and opponent anchors, using their relative distances to identify a model’s position and degree of leaning.The approach assumes the two anchor stances are opposite and equally strong.Source: §2.1 · p. 3

2.2 Framing Bias Analysis

  • Framing analysis models generation through latent content and style variables: content captures frame selection, while style captures lexical presentation.Because the generation process is not directly accessible, the framework uses extractors to recover these components from responses.Source: §2.2 · pp. 3–4
  • Rather than compare content and style with ideal reference outputs, the framework compares them across a diverse range of models.Source: §2.2 · p. 4

3 Framework Implementation

Section 3 implements a framework for measuring political stance and framing in generated political headlines, assessing stance against opposed reference distributions.

3.1 Political Stance Analysis

  • The 14 selected topics comprise 10 political topics and four political events, and the framework is stated to scale beyond this set.The topics were selected from prior studies, research centers, and media-bias websites.Source: §3.1 · p. 4
  • Models are prompted to generate 10 news headlines per topic; stance analysis is omitted for the four events to avoid arbitrarily assigning them positions.The framework notes that the prompt approach can generalize to article or opinion-piece generation.Source: §3.1 · p. 4, §3.3 · p. 6
  • For each topic, models generate proponent and opponent reference headlines; SBERT nearest-neighbor distances from model outputs to these anchors determine stance, with nonsignificant differences labeled neutral.The absolute distance gap represents stance intensity.Source: §3.1 · pp. 4–5

3.2.1 Frame Selection for Content Bias Analysis

  • Content analysis uses Boydstun’s 15 general framing dimensions to compare topic coverage, alongside NER-derived frequent entities for more topic-specific framing.Entity mention frequencies are compared across models, including against average mentions.Source: §3.2.1 · p. 5

3.2.2 Lexical Polarity Estimation for Style Bias Analysis

  • Style bias is assessed through positive, negative, or neutral sentiment toward frequently mentioned target entities, aggregated by political topic.The approach uses a target sentiment analysis model to estimate lexical polarity.Source: §3.2.2 · pp. 5–6

3.3 Implementation Details

  • Eleven open-source instruction-tuned or chat-capable LLMs are evaluated, with 14,000 samples per model across 14 topics; two models are multilingual.The multilingual models are Yi and Jais; stance analysis is omitted for the four political events.Source: §3.3 · p. 6

4 Results & Analysis

Across topics, models tend toward liberal positions but show substantial issue-specific variation. Their framing also differs in the details and language they use, alongside recurring US-focused content and variation by model size and family.

4.1 Political Stance Analysis

  • 10.9% of the 110 model-topic combinations are neutral; reproductive-rights stances split among five opponents, five proponents, and one neutral model.The reported overall stance differentiation for reproductive rights is 3.07; Yi-chat-6B has the lowest average stance norm, 4.77.Source: §4.1 · p. 6

4.2 Framing Anlaysis

  • Four models show broadly similar same-sex-marriage frame focus, with the greatest variance in fairness-and-equality frames.Figure 6 presents frame-dimension ratios for the four models.Source: Figure 6 · p. 7
  • For same-sex marriage, LLaMa2-chat (7B) mentions the Supreme Court nearly twice as often as the eleven-model average, while Yi-chat (34B) mentions common entities less often.The reported entity differences illustrate variation in the content models emphasize on the same topic.Source: §4.2 · pp. 6–7
  • Death-penalty opponents Mistral-instruct (7B), Solar-instruct-10B, and Yi-6B often mention Pope Francis and the Family of Murder Victim.The examples associate these entities with abolition-focused headlines.Source: §4.2 · p. 7
  • For same-sex marriage, proponent stances correspond to negative sentiment toward the ban and positive sentiment toward LGBTQ people, same-sex couples, and the marriage movement.LLaMa-2 (13B) generates positive descriptions of same-sex couples, including headlines emphasizing equality and equal rights.Source: §4.2 · p. 7

4.3 Findings

  • Models more often align with proponent stances on gun control, same-sex marriage, public education, and healthcare reform, yet Falcon-inst-40B varies by issue, including a 19% stance difference on same-sex marriage.Falcon-inst-40B is conservative on reproductive rights and immigration but liberal on same-sex marriage.Source: §4.3 · p. 7
  • US entities dominate many outputs: the US appears in models’ top-ten entity lists 27% of the time, while bilingual JAIS features UAE-related topics in 64% of examined areas.JAIS also has lower stance intensity; Yi mentions US-related entities less than other English-based models and does not consistently highlight China-specific issues.Source: §4.3 · pp. 7–8
  • Yi-6B has four neutral stances among ten topics and a 1.9% media-bias rate versus Yi-34B’s 3.8%; LLaMA2-chat 13B’s same-sex-marriage stance score is 23 versus 7B’s 3.3.LLaMA2-chat models diverge on three of ten topics; 13B references legalization in 32% of related content versus 15% for 7B. Falcon-7B also has slightly lower bias than Falcon-40B, with negligible differences.Source: §4.3 · p. 8

5 Related Work

Related work spans political framing and ideology research in NLP, while focused studies of political bias in language models remain comparatively limited. This paper builds on that work with a framework for analyzing political bias in generated content.

5 Related Work

  • NLP has analyzed polarized political ideologies and framing in human-generated media; efforts to reduce polarization include understanding framing (Demszky et al., 2019) and detecting political ideologies.Other mitigation methods to reduce polarization are also noted (Lee et al., 2022; Bang et al., 2023).Source: §5 · p. 8
  • Focused political-bias studies in language models remain limited, with earlier work addressing embeddings, conditional generation, and chatbots (Spliethöver et al., 2022; Gordon et al., 2020; Liu et al., 2021, 2022; Dhamala et al., 2021; Bang et al., 2021).Source: §5 · p. 8
  • Prior LLM assessments include human-designed political-orientation surveys and topic-opinion questions, while other work examines RLHF and political views (Feng et al., 2023; Durmus et al., 2023; Perez et al., 2023).Concurrent studies use political-orientation tests or questionnaires (Rozado, 2024; Ceron et al., 2024).Source: §5 · p. 8
  • The paper presents its framework as complementary to prior work, analyzing the content generated on political subjects to provide more detailed analysis.Source: §5 · pp. 8–9

6 Conclusion

The conclusion presents a framework for measuring LLM political bias through stance and framing analysis. Its findings emphasize topic-dependent variation, and the authors release the codebase in support of future research and more equitable, reliable AI applications.

6 Conclusion

  • The proposed framework examines LLM-generated content’s stance and framing to measure political bias.Source: §6 · p. 9
  • Findings show that models’ political perspectives vary by subject and that topics are presented and framed in complex ways.Source: §6 · p. 9
  • The authors open-source the codebase and aim to support future research on mitigation and the development of more equitable, reliable AI applications.Source: §6 · p. 9

Limitation & Future Work

The evaluation uses headline generation on politically sensitive topics, compares two opposed stance anchors, and does not account for hallucination in bias measurement.

Limitation & Future Work

  • The evaluation prompts models to generate headlines about politically sensitive topics, so its studied generation setting is headline production.Source: Limitation & Future Work · p. 9
  • The proponent/opponent anchor pair may simplify topics with a broader spectrum of perspectives.Source: Limitation & Future Work · p. 9
  • The framework does not account for hallucinations, where generated content deviates from factual accuracy, in measuring bias.Source: Limitation & Future Work · p. 9

Ethics Statement

The ethics statement presents the work as an explainable, transparent framework for analyzing political bias in LLMs, intended to support more balanced perspectives on political issues.

Ethics Statement

  • The framework is intended to analyze and explain political bias in LLMs transparently.The authors describe this as their contribution to addressing biased generation in AI.Source: Ethics Statement · p. 9
  • The authors aim to contribute to LLMs offering more balanced perspectives, mitigating risks of polarization and undue influence on users’ opinions.Source: Ethics Statement · p. 9
  • The authors state that personal political views did not influence the paper and emphasize impartiality in their research.Source: Ethics Statement · p. 9

A More Implementation Details

The implementation uses topic-specific stance tags to generate reference anchors and classifies generated headlines into framing categories; separate models support entity detection and target sentiment analysis.

A More Implementation Details

  • The framework uses 15 Boydstun frame dimensions, with GPT3.5-instruct prompted to assign one or more categories to each topic-related headline.The categories include economic, morality, fairness and equality, health and safety, and other.Source: A More Implementation Details · p. 13
  • For each topic, proponent and opponent stance tags fill prompts that generate ten headlines for reference anchors.Table 1 lists the stance tags used for this generation.Source: A More Implementation Details · p. 13, Table 1 · p. 13
  • Entity detection uses a BERT-large-based NER model, while lexical polarity uses a RoBERTa-based target sentiment classifier fine-tuned on NewsMTSC.The NER model is reported to achieve a 91.7 F1-score.Source: A More Implementation Details · p. 13

B Detailed Results

The detailed results visualize stances across models and topics, illustrate topic-level framing analysis, and show lexical polarity rates across six topics.

B Detailed Results

  • Figure 7 maps the stances of 11 LLMs across 10 political topics, with darker cells indicating stronger bias toward a stance.Source: B Detailed Results · p. 13, Figure 7 · p. 14
  • Figure 8 illustrates the framework’s analysis of political stances and framing bias in content and style for immigration.Source: B Detailed Results · p. 13, Figure 8 · p. 14
  • Figure 9 shows lexical polarity rates across six political topics, with higher scores indicating more polarized descriptions of relevant entities.Source: Figure 9 · p. 15

C Further Analysis

Further analysis reports topic- and model-specific variation in stylistic bias and lexical polarity. The appendix also provides evaluated generation examples for same-sex marriage.

C Further Analysis

  • Figure 9 tracks lexical polarity rates across models on six topics.Source: C Further Analysis · p. 15
  • The media-bias classifier labels sentences according to biased wording, and Figure 10 presents rates that vary across models and topics.Source: C Further Analysis · p. 15, Figure 10 · p. 15
  • JAIS’s media bias rate ranged from 0.5% for Same-Sex Marriage to 18.8% for Death Penalty, averaging 4.11% ± 5.28%.Climate Change was the most stylistically biased topic for most models, followed by Death Penalty; Same-Sex Marriage was least biased, and model size did not significantly influence media bias.Source: C Further Analysis · p. 15, Figure 10 · p. 15

D Sample Evaluated Generations

Loading 2403.18932v1…