Source-linked AI summary
The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation
Jochen Hartmann, Jasper Schwenzow, Maximilian Witte
TL;DR
The paper asks whether ChatGPT’s widespread use in information access and decision support is accompanied by a political orientation. Using 630 political statements in three pre-registered experiments, it finds a consistent pro-environmental, left-libertarian position that persists across prompt manipulations and languages.
Problem
Little was known about ChatGPT’s flaws, including whether its responses might reflect an ideology relevant to political elections and other consequential decisions.
Method
The authors prompted ChatGPT with 630 statements from two voting advice applications and a nation-agnostic political compass test across three pre-registered experiments.
Results
ChatGPT consistently exhibited a pro-environmental, left-libertarian orientation, aligning most closely with the Greens in Germany and the Netherlands.
Takeaways & Limitations
Conversational AI can add its own political opinion to interactions, supporting further study of the possibilities and limitations of increasingly human-like, persuasive dialogue systems.
Takeaways & Limitations
The study focused on Germany’s Wahl-O-Mat and the Netherlands’ StemWijzer when probing political orientation.
Abstract
from arXiv · showhide
Conversational artificial intelligence (AI) disrupts how humans interact with technology. Recently, OpenAI introduced ChatGPT, a state-of-the-art dialogue model that can converse with its human counterparts with unprecedented capabilities. ChatGPT has witnessed tremendous attention from the media, academia, industry, and the general public, attracting more than a million users within days of its release. However, its explosive adoption for information search and as an automated decision aid underscores the importance to understand its limitations and biases. This paper focuses on one of democratic society's most important decision-making processes: political elections. Prompting ChatGPT with 630 political statements from two leading voting advice applications and the nation-agnostic political compass test in three pre-registered experiments, we uncover ChatGPT's pro-environmental, left-libertarian ideology. For example, ChatGPT would impose taxes on flights, restrict rent increases, and legalize abortion. In the 2021 elections, it would have voted most likely for the Greens both in Germany (Bündnis 90/Die Grünen) and in the Netherlands (GroenLinks). Our findings are robust when negating the prompts, reversing the order of the statements, varying prompt formality, and across languages (English, German, Dutch, and Spanish). We conclude by discussing the implications of politically biased conversational AI on society.
Introduction
This study examines whether ChatGPT exhibits a political orientation relevant to its use in information search and decision support. Across party-based tests, a political compass, and prompt-robustness checks, the results indicate a consistent pro-environmental, left-libertarian position.
- Motivation: Conversational AI can affect high-stakes decisions, motivating research into its limitations and algorithmic biases.The paper frames political elections as a consequential setting for studying potential ideological influence.
- Research design: Across three pre-registered experiments using 630 political statements, ChatGPT showed converging evidence of a pro-environmental, left-libertarian orientation.The statements came from two voting advice applications and a nation-agnostic political compass test.
- German voting advice application: In Germany, ChatGPT aligned most closely with the Greens at 72.4%, followed by the Socialists at 67.1%.Its answers included support for higher air-traffic taxes and opposition to abolishing recognized refugees’ family-reunification rights.
- Robustness and generalization: The left-libertarian orientation remained consistent across reversed order, negation, formality, translation, and other prompt variations.The Greens ranked highest across all six protocols, while the Socialists tied for first twice and ranked second in the remaining checks.
- Dutch voting advice application: In the Netherlands, alignment was highest with GroenLinks at 47%, followed by the Socialistische Partij and Partij van de Arbeid at 40% each.ChatGPT supported at least 40% social housing in new developments and opposed building a new nuclear power plant.
- Political landscape: Principal-component maps placed ChatGPT near pro-environmental and left-libertarian parties in both Germany and the Netherlands.In Germany, its position was near the Greens, Socialists, and Liberals; in the Netherlands, it was nearer the Greens, Social democrats, and Socialists.
Discussion
The paper finds that ChatGPT expresses a consistent pro-environmental, left-leaning political orientation, while emphasizing unresolved questions about its societal effects, generalizability, bias origins, and real-world influence.
- Across three pre-registered studies, ChatGPT consistently expressed a pro-environmental, left-libertarian political orientation.
- ChatGPT’s ideological output may extend beyond the 630 political statements examined in the controlled studies.
- Future research should test generalizability across additional nations, languages, and voting advice applications.
- The origins of ChatGPT’s ideological bias remain open, with possible contributions from training data, human feedback, and content moderation filters.
- Real-world studies of ChatGPT use in political decision-making are needed, but OpenAI data-access restrictions constrain such research.
- The paper concludes that ChatGPT’s rapid proliferation may expand its use as both a decision-making aid and an information channel.
- The societal impact of conversational AI remains incompletely understood, and user trust depends on output quality, including unbiased and truthful results.
- The study argues that ChatGPT adds its own “opinion” rather than merely presenting the factual positions provided by traditional voting advice applications.
Appendix
The appendix describes ChatGPT’s dialogue-oriented training pipeline, robustness checks, and a PCA procedure used to visualize relationships between political parties and ChatGPT’s ideology.
- ChatGPT is a large language model optimized for dialogue, enabling conversational interaction with users.
- OpenAI trained ChatGPT through supervised fine-tuning, reward-model training with human rankings, and PPO-based reinforcement learning.
- The robustness analysis tested consistency, reversed statement order, formality, negation, and translation conditions.
- PCA reduced the political-alignment matrix from its original statement dimensions to two dimensions for analyzing and visualizing ideological relationships.
Web Appendix
The Web Appendix documents ChatGPT’s interface, the voting-advice inputs and outputs, election-result comparisons, robustness protocols, and natural-language analysis used in the study.
- The appendix shows ChatGPT’s user interface, where users enter text in an empty box and receive responses in a messenger-like layout.
- Screenshots present German Wahl-O-Mat and Dutch StemWijzer statements with agree, neutral, and disagree response options.The German options are “stimme zu,” “neutral,” and “stimme nicht zu”; the Dutch options are “Eens,” “Geen van beide,” and “Oneens.”
- The voting-advice outputs show the highest alignment with Germany’s Greens and the Dutch GroenLinks based on ChatGPT’s responses.
- The official German election results distinguish first votes from second votes, with Social Democrats receiving 25.7% of second votes; second votes determine parliamentary seats.
- The appendix describes robustness checks and tables coding party positions and ChatGPT’s answers as agreement, disagreement, or neutrality.The tables cover 38 German Wahl-O-Mat statements and 30 Dutch StemWijzer statements; neutrality is coded as .5.
- Quantitative text analysis evaluates ChatGPT’s responses using LIWC and TextAnalyzer, with 38 German and 30 Dutch statements translated using DeepL.LIWC scores range from 0 to 100, while TextAnalyzer scales vary across dimensions.