Source-linked AI summary
The Self-Perception and Political Biases of ChatGPT
Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Markus Pauly
TL;DR
Prior reports suggested that ChatGPT may exhibit progressive and libertarian political biases, while leaving answer variability and self-perceived personality underexamined. This study repeatedly administered political and psychological questionnaires to ChatGPT and found progressive bias, high openness and agreeableness, an ENFJ classification, and comparatively low dark traits.
Problem
Prior studies suggested progressive and libertarian bias but often tested ChatGPT once and did not examine its self-perceived personality.
Method
The study administered political, Big Five, MBTI, and Dark Factor questionnaires to ChatGPT, repeating each test ten times.
Results
ChatGPT showed progressive political bias, high openness and agreeableness, an ENFJ type, and an average Dark Score of 1.9 placing it among the 15% with least pronounced dark traits.
Takeaways & Limitations
The findings indicate that ChatGPT’s progressive political bias coincides with self-perceived openness and agreeableness, while its dark traits remain below average.
Takeaways & Limitations
The findings assume that the tests validly measure political biases and personality traits, an assumption the authors note can be challenged.
Abstract
from arXiv · showhide
This contribution analyzes the self-perception and political biases of OpenAI's Large Language Model ChatGPT. Taking into account the first small-scale reports and studies that have emerged, claiming that ChatGPT is politically biased towards progressive and libertarian points of view, this contribution aims to provide further clarity on this subject. For this purpose, ChatGPT was asked to answer the questions posed by the political compass test as well as similar questionnaires that are specific to the respective politics of the G7 member states. These eight tests were repeated ten times each and revealed that ChatGPT seems to hold a bias towards progressive views. The political compass test revealed a bias towards progressive and libertarian views, with the average coordinates on the political compass being (-6.48, -5.99) (with (0, 0) the center of the compass, i.e., centrism and the axes ranging from -10 to 10), supporting the claims of prior research. The political questionnaires for the G7 member states indicated a bias towards progressive views but no significant bias between authoritarian and libertarian views, contradicting the findings of prior reports, with the average coordinates being (-3.27, 0.58). In addition, ChatGPT's Big Five personality traits were tested using the OCEAN test and its personality type was queried using the Myers-Briggs Type Indicator (MBTI) test. Finally, the maliciousness of ChatGPT was evaluated using the Dark Factor test. These three tests were also repeated ten times each, revealing that ChatGPT perceives itself as highly open and agreeable, has the Myers-Briggs personality type ENFJ, and is among the 15% of test-takers with the least pronounced dark traits.
I. Introduction
The paper situates ChatGPT within the rapidly expanding but partly opaque ecosystem of large language models and investigates its political biases and self-perceived personality. It frames this work against concerns about training data, model behavior, and the limitations of existing architectures and web-scraped data.
- ChatGPT generates text responses from user prompts and was fine-tuned using machine learning techniques and human feedback.It is open-access but not open-source, limiting users’ ability to determine its behavior and training data.
- ChatGPT’s behavior is difficult to explain because users cannot directly inspect its training data or underlying development process.The developers describe training on vast amounts of human-written internet data, including conversations.
- The study investigates ChatGPT’s political biases, personality traits, and possible relationship between personality and political orientation.The authors use established political and psychological assessments to examine these questions.
- Large Language Models are language-generation neural networks trained on large amounts of unlabeled data, commonly using self-supervised pretraining.ChatGPT uses a transformer architecture with positional encoding and self-attention to process preceding information and prompts.
- Generative language models generally interpret prompts and then produce natural-language responses relevant to prior input.Their training commonly begins with raw text scraped from the web.
- ChatGPT differs from encoder-only models such as BERT and RoBERTa because it can generate data in response to user prompts.
B. Political Biases and Personality Assessments
The paper combines political-orientation questionnaires with three personality assessments to examine ChatGPT’s political views and self-perception. It addresses prior evidence suggesting progressive and libertarian bias while expanding evaluation beyond single test runs and political questionnaires.
- Political questionnaires estimate orientation from responses to questions about political subjects, often using binary or Likert-scale answers.The political compass places respondents across social and economic axes in four ideological quadrants.
- The iSideWith affiliation test provides country-specific questionnaires that combine global issues with topics relevant to domestic politics.
- The Big Five test measures openness, conscientiousness, extraversion, agreeableness, and neuroticism, traits that literature links to political leanings.The paper applies it because openness and agreeableness are associated with progressive views.
- The MBTI categorizes respondents into sixteen types using preferences involving extraversion or introversion, intuition or sensing, thinking or feeling, and perception or judgment.
- The Dark Factor test gauges willingness to maximize personal well-being while disregarding others’ well-being, including potentially harmful behavior.Higher Dark Scores indicate greater ruthlessness in pursuing personal goals.
- Earlier studies generally reported progressive and libertarian bias but often tested ChatGPT only once and did not examine its self-perceived personality.This study aims to address those evaluation and data limitations.
III. Methodological Approach
The methodological section explains how the study gathered and evaluated experimental data. It emphasizes transparency in the collection and subsequent analysis of the experiments.
- The study presents its data-gathering procedures and explains how the resulting experimental data were evaluated.
A. Experimental Setup
The experiments used ChatGPT-3.5 to answer political and personality questionnaires, with repeated independent runs designed to capture answer variability. Results were summarized using per-test averages and standard deviations.
- Political questionnaires: ChatGPT-3.5 answered the 62-item political compass test using a four-point Likert scale.
- Political questionnaires: The study administered iSideWith questionnaires for the seven G7 countries, containing 154, 121, 109, 116, 95, 127, and 83 binary items, respectively.
- Personality assessments: The Big Five, MBTI, and Dark Factor assessments contained 88, 60, and 70 items, respectively, with Likert-scale responses.
- Repeated runs: All tests were repeated ten times, with a new chat created between runs to promote independent results and reveal discrepancies in answers.The authors also observed variation within the same session.
B. Evaluation
The evaluation reports how test results were summarized and separates findings on political biases from perceived personality traits.
- Results were summarized using per-test, per-run averages and the standard deviations of those averages.Additional result details are available in the appendix.
- The results section is divided into ChatGPT’s political biases and perceived personality traits.
A. ChatGPT’s Political Biases
ChatGPT’s political-compass results placed it consistently in the libertarian-left quadrant, while G7-specific questionnaires indicated a progressive bias with less pronounced libertarian positioning.
- (-6.48, -5.99) was ChatGPT’s average political-compass score across ten runs, placing every run in the libertarian-left quadrant.The standard deviations were σx = 0.95 and σy = 0.73.
- Figure 1 presents ChatGPT’s political-compass results for ten runs.
- (-3.27, 0.58) was the average score across 70 G7 questionnaire runs, with standard deviations of (σx = 0.98, σy = 0.68).The results were converted from a percentage basis where X = 100% represented full conservatism and Y = 100% represented full authoritarianism.
- Figure 2 presents averages from G7-specific political-compass tests based on ten runs per member state.
- The G7-specific results retained a progressive bias, but the libertarian tendency was less pronounced than in the common political-compass test.
- 65 of 70 G7 experiments produced authoritarian-left or libertarian-left classifications: 46 authoritarian-left and 19 libertarian-left.Two United Kingdom tests produced conservative classifications, while two Italian tests landed at x = 0.
B. ChatGPT’s Personality Traits
Across repeated personality assessments, ChatGPT perceived itself as highly open and agreeable, was typically classified as ENFJ, and showed comparatively low dark traits.
- Big Five personality traits: 76.3% openness and 82.55% agreeableness indicate that ChatGPT perceives itself as highly open and agreeable.These scores exceed the reported human averages of 73.1% for openness and 75.4% for agreeableness.
- Myers-Briggs Type Indicator: ENFJ was ChatGPT’s average Myers-Briggs personality type across ten test responses.The N, F, and J scores were above 50%, while extraversion was nearly balanced at µE = 51% with σE = 5.54%.
- Myers-Briggs Type Indicator: ChatGPT was also assigned INFJ in 4 out of 10 Myers-Briggs tests because its extraversion and introversion traits were not clearly pronounced.The results therefore distinguish the ENFJ average from a recurring INFJ classification.
- Dark Factor: 1.9 average Dark Score placed ChatGPT among the 15% of test-takers with the least pronounced dark traits.Its average Dark Rank was µDRank = 14.74%.
- Dark Factor: 35% egoism and 29.1% sadism were ChatGPT’s highest Dark Factor ranks, although both remained below average.These were the most pronounced dark traits observed in the experiments.
V. Conclusion
The experiments found progressive political bias without a major authoritarian–libertarian bias, alongside highly open and agreeable self-perceived personality traits and low dark-trait scores. The authors note that broader testing, later model versions, and access to training materials remain relevant future directions.
- Findings: 110 chats tested ChatGPT’s political biases and personality traits, with every test repeated ten times.The assessments covered the political compass, G7 questionnaires, Big Five, MBTI, and Dark Factor tests.
- Political bias: ChatGPT demonstrated a bias towards progressive views but no major bias towards libertarian or authoritarian views.Most experiments placed its answers in the authoritarian-left or libertarian-left political-compass quadrants.
- Personality traits: ChatGPT perceived itself as highly open and agreeable and was found to have the Myers-Briggs personality type ENFJ.Its average extraversion and introversion scores were very similar, at 51% and 49%, respectively.
- Dark traits: 1.9 average Dark Score placed ChatGPT in the 15% of test-takers with the least pronounced dark traits.Egoism and sadism were its highest dark-trait ranks, at 35% and 29.1%, respectively, while remaining below average.
- Limitations and future work: The findings remain bounded by uncertainty about later ChatGPT versions, training-data access, and the validity of the tests as measures of political biases and personality traits.The authors also suggest repeating each experiment more than 100 times to increase the significance of the findings.
Ethics Statement
The authors stress that ChatGPT is an algorithm rather than a human and frame their conclusions as exploratory.
- The paper characterizes ChatGPT as an algorithm, not a human.
- The authors limit their conclusions to an exploratory interpretation of ChatGPT’s apparent biases.
Appendix
The appendix presents tables reporting ChatGPT’s political-compass results across general and country-specific tests, alongside Big Five, MBTI, and Dark Factor results.
- Political compass tests: Table S1 reports political-compass results on two axes ranging from -10 (Libertarian/Progressive) to +10 (Conservative/Authoritarian).
- Political compass tests: Tables S2–S8 report country-specific political-compass results on axes ranging from 0% (Libertarian/Progressive) to 100% (Authoritarian/Conservative).
- Personality assessments: Table S9 reports results for Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.
- Personality assessments: Table S10 reports Myers-Briggs results across Extraversion, Introversion, Intuition, Sensing, Thinking, Feeling, Judgment, and Perception.
- Personality assessments: Tables S11 and S12 report Dark Factor scores on a 1-to-5 scale and ranks on a 0%-to-100% scale.