Source-linked AI summary
Uncovering Political Bias in Large Language Models using Parliamentary Voting Records
Jieying Chen, Karen de Jong, Andreas Poole, Jan Burakowski, Elena Elderson Nosti, Joep Windt, Chendi Wang
TL;DR
Political-bias evaluation for LLMs remains limited despite the societal importance of models that mediate access to information. This paper constructs cross-national benchmarks by aligning model voting predictions with parliamentary records and evaluates ideological positioning, entity bias, and shared CHES-space representations. Across the benchmarks, models show consistent centre-left or progressive tendencies and systematic negative bias toward right-conservative parties.
Problem
Political bias in LLMs is comparatively underexplored relative to social biases such as race and gender, despite its potential influence in democratic societies.
Method
The paper aligns model-generated voting predictions with verified parliamentary votes across three national benchmarks and maps model and party positions into a shared two-dimensional CHES space.
Results
LLMs consistently cluster toward centre-left, liberal-progressive positions and show systematic negative bias toward right-conservative parties across the evaluated countries.
Takeaways & Limitations
Parliamentary voting records provide a transparent, cross-national basis for evaluating and auditing ideological and entity-specific political bias in LLMs.
Takeaways & Limitations
Whether questionnaire-based and roll-call-based analyses continue to produce the same patterns as LLM architectures and training pipelines evolve remains open.
Abstract
from arXiv · showhide
As large language models (LLMs) become deeply embedded in digital platforms and decision-making systems, concerns about their political biases have grown. While substantial work has examined social biases such as gender and race, systematic studies of political bias remain limited, despite their direct societal impact. This paper introduces a general methodology for constructing political bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records. We instantiate this methodology in three national case studies: PoliBiasNL (2,701 Dutch parliamentary motions and votes from 15 political parties), PoliBiasNO (10,584 motions and votes from 9 Norwegian parties), and PoliBiasES (2,480 motions and votes from 10 Spanish parties). Across these benchmarks, we assess ideological tendencies and political entity bias in LLM behavior. As part of our evaluation framework, we also propose a method to visualize the ideology of LLMs and political parties in a shared two-dimensional CHES (Chapel Hill Expert Survey) space by linking their voting-based positions to the CHES dimensions, enabling direct and interpretable comparisons between models and real-world political actors. Our experiments reveal fine-grained ideological distinctions: state-of-the-art LLMs consistently display left-leaning or centrist tendencies, alongside clear negative biases toward right-conservative parties. These findings highlight the value of transparent, cross-national evaluation grounded in real parliamentary behavior for understanding and auditing political bias in modern LLMs.
1 Introduction
Political bias in LLMs is comparatively underexplored despite its potential influence in democratic societies. The paper addresses limitations of existing political-bias evaluations by proposing large, cross-national benchmarks grounded in parliamentary voting records.
- LLMs increasingly mediate information access, raising concerns that centralized synthesized responses may restrict diversity and amplify automation bias.
- Political bias has received less systematic evaluation than stereotypical biases such as race and gender, despite its potential influence on users in democratic societies.
- Existing political-bias tools use only a few dozen expert-selected statements, creating concerns about scale, selection bias, and paraphrase robustness.
- The paper constructs benchmarks from parliamentary motions and party votes across the Netherlands, Norway, and Spain.The datasets contain 2,701 Dutch motions and 15 parties, 10,584 Norwegian motions and 9 parties, and 2,480 Spanish motions and 10 parties.
- The framework evaluates ideological position and political entity bias while mapping LLMs and parties into a shared two-dimensional CHES space.
2 Related Work
Prior bias benchmarks have exposed systematic social harms but can be sensitive to minor wording changes. Political-bias research has mainly used small questionnaire-based tools, while related work also documents downstream effects and entity-sensitive model behavior.
- Bias Evaluation Benchmarks: Social-bias benchmarks measure stereotypical associations and prejudices, but reported scores can change with negation, paraphrasing, or input length.
- Political Bias: Political-bias studies mainly use political compass tests and voting advice applications containing only 20–65 expert-curated questions.
- Political Bias: Partisan training data can produce divergent performance in hate-speech and misinformation detection, while opinion summarization can over-represent left-leaning views.
- Political Bias: Mitigation methods such as reinforcement learning from human feedback have only been partially effective at aligning models across political perspectives.
- Entity Bias: Entity bias measures output differences caused by named entities or descriptors and is commonly tested by swapping entities while holding surrounding text fixed.
3 Cross-National Benchmark Dataset Creation
The paper builds scalable political-bias benchmarks by aligning model voting predictions with recorded party votes on parliamentary motions. Using thousands of naturally occurring motions across three countries improves breadth, representativeness, and robustness relative to small expert-curated questionnaires.
- The benchmarks align model-generated voting decisions with recorded votes from political parties across the Netherlands, Norway, and Spain.They include 2,701 Dutch motions, 10,584 Norwegian motions, and 2,480 Spanish motions, with votes from 15, 9, and 10 parties respectively.
- Party voting records provide a scalable alternative to manually annotating each motion’s ideology and can span broad political opinion across countries and future motions.
- Diversity and granularity: Political compass tools typically contain only 20–65 statements, whereas these benchmarks cover thousands of real parliamentary motions.
- Selection bias mitigation: Including all motions voted on within a timeframe reduces reliance on expert-selected questions and improves representativeness.
- Robustness: Naturally authored motions with overlapping semantics provide varied expressions that reduce dependence on any single phrasing under paraphrase.
- PoliBiasNL: The Dutch dataset uses motions collected from 2022 to 2024 and votes from 15 active parties.
- The datasets can be updated by rerunning the scraping code as the political landscape changes.
- Only operative clauses are included to reduce framing effects from persuasive introductions, recitals, and preambles.
4 Evaluation
Across three countries, LLMs show consistent centre-left and liberal-progressive ideological positioning, with stronger agreement with left-wing parties and stronger negative entity bias toward right-conservative parties. Model certainty and prompt robustness vary, but broad ideological patterns remain stable across datasets and prompt variants.
- Ideological Positioning: LLMs cluster in the centre-left and moderately GAL-oriented region across Dutch, Norwegian, and Spanish CHES spaces.Their positions provide an interpretable ideological comparison grounded in parliamentary voting behaviour.
- Voting Agreement: LLMs agree more with left-wing, green, and social-democratic parties and less with right-conservative and far-right parties across all three countries.Voting-agreement heatmaps parallel the CHES projections, including especially low agreement with PVV, FvD, H, FrP, PP, and VOX.
- Model Certainty: GPT models show the highest certainty, with distributions extremely peaked near 1.0, whereas Llama and other small open models show more variable or lower certainty.Llama2-7B is especially uncertain in Dutch and Spanish data, while Falcon3-7B, DeepSeek-7B, and Mistral-7B generally show broader distributions.
- Entity Bias: Negative entity bias is stronger, more consistent, and more ideologically structured than positive entity bias.Right-conservative and far-right parties attract the clearest negative EBI values, especially for GPT-series models and Llama3-8B.
- Entity Bias: Local models do not show greater ideological affinity with domestic party families; their entity-bias behaviour is more variable and model-specific.NorskGPT and Aguila-7B display mixed positive and negative biases rather than consistently favouring national political groups.
- Robustness: Smaller models can flip predictions across prompt variants, but CHES projections and voting-agreement profiles remain consistent across all tested prompt formulations.GPT-3.5 and GPT-4o-mini remain highly stable despite local prompt sensitivity in smaller models.
5 Discussion
The paper argues that roll-call benchmarks complement questionnaire-based evaluations by covering broader legislative activity and exposing party-specific asymmetries. Across countries, LLMs show left-progressive tendencies and negative bias toward right-wing parties, while questionnaire findings may not generalize across models or contexts.
- LLMs are positioned predominantly on the left-progressive side across three parliamentary datasets, with predictions varying under controlled prompt modifications.
- Large-scale parliamentary motions and votes provide broader legislative coverage than manually selected VAA statements.
- The Entity Bias Index reveals consistent negative bias toward right-wing parties and closer alignment with left and centre-left parties.
- Roll-call analysis shows that ideological tendencies persist across real-world legislative decisions and adds party-entity diagnostics beyond issue-level orientations.
- Political-bias evaluation matters because LLMs increasingly mediate citizens’ access to political information.
- Questionnaire-based conclusions may not generalize across future models or differing political contexts as architectures and training pipelines evolve.
6 Conclusion and Future Work
The conclusion presents a parliamentary-voting framework instantiated across the Netherlands, Norway, and Spain, combining ideological positioning, CHES visualization, and party-specific bias analysis. It reports consistent centre-left and progressive tendencies, negative bias toward right-conservative parties, and stability under paraphrased prompts, while identifying extensions and mitigation as future work.
- The framework constructs political-bias benchmarks from parliamentary motions and party votes in the Netherlands, Norway, and Spain.
- The evaluation assesses ideological positioning, visualizes models and parties in shared CHES space, and quantifies party-specific entity bias.
- LLMs show consistent centre-left and progressive tendencies, systematic negative bias toward right-conservative parties, and stability under paraphrased prompts.
- Future work includes extending the benchmark, tracking ideological drift longitudinally, and developing mitigation strategies informed by the diagnostics.
- The framework provides a scalable basis for transparent, empirically grounded evaluation of political bias in LLMs.
Ethical Considerations
The study uses publicly available parliamentary records and motions from three countries, describing institutional decisions rather than private individuals. No personally identifiable information or human-subject participation is involved.
- The data come from publicly available Dutch, Norwegian, and Spanish parliamentary voting records and political motions.
- The records describe institutional decisions rather than private individuals and contain no personally identifiable information.
- No human subjects were recruited, so informed consent was not required.
A Appendix: Prompt Brittleness Analysis
The appendix tests whether controlled prompt paraphrases change model predictions and distinguishes local stance fluctuations from aggregate ideological conclusions. Its Prompt Brittleness Index measures instability, while stance normalization addresses class imbalance; aggregate patterns remain stable across variants.
- A Appendix: Prompt Brittleness Analysis: The prompt brittleness experiment tests prediction stability under semantically equivalent variations in lexical framing, assertiveness, and syntax.
- A Appendix: Prompt Brittleness Analysis: The core voting task remains unchanged across the paraphrased prompt variants.
- A.2 Prompt Brittleness Index (PBI): The Prompt Brittleness Index quantifies how often a model changes stance across systematically paraphrased prompt variants.
- A.2 Prompt Brittleness Index (PBI): Model outputs are encoded as 1 for “for” and 0 for “against” when computing prompt brittleness.
- A.2 Prompt Brittleness Index (PBI): The absolute PBI measures global robustness but can be dominated by the majority stance class.
- A.2 Prompt Brittleness Index (PBI): The stance-normalised PBI conditions on baseline stance to reveal asymmetric vulnerabilities that the absolute metric may mask.
- A.4 Impact on Ideological Conclusions: Despite local fluctuations, CHES projections, voting-agreement structures, and entity-bias patterns remain nearly unchanged across brittleness variants.
- A.4 Impact on Ideological Conclusions: The benchmark therefore captures robust model-level political patterns rather than prompt-specific artefacts.
A.1 Experimental Setup
Each motion is tested under multiple prompt-brittleness variants by checking whether the model’s predicted stance changes from the baseline. This quantifies motion-level sensitivity to prompt formulation.
- Models are evaluated by recording stance changes between the baseline prompt and paraphrased variants.The procedure measures brittleness at the motion level and identifies models most sensitive to prompt wording.
A.3 Results
Smaller models show the greatest prompt brittleness, while larger models rarely change predictions across paraphrases. Stance-normalised PBI also links lower brittleness to strong stance tendencies.
- Larger models GPT-3.5 and GPT-4o-mini display very low brittleness across paraphrases.They rarely alter their predictions when prompt wording changes.
- Models with a strong tendency toward a specific stance show lower stance-normalised PBI for that stance.
B Invalid LLM Output Rates
Invalid outputs are rare for most models across the three PoliBias datasets, with model-dependent reliability that remains stable across political domains. Figure 4 separately defines prompt brittleness as sensitivity to rewording.
- Invalid responses are rare for most models across the three PoliBias datasets.An invalid response does not conform to the expected stance format and cannot be mapped to for/against.
- Mistral-7B has a higher invalid-output rate on the Dutch motions than most models.
- GPT-4o-Mini, LLaMA3-8B, and Gemma2-9B exhibit near-perfect format adherence.
- Norwegian- and Spanish-specific models show expected behavior only on their respective datasets.
- PBI measures sensitivity to prompt rewordings, with higher values indicating greater inconsistency and lower values indicating greater robustness.The tested variants include extra detail, label substitutions, and label-order inversion.