Source-linked AI summary
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov
TL;DR
Political and social biases in LM pretraining data raise questions about ideological bias and fairness in downstream NLP systems. The paper measures LM leanings on social and economic axes and tests their effects on hate speech and misinformation detection, finding polarized behavior across identity groups and partisan sources despite stable overall performance.
Problem
The paper addresses the limited evidence on how naturally occurring media biases in pretraining data affect LM political leanings and fairness in downstream social-oriented tasks.
Method
The authors quantify LM political leanings on social and economic axes, continue pretraining on partisan corpora, and evaluate downstream classifiers under controlled conditions.
Results
Different pretrained LMs reinforce pretraining-corpus polarization and show significantly different behavior across hate speech identity groups and partisan misinformation sources despite consistent overall performance.
Takeaways & Limitations
Political biases in pretraining data can propagate through LMs into downstream social-oriented tasks and directly affect model fairness.
Takeaways & Limitations
The proposed political-leaning representation is coarse because a model can be liberal on one issue and conservative on another.
Abstract
from arXiv · showhide
Language models (LMs) are pretrained on diverse data sources, including news, discussion forums, books, and online encyclopedias. A significant portion of this data includes opinions and perspectives which, on one hand, celebrate democracy and diversity of ideas, and on the other hand are inherently socially biased. Our work develops new methods to (1) measure political biases in LMs trained on such corpora, along social and economic axes, and (2) measure the fairness of downstream NLP models trained on top of politically biased LMs. We focus on hate speech and misinformation detection, aiming to empirically quantify the effects of political (social, economic) biases in pretraining data on the fairness of high-stakes social-oriented tasks. Our findings reveal that pretrained LMs do have political leanings that reinforce the polarization present in pretraining corpora, propagating social biases into hate speech predictions and misinformation detectors. We discuss the implications of our findings for NLP research and propose future directions to mitigate unfairness.
1 Introduction
The paper examines how political and social biases in naturally occurring pretraining data shape language models and fairness in downstream hate speech and misinformation detection. It proposes measuring LM political leanings and tracing their effects across these tasks.
- Political discussions can promote diverse perspectives while also reflecting and reinforcing societal stereotypes that enter LM pretraining data.
- Prior work studied encoded language biases and annotator effects, but rarely analyzed naturally occurring media biases in pretraining data.
- The study evaluates fairness in hate speech detection across identity groups and misinformation detection across partisan leanings.
- The authors measure LM political leanings, continue pretraining on partisan corpora, and evaluate classifiers built on LMs with different leanings.
- Experiments show that LMs reinforce pretraining-corpus polarization and produce different behaviors across identity groups and partisan media sources despite consistent overall task performance.
- The paper contributes methods for quantifying LM political bias and discusses how ideological polarization propagates into social-oriented downstream tasks.
2 Methodology
The methodology measures LM ideology on social and economic axes, then tests whether partisan pretraining changes downstream behavior under controlled conditions. It combines political-compass-based probing with hate speech and misinformation evaluations.
- The study uses a two-step methodology: measuring pretrained LM political leanings and testing their effects on downstream social-oriented tasks.
- The political-leaning framework assesses social liberalism–conservatism and economic left–right positions rather than only a single left–right distinction.
- The political compass test maps responses to 62 political statements onto social and economic scores ranging from −10 to 10.
- Generation-based probing uses stance detection across 10 random seeds, filters low-confidence responses, and averages scores for evaluation.
- Partisan LMs are created through continued pretraining on established media corpora from different political viewpoints, followed by ideological-coordinate evaluation.
- Downstream analysis compares overall and per-category performance while holding the starting checkpoint, fine-tuning data, and hyperparameters constant.
3 Experiment Settings
The experiments span diverse LM architectures, partisan news and social-media corpora, and hate speech and misinformation datasets. They compare both aggregate task performance and behavior across socially or politically defined categories.
- Political-bias evaluation covers 14 language models and their variants, including encoder-only, autoregressive, and decoder architectures.
- Human evaluation found stance-detector accuracy of 0.97 on clear LM responses, with Fleiss’ Kappa of 0.85 among three annotators.
- Partisan pretraining corpora vary by domain—news or social media—and political leaning—left, center, or right.
- The downstream tasks are hate speech detection using identity-group splits and misinformation detection using a dataset containing news-source information.
- The experiment evaluates RoBERTa and four variants further pretrained on REDDIT-LEFT, REDDIT-RIGHT, NEWS-LEFT, and NEWS-RIGHT corpora.
- The selected downstream tasks and datasets are not exhaustive, with other possible task and dataset choices left for future work.
4 Results and Analysis
The results show that pretrained language models occupy distinct political positions, acquire bias from partisan corpora, and transmit those differences into downstream fairness behavior. Political leaning varies across architectures, issue axes, corpus types, and training periods, while overall task performance can remain similar.
- Political Leanings of Pretrained LMs: Language models occupy all four political-compass quadrants and differ in their underlying ideological leanings.
- Political Leanings of Pretrained LMs: BERT variants are generally more socially conservative than GPT variants, possibly reflecting differences in pretraining corpora and reinforcement-learning data.
- Political Leanings of Pretrained LMs: Pretrained LMs show stronger bias toward social than economic issues, with average magnitudes of 2.97 and 0.87, respectively.The corresponding standard deviations are 1.29 for social issues and 0.84 for economic issues.
- The Effect of Pretraining with Partisan Corpora: Partisan continued pretraining shifts LMs toward the political orientation of the corpus, although most shifts are relatively small.RoBERTa pretrained on REDDIT-LEFT shifted in social values from 2.97 to −3.03.
- Pre-Trump vs. Post-Trump: Post-Trump corpora generally move RoBERTa further from the political center, indicating that LMs can pick up heightened polarization in news and social media.
- The Effect of Pretraining with Partisan Corpora: Increasing partisan corpus size and training epochs does not push models toward social extremes of 10 or −10, while economic scores remain near the center.
- Political Leaning and Downstream Tasks: Overall hate-speech and misinformation performance remains consistent across politically biased LMs, but REDDIT-RIGHT greatly trails vanilla RoBERTa and left-leaning LMs generally slightly outperform right-leaning ones.
- Political Leaning and Downstream Tasks: Political leanings produce different identity- and source-specific behaviors: left-leaning models favor minority-group hate-speech detection and scrutinize right-leaning misinformation more strictly, with right-leaning models showing opposite patterns.
5 Reducing the Effect of Political Bias
The paper discusses partisan ensembles and strategic pretraining as ways to reduce or exploit political bias in downstream NLP tasks. These approaches can improve performance or scenario-specific sensitivity, but ensembles add computational and human-evaluation costs, while suitable corpora can be difficult to curate.
- Different political biases produce different downstream predictions, including differing judgments about offensive content and misinformation.
- Partisan Ensemble: Partisan ensembles combine LMs with different political leanings to incorporate multiple perspectives into downstream decisions.The approach is motivated by models’ differing strengths and weaknesses across tasks.
- Partisan Ensemble: Table 6 reports that partisan ensembles improve task performance by engaging diverse political perspectives.
- Partisan Ensemble: Partisan ensembles may require additional computation and human evaluation to resolve differences among models.
- Strategic Pretraining: Strategic pretraining can increase sensitivity to political perspectives that differ from a model’s own, such as identifying inconsistencies in New York Times news after right-leaning pretraining.
- Strategic Pretraining: Scenario-specific strategic pretraining may yield substantial improvements, but curating ideal corpora for each scenario is challenging.
6 Related Work
Prior NLP research has examined political and societal bias in users, news, entities, generative models, datasets, and downstream systems. This study extends that work by tracing political bias from training data through language models to fairness in hate speech and misinformation detection.
- Earlier studies measured political bias in individual users, news articles, political entities, and generative models.
- Prior fairness research showed that downstream model performance can vary substantially across identity groups.
- Existing work commonly attributed downstream effects to annotator and data bias, with some studies connecting training data to model behavior.
- This study links political data bias to model bias and downstream-task fairness in a more complete pipeline.
- The study also highlights fairness concerns in misinformation detection, an area less emphasized by prior work focused mainly on hate speech detection.
7 Conclusion
The paper presents a systematic framework for measuring political leanings in language models and examining how those leanings affect downstream fairness. It concludes that subtle imbalances in otherwise non-toxic, diverse data can produce unfair downstream behavior, motivating mitigation strategies that preserve opinion diversity.
- The study probes language models with politically grounded prompts to measure ideological positions along social and economic dimensions.
- It examines how political biases in pretraining data influence LM leanings and downstream behavior in hate speech and misinformation tasks.
- Models with different political biases apply different standards to hate speech targets and misinformation sources.
- Non-toxic data containing diverse opinions can still produce downstream unfairness when its distributions contain subtle imbalances.
- The paper discusses mitigation strategies intended to reduce negative impacts while preserving diversity of opinions in pretraining data.
Limitations
The paper identifies limitations in its political-leaning measurement framework and probing methodology. Its two-axis test is geographically and theoretically incomplete, while response evaluation can be subjective; political leaning may also require finer-grained, issue-specific analysis.
- The Political Compass Test: The political compass test uses only social and economic axes, although other political theories offer alternative ideological categorizations.
- The Political Compass Test: The test focuses heavily on Western ideological issues and has been criticized for unclear scoring, libertarian bias, and vague statements.
- The Political Compass Test: The authors’ general probing methodology is designed to remain compatible with other ideological theories, tests, and questionnaires.
- Probing Language Models: For generation models, prompted responses are evaluated with a stance detector, making response mapping and hyperparameter choices potentially subjective.
- Probing Language Models: Multi-choice probing gives more direct answers but is less interpretable and performs poorly with smaller models such as GPT-2.
- Fine-Grained Political Leaning Analysis: Representing political leaning with two numerical values can obscure issue-specific variation, since a model may be liberal on one issue and conservative on another.
Ethics Statement
The ethics statement identifies limits on the paper’s cultural scope, interpretation, and potential misuse. It also cautions that the authors’ own political leanings may have influenced the analysis.
- The paper’s U.S.-centric framing of minority identity groups may not apply universally across cultures and contexts.
- Although the study found that more partisan pretraining does not simply produce hyperpartisan LMs, future malicious attempts may still succeed.Potential harms include partisan misinformation and adversarial attacks, motivating restricted release of checkpoints and partisan corpora.
- The authors caution that differences in downstream behavior should not be interpreted as judgments of people or communities with particular political leanings.
- The authors acknowledge that their inherent political leanings may have affected experiment interpretation and analysis in unperceived ways.
A.2 Decoder-Based LMs
The decoder-based LM procedure generates responses to political propositions and uses stance detection to classify agreement. Human evaluation supports the detector’s accuracy and motivates filtering low-confidence responses.
- Decoder-based LMs are evaluated through prompted text generation followed by a BART-based stance detector that maps responses to four agreement categories.
- 80 of 110 sampled LM responses expressed a clear stance, with Fleiss’ Kappa of 0.85 among annotators.
- 97% accuracy was achieved by the BART-based stance detector on the 80 responses with clear stances.
- The authors describe the political-leaning probing framework and commit to making its code and data publicly available upon acceptance.
- The framework uses additional reporting of false positives and false negatives through precision and recall on the downstream tasks.
C Experiment Details
The experiment details document the model checkpoints, corpus statistics, hyperparameters, computational resources, and approximate training times used in the study.
- Specific language-model checkpoints are listed in Table 10, while social-media corpus statistics appear in Table 8 and news-corpus statistics are referenced elsewhere.
- Table 8 notes that pre- and post-Trump social-media counts may not sum to the total because some PushShift posts lack timestamps.
- Table 9 reports the hyperparameter settings for pretraining and fine-tuning, with defaults used when settings are not included.
D Stability Analysis
The stability analysis tests whether political-leaning measurements persist under changes to prompts and statement phrasing, alongside qualitative downstream-task comparisons and implementation details. Political opinions are moderately stable overall, while GPT-3 DaVinci is most consistent under prompt changes, and politically different LMs can behave very differently on identity-targeted hate speech.
- D Stability Analysis: Because pretrained LMs can be sensitive to minor input perturbations, the study tests political-leaning stability under changes to prompts and political statements.
- D Stability Analysis: Seven prompt formats are evaluated; GPT-3 DaVinci gives the most consistent responses, while all pretrained LMs are moderately stable.
- D Stability Analysis: The analysis uses three paraphrasing models and three online paraphrasing tools, with seven manually designed prompts listed in Table 11.
- Qualitative Analysis: Qualitative examples show that pretrained LMs with different political leanings exhibit vastly different behavior when evaluating hate speech targeting different identities.
- Experiment Setup: The study further pretrains LM checkpoints on partisan corpora and fine-tunes them on downstream tasks using reported hyperparameters.
- Experiment Setup: Pretraining RoBERTa-base and GPT-2 takes approximately 48 and 83 hours, respectively, while downstream fine-tuning takes approximately 30 and 20 minutes.