PaperlayerView exact revision on arXiv

Source-linked AI summary

Ideology Prediction of German Political Texts

Sinclair Schneider, Florian Steuber, Joao A. G. Schneider, Gabi Dreo Rodosek

arXiv:2605.14352v1cs.CL

TL;DR

The paper addresses the challenge of measuring German political orientation continuously rather than through fixed categories. It trains transformer-based multilabel party classifiers, projects their outputs onto a -1 to 1 spectrum, and tests them across independent datasets. DeBERTa-large performs best in-domain, while out-of-domain performance varies by dataset, with Gemma-2-2B strongest on newspapers.

  • Problem

    Existing political ideology classifiers rely on categorical outputs that cannot measure orientation continuously or transfer straightforwardly across political contexts.

  • Method

    The paper trains multilabel transformer classifiers, associates German parties with normalized vectors, and converts classifier outputs into a continuous left-right score.

  • Results

    DeBERTa-large achieves the strongest in-domain performance, while Gemma-2-2B outperforms it on an out-of-domain test, showing that architecture and training data matter alongside model size.

  • Takeaways & Limitations

    The method recognizes German political stances at a level comparable to public-opinion polls and supports periodic retraining by updating training datasets rather than manually labeling anew.

  • Takeaways & Limitations

    The classifier lacks reasoning and interpretability, may misclassify quoted opinions or novel concepts, and should not be used to block texts without additional verification.

Abstract

from arXiv · show

Elections represent a crucial milestone in a nation's ongoing development. To better understand the political rhetoric from various movements, ranging from left to right, we propose a transformer-based model capable of projecting the political orientation of a text on a continuous left-to-right spectrum, represented by a normalized scalar d between -1 and 1. This approach enables analysts to focus on specific segments of the political landscape, such as conservatives, while excluding liberal and far-right movements. Such a task can only be achieved with multiclass classifiers, provided that the desired orientation is incorporated within one of their predefined classes. To determine the most suitable foundation model among 13 candidate transformers for this task, we constructed four distinct corpora. One corpus comprised annotated plenary notes from the German Bundestag, while another was based on an official online decision-making tool, Wahl-O-Mat. The third corpus consisted of articles from 33 newspapers, each identified by its political orientation, and the fourth included 535,200 tweets from 597 members of the 20th and 21st German Bundestag. To mitigate overfitting, we used two distinct corpora for training and two for testing, respectively. For in-domain performance, DeBERTa-large achieved the highest F1 score F1=0.844 as well as for the X (Twitter) out-of-domain test ACC=0.864. Regarding the newspaper out-of-domain test, Gemma2-2B excelled (MAE = 0.172). This study demonstrates that transformer models can recognize political framing in German news at the level of public opinion polls. Our findings suggest that both the model architecture and the availability of domain-specific training data can be as influential as model size for estimating political bias. We discuss methodological limitations and outline directions for improving the robustness of bias measurement.

Introduction

The paper develops a transformer-based method that projects German political texts onto a continuous left-to-right spectrum rather than predefined categories. It combines party-based classification with vector projection and evaluates out-of-domain performance across German political data.

  • Motivation: Social media can substantially influence political discourse, motivating tools that analyze political rhetoric and constituents’ preferences.The paper cites the Team Jorge case as an example of social media’s political influence.
  • Contribution: The proposed algorithm maps political texts onto a continuous scale from -1 to 1, with liberal orientation at 0.This design addresses the limited directional focus of topic models and classifiers with predefined categories.
  • Contribution: The paper targets three challenges: continuous left-to-right mapping, adaptation to local political biases, and testing on distinct out-of-domain datasets.The stated approach uses semi-supervised labeling and separate datasets for evaluation.
  • Approach: Political parties are represented by normalized two-dimensional vectors whose angles encode positions on the left-right spectrum.Classifier outputs are multiplied by party vectors, summed, and converted into an angular classification result.
  • Contribution: The study extends categorical ideology prediction toward a continuous spectrum and reports out-of-sample evaluation on German political texts.Its contribution includes a German-context evaluation using multiple self-collected datasets.

Related Work

Prior work commonly represents political ideology with discrete categories or scales that are difficult to transfer across political contexts. German-language research and cross-domain studies motivate a continuous, locally adapted approach.

  • Categorical approaches: Political ideology detection typically uses manually annotated left, center, and right classes.Such approaches represent orientation categorically rather than continuously.
  • Alternative scales: Existing studies range from detecting extreme orientations to using broader or increasingly fine-grained multi-class spectra.Examples include lean-left and lean-right categories and classifications with seven or more classes.
  • German context: English-language foundational research often reflects the United States, while translating those datasets is insufficient for German political applications.Researchers therefore collect and label German-specific datasets, including newspaper data.
  • Domain generalization: Ideology models trained in one domain can perform poorly in another, motivating mixed-domain training and comparisons of identical stories across differently biased outlets.This issue has been reported for news-site and social-media settings.
  • Expanded representations: Some related work adds network information or dimensions such as anti-elitism, peoplecentrism, and trustworthiness beyond a traditional political spectrum.These approaches still include frameworks based on hand-labeled discrete categories or multiple dimensions.
  • Limitations: Categorical and ordinal outputs cannot measure the extent of left- or right-leaning perspectives and depend on predefined contextual categories.A left-wing opinion may differ substantially between countries such as the United States and Germany.

Methodology

The pipeline combines multiple German political datasets, automated or existing labels, a binary political-content filter, and multilabel party classification. Party outputs are converted to a continuous spectrum and evaluated on independent domains.

  • Pipeline: The pipeline collects and enriches data, trains binary and multilabel classifiers, converts party outputs to a -1 to 1 spectrum, and evaluates independent test sets.It also compares results before and after vector optimization.
  • Evaluation design: Two training datasets were supplemented with newspapers and tweets for out-of-domain evaluation to reduce overfitting concerns.The Bundestag and Wahl-O-Mat data used an 80:20 split, while the additional datasets supplied independent evaluation domains.
  • Bundestag dataset: Bundestag speeches were auto-labeled from recorded interruptions, producing 32,246 annotated statements before linguistic enrichment.A Llama 3.1 model generated five stylistic versions of each text, expanding the dataset to 449,209 statements.
  • Wahl-O-Mat dataset: The Wahl-O-Mat corpus contains 1,751 statements from elections between 1998 and 2021, with party attitudes coded as approval, neutral, or rejection.Synthetic enrichment increased the labeled set to 87,210 statements.
  • Training data: The combined Bundestag and Wahl-O-Mat training dataset contains 570,416 samples.The resulting dataset was made publicly available.
  • Twitter dataset: The tweet test set contains 535,200 political tweets from 597 Bundestag members, with 89,200 tweets per party and labels based on account affiliation.Each tweet is assigned to one political party.
  • Newspaper dataset: Newspaper ideology labels were based on ratings from 1,148 participants covering 39 newspapers and were checked against independent media-bias and scientific sources.The ratings were z-transformed for comparability and used to estimate convergent validity.
  • Label validation: The newspaper ratings correlated highly with independent sources, including ρ = .95 for 19 outlet pairs and r = .95 for 21 outlets from a scientific study.The authors therefore considered the validity of the newspaper ratings sufficient.

Models

The study compares 13 transformer classifiers spanning smaller German-focused encoders and larger multilingual decoders. Models are trained for multilabel party prediction before evaluation on political text domains.

  • Encoder models: Encoder candidates include DeBERTa Large, GottBERT Large, GBERT, GELECTRA Large, XLM-RoBERTa Large, and EuroBERT.Several are German foundation models trained on German text collections.
  • Decoder models: The decoder candidates include Gemma and Llama models trained on multilingual data that includes German.Gemma models use a 256,000-entry tokenizer and smaller versions use knowledge distillation.
  • Preprocessing: A binary classifier first filters political from non-political newspaper content so non-political articles do not shift average ideology estimates toward the center.The filtered data then supports political-orientation classification.
  • Political party classifiers: The 13 classifiers use multilabel outputs corresponding to six major German parties and are evaluated on out-of-domain newspaper data after training.Training uses lines fed into pretrained foundation models followed by four epochs of fine-tuning.
  • Implementation: Training resources range from four A6000 Ada GPUs to eight H200 GPUs, with model-specific parameter and evaluation details summarized in Table 1.The table also reports training hours and hardware configurations.
  • Training constraints: Because larger models required several days to train, some configurations were stopped when training loss no longer decreased instead of receiving full runs.DeBERTa was executed multiple times to identify an optimal hyperparameter configuration.

From Multilabel to a Continuous Scale

The method maps six party positions onto a continuous left-right spectrum by combining party vectors with multilabel classifier outputs. Party placement is anchored geometrically and refined using Wahl-O-Mat response similarities.

  • Party-vector representation: Six German parties are represented by unit vectors positioned on a left-right geometric continuum, with Die Linke, FDP, and AfD serving as fixed reference points.Intermediate party positions are assigned relative to these references.
  • Party positioning: Wahl-O-Mat response overlap determines whether Grüne is closer to Die Linke or FDP.Identical, partially differing, and completely opposing responses receive distances of 0.0, 0.5, and 1.0, respectively.
  • Party positioning: The party distance is d(a,b) := (0.5 · P + O)/T, where T = I + P + O.For the example, the distances are dB′90,Linke = 0.208 and dB′90,FDP = 0.547.
  • Party positioning: Relative proximity to reference parties is converted into party angles, yielding θB′90 ≈ −65.2°, θSPD ≈ −53.9°, and θCDU ≈ 37.9°.The arctan2 function determines the appropriate circle quadrant during implementation.
  • Continuous-score calculation: Classifier probabilities pi are multiplied by party vectors vi, summed as vres = Σ pi vi, and converted into a normalized score using atan2.The resulting angle is divided by π/2 to obtain the final classification score.

Overall Architecture

The architecture first identifies political text, then applies party classifiers whose probability outputs are combined with party vectors. The resulting angle is normalized to produce a continuous ideological score.

  • Political text detection: A DeBERTa political classifier first determines whether the example sentence is political, producing a score of 0.99.The example is therefore passed to the party-classification stage.
  • Party classification: Political text is processed by trained party classifiers that output probabilities for six German parties.In the example, Gemma2-9b produces probabilities for Linke, B′90, SPD, FDP, CDU, and AfD.
  • Vector aggregation: The party probabilities are multiplied by their vectors and summed into a combined result vector vres = (−0.159, 0.277).The combined vector is then converted into an angle using atan2.
  • Score normalization: The angle is divided by 90° or π/2 to convert it from the interval −90° to 90° into a score between −1 and 1.For the family-policy example, the resulting score is approximately −0.332, indicating a slightly left-leaning vector.

Evaluation Using Newspapers

Newspaper evaluation compares article-level classifier predictions with newspaper-level ideological ratings. Mean absolute error aggregates the differences across tested newspapers.

  • Reference ratings: Bild receives a Mediencompass.org rating of 0.4 on the [−1, 1] scale, indicating a slightly right-wing orientation.The classifier produces article-level estimates, whereas the reference rating is defined at newspaper level.
  • Aggregation: The evaluation aggregates political-leaning predictions for each article within a newspaper before comparing them with the newspaper’s expected leaning.This aggregation addresses the mismatch between article-level predictions and newspaper-level ground truth.
  • Evaluation metric: Mean absolute error is computed as the average absolute difference between predicted and expected leanings across tested newspapers.The metric summarizes newspaper-level out-of-domain performance.

Final Optimization

The final optimization adjusts party-vector positions to reduce newspaper mean absolute error while constraining the magnitude of each adjustment. Extreme party positions remain fixed to preserve model functionality.

  • Optimization design: Party vectors are optimized against newspaper data, with each adjustable vector limited to a maximum shift of 0.25 in either direction.The procedure builds on the initial Wahl-O-Mat-based vector alignment.
  • Optimization design: The optimization holds Die Linke and AfD fixed while allowing adjustments to the liberal FDP and other non-extreme positions.This preserves the leftmost and rightmost anchors during refinement.
  • Evaluation: The comparison evaluates used models before and after optimization using MSE and MAE, ordered by post-optimization MSE.The table reports both error metrics for the pre/post comparison.
  • Technical constraint: Positive classifier outputs restrict reachable vectors, so moving extreme vectors upward would require infeasible negative contributions.Consequently, adjusting the extreme vectors would restrict the set of reachable vectors.

Results

The transformer classifiers recognized political stances in German texts, with performance varying across in-domain and out-of-domain settings. DeBERTa-large led in-domain evaluation, while Gemma2-2b performed best on newspapers, and tweet accuracy improved with longer texts.

  • DeBERTa-large achieved the highest in-domain performance with F1 = 0.84.
  • 50 or more words raised tweet accuracy to over 80%, compared with 50%–65% for shorter tweets.
  • Gemma2-2b achieved the best newspaper out-of-domain result with MAE of 0.1852 (9.26%).
  • Newspaper model rankings differed from in-domain results, indicating that models varied in how effectively they generalized across text domains.
  • Vector optimization decreased mean absolute error by 0.0239 on average across models.

Discussion

The discussion reports that domain-specific performance varies across transformer models and that the approach supports adaptation without manual relabeling. It also identifies limitations involving reasoning, short texts, ideological representation, political change, cultural transfer, and misuse.

  • Model performance: Training data and model architecture both affect cross-domain performance; Gemma2 models are attributed better generalization to knowledge distillation.DeBERTa-large performed best in-domain, while Gemma2-2B and Gemma2-9B generalized well out of domain.
  • Adaptability: Optimizing the party vectors can further improve out-of-domain accuracy and enables retraining by updating datasets rather than manually relabeling texts.The method is presented as adaptable to changing political landscapes and different countries or use cases.
  • Model limitations: The classifier may misread quoted or foundational statements because it lacks reasoning and cannot determine whether an author endorses quoted content.The paper gives examples involving a quoted SPD-program statement and Proudhon’s “Property is theft.”
  • Model limitations: Very short texts may be difficult to classify, particularly when interpreting them requires background knowledge.The discussion links higher accuracy to tweets containing more words.
  • Methodological limitations: A one-dimensional spectrum can collapse distinct ideological positions, including similar issue responses from Die Linke and AfD.The paper attributes this possibility to shared opposition status or different motivations producing similar positions.
  • Scope limitations: Political shifts and cultural differences constrain applicability, making periodic retraining and adaptation to new linguistic and cultural contexts necessary.The scale is bounded by the six reference parties, so positions beyond Die Linke or AfD cannot be distinguished.
  • Future work and misuse: The authors recommend reasoning-capable classifiers that explain assignments and generalize to unfamiliar topics, while warning that the model could enable discrimination through political-belief monitoring.Explainability would help users identify and contest classification errors.

Ethical Statement

The ethical statement limits the system to academic use of publicly available data and warns against using it to evaluate or discriminate against people based on political beliefs.

  • Data use: The training data came from publicly available materials, with no paywalled or personal data collected except public figures’ political speeches.The authors state that the data is intended solely for academic use.
  • Responsible use: The authors acknowledge that errors remain possible and advise against evaluating or discriminating against individuals based on political beliefs.They also express support for political diversity and respectful free speech.

Paper Checklist

The checklist records affirmative documentation for ethics, reproducibility, training details, evaluation, asset use, and limitations, while several non-applicable items concern theoretical proofs or human participants.

  • Ethics and impact: The paper reports addressing social contracts, data and model documentation, privacy, negative societal impacts, and potential misuse through its ethics and discussion sections.The checklist points readers to the Ethical Statement and relevant Discussion subsections.
  • Theoretical requirements: The checklist marks theoretical-result requirements as not applicable, including assumptions, proofs, competing theories, alternative mechanisms, and links to social-science literature.These entries apply to theoretical results rather than the reported machine-learning experiments.
  • Reproducibility: The paper reports providing code, data, training details, hyperparameters, error bars, compute information, and justification of the evaluation.The checklist refers to the GitHub repository, Figure 7, Table 1, and the Methods and Limitations sections.
  • Assets and datasets: Existing assets are cited and licensed, while new datasets, consent, personally identifiable information, FAIR practices, and model cards are reportedly documented.The checklist directs readers to the References, Methods, Ethical Statement, supplemental links, and HuggingFace model cards.
  • Study participants: Requirements concerning crowdsourcing or human-subject research are marked not applicable, while data storage and deidentification are referenced in the Methods.The checklist also records that the paper describes its limitations and potential negative impacts.
  • Data examples: The checklist identifies Table 4 as presenting party agreement or disagreement for three example statements.The examples concern a wealth tax, retaining the euro, and introducing a minimum wage.
  • Figures: The paper also includes figures for sentiment extraction and optimization effects across models and media outlets, with error bars representing standard errors.Figure 7 reports MAE and MSE comparisons across 13 models and 33 news media outlets.
Loading 2605.14352v1…