PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

Mark Mets, Andres Karjus, Indrek Ibrus, Maximilian Schich

arXiv:2305.13047v1cs.CL

TL;DR

The paper asks whether automated stance detection can work for politically complex immigration coverage in Estonian, a lower-resource and morphologically complex language. It annotates sentences, fine-tunes language models, compares them with ChatGPT zero-shot classification, and applies the best model to seven years of news. Est-RoBERTa achieves F1 macro 0.66 and ChatGPT 0.65, while the analysis identifies stance differences and temporal changes across mainstream and radical-right media.

  • Problem

    The paper asks whether automated stance detection can work for politically complex topics in lower-resource languages with limited annotated NLP resources.

    Source: Introduction · p. 1

  • Method

    The authors annotate Estonian immigration sentences, fine-tune multiple pretrained language models, compare them with ChatGPT zero-shot classification, and apply the best model to news corpora.

    Source: Introduction · p. 3, Automatic classifiers · p. 10

  • Results

    Est-RoBERTa achieves F1 macro 0.66, while zero-shot ChatGPT achieves 0.65, with stance differences and temporal changes identified across the news sources.

    Source: Results · p. 11, Discussion · pp. 17–18

  • Takeaways & Limitations

    The results support automated stance detection as a potentially useful approach for media monitoring and analytics beyond large languages.

    Source: Discussion · p. 17, Conclusions · p. 19

    Performance is constrained by annotation ambiguity, annotator knowledge and bias, implicit opinions, quotations, sarcasm, metaphor, and missing paragraph-level context.

    Source: Limitations · p. 13

Abstract

from arXiv · show

Automated stance detection and related machine learning methods can provide useful insights for media monitoring and academic research. Many of these approaches require annotated training datasets, which limits their applicability for languages where these may not be readily available. This paper explores the applicability of large language models for automated stance detection in a challenging scenario, involving a morphologically complex, lower-resource language, and a socio-culturally complex topic, immigration. If the approach works in this case, it can be expected to perform as well or better in less demanding scenarios. We annotate a large set of pro and anti-immigration examples, and compare the performance of multiple language models as supervised learners. We also probe the usability of ChatGPT as an instructable zero-shot classifier for the same task. Supervised achieves acceptable performance, and ChatGPT yields similar accuracy. This is promising as a potentially simpler and cheaper alternative for text classification tasks, including in lower-resource languages. We further use the best-performing model to investigate diachronic trends over seven years in two corpora of Estonian mainstream and right-wing populist news sources, demonstrating the applicability of the approach for news analytics and media monitoring settings, and discuss correspondences between stance changes and real-world events.

Introduction

The paper tests automated immigration-stance detection in Estonian, a morphologically complex lower-resource language, using supervised and zero-shot language-model approaches. It situates the task within media monitoring and examines immigration discourse across mainstream and radical-right media.

Introduction

  • Estonian immigration news provides a challenging test case because the language is lower-resource and morphologically complex, while the topic is politically polarized.The study targets a topic requiring cultural and contextual interpretation in a language with limited NLP resources.Source: Introduction · pp. 1–2
  • The study aims to assess whether automated stance detection can estimate the balance of immigration views in news media and support media monitoring.The corpus contrasts mainstream and right-wing populist sources expected to express different positions.Source: Introduction · p. 2
  • The paper compares supervised language-model classifiers with ChatGPT zero-shot classification and evaluates their feasibility for lower-resource stance detection.It also makes an annotated dataset public and compares the approach with cheaper sentiment-based bootstrapping.Source: Introduction · pp. 2–3

Analytic approach

  • The analysis treats stance as favorability toward a specified target, distinguishing the topic-dependent task from general sentiment analysis.The target selected here is immigration, with sentiment analysis used as a comparison approach.Source: Analytic approach · pp. 3–4
  • Prior work spans supervised, multi-shot, and zero-shot stance detection, but evidence for lower-resource languages remains comparatively limited.Existing studies have applied multilingual and cross-topic approaches beyond English, including Italian, French, German, and under-resourced settings.Source: Analytic approach · pp. 4–5

Object of analysis

  • Immigration is examined because media representations can shape perceptions and voting, and the topic is central to radical-right populist rhetoric.In Estonia, Uued Uudised is described as a relevant source for radical-right perspectives on immigration.Source: Object of analysis · p. 5
  • Automated stance and sentiment detection offers a complementary large-scale quantitative perspective on under-researched Estonian radical-right discourse.The study therefore connects methodological testing with analysis of media polarization and immigration discourse.Source: Object of analysis · p. 6

Methods and materials

The authors construct and annotate an Estonian immigration-sentence dataset, then compare fine-tuned transformer classifiers with ChatGPT prompting. The design uses sentence-level analysis, balanced sampling by publisher and keywords, and explicit handling of annotation and contextual ambiguity.

Dataset

  • The corpus comprises articles from 2015 to early April 2022, contrasting mainstream Ekspress Grupp media with the populist radical-right online portal Uued Uudised.It includes 21 667 Uued Uudised articles (April 2015 to April 2022) and 244 961 Ekspress Group articles (January 2015 to March 2022), with an incomplete gap in October-December 2019.Source: Dataset · pp. 6–7
  • Sentence-level analysis limits cross-sentence context but helps isolate a single stance and fit BERT’s input-length constraints.The authors expect sentences to indicate rhetorical and large-scale temporal changes despite this limitation.Source: Dataset · p. 7
  • Immigration-related sentences were retrieved with morphologically aware keyword regular expressions covering migration, refugees, nationality, race, and related political topics.A negative filter removed unrelated uses such as bird migration.Source: Dataset · p. 7

Annotations

  • Two Estonian-speaking graduate students annotated 8000 sentences on a five-point pro-to-anti scale, yielding 7345 retained examples across Against, Neutral, Supportive, and Ambiguous classes.Sampling was balanced by keyword prevalence and publisher, while ambiguous or unsuitable sentences were excluded from model tuning.Source: Annotations · p. 8
  • Annotation agreement was substantial for Supportive, Against, and Neutral classes, with κ = 0.69 and 0.66, and very strong for Pro versus Against at κ = 0.97.Most disagreement occurred between an extreme stance and Neutral.Source: Annotations · p. 9

Automatic classifiers

  • The study fine-tuned multilingual and Estonian-specific BERT-family models, using class weights and a 20% evaluation set.Models included mBERT, XLM-RoBERTa, EstBERT, and Est-RoBERTa.Source: Automatic classifiers · pp. 9–10
  • ChatGPT 3.5 was evaluated as a zero-shot classifier through natural-language prompts that assigned Against, Neutral, or Supportive labels to sentence batches.Invalid labels or tag counts were resubmitted until the requested label format was returned.Source: Automatic classifiers · p. 10

Results

Est-RoBERTa achieved acceptable stance-detection performance, while the error analysis identified contextual and annotation challenges. Applied across seven years of Estonian news, the model revealed publisher differences and stance changes corresponding to major media events.

Results

  • Est-RoBERTa achieved F1 macro 0.66, compared with 0.64 for both EstBert and XLM-RoBERTa, while zero-shot ChatGPT showed comparable performance.Est-RoBERTa scored F1 0.74 for Against, 0.69 for Neutral, and 0.55 for Supportive; the confusion was mostly between Neutral and one extreme.Source: Results · p. 11, Table 1 · p. 11
  • Most classifier errors occurred between Neutral and one stance extreme rather than between Against and Supportive.The confusion matrix describes small values for the two extreme-confusion cells as a favorable pattern.Source: Results · p. 11, Figure 2 · p. 12

Limitations

  • Annotation and classification errors involved mistaken labels, sarcasm, ambiguous or context-dependent sentences, and references to third-party opinions.The authors note that paragraph-level analysis might provide more context than sentence-level analysis.Source: Results · p. 12
  • Classifier performance was partly constrained by annotation ambiguity, annotator knowledge and biases, and possible biases in language-model training data.Neutral-versus-ambiguous distinctions could reduce the effective training set, while annotators sometimes relied on contextual knowledge despite instructions to rate sentences alone.Source: Limitations · p. 13

Exemplary analysis

  • Uued Uudised focused more on immigration and reacted more strongly to events such as the 2015–2016 migration crisis, the 2018 UN migration pact, and Russia’s 2022 invasion of Ukraine.The relative amount of immigration-related coverage differed between publishers, with Uued Uudised showing stronger event-related fluctuations.Source: Exemplary analysis · p. 14, Figure 3 · p. 14
  • Uued Uudised was generally Against immigration, whereas Ekspress Grupp was predominantly Neutral and more stable over time.Stance distributions varied by keyword group, with multiculturalism, xenophobia, and race-related terms having the highest Against percentages.Source: Exemplary analysis · p. 15
  • The Against stance increased around the 2018–2019 UN migration pact debates and elections, while the 2022 Ukraine war corresponded to increased Supportive stance, especially in Ekspress Grupp.Sentence-level cosine similarity also spiked in late 2018 and 2022, possibly reflecting shared attention to events or similar rhetoric.Source: Exemplary analysis · pp. 15–17, Figure 6 · p. 17

Discussion

Automated stance detection is feasible for complex immigration discourse in Estonian, but annotation costs, contextual ambiguity, and reproducibility concerns constrain its use.

Discussion

  • Automated stance detection was feasible, achieving satisfactory F1 macro 0.66 with Est-RoBERTa and a similar 0.65 with zero-shot ChatGPT for media monitoring and analytics.Source: Discussion · p. 17
  • Against stances were more accurately classified than Supportive stances, while radical-right media was generally more anti-immigration than mainstream news.The classifier also identified stance changes associated with major immigration-related events between 2015 and 2022.Source: Discussion · p. 18
  • Sentence-level analysis is efficient but may lack context, whereas paragraph-level analysis can contain multiple stances and complicate classification.The unit of analysis therefore constrains both human annotation and machine learning.Source: Discussion · p. 18

Future research

  • Annotated data make supervised stance detection time consuming, expensive, and limited to one topic at a time.This motivates interest in generative models that can be instructed across languages, tasks, and topics.Source: Future research · p. 18
  • Zero-shot performance requires continued evaluation because prompt variation, model stochasticity, and cloud-model updates can hinder replication.The authors also emphasize evaluating accuracy and model bias when using generative models.Source: Future research · p. 19

Conclusions

The paper demonstrates pretrained-LLM stance detection for complex immigration discourse in Estonian and applies it to seven years of mainstream and radical-right news coverage.

Conclusions

  • Pretrained LLMs and an instructable zero-shot ChatGPT classifier provide a proof of concept for stance detection in smaller languages.The paper also releases a stance-annotated dataset.Source: Conclusions · p. 19
  • Applying one classifier to Estonian news from 2015–2022 revealed changes in immigration discourse across mainstream and radical-right sources.The findings support prior literature while providing additional insights into media trends.Source: Conclusions · p. 19

Declarations

The study received co-funding for annotation, data acquisition, and preliminary analysis, while the funder did not influence study design or conclusions.

Declarations

  • Ekspress Grupp co-funded annotation, data acquisition, and preliminary analysis without influencing the study design or conclusions.Several authors were also supported by the CUDAN ERA Chair project funded through Horizon 2020.Source: Declarations · p. 19

1. Keywords details

The keyword search organizes immigration-related language into topical groups covering migration, affected populations, and ideological debate.

1. Keywords details

  • The search uses regular expressions to capture word forms while filtering non-topical matches across eight original-language topics.The keyword design includes both topical stems and exclusion filters for unrelated uses.Source: §1 · p. 25
  • The topical groups cover migration, refugees, foreign workers, foreign students, and noncitizens.Examples include migration, asylum seekers, border control, foreign labor, international students, residence permits, and non-Estonians.Source: §1 · p. 25
  • Additional groups capture race, ethnicity, and radical-right versus liberal-opposition discourse.The latter includes terms related to globalism, open borders, xenophobia, and multiculturalism.Source: §1 · pp. 25–26

2. Data distribution

The dataset distribution compares Estonian mainstream and populist radical-right sources across periods, publications, and immigration-related topics. Delfi dominates the Ekspress Grupp material, while the outlets differ in topical emphasis and weekly-to-monthly variation.

  • Delfi provides the main share of Ekspress Grupp articles, while later data includes more of the group’s other publications.The Ekspress Grupp data covers multiple periodicals, but the online platform Delfi is the dominant source.Source: Figure S1 · p. 26, §2 · p. 27
  • Refugee and migration keywords account for most keyword-group mentions, whereas foreign-student mentions are relatively rare.Source: Figure S2 · p. 27
  • Uued Uudised has about twice as many sentences on refugee and foreign-workforce topics, while Ekspress Grupp uses more xenophobia and multiculturalism keywords.Source: Figure S2 · p. 27
  • Weekly distributions provide a more detailed view of immigration-related events than monthly trends.Monthly trends depend on more specific weekly changes.Source: Figure S4 · p. 28, §2 · p. 29

3. Explanations on LLM and automatic classifier

The paper explains LLM classification through context-sensitive text embeddings and task-specific fine-tuning. It contrasts context-sensitive models with one-vector-per-word representations and describes evaluation on held-back data.

  • Text is transformed into multidimensional embeddings whose distances represent relations between words or whole sentences.The embedding dimensions encode abstract features, although their individual meanings are difficult to identify.Source: §3 · p. 29
  • The classifier learns category assignments through masked-language-model training and is fine-tuned for a specific task.Generalization is assessed using data withheld from training.Source: §3 · p. 29
  • Context-sensitive models such as ELMo and BERT assign word representations according to surrounding text, unlike Word2Vec and GloVe.BERT considers text before and after a word simultaneously.Source: §3 · pp. 29–30

3. Examples of stance annotations

The section provides examples from the final stance training set to illustrate the categories used for classification.

  • Table S1 presents original-punctuation examples representing each category in the final stance training set.Source: Table S1 · p. 30
  • The examples document how sentences were represented in the finalized training data.Source: Table S1 · p. 30
  • The table preserves the examples’ original punctuation.Source: Table S1 · p. 30

4. Annotation interrater scores

The section reports annotation, model-evaluation, error-analysis, and longitudinal stance findings. Results include model metrics, confusion patterns, keyword-group differences, threshold robustness, and sentence-embedding trends.

8. Detailed results of models

  • Supportive-class results vary across reported model blocks, including F1-score values of 0.57, 0.58, and 0.64.Source: §8 · p. 35
  • Table S5 reports detailed classification results per model, including macro-average F1, precision, and recall values of 0.65, 0.66, and 0.66.Source: §8 · p. 35, Table S5 · p. 36

9. Confusion matrix of sentiment analysis

  • The sentiment-analysis confusion matrix shows more mistakes between supportive and Against classes and more neutral misclassification than stance detection.A larger percentage of sentences was correctly classified as negative.Source: Figure S5 · p. 36

10. Stances of keyword groups

  • Across keyword groups, average yearly stance was relatively stable, with the most negative framing for race, followed by nationalities and migrants more generally.The authors caution that this ordering averages years and uses relatively fixed keywords, while topics can be more nuanced than the approach distinguishes.Source: §10 · p. 37
  • In 2018–2019, Uued Uudised became more negative across nearly all keyword groups, while Ekspress changes varied by topic.In 2022, Uued Uudised showed more negative immigration sentences amid a possible shift toward war and refugees, while Ekspress showed a sharp decrease in Against stances related to radical-right liberal opposition.Source: §10 · p. 37
  • Figures S6–S9 track yearly Against-stance changes by keyword group, with Ekspress becoming more positive in 2022 for all topics except foreign students.The supplied caption highlights especially large changes for xenophobia/multiculturalism and refugee keywords, plus a slow increase in supportive foreign-workforce stances.Source: §10 · pp. 38–39

11. Stance trends with threshold

  • Using probability thresholds has little effect on general stance trends, while below-threshold sentences comprise about 5–10% and are more common in Ekspress Grupp.Source: §11 · pp. 40–41

12. Sentence embeddings trends

  • Figure S12 compares monthly within-stance sentence cosine similarities separately for each stance.Source: Figure S12 · p. 41
Loading 2305.13047v1…