PaperlayerView exact revision on arXiv

Source-linked AI summary

Ideological Stance Detection in a Low-Resource Language: Polarization in Bangladeshi Public vs Private University Discourse on Social Media

Safaruzzaman Shovo, Monowar Islam, Asif Hossain, Sameya Akhter, Md. Shamsul Islam

arXiv:2610.04401v1cs.CL

TL;DR

Bangladeshi university debates are polarized on social media, but research has paid limited attention to Bengali educational discourse. This paper analyzes annotated Bengali comments and compares stance-detection models, finding that zero-shot Llama 4 Maverick outperforms the supervised baselines.

Abstract

from arXiv · show

Public vs. private universities is a debatable issue, and it creates polarization on social media in Bangladesh. Debate on quality, jobs, and prestige is passionate among the students, parents, and graduates, the majority of whom speak Bengali, a low-resource language. To measure this polarization, this paper introduces a manually annotated dataset of 4,060 Bengali comments labeled as Pro-Public, Pro-Private, or Neutral. We evaluated the quality of our annotations by Fleiss's Kappa agreement that was 0.89, corresponding to a high agreement among annotators. The classical ML (SVM, Random Forest, XGBoost), BiLSTM network, hybrid BanglaBERT+XGBoost models and the state-of-the-art zero-shot LLMs (Claude Sonnet 4, DeepSeek-V3.1, Llama 4 Maverick, Kimi K2 Thinking, Qwen3-235B Thinking) models are evaluated. The accuracy of BanglaBERT+XGBoost is 91.81% and macro F1 score is 91.70%, which is higher than all the supervised baselines. The zero-shot Llama 4 Maverick Thinking achieves a macro F1 of 0.931 (overall accuracy of 93.31%) without any fine-tuning. All machine learning (ML), deep machine learning (DL) and transformer models were outperformed by the zero-shot Llama 4 Maverick model. Polarization also is evident, in some ways more clearly in the Pro-Private comments, which emphasize modern facilities and timely graduation, versus the Pro-Public comments, which emphasize affordability and government jobs. Our findings open new directions for analyzing social media polarization in low-resource languages.

I. INTRODUCTION

The paper examines polarized debate over Bangladeshi public and private universities in Bengali social-media comments, addressing limited research on organic educational discourse. It introduces an annotated dataset and compares multiple stance-detection models.

I. INTRODUCTION

  • Public universities are associated with low tuition and prestige but also with session delays, conflict, limited seats, and outdated infrastructure.Private universities are described as offering modern facilities and timely graduation, while facing criticism for high tuition and quality.Source: §I · p. 1
  • University comparisons fuel personal social-media disputes, including accusations that private students buy degrees and public-university students are unemployable rote learners.Source: §I · p. 1
  • Existing survey research examines educational preferences, but little work studies organic educational polarization, especially in non-Western contexts.Social-media comments can reveal arguments made in discussion with others rather than in response to a researcher’s questions.Source: §I · p. 1
  • The study analyzes 4,060 Bengali comments and compares machine-learning, neural-network, pretrained-language-model, and LLM prompting approaches.The dataset is manually annotated with high inter-annotator reliability.Source: §I · p. 1
  • The manually annotated dataset covers public-versus-private university debates and has Fleiss’s Kappa agreement of 0.89.The paper also makes the source code publicly available.Source: §I · pp. 1–2
  • The paper compares classical machine learning, neural networks, Bengali language models, and large language models for stance detection in Bengali educational discourse.The comparison includes task-specific fine-tuning of multilingual and Bengali language models.Source: §I · p. 2

II. RELATED WORK

Related work discusses stance detection methods and educational preferences, while noting the limited attention paid to organic social-media discourse about education in Bengali.

II. RELATED WORK

  • Stance detection identifies a text author’s opinion toward a target, and research has expanded from traditional methods to neural networks, pretrained models, and LLMs.Source: §II · p. 2

A. Educational Discourse and Polarization in Social Media

  • Research on university preferences in Bangladesh has used surveys, while social-media studies of educational discourse are mostly Western or English-language and leave Bengali discourse underexplored.Survey studies based on closed-ended questions do not capture the subtleties of real-world debate.Source: A. Educational Discourse and… · p. 2

B. Stance Detection in Social Media Polarization

Stance detection identifies an author’s opinion toward a target and is used to analyze social, political, and health issues, including opinion polarization. Prior work compares modeling approaches, while Bengali educational discourse on public versus private universities remains an unbenchmarked topic.

B. Stance Detection in Social Media Polarization

  • Stance detection identifies a text author’s opinion toward a specific target and is used in social media analysis of political, social, and health issues.NLP has also been applied to belief-system modeling and the evolution of opinion polarization.Source: B. Stance Detection in Social Media… · p. 2
  • Table I is presented as a comparison of related stance-detection datasets.Source: Table I · p. 2
  • LLMs with prompt engineering outperformed earlier methods on Reddit Israel-Palestine stance detection, while RoBERTa reached 91% accuracy on a Bengali political text-classification task.Other transformer-based models have also been evaluated for Bengali text classification.Source: B. Stance Detection in Social Media… · p. 2
  • This study addresses the limited Bengali stance-detection research by introducing the first benchmark dataset and extensive model comparison for public-versus-private university discourse.Earlier BanglaBias work classified 200 political news articles, whereas this study examines educational discourse.Source: B. Stance Detection in Social Media… · p. 2

C. Research Gap and Motivation

The paper addresses gaps in Bengali educational stance detection by constructing a manually labeled dataset of public and private university comments collected from Facebook and YouTube.

C. Research Gap and Motivation

  • The authors identify limited research on educational stance detection, Bengali-language benchmark datasets, and organic social-media arguments compared with survey-based university preference studies.They position their work as addressing all three gaps.Source: C. Research Gap and Motivation · p. 2

A. Dataset

  • The dataset contains 4,060 comments labeled Pro-Public, Pro-Private, or Neutral after irrelevant and duplicate comments were removed.Comments were gathered from Facebook and YouTube, filtered by topic-related keywords, and independently labeled by three annotators; majority voting determined final labels, with Fleiss’s Kappa of 0.89.Source: A. Dataset · p. 3

B. Dataset Preprocessing

The preprocessing section describes cleaning and augmentation of the Bengali comments, including steps to address class imbalance and remove entries made redundant by cleaning. It reports the final size and composition of the combined dataset.

B. Dataset Preprocessing

  • The first cleaning removed English letters, stopwords, and emoji; a second cleaning removed empty strings and duplicates created when different comments became identical after cleaning.The second cleaning was performed on the combined original and augmented dataset.Source: B. Dataset Preprocessing · p. 3
  • After both cleaning iterations, the combined dataset contained 4,060 comments: 3,641 original and 419 augmented entries, including 189 Pro-Public and 230 Pro-Private augmented entries.The augmentation process initially yielded 561 augmented labels; the enlarged set was reduced by removing 79 duplicates and 63 empty strings.Source: B. Dataset Preprocessing · p. 3

C. Data Augmentation Techniques

  • BDA augmentation targeted the less frequent Pro-Public and Pro-Private classes, using synonym replacement, random word swaps, back translation, and paraphrasing.Synonyms came from a pre-trained FastText model; BanglaT5 was used for back translation and paraphrasing.Source: C. Data Augmentation Techniques · p. 3

D. Model Architectures

The section describes simple models with few extra steps and focuses on whether data augmentation helps, rather than on improving the models themselves.

D. Model Architectures

  • The models are described as simple and having few extra steps.The stated focus was to assess data augmentation's effect on machine-learning techniques.Source: D. Model Architectures · p. 3
  • Table IV is identified as a comparison of model results for multi-class stance classification.Source: Table IV · p. 4
  • The authors state that they did not try to improve the models themselves.The preceding passage frames the work as testing whether augmentation helps machine-learning techniques.Source: D. Model Architectures · pp. 3–4

1) Traditional Machine Learning Models:

The section covers traditional classifiers, a BiLSTM, a BanglaBERT-based classifier, and zero-shot LLMs. It specifies the models and key design choices used for each approach.

1) Traditional Machine Learning Models:

  • SVM, Random Forest, and XGBoost were evaluated with five-fold stratified cross-validation, reporting average performance across folds.Their hyperparameter values were selected from the literature.Source: 1) Traditional Machine Learning Models · p. 4

2) Neural Network Architectures:

3) Transformers Architecture:

  • BanglaBERT’s last four hidden layers were mean-pooled into embeddings for XGBoost; its hyperparameters matched the standalone XGBoost model to assess the pooling technique.The technique is called boost pooling and is described as using last-layer weights to represent token semantic relationships.Source: 3) Transformers Architecture · p. 4

4) Large Language Models (LLMs):

  • Five instruction-tuned LLMs were prompted to classify comments as pro-public, pro-private, or neutral, with no fine-tuning or few-shot examples.The prompt was kept short to test raw zero-shot ability.Source: 4) Large Language Models (LLMs) · p. 4

E. Experimental Setup

The experimental setup combines text cleaning and augmentation with distinct model inputs, then evaluates models on a held-out test set using four classification metrics.

E. Experimental Setup

  • Preprocessing used text cleaning to reduce noise and inconsistency, followed by augmentation for diversity and class imbalance; duplicated originals and invalid samples were cleaned before the sets were combined.Source: E. Experimental Setup · p. 4
  • The data was split into 80% training and 20% testing, with the test set held aside for final evaluation.Source: E. Experimental Setup · p. 4
  • Traditional classifiers received TF-IDF vectors, while BiLSTM was trained directly on processed text.Source: E. Experimental Setup · p. 4
  • For transformer-based modeling, pooled BanglaBERT features were input to XGBoost; the LLMs were evaluated on the test set without task-specific training.Source: E. Experimental Setup · p. 4
  • All models were evaluated using accuracy, precision, recall, and macro F1-score.The general experimental scheme is shown in Fig. 2.Source: E. Experimental Setup · p. 4

A. Stance Classification and Polarization Findings

Zero-shot Llama 4 Maverick led the evaluated LLMs, while BanglaBERT+XGBoost also outperformed the classical and BiLSTM results reported here. Neutral comments were typically harder to classify than either pro class.

A. Stance Classification and Polarization Findings

  • 89.27% accuracy and 89.09% macro F1 made SVM the strongest classical model, ahead of Random Forest and XGBoost.SVM's per-class F1 scores were 91.97 for Pro-Public, 85.41 for Pro-Private, and 89.89 for Neutral.Source: A. Stance Classification and… · pp. 4–5
  • 93.11 macro F1 and 93.31% accuracy made zero-shot Llama 4 Maverick the best of five LLMs; BanglaBERT+XGBoost scored 91.70 macro F1 and 91.81% accuracy.The reported per-class F1 scores for Llama were 96.43 for Pro-Public, 93.22 for Pro-Private, and 89.67 for Neutral.Source: A. Stance Classification and… · p. 5
  • Neutral was typically the most difficult class, with lower F1 scores than the two pro classes across models.Examples include BiLSTM F1 scores of 89.98 for Neutral versus 92.36 for Pro-Public and 88.62 for Pro-Private, and Claude Sonnet 4 scores of 87.67 versus 89.08 and 88.71.Source: A. Stance Classification and… · p. 5

1) Overview of the Augmented Dataset:

The study augmented underrepresented Pro-Public and Pro-Private comments, but duplicate and near-duplicate samples constrained the process. Zero-shot Llama 4 Maverick outperformed the reported trained models, while proposed explanations for that result were not independently tested.

1) Overview of the Augmented Dataset:

  • BanglaT5 paraphrasing and back-translation added 419 non-duplicate comments to the 3,641 originals, targeting Pro-Public and Pro-Private but not Neutral.The final training set contained 4,060 comments.Source: 1) Overview of the Augmented Dataset · p. 5

2) Data Augmentation Process Critique:

  • 79 of 561 generated samples were duplicates, and SacreBLEU filtering did not impose an upper similarity limit to prevent near-duplicates.The authors propose maximum-similarity filtering and Bengali WordNet-based word replacement as future improvements.Source: 2) Data Augmentation Process Critique · p. 5

3) Why LLMs (Zero-Shot) Perform Better:

  • Macro F1 0.931 for zero-shot Llama 4 Maverick exceeded BanglaBERT+XGBoost and the other reported ML/DL models.The authors suggest multilingual exposure and pretraining scale as possible explanations, but note these factors and prompt design were not independently controlled.Source: 3) Why LLMs (Zero-Shot) Perform Better · p. 5

4) Threats to Validity:

The paper reports a zero-shot Llama 4 Maverick macro F1 of 0.931, while noting annotation ambiguity, augmentation duplicates, and limits on generalizing beyond Bengali Facebook and YouTube comments. It aims to improve BanglaT5 augmentation and develop an open-source stance detector.

4) Threats to Validity:

  • Subjective annotation leaves ambiguous cases that humans and LLMs could interpret differently, despite high agreement for neutral labels (Fleiss’s Kappa = 0.89).The paper identifies this as a possible threat to construct validity.Source: 4) Threats to Validity · p. 5
  • About 14% of generated samples were duplicates, which might raise overfitting risk; the lack of significance testing, augmentation ablation studies, and quantitative error analysis reduced comparative evaluation quality.Source: 4) Threats to Validity · pp. 5–6
  • Because the dataset contains Bengali code-mixed comments from Facebook and YouTube, results may not apply to other platforms, languages, or domains.The paper notes that using a common augmented dataset and evaluation conditions allows fair comparison among the evaluated models.Source: 4) Threats to Validity · p. 6

V. CONCLUSION AND FUTURE WORK

  • 0.931 macro F1 was achieved by Llama 4 Maverick, outperforming all supervised baselines.The paper reports this result alongside a manually annotated dataset of 4,060 Bengali comments.Source: §V · p. 6
  • The dataset contains 4,060 Bengali comments; future work will fine-tune on Bengali and filter near-duplicates toward an open-source stance detector.The stated aim is to study social media polarization without costly commercial APIs.Source: §V · p. 6

Related papers

Loading 2610.04401v1…