PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Follow the Pulse of a Crisis? Evaluating Crisis Sentiment in Bangladesh's July Uprising

Md. Samiul Alim, Mahir Shahriar Tamim, Tanvir Ahmed Khan, Sharjil Khan, Rafia Ferdous Duti, Shahriyar Zaman Ridoy, Mohammad Ali Moni

arXiv:2609.16997v1cs.CL

TL;DR

Crisis sentiment analysis must account for fast-changing language and context in low-resource Bangla, where existing benchmarks often evaluate isolated, static text. The paper introduces a human-annotated, phase-aligned dataset and benchmarks encoders and LLMs; parent-post context helps, but performance drops under temporal shift.

  • Problem

    Crisis sentiment benchmarks often treat language as stable over time, evaluate comments in isolation, and focus on English, despite shifting crisis discourse and context-dependent meaning.

    Source: §1 · p. 1

  • Method

    The authors build UNRESTSENT200K from approximately 200K Facebook and YouTube comments across five uprising phases, link comments to parent posts, and benchmark encoders, prompted LLMs, and LoRA-tuned LLMs.

    Source: §1 · p. 2, §4 · p. 5, §4.2 · p. 6

  • Results

    Parent-post context consistently improves classification, temporal shift causes large performance drops, and supervised adaptation yields the strongest results while the benchmark remains challenging.

    Source: §6 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    UNRESTSENT200K supports research on context-aware and temporally robust sentiment analysis in low-resource crisis discourse.

    Source: §6 · p. 9

    The dataset covers one Bangladesh event and only Facebook and YouTube, so its results are not universal claims about all crises or low-resource languages.

    Source: §6 · p. 10

Abstract

from arXiv · show

Crisis sentiment analysis is especially challenging for low-resource languages such as Bangla, where language, context, and public reaction shift rapidly. We introduce UNRESTSENT200K, a Bangla crisis sentiment dataset with approximately 200K Facebook and YouTube comments from the July-August 2024 Bangladesh uprising. The dataset covers five event-aligned phases, from early escalation and internet blackout to regime transition and a later flood crisis. Each comment is linked to its parent post, enabling evaluation with and without discourse context. All comments are annotated through a fully human process involving 14 native Bangla-speaking annotators and senior validation, achieving substantial agreement (kappa = 0.73, alpha = 0.71) and 94.2% blind-audit agreement. We benchmark fine-tuned encoders, prompted LLMs, and LoRA-tuned LLMs. Results show that parent-post context consistently improves performance, while temporal shift across phases causes large performance drops. Strong LLMs perform well, but still struggle with sarcasm, implicit political references, and phase-dependent meaning. UNRESTSENT200K provides a benchmark for studying context-aware and temporally robust sentiment analysis in low-resource crisis discourse. UNRESTSENT200K is available at https://sami0055.github.io/UNRESTSENT200K/

1 Introduction

The introduction argues that crisis sentiment benchmarks need to account for changing event context and comments’ dependence on parent posts. It presents UNRESTSENT200K as a human-annotated, phase-aligned Bangla benchmark and previews gains from context alongside substantial losses under temporal shift.

Motivation

  • Crisis comments can change meaning with major events or the news post they respond to, yet most benchmarks treat language as temporally stable, isolated, and English-focused.The introduction links crisis sentiment systems to situational awareness, journalism, and policy decisions.Source: §1 · pp. 1–2

Contributions

  • Parent-post context improves encoder performance by 7–11 percentage points, while training on earlier phases and testing on later ones causes drops of 19–28 F1 points.The introduction reports these patterns across fine-tuned encoders, prompted LLMs, and LoRA-tuned LLMs.Source: §1 · p. 2
  • UNRESTSENT200K contains approximately 200K comments from the July–August 2024 Bangladesh uprising, the largest Bangla sentiment dataset to date.The introduction reports a size 2.8× that of SentiGOLD.Source: §1 · p. 2
  • Each comment is timestamped, assigned to one of five event-aligned phases, and paired with its parent post for context and temporal evaluation.Source: §1 · p. 2
  • The five-stage PiLA process uses 14 native Bangla-speaking annotators and achieves κ = 0.73, α = 0.71, and 94.2% blind-audit agreement.Senior validation is part of the annotation process.Source: §1 · p. 2

2 Related Work

The related work describes Bangla sentiment resources as largely static, decontextualised, and consumer-oriented, leaving within-event temporal change insufficiently addressed. UNRESTSENT200K pairs event phases with parent posts and uses a human annotation protocol for sensitive crisis discourse.

Sentiment benchmarks and the low-resource

  • Existing Bangla resources cover social-media sentiment, but are not anchored to a single event with phases for probing within-episode shift.The cited resources include SentNoB, SentiGOLD, and BnSentMix.Source: Sentiment benchmarks and the… · p. 2

Temporal and contextual robustness

  • UNRESTSENT200K holds language, platform, and population largely fixed while testing temporal shift within a seven-week crisis and comparing comment-only with post-plus-comment evaluation.Source: Sentiment benchmarks and the… · p. 2

Sentiment benchmarks and the low-resource

  • Table 1 compares Bangla social-media datasets across sentiment, hate-speech, and offensive-language benchmarks.Source: Table 1 · p. 3

Human annotation for sensitive discourse

  • Its fully human five-stage PiLA protocol involved 14 native-Bangla annotators and achieved κ = 0.73, α = 0.71, and 94.2% blind-audit agreement.The paper presents the protocol as a replicable approach for low-resource, politically sensitive NLP.Source: Sentiment benchmarks and the… · p. 3

3 Dataset

UNRESTSENT200K is a roughly 200K-comment Bangla dataset of public Facebook and YouTube discussion about the July–August 2024 uprising. Its event phases, parent-post links, human annotation, and stratified splits support contextual and temporal evaluation.

3 Dataset

  • UNRESTSENT200K pairs each of approximately 200K comments with its parent post and retains timestamp, platform, event phase, and one of three sentiment labels.Source: §3 · p. 3

3.1 Data Collection

  • Data collection used the Facebook Graph API and YouTube Data API v3 from July 5 to August 31, 2024; the period was divided into five phases for temporal analysis and split construction.More than 2,000 source posts and their comment threads were collected.Source: §3.1 · p. 3

3.2 Preprocessing

  • Preprocessing normalized Unicode and whitespace, replaced URLs and mentions, retained emojis, and removed short, mostly non-Bangla, spam, and near-duplicate comments.The final dataset contains approximately 200K comments.Source: §3.2 · p. 3

3.3 Annotation Guidelines

  • Annotators used the parent post as context but assigned sentiment to the comment, with guidelines covering positive, neutral, and negative labels.The negative category includes sarcasm, fear, and criticism; the neutral category includes unclear or mixed comments.Source: §3.3 · p. 3

3.4 Manual Annotation

  • Fourteen native Bangla-speaking annotators independently labeled each comment, with senior validators adjudicating disagreements through a five-stage process.The process included pilot guideline development, training, double annotation, senior adjudication, and blind audit, and took approximately eight months.Source: §3.4 · p. 4

3.5 Annotation Agreement

  • Annotation achieved Cohen’s κ = 0.73 and Krippendorff’s α = 0.71, with 94.2% exact-label agreement in the blind audit.Source: §3.5 · p. 4

3.6 Dataset Split

  • 70/15/15 train, validation, and test splits were jointly stratified by sentiment and event phase to preserve their distributions.The phasewise and sentiment-class split is shown in Table 2.Source: §3.6 · p. 4, Table 2 · p. 5

4 Experiments and Analysis

The experiments assess prompted LLMs alongside encoder baselines on Bangla crisis sentiment, using zero-shot and few-shot prompts. GPT-4o-mini performs slightly above the strongest post-plus-comment encoder, but the reported improvement is small.

4.1 Prompted LLMs

  • The experiments frame prompted LLMs as a baseline for settings where labeled data is limited and compare zero-shot with few-shot performance.Source: §4.1 · p. 5
  • The evaluation covers 15 open-source and proprietary LLMs with zero-shot and few-shot Bangla-native prompts, using post-plus-comment inputs for comparability with encoders.Few-shot prompts include seven examples spanning positive, negative, neutral, sarcastic, and politically implicit comments.Source: §4.1 · p. 5, §4 · p. 5
  • Few-shot examples raise GPT-4o-mini’s κ from 0.47 to 0.55 and GPT-4.1-mini’s κ from 0.34 to 0.40, while several smaller open-source models show little improvement.Source: §4.1 · p. 5
  • GPT-4o-mini few-shot prompting reaches 76.0% accuracy, 75.0 macro-F1, and κ = 0.55, slightly above BanglaBERT’s 74.72% accuracy and 0.7167 macro-F1.The authors characterize the improvement over the strongest post-plus-comment encoder as small.Source: §4.1 · p. 5, Table 3 · p. 6

LoRA SFT

LoRA supervision gives the strongest reported model, but parent-post context and temporal change remain central to crisis sentiment performance. The section also finds that sentiment shifts cluster around major event transitions rather than changing smoothly.

LoRA SFT

  • Sarcasm, neutral-to-negative errors, and politically charged surface cues remain difficult, with the same patterns also appearing in encoder models.This overlap suggests the errors are properties of the task rather than one model family.Source: LoRA SFT · p. 6

4.2 Supervised Adaptation with Mid-Scale LLMs

  • Gemma-4B reaches 78.4% accuracy and 77.6 macro-F1, outperforming prompted models; it exceeds GPT-4o-mini by 2.4 accuracy points and grounded BanglaBERT by 3.7.Gemma-4B also outperforms the larger LLaMA-3.1-8B, suggesting backbone choice and fine-tuning recipe matter beyond parameter count alone.Source: §4.2 · p. 6

4.3 Qualitative Effect of Fine-Tuning

  • LoRA SFT recovers sarcastic criticism, political accusation, and solidarity meanings missed by zero-shot Gemma-4B on the same inputs.The passage attributes this qualitative pattern to supervised exposure to gold PiLA labels.Source: §4.3 · pp. 6–7, Table 6 · p. 8

4.4 Effect of Parent-Post Context

  • Adding the parent post consistently improves encoder accuracy, including BanglaBERT from 64.19% to 74.72%.XLM-RoBERTa gains 9.55 points, mBERT 8.84, and BanglaElectra 6.96; the paired ablation holds other training settings fixed.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 Temporal Robustness

  • Models lose 19–28 macro-F1 points on later phases; BanglaBERT reaches 54.83% accuracy on Test-B versus 74.72% in-distribution post+comment.Training covers July 5–August 5, with later non-overlapping tests in August 6–20 and August 21–31.Source: §4.5 · pp. 7–8, Table 5 · p. 7

4.6 Sentiment Change Across Phases

  • Two phase comparisons show medium sentiment shifts: Crisis and Blackout versus Post-Blackout (d = −0.556), and Post-Blackout versus Flood Crisis (d = 0.567).Seven of ten pairwise phase comparisons are negligible; the strongest shifts center on Post-Blackout.Source: §4.6 · p. 8, Table 7 · p. 9

4.7 Findings

  • 78.4% accuracy makes LoRA-tuned Gemma-4B the strongest overall model, ahead of prompted LLMs and fine-tuned encoders.The authors also report that it remains below 80% accuracy.Source: §4.7 · p. 9

5 Discussion

The discussion argues that crisis sentiment analysis depends on both discourse context and time. Strong LLMs do not eliminate these challenges, supporting evaluation that includes parent posts and temporal splits.

5 Discussion

  • Including parent posts yields consistent 7–11 percentage-point gains across encoders, while testing on later uprising phases substantially lowers performance.The discussion warns that standard random splits can overestimate reliability as public conversation changes.Source: §5 · p. 9
  • GPT-4o-mini performs well under few-shot prompting but remains below the best fine-tuned LLM and only slightly exceeds the best post+comment encoder.Its errors overlap with smaller models on sarcasm, implicit political references, and phase-dependent meaning.Source: §5 · p. 9
  • The authors recommend temporal splits and discourse context in future crisis benchmarks, particularly for low-resource languages such as Bangla.UNRESTSENT200K provides a controlled setting for studying these issues.Source: §5 · p. 9

6 Conclusion

UNRESTSENT200K links roughly 200K Bangla comments to parent posts and event phases, supporting research on context-aware and temporally robust sentiment analysis. The conclusion presents human annotation, preprocessing, and post-level framing metadata, while noting that the dataset and evaluated models have defined scope limits.

6 Conclusion

  • UNRESTSENT200K contains approximately 200K Facebook and YouTube comments from the July–August 2024 uprising, linked to parent posts and assigned POSITIVE, NEUTRAL, or NEGATIVE labels.The conclusion says the dataset supports context-aware and temporally robust sentiment research; experiments find context gains, temporal-shift losses, and the strongest results from supervised adaptation.Source: §6 · p. 9
  • The release package is planned to include the de-identified corpus, fixed splits, parent-post context, timestamps, platform and phase metadata, annotation guidelines, and experiment materials.It will also include preprocessing and evaluation scripts and encoder and LoRA configurations.Source: §6 · p. 10

A.3 Preprocessing Details

  • Preprocessing normalizes text to NFC, cleans whitespace, masks URLs and mentions, retains emojis, and removes near-duplicates within threads at Jaccard similarity 0.85.Duplicates across different posts are retained; comments under three tokens, mainly non-Bangla text, obvious spam, and substantial English or Bangla–English transliteration are excluded.Source: §A.3.1 · p. 12, §A.3.3 · p. 12, §A.3.2 · p. 12

A.3.2 Filtering

  • Among retained comments, short comments of at most 10 words skew positive, neutral expression rises with length, and negative sentiment stays relatively flat.These distributions are reported across word-length bins.Source: §A.3.2 · p. 12

A.3.4 Post-Level Framing Labels

  • Each source post receives a coarse HOPE, DESPAIR, or OUTRAGE framing label as annotation context and metadata; the prediction target remains comment-level sentiment.HOPE captures optimistic or change-oriented framing, DESPAIR grief or uncertainty, and OUTRAGE anger, blame, or calls for accountability.Source: §A.3.4 · pp. 12–13

A.4 PiLA: Annotation Protocol Details

  • PiLA is a fully human process that took approximately eight months and involved 14 primary annotators and 10 senior validators.The primary annotators were native Bangla speakers familiar with the events, and validators did not overlap with that pool.Source: §A.4 · p. 13, §A.4.1 · p. 13

A.4.2 Guideline Document

  • The guideline defines three comment sentiment labels and gives rules for ambiguous cases, while directing annotators to use—but not copy—the parent post’s sentiment.Guidelines were refined during the pilot and fixed before main annotation.Source: §A.4.2 · p. 13

A.4.3 Qualification and Quality Checks

  • All 14 annotators met the qualification threshold of Cohen’s κ ≥0.65, and ongoing gold checks triggered retraining and review when agreement fell below requirements.Each comment received two independent labels; disagreements went to a senior validator, with difficult cases escalated to a panel.Source: §A.4.3 · p. 13, §A.4.4 · p. 13

A.5 Quality Metrics Details

The annotation checks combine pairwise and multi-rater reliability with confidence metadata, adjudication of disagreements, and a blind audit. The appendix also reports how sentiment composition varies with comment length.

A.5.1 Pairwise Inter-Annotator Agreement

  • κ = 0.73 indicates substantial pairwise agreement, with per-phase κ ranging from 0.69 to 0.78.The multi-rater reliability score was α = 0.71.Source: §A.5.1 · p. 13, §A.5.2 · p. 13

A.5.3 Annotator Confidence

  • The mean annotator confidence was 0.82 (SD 0.14), and 87.3% of labels had confidence of at least 0.7.Confidence scores are included as metadata for filtering examples.Source: §A.5.3 · p. 14

A.5.4 Pre-Adjudication Agreement

  • 78.4% of comments received matching labels from the two primary annotators; disagreements went to senior validators, and 3.1% required a three-validator panel.The remaining 21.6% were sent for adjudication.Source: §A.5.4 · p. 14

A.5.5 Independent Blind Audit

  • 94.2% exact-label agreement was achieved in the blind audit, with 96.8% agreement within one sentiment-scale step.Two senior validators who had not adjudicated relabeled 2,000 comments, sampled equally across the five phases; disagreements often involved rhetorical questions and emoji-mediated sentiment.Source: §A.5.5 · p. 14

A.6 Effect of Comment Length

  • Positive sentiment fell from 50% in 1–10-word comments to 20% in comments longer than 50 words, while neutral sentiment increased with length.Negative sentiment remained comparatively stable across length ranges.Source: §A.6 · p. 14

A.7 Quantifying Temporal Non-Stationarity

The appendix measures lexical and sentiment changes across event phases and describes the topic-modeling procedure used to examine the discussion. It also reports sentiment differences by comment length.

A.7 Quantifying Temporal Non-Stationarity

  • Short comments were more often positive, neutral sentiment increased with length, and negative sentiment stayed comparatively stable.Figure 3 shows the distribution across word-count intervals.Source: Figure 3 · p. 14
  • JS divergence ranged from 0.3713 to 0.3832, top-word Jaccard similarity from 0.4235 to 0.4652, and Shannon entropy rose from 7.5989 to 7.7872 across phases.Together, the measures indicate changing word distributions, vocabulary turnover, and greater lexical diversity; Figure 4 summarizes the temporal drift.Source: §A.7 · p. 14, Figure 4 · p. 14
  • Seven of ten pairwise phase contrasts had negligible sentiment-level effects, with the strongest effects clustered around the post-blackout transition.This pattern mirrors the reported lexical-level drift.Source: §A.7 · p. 14

Detailed Methodology

  • The topic workflow embeds comments with a multilingual sentence transformer, reduces representations with UMAP, and clusters them with HDBSCAN after stopword removal.The stopword list contains 619 Bangla terms, including discourse markers and platform artifacts.Source: Detailed Methodology · p. 15
  • The global topic model used UMAP and HDBSCAN settings intended to retain detailed but interpretable topics, with CountVectorizer phrases spanning one to three words.CountVectorizer used min_df=10 and max_df=0.9.Source: Detailed Methodology · p. 15
  • Phase-specific HDBSCAN cluster sizes were scaled from 25 to 100, and outliers were assigned to the nearest topic centroid using cosine similarity.Varying min_cluster_size by ±20% changed topic coherence by less than 0.05 (∆Cv < 0.05).Source: Detailed Methodology · p. 15

Extended Results

The extended results describe bursty but connected topic development during the uprising, alongside topic-specific sentiment volatility. Attention and polarity changes are concentrated around particular themes and phase transitions.

Extended Results

  • 75% of topics had an activity coefficient of variation above 1.0, and Topic T1 peaked at 1,063 posts on August 9, three days after the collapse.Topics remained active for 44.8 days on average, while dominant Topic T0 changed momentum 15 times.Source: Extended Results · p. 15
  • Across 100 topic transformations, mean semantic similarity was 0.93, indicating that themes usually evolved from earlier themes rather than disappearing and reappearing unrelated.Student-protest discussion developed into state-violence narratives with similarity 0.902; confrontation rhetoric became celebratory discourse with similarity 0.954 as its volume grew 462% after the collapse.Source: Extended Results · p. 15
  • Only Topic T8 had positive polarity (+0.246); the most negative topics, T15 and T6, had polarities of −0.486 and −0.294.The discussion included 38 significant co-occurrence relationships with r > 0.5; the strongest was T3 ↔ T1 (r = 0.988).Source: Extended Results · p. 15
  • The results characterize crisis discussion as mostly negative, focused on a small set of grievances, and connected despite rapid changes across phases.The authors conclude that crisis discourse is not simply a set of disconnected conversations.Source: Extended Results · p. 15

A.9 Failure Mode Discovery: Topic-Level Sentiment Instability

  • Six topics were highly volatile (σ > 30), 14 had medium volatility (15 ≤ σ ≤ 30), and five were stable (σ < 15).Quota-reform Topic T22 shifted from +57.1% to −33.3% between P3 and P4 (σ = 45.76), while humanitarian topics had σ ≈ 12–15.Source: §A.9 · p. 16
  • The largest sentiment shifts clustered after the blackout and during regime collapse: four topics changed by over 60% at P2→P3, and five reversed by over 45% at P3→P4.Overall, 15 topics showed a dramatic sentiment flip (∆ > 40%); the authors suggest topic-aware temporal recalibration as a response.Source: §A.9 · p. 16

A.10 Failure Mode Discovery through Topic Modeling

Topic-level analysis of classification errors finds a substantial neutral-class problem, including confident mistakes and complete polarity reversals. The supplied topic-level sentiment figure also shows that shifts vary by topic and event phase.

A.10 Failure Mode Discovery through Topic Modeling

  • The error-set analysis found 33 semantically coherent topics (Cv = 0.9499) among 2,789 misclassified comments.Topic 0 contained 322 neutral errors, of which 261 (81.1%) were predicted as negative.Source: §A.10 · p. 16
  • The topic-level figure marks initial and final polarity with hollow and filled circles, and shows T22 moving from +57.1% to −33.3%.Flood-relief topics were comparatively stable (σ ≈ 12–15), while the largest reversals occurred at P2→P3 and P3→P4.Source: Figure 5 · p. 16
  • 57.46% of neutral comments were misclassified, and the mean confidence across incorrect predictions was 0.977.Table 9 reports 676 errors in Topic 0 (24.2% of all errors); neutral-to-negative errors occurred 895 times, or 32.1% of all errors.Source: §A.10 · pp. 16–17, Table 9 · p. 17
  • In 531 cases (19.0% of errors), predictions flipped directly between negative and positive; Topic 4 had a 41.3% flip rate.The authors note that BanglaBERT also struggled to distinguish hopeful and critical framings of similar political or educational issues.Source: §A.10 · p. 17

A.11 Structural Diagnosis: Where Do Models Fail?

Models’ errors cluster around dominant political themes, while crisis phases differ sharply in topic coherence. The section also describes frozen-model prompting evaluations and the study’s data-handling and prompt conventions.

A.11 Structural Diagnosis: Where Do Models Fail?

  • T0 contains 56.3% of documents (n = 105,226), and 33% of all errors fall within its thematic orbit.The global model finds 29 topics, but the effective number of topics is only 1.7–2.0 across phases; politically charged topics show three times the sentiment volatility of humanitarian topics.Source: §A.11 · pp. 17–18
  • Cv = 0.781 in the Crisis & Blackout phase, versus 0.384–0.422 in every other phase; coherence falls as post-crisis attention spreads.The authors call this pattern the crisis coherence paradox; the largest phase, Post-Revolution, reaches Cv = 0.420.Source: §A.11 · pp. 17–18
  • T0’s dominance may lead models to associate its political vocabulary with negative sentiment, extending that tendency to neutral comments or other topics.The authors connect this risk to the neutral→negative bias reported in Appendix A.10.Source: §A.11 · p. 17

A.12 Prompting Techniques

  • Zero-shot and few-shot prompting keep model parameters frozen; the few-shot setting supplies seven labeled training examples before held-out post–comment classification.Greedy decoding uses temperature = 0, and outputs outside the allowed labels count as errors without another attempt.Source: §A.12 · p. 18

A.13 Expert Prompting with LLMs

  • Table 3 reports zero-shot and few-shot performance for open-source and proprietary LLMs prompted with Bangla-native instructions.The supplied passage states the table’s scope but does not provide its results.Source: §A.13 · p. 18

B Ethics Statement

  • Public Facebook and YouTube data were collected under platform terms of service, with usernames and profile links removed during preprocessing.Source: B Ethics Statement · p. 18

B.1 Sample Prompts

  • The sample prompt asks for one of three sentiment labels, with examples covering positive, neutral, and negative comments and instructions to consider tone and context.Its guidance treats factual, question-like, ambiguous, or sentiment-neutral comments as neutral, and asks models to judge comment sentiment rather than political ideology.Source: §B.1 · p. 20
Loading 2609.16997v1…