AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages

1 Introduction

• African languages are underrepresented in NLP, and sentiment research on them is limited partly by a lack of high-quality annotated data.Sentiment analysis has applications including literary analysis, e-commerce, psychology, and social science.Source: §1 · pp. 1–2
• 110,000+ tweets in 14 African languages from four language families form the AfriSenti sentiment benchmark, with positive, negative, and neutral annotations.The paper presents it as the largest sentiment analysis benchmark for under-represented African languages.Source: §1 · p. 2
• The authors created ten datasets, curated four existing ones, discussed collection and annotation in 14 languages, released sentiment lexicons, and reported classification baselines.The datasets were used in the first Afrocentric SemEval shared task, AfriSenti-SemEval.Source: §1 · p. 2

2 Related Work

• Sentiment methods have developed from lexicon-based approaches through machine learning and deep learning to pretrained language models.The cited pretrained models include XLM-R, mDeBERTaV3, AfriBERTa, AfroXLMR, and XLM-T.Source: §2 · pp. 2–3
• Polarity classification is often considered solved in clean, high-resource domains, but noisy user-generated text in under-represented languages remains challenging.African-language sentiment analysis also faces tone, code-switching, and digraphia challenges.Source: §2 · p. 3
• AfriSenti is described as the largest multilingual dataset for sentiment analysis in African languages.Prior work in African-language sentiment analysis has mainly focused on polarity classification.Source: §2 · p. 3

3 Overview of the AfriSenti Datasets

• AfriSenti covers 14 languages across six Afro-Asiatic, six Niger-Congo, one English Creole, and one Indo-European language.The benchmark includes languages with Latin, Arabic, and Ethiopic scripts; Moroccan and Algerian Arabic are digraphic in social-media use, as is Amharic.Source: §3 · pp. 3–4
• The benchmark covers most African sub-regions and includes Swahili, Amharic, and Hausa, the continent’s three languages with the highest speaker numbers.Hausa and Swahili variants are spoken across neighboring countries in their respective sub-regions.Source: §3 · p. 4
• AfriSenti combines newly created and previously available datasets, with Table 3 listing their sources.For existing datasets with public test sets, the authors created new test sets for the shared task evaluation.Source: §3 · p. 4, Table 3 · p. 4

4 Data Collection and Processing

• Twitter’s language API supports only Amharic among more than 2,000 African languages, so the authors used location-based and vocabulary-based heuristics to collect tweets.The collection used the Twitter Academic API, which did not provide language identification for African-language tweets.Source: §4 · p. 4, §4.1 · p. 4

4.1.1 Location-based data collection

• Location-based collection used countries where most target-language speakers live, except for Algerian Arabic and Afaan Oromo; Oromo tweets were collected globally after Ethiopia yielded few.Collection took place before December 20, 2022, before a Twitter policy change in 2023.Source: §4.1.1 · pp. 4–5

4.1.2 Vocabulary-based Data Collection

• Because one location can return tweets in several languages, the authors combined location filters with stopwords, sentiment lexicons, and language-detection tools.Geez-script languages also used the Ethiopic Twitter Dataset for Amharic, which contains tweets collected since 2014.Source: §4.1.2 · p. 5
• Stopword lists were built from cross-domain word frequencies and co-occurrence, then checked by native speakers; this approach worked best for Xitsonga.Sentiment lexicons and native-speaker-provided mixed word lists supplemented collection; sentiment lexicons were used because stopword collection could yield too many neutral tweets.Source: §4.1.2 · p. 5

4.2 Language Detection

• Language detection followed collection because heuristics admitted other languages, including Tigrinya in Amharic searches and other Arabic varieties in Algerian Arabic searches.Existing tools were used for Amharic, Oromo, and Tigrinya; native speakers and a model fine-tuned on 2,000 manually annotated tweets helped identify Twi and Xitsonga, but some code-mixed tweets remained.Source: §4.2 · pp. 5–6

4.3 Tweet Anonymization and Pre-processing

• Preprocessing replaced mentions with @user and removed URLs; Nigerian-language test-set tweets were also lower-cased.Source: §4.3 · p. 6

5 Data Annotation Challenges

• Most tweets were annotated by three native speakers, while Amharic, Tigrinya, and Oromo used two annotators and an experienced third person to set gold labels.Majority voting determined final labels; cases with complete disagreement were discarded.Source: §5 · p. 6
• In Hausa, Igbo, and Yorùbá, tone is often absent from writing, while nearly identical text can carry opposite sentiment when intonation or context differs.The authors note punctuation or missing context may obscure the intonation distinction.Source: §5 · p. 6
• Twi tweets were often ambiguous, and some were labeled negative because common expressions could be offensive or playful depending on context.The text notes that “Tweaa” can be playful despite having been considered an insult.Source: §5 · pp. 6–7
• 0.40–0.75 free marginal multi-rater kappa scores indicate intermediate-to-good agreement across languages, except Afaan Oromo, where agreement was relatively low.The authors attribute the Oromo result to annotation challenges.Source: §5 · p. 6, Table 5 · p. 7
• Afaan Oromo and Tigrinya have no training splits because their datasets were limited in size; evaluation uses zero-shot transfer.Source: Table 6 · p. 7
• Code-mixing, unfamiliar local languages, Changana–Ronga variants, and sarcasm complicated annotation of Mozambican Portuguese and Xitsonga tweets.Many tweets were excluded because sarcasm contributed to annotator disagreement.Source: §5 · p. 7
• For Afaan Oromo and Tigrinya, recruiting annotators and limited internet and computer access posed challenges; Oromo annotations had severe quality problems.Source: §5 · p. 7
• Algerian Arabic sarcasm led to additional labeling or tweet discard when annotators could not agree, and offensive tweets were removed.Offensive tweets were removed to protect annotators and keep such instances out of the sentiment dataset.Source: §5 · p. 7

6.1 Setup

• Baselines cover monolingual fine-tuning for 12 languages, multilingual training and evaluation on their combined data, and zero-shot transfer to Oromo and Tigrinya.The experiments fine-tune massively multilingual and Africa-centric pretrained models, as well as the Twitter-adapted XLM-T.Source: §6.1 · pp. 7–8

6.2 Experimental Results

• AfriBERTa averaged 61.7 and scored below 50 on Arabic dialects outside its pretraining, while AfroXLMR-base and mDeBERTaV3 performed similarly.XLM-R-base was comparable to AfriBERTa on average but worse for most African languages except the Arabic dialects and pt-MZ.Source: §6.2 · p. 8
• AfroXLMR-large achieves the best overall performance, improving over XLM-T by 2.5 F1 points; XLM-T leads among models with up to 270M parameters.Together, these results highlight domain-specific pre-training and the benefits of scale for appropriately pretrained models.Source: §6.2 · p. 8
• In multilingual evaluation on combined data, AfroXLMR-large outperformed AfroXLMR-base, XLM-R-large, and XLM-T-base by more than 2.5 F1 points.Source: §6.2 · p. 8
• For zero-shot transfer, Hausa or Amharic were the best sources for Oromo, and Hausa or Yorùbá for Tigrinya; Hausa even outperformed a multilingual model.Hausa–Oromo transfer may benefit from their shared language family and Latin script, as well as Hausa's largest training dataset; why Yorùbá transfers well to Tigrinya remains unclear.Source: §6.2 · pp. 8–9

7 Conclusion and Future Work

• AfriSenti comprises native-speaker-annotated sentiment Twitter datasets in 14 African languages and was used in the first Afro-centric SemEval sentiment-analysis task.Source: §7 · p. 9
• The authors publicly release the datasets and collection lexicons, and plan to extend AfriSenti to more African languages and sentiment-analysis subtasks.Source: §7 · p. 9

8 Ethics Statement

• The authors forbid commercial or state-actor use of the datasets unless the creators explicitly approve it.Source: §8 · p. 9
• Because sentiment systems are unreliable at the individual-instance level and affected by domain shifts, the authors advise against high-stakes decisions about individuals.Health applications are given as an example.Source: §8 · p. 9

9 Limitations

• Because sentiment annotation is subjective, the data may retain label bias; the authors share attributed labels to help researchers study disagreements.Source: §9 · p. 9
• Data scarcity required keyword- and location-based collection, and the resulting datasets are sometimes imbalanced.Source: §9 · p. 9
• The study covers 14 languages, with collection for additional African languages left for future work.Source: §9 · p. 9

• Table 10 presents annotated tweets alongside their English translations.Source: Table 10 · p. 14