Source-linked AI summary
LABR: A Large Scale Arabic Sentiment Analysis Benchmark
Mahmoud Nabil, Mohamed Aly, Amir Atiya
TL;DR
Arabic sentiment analysis lacks large, standardized datasets despite the language’s complexity and importance. The paper introduces LABR, evaluates classifiers and dataset-derived lexicon features across sentiment tasks, and reports competitive lexicon-based results with far fewer features. LABR is released with standard splits and reproducible experimental resources for future comparison.
Problem
Arabic sentiment analysis has very few large-scale datasets with standard benchmarks, while Arabic morphology and dialect variation create additional challenges.
Method
The paper introduces LABR, analyzes its properties, evaluates a wide range of classifiers, and extracts a seed sentiment lexicon from the dataset.
Results
SVM and logistic regression are the best two classifiers, while the constructed lexicon achieves competitive results using only 667 features.
Takeaways & Limitations
LABR provides a public Arabic sentiment benchmark with standard splits, baseline results, and a compact domain-specific lexicon for future research comparisons.
Abstract
from arXiv · showhide
We introduce LABR, the largest sentiment analysis dataset to-date for the Arabic language. It consists of over 63,000 book reviews, each rated on a scale of 1 to 5 stars. We investigate the properties of the dataset, and present its statistics. We explore using the dataset for two tasks: (1) sentiment polarity classification; and (2) ratings classification. Moreover, we provide standard splits of the dataset into training, validation and testing, for both polarity and ratings classification, in both balanced and unbalanced settings. We extend our previous work by performing a comprehensive analysis on the dataset. In particular, we perform an extended survey of the different classifiers typically used for the sentiment polarity classification problem. We also construct a sentiment lexicon from the dataset that contains both single and compound sentiment words and we explore its effectiveness. We make the dataset and experimental details publicly available.
I. INTRODUCTION
Arabic sentiment analysis has received far less attention and data support than English, despite Arabic’s broad use and linguistic complexity. LABR addresses this gap with a large, reproducible benchmark, baseline experiments, and a dataset-derived sentiment lexicon.
- Motivation: Arabic sentiment analysis is underrepresented because English dominates available websites and Arabic has complex morphology and diverse dialects.The paper argues that developing Arabic sentiment tools remains important because Arabic is the sixth most widely spoken language.
- Dataset: LABR contains over 63,000 Arabic book reviews rated from 1 to 5 stars.The dataset is introduced as the Large-scale Arabic Book Review dataset.
- Contributions: The authors extract a domain-specific seed sentiment lexicon from the training data and examine its effectiveness.The approach targets both single and compound sentiment expressions and is designed to reduce classification complexity.
- Contributions: The paper provides standard training, validation, and testing splits to support comparable future experiments.The splits and reproduction scripts are publicly available.
- Contributions: LABR evaluates a wide range of classifiers to establish a baseline benchmark for Arabic sentiment classification.The benchmark is intended for comparison with future algorithms.
TAGREED (TGRD), TAHRIR (THR) and MONTADA
Earlier Arabic sentiment datasets were limited in size, availability, and experimental standardization. LABR is presented as a publicly available benchmark that addresses these weaknesses with a substantially larger dataset, standard splits, and baseline experiments.
- Limitations of prior datasets: Earlier Arabic sentiment datasets were small, with the largest containing just over 3,000 examples.The paper identifies dataset size as the first major weakness of prior resources.
- Limitations of prior datasets: Most earlier datasets were not publicly available, limiting access to the underlying resources.This is identified as the second weakness of prior Arabic sentiment datasets.
- Limitations of prior datasets: Earlier datasets generally lacked standard training and testing splits for benchmarking future research.The paper treats standardized splits as necessary for comparable evaluation.
- LABR’s role: LABR addresses these weaknesses by offering a publicly available dataset an order of magnitude larger, with standard benchmarks and baseline experiments.The comparison is made directly against the earlier Arabic datasets discussed in the paper.
III. SENTIMENT ANALYSIS CHALLENGES
Arabic sentiment analysis is difficult because sentiment depends on context, language variation, morphology, dialect, and compound expressions. The paper responds with a large dataset, baseline experiments, and a seed lexicon, while documenting substantial rating imbalance.
- General challenges: Sentiment words can take positive, negative, or neutral orientations, and their polarity may depend on sentence context.This makes sentiment analysis more difficult than ordinary text categorization.
- Arabic-specific challenges: Arabic-specific challenges include morphological richness, coexistence of Modern Standard Arabic and dialects, and limited language-processing resources.The paper also notes scarce datasets and limited colloquial Arabic parsers.
- Arabic-specific challenges: Sentiment in compound phrases may not correspond to the sentiment of their individual constituent words.This is identified as a language-specific challenge requiring suitable sentiment resources.
- Dataset properties: The dataset contains substantially more positive reviews than negative or neutral reviews.Figure 3 reports the imbalance across ratings, with ratings 4 and 5 dominating ratings 1, 2, and 3.
- Proposed resources: The paper provides baseline sentiment experiments and proposes a seed sentiment lexicon extracted from the dataset.These contributions are presented as responses to the documented Arabic sentiment-analysis challenges.
IV. DATASET COLLECTION AND PROPERTIES
LABR contains 63,257 Arabic book reviews collected and cleaned from Goodreads, with substantial variation in review structure, user/book activity, ratings, and sentiment alignment. The dataset is positively skewed, while some ratings are noisy or ambiguous, especially rating 3.
- Collection: 63,257 Arabic reviews remained after filtering more than 220,000 Goodreads reviews for non-Arabic characters.The reviews came from 16,486 users and 2,131 books; only cleaned Unicode reviews were publicly released.
- Ratings: Positive reviews substantially outnumber negative reviews, likely because popular books attract more reviews, especially positive ones.The authors associate this imbalance with the popularity of the reviewed books.
- Rating quality: Examples expose rating noise and ambiguity: rating-3 reviews may express positive, negative, or neutral sentiment, and some text-rating pairs conflict.The figure identifies noisy examples including reviews 4, 9, and 11.
- Users and books: Users averaged 3.84 reviews, while books averaged 29.68 reviews; most users and books had few reviews, with a long tail of higher activity.The corresponding medians were 2 reviews per user and 6 per book.
- Users and books: Positive reviews per user had a median of 2, compared with 1 for negative reviews.This supports the broader observation that users tend to provide more positive than negative reviews.
- Tokens and sentences: Reviews averaged 33 tokens and 3.5 sentences, with similar token and sentence distributions for positive and negative reviews.The average review sentence contained 9 tokens, and vocabulary frequencies followed Zipf’s law.
V. EXPERIMENTS
The experiments extend prior LABR work by broadening classifier coverage, adding neutral polarity, and constructing a dataset-derived sentiment lexicon. They evaluate these approaches across balanced and unbalanced settings using n-gram features.
- Experimental scope: The paper extends earlier experiments on binary polarity and five-way ratings classification with a broader survey of sentiment classifiers.The earlier work introduced LABR and conducted only a limited set of experiments.
- Experimental setup: Both balanced and unbalanced datasets are evaluated using combined unigram, bigram, and trigram feature ranges.The trigram range includes all lower-order n-grams as well as trigrams.
- Experimental scope: Neutral sentiment is added as a third polarity class alongside positive and negative.This extends the previous binary polarity formulation.
- Lexicon: A sentiment lexicon containing single and compound sentiment words is generated from the dataset and evaluated for effectiveness.The lexicon is intended to reduce classification time and space complexity.
A. Data Preparation
Data preparation creates reproducible train, validation, and test partitions, then compares balanced and proportionally unbalanced three-class polarity datasets. The unbalanced setting produces a very large n-gram feature space that motivates lexicon-based dimensionality reduction.
- Partitioning: Training, validation, and test sets are partitioned in a 6:2:2 ratio.The validation set serves as a mini-test for comparing models before selecting a final model.
- Class construction: Reviews are mapped to positive, neutral, and negative classes using ratings 4–5, 3, and 1–2, respectively.The neutral class captures mixed, objective, or otherwise unswayed opinions.
- Class construction: The balanced dataset equalizes class sizes at the smallest class count, whereas the unbalanced dataset preserves collected class proportions.Both settings are represented in the dataset splits and feature statistics.
- Feature construction: The unbalanced setting exceeds 3.7 million unigram, bigram, and trigram features, creating training challenges.This feature explosion motivates reducing dimensionality with sentiment lexicons.
B. Sentiment Analysis
LABR evaluates Arabic sentiment through polarity and rating classification using standard dataset splits, n-gram representations, and a broad set of baseline classifiers. The experiments report weighted accuracy and F1 on held-out test sets, with polarity generally easier than five-class rating prediction.
- LABR evaluates sentiment polarity and rating classification on balanced and unbalanced datasets.Polarity uses positive, negative, and neutral classes; rating classification predicts one of five star ratings.
- All experiments use unigram, bigram, and trigram feature ranges with token counts and TF-IDF representations.TF-IDF emphasizes terms frequent in the current document but less frequent across the document collection.
- The benchmark compares widely used classifiers, including Naive Bayes, SVM, Passive Aggressive, SGD, logistic regression, perceptron, and KNN.The classifiers are treated as baseline methods for future experiments on LABR.
- Weighted accuracy combines class-specific accuracies using class weights, while weighted F1 combines class-weighted precision and recall.The evaluation defines class accuracy from true positives and recall using true positives and false negatives.
- Rating classification is more challenging than polarity classification, while balanced datasets are more challenging than unbalanced datasets.The authors attribute the balanced-setting difficulty to fewer reviews and therefore less reliable n-gram training examples.
- SVM and logistic regression achieve good overall accuracy and F1, exceeding 70% for polarity classification.The reported comparison concerns the polarity task results in Table V.
A. Seed Lexicon Generation
The paper generates a domain-specific sentiment lexicon from classifier feature weights rather than constructing one manually. It ranks informative n-grams, selects extreme-weight candidates, and manually removes errors.
- Manual lexicon construction is difficult because of limited coverage, ambiguity, multiple meanings, and compound-word permutations.
- Linear SVM and logistic regression weights provide an automatic signal for selecting informative sentiment n-grams.Negligible weights indicate features considered unimportant or ineffective, particularly under the ℓ1 error measure.
- The method selects the highest 1000 weights as positive candidates and the lowest 1000 weights as negative candidates.The candidates are then manually reviewed to remove erroneous n-grams.
B. Lexicon Experiments
The lexicon experiments test the automatically generated lexicon alone and alongside trigram features on the unbalanced polarity task. The constructed domain-specific lexicon uses far fewer features while achieving competitive results and outperforming a general-purpose Arabic lexicon.
- The generated lexicon is evaluated as a stand-alone feature vector and combined with trigram features on the unbalanced training set.The comparison includes the Arabic lexicon developed by El-Beltagy and Ali.
- The generated lexicon contains 667 features: 348 negative n-grams and 319 positive n-grams.This is contrasted with trigram representations containing several million features.
- The constructed lexicon achieves competitive polarity-classification results using only 0.02% of the comparison feature count.The paper also reports that it outperforms the general-purpose lexicon by El-Beltagy and Ali.
- Domain-specific expressions such as “worth reading” and “I felt the novel” contribute book-review-specific sentiment information.The lexicon also captures compound expressions that may be represented as one word in Arabic.
VII. SUMMARY AND CONCLUSION
The paper presents LABR as a large Arabic sentiment dataset with standardized splits, broad classifier benchmarks, and a dataset-derived sentiment lexicon. Its experiments identify SVM and logistic regression as the strongest classifiers and show competitive lexicon results with far fewer features.
- LABR is presented as the largest Arabic sentiment analysis dataset to date.
- The paper provides standard splits and a comprehensive classifier study to support future comparisons.
- SVM and logistic regression are reported as the two best classifiers in the experiments.
- The constructed sentiment lexicon obtains competitive results with only a fraction of the feature count.