PaperlayerView exact revision on arXiv

Source-linked AI summary

100,000+ Movie Reviews from Kazakhstan: Russian, Kazakh, and Code-Switched Texts

Rustem Yeshpanov

arXiv:2605.08600v2cs.CL

TL;DR

The paper addresses limited evidence for Kazakhstan-specific multilingual movie-review sentiment by constructing and annotating a large corpus and benchmarking two supervised tasks. Multilingual transformers perform best on polarity classification, whereas leakage-controlled score classification remains difficult under severe imbalance and subtle adjacent-rating distinctions.

  • Problem

    Publicly available Kazakh movie-review sentiment datasets are scarce, limiting reproducible sentiment modelling in an underresourced language and multilingual Kazakhstan setting.

  • Method

    The paper constructs a kino.kz corpus and benchmarks three-way polarity and rating-based score classification using classical BoW/TF–IDF baselines and multilingual transformer models.

  • Results

    RemBERT achieved macro-averaged F1 = 0.82 and κ = 0.88 for polarity classification, while leakage-controlled score classification reached only macro-averaged F1 scores of 0.51–0.55.

  • Takeaways & Limitations

    The corpus provides a Kazakhstan-specific benchmark for multilingual sentiment analysis, while score inference from text alone remains difficult under class imbalance.

  • Takeaways & Limitations

    Stronger multilingual generalisation conclusions require more balanced language coverage or dedicated evaluation subsets.

Abstract

from arXiv · show

We present a new publicly available corpus of 100,502 movie reviews from Kazakhstan collected from kino.kz, spanning 2001-2025 and covering 4,943 unique titles. The dataset is multilingual, consisting mainly of Russian reviews alongside Kazakh and code-switched texts. Reviews are manually annotated for language and sentiment polarity, and 11,309 reviews additionally contain explicit user-provided ratings. We define two sentiment tasks -- three-way polarity classification and five-class score classification -- and benchmark classical BoW/TF-IDF baselines against multilingual transformer models (mBERT, XLM-RoBERTa, RemBERT). Experimental results show that transformer models consistently outperform classical baselines on polarity classification, while score classification remains challenging under leakage-controlled evaluation due to severe class imbalance and subtle distinctions between adjacent rating levels.

1 Introduction

The paper addresses the scarcity of publicly available Kazakh movie-review sentiment data by introducing a large multilingual corpus from Kazakhstan. It uses polarity and rating prediction as diagnostic tasks, revealing dataset-specific challenges in neutral sentiment and leakage-prone fine-grained scoring.

  • Publicly available Kazakh movie-review sentiment datasets remain scarce, despite reviews offering naturally evaluative language and substantial context.
  • 100,502 reviews from kino.kz span 2001–2025 and 4,943 unique titles, forming the paper’s new publicly available corpus.
  • The corpus contains Russian, Kazakh, and code-switched texts, with reviews manually annotated for language and sentiment polarity.
  • 11,309 reviews include explicit user-provided ratings, enabling fine-grained score prediction alongside three-way polarity classification.
  • Neutral polarity is rare and often implicit, while explicit rating mentions create leakage risks that motivate leakage-controlled evaluation for score prediction.

2 Related Work

Prior sentiment resources provide broad English coverage and some Russian and Kazakh data, but limited coverage of Kazakhstan-specific long-form multilingual movie reviews. This gap motivates the corpus and its diagnostic sentiment benchmarks.

  • English movie-review resources have supported extensive comparisons of classical and neural methods across binary and fine-grained sentiment settings.
  • Russian sentiment datasets exist, but relatively few have become standard movie-review benchmarks, and many alternatives target shorter social-media text.
  • KazSAnDRA offers 180,064 Kazakh reviews with 1–5-star ratings across four online domains, including code-switching and mixed Cyrillic/Latin writing.
  • Code-switched sentiment research has focused mainly on short-form social media, whereas this corpus targets long-form Kazakhstan movie reviews with naturally occurring Kazakh–Russian mixing.
  • Prior datasets therefore provide limited coverage of Kazakhstan-specific movie-review sentiment with long temporal span and naturally occurring multilingual review text.

3 Dataset Development

The dataset combines multilingual movie reviews from kino.kz with manual language and sentiment annotation, explicit ratings, and two classification tasks. Its construction addresses multilingual review analysis while documenting annotation, rating, and class-distribution constraints.

  • Source Data: 100,567 deduplicated reviews were collected from kino.kz with text, dates, authors, multilingual titles, and film metadata.The corpus covers reviews from a major Kazakh entertainment platform.
  • Source Data: Kazakh-language reviews are more common for Kazakhstan-produced titles, with median per-title shares of 0.10 versus 0.00 for other titles.Among titles with known production-country labels, Kazakhstan accounts for 18.3% of titles.
  • Review Language Identification and Annotation: Reviews were manually labelled as Russian, Kazakh, English, code-switched, or other, distinguishing multiword language segments from integrated loanwords.Code-switched reviews most commonly combine Kazakh and Russian, while English and other-language reviews were extremely rare and excluded from later analyses.
  • Review Language Identification and Annotation: Russian reviews dominate the corpus, positive labels are strongly skewed, and neutral reviews are comparatively rare.Median review lengths are 30 words in Russian, 24 in Kazakh, and 33 in code-switched texts.
  • Review Language Identification and Annotation: GPT-generated sentiment labels matched single-annotator labels with 89.54% accuracy and Cohen’s κ = 0.78, while released labels remain human annotations.GPT-4.1-nano was used to support reliability under single-annotator constraints.
  • Sentiment Classification Tasks: Score classification uses an 11-class 0–10 formulation, but severe imbalance caused unstable training and near-random macro-averaged F1 scores.Ambiguous rating formats were manually resolved, so rare borderline scores remain approximate.

4.1 Sentiment Classification Models

The paper evaluates multilingual transformer encoders and classical BoW/TF–IDF baselines for sentiment classification in Russian, Kazakh, and code-switched reviews.

  • Multilingual Transformers: mBERT, XLM-RoBERTa, and RemBERT are multilingual transformer models selected to support Kazakh and Russian.mBERT uses Wikipedia pretraining in 100+ languages, XLM-RoBERTa uses CC-100 data in 100 languages, and RemBERT is designed for underrepresented languages.
  • Model Scope: The study focuses on widely used pre-trained multilingual transformers to provide stable, reproducible supervised baselines.Instruction-tuned generative models and stronger multilingual encoders are identified as future work.
  • Classical Baselines: Classical baselines include linear SVM, logistic regression, and multinomial naïve Bayes with BoW and TF–IDF representations.These remain widely adopted baselines for text classification.

4.2 Experimental Setup

The experiments fine-tune multilingual transformers and tune classical text classifiers separately for polarity and score classification, using validation-based model selection and masking explicit ratings.

  • Transformer Training: All three transformer models were fine-tuned separately for the PC and SC tasks using corresponding training splits.The final configurations with the best validation macro-averaged F1 were evaluated on held-out test sets.
  • Classical Training: Classical models used sparse BoW or TF–IDF features with vocabularies capped at 50,000 terms and validation-tuned n-gram ranges.Task-specific configurations varied across SVM, logistic regression, and naïve Bayes.
  • Model Selection: Hyperparameters were selected by validation-set grid search using macro-averaged F1 as the primary criterion because of class imbalance.Class weights were applied to discriminative classical models.
  • Leakage Control: Explicit score mentions were replaced with a scoretoken placeholder in the SC task to control label leakage.The placeholder was added to transformer tokenisers and kept alphanumeric for BoW/TF–IDF compatibility.

4.3 Sequence Length

Transformer input length was set to 256 tokens after examining tokenised review lengths across all data splits.

  • Length Analysis: Approximately 95–97% of reviews contain fewer than 256 tokens, depending on the model tokeniser.About 2–5% of instances exceed this length and are truncated.
  • Length Selection: The maximum input length was set to 256 tokens for all transformer models because self-attention has quadratic complexity in sequence length.This setting balances coverage of most reviews with computational cost.

4.4 Performance Metrics

Performance is assessed with accuracy, precision, recall, macro-averaged F1, and Cohen’s kappa, with macro-F1 serving as the primary metric under class imbalance.

  • Evaluation Metrics: The evaluation reports accuracy, precision, recall, macro-averaged F1-score, and Cohen’s kappa.Macro-F1 gives equal importance to all classes, while κ measures agreement between model predictions and gold labels.

4.5 Experiment Results

Transformer encoders perform best on three-way polarity classification, while masked five-class score classification is substantially harder and approaches classical baselines.

  • Polarity classification: RemBERT and XLM-RoBERTa achieve test-set macro-averaged F1 scores of 0.82 and 0.81 on polarity classification, compared with 0.74 for mBERT.
  • Polarity classification: Transformer encoders maintain a consistent advantage over classical BoW/TF–IDF methods on the three-way polarity task.SVM, LR, and MNB reach F1 scores of 0.73, 0.71, and 0.70, respectively.
  • Polarity classification: RemBERT performs strongly on positive and negative reviews but reaches only F1 = 0.56 for the rare neutral class.Neutral instances comprise about 4–5% of the corpus and often use ambiguous language.
  • Score classification: Masked score classification produces macro-averaged F1 values of 0.50–0.55 across models, with RemBERT and SVM reaching 0.54–0.55.Masking explicit score mentions narrows the gap between transformer and classical models.
  • Score classification: RemBERT reaches F1 = 0.88 for score 5 but only F1 = 0.09 for score 2, while overall accuracy is 69%.The disparity reflects extreme class imbalance across rating levels.

5 Discussion

The discussion finds that contextual encoders provide a modest polarity advantage, whereas leakage-controlled score classification remains difficult because ratings are imbalanced and adjacent levels are linguistically similar.

  • Polarity classification: Multilingual transformers offer a consistent but modest advantage over strong linear baselines on polarity classification.
  • Polarity classification: Positive and negative sentiment are detected reliably, while the minority neutral class remains difficult because its language is ambiguous and underrepresented.
  • Polarity classification: Surface lexical cues capture much of the polarity signal, but contextual modelling benefits borderline cases and robustness under class imbalance.
  • Score classification: After masking explicit rating mentions, score classification converges across model families and falls to modest macro-averaged F1 values.
  • Score classification: Models perform well on frequent high-score bins but struggle on intermediate bins, where severe imbalance and similar adjacent ratings complicate prediction.
  • Limitations and future work: The score distribution is skewed toward favourable ratings, mid-range classes remain sparse, and residual label noise may persist despite leakage handling.The paper identifies rebalancing, ordinal modelling, and improved score extraction as directions for future work.
  • Limitations and future work: Stronger conclusions about multilingual generalisation require more balanced language coverage or dedicated evaluation subsets.

6 Conclusion

The paper releases a multilingual Kazakhstan movie-review corpus and benchmarks polarity and score classification, finding stronger transformer performance for polarity but modest results for masked score prediction.

  • The released corpus contains 100,502 reviews spanning 2001–2025 and 4,943 titles, with Russian, Kazakh, and code-switched texts.
  • Reviews are manually annotated for language and sentiment polarity, while 11,309 also contain explicit user-provided ratings.
  • The study defines three-way polarity and five-class score classification tasks and evaluates classical baselines alongside multilingual transformers.
  • RemBERT achieves the strongest polarity result with macro-averaged F1 = 0.82 and κ = 0.88, whereas masked score classification reaches only 0.51–0.55 macro-averaged F1.
  • The dataset, annotation guidelines, and trained models are released on the Hugging Face Hub to support reproducibility.
Loading 2605.08600v2…