Source-linked AI summary
ParsBERT: Transformer-based Model for Persian Language Understanding
Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, Mohammad Manthouri
TL;DR
Persian NLP lacks a dedicated pre-trained BERT despite limitations in multilingual coverage and available Persian data. The paper builds ParsBERT, a monolingual Persian BERT model, and evaluates it across three downstream tasks using assembled corpora and datasets. ParsBERT is reported to achieve state-of-the-art performance across the evaluated tasks and outperform multilingual BERT and other prior approaches.
Problem
Persian has limited language-specific NLP resources, and multilingual models may fall behind models trained with Persian-specific vocabulary and larger Persian text collections.
Method
The paper builds ParsBERT, a monolingual Persian language model based on the BERT architecture, and evaluates it on three Persian NLP downstream tasks.
Results
ParsBERT achieves state-of-the-art performance across Sentiment Analysis, Text Classification, and Named Entity Recognition, outperforming multilingual BERT and other prior models.
Takeaways & Limitations
ParsBERT provides a lighter Persian-specific model and associated Persian corpora and datasets for downstream NLP use.
Abstract
from arXiv · showhide
The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones as well as composed ones and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification and Named Entity Recognition tasks.
1 Introduction
ParsBERT addresses the limited Persian-specific pre-training resources by adapting BERT into a monolingual Persian language model and assembling Persian corpora and task datasets. It is reported to outperform multilingual and other competing architectures across several Persian NLP tasks while remaining lighter than multilingual BERT.
- Motivation: Persian NLP has limited language-specific resources, while multilingual models may not adequately benefit languages differing from Latin-based languages.The paper argues that Persian-specific vocabulary and larger Persian text collections are needed.
- Contribution: ParsBERT is a monolingual Persian language model based on the BERT architecture.The model is designed for Persian downstream evaluation.
- Evaluation: The study evaluates ParsBERT on Sentiment Analysis, Text Classification, and Named Entity Recognition.These tasks represent the downstream benchmarks used to assess the model.
- Results: ParsBERT achieves better performance than multilingual and deep-hybrid architectures.The introduction summarizes this comparison as a central contribution.
- Results: ParsBERT is lighter than the original multilingual BERT model.This is reported alongside its downstream performance results.
- Resources: The research provides a massive set of Persian text corpora and NLP task datasets for other use cases.The paper presents these resources as part of its contribution.
2 Related Work
Prior Persian NLP work includes contextual, recurrent, character-level, embedding-based, and task-specific approaches, but Persian monolingual pre-trained networks remain underdeveloped. ParsBERT is positioned against this gap and the limitations of multilingual or task-specific methods.
- Language Modeling: Earlier language-modeling research includes character-level recurrent and multi-task attentional models addressing spelling, grammar, out-of-vocabulary, and morphological information.These approaches model character sequences or medical concepts rather than introducing a Persian monolingual BERT.
- Language Modeling: Contextualized models represent words according to their surrounding context, with ELMo and ULMFiT using LSTM-based architectures.ELMo is bidirectional, while ULMFiT uses a regular multi-layer LSTM.
- Transformer Models: Transformer architectures use attention to evaluate sequence dependencies, and BERT builds bidirectional representations from transformer encoders and masked language modeling.GPT is described as unidirectional, whereas BERT conditions jointly on left and right contexts.
- Monolingual Models: Monolingual pre-trained models had been released for several languages, including Dutch, Italian, Arabic, Finnish, Russian, and Portuguese.The related work establishes that monolingual BERT-style modeling was already being pursued beyond English.
- Persian Prior Work: Persian resources included Wikipedia-trained word embeddings and an LSTM language model, but not a Persian monolingual BERT.The cited Persian LSTM used word embeddings and a two-layer bidirectional LSTM.
- Persian Downstream Tasks: Persian downstream systems largely used task-specific machine-learning or neural methods because they could not be fine-tuned as pre-trained models.The paper identifies multilingual Google BERT fine-tuning for Persian NER as the closest prior work, but it did not develop a monolingual Persian BERT.
3 ParsBERT: Methodology
ParsBERT’s methodology builds a Persian pre-training corpus, cleans and segments it with Persian-specific procedures, then trains and fine-tunes a BERT-based model for downstream NLP tasks.
- The methodology comprises data gathering, data pre-processing, sentence segmentation, pre-training setup, and fine-tuning.
- Data Gathering: The corpus was composed from diverse Persian sources because existing corpora were erroneous and covered limited writing styles and subjects.Sources included Persian Wikipedia, Eligasht, Digikala, Ted Talks subtitles, fictional books and novels, and MirasText.
- Data Pre-processing: Pre-processing removes trivial and junk characters and standardizes Persian characters before sentence segmentation.
- Sentence Segmentation: POS-based segmentation addresses errors caused by splitting Persian text only at writing notations and produces 38,269,471 True Sentence lines.Notation-only splitting can create short, meaningless sentences when abbreviations contain periods.
- Pre-training Setup: ParsBERT uses the BERT BASE architecture with 12 hidden layers, 12 attention heads, 768 hidden sizes, and 110M parameters.Its pre-training uses masked language modeling and next sentence prediction.
- Fine-tuning: For downstream tasks, sequence classification feeds the [CLS] representation into a Softmax layer, while NER predicts IOB-style entity tags.Fine-tuning adjusts both the classifier and pre-trained model weights for sequence classification.
4 Evaluation
ParsBERT is evaluated on sentiment analysis, text classification, and named entity recognition using existing and newly prepared Persian datasets with task-specific baselines.
- The evaluation covers Sentiment Analysis, Text Classification, and Named Entity Recognition, each requiring task-specific datasets.
- Sentiment Analysis: Sentiment Analysis uses Digikala, Snappfood, and DeepSentiPers comment datasets with two-, three-, and five-class labeling schemes.DeepSentiPers contains 12,138 opinions and can support binary classification after removing the neutral class.
- Dataset Preparation: The study prepared the Snappfood sentiment dataset and both text-classification datasets using the authors’ tools.Figure 2 concerns sentence segmentation based on writing notations versus POS, while the evaluation uses the resulting task datasets.
- Baselines: Baselines vary by task: multilingual BERT is used for Digikala and Snappfood sentiment data, prior DeepSentiPers results for that dataset, monolingual BERT for text classification, and Beheshti-NER for NER.
- Text Classification: Text Classification uses 8,515 Digikala magazine articles across seven classes and 16,438 news articles across eight classes.
- Named Entity Recognition: NER evaluation uses PEYMA and ARMAN, containing 7,145 and 7,682 sentences with seven and six entity classes, respectively.
5 Results
ParsBERT outperforms multilingual BERT and prior methods across sentiment analysis, text classification, and named entity recognition benchmarks. The reported results include strong F1 scores on PEYMA and ARMAN, while the discussion attributes the gains to broader Persian pre-training data and vocabulary.
- Sentiment Analysis: ParsBERT outperforms multilingual BERT on Digikala and SnappFood sentiment datasets in accuracy and F1 score.
- Sentiment Analysis: ParsBERT achieves significantly higher F1 scores than prior DeepSentiPers methods in both multi-class and binary sentiment analysis.
- Text Classification: ParsBERT achieves better accuracy and scores than multilingual BERT on Digikala Magazine and Persian news classification datasets.
- Named Entity Recognition: 93.10 and 98.79 F1 scores are reported for ParsBERT on PEYMA and ARMAN NER datasets, respectively, outperforming prior works.
- Discussion: The discussion links ParsBERT’s results to Persian-specific preprocessing, a diverse 14GB corpus with more than 3.9M documents, and a 100K-token vocabulary.
6 Conclusion
ParsBERT is presented as a lightweight Persian language model achieving state-of-the-art results across downstream tasks. The paper also contributes newly composed datasets intended for public use and future Persian NLP benchmarking.
- ParsBERT is lighter than multilingual BERT and achieves state-of-the-art results in sentiment analysis, text classification, and NER.
- ParsBERT outperforms prior Persian NER models with F1 scores of 93% on PEYMA and 98% on ARMAN.
- ParsBERT reaches F1 scores as high as 92% and 71% in binary and multilabel sentiment-analysis scenarios, respectively.
- The paper composes additional Persian downstream-task datasets because the number of existing datasets is limited and announces plans for public release.