Source-linked AI summary
Arabizi Detection and Conversion to Arabic
Kareem Darwish
TL;DR
Arabizi's variable spellings and use alongside English complicate identifying and converting it, despite the need to process Arabic-script and Arabizi text. The paper applies sequence labeling and transliteration mining with language modeling, reporting 98.5% identification accuracy and 88.7% conversion accuracy.
Problem
Identifying Arabizi amid English and converting it to Arabic script is complicated by dialectal spelling variation and creative English spellings.
Source: §1 · pp. 1–2Method
The paper uses CRF sequence labeling with word- and sequence-level features for detection, and transliteration mining with language modeling for conversion.
Source: §1 · p. 2Results
98.5% overall language-identification accuracy and 88.7% transliteration accuracy were achieved.
Source: §5 · p. 7Takeaways & Limitations
Detecting and converting Arabizi to Arabic script would help reading, enable processing with existing NLP tools, and support text-processing and search normalization.
Source: §1 · p. 1Conversion is limited to Arabic words present in the word list, although the authors expect missing correctly generated words to be negligible because the list was built from many tweets.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Arabizi is Arabic text that is written using Latin characters. Arabizi is used to present both Modern Standard Arabic (MSA) or Arabic dialects. It is commonly used in informal settings such as social networking sites and is often with mixed with English. In this paper we address the problems of: identifying Arabizi in text and converting it to Arabic characters. We used word and sequence-level features to identify Arabizi that is mixed with English. We achieved an identification accuracy of 98.5%. As for conversion, we used transliteration mining with language modeling to generate equivalent Arabic text. We achieved 88.7% conversion accuracy, with roughly a third of errors being spelling and morphological variants of the forms in ground truth.
1 Introduction
The paper addresses detecting Arabizi amid English and converting it to Arabic script. Unconventional spelling and shared English-Arabizi forms make both tasks challenging.
1 Introduction
- Arabizi detection and conversion are intended to support reading, NLP processing, and normalization of Arabic and Arabizi for text processing and search.Arabizi can represent MSA or dialects, and users may switch between Arabizi and English within sentences.Source: §1 · p. 1
- Lack of spelling conventions makes comprehensive Arabizi dictionaries prohibitive, with many spellings possible for the same word.The MSA word for “liberty” has spellings including ta7rir, t7rir, and ta7reer; dialectal forms also vary.Source: §1 · pp. 1–2
- Shared spellings can make English dictionaries insufficient for distinguishing English from Arabizi, while creative English spellings further complicate the distinction.The examples include “men” used for “from” and shortened or lengthened English spellings.Source: §1 · p. 2
- The paper uses CRF sequence labeling with word- and sequence-level features to identify Arabizi in context, and transliteration mining with an Arabic language model to convert it.The language model covers MSA and other Arabic dialects.Source: §1 · p. 2
- The authors plan to publicly release their training and test data.The paper is organized around related work, detection, conversion, and a conclusion.Source: §1 · p. 2
2 Related Work
Prior work addresses language identification with character-based models and transliteration through learned mappings or other approaches. The paper uses CRF sequence labeling for offline word-level identification and adopts a transliteration-mining baseline.
2 Related Work
- Language identification systems have used character n-gram models with HMMs or Bayesian classification, while another system reported up to 97% accuracy on a synthetic two-language test set.That system used HMM-based identification with an n-gram character language model.Source: §2 · p. 2
- The authors choose CRF sequence labeling to combine word-level and sequence-level features for offline word-level language identification.CRF implementations such as CRF++ use nominal features, requiring real-valued features to be quantized.Source: §2 · p. 3
- Transliteration mining learns character mappings from paired forms and searches a target-language lexicon; this work uses the baseline system of El-Kahky et al. (2011).Prior mapping approaches include EM and HMM alignment.Source: §2 · p. 3
- Commercial Arabizi-to-Arabic IMEs are difficult to compare because they operate interactively rather than in batch mode.The authors' interactive use suggested these IMEs may use unigram language modeling.Source: §2 · p. 3
3 Identifying Arabizi
The authors train a CRF sequence labeler on tweet tokens marked as English, Arabizi, or other, using word- and sequence-level features. They report 98.5% word-level identification accuracy.
3 Identifying Arabizi
- The training data contained 5,207 tokens (3,307 English, 1,203 Arabizi, 697 other); the test set contained 3,491 tokens across 101 tweets (797 English, 1,926 Arabizi, 768 other).The authors queried common Arabizi words every 30 seconds for roughly 1 hour, then queried Twitter with the authors’ user IDs to retrieve their remaining tweets.Source: §3 · pp. 3–4
- The CRF++ sequence labeler used word-level features together with the previous and next words.Training and test tokens were manually labeled as English, Arabizi, or other.Source: §3 · pp. 3–4
- 98.5% word-level language identification accuracy was achieved on the test set.Reported mislabelings included Arabized English, Arabizi words that are also English words, isolated language switches, and misspelled English words.Source: §3 · p. 4
4 Arabizi to Arabic
The section presents a word-list-based approach to converting Arabizi into Arabic script, with candidate transliterations selected using context. It evaluates candidate ranking and in-context conversion on Arabizi tweets.
4 Arabizi to Arabic
- The conversion training collection contains 3,452 Arabizi–Arabic training pairs; evaluation uses 127 random Arabizi tweets containing 1,385 words, which a native Arabic speaker transliterated into Arabic script.Source: §4 · p. 4
- Preprocessing reduces repeated letters beyond two repetitions and concatenates short coordinating-conjunction and preposition tokens with following tokens.For example, “salaaaam” becomes “salaam”; the listed short tokens include “w,” “el,” and “b.”Source: §4 · p. 4
- The system aligns word pairs at character level to learn Latin-to-Arabic letter-sequence mappings, then ranks the top 10 candidate transliterations using mapping probabilities.Candidate probabilities combine the mapping posterior with the word prior.Source: §4 · pp. 5–6
- Generation is restricted to words in a large word list, built from more than 112 million Arabic tweets and containing 5.1 million unique words.The authors note this limits generation to previously seen words.Source: §4 · pp. 5–6
- A trigram language model containing MSA and dialectal text selects among candidates by combining transliteration and path probabilities.The chosen path maximizes the product of candidate transliteration probabilities and the trigram path probability.Source: §4 · p. 6
- In out-of-context evaluation, the system achieved an MRR of 0.84; the correct candidate ranked first in 77.1% of test cases, and no correct candidate was found in 4.9%.Source: §4 · p. 6
- 88.7% in-context accuracy, versus 77.1% for the top out-of-context choice, with a trigram language model; 46 errors were spelling or morphological variants.The evaluation had 157 wrong outputs among 1,385 words.Source: §4 · p. 7
5 Conclusion
The paper presents methods for detecting Arabizi mixed with English and converting Arabizi into Arabic. Detection uses sequence labeling with word- and character-level features, while conversion uses transliteration mining and language modeling.
5 Conclusion
- Detection used a sequence labeler with word- and character-level features, trained and tested on datasets constructed from tweets.Source: §5 · p. 7
- 98.5% overall accuracy was achieved for detection, and 88.7% transliteration accuracy for Arabizi-to-Arabic conversion.The authors identify additional training data and improved language models that account for Arabic morphological complexity as future work.Source: §5 · p. 7
- Conversion used a transliteration miner to find likely Arabic words from Arabizi, combining character transliteration probabilities with language modeling.The authors also suggest exploring variant-spelling detection and dialect-to-MSA conversion.Source: §5 · p. 7