PaperlayerView exact revision on arXiv

Source-linked AI summary

WangchanBERTa: Pretraining transformer-based Thai Language Models

Lalita Lowphansirikul, Charin Polpanumas, Nawat Jantrakulchai, Sarana Nutanong

arXiv:2101.09635v2cs.CL

TL;DR

Thai has limited monolingual language-model choices, while multilingual pretraining does not account for Thai-specific features. The paper trains a RoBERTa-based model on a large, cleaned Thai corpus with Thai-specific preprocessing, and reports that it outperforms strong baselines and multilingual models on sequence and token classification tasks.

  • Problem

    For Thai, available choices include models trained on smaller datasets or multilingual models that do not account for Thai-specific features.

    Source: §1 · pp. 1–2

  • Method

    The paper trains a RoBERTa-based model on a large cleaned and deduplicated Thai corpus and applies Thai-specific processing, including preserving spaces before subword tokenization.

    Source: §1 · p. 2

  • Results

    WangchanBERTa outperforms NBSVM, CRF, ULMFit, XLMR, and mBERT on sequence and token classification tasks in human-annotated, monolingual contexts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported comparisons support WangchanBERTa as a strong Thai model for the evaluated sequence and token classification tasks.

    Source: §1 · p. 2

    The authors identify measuring social biases in their models as future work, so bias measurement in Thai contexts remains to be developed.

    Source: §5 · p. 11

Abstract

from arXiv · show

Transformer-based language models, more specifically BERT-based architectures have achieved state-of-the-art performance in many downstream tasks. However, for a relatively low-resource language such as Thai, the choices of models are limited to training a BERT-based model based on a much smaller dataset or finetuning multi-lingual models, both of which yield suboptimal downstream performance. Moreover, large-scale multi-lingual pretraining does not take into account language-specific features for Thai. To overcome these limitations, we pretrain a language model based on RoBERTa-base architecture on a large, deduplicated, cleaned training set (78GB in total size), curated from diverse domains of social media posts, news articles and other publicly available datasets. We apply text processing rules that are specific to Thai most importantly preserving spaces, which are important chunk and sentence boundaries in Thai before subword tokenization. We also experiment with word-level, syllable-level and SentencePiece tokenization with a smaller dataset to explore the effects on tokenization on downstream performance. Our model wangchanberta-base-att-spm-uncased trained on the 78.5GB dataset outperforms strong baselines (NBSVM, CRF and ULMFit) and multi-lingual models (XLMR and mBERT) on both sequence classification and token classification tasks in human-annotated, mono-lingual contexts.

1 Introduction

Thai language modeling is constrained by smaller monolingual training data and multilingual models that do not account for Thai-specific features. The paper introduces a RoBERTa-based model trained on a large, cleaned Thai corpus and reports stronger performance across sequence and token classification tasks.

1 Introduction

  • Thai model choices were limited to training on smaller datasets or finetuning multilingual models, which omit Thai-specific features.The introduction notes that BERT-th underperforms ULMFit on Wongnai Reviews sequence classification.Source: §1 · pp. 1–2
  • The main model uses RoBERTa-base and a cleaned, deduplicated 78GB corpus of social media, news, and other publicly available data.Thai-specific preprocessing preserves spaces as chunk and sentence boundaries before subword tokenization.Source: §1 · p. 2
  • Four additional models trained on Thai Wikipedia compare SentencePiece, word-level, syllable-level, and SEFR tokenization.The tokenizer experiments explore how tokenization affects downstream performance.Source: §1 · p. 2
  • WangchanBERTa outperforms NBSVM, CRF, ULMFit, XLMR, and mBERT on sequence and token classification tasks.The reported evaluations are in human-annotated, monolingual contexts.Source: §1 · p. 2
  • The report makes pretrained and finetuned models and experiment code publicly available.Models are hosted on Hugging Face’s Model Hub, and source code is available in a GitHub repository.Source: §1 · p. 2

2 Methodology

The methodology trains one model on assorted Thai texts and four Wikipedia-only models with different tokenizers. It combines Thai-aware preprocessing and sentence segmentation with RoBERTa-base pretraining.

2.1 Raw Data

  • The assorted corpus combines publicly available Thai material, including social posts, newspaper articles, and parallel English–Thai data.The study trains one model on all available raw datasets and four additional models on Wikipedia only.Source: §2 · p. 2, §2.1 · pp. 2–3

2.2 Preprocessing

  • Preprocessing cleans noisy text and explicitly marks spaces because Thai spaces can signal sentence boundaries and SentencePiece may otherwise combine them with tokens.Rules include normalizing HTML characters, removing empty brackets and repetitive characters, and replacing repetitive words after tokenization.Source: §2.2 · p. 4
  • CRFCut sentence breaking limits sequences to 5–300 words, keeping them within the 416-subword maximum while avoiding sequences too short for language modeling.Its sentence-boundary F1 scores are 0.69 on translated texts, 0.71 on TED transcripts, and 0.96 on generated product reviews.Source: §2.2 · p. 4
  • The assorted-corpus tokenizer is a 25,000-subword SentencePiece unigram model trained on 15 million sampled sentences; Wikipedia-only models compare four tokenizers.The Wikipedia tokenizers are SentencePiece, newmm word-level, syllable-level, and SEFR.Source: §2.2 · pp. 4–5

2.3 Train-Validation-Test Splits

  • The assorted dataset contains 381,034,638 unique sentences (78.5GB), with separate validation and test samples; Wikipedia-only splits use 944,782 training sentences.The assorted corpus also has 99,181 validation sentences and 42,238,656 test sentences; Wikipedia-only validation and test sets have 24,863 and 24,862 sentences.Source: §2.3 · p. 5

2.4 Language Modeling

  • RoBERTa-base models use masked language modeling, predicting masked subword tokens; pretraining settings are tuned separately for the large assorted corpus and smaller Wikipedia corpus.The architecture has 12 layers, 768 hidden dimensions, and 12 attention heads; 15% of tokens are sampled for masking.Source: §2.4 · p. 5

3 Downstream Tasks

The section evaluates Thai language models on existing sequence- and token-classification benchmarks, describing the datasets and the traditional, recurrent, and transformer-based models used for comparison.

3.1 Datasets

  • Dataset splits follow Hugging Face Datasets where available; Wongnai Reviews and ThaiNER splits are sampled uniformly, with rare ThaiNER tags replaced and overlength mBERT test examples removed.Source: §3.1 · p. 7

3.1.1 Sequence Classification

  • Sequence-classification datasets include sentiment and rating tasks, plus translated Thai reviews used to predict one-to-five-star ratings.Source: §3.1.1 · p. 7
  • Prachathai-67k provides multi-label topic classification from news headlines, with labels spanning politics, human rights, and other topics.Source: §3.1.1 · p. 7

3.1.2 Token Classification

  • Token-classification data comprise human-annotated ThaiNER named entities in IOB format and LST20, used for POS tagging and NER.Source: §3.1.2 · p. 7

3.2 Benchmarking Models

  • The benchmarks compare traditional models, sequence-classification-only ULMFit, and transformer-based models.Source: §3.2 · p. 8
  • NBSVM uses unigram and bigram TF-IDF features, while CRF uses unigram, bigram, and trigram features in a three-timestep window.Source: §3.2 · p. 8
  • ULMFit is finetuned for sequence classification, while transformer models share a finetuning scheme with three epochs for sequence tasks and six for token tasks.Source: §3.2 · p. 8

4 Results

On downstream tasks, the Assorted Thai Texts model generally outperforms strong baselines and other transformer models, with stated exceptions in one sequence-classification task and one token-classification task. The results also compare tokenization variants on Wikipedia-only data, while the larger dataset was trained only with SentencePiece.

4.1 Language Modeling

  • Wikipedia-only RoBERTaBASE results cover SentencePiece, newmm word-level, syllable-level, and SEFR stacked-ensemble tokenization variants.Table 5 covers vocabulary size, training-example count, and best checkpoint for these models and the Assorted Thai Texts dataset.Source: §4.1 · p. 9, Table 5 · p. 9
  • The Assorted Thai Texts model was trained only with SentencePiece because computational resources were limited.The reported downstream models are selected using validation performance; multi-class and token classification optimize micro-F1, while multi-label classification optimizes macro-F1 and label-specific thresholds.Source: §4.1 · p. 9, §4.2 · p. 9

4.2 Downstream Tasks

  • The Assorted Thai Texts model outperforms strong baselines and other transformer architectures on sequence-classification tasks except Generated Reviews (EN-TH).The authors suggest XLMR may benefit because the reviews were translated from generated English texts.Source: §4.2 · p. 9, Table 6 · p. 10
  • The model achieves the highest micro-averaged F1 on token-classification tasks except POS tagging in the ThaiNER dataset.The authors suggest machine-generated ThaiNER POS tags may suit CRF better; Table 7 reports test-set micro- and macro-averaged F1 scores.Source: §4.2 · p. 9, Table 7 · p. 10

5 Discussions and Future Works

The authors report better downstream performance from larger training datasets, while XLMR's stated advantage occurs when training data include English-to-Thai translated text. They also identify Thai bias measurement as future work and discuss privacy and copyright considerations for training on public data.

5 Discussions and Future Works

  • Larger training datasets such as Assorted Thai Texts yield better downstream performance, while Wikipedia-only experiments show no notable difference across sequence- and token-classification tasks.XLMR outperforms the largest monolingual model only when training data include English-to-Thai translated Generated Reviews.Source: §5 · p. 10
  • The authors plan to create Thai-context bias-measuring datasets to investigate inherent biases in their relatively large language models.They cite prior detection of social biases in large English-trained language models.Source: §5 · p. 11
  • The models use publicly available training data and are provided under CC-BY-SA 4.0; the authors train them as feature extractors rather than generative models.They note that training-data reproduction remains possible, though at a much lower chance than for models trained specifically for generative tasks.Source: §5 · p. 11

7 Appendix

The appendix collects hyperparameter-tuning tables and per-class precision, recall, and F1-score tables for ThaiNER and LST20. The tables cover CRF, several WangchanBERTa variants, the final WangchanBERTa model, mBERT, and XLMR.

7 Appendix

  • Hyperparameter-tuning results are included for NBSVM, ULMFit (thai2fit), and CRF.The NBSVM tuning passage describes selection by F1, with micro-averaging for multi-class and macro-averaging for multi-label classification.Source: Table 8 · p. 14, Table 9 · p. 15, Table 10 · p. 16
  • Per-class precision, recall, and F1-score test results are reported for CRF on ThaiNER and LST20.Additional tables report these metrics for WangchanBERTa variants using SentencePiece, newmm, syllable, and sefr tokenization.Source: Table 11 · p. 17, Table 12 · p. 17, Table 13 · p. 18
  • Per-class precision, recall, and F1-score test results are tabulated for WangchanBERTa variants using syllable and sefr tokenization on ThaiNER and LST20.These results appear in separate tables for each variant and dataset.Source: Table 17 · p. 20, Table 18 · p. 20, Table 19 · p. 21
  • Per-class precision, recall, and F1-score test results are tabulated for the final WangchanBERTa model on ThaiNER and LST20.The appendix also provides corresponding tables for mBERT and XLMR on both datasets.Source: Table 21 · p. 22, Table 22 · p. 22, Table 23 · p. 23
Loading 2101.09635v2…