Source-linked AI summary
PyThaiNLP: Thai Natural Language Processing in Python
Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, Can Udomcharoenchaikit
TL;DR
Thai NLP has been constrained by limited resources, fragmented tooling, and access barriers. PyThaiNLP addresses this need through an open-source Python library offering Thai NLP software, datasets, and pre-trained models, and it is used by research and industry communities. Its development remains constrained by the computational cost of testing large language model components in continuous integration.
Problem
Thai NLP has limited resources, fragmented tooling, and access barriers that make it harder for researchers to build on existing tools and datasets.
Method
PyThaiNLP provides an open-source Python library combining Thai NLP tools, datasets, and pre-trained language models.
Results
PyThaiNLP supports Thai NLP research and industrial applications, while WangchanBERTa is reported as state-of-the-art on public benchmarks.
Takeaways & Limitations
The library serves as a shared Thai NLP resource used in research and industry and includes models such as WangchanBERTa and WangChanGLM.
Takeaways & Limitations
Large language models can exhaust free-tier continuous-integration memory, forcing the project to drop some LLM-related tests and reduce code coverage.
Abstract
from arXiv · showhide
We present PyThaiNLP, a free and open-source natural language processing (NLP) library for Thai language implemented in Python. It provides a wide range of software, models, and datasets for Thai language. We first provide a brief historical context of tools for Thai language prior to the development of PyThaiNLP. We then outline the functionalities it provided as well as datasets and pre-trained language models. We later summarize its development milestones and discuss our experience during its development. We conclude by demonstrating how industrial and research communities utilize PyThaiNLP in their work. The library is freely available at https://github.com/pythainlp/pythainlp.
1 Introduction
Thai NLP remains underserved because limited resources compound the challenges of processing Thai text without clear word and sentence boundaries. PyThaiNLP addresses this gap with an open-source Python library spanning Thai NLP tools, models, and datasets.
- Thai has remained comparatively underserved in NLP because language resources are limited.
- Thai is Thailand’s de facto national language, with substantial Central Thai use and additional Northern, Northeastern, and Southern varieties.
- Thai writing commonly lacks spaces or other marks between words and sentences, creating ambiguity that grammar alone cannot resolve.
- An open-source toolbox can provide transparent access to Thai NLP capabilities, training data, and algorithms while allowing community adaptation.
- PyThaiNLP is a Python library whose features range from dictionary-based tokenization and statistical named-entity recognition to an instruction-following large language model.
2 Open-source Thai NLP before PyThaiNLP
Before PyThaiNLP, Thai NLP had separate tools and limited datasets but lacked a unified, well-maintained open-source Python ecosystem. Restricted access, documentation gaps, weak benchmarking, and maintenance problems increased the effort required for research and reduced reproducibility.
- Before 2016, Thai NLP lacked a unified open-source toolkit combining multiple tools and tasks, while datasets were scarce relative to high-resource languages.
- NLTK did not support Thai, and OpenNLP’s Thai support appeared in version 1.4 but was no longer listed in version 1.5.
- Open Thai resources were limited in size and number, and public datasets often imposed restrictive licenses, registration requirements, or both.
- Limited tool availability, documentation, benchmarking, performance, and maintenance forced researchers to build basic components or collect datasets before tackling advanced problems.
- Earlier resources covered tokenization, tagging, named-entity recognition, and speech recognition, but access and licensing varied across tools and corpora.
- BEST evaluations produced additional tokenizers, although many later became inaccessible or unmaintained; the BEST-2010 segmentation dataset remained an important basis for modern software.
- TLTK and PyThaiNLP are described as the only two comprehensive Thai NLP libraries for Python, while TLTK’s documentation remained limited.
3 PyThaiNLP and Its Ecosystem
PyThaiNLP emphasizes a simple, user-friendly Python interface while extending beyond software components to provide Thai datasets and pre-trained models. Its ecosystem covers a broad range of language-processing functionality.
- PyThaiNLP prioritizes userfriendliness and simplicity through established interfaces inspired by NLTK, such as word_tokenize and pos_tag.
- The library also creates Thai-language datasets and pre-trained models, with its functionality and ecosystem summarized in Figure 1.
- The paper focuses on popular features and major datasets and models rather than documenting every component in the ecosystem.
3.1 Features
PyThaiNLP provides a broad set of Thai NLP features, combining multiple tokenization, modeling, conversion, and linguistic-analysis tools. Its components include rule-based and statistical methods, neural models, and community-created resources.
- 3.1.1 Word and Sentence Tokenization: NewMM is PyThaiNLP’s default dictionary-based maximum-matching tokenizer and uses Thai character clusters.
- 3.1.1 Word and Sentence Tokenization: 71.18 % accuracy made the pure-Python tokenizer the fastest on the BEST 2010 benchmark, compared with 95.60 % for the state-of-the-art system.
- 3.1.1 Word and Sentence Tokenization: Sentence tokenization uses a conditional random field model trained on translated TED transcripts, assuming Thai boundaries align with English sentence boundaries.
- The library includes multiple spell-checking engines, including Norvig, SymSpell, and Hunspell ports using Thai lexical resources.
- PyThaiNLP supports grapheme-to-phoneme conversion, Thai Soundex algorithms, and transliteration for Thai and several other languages.
- Its linguistic-analysis tools include Thai NER, several POS taggers, the WangchanBERTa model, Han-Coref, and BELA-based entity linking.
3.1.6 Word Embeddings
PyThaiNLP’s ecosystem includes Thai word embeddings, translation and ASR resources, supported by models, datasets, and collaborations.
- 3.1.6 Word Embeddings: thai2fit provides word-level ULMFiT token embeddings trained on Thai Wikipedia for use as PyThaiNLP word embeddings.
- 3.1.7 Machine Translation: The English-Thai translation model, developed with AIResearch.in.th, outperformed Google Translate on an out-of-sample test set at release.
- 3.1.8 Automatic Speech Recognition: PyThaiNLP members contributed Thai sentence cleanup, validation rules, sentence collection, and voice-validation activities to the Common Voice corpus.
- 3.1.8 Automatic Speech Recognition: The Thai ASR model, created with AIResearch.in.th using Common Voice Corpus 7.0, achieved the lowest character error rate in a benchmark.
3.2 Datasets
PyThaiNLP provides datasets for Thai social-media processing, named-entity recognition, coreference resolution, and English-Thai translation.
- 3.2.1 VISTEC-TPTH-2020: VISTEC-TPTH-2020 contains 50,000 Thai Twitter sentences and supports mention detection, spell checking, and spell correction.It is described as the largest Thai word-tokenization and spell-checking dataset to date.
- 3.2.2 Thai NER: Thai NER combines text from news, Wikipedia, government documents, and other datasets that were manually relabeled for consistency.
- 3.2.3 Han-Coref: Coreference Resolution: Han-Coref contains 1,339 documents from news and Wikipedia domains for Thai coreference resolution.
- 3.2.4 scb-mt-en-th-2020: scb-mt-en-th-2020 is a collaborative English-Thai dataset containing 1,001,752 sentence pairs.
3.3 Pre-trained Language Models
PyThaiNLP includes pretrained Thai language models spanning encoder-only and multilingual instruction-following architectures.
- 3.3 Pre-trained Language Models: WangchanBERTa is an encoder-only pretrained Thai language model that achieves current state-of-the-art performance on public benchmarks.
- 3.3 Pre-trained Language Models: WangChanGLM is a multilingual instruction-following model finetuned from XGLM.
4 Community and Project Milestones
PyThaiNLP grew from a hobby project into a community-supported Thai NLP library through contributor growth, expanding models and datasets, and increasingly systematic software engineering.
- 4.1 Foundation Years (2016-2019): The project began in 2016 after its creator found no comprehensive Thai NLP toolkit in Python comparable to NLTK.It was initially created as a hobby project for building a simple Thai chatbot.
- 4.1 Foundation Years (2016-2019): The Thai NLP Facebook group became a major discussion, showcasing, and recruiting venue, growing to over 16,000 members.Many major contributors joined through this community.
- 4.1 Foundation Years (2016-2019): PyThaiNLP’s adoption is reflected in GitHub stars, while repository activity peaked in Q4 2019.The star-count trend is presented in Figure 2.
- 4.1 Foundation Years (2016-2019): PyThaiNLP expanded from foundational tokenization and tagging capabilities to translation and multilingual language models through community and institutional collaboration.Contributors joined organically, while the AIResearch.in.th collaboration added computational resources and developers.
- 4.2 AIResearch.in.th Collaboration: Resource constraints led the project to prioritize high-impact features and choose encoder-only WangchanBERTa over decoder-only models during 2019–2020.The choice reflected lower training-resource requirements and stronger performance on impactful tasks at the time.
5 PyThaiNLP in the Wild
PyThaiNLP is used across Thai NLP research and production systems, where its preprocessing tools, vectors, and models support applications from language modeling to search, recommendation, and customer-service automation.
- 5.1 PyThaiNLP in Research: Researchers use PyThaiNLP for cross-lingual pretraining, dependency parsing, and representation learning, while SEACoreNLP and spaCy include it in their toolkits.These examples show adoption across both research workflows and broader NLP software ecosystems.
- 5.2 PyThaiNLP and Its Industry Impact: Industrial users report faster Thai NLP development cycles and effective business outcomes, with tokenization and text normalization among the most-used functions.Reported use cases span banking, telecommunications, insurance, retail, and software development.
- 5.2 PyThaiNLP and Its Industry Impact: True Digital Group used thai2fit word vectors for media analysis and recommendation, reporting noticeable gains in user engagement and subsequent business outcomes.The organization also combined the vectors with Top2Vec for topic modeling and customer-experience improvements.
- 5.2 PyThaiNLP and Its Industry Impact: Central Retail used tokenization and normalization for Thai product search, enabling performance beyond out-of-the-box systems incompatible with Thai.The team also featurized product information for content-based recommendations across five business units and over six million customers.
- 5.2 PyThaiNLP and Its Industry Impact: AIA Thailand analyzed call logs with PyThaiNLP to identify conversation topics, improving voice-bot fulfillment and reducing call waiting time.The workflow supported both automated voice-bot training and human-staff allocation.
- 5.2 PyThaiNLP and Its Industry Impact: VISAI fine-tuned WangchanBERTa for classification, named-entity recognition, and relation extraction to build queryable knowledge graphs from unstructured client data.It also uses tokenization and normalization across its NLP products.
6 Conclusion and Future Works
PyThaiNLP’s future work focuses on expanding Thai NLP coverage, benchmarking, correctness, resource efficiency, and integration with language-agnostic tools.
- 6 Conclusion and Future Works: The paper introduces PyThaiNLP, explains its features and datasets, and discusses the community and engineering project supporting the library.The library’s ecosystem and development milestones frame the future-work priorities.
- 6 Conclusion and Future Works: Future milestones include domain-specific datasets and models, robust Thai NLP benchmarks, correctness improvements, efficient resource management, and language-agnostic integration.These priorities address specialized use cases, model and dataset comparison, deterministic behavior, restricted-network deployment, and integration with tools such as spaCy.
- 6 Conclusion and Future Works: PyThaiNLP aims to integrate its text-processing functions and models with standard NLP libraries such as spaCy and Hugging Face.The stated long-term goal is for developers to use standard language-agnostic libraries rather than require PyThaiNLP directly.
Limitations
PyThaiNLP’s continuous testing is constrained by the memory and cost demands of large language models in free-tier infrastructure.
- Limitations: Free-tier continuous-integration infrastructure cannot reliably run all large-language-model tests because a single model can exhaust available memory.Some components can be cached to reduce build time, but they still must be loaded into memory.
- Limitations: The project dropped some LLM-related tests, reducing library code coverage.The limitation arises within the automated test workflow that runs for every repository commit across supported platforms.
- Limitations: Offline fake-inference tests using stubs, mocks, or spies could reduce repeated LLM loading and make testing more economical and sustainable.Remaining inference could run on a non-free-tier provider or an external computer after reducing the number of actual model calls.