Source-linked AI summary
PhoBERT: Pre-trained language models for Vietnamese
Dat Quoc Nguyen, Anh Tuan Nguyen
TL;DR
Vietnamese NLP lacks large, representative monolingual pretraining data and models that distinguish syllables from words. PhoBERT addresses these gaps with released base and large Vietnamese models pretrained on a segmented 20GB corpus, achieving state-of-the-art results and outperforming XLM-R across four tasks. A reported limitation is that PhoBERT-large scores lower than PhoBERT-base on dependency parsing.
Problem
Vietnamese monolingual pretraining has relied on limited Wikipedia data, while existing BERT-based models do not distinguish Vietnamese syllables from word tokens.
Method
PhoBERT trains base and large monolingual BERT-based models on a 20GB word-level Vietnamese corpus and evaluates them across four downstream tasks.
Results
PhoBERT achieves state-of-the-art performance on four Vietnamese NLP tasks and outperforms XLM-R across these tasks.
Takeaways & Limitations
The released PhoBERT models provide a strong baseline for future Vietnamese NLP research and applications.
Takeaways & Limitations
PhoBERT-large achieves 0.9% lower dependency parsing scores than PhoBERT-base, possibly because the last Transformer layer is not optimal for syntactic information.
Abstract
from arXiv · showhide
We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. We release PhoBERT to facilitate future research and downstream applications for Vietnamese NLP. Our PhoBERT models are available at https://github.com/VinAIResearch/PhoBERT
1 Introduction
PhoBERT addresses limited Vietnamese pretraining data and the lack of syllable–word awareness in existing models. It introduces large-scale monolingual Vietnamese models that achieve strong downstream performance and outperform XLM-R across four tasks.
- Motivation: Existing Vietnamese language models rely mainly on a small Wikipedia corpus that is not representative of general language use.The Vietnamese Wikipedia data is about 1GB uncompressed, while more pretraining data can significantly improve pretrained language models.
- Motivation: Existing BERT-based models do not distinguish Vietnamese syllables from word tokens, complicating word-level NLP.Vietnamese whitespace separates syllables within words, and applying BPE directly to syllable-level data may reduce performance on word-level tasks.
- Contribution: PhoBERT presents the first large-scale monolingual BERT-based language models pretrained for Vietnamese.The models include “base” and “large” versions trained on a 20GB word-level Vietnamese corpus.
- Results: PhoBERT achieves state-of-the-art performance on POS tagging, dependency parsing, NER, and NLI.These results cover three word-level tasks and one language-understanding task formulated at syllable or word level.
- Results: PhoBERT outperforms the multilingual model XLM-R across four Vietnamese-specific tasks, supporting dedicated language-specific pretraining.The paper describes this as the first multi-task comparison between monolingual and multilingual language models in this setting.
- Contribution: The authors publicly release PhoBERT for use with fairseq and transformers to support Vietnamese NLP research and applications.They intend the models to serve as strong baselines for future work.
2 PhoBERT
PhoBERT uses BERT-base and BERT-large architectures with RoBERTa-based pretraining. Its 20GB corpus combines Vietnamese Wikipedia and deduplicated news, which is word- and sentence-segmented before subword tokenization and optimization.
- Architecture: PhoBERT provides base and large versions using the corresponding BERT-base and BERT-large architectures.Its pretraining approach is based on RoBERTa, which optimizes the BERT pretraining procedure for robust performance.
- Pre-training data: The pretraining corpus contains 20GB of uncompressed Vietnamese text from Wikipedia and deduplicated news articles.The corpus combines approximately 1GB of Vietnamese Wikipedia with approximately 19GB generated from a 50GB Vietnamese news corpus.
- Pre-training data: RDRSegmenter produces approximately 145M word-segmented sentences and approximately 3B word tokens before fastBPE subword segmentation.PhoBERT uses a 64K subword vocabulary, averaging 24.4 subword tokens per sentence.
- Optimization: Training uses RoBERTa’s fairseq implementation, Adam optimization, GPU batches, learning-rate warmup, and 40 training epochs.The setup uses maximum sequences of 256 subword tokens and differs in batch size and peak learning rate between the base and large models.
3 Experimental setup
PhoBERT is evaluated on four Vietnamese NLP tasks using standard benchmarks and Vietnamese XNLI data. Task-specific prediction layers, parsers, and fine-tuning procedures are applied across datasets.
- Evaluation tasks: PhoBERT is evaluated on POS tagging, dependency parsing, NER, and NLI.The first three are word-level Vietnamese tasks, while NLI uses Vietnamese validation and test sets from XNLI.
- Datasets: POS tagging, dependency parsing, and NER use the VLSP 2013, VnDT v1.1, and VLSP 2016 benchmarks.The setup follows the VnCoreNLP evaluation configuration.
- Datasets: For NLI, Vietnamese validation and test sets are manually constructed, while the Vietnamese training set is machine-translated from English.RDRSegmenter is applied before BPE because these NLI data do not provide gold word segmentation.
- Task models: POS tagging and NER add a linear prediction layer over PhoBERT, while dependency parsing uses an extended biaffine parser.The POS and NER layer uses the contextualized representation of each word’s first subword.
- Fine-tuning: POS tagging, NER, and NLI are fine-tuned independently with transformers using AdamW, fixed learning rates, early stopping, and five random-seed runs.The reported test score is selected from the best validation checkpoint after training for up to 30 epochs.
4 Experimental results
PhoBERT achieves new state-of-the-art results across four Vietnamese NLP tasks and outperforms XLM-R on word-level tasks despite using substantially less pre-training data. Results also show a small dependency-parsing disadvantage for PhoBERTlarge relative to PhoBERTbase.
- PhoBERT produces new state-of-the-art results on POS tagging, Dependency parsing, NER, and NLI.
- POS tagging: 0.8% absolute higher accuracy gives PhoBERT an advantage over the previous approximately 96.0% POS-tagging state of the art.
- Dependency parsing: 4% absolute improvement boosts the Biaffine parser to 78.8% LAS and 85.2% UAS.
- NER: PhoBERTlarge achieves 1.1 points higher F1 than PhoBERTbase, while PhoBERTbase exceeds previous NER state-of-the-art models by 2+ points.
- Discussion: PhoBERTlarge scores 0.9% lower on dependency parsing than PhoBERTbase, possibly because the final Transformer layer is not optimal for syntactic information.
- Comparison with XLM-R: PhoBERT outperforms XLM-R on POS tagging, Dependency parsing, and NER despite XLM-R using about seven times more Vietnamese pre-training text.
5 Conclusion
The paper introduces and publicly releases PhoBERT, the first large-scale monolingual Vietnamese language models. PhoBERT outperforms XLM-R and supports state-of-the-art performance across four Vietnamese NLP tasks, with the authors targeting future research and applications.
- PhoBERT is presented as the first large-scale monolingual language model pre-trained for Vietnamese.
- PhoBERT performs better than XLM-R and helps produce state-of-the-art results for POS tagging, Dependency parsing, NER, and NLI.
- The publicly released PhoBERT models are intended to foster future Vietnamese NLP research and applications.