PaperlayerView exact revision on arXiv

Source-linked AI summary

GREEK-BERT: The Greeks visiting Sesame Street

John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, Ion Androutsopoulos

arXiv:2008.12014v2cs.CL

TL;DR

Pre-trained Transformers have delivered strong NLP results but have mostly targeted resource-rich English, while Greek lacked a publicly available pre-trained Transformer model. This paper introduces and evaluates monolingual GREEK-BERT on three Greek NLP tasks. It reports state-of-the-art results on all datasets and 5–10% margins over competitors in two tasks.

  • Problem

    Transformer language models have mostly targeted English, and no publicly available pre-trained Transformer model was available for modern Greek to the authors’ knowledge.

    Source: §1 · p. 1

  • Method

    The authors introduce a monolingual BERT-based model for Greek, trained on 29 GB of text with a 35k sub-word BPE vocabulary, and evaluate it on three NLP tasks.

    Source: §1 · p. 1

  • Results

    GREEK-BERT achieves state-of-the-art results across all datasets and outperforms competitors by 5–10% in two tasks.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors publicly release GREEK-BERT, training code, and fine-tuning code, expecting these resources to boost Greek NLP research and applications.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Transformer-based language models, such as BERT and its variants, have achieved state-of-the-art performance in several downstream natural language processing (NLP) tasks on generic benchmark datasets (e.g., GLUE, SQUAD, RACE). However, these models have mostly been applied to the resource-rich English language. In this paper, we present GREEK-BERT, a monolingual BERT-based language model for modern Greek. We evaluate its performance in three NLP tasks, i.e., part-of-speech tagging, named entity recognition, and natural language inference, obtaining state-of-the-art performance. Interestingly, in two of the benchmarks GREEK-BERT outperforms two multilingual Transformer-based models (M-BERT, XLM-R), as well as shallower neural baselines operating on pre-trained word embeddings, by a large margin (5%-10%). Most importantly, we make both GREEK-BERT and our training code publicly available, along with code illustrating how GREEK-BERT can be fine-tuned for downstream NLP tasks. We expect these resources to boost NLP research and applications for modern Greek.

1 INTRODUCTION

Greek NLP lacked a publicly available pre-trained Transformer, despite strong results from Transformer models on English benchmarks. GREEK-BERT addresses this gap with a monolingual Greek model, evaluates it across three tasks, and releases the model and supporting code.

1 INTRODUCTION

  • GREEK-BERT is a monolingual BERT-based model for Greek, trained on 29 GB of Greek text with a 35k sub-word BPE vocabulary.The vocabulary was created from scratch.Source: §1 · p. 1
  • GREEK-BERT achieves state-of-the-art results in all datasets across three core tasks—PoS tagging, NER, and NLI—and outperforms its competitors by 5–10% in two datasets.Comparisons include multilingual Transformer models and strong neural baselines using pre-trained word embeddings.Source: §1 · p. 1
  • The paper evaluates GREEK-BERT on part-of-speech tagging, named entity recognition, and natural language inference.Source: §1 · p. 1
  • The authors publicly release the pre-trained model, training code, and code illustrating downstream fine-tuning.Source: §1 · p. 1

2 RELATED WORK

Prior work established BERT-style pre-training, multilingual Transformer models, and monolingual models for other languages, while Greek NLP lacked a publicly available pre-trained Transformer model. The paper situates GREEK-BERT within these approaches and resource gaps.

2.1 BERT: A Transformer language model

  • BERT uses Transformer-based contextual representations and is pre-trained with masked language modeling and next sentence prediction.Its inputs can be sentence pairs marked with [CLS] and [SEP].Source: §2.1 · p. 2

2.2 Multilingual language models

  • M-BERT and XLM-R cover Greek within multilingual models, but Greek uses only a small portion of their shared vocabularies.The passage reports 1,208 Greek WordPieces for M-BERT and 4,862 sub-words for XLM-R.Source: §2.2 · pp. 2–3

2.3 Monolingual language models

  • Monolingual Transformer models for French and Finnish reported state-of-the-art results across several downstream NLP tasks.CAMEMBERT and FinBERT were among the cited models.Source: §2.3 · p. 3

2.4 NLP in Greek

  • Greek NLP resources, especially deep-learning resources, remained limited, and no Greek-specific pre-trained Transformer model was publicly available to the authors’ knowledge.The authors also argue that comparisons should include neural methods relying on pre-trained word embeddings, not only multilingual Transformers.Source: §2.4 · p. 3

3 GREEK-BERT

GREEK-BERT adapts the BERT-BASE-UNCASED architecture to Greek and pre-trains on 29 GB of Greek text. The authors normalize the corpora and derive a 35k BPE vocabulary from them.

3 GREEK-BERT

  • GREEK-BERT uses BERT-BASE-UNCASED because BERT-LARGE-UNCASED is computationally heavy for pre-training and fine-tuning.Source: §3 · p. 3
  • The 29 GB pre-training corpus combines Greek Wikipedia, Europarl proceedings, and the Greek portion of OSCAR.OSCAR is described as a clean version of Common Crawl.Source: §3 · p. 3
  • The authors removed accents and other diacritics, lower-cased the text, and extracted a 35k BPE vocabulary from the same corpora.They used SentencePiece to extract the vocabulary.Source: §3 · p. 3
  • Pre-training used the MLM and NSP tasks for 1M steps with Adam on a Google Cloud TPU v3-8 and took about five days.The initial learning rate was 1e-4.Source: §3 · p. 3

4 BENCHMARKS

The benchmarks cover Greek part-of-speech tagging, named entity recognition, and natural language inference. Their datasets range from a treebank to merged entity annotations and translated inference pairs.

4.1 Part-of-Speech tagging

  • The Greek Universal Dependencies Treebank provides 2,521 sentences, split into 1,622 training, 403 development, and 456 test sentences, with 17 universal PoS tags.The benchmark uses the PoS annotations and ignores syntactic dependencies.Source: §4.1 · p. 3

4.2 Named Entity Recognition

  • The NER benchmark merges two unpublished datasets of 1,798 and 2,521 sentences into 4,189 unique sentences, using only Person, Organization, and Location labels.Duplicate sentences were eliminated; other entity types were not shared across the datasets.Source: §4.2 · p. 3

4.3 Natural Language Inference

  • The Greek XNLI benchmark uses 2,500 development and 5,000 test pairs, while its 340k training pairs were automatically translated from English.The authors note that training-pair quality is questionable and consider only the Greek portion of XNLI.Source: §4.3 · p. 4

5 EXPERIMENTAL SETUP

The experiments compare GREEK-BERT with multilingual Transformers and word-embedding baselines across Greek NLP tasks. They also examine XNLI training-data quality and tune models using development loss.

5.1 Multilingual models

  • Multilingual models are compared with GREEK-BERT on Greek tasks; they fragment words more, with M-BERT variants averaging about twice GREEK-BERT’s fragmentation ratio.The word fragmentation ratio is the average number of sub-word tokens per word; XLM-R fragments less than M-BERT, and its Greek vocabulary coverage is four times larger.Source: §5.1 · p. 4, Table 2 · p. 4
  • Long sequences of meaningless sub-words may be difficult to reassemble into meaningful units, while whole-word embedding baselines do not have this problem.The paper illustrates over-fragmentation with Greek words split into several pieces by M-BERT-UNCASED, although those words appear whole in GREEK-BERT’s vocabulary.Source: §5.1 · p. 4

5.2 Baselines operating on word embeddings

  • PoS tagging and NER use BILSTM-CNN-CRF, which combines pre-trained word and character-CNN embeddings before bidirectional LSTM context encoding and CRF prediction.Each word’s character representation is produced by a CNN; the concatenated word representations are passed through stacked bidirectional LSTMs.Source: §5.2 · p. 4
  • For NLI, the re-implemented Decomposable Attention Model uses attention, comparison, and aggregation components to predict from premise and hypothesis representations.Attention uses projected word-embedding dot products and softmax-normalized similarities; aggregation uses an MLP classifier.Source: §5.2 · p. 4

5.3 Implementation details and hyper-parameter tuning

  • Experiments use Greek FastText 300-dimensional embeddings for embedding-based baselines and select architectures by grid search on development loss, with early stopping.Models are run with three random seeds, and mean test scores and unbiased standard deviations are reported; the full XNLI training-set experiments are not repeated across seeds.Source: §5.3 · pp. 4–5

5.4 Denoising XNLI training data

  • For XNLI, the study compares fine-tuning on the full machine-translated training set with a high-quality subset of 40,000 pairs selected by GREEK-BERT perplexity.The subset is about 10% of the 340,000 training pairs; full-set experiments use one run because repeated runs are computationally more expensive.Source: §5.4 · p. 5

6 EXPERIMENTAL RESULTS

GREEK-BERT leads the NLI and NER comparisons, while Transformer models perform similarly on PoS tagging. The results also identify persistent difficulties in specific PoS tags, entity types, and NLI classes.

PoS tagging

  • Transformer models achieve 97.8–98.2 accuracy in PoS tagging, with XLM-R marginally (+0.1%) ahead of GREEK-BERT and M-BERT-CASED.The authors attribute the similar scores partly to Greek suffixes and short function words often indicating PoS tags without much context.Source: §6 · p. 5, Table 3 · p. 5
  • Both GREEK-BERT and XLM-R struggle most with the other (X) tag, followed by proper nouns (PROPN) and numerals (NUM), whose morphology or context can overlap.For example, a Greek word may be either a foreign proper noun or the common noun for “castle.”Source: §6 · p. 5, Table 4 · p. 5

NER

  • GREEK-BERT leads in NER by 9.3% over BILSTM-CNN-CRF, 3.9% over M-BERT on average, and 0.9% over XLM-R.NER results are farther from perfect than PoS results, leaving more room for methods to distinguish themselves.Source: §6 · p. 5, Table 5 · p. 6
  • GREEK-BERT performs better on persons and organizations, while XLM-R is marginally better on locations; both models struggle with organizations that contain person or location names.Both models are more accurate on persons and locations than on organizations.Source: §6 · pp. 5–6, Table 6 · p. 6

NLI

  • GREEK-BERT beats DAM by 10.1%, M-BERT by 4.9% on average, and XLM-R by 1.3% in NLI; all models improve with the full training set.The authors suspect noise may act as a regularizer, but leave careful error analysis for future work.Source: §6 · p. 6, Table 7 · p. 6
  • GREEK-BERT is better in all three NLI classes, although both models have difficulty identifying neutral pairs, which are often confused with contradiction pairs.Source: §6 · p. 6

7 CONCLUSIONS AND FUTURE WORK

GREEK-BERT reaches state-of-the-art across three Greek NLP tasks and outperforms strong baselines on NER and NLI. The authors release the model and code and outline broader-corpus and task directions.

7 CONCLUSIONS AND FUTURE WORK

  • GREEK-BERT achieves state-of-the-art performance in PoS tagging, NER, and NLI, outperforming strong baselines in the latter two tasks.Table 8 reports test-set F1 scores per NLI class for GREEK-BERT and XLM-R, both trained on all training data.Source: §7 · p. 6, Table 8 · p. 6
  • The comparison includes M-BERT and XLM-R, plus shallower neural methods BILSTM-CNN-CRF and DAM operating on word embeddings.The authors release the pre-trained model, experiment-replication code, and code illustrating fine-tuning; they expect these resources to boost Greek NLP, where public resources, especially for deep learning, remain scarce.Source: §7 · p. 6
  • The authors plan another GREEK-BERT version trained on larger corpora, including Greek and EU legislation, while filtering noisy material such as documents containing tables.The planned legislative corpora contain well-written Greek policy text across domains including economy, health, education, and agriculture.Source: §7 · p. 6
  • They plan to evaluate more downstream tasks, including dependency parsing, subject to the availability of additional public Greek datasets.Source: §7 · p. 6
  • They also propose pre-training BERT-based models for earlier Greek, especially classical Greek, for which large datasets are available.The authors say this could potentially improve NLP tools for classical studies.Source: §7 · p. 6

A EXAMPLES FROM THE GREEK PART OF XNLI CORPUS

The appendix presents examples from the Greek XNLI corpus, including noisy samples and samples selected by GREEK-BERT perplexity.

A EXAMPLES FROM THE GREEK PART OF XNLI CORPUS

Loading 2008.12014v2…