PaperlayerView exact revision on arXiv

Source-linked AI summary

Small and Practical BERT Models for Sequence Labeling

Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, Amelia Archer

arXiv:1909.00100v1cs.CL

TL;DR

The paper addresses multilingual sequence labeling across diverse languages, where separate language models are often used. It fine-tunes multilingual BERT and distills the model for practical deployment; the distilled model is smaller and faster while retaining quality gains and performing well against multilingual and BERT baselines.

  • Problem

    Prior morphosyntactic tagging improvements largely use separate models for each language or small groups, while the paper examines one model across almost 50 languages.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes pretrained BERT for multilingual sequence labeling and distills the multilingual model into a smaller version for single-CPU use.

    Source: §1 · p. 1, §5 · p. 4

  • Results

    The distilled model is 27x faster and 6 times smaller than a BERT-based baseline, outperforms a multilingual baseline, and matches or exceeds large BERT performance for most languages.

    Source: §1 · p. 1, §5 · p. 5

  • Takeaways & Limitations

    The distilled model is fast enough for single-CPU use and performs well on low-resource languages.

    Source: §5 · p. 4, §4.4 · p. 4

    The Universal Dependencies data has no explicit tuning data for Armenian or Kazakh.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.

1 Introduction

The paper studies whether one multilingual sequence-labeling model can serve diverse languages where prior systems often use separate language-specific models. It presents a fine-tuned BERT approach with accuracy gains, especially for low-resource languages, and a smaller, faster version intended for single-CPU deployment.

1 Introduction

  • One multilingual model replaces separate models for a diverse set of almost 50 languages, evaluated on part-of-speech tagging and morphological attribute prediction.The evaluation uses data from the Universal Dependencies repository.Source: §1 · p. 1
  • Parameter sharing reduces model size and enables cross-lingual transfer, with accuracy improvements especially for low-resource languages.Source: §1 · p. 1
  • A multilingual model can process codemixed input without explicit training on codemixed labeled examples.Source: §1 · p. 1
  • The model matches or exceeds language-specific state-of-the-art accuracy, outperforms multilingual baselines, and is especially stronger on low-resource languages.It is evaluated on part-of-speech tagging and morphological attribute prediction using Universal Dependencies data.Source: §1 · p. 1
  • The final model is 27x faster and 6 times smaller than a BERT-based baseline while preserving most quality gains.The stated target is practical use on a single CPU.Source: §1 · p. 1

2 Multilingual Models for Sequence Labeling

The section introduces Meta-LSTM and multilingual BERT as sequence-labeling models for multilingual training. It describes their model setups and the comparisons used to assess F1, parameter counts, and inference speed.

2 Multilingual Models for Sequence Labeling

  • Table 1 compares macro-averaged F1 for per-language and multilingual models over 48 languages; non-multilingual scores average the trained per-language models.Source: Table 1 · p. 2

2.1 Meta-LSTM

  • The Meta-LSTM combines character- and word-BiLSTMs through a joint BiLSTM, and is identified as the best-performing CoNLL 2018 Shared Task model for these tasks.Source: §2.1 · p. 2

2.2 Multilingual BERT

  • The multilingual BERT experiments use a cased checkpoint pretrained on 104 languages, with a softmax over each token’s first wordpiece and examples combined across languages.Training concatenates examples from all treebanks and randomly shuffles them.Source: §2.2 · p. 2
  • Table 2 compares model parameter counts, separating embedding-row tokens from hidden units outside the embedding and output layers.Source: Table 2 · p. 2
  • MiniBERT is the fastest model on CPU and GPU in the relative-speed comparison over BERT.The comparison specifies an Intel Xeon CPU and Nvidia Titan V GPU.Source: Table 3 · p. 2

3 Small and Practical Models

The section frames deployment as a CPU speed and memory problem for BERT while seeking to preserve its task capability. It presents a smaller MiniBERT and a teacher-to-student distillation procedure.

3 Small and Practical Models

  • The goal is a model fast enough for a single CPU while maintaining the large model’s capability on the tasks.Source: §3 · p. 2

Size and speed

  • MiniBERT is a three-layer BERT with fewer embedding parameters and hidden units than both comparison models, and it has the fewest total parameters.Source: Size and speed · pp. 2–3
  • MetaLSTM’s bottleneck is sequential character-LSTM unrolling, whereas BERT’s is large feedforward layers and attention with quadratic sequence-length complexity.Source: Size and speed · p. 3
  • MiniBERT is significantly faster than the other models on both GPU and CPU, while BERT is slower than MiniBERT and MetaLSTM on CPU.Source: Size and speed · p. 3

Distillation

  • Wikipedia sentences in languages for which multilingual BERT is pretrained supply unlabeled teacher-logit distillation data before student finetuning on the teacher’s labeled data.Source: Distillation · p. 3

4 Experiments

The experiments compare multilingual and per-language sequence-labeling models, then evaluate transfer on low-resource languages and performance on codemixed input. They report strong multilingual-model results, including for codemixed data without supervised codemixed training.

4.1 Data

  • Experiments use CoNLL 2018 part-of-speech and morphology data, excluding languages absent from the multilingual BERT checkpoint.POS uses 17 labels; morphology unions its groups into an 18,334-label output. Models are trained and tuned on gold-segmented data, with UDPipe v2.2 segmenting raw test text.Source: §4.1 · p. 3

4.3 Multilingual Models

  • mBERT performs best among multilingual models; mMiniBERT performs comparably to mBERT and outperforms state-of-the-art mMeta-LSTM.The table reports macro-averaged F1 for multilingual models.Source: Table 4 · p. 3, §4.3 · p. 3

4.2 Tuning

  • Among tested distillation temperatures T=1, 2, and 3, T=3 consistently gives the best teacher-student accuracy.The authors report that BERT is very confident in its predictions.Source: §4.2 · p. 3

4.4 Low Resource Languages

  • With fewer than 50 examples, mBERT is significantly better than BERT; distilled mMiniBERT beats individual BERT models, with smaller gains as training data increases.Source: Table 5 · p. 4, §4.4 · p. 4
  • mMiniBERT outperforms state-of-the-art Meta-LSTM on POS tagging and on four of eight languages in the morphology task.Source: Table 5 · p. 4, §4.4 · p. 4

Multilingual Models vs Per-Language Models

4.5 Codemixed Input

  • All multilingual models handle Hindi-English codemixed POS reasonably without supervised codemixed training; when the top label is X, evaluation uses the usually-correct second choice.The evaluation uses the Devanagari-script tokens supplied with the dataset.Source: §4.5 · p. 4

5 Conclusion

The conclusion presents a distilled multilingual model as a practical way to run BERT-based sequence labeling on a single CPU. It reports preserved quality gains, faster inference, and better performance than a multilingual strong baseline.

  • The distilled multilingual model runs fast enough for a single CPU while maintaining the quality wins of the larger BERT-based model.Source: §5 · p. 4
  • The final model outperforms a multilingual version of a very strong baseline and gives most languages performance comparable to or better than large BERT.Source: §5 · p. 5

A.1 Training Hyperparameters

The appendix gives multilingual BERT fine-tuning settings and describes modifications to its configuration for mMiniBERT.

A.1 Training Hyperparameters

  • The POS task is trained for 10 epochs and the morphology task for 50 epochs, using the public multilingual BERT fine-tuning hyperparameters.The same hyperparameters are used for all tasks during distillation.Source: §A.1 · p. 6
  • The Wikipedia pretraining data is used as-is after dropping sentences shorter than 10 characters.Source: §A.1 · p. 6

A.2 Small BERT structure

  • mMiniBERT uses the cased public multilingual model’s vocabulary and wordpiece model, with modifications to the public multilingual BERT configuration.Source: §A.2 · p. 6

B.1 The Importance of Distillation

The ablation compares distillation with training the MiniBERT structure from scratch on labeled multilingual data. It reports that distillation helps close the accuracy gap between teacher and student.

B.1 The Importance of Distillation

  • Without distillation, the MiniBERT-structure model is trained from scratch using only the labeled multilingual data used to train the teacher.Source: §B.1 · p. 6
  • Distillation plays an important role in closing the accuracy gap between the teacher and student.Source: §B.1 · p. 6
  • The ablation tests distillation’s importance by comparing the distilled model with a MiniBERT-structure model trained from scratch.Source: §B.1 · p. 6

B.2 Per-Language Results

The section reports per-language F1 results for POS tagging and morphology. Its macro-averaged results exclude languages without tuning data for per-language models.

B.2 Per-Language Results

  • Per-language F1 results are shown in Tables 8 and 9, but languages without tuning data for per-language models are not reported and are excluded from macro averages.Source: §B.2 · p. 6
  • 3.5 points lower macro-averaged POS F1 and 3.1 points lower morphology F1 are reported for the model without distillation.Source: Table 7 · p. 6
  • Table 8 reports POS-tagging F1 for all models.The listed models are BERT, Meta-LSTM, mBERT, mMeta-LSTM, and mMiniBERT.Source: §B.2 · p. 7, Table 8 · p. 9
  • Table 9 reports morphology F1 for all models.Source: Table 9 · p. 11

Related papers

Loading 1909.00100v1…