PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Multilingual Models with Language-Clustered Vocabularies

Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, Jason Riesa

arXiv:2010.12777v1cs.CLcs.LG

TL;DR

Multilingual vocabulary-generation methods can favor frequent subwords across languages, potentially disadvantaging low-resource languages. The paper clusters languages by individual vocabulary similarity and combines separately trained cluster vocabularies; it reports consistent gains over the standard recipe across evaluations without increasing model size, compute, or data.

  • Problem

    Joint frequency-based vocabulary generation may yield suboptimal decompositions for low-resource languages and favor subwords from common scripts.

    Source: §1 · p. 1

  • Method

    The method groups languages by similarities in their individual subword vocabularies, trains SentencePiece separately for each cluster, and combines the resulting vocabularies.

    Source: §1 · p. 2

  • Results

    The method consistently outperforms the standard vocabulary recipe, reaching state of the art on TYDI QA and zero-shot cross-lingual NER without increasing model size, compute, or data.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    The clustering-based vocabulary method can be applied to any model that uses a vocabulary because it does not depend on model architecture.

    Source: §1 · p. 2

    The stated study scope is investigating the effect of vocabulary composition on multilingual models.

    Source: §4 · p. 3

Abstract

from arXiv · show

State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively multilingual applications. In this work, we introduce a novel procedure for multilingual vocabulary generation that combines the separately trained vocabularies of several automatically derived language clusters, thus balancing the trade-off between cross-lingual subword sharing and language-specific vocabularies. Our experiments show improvements across languages on key multilingual benchmark tasks TyDi QA (+2.9 F1), XNLI (+2.1\%), and WikiAnn NER (+2.8 F1) and factor of 8 reduction in out-of-vocabulary rate, all without increasing the size of the model or data.

1 Introduction

Multilingual models use a shared subword vocabulary, whose embedding matrix can account for a substantial share of model parameters. The paper proposes language clustering to balance cross-lingual sharing with language-specific representation and reports gains on three downstream tasks.

1 Introduction

  • 47% of XLM-R’s parameters are in its embedding matrix, making vocabulary size a consequential part of model scaling.The paper argues that scaling vocabulary size therefore requires inductive biases that guide parameter learning.Source: §1 · p. 1
  • Joint frequency-based vocabulary generation may produce suboptimal decompositions for low-resource languages and favor subwords from common scripts.For example, a subword useful in French and Spanish need not be split from words in every language.Source: §1 · p. 1
  • The proposed method clusters languages by similarities in their individual subword vocabularies, trains SentencePiece separately per cluster, then combines the cluster vocabularies.It aims to balance cross-lingual subword sharing with robust representations of individual languages.Source: §1 · p. 2
  • The method improves performance on TYDI QA, XNLI, and WikiAnn NER, reaching state of the art on TYDI QA and zero-shot cross-lingual NER without model changes.The approach does not depend on model architecture and can be applied to any model that uses a vocabulary.Source: §1 · p. 2

2 Clustered Vocabularies

The method represents each language by the subwords in its own vocabulary, clusters these representations, and trains one vocabulary per cluster. It combines the cluster vocabularies into a unified multilingual vocabulary, allocating capacity according to cluster vocabulary overlap.

2 Clustered Vocabularies

  • Each language is represented by a binary vector over the union of individual vocabularies, with shared subwords giving languages more similar representations.The vector marks which subwords occur in that language’s vocabulary.Source: §2 · p. 2, Figure 1 · p. 2
  • The authors use k-means with cosine distance and fix k = 8 for their 104-language pretraining data based on preliminary experiments.Table 1 frames k = 1 as the JOINT baseline and k = 104 as a separate vocabulary for each language.Source: §2 · p. 2, Table 1 · p. 2
  • SentencePiece is trained on pooled data within each cluster, with target vocabulary size proportional to the union of member languages’ individual vocabularies.This allocation grows with more languages in a cluster and shrinks when their vocabularies overlap more.Source: §2 · p. 2
  • The unified vocabulary is the union of all cluster vocabularies, and the method generalizes the conventional single-cluster approach.The conventional approach corresponds to |C| = 1.Source: §2 · p. 2

3 Intrinsic Analysis

The intrinsic analysis examines whether CLUSTER balances subword sharing with languages’ freedom to contribute useful subwords. It compares vocabularies trained on the same multilingual data using distributional distances and the representation of rare scripts.

3 Intrinsic Analysis

  • The analysis evaluates how well JOINT and CLUSTER balance cross-language sharing against each language’s ability to contribute effective subwords.The paper frames this as an examination of the vocabularies produced by the two recipes.Source: §3 · p. 2
  • The analysis uses Wikipedia articles from 104 languages, with 906M sentences, and sets each multilingual vocabulary to 488k subwords.The cluster assignments and sizes are presented in Table 2.Source: §3 · p. 2, Table 2 · p. 3
  • CLUSTER shows larger Wasserstein-1 distances for empirically different languages and smaller distances for languages in the same cluster, unlike JOINT’s relatively small distances even across distinct scripts.The analysis interprets this pattern as less sharing between dissimilar languages and more sharing when appropriate.Source: §3 · p. 3, Table 3 · p. 3
  • CLUSTER contains a higher percentage of Chinese, Japanese, and Korean subwords because those languages share a cluster and can select subwords independently of other languages.The passage reports a similar pattern for Arabic script.Source: §3 · p. 3

4 Experiments

The experiments compare CLUSTER with JOINT across downstream tasks, analyze how vocabulary composition relates to segmentation and OOV rates, and test CLUSTER in a full-scale model.

4 Experiments

  • The evaluation used TYDI QA, XNLI, and WikiAnn NER, while keeping the number of languages and model parameters constant across comparisons.Pretraining used masked language modeling on raw text without language-specific preprocessing.Source: §4 · p. 3
  • CLUSTER increased macro-average performance over JOINT across all datasets and tasks; gains covered all TYDI QA languages and all but French on XNLI, while NER language-level results were mixed.For NER, CLUSTER had large gains on low-resource or rare-script languages and small losses on high-resource languages.Source: §4 · p. 3, Table 5 · p. 4, Table 6 · p. 4

4.1 Analysis

  • CLUSTER reduced training-corpus description length, which the authors report correlates with downstream gains, and had an 8-times smaller OOV rate than JOINT.The OOV reductions were especially large for Korean (factor of 26), Japanese (18), and Chinese (17); the authors also propose description length as a proxy for comparing vocabularies.Source: §4.1 · p. 4
  • For low-resource Latin-script languages, CLUSTER had 9.4% shorter description length for Swahili and 11.1% for Yoruba than JOINT, alongside 7.5 and 8.0 higher NER F1, respectively.The authors attribute the results they report for Swahili and Yoruba to better segmentation; for example, CLUSTER keeps Swahili lazima intact where JOINT splits it into la and zima.Source: §4.1 · p. 4
  • For rare-script languages, CLUSTER strongly outperformed JOINT across tasks; the authors hypothesize that greater rare-script subword coverage and lower OOV rates contribute to the gains.NER gains included 28.9 F1 in Urdu; Chinese, Japanese, and Korean also showed strong improvements across tasks and large OOV-rate reductions.Source: §4.1 · p. 5
  • CLUSTER grouped some different-script Indic languages, whose Wikipedia articles share names and places, and the authors suggest those shared entity representations may relate to strong NER results in cluster c8.They describe such data-driven capture of corpus characteristics as a strength of the approach.Source: §4.1 · p. 5

4.2 Full-scale model

  • 10.7 F1 absolute improvement on MINSPAN and 13.3 F1 on SELECTP, plus 8.2 F1 absolute improvement over XLM-R on NER, were reported for the 24-layer CLUSTER model; XNLI had a small loss.The authors suggest the XNLI loss may reflect less training data and Wikipedia-only training.Source: Table 9 · p. 5, §4.2 · p. 5

5 Conclusion

The paper presents clustering-based multilingual vocabulary generation and reports consistent gains over the standard recipe without increasing model size, compute, or data.

5 Conclusion

  • The clustering-based vocabulary method consistently outperforms the standard multilingual vocabulary recipe without increasing model size, compute, or data.The conclusion describes it as empirically motivated.Source: §5 · p. 5

A Experiment details

The appendix details pretraining and SentencePiece configurations, including language-data sampling, masking, and model-training settings.

A.1 Pretraining

  • Pretraining used LAMB with a batch size of 4096, learning rate 0.0018, linear warm-up and decay, and gradient clipping at norm 1.0.The warm-up proportion was 2.5% of total steps.Source: §A.1 · p. 8
  • Most experiments used 64 TPUs; sequence lengths of 128 and 512 ran for 500k and 125k steps, respectively, while the full-scale model used 256 TPUs for 1.5 million steps.The 512-length model was used for WikiAnn NER and TYDI QA; the 128-length model was used for XNLI. SentencePiece used a unigram model, 0.9995 character coverage, and 1M seed sentences.Source: §A.1 · p. 8, §A.2 · p. 8
  • Language sampling used sentence-count proportions with exponential smoothing of 0.7, and whole-word masking defined words by spans between SentencePiece U+2581 markers.The word-boundary heuristic accommodates languages that do not typically use whitespace between words.Source: §A.1 · p. 8

A.3 Model architecture

The appendix contrasts the standard experimental model architecture with the larger full-scale model.

A.3 Model architecture

  • Most models had 12 Transformer layers, hidden size 768, 12 attention heads, and 128-dimensional embeddings, totaling 150M parameters.The embedding size was chosen for faster experimentation.Source: §A.3 · p. 8
  • The full-scale model had 24 layers, hidden size 1024, 16 attention heads, and 512-dimensional embeddings, totaling 550M parameters to mimic XLM-R.The architecture description distinguishes it from the smaller experimental models.Source: §A.3 · p. 8

A.4 Fine-tuning

Fine-tuning used task-specific settings for WikiAnn NER, XNLI, and TYDI QA, with model selection based on development-set metrics.

A.4 Fine-tuning

  • Fine-tuning experiments used two seeds and selected the best model based on the average of the two runs.Adam was used for fine-tuning.Source: §A.4 · p. 8
  • WikiAnn NER used learning rate 4 × 10−5, batch size 32, and 2 epochs, with span-level F1 as its evaluation metric.The same hyperparameters were used for CLUSTER and JOINT on the development set.Source: §A.4 · p. 8, Table 11 · p. 9
  • XNLI used batch size 32, searched learning rates [1×10−5, 2× 10−5, 3 × 10−5] for 3 epochs, and selected models by development macro-averaged accuracy.The evaluation metric was classification accuracy; the table caption gives the best CLUSTER and JOINT settings as 2 × 10−5, batch size 32, and 3 epochs.Source: §A.4 · p. 8, Table 10 · p. 9
  • TYDI QA used batch size 512 and searched learning rates [3×10−5, 4×10−5, 5×10−5] and epochs [7, 8, 9], selecting by macro-averaged F1 across 10 non-English languages.Its settings also included 45 maximum passages, 0.1 include unknown rates, sequence length 512, and window stride 128.Source: §A.4 · p. 8

B Datasets

Pretraining uses Wikipedia across 104 languages, with XNLI, TyDi QA, and WikiAnn NER as evaluation datasets. The appendix describes dataset handling and development-set reporting, including a special evaluation constraint for TyDi QA.

B Datasets

  • 906M Wikipedia sentences across 104 languages support pretraining; evaluation covers XNLI in 15 languages, TyDi QA in 11, and WikiAnn NER in 40.English is excluded from TyDi QA's official evaluation.Source: B Datasets · p. 9
  • WikiAnn NER's Greek, Thai, Japanese, Korean, and Chinese IOB2 tags were corrected when an I-tag followed O; XNLI and TyDi QA had no preprocessing.No examples were excluded from the three datasets.Source: B Datasets · p. 9

C Performance on development set

  • The paper reports development-set results for XNLI and WikiAnn NER so researchers can try to reproduce the results without consulting the test set.Source: C Performance on development set · p. 9
  • Because TyDi QA's test set is private, experiments used its development set except for the full-scale model, which was submitted to the official leaderboard.The appendix also reports the full-scale model's development results for TyDi QA, XNLI, and WikiAnn NER.Source: C Performance on development set · p. 9

C.1 Development set results for the full-scale model

The full-scale Transformer required task-specific hyperparameters. The appendix used LAMB and searched different learning-rate and epoch ranges for XNLI and TyDi QA, while WikiAnn NER used a fixed two-epoch setup.

C.1 Development set results for the full-scale model

  • The full-scale Transformer used LAMB and task-specific settings: XNLI searched learning rates 9 × 10−5–1 × 10−4 and 9–11 epochs; TyDi QA searched 8 × 10−5–9 × 10−5 and 13–15 epochs.Both used batch size 512; WikiAnn NER used a learning rate from 2 × 10−5–3 × 10−5, batch size 32, and 2 epochs.Source: §C.1 · p. 9

C.2 WikiAnn NER test set results on all 40 languages

The appendix expands WikiAnn NER test results to all 40 languages and lists the 104 pretraining languages with their scripts.

C.3 Language information

  • Table 18 lists the 104 pretraining languages and their script information.The listed scripts include Latin, Arabic, Cyrillic, and other scripts.Source: §C.3 · pp. 9, 11, Table 18 · p. 11

C.2 WikiAnn NER test set results on all 40 languages

  • 64.5 average F1 for CLUSTER versus 61.7 for JOINT is reported in Table 16; Table 17 reports an average F1 of 73.6.Tables 16 and 17 expand the results to all 40 languages considered in XTREME.Source: §C.2 · p. 9, Table 16 · p. 10, Table 17 · p. 10

Related papers

Loading 2010.12777v1…