PaperlayerView exact revision on arXiv

Source-linked AI summary

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

arXiv:2012.15562v3cs.CL

TL;DR

The paper addresses poor cross-lingual transfer to low-resource languages, especially those absent from pretraining and written in unseen scripts. It develops data-efficient embedding and adapter adaptations that reuse pretrained lexical and latent information, achieving substantial gains for unseen scripts and improvements for low-resource languages with covered scripts.

  • Problem

    Pretrained multilingual models transfer poorly to low-resource languages, with the greatest challenge arising for languages and scripts absent from pretraining.

  • Method

    The paper adapts pretrained models with dedicated target-language embeddings, lexically overlapping tokens, matrix factorization, and language-cluster information while keeping most pretrained parameters fixed.

  • Results

    The methods outperform previous approaches and achieve the largest gains for languages with unseen scripts, while also improving low-resource languages written in covered scripts.

  • Takeaways & Limitations

    Pretrained embedding information can support efficient adaptation to resource-poor languages across both unseen and covered scripts.

  • Takeaways & Limitations

    Lexical-overlap initialization is less useful for languages written in different scripts, and named-entity recognition may overstate generalization when overlapping tokens are named entities.

Abstract

from arXiv · show

Massively multilingual language models such as multilingual BERT offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. However, due to limited capacity and large differences in pretraining data sizes, there is a profound performance gap between resource-rich and resource-poor target languages. The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts unseen during pretraining. In this work, we propose a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts. Relying on matrix factorization, our methods capitalize on the existing latent knowledge about multiple languages already available in the pretrained model's embedding matrix. Furthermore, we show that learning of the new dedicated embedding matrix in the target language can be improved by leveraging a small number of vocabulary items (i.e., the so-called lexically overlapping tokens) shared between mBERT's and target language vocabulary. Our adaptation techniques offer substantial performance gains for languages with unseen scripts. We also demonstrate that they can yield improvements for low-resource languages written in scripts covered by the pretrained model.

1 Introduction

Pretrained multilingual models transfer poorly to low-resource languages, especially those written in scripts absent from pretraining. The paper introduces data-efficient adaptation methods that exploit pretrained embedding information and improve performance across unseen and covered scripts.

  • Unseen scripts cause pretrained multilingual models to fail off-the-shelf because their tokenizers cannot adequately represent the target language.Many tokens are replaced by UNKs, preventing effective transfer without adaptation.
  • The proposed methods adapt pretrained models using dedicated target-language embeddings, lexically overlapping tokens, matrix factorization, and token groupings.These techniques aim to align new target-language representations with knowledge encoded in the existing embedding matrix.
  • The study evaluates transfer from four resource-rich source languages to 17 NER and 6 dependency-parsing target languages, including five with unseen scripts.The unseen-script targets are Amharic, Tibetan, Khmer, Divehi, and Sinhala.
  • The adaptation techniques outperform previous approaches and deliver substantial gains for unseen-script languages while also improving low-resource languages with covered scripts.The methods are presented as data-efficient alternatives for adapting multilingual models to resource-poor languages.

2 Background: Multilingual Model Adaptation for Cross-lingual Transfer

Cross-lingual transfer commonly adapts multilingual models through language- and task-specific parameters while keeping the pretrained model fixed. However, shared vocabularies and representation spaces remain poorly suited to resource-poor languages, especially those using unseen scripts.

  • Standard cross-lingual transfer fine-tunes a model on labeled source-language data and applies it directly to a target language.Because the model balances many languages in one representation space, it may not excel for a specific target language without adaptation.
  • Adapter-based methods store language-specific knowledge in dedicated parameters while keeping the pretrained multilingual model’s parameters fixed.MAD-X uses language, task, and invertible adapters trained for different objectives and swaps source and target language adapters at inference.
  • Adapters consist of a down-projection, GeLU activation, and up-projection inserted at each Transformer layer.The down-projection maps the hidden size h to adapter dimension d, and the up-projection maps d back to h.
  • Existing adapter methods do not directly address extended vocabularies or adequately adapt the representation space for resource-poor and unseen-script languages.The paper therefore trains target-language embeddings alongside language adapters while retaining the rest of the model.

3 Cross-lingual Transfer of Lexical Information

The section examines how vocabulary mismatch limits multilingual transfer and presents embedding-based adaptations that reuse pretrained lexical information for new languages and scripts.

  • 3 Cross-lingual Transfer of Lexical Information: The embedding matrix contains substantial multilingual information, but vocabulary mismatch leaves many target-language tokens unknown or unavailable to mBERT.The embedding matrix accounts for around 50% of multilingual models’ parameters, while UNKs cannot be composed from mBERT’s subword vocabulary.
  • 3.1 Target-Language Embedding Learning: EL-RAND learns a dedicated target-language embedding matrix, initializing only shared special tokens from pretrained representations and training the remaining embeddings with the pretraining task.This approach adapts the embedding layer but uses pretrained lexical information only implicitly beyond shared special tokens.
  • 3.2 Initialization with Lexical Overlap: EL-LEX initializes lexically overlapping target tokens with their pretrained representations while randomly initializing non-overlapping tokens, then fine-tunes all target embeddings.The method selectively transfers pretrained information using surface-level token similarity, which is rarer across different scripts.
  • 3.3 Embedding Matrix Factorization: Matrix factorization represents pretrained embeddings with lower-dimensional token embeddings and shared up-projection matrices that encode general cross-lingual information.In the single-projection case, Semi-NMF restricts the projection matrix to non-negative values while leaving token-specific factors unconstrained.
  • 3.3 Embedding Matrix Factorization: Factorized embeddings reduce trainable parameters and storage for each additional language, especially when the reduced dimension D′ is much smaller than the original dimension D.The approach can also use multiple up-projection matrices by grouping tokens into clusters, with KMeans providing one clustering strategy.

4 Experiments

The experiments evaluate multilingual transfer methods across low-resource languages, including settings with seen and unseen scripts, using NER and dependency parsing. They compare adapter-based transfer, new tokenizers, embedding initialization strategies, and factorized low-dimensional embeddings.

  • Data and Languages: The study evaluates NER and dependency parsing across low-resource languages with scripts either covered or uncovered by mBERT pretraining.The language selection is designed to compare transfer under different script-coverage conditions.
  • Transfer Methods: MAD-X 2.0 removes the adapter from the last transformer layer to avoid constraining the final representation during zero-shot language-adapter replacement.This is the novel architectural variant evaluated alongside the original MAD-X framework.
  • Embedding Adaptation: New-tokenizer methods compare random and lexical-overlap initialization for full-dimensional and factorized embeddings.The factorized variants use either a shared projection matrix or multiple cluster-specific up-projection matrices, with assignments learned by k-means or Gumbel-Softmax.
  • Transfer Methods: The experiments primarily use adapter-based training, jointly training target-language adapters and embeddings while keeping the remaining pretrained model fixed.For zero-shot transfer, the source adapter and entire embedding layer are replaced with target-language components.
  • Training Setup: Each target language receives a 10k-vocabulary WordPiece tokenizer, while Semi-NMF initializes 100-dimensional factorized embeddings from the pretrained mBERT embedding matrix.The models use Wikipedia for masked-language-model pretraining and are fine-tuned for 10 epochs using the best source-language development F1 score.
  • Results: The reported NER and dependency-parsing results compare original-tokenizer systems with new-tokenizer systems, separating random initialization from lexical initialization across source languages en, ar, zh, and ja.Mean test results are averaged over the evaluation settings described for the two tasks.

5 Results and Discussion

The proposed adaptation methods improve transfer across low-resource languages, with especially large gains for unseen scripts and additional benefits from lexical overlap and matrix factorization.

  • Overall results: MAD-X 2.0 considerably outperforms MAD-X, although both remain weak on unseen scripts because mBERT tokenization produces many UNKs.The main NER and dependency-parsing results are reported in Tables 3a and 3b.
  • Overall results: The largest gains occur for languages with unseen scripts, including Amharic, Tibetan, Khmer, Divehi, and Sinhala, while Latin-script languages remain competitive.Proposed methods also improve under-represented languages such as Min Dong, Mingrelian, and Sindhi.
  • Initialization: Lexical-overlap initialization improves performance for most languages, adding gains even though shared vocabulary is not required for multilingual generalization.The result indicates that lexical overlap is useful without being necessary.
  • Matrix factorization: Matrix-factorization methods outperform full-sized embedding methods, especially without lexical-overlap initialization, showing that factorized embeddings extract useful information from the original matrix.Combining factorization with lexical initialization leverages both surface similarity and latent embedding knowledge.
  • Matrix factorization: For covered-script unseen languages, MF10*-* reaches the peak score for 6/9 languages and also outperforms MAD-X 2.0 on average.The KMEANS variant performs better for NER, whereas the NEURAL variant performs better for UD.
  • Unseen scripts: Factorized information has smaller impact on unseen scripts, whereas lexical overlap remains helpful because those languages are too distant to benefit substantially from latent multilingual knowledge.This contrasts with the stronger factorization gains for languages more similar to those seen during pretraining.

6 Further Analysis

Further analysis examines lexical overlap, sample efficiency, embedding dimensionality, and structure in the pretrained embedding matrix. The findings indicate that overlap and factorized representations support efficient adaptation, while lexical overlap may bias NER evaluation.

  • 6.1 Lexically Overlapping (Sub)Words: Lexically overlapping tokens in unseen scripts are mostly single non-Latin characters, while frequent overlapping subwords are often Latin-script named entities.This pattern may help explain stronger gains on NER than dependency parsing and limits NER as an objective measure of generalization.
  • 6.2 Sample Efficiency: EL-LEX is slightly more sample-efficient than MF10 KMEANS-LEX, but MF10 KMEANS-LEX performs better when more data are available.The comparison uses zero-shot transfer after adapters and embeddings are pretrained on different amounts of data.
  • 6.3 Embedding Dimensionality: Lower-dimensional embeddings tend to outperform higher-dimensional embeddings for most languages.Table 6 compares 100- and 300-dimensional embeddings using mean F1 averaged over five runs and four high-resource source languages.
  • 6.4 KMeans Analysis: KMeans clustering produces distinct script-based groups, including predominantly Latin, Chinese, Korean, Cyrillic, Arabic, and Indian-script clusters.One additional cluster contains tokens from all scripts except Latin.

7 Conclusion

The conclusion reports a systematic evaluation of adaptation strategies for unseen languages and scripts. It attributes the approach’s gains to exploiting lexical overlap and latent semantic information from the original embedding matrix, including lower-dimensional embeddings and up-projections.

  • 7 Conclusion: The study evaluates original-embedding information through lexically overlapping tokens and latent semantic concepts extracted into lower-dimensional embeddings and up-projections.These analyses support the proposed adaptation methods for languages with seen and unseen scripts.
  • 7 Conclusion: The methods outperform previous approaches on NER and dependency parsing in both resource-rich and resource-poor scenarios.The conclusion states that they reduce the gap between random and lexical-overlap initialization and enable more effective adaptation to unseen scripts.

A.1 Full Model Transfer

The full-model transfer setup follows prior work by learning a target-language embedding matrix while freezing pretrained transformer weights. Task fine-tuning and zero-shot transfer then use different combinations of frozen and replaced embedding components.

  • A.1 Full Model Transfer: The adaptation learns a new target-language embedding matrix while freezing the pretrained transformer weights for comparability with previous work.This follows the setup attributed to Artetxe et al. (2020).
  • A.1 Full Model Transfer: For target-task training, transformer weights are fine-tuned with the original embedding layer frozen; for zero-shot transfer, the original embedding is replaced by the new matrix.

A.2 Results: Named Entity Recognition

The appendix reports non-aggregated NER transfer results for four source languages and describes adapter placement and full-model results. These tables provide source-specific transfer performance beyond the aggregated analyses.

  • A.2 Results: Named Entity Recognition: NER transfer performance is reported separately for English, Chinese, Japanese, and Arabic source languages in Tables 7–10.
  • A.2 Results: Named Entity Recognition: The appendix indicates whether an adapter is placed in the 12th transformer layer and additionally presents full-model results.

A.3 Results: Dependency Parsing

The dependency-parsing results report mean UAS and LAS over five runs for transfers from English, Chinese, Japanese, and Arabic. The appendix also organizes script clusters, pretrained languages, and related NER evaluation tables.

  • A.3 Results: Dependency Parsing: The appendix lists the 104 languages and corresponding scripts used for mBERT pretraining.This information is provided in Table 12.
  • A.3 Results: Dependency Parsing: Figure 4 groups script tokens into 10 KMeans clusters, following prior procedures for splitting UTF-8 tokens into scripts.The accompanying text describes grouping scripts into languages.
  • A.3 Results: Dependency Parsing: Related NER tables report mean F1 results averaged over five runs for transfers from English, Chinese, Japanese, and Arabic.These tables describe tokenizer groups and adapter placement in the 12th transformer layer.
  • A.3 Results: Dependency Parsing: Mean UAS and LAS dependency-parsing results are averaged over five runs for transfers from English, Chinese, Japanese, and Arabic.The results are presented in Tables 11a–11d.
  • A.3 Results: Dependency Parsing: The dependency-parsing tables distinguish models using the original tokenizer from models using new tokenizers.The new-tokenizer group further separates randomly initialized embeddings from lexical initialization.
Loading 2012.15562v3…