PaperlayerView exact revision on arXiv

Source-linked AI summary

Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?

Tim Isbister, Fredrik Carlsson, Magnus Sahlgren

arXiv:2104.10441v1cs.CLcs.LG

TL;DR

Because Scandinavian monolingual models are much smaller than leading English models, the paper tests translating sentiment data into English for use with large language models. It compares native and translated data and finds that translation-based approaches outperform native models in most tested Scandinavian languages, with Finnish as an exception.

  • Problem

    Scandinavian models are far smaller than leading English models, prompting the question of whether data can instead be translated into English to use those larger models.

    Source: §1 · p. 1

  • Method

    The authors translate Scandinavian sentiment data into English before experiments, then evaluate English models on the translated data alongside models evaluated on native data.

    Source: §3.1 · p. 2, §3 · p. 2

  • Results

    Large English models on translated data outperform native models in most tested Scandinavian sentiment settings, with Finnish the exception; English BERT-Base on translated data is about 1% unit higher than Swedish BERT on native data.

    Source: §5.3 · p. 4, §6 · p. 4

  • Takeaways & Limitations

    The results provide a counter-argument to routinely developing native solutions and suggest hosting one large English model for practical applications that receive requests in different languages.

    Source: §6 · p. 5

    The approach's applicability depends on translation quality and the availability of large-scale models in the target language; Finnish was the exception, with insufficient translation quality a likely factor.

    Source: §6 · pp. 4–5

Abstract

from arXiv · show

Most work in NLP makes the assumption that it is desirable to develop solutions in the native language in question. There is consequently a strong trend towards building native language models even for low-resource languages. This paper questions this development, and explores the idea of simply translating the data into English, thereby enabling the use of pretrained, and large-scale, English language models. We demonstrate empirically that a large English language model coupled with modern machine translation outperforms native language models in most Scandinavian languages. The exception to this is Finnish, which we assume is due to inferior translation quality. Our results suggest that machine translation is a mature technology, which raises a serious counter-argument for training native language models for low-resource languages. This paper therefore strives to make a provocative but important point. As English language models are improving at an unprecedented pace, which in turn improves machine translation, it is from an empirical and environmental stand-point more effective to translate data from low-resource languages into English, than to build language models for such languages.

1 Introduction

Large English language models have advanced rapidly, while Scandinavian models remain far smaller. The paper asks whether translating data into English could make large English models a better alternative to native models.

1 Introduction

  • Leading language models have grown in size and performance, culminating in GPT-3 with 175 billion parameters trained on about 45 TB of data.The passage links model size and performance to Kaplan et al. (2020).Source: §1 · p. 1
  • Around 300 Scandinavian models were available in Hugging Face Transformers at the time of writing, but none approached leading English models in parameter size or training-data scale.Many of the Scandinavian models were translation models, alongside a significant number of monolingual models.Source: §1 · p. 1
  • The paper asks whether translating data and tasks into English could replace the need to build Scandinavian models.The question arises because Scandinavian models are much smaller than leading English models, and the authors expect their relative performance to be worse.Source: §1 · p. 1

2 Related work

Prior work studies transferring multilingual models to new languages, while this paper studies transferring data into English to use state-of-the-art models. It contrasts modern machine translation with an earlier study that reported degradation even under perfect translation.

2 Related work

  • Multilingual-model research examines cross-lingual transfer to apply models to original-language data, with smaller-data languages benefiting from transfer from related languages.The passage cites Conneau et al. (2020a), Xue et al. (2020), Ruder et al. (2019), Artetxe et al. (2020), Lauscher et al. (2020), Conneau et al. (2020b), Karthikeyan et al. (2020), and Nooralahzadeh et al. (2020).Source: §2 · p. 2
  • This paper instead transfers data into another language to enable use of state-of-the-art models for the task.The authors identify Duh et al. (2011) as the only prior study they know in this direction; it used outdated methods and claimed degradation even with perfect translation.Source: §2 · p. 2
  • Using modern machine translation, the authors report no observable performance degradation when applying large state-of-the-art models.This finding contrasts with the claim attributed to Duh et al. (2011).Source: §2 · p. 2

3 Data

The authors construct comparable sentiment datasets for four Scandinavian languages and translate them into English before experiments.

3 Data

  • ScandiSent comprises Trustpilot reviews in Swedish, Danish, Norwegian, and Finnish, spanning 22 topics with reviews evenly distributed across categories for each language.The reviews were gathered from the corresponding language subdomains.Source: §3 · p. 2
  • Review ratings were converted to positive labels for ratings 4 or 5 and negative labels for ratings 1 or 2, with neutral ratings discarded.The review ratings ranged from 1 to 5.Source: §3 · p. 2
  • Language identification filtering produced 10,000 texts per language, split into 7,500 training and 2,500 test samples.The filtering used fastText's language identification model to remove reviews in an incorrect language.Source: §3 · p. 2

3.1 Translation

  • All four language datasets were translated into English with Google's neural machine translation model before fine-tuning, and the translation model was not updated during training.The authors fixed translations in advance to isolate the effects of machine translation.Source: §3.1 · p. 2

4 Models

The experiments compare popular native BERT-Base models with English BERT models and XLM-R-Large, using models with different architectures and training-corpus sizes.

4 Models

  • For each Scandinavian language, the authors selected the most popular native Hugging Face model, which was BERT-Base.The native models share an architecture; they differ mainly in training-text quantity and type, as well as potentially in hyperparameters.Source: §4 · p. 2
  • The comparison includes English BERT-Base, BERT-Large, and XLM-R-Large alongside the Scandinavian models.English BERT-Base matches the native models' architecture; BERT-Large is twice as deep and has more than three times BERT-Base's parameters, while XLM-R-Large was trained on significantly larger multilingual corpora.Source: §4 · p. 2
  • Table 1 reports vocabulary size, lexical richness, average word length, and average sentence length for each language's Trustpilot sentiment data.Source: Table 1 · p. 3
  • Table 2 lists the experimental models and their training-data sizes, while Tables 3 and 4 report accuracy results for monolingual models and XLM-R-Large, respectively.Table 3 separates results on native and machine-translated sentiment data.Source: Table 2 · p. 3, Table 3 · p. 3, Table 4 · p. 3

5 Experiments

The experiments compare fine-tuned models on sentiment datasets using binary accuracy. English BERT-Large on machine-translated data beats native models in all considered languages except Finnish, while XLM-R-Large leads on sentiment data for all languages except Finnish.

5.1 Setup

  • The setup fine-tunes and evaluates each model on the sentiment datasets and reports binary accuracy for BERT and XLM-R.Results are presented in Tables 3 and 4, respectively.Source: §5.1 · pp. 3–4

5.2 Monolingual Results

  • Native models form a clear diagonal, and performance correlates with pretraining-data amount, with the least-data Danish model performing worst on its native language.Bert-Large significantly outperforms BERT-Base on all non-English datasets and performs slightly better on English data.Source: §5.2 · p. 4
  • In cross-lingual evaluation, BERT-Large leads the non-native models; Swedish performs best on Norwegian and Finnish, while Norwegian leads on Danish.The Norwegian model performs considerably worse than other Scandinavian models on English data.Source: §5.2 · p. 4

5.3 Translation Results

  • BERT-Large on machine-translated data outperforms every native model on its native data except Finnish; English BERT-Base also exceeds Norwegian and Danish native models.English BERT-Base is about 1% unit below Swedish BERT on native data.Source: §5.3 · p. 4
  • Scandinavian models generally perform significantly worse on their own machine-translated data, but show no clear cross-language trend; Danish and Finnish have specific translated-data gains.Danish does better on translated Swedish than original Swedish; Finnish improves on other translated sets except Swedish, and all non-Finnish models do better on translated Finnish.Source: §5.3 · p. 4
  • XLM-R-Large achieves top sentiment scores for all languages except Finnish, with its best Danish result on machine-translated data.For Swedish, Norwegian, and Finnish, its native-data results are slightly better.Source: §5.3 · p. 4

6 Discussion & Conclusion

In sentiment analysis, translating data into English and using a large pretrained English model can outperform smaller native models, though Finnish is an exception. The authors frame mature machine translation as a counterargument to training native models where translation to English is high-quality and native-model training data is scarcer.

6 Discussion & Conclusion

  • English translation paired with a large pretrained model outperforms a smaller native model on sentiment analysis, though the result’s reach beyond that task remains unshown.The authors report no theoretical reason it would not extend to other tasks, but say this remains to be shown.Source: §6 · p. 4
  • Downstream performance strongly correlates with pretraining-data quantity, while XLM-R’s results may also reflect cross-language transfer.The authors present data size and transfer as possible explanations for XLM-R’s performance, not established causes.Source: §6 · p. 4
  • Finnish is the only task where the native model performs best; insufficient translation quality may explain BERT-Large’s poor results on translated Finnish.The authors also hypothesize that cross-lingual transfer may favor typologically similar languages and disadvantage typologically unique Finnish.Source: §6 · p. 4
  • The proposed alternative depends on high-quality machine translation to English and less training data for building a native language model.These are the stated conditions for considering it unnecessary to train a model in that language.Source: §6 · pp. 4–5
  • The authors argue that mature machine translation makes relying on existing large English models a practical counterargument to habitual native-language development.They suggest hosting one large English model and translating incoming requests from different languages.Source: §6 · p. 5

A Training Details

Table 5 is identified as the location of training hyperparameters for the sentiment-classification experiments.

A Training Details

  • Table 5 summarizes training hyperparameters for the sentiment-classification experiments.Source: Table 5 · p. 6
Loading 2104.10441v1…