Multilingual Neural Machine Translation with Language Clustering

• Prior multilingual NMT work studied how to share model components across language pairs but did not determine which pairs should share a model.A separate model for every language pair is unaffordable given the number of languages worldwide.Source: §1 · p. 1
• The framework groups languages into clusters and trains one multilingual model for each cluster, rather than placing all languages in one model.It considers many-to-one translation into English and one-to-many translation from English.Source: §1 · pp. 1–2
• The paper compares clustering by language family with clustering learned from language embeddings trained in a universal NMT model.Language tags are learned end-to-end; the authors argue this task-derived clustering may better serve multilingual NMT.Source: §1 · p. 2
• On 23 languages→English and English→23 languages, language-embedding clustering captures language similarity and outperforms language-family clustering in BLEU scores.The embeddings also correlate with fine-grained language-family structure, and similar languages clustered together can boost multilingual performance.Source: §1 · p. 2

• Prior multilingual NMT methods include one-to-many systems with a shared encoder and separate decoders, and many-to-many systems with multiple encoders and decoders.Other approaches share attention or use a universal encoder and decoder.Source: §2 · p. 2
• Some prior systems use a universal encoder and decoder to handle multiple source and target languages, with a source-side tag identifying the target language.Source: §2 · p. 2
• The paper identifies a gap: previous work assumes the language set is given rather than studying which languages should share one model.Source: §2 · p. 2

3 Multilingual NMT with Language Clustering

• Grouping languages into clusters and training one multilingual NMT model per cluster is intended to balance translation accuracy and computational cost.Compared with one model per language pair, clustering reduces computational cost; compared with one model for all pairs, it aims for better accuracy.Source: §3 · pp. 2–3

3.1 Prior Knowledge Based Clustering

• The prior-knowledge method groups languages from the same language family into one cluster, using a commonly accepted taxonomy as its basis.The study selects language family as its comparison method rather than comprehensively evaluating other prior knowledge sources.Source: §3.1 · p. 3

3.2 Language Embedding Based Clustering

• The embedding method trains a universal NMT model, uses its learned language vectors as language representations, and clusters those vectors.The encoder receives both word and language embeddings; hierarchical clustering is used in the experiments.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 Discussions

• Family-based clustering is understandable and stable, but its taxonomy misses isolate languages, still implies many families, and cannot capture all language features.Because embeddings are learned in the downstream NMT task, their clusters are intended to capture language similarities relevant to NMT; the experiments report that these clusters combine multiple language characteristics and boost multilingual-model performance.Source: §3.3 · pp. 3–4

• The experiments use IWSLT data from 2011–2018 for 23 language pairs with English, tokenized with a shared 90K-token BPE vocabulary.The language-family clustering of the 23 languages yields eight groups.Source: §4 · p. 4, Figure 2 · p. 4
• Each cluster uses the same two-layer Transformer configuration, with hidden size 256, feed-forward size 1024, and language-embedding size 256.Source: §4 · p. 4
• Training upsamples each language's data to roughly equal amounts and concatenates language pairs in mini-batches; decoding uses beam size 6 and length penalty α = 1.1.Translation quality is evaluated using tokenized case-sensitive BLEU.Source: §4 · p. 4

5.1 Results of Language Clustering

• Embedding clustering yields seven clusters by the elbow method, compared with eight language-family groups, and includes both family-related and cross-family groupings.Source: §5.1 · pp. 4–5, Figure 4 · p. 5, Figure 3 · p. 5
• The embeddings capture family branches, shared morphology, and regional or historical influence, including the Ja/Tr/Hu and He/Ar/Fa clusters.Source: §5.1 · pp. 5–6

5.2 Translation Accuracy

• Embedding outperforms Individual (mostly by more than 1 BLEU score) on 12 out of 23 languages, especially low-resource cases; Universal generally performs worst.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Clustering Results with respect to Training Data Scale

• Clustering remains stable for most languages at 50% or 20% data, but 5% disrupts more; Individual is slightly better on 8 data-rich languages, mostly within 0.5 BLEU.Source: §5.3 · pp. 6–7, Figure 5 · p. 7

5.4 Results of One-to-Many Translation

• In English→23, clusters align with Romance, Germanic, and mostly Slavic branches, while also grouping Chinese with Japanese and Thai with Vietnamese.Source: §5.4 · pp. 7–8, Figure 6 · p. 8

5 Results

• Embedding clustering produces more reasonable clusters and higher BLEU for most languages than family-based clustering in both translation directions; Random performs worst in 23→English.Source: Table 1 · p. 6, §5.2 · p. 6, Table 3 · p. 8

5.4 Results of One-to-Many Translation

• In one-to-many translation, Individual performs best on nearly 10 languages, but clustering methods outperform it for several smaller-data languages.Source: Table 3 · p. 8, §5.4 · p. 8

• Across 23 languages→English and English→23 languages, language embeddings outperform language-family knowledge for clustering in BLEU scores.Source: §6 · p. 8
• The authors plan to test the methods for many-to-many translation and expand evaluation to hundreds or thousands of languages.Source: §6 · p. 8
• Future work will explore obtaining embeddings from monolingual data and other pre-training methods for multilingual and low-resource NMT.Source: §6 · p. 9

• Evaluation covers 23 IWSLT language↔English translation pairs from campaigns 2011–2018; default validation and test sets are used except for Bulgarian, Greek, Hungarian, and Japanese, which use randomly split 1K training sentence pairs for validation and test.Source: A Dataset Description · p. 9
• For Bulgarian, Greek, Hungarian, and Japanese, the authors randomly split 1K training sentence pairs for validation and test data, respectively.Source: A Dataset Description · p. 9

• The appendix lists each language with its corresponding ISO 639-1 code in Table 5.Source: B Language Name and Code · p. 9

• The elbow method automatically selects 5 clusters for the one-to-many setting, as shown in Figure 7.Source: C The Elbow Method for One-to-Many… · p. 9, Figure 7 · p. 9