Source-linked AI summary
Contextual Parameter Generation for Universal Neural Machine Translation
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, Tom Mitchell
TL;DR
Multilingual NMT must share information across languages without discarding language-specific parameterization. The paper introduces a contextual parameter generator that conditions shared encoder and decoder parameters on language embeddings, achieving stronger results across IWSLT-15 and IWSLT-17 while supporting monolingual and zero-shot learning.
Problem
Existing multilingual NMT approaches either lack language-specific parameterization or fail to share information across languages, creating limitations for differing languages, limited data, or similar languages.
Method
A contextual parameter generator uses source and target language embeddings to generate encoder and decoder parameters for a shared NMT model.
Results
The method consistently outperforms corresponding pairwise and multilingual baselines on IWSLT-15 and IWSLT-17, including supervised and zero-shot settings.
Takeaways & Limitations
The framework supports multilingual, semi-supervised, and zero-shot translation while allowing controllable parameter sharing and adaptation to new languages.
Takeaways & Limitations
The multilingual setting assumes source and target language sets with available pairwise parallel corpora, potentially covering only some language combinations.
Abstract
from arXiv · showhide
We propose a simple modification to existing neural machine translation (NMT) models that enables using a single universal model to translate between multiple languages while allowing for language specific parameterization, and that can also be used for domain adaptation. Our approach requires no changes to the model architecture of a standard NMT system, but instead introduces a new component, the contextual parameter generator (CPG), that generates the parameters of the system (e.g., weights in a neural network). This parameter generator accepts source and target language embeddings as input, and generates the parameters for the encoder and the decoder, respectively. The rest of the model remains unchanged and is shared across all languages. We show how this simple modification enables the system to use monolingual data for training and also perform zero-shot translation. We further show it is able to surpass state-of-the-art performance for both the IWSLT-15 and IWSLT-17 datasets and that the learned language embeddings are able to uncover interesting relationships between languages.
1 Introduction
Multilingual NMT must balance sharing across languages with language-specific modeling. The paper introduces contextual parameter generation as a shared, adaptable alternative that supports multilingual, semi-supervised, and zero-shot translation.
- Motivation: Universal NMT shares one model but lacks language-specific parameters, whereas per-language encoder-decoders do not share information across languages.The former can be detrimental for very different languages and limited data, while the latter can overparameterize similar languages.
- Contribution: The contextual parameter generator generates language-conditioned parameters for a shared translation model, balancing separate language modeling with information sharing.It uses language embeddings as context and can enhance existing NMT systems with minor modifications.
- Capabilities: The approach supports multilingual translation, monolingual-data training, zero-shot translation, controlled sharing, and adaptation to new languages without complete retraining.These capabilities follow from extensive but controllable parameter sharing across languages.
- Results: The method surpasses pairwise NMT and Johnson et al. (2017), including reported state-of-the-art performance.The paper evaluates these claims on IWSLT-15 and IWSLT-17 experiments.
2 Background
The paper frames multilingual NMT as learning translation across source-target language pairs from available parallel corpora. It decomposes NMT into preprocessing, encoder-decoder, and parameter-generator modules to characterize existing systems and motivate multilingual parameter generation.
- Setting: Multilingual NMT seeks one model that translates between language pairs using a subset of available pairwise parallel corpora.The setting allows up to |S| × |T| source-target combinations, while most prior work considers only one pair.
- Preprocessing Pipeline: Preprocessing tokenizes and normalizes text, builds vocabularies, and maps sentences to word-index sequences; BPE creates subword vocabularies and eliminates out-of-vocabulary words.The passage associates BPE with increased translation quality.
- Encoder/Decoder: The encoder maps indexed source sentences to intermediate representations, and the decoder generates target sentences using learned parameter sets.Attention over the input sequence is cited as an encoder-decoder design that improves translation performance.
- Parameter Generator: A parameter generator produces encoder and decoder parameters for each source-target language choice, with different generators determining how much information languages share.This module provides a common description of pairwise, per-language, and universal multilingual systems.
P P P
The modular framework distinguishes parameter-sharing choices for multilingual NMT. It positions the proposed approach between pairwise or per-language specialization and a fully universal model.
- Parameterization Options: Pairwise models use separate encoder and decoder parameters for each language pair, while per-language models share encoder parameters by source language and decoder parameters by target language.The parameter-count scaling changes from O(ST) for pairwise parameters to O(S+T) for per-language parameters.
- Parameterization Options: Universal NMT uses the same parameters for all language pairs but can perform poorly with limited training data and embeds languages alongside words.The paper instead places language embeddings in a separate space.
- Proposed Alternative: The proposed parameter-generator alternative learns separate language representations while sharing information across languages, enabling semi-supervised and zero-shot learning.Language embeddings act as translation context and define parameter sharing across encoders and decoders.
3 Proposed Method
The contextual parameter generator conditions encoder and decoder parameters on language embeddings while preserving a shared NMT architecture. Its decoupled form supports interlingual representations, controlled low-rank sharing, monolingual training, zero-shot translation, and parameter-efficient expansion.
- Contextual Parameter Generator: The decoupled generator defines encoder parameters from the source embedding and decoder parameters from the target embedding, leaving the intermediate representation independent of the target language.This supports translating one encoded sentence into any target language and motivates an interlingua interpretation.
- Parameter Generator Network: The contextual parameter generator is a general module whose functional form is not fixed; experiments use simple linear transforms for encoder and decoder parameters.The formulation can also couple both stages using source and target embeddings, although experiments use the decoupled form.
- Parameter Sharing: The linear generator imposes a low-rank parameterization, replacing pairwise parameter vectors with one shared matrix and one embedding per language.For L languages, pairwise models require L(L−1) parameter vectors, while the proposed representation uses L language vectors and a shared matrix.
- Controlled Parameter Sharing: Grouped parameter generation controls sharing by extracting a smaller M′-dimensional subrepresentation for each encoder or decoder parameter group from an M-dimensional language embedding.Increasing M while keeping M′ small increases language-specific capacity without causing the total parameter count to explode.
- Semi-supervised and Zero-shot Learning: Monolingual data can train shared networks through self-translation, and zero-shot translation is possible when language-specific parameters are reused across translation directions.A new language can be added by learning its embedding while fixing parameters shared across existing languages.
- Parameter Efficiency: O(PM + LWV) parameters are required by the linear generator, compared with O(L2P + 2L2WV) for multiple pairwise models.The proposed count removes the quadratic dependence on L from encoder-decoder parameters.
4 Experiments
Experiments evaluate CPG across supervised, low-resource, and zero-shot multilingual translation settings. Across IWSLT-15 and IWSLT-17, CPG improves over pairwise and multilingual baselines, supports monolingual-data training and zero-shot translation, and yields linguistically meaningful language embeddings.
- Setup: The base system is a bidirectional-LSTM encoder with a two-layer LSTM attention decoder, used without extensive hyperparameter tuning.Training uses label smoothing, AMSGrad, batch size 128, and beam search with beam size 10.
- Experimental Settings: The experiments compare supervised, low-resource, and zero-shot translation, with CPG using monolingual data through auto-encoding in low-resource training.Zero-shot evaluation tests language pairs absent from training; PNMT instead relies on pivoting, which can propagate translation errors.
- Results: Up to 5.06 BLEU points, or a 2.4× increase, separates CPG from corresponding baselines in IWSLT-15 low-resource experiments.The advantage over pairwise NMT and GML grows when less parallel data is available.
- Results: CPG achieves 29.03 BLEU on IWSLT-15 En→Vi and 26.77 on En→De, exceeding the cited prior results of 28.07 and 25.87, respectively.These comparisons concern additional published results not included in the main tables.
- Results: CPG consistently outperforms PNMT and GML on IWSLT-17 in both supervised and zero-shot settings.Performance remains robust across language-embedding sizes and controlled-sharing choices, except when the sharing rank is reduced to M′ = 1.
- Language Embeddings: Learned language embeddings recover relationships such as German–Dutch similarity and Romanian–German separation in IWSLT-17.The paper also reports a French–Vietnamese relationship in IWSLT-15, which it associates with French influence in Vietnam.
5 Related Work
Multilingual NMT research has explored shared models, monolingual-data training, and zero-shot translation, but the paper positions contextual parameter generation as a distinct way to condition generated neural parameters on language context.
- Multilingual NMT includes shared-encoder, many-to-one, unsupervised, and zero-shot approaches, with parameter sharing supporting monolingual-data training and zero-shot translation.
- Earlier multilingual systems faced requirements including multi-way parallel corpora, pseudo-parallel data, iterative training, or shared representations, depending on the approach.
- The paper distinguishes its method by generating deep neural-network parameters from source and target language context, rather than using structural network features or generating only linear-model parameters.
6 Conclusion and Future Directions
The paper concludes that contextual parameter generation provides a shared yet language-conditioned NMT system, and identifies broader applications in multimodal translation and adaptable lifelong learning.
- The proposed system generates encoder and decoder parameters from source and target languages while retaining a standard pairwise encoder-decoder architecture.
- The approach improves the parameter-to-example ratio through shared structure and achieves state-of-the-art performance without back-translation, dual learning, pivoting, multilingual word embeddings, or tuning.
- Future work extends contextual parameter generation to multimodal translation and adaptable, never-ending NMT learning.