Unsupervised Neural Machine Translation with Weight Sharing

1 Introduction

• Prior unsupervised NMT often uses a shared encoder that may not preserve language-specific style, terminology, and sentence structure.The authors conjecture that this may limit translation potential.Source: §1 · pp. 1–2
• The proposed system uses independent encoders and decoders, tying the last few encoder layers and first few decoder layers to map into a shared-latent space.The tied encoder layers extract high-level representations; the tied decoder layers also enforce the shared-latent-space assumption.Source: §1 · p. 2
• Embedding-reinforced encoders and local and global GANs are proposed to support shared-latent alignment and improve cross-language translation.The local GAN aligns latent-representation distributions, while the global GAN guides generated sentences toward the true data distribution.Source: §1 · p. 2
• The approach reports consistent success on English-German, English-French, and Chinese-to-English translation tasks.Experimental results are described as consistently successful.Source: §1 · p. 2
• Directional self-attention models temporal order, and experiments suggest further investigation of temporal order within NMT attention layers.The authors introduce it as an additional component of their proposed model.Source: §1 · p. 2

2 Related Work

• For language pairs with little parallel data but a shared pivot, translation commonly proceeds through the pivot; Johnson et al. (2016) extend multilingual NMT to pairs without direct-pair training data.The cited approaches translate source-to-pivot and pivot-to-target independently.Source: §2 · p. 2
• Lample et al. (2017) and Artetxe et al. (2017b) train from monolingual corpora, and both use one shared encoder that may not preserve each language’s uniqueness.Lample et al. use one decoder as well, while Artetxe et al. use two independent decoders.Source: §2 · p. 2
• Figure 1 depicts tied encoder and decoder layers, distinguishing each language’s self-reconstruction path from its cross-language translation path.The last few encoder layers and first few decoder layers are tied.Source: Figure 1 · p. 3
• The architecture’s shared-latent space is labeled Z; its local discriminator identifies an encoding’s language, while global discriminators assess translated-sentence realism.The caption identifies the two global discriminators as assessing sentences in their respective languages.Source: §2 · p. 3

3.1 Model Architecture

• The architecture combines two language-specific auto-encoders with local and global discriminators, using Transformer encoders and decoders.The local discriminator is an MLP, while the global discriminators use CNNs.Source: §3.1 · p. 3, Table 1 · p. 3
• Directional self-attention uses forward and backward positional masks to encode token order, with successive attention layers processing the sequence in forward and reverse directions.The forward mask limits later tokens to attending to earlier tokens; the backward mask reverses that direction.Source: §3.1 · pp. 3–4
• The model shares the encoders’ last few layers and decoders’ first few layers, retaining independent weights for language-specific features while mapping them to a shared latent space.The independent weights are intended to encode language characteristics such as terminology, style, and sentence structure.Source: §3.1 · p. 4
• The encoders reinforce their outputs with fixed pre-trained cross-lingual embeddings through a gate whose trainable parameters are shared across languages.The approach motivates this reinforcement as supporting the shared latent space and giving the decoder representations at different composition levels.Source: §3.1 · p. 4

3.2 Unsupervised Training

• Denoising auto-encoding shuffles input sentences so the auto-encoders must recover word order rather than merely copy words.The shuffle uses tunable parameters k and s; the paper sets k = 2 and s = 100000.Source: §3.2 · p. 4
• Back-translation translates a monolingual sentence with the other language’s decoder, then uses the translation and original sentence to train reconstruction of the original.This produces a pseudo-parallel corpus for cross-language training.Source: §3.2 · pp. 4–5
• The local GAN trains encoders to fool a discriminator predicting sentence language, while the global GAN discriminator assesses generated against true sentences and fine-tunes the model.The global GAN uses policy-gradient reinforcement training to back-propagate the discriminator’s assessment.Source: §3.2 · p. 5
• Training has two stages: denoising auto-encoding, back-translation, and local GAN training continue until development-set improvement stops, followed by global-GAN fine-tuning.In the first stage, batches for both languages are alternated across the three training strategies.Source: §3.2 · p. 5

4.1 Data Sets and Preprocessing

• Experiments cover English-German, English-French, and Chinese-to-English; English-to-Chinese was not tested because no public test set was available.English-French and English-German use WMT data; Chinese-to-English uses LDC data.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Model Hyper-parameters and Evaluation

• Model selection stops after ten development evaluations without improvement; BLEU is averaged across both translation directions after translating source sentences out and back.The development set contains 3,000 source and target sentences sampled from the monolingual training corpora.Source: §4.2 · p. 6

4.3 Baseline Systems

• The baselines include word-by-word translation, Lample et al. (2017), and a supervised version of the proposed model treated as an upper bound.The word-by-word baseline replaces each word with its nearest neighbor in the other language using an inferred bilingual dictionary.Source: §4.3 · pp. 6–7

4.4.2 Translation results

• At least +5.01 BLEU points on English-to-German and up to +13.37 BLEU points on English-to-French over the word-by-word baseline; the study also reports English-to-Chinese results.Table 2 covers English-German, English-French, and Chinese-to-English test sets.Source: Table 2 · p. 7, §4.4.2 · p. 7

4.4.1 Number of weight-sharing layers

• One shared layer gives the best performance; zero shared layers and four shared layers both perform poorly across the three translation tasks.The encoder-shared model declines by 0.53 BLEU points on English-to-French, 0.85 on English-to-German, and 1.66 on Chinese-to-English relative to the one-layer model.Source: §4.4.1 · p. 7, Figure 2 · p. 7

4.4.2 Translation results

• The unsupervised model remains 12.3-25.5 BLEU points below the supervised upper bound, depending on language pair and translation direction.Compared with Lample et al. (2017), the model improves English-to-French by up to +1.92 BLEU points.Source: §4.4.2 · p. 8

4.4.3 Ablation study

• Ablations favor using all tested components together; weight sharing is most critical, while global and local GANs add up to +0.78 and +0.57 BLEU on English-to-French.Embedding-reinforced encoders improve all translation tasks; removing directional self-attention causes up to -0.3 BLEU points decline.Source: §4.4.3 · p. 8

• The system combines independent encoders with partial weight sharing, embedding-reinforced encoders, local and global GANs, and directional self-attention.Source: §5 · p. 8
• The approach significantly improves unsupervised NMT on English-German, English-French, and Chinese-to-English tasks.Source: §5 · p. 8
• Unsupervised NMT still has substantial room for improvement compared with supervised NMT.Source: §5 · p. 8
• Future work will investigate more effective use of monolingual data, including language models and syntactic information, and stronger temporal-order modeling.Source: §5 · pp. 8–9

• Increasing encoder and decoder depth from two to four layers substantially improved English-German translation, while depths above four yielded little improvement.Source: A Experiments on the layer number for… · p. 10
• The authors selected four layers for both encoders and decoders to trade off translation performance against computation complexity.Source: A Experiments on the layer number for… · p. 10
• Table 4 reports experiments on the number of encoder and decoder layers.Source: Table 4 · p. 10

• The global discriminator classifies sentences as source language, target language, or generated, using a CNN on fixed-length padded sequences.Source: B The architecture of the global… · p. 10
• The CNN applies kernels to word-embedding windows to produce feature maps, then uses max-over-time pooling to obtain each kernel’s feature.Source: B The architecture of the global… · p. 11
• Features from kernels with different window sizes are concatenated into a sentence representation and passed through fully connected and softmax layers.Source: B The architecture of the global… · p. 11
• The final softmax layer generates the probability of the sentence being true or generated.Source: B The architecture of the global… · p. 11

• The procedure generates source- and target-language sentences from monolingual data using the corresponding encoder-decoder pairs and greedy sampling.Source: C The training procedure of the global… · p. 11
• Each global discriminator is pre-trained on true monolingual data combined with generated data in its language.Source: C The training procedure of the global… · p. 11
• The generators and discriminators are then jointly trained, with policy-gradient methods used for the generators.Source: C The training procedure of the global… · p. 11

D The configurations for the open-source toolkits

• The word2vec training command sets size 512, window 10, negative 10, threads 50, minimum count 5, and 10 iterations.The command also specifies -cbow 0 and -hs 0.Source: D The configurations for the… · p. 11
• VecMap maps the embeddings to a shared-latent space with orthogonal mapping, numerals, and self-learning options.Source: D The configurations for the… · p. 11
• Source and target embeddings are normalized separately using unit and center options before mapping.Source: D The configurations for the… · p. 11