Code-Switching for Enhancing NMT with Pre-Specified Translation

1 Introduction

• Replacing source phrases with target translations creates code-switched training pairs, teaching the model to copy embedded translations without decoding constraints or post-processing.At decoding, the source is modified in the same way before translation.Source: §1 · pp. 1–2
• Unlike placeholders, the modified source retains target-word meaning; unlike lexical constraints, the translations are available during both training and decoding.The approach is data augmentation and can be used with any NMT architecture.Source: §1 · p. 2
• The paper also proposes shared target-side embeddings and a pointer network to strengthen copying of source-side target words.The pointer network is further used to copy target words embedded in the source.Source: §1 · p. 2
• Across En–Ru and Ch–En test sets in different domains, the method outperforms placeholder and lexical-constraint approaches on a Transformer.The authors also report that shared embeddings and a pointer network improve copying.Source: §1 · p. 2
• The authors release four expert-translated En–Ru e-commerce test sets, totaling 7,169 sentences with an average length of 211.These sets accompany the evaluation across different domains.Source: §1 · p. 2

2 Related Work

• Placeholder-tag methods replace unknown words with tags, then restore outputs through dictionary lookup or source-word copying; for general pre-specified translations, they are limited because tags lose word meaning (Crego et al., 2016; Li et al., 2016).In contrast, the paper’s method fully keeps word meaning in the modified source.Source: §2 · p. 2
• Lexical-constraint decoding inserts specified translations, but may neglect source correspondence and slow decoding; this method leaves decoding unchanged and does not guarantee every constraint [Hokamp and Liu, 2017; Post and Vilar, 2018; Hasler et al., 2018].The paper describes its approach as soft insertion with better overall translation quality than those methods.Source: §2 · p. 2
• Unlike probabilistic lexicons that suggest one-to-many translations, this work uses one-to-one specified translations to constrain output without integrating a lexicon into the NMT model [Arthur et al., 2016; Tang et al., 2016; Wang et al., 2017a].The paper characterizes its data augmentation as model-free.Source: §2 · pp. 2–3
• Currey et al. (2017) augment training by copying target-side sentences to the source side, producing identical source-target sentences; this improves translation, especially for proper nouns and words identical across languages.Alkhouli et al. (2018) use one-to-one word dictionaries for decoding, with entries limited to one word on each side and translations applied only after successful source-word alignment.Source: §2 · p. 3

3 Data augmentation

• Training augmentation replaces source words or phrases with their target translations, while decoding applies the same source-side replacement so the trained model can copy them into output.The lexicon consists of one-to-one source–target phrase pairs; decoding is not restricted to replacing one or two phrases.Source: §3 · p. 3, §3.2 · p. 3

3.1 Training

• Using a phrase table, the method indexes bilingual training sentences matching one phrase pair or a pair combination, replaces the source phrase(s) with target translations, and adds sampled pairs to original training data.Source: §3.1 · p. 3

3.2 Decoding

• At decoding, pre-specified target phrases replace corresponding source phrases before translation; the translations may come from customers or user feedback.The phrase pairs are one-to-one, and decoding may replace more than one or two phrases in a sentence.Source: §3.2 · p. 3

4 Model

• The Transformer encoder and decoder use stacked self-attention layers, with encoder states attending over source positions and decoder states attending over previous target states and the source.The model uses multi-head attention and positional encoding.Source: §4 · p. 4

4.1 Shared Target Embeddings

• Shared target embeddings make source-side target words use the same embeddings as target-vocabulary words, facilitating copying into the output.This aligns the source-side and target-side expressions at the embedding level.Source: §4.1 · p. 4

4.2 Pointer Network

• The pointer network uses target-to-source attention as a copy distribution and combines it with vocabulary prediction to form an output distribution.A gate computed from the context vector and decoder state controls the contributions of copying and prediction.Source: §4.2 · pp. 4–5

5.1 Data

• 13.88M En-Ru and 7.42M Ch-En sentences form the baseline training corpora, with test sets spanning news, e-commerce, and spoken-language data.The En-Ru corpus combines bilingual and back-translation data; the Ch-En corpus does likewise.Source: §5.1 · p. 5

5.2 Experimental Settings

• Pre-specified translations are sampled from references for all systems, and an SMT phrase table with count-based pruning supports augmented-data generation.The phrase-table pruning threshold is 10.Source: §5.2 · p. 5

5.4 Results

• 3.48 BLEU over placeholder (En-Ru news) and 1.38 BLEU over placeholder (Ch-En news); our method beats both baselines across tested domains.The En-Ru and Ch-En results cover news, e-commerce, and spoken-language domains.Source: Table 1 · p. 6, Table 2 · p. 6, §5.4 · pp. 6–7

5.3 System Configurations

• Training combines original data with about 10M augmented Ch-En and 6M En-Ru sentences containing one or two source-side replacements.The systems include placeholder and lexical-constraint baselines alongside the augmented-data method.Source: §5.3 · p. 6

5.4 Results

• 1.34 BLEU over placeholder and 2.63 over lexical constraints are En-Ru e-commerce gains; Ch-En gains are 1.38 and 0.74 BLEU in news, and 1.35 and 0.42 in spoken language.Placeholder tags improve only En-Ru e-commerce results, where the pre-specified translations are mostly entities; vocabulary words lose meaning when represented by tags.Source: §5.4 · p. 7

5.5 Analysis

• On Ch-En WMT test sets with one reference, BLEU rises from 18.49, 20.01, and 19.05 to 20.56, 22.3, and 21.08, respectively.The paper reports larger improvements on WMT than NIST because translations are sampled from a single reference, encouraging output to match it.Source: §5.5 · p. 8
• Copy success reaches 90.54% in the presented experiments and 98% with unpruned phrase-table matching, even without shared embeddings or a pointer network.The authors suggest insufficient training examples for some target phrases may explain failures to copy.Source: §5.5 · pp. 8–9
• Shared embeddings and a pointer network improve copying, while the augmented-data model retains comparable strength on un-replaced inputs and is sometimes slightly better.The paper attributes the latter to robustness to perturbed inputs and copying source-side words such as nontransliterated named entities.Source: §5.5 · p. 9

• The approach augments training with code-switched source sentences and their translations.Source: §6 · p. 9
• The method achieves consistent improvements over previous lexical-constraint methods on large NMT test sets.Source: §6 · p. 9
• The model learns to translate source-side target phrases by copying them into the output.Source: §6 · p. 9
• The authors describe this as the first use of code switching for NMT with pre-specified translations.Source: §6 · p. 9

7 Future Work

• The authors plan to study how copy success rate and BLEU scores interact across sampling strategies and as the amount of augmented training data grows.Source: §7 · p. 9
• They also plan to validate the approach on language pairs with identical letters in their alphabets, including English–French and English–Italian.Source: §7 · p. 9