Neural Machine Translation

Index .

• The dotted locator ends at page 117.Source: Index · p. 4

13.1 A Short History

• Early neural translation models from the 1980s and 1990s lacked enough training data and computational resources for non-toy results, so the approach was abandoned for almost two decades.The cited examples are Forcada and Ñeco (1997) and Castaño et al. (1997).Source: §13.1 · p. 5
• During that pause, phrase-based statistical machine translation rose to dominance and supported applications from information gisting to translator productivity.Source: §13.1 · p. 5
• Neural methods re-entered through language models in statistical systems, then extended to scoring, translation tables, reordering, and pre-ordering.Schwenk (2007) showed large improvements in public evaluations; Devlin et al. (2014) showed large quality improvements over a competitive statistical system.Source: §13.1 · pp. 5–6
• Pure neural convolutional and sequence-to-sequence models handled short sentences but degraded as sentence length increased; attention yielded competitive results, followed by refinements including byte pair encoding and back-translation.Bahdanau et al. (2015) and Jean et al. (2015b) are cited for attention.Source: §13.1 · p. 6
• By 2017, almost all WMT submissions were neural, after one pure neural system competed but was outperformed in 2015 and neural systems won in almost all language pairs in 2016.Source: §13.1 · p. 6
• At the time of writing, research was exploring deeper and more linguistically informed models while gathering insight into NMT's strengths and weaknesses.Source: §13.1 · p. 6
• The toolkit landscape was multiplying rather than consolidating, making specific recommendations premature; examples included Nematus, Marian, and OpenNMT.Source: §13.1 · p. 6

13.2.1 Linear Models

• Linear models weight and sum translation features, but cannot model feature dependencies, nonlinear feature effects, or non-linearly separable cases such as XOR.Examples include language-model importance varying with sentence length and a threshold-like change in the impact of phrase-translation probabilities.Source: §13.2.1 · p. 7

13.2.2 Multiple Layers

• A hidden layer transforms input values into hidden-node values, then combines those values to produce outputs, using weight matrices between successive layers.The hidden mechanism connects observed inputs and outputs; networks can also have multiple output nodes.Source: §13.2.2 · p. 8

13.2.3 Non-Linearity

• Nonlinear activation functions make hidden layers more expressive; ReLU floors negative values at 0 and leaves positive values unchanged, while deeper stacks can support more intricate computations.Hidden nodes can act as feature detectors, reducing the need for manual feature engineering.Source: §13.2.3 · pp. 9–10

13.2.4 Inference

• Bias units fixed at 1 let a network produce nonzero weighted sums even when all input values are 0.The worked example uses sigmoid activations to compute node values.Source: §13.2.4 · p. 10
• For input (0,1), output y0 is 0.743; hidden nodes implement OR and AND, whose subtraction yields XOR.The OR-like node is high when either input is 1, while the AND-like node is high only when both are 1; nonlinearity lets the AND activation lower the output for (1,1).Source: §13.2.4 · pp. 10–11

13.2.5 Back-Propagation Training

• Back-propagation trains networks by comparing predictions with targets, computing node error terms, and updating weights against the error gradient over repeated training passes.Each pass through the training data is an epoch.Source: §13.2.5 · pp. 11–12

Weights to the output nodes

• For output-node weights, the update uses the node’s error term and incoming hidden-node values, with a learning rate scaling the gradient-based change.The output error is defined from computed and target values using the L2 norm; a node’s error term can be reused for each incoming weight.Source: Weights to the output nodes · pp. 13–14

Weights to the hidden nodes

• For hidden-node weights, back-propagation applies the chain rule: each hidden error depends on subsequent node errors weighted by the hidden node’s impact on those outputs.Training processes examples one at a time and updates weights using error terms associated with each non-input node.Source: Weights to the hidden nodes · pp. 14–15, Summary · p. 16

Summary

• Back-propagation computes hidden-node error terms from downstream errors, and those terms drive weight updates scaled by a learning rate across training examples and epochs.The example section illustrates the procedure with a single training example and summarizes its weight updates in Table 13.2.Source: Summary · p. 16, Table 13.2 · p. 17

13.2.6 Refinements

• Too-high learning rates can overshoot optima, too-low rates slow convergence, poor initialization can prolong progress, and local optima can trap gradient descent.These practical problems motivate refinements to gradient-descent training.Source: §13.2.6 · p. 18

Validation Set

• Training error may keep decreasing as the model overfits, so training stops when error on a separate validation set reaches its minimum.Figure 13.7 illustrates training error decreasing while validation error eventually rises.Source: Validation Set · p. 18, Figure 13.7 · p. 18

Weight Initialization

• Weights are randomly initialized from a uniform distribution, with preferred values that place activations in the function’s transition region rather than shallow-slope regions.For sigmoid, inputs in [−1; 1] yield activations in [0.269; 0.731].Source: Weight Initialization · p. 19

Momentum Term

• Momentum combines the previous momentum value with the current raw update, while Adagrad reduces a parameter’s learning rate according to accumulated squared gradients; Adam combines momentum with exponentially decayed squared gradients.Adam uses exponential decay to avoid the accumulated squared gradients growing so large that they permanently depress the learning rate.Source: Momentum Term · p. 19, Adapting Learning Rate per Parameter · pp. 19–20

Dropout

• Dropout randomly sets selected nodes to zero during training and suspends their parameter updates, encouraging other nodes to take over their roles and yielding a more robust model.The selected nodes may account for 10%, 20%, or more of the network, and the dropped set changes over training.Source: Dropout · p. 21

Layer Normalization

• Layer normalization addresses exploding or diminishing gradients by normalizing each layer’s weighted-sum values around their mean and then scaling them with additional bias vectors.The added vectors provide flexibility and may be shared across layers of the same type, including recurrent-network time steps.Source: Layer Normalization · pp. 21–22

Mini Batches

• Mini-batches accumulate updates before applying them to support parallel computation, while partitioned online training with averaged weights can improve results; GPUs accelerate the vector and matrix operations these computations require.Hogwild uses concurrent threads that update weights without waiting for other threads to finish using them, and the text reports that this does not hurt in practical experience.Source: Mini Batches · p. 22, Vector and Matrix Operations · p. 23

13.3.1 Neural Networks as Computation Graphs

• A computation graph represents a network’s parameters, inputs, and operations as nodes in an acyclic directed graph that can process inputs forward.Each computation node has an operation, links to inputs, and a value computed for the example.Source: §13.3.1 · pp. 24–25

13.3.2 Gradient Computations

• Training adds an error computation to the forward graph, then propagates gradients backward along its operations to derive parameter updates.For a parameter such as W2, the error is differentiated through the graph using the chain rule while other variables are treated as constants.Source: §13.3.2 · pp. 25–26
• When a computed value feeds multiple later operations, its backward-pass gradient is the sum of the gradients passed back from those descendants.Nodes carry forward values and backward gradients, alongside the functions and connections needed for those computations.Source: §13.3.2 · p. 29

13.3.3 Deep Learning Frameworks

• Theano computes parameter derivatives automatically, and the chapter’s example reports that a second training call improves predictions and lowers cost.The example uses a learning rate of 0.1 and describes repeating training until convergence, optionally with mini-batches.Source: §13.3.3 · p. 31

13.4 Neural Language Models

• Neural networks model conditional probabilities with multiple inputs and are robust to unseen data points, avoiding the insight and arbitrary choices required by traditional back-off and clustering.N-gram language models face sparse estimates for long contexts and use discounting and back-off to balance them against lower-order evidence.Source: §13.4 · pp. 31–32

13.4.1 Feed-Forward Neural Language Models

• A feed-forward N-gram language model maps one-hot context words through shared word embeddings and a hidden layer to predict the next word, pooling evidence across similar words.A limited vocabulary with an OTHER token or word classes can reduce the dimensionality of the large one-hot inputs.Source: §13.4.1 · pp. 32–33

13.4.2 Word Embedding

• Word embeddings represent words by their distributional contexts, so words occurring in similar contexts receive similar representations that can support semantic inference.Embedding similarity can be measured by a distance function such as cosine distance.Source: §13.4.2 · p. 36

13.4.3 Efficient Inference and Training

• Training neural language models is computationally expensive, and decoder scoring is costly; caching can reuse embeddings and context-slot computations, though exact softmax normalization still requires all output values.For a candidate word, the decoder can compute only its output node, but without normalization that score is not a proper probability.Source: §13.4.3 · p. 37
• Noise contrastive estimation trains a self-normalizing model to distinguish correct examples from noise, avoiding computation of all output nodes and making training faster.It compares examples from the training set with artificially generated noise examples.Source: §13.4.3 · p. 38

13.4.4 Recurrent Neural Language Models

• Recurrent neural language models reuse the previous prediction’s hidden state, allowing each next-word prediction to condition on the full sentence history rather than a fixed context window.The model has fewer weights than a 3-gram feed-forward neural language model.Source: §13.4.4 · pp. 39–40
• Back-propagation through time trains recurrent models by unrolling multiple predictions; mini-batch updates or full-sentence unrolling can be used, with full unrolling requiring dynamic computation graphs.Even when unrolled for a fixed number of steps, the network can learn dependencies over longer distances.Source: §13.4.4 · p. 40

13.4.5 Long Short-Term Memory Models

• Vanilla recurrent networks use the hidden layer both as memory and as the output representation, without a clear mechanism to choose between recent words and longer context.Back-propagating over long sequences can let recent information drown out older information; gradients can also grow too large, a problem typically suppressed by clipping.Source: §13.4.5 · p. 41
• LSTMs use an explicit real-valued cell memory, with input, forget, and output gates regulating new information, retained memory, and what is passed onward.The gates are conditioned on the input, previous hidden state, and previous memory state.Source: §13.4.5 · pp. 41–42

13.4.6 Gated Recurrent Units

• GRUs simplify LSTMs with one hidden state and two gates: a reset gate scales prior-state influence in the input combination, while an update gate balances that combination with the previous state.The update gate can bypass new input, allowing information and gradients to pass through across long distances.Source: §13.4.6 · pp. 43–44

13.4.7 Deep Models

• Experiments in vision and speech report increasingly better quality with a handful or even dozens of layers; recurrent models can be stacked or connected as deep transitional networks.In stacked models, each layer recurs over time; in transitional models, only the first layer receives the previous time step's final-layer state. Deep models stretch computational resources and typically train more slowly.Source: §13.4.7 · pp. 45–46, Figure 13.17 · p. 45
• Neural language models in statistical machine translation have shown benefits from 3–4 hidden layers (Luong et al., 2015a).Adding skip connections sometimes speeds training, but deep models still take several times longer to train than shallow networks.Source: §13.4.7 · p. 46
• Gülçehre et al. (2015) add a recurrent language model trained on extra monolingual data to neural machine translation, comparing reranking with gated integration alongside the translation model.The gate regulates the language model's relative contribution when predicting a word.Source: §13.4.7 · pp. 46–47

13.5.1 Encoder-Decoder Approach

• The basic encoder-decoder compresses the input into its final hidden state, but must retain sentence information and track translation coverage, so it works for short sentences (about 10–15 words) and fails on long ones.Proposed refinements include supplying the sentence embedding to every decoder state and reversing the output sentence order.Source: §13.5.1 · pp. 47–48

13.5.2 Adding an Alignment Model

• Attention supplies the explicit alignment that connects the encoder’s input-word representations to each decoder step, forming the central refinement of the encoder-decoder model.The model computes relevance scores from the previous decoder state and each input representation, normalizes them with softmax, and uses them to form a weighted context vector.Source: §13.5.1 · p. 48, §13.5.2 · p. 48, Attention Mechanism · p. 51

Encoder

• A bidirectional recurrent encoder processes embedded input words in both directions, combining left- and right-context hidden states into each word representation.The recurrent cell may use a feed-forward layer, GRU, or LSTM; the original approach used GRUs, while LSTMs had become more popular at the time of writing.Source: Encoder · pp. 48–49, Figure 13.19 · p. 49

Decoder

• At each step, the recurrent decoder uses its previous state, the preceding output-word embedding, and input context to predict a probability distribution over the output vocabulary.Training maximizes probability assigned to the correct next word, using its negative log probability as cost; sentence cost sums word costs, while inference can select the top word or explore alternatives.Source: Decoder · pp. 49–50

Attention Mechanism

• Attention scores each input representation against the previous decoder state, normalizes the scores across input words, and combines representations into the context for the next output.The scores indicate each input word’s relevance to producing the current output word.Source: Attention Mechanism · p. 51

13.5.3 Training

• Training dynamically unrolls computation graphs for variable-length sentence pairs, uses correct previous output words as context, and sums per-word errors; batching uses padding and masks to handle differing lengths.Grouping similarly sized examples into mini-batches reduces computation on padding; GPU batching exploits parallelism, and validation error is used to stop training when it stops improving.Source: §13.5.3 · pp. 52, 54

13.5.4 Beam Search

• Greedy decoding commits to the most likely word at each step, but can miss a better complete translation when an initially unlikely word choice is redeemed by later context.The passage illustrates ambiguity with believe (68.4%) versus think (28.6%), and different (41.5%) versus various (22.7%); a sentence may also begin with but (42.1%) or I (20.4%).Source: §13.5.4 · p. 55, Figure 13.25 · p. 56

Output Word Predictions

• Beam search keeps the top n partial translations, expands each using its next-word probabilities, and ends when all hypotheses produce an end-of-sentence token.Completed hypotheses leave the beam; the highest-scoring complete path yields the translation, recoverable through back-pointers.Source: Output Word Predictions · p. 57, Figure 13.26 · p. 57
• Complete paths are scored by multiplying word prediction probabilities, with better results from dividing by translation length after search.During search, all beam translations have equal length, so normalization would not affect their scores.Source: Output Word Predictions · p. 57
• Because recurrent models condition on the entire output sequence, hypotheses cannot be combined by shared future-feature context, making search a tree with as many complete paths as beam size.This makes the search graph less diverse than in statistical machine translation models.Source: Output Word Predictions · pp. 57–58
• Beam search retains the n best partial translations at each step and terminates after n complete sentences are found.The figure illustrates n = 6; end-of-sentence tokens and back-pointers identify complete paths.Source: Figure 13.27 · p. 58
• Cho et al. (2014) and Sutskever et al. (2014) established recurrent sequence-to-sequence approaches; Bahdanau et al. (2015) added attention, and Luong et al. (2015b) proposed global and local variants.Sutskever et al. reverse source order; Bahdanau et al. use bidirectional source states and condition on the preceding target-word state. Tu et al. (2016a) introduced a context gate to scale source and target context, while Tu et al. (2017) added reconstruction likelihood to the training objective.Source: Output Word Predictions · p. 58

13.6 Refinements

• The best-performing WMT 2017 system used ensemble decoding, byte pair encoding, synthetic data, and deeper models.Source: §13.6 · p. 59

13.6.1 Ensemble Decoding

• Ensembling combines multiple systems because they can make different errors, addressing alternative-system generation and prediction combination.The chapter presents agreement between systems as making joint correctness more likely.Source: §13.6.1 · p. 59

Generating Alternative Systems

• Checkpoint ensembling selects strong models from training stages, while multi-run ensembling uses separate runs and usually performs better at greater computational cost.Multi-run systems vary initialization and training-data order; checkpoint ensembles can also be combined across runs.Source: Generating Alternative Systems · pp. 59–60, Figure 13.28 · p. 59

Combine System Output

• Ensemble decoding averages models’ output-word probability distributions; weighting systems differently is possible but typically unused when their quality is similar.Source: Combine System Output · p. 60, Figure 13.29 · p. 60

Reranking with Right-to-Left Decoding

• Right-to-left systems rerank candidates from a left-to-right n-best list by averaging each candidate’s scores across both directions.The left-to-right ensemble generates candidates; right-to-left scoring uses forced decoding.Source: Reranking with Right-to-Left Decoding · p. 61

13.6.2 Large Vocabularies

• Rare words and large vocabularies challenge neural models, whose output computation grows with vocabulary size; proposed approaches include source-word copying and mini-batch vocabulary filtering.Models typically restrict vocabulary size, while copying methods address source words such as names and numbers.Source: §13.6.2 · pp. 61, 63
• Byte pair encoding repeatedly merges frequent character pairs to form subword units, leaving common words intact and splitting many rarer words.The illustrated English model uses 49,500 operations; splits are marked with @@.Source: §13.6.2 · pp. 62–63, Figure 13.30 · p. 62
• Further rare-word approaches use separate dictionaries or interpolate a bilingual dictionary with neural predictions linked to source words through attention.Source: §13.6.2 · pp. 62–63

13.6.3 Using Monolingual Data

• Monolingual data can be used by synthesizing the missing side of parallel data or integrating a language model into the neural architecture.Source: §13.6.3 · p. 63

Back Translation

• Back-translation uses a reverse system to translate target-side monolingual text, then combines the synthetic pairs with true parallel data for final training.The chapter describes balancing synthetic and true data, including equal amounts or oversampling true data.Source: Back Translation · pp. 63–64, Figure 13.31 · p. 64

Adding a Language Model

• A separately trained recurrent language model is joined to the translation model at output prediction by concatenating their conditioning contexts; its parameters remain fixed during combined training.Only the translation model and combination layer are updated, to avoid overfitting the language model to parallel data.Source: Adding a Language Model · pp. 64–65

13.6.3 Using Monolingual Data

• Round-trip training adds an objective to translate a target-language sentence into the source language and back while preserving the original target sentence’s meaning.Source: Figure 13.32 · p. 65

Adding a Language Model

• A gate predicted from the language-model state can scale that state before output prediction, allowing its contribution to be adjusted.Source: Adding a Language Model · p. 65

Round Trip Training

• Round-trip training asks a translation to be valid in its new language and reconstructable back into the original language.The two objectives can update both translation models using monolingual sentences.Source: Round Trip Training · pp. 65–66
• For monolingual f sentences, an n-best list of e translations supplies training pairs and costs for updates in both translation directions.Updates to the f→e model are scaled by language-model and forward-translation costs; monolingual e data uses the reverse round trip.Source: Round Trip Training · p. 66

13.6.4 Deep Models

• A deep decoder replaces one hidden state per time step with several layers, combining stacked recurrent layers and deep transitions.The illustrated architecture omits word prediction, selection, and output-word embedding steps, which remain as in the original model.Source: §13.6.4 · p. 67, Figure 13.33 · p. 67
• The alternating recurrent encoder stacks layers while conditioning them alternately on the previous or next time step.This extends bidirectional context to a deep encoder; even-numbered states use left context and odd-numbered states use right context.Source: §13.6.4 · p. 68, Figure 13.34 · p. 68
• Residual connections help training most in its early stages, when they speed the initial reduction of perplexity, and less in the converged model.Direct input-to-output connections may link embeddings to the final encoder layer or pass input to each layer's output.Source: §13.6.4 · pp. 68–69
• Four encoder and decoder stacks with two deep transitions each, plus alternating encoder networks, have shown good results, while many variations remain to be explored across data conditions.The cited configuration is reported by Miceli Barone et al. (2017).Source: §13.6.4 · p. 69

13.6.5 Guided Alignment Training

• Attention weights typically match traditional statistical word alignments, though an output word may attend to an entire input verb group.The example shows German “sind” attending to “have been strained.”Source: §13.6.5 · p. 69
• Given word alignments can support terminology handling and let users inspect where output words originate, motivating explicit alignment supervision.The passages also identify possible benefits for faster convergence or low-resource data conditions.Source: §13.6.5 · pp. 69–70
• Training can retain the translation model and add an objective that matches its attention scores to supplied word alignments.The supplied alignment and attention scores each sum to 1 across input words for each output word; their mismatch can be measured by cross entropy.Source: §13.6.5 · p. 70

13.6.6 Modeling Coverage

• Neural translation sometimes translates input words multiple times or misses them, and misallocated attention can produce hallucinated words or leave phrases untranslated.The example over-attends to “Social Housing” and fails to attend to “a fresh start.”Source: §13.6.6 · pp. 70–71, Figure 13.36 · p. 71
• Coverage can be defined as accumulated attention, with roughly similar attention expected across input words in a complete sentence translation.Too little or too much attention to an input word signals a possible translation problem.Source: §13.6.6 · p. 71
• Coverage can be enforced during beam search by discouraging hypotheses that over-attend to input words or penalizing completed hypotheses that under-attend.Choosing weights for multiple scoring functions is a challenge; grid search is suggested for two or three weights, with MERT or MIRA for more.Source: §13.6.6 · pp. 71–72
• A coverage model conditions attention on each input word's accumulated attention, and a weighted coverage penalty can also be added to the training objective.The added objective functions may distract from the primary goal of producing good translations.Source: §13.6.6 · p. 72
• Fertility models predict how many output words each input word generates and can use that estimate to normalize coverage.Fertility varies: English “do” may have no equivalent in another language, while German “natürlich” may translate as “of course.” The example predicts fertility from the input representation with a sigmoid scaled to a predefined maximum.Source: §13.6.6 · pp. 72–73
• Coverage modeling illustrates a choice between engineering a targeted fix for observed errors and expecting machine learning to acquire coverage from data.The text says it remains to be seen which direction neural machine translation will take; generic adjustments are difficult for a complex task.Source: §13.6.6 · p. 73

13.6.7 Adaptation

• A common neural adaptation method trains on all available data to convergence, then continues on in-domain data until in-domain validation performance peaks.The final model thus draws on all training data and is specialized to the in-domain data.Source: §13.6.7 · p. 74
• In-domain fine-tuning may converge quickly, often requiring only a handful of epochs because in-domain data is typically small.Small in-domain sets can also risk overfitting when used alone or in a secondary adaptation stage.Source: §13.6.7 · p. 74
• A less commonly used approach combines separate in-domain and out-of-domain models, with their weights chosen by line search in the two-model case.Choosing model weights is not trivial in general.Source: §13.6.7 · p. 74
• In-domain and out-of-domain detectors can select relevant sentence pairs from large collections, which can be used for domain-specific training or a secondary adaptation stage.The selection approach scores out-of-domain pairs with both detectors and favors those judged relatively relevant by the in-domain detector.Source: §13.6.7 · p. 75
• Without parallel data in the target domain, monolingual data can guide selection of general parallel data or support adaptation through back-translated synthetic data.The latter approach trains an out-of-domain model, back-translates data to form a synthetic in-domain corpus, then adapts the initial model.Source: §13.6.7 · p. 75
• For multiple domains, systems can select a domain-specific model or combine models using domain-relevance probabilities as ensemble weights.When a test sentence's domain is unknown, a classifier can choose a model; domain classification may use a whole document rather than a sentence.Source: §13.6.7 · pp. 75–76
• Adaptation advice is not conclusive because domains vary in style, similarity, data amounts, separation, and translation quality.Training collections may also contain noise such as mistranslations, misalignments, or machine-generated translations.Source: §13.6.7 · p. 76
• Farajian et al. (2017) report that adaptation lets neural machine translation catch up after traditional systems outperform it on niche domains.The comparison concerns general-purpose systems trained on a collection of data and tested on niche domains.Source: §13.6.7 · p. 76

• Input words can carry part-of-speech, lemma, morphology, syntax, and semantic annotations, encoded as factor embeddings summed into the word embedding.The richer representations can be supplied without changing the neural translation architecture.Source: §13.6.8 · pp. 77–78
• Linguistic annotations may provide relevant knowledge when training data is insufficient for a model to induce it automatically, though their value must be established empirically.The text leaves open whether these features can instead be learned in word embeddings or contextualized encoder states.Source: §13.6.8 · pp. 78–79
• Phrase-structure trees can be linearized as sequences of words and structural tokens marking phrase openings and closings, so a translator can generate both.The resulting sequence mixes output words with special tokens such as “(NP” and “)”.Source: §13.6.8 · p. 79, Figure 13.39 · p. 79
• Linearized syntax is intended to encourage well-formed output, but recursive structure is not naturally captured by simple word-level BEGIN, CONTINUATION, and OTHER tags.Phrase-structure parsing is generally recursive and often built bottom-up.Source: §13.6.8 · pp. 78–80
• The best-performing syntactic parsers use recursive model structures, while unified syntax-parsing and translation frameworks have no agreed best practices.The text identifies integration of parsing and translation as a challenge for future work.Source: §13.6.8 · p. 80

• Training data is unevenly distributed across language pairs, with abundant data for some directions and very little for others, including Chinese–German and Japanese–Spanish.Available corpora may overlap substantially or be unique to a language pair.Source: §13.6.9 · p. 80
• Concatenating corpora can train one model on multiple input languages, while a leading input token such as [SPANISH] specifies the desired output language.The combined model sees both English sides of German–English and French–English data, which may improve its language model.Source: §13.6.9 · p. 81
• A model trained on German–English, French–English, and French–Spanish can translate German to Spanish without ever seeing a German–Spanish training pair.Good quality still requires some parallel data in the desired language pair, though less than a standalone model (Johnson et al., 2016).Source: §13.6.9 · p. 81
• Separate language-pair models can share an encoder for the same input language, a decoder for the same output language, or attention across pairs.Shared components use the same parameters, so updates from training one pair also change the corresponding component in other models.Source: §13.6.9 · p. 82
• Monolingual data can train an encoder or decoder separately, but a decoder trained without context may learn to ignore the input and act only as a target-side language model.The decoder's context states must be blanked out for this training setup.Source: §13.6.9 · p. 82
• Johnson et al. (2016) report small benefits for several input languages sharing an output language, mixed results for multiple output languages, and weaker zero-pair translation than traditional pivot methods.Their results indicate that a model learns some interlingual meaning representation.Source: §13.6.9 · p. 82

13.7.1 Convolutional Neural Networks

• Convolutional encoders merge neighboring word representations bottom-up, and the decoder reverses this process after determining the output length.In the illustrated design, kernel sizes depend on sentence length, so longer sentences require larger kernels.Source: §13.7.1 · pp. 83–84
• The original convolutional translation model compresses the source sentence into one vector and was not shown to match traditional systems, though it was used to rerank their candidate translations.Single-vector compression is especially problematic for long sentences.Source: §13.7.1 · p. 84

13.7.2 Convolutional Neural Networks With Attention

• Convolutional attention models replace recurrent layers with convolutions that preserve the sequence length and allow all words at a depth to be processed in parallel.Their word representations use a limited context window, which may exclude relevant disambiguating words.Source: §13.7.2 · p. 85
• In the convolutional decoder, stacked layers condition each state on recent output words and input context computed through attention.Unlike a recurrent decoder, its computation does not depend on the preceding decoder state.Source: §13.7.2 · pp. 86–87, Figure 13.44 · p. 86

13.7.3 Self-Attention

• Self-attention builds each input-word representation from associations with other input words, using normalized dot-product scores to weight their representations.This connects words across the sentence rather than restricting them to a convolution’s local window.Source: §13.7.3 · p. 87
• The self-attention architecture replaces recurrent layers in both encoder and decoder, with decoder self-attention restricted to previously produced output words and separate attention to encoder states.The model stacks sublayers with residual connections and layer normalization.Source: §13.7.3 · pp. 88–90

13.8 Current Challenges

• The chapter frames robustness as a challenge when conditions differ from training, including limited training data, unusual out-of-domain inputs, and unlikely initial choices in beam search.It also identifies lower interpretability as an unexamined challenge: decisions are buried in large matrices of real-valued parameters.Source: §13.8 · p. 90

13.8.1 Domain Mismatch

• NMT’s out-of-domain performance is worse in almost all cases, while in-domain NMT and SMT are similar; on the Law test set, Medical-trained NMT scores 3.9 BLEU versus 10.2 for SMT.NMT output can also be fluent yet unrelated to the input, a concern for information gisting because users may be misled by hallucinated content.Source: §13.8.1 · pp. 91, 93, Figure 13.46 · p. 92

13.8.2 Amount of Training Data

• With the full English–Spanish data set, NMT scores 31.1 BLEU, versus 28.4 for SMT and 30.4 for SMT with a big language model; with 24.1 million words, it scores 25.7 versus SMT’s 24.7.NMT has a steeper learning curve but cannot get off the ground with training corpora of a few million words or less; with one 1/1024 subset, it scores 1.6 BLEU versus SMT’s 16.4.Source: Figure 13.48 · p. 94, §13.8.2 · p. 95

13.8.3 Noisy Data

• With 50% of training data perturbed, NMT drops from 35.4 to 30.1 BLEU, while SMT drops from 32.7 to 32.0; the chapter suggests noisy inputs may shift NMT toward language-model behavior and fluent but inadequate output.The noise experiment perturbs data by shuffling target sentences to create misaligned pairs.Source: §13.8.3 · p. 96, Table 13.4 · p. 96

13.8.4 Word Alignment

• Attention alignments often match fast-align reasonably well, but German–English shows a marked divergence that the authors also observed under other data conditions.The chapter cautions that attention may have a broader role than word alignment, since contextual word representations can inform attention to a verb’s subject and object.Source: §13.8.4 · pp. 96–98

13.8.5 Beam Search

• Translation quality usually worsens beyond an optimal beam size, which ranges from 4 to around 30 across the cited language pairs; length normalization alleviates the problem and improves optimal quality in 5 of 8 pairs.Quality can still fall with larger beams, with shorter translations identified as the main cause.Source: Figure 13.51 · p. 99, §13.8.5 · p. 100

13.8.6 Further Readings

• Prior comparisons examined linguistic categories for English–German and broad aspects such as fluency and reordering across nine language directions.The chapter cites Bentivogli et al. (2016) and Toral and Sánchez-Cartagena (2017).Source: §13.8.6 · p. 100

13.9 Additional Topics

• Earlier neural machine translation work often built neural components for traditional statistical machine translation systems.Source: §13.9 · p. 100

Translation Models

• Translation models add source context through aligned words, sentence embeddings, or convolutional encoders incorporating output context.The approaches described include feed-forward, convolutional, and gated-layer models.Source: §13.9 · p. 100

Reordering Models

• A neural reordering model conditions orientation classification on the current and previous phrase pair, encoded with a recursive neural network auto-encoder.The approach addresses sparse-data problems faced by lexicalized reordering models conditioned on rich context.Source: §13.9 · p. 100

Pre-Ordering

• Pre-ordering learns to reorder an input dependency tree into output word order, using either child swaps or decisions made during a tree traversal.The described models use feed-forward or recurrent neural networks, with the latter conditioning on past decisions.Source: §13.9 · p. 101

N-Gram Translation Models

• N-gram translation models treat minimal translation units as symbols or represent them with bags of words, individual words, word pairs, or phrase embeddings.These approaches model minimal translation units in relation to preceding units.Source: §13.9 · p. 101

Author Index

• The index lists authors from Callison-Burch through the D entries, including Castaño, de Gispert, and Devlin.Source: Author Index · pp. 113–114
• The G-through-N entries include Gehring, Hinton, Koehn, and Ñeco, among other authors.Source: Author Index · pp. 114–115
• The later entries include authors from Rebollo through Zoldan, with page references alongside each name.Source: Author Index · pp. 116–117