Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization

1 Introduction

• Neural networks represent language structures with continuous vectors, while prior knowledge such as dictionaries and rules uses discrete symbols.The paper notes that these symbolic representations are difficult to transform into the continuous representations NMT requires.Source: §1 · p. 1
• Combining multiple overlapping, arbitrary knowledge sources remains difficult because architecture changes must model source interdependence explicitly.Neural networks impose strong independence assumptions between hidden states.Source: §1 · p. 1
• Adding knowledge as training-objective terms can partly address the challenge, but existing terms are restricted to a limited number of simple constraints.The introduction also describes prior approaches that modify model architectures or training objectives.Source: §1 · pp. 1–2
• The framework integrates multiple overlapping, arbitrary prior knowledge sources into neural machine translation using posterior regularization.It represents prior sources as real-valued features in a log-linear desired distribution.Source: §1 · p. 2
• NIST Chinese-English experiments show significant improvements over posterior regularization using constrained posterior sets.The proposed framework is described as incorporating a variety of features.Source: §1 · p. 2

2.1 Neural Machine Translation

• NMT models translation with word-level probabilities, while prior approaches add knowledge through external phrase-table memory or coverage vectors.Coverage constraints encode the intuition that each source phrase should translate into exactly one target phrase.Source: §2.1 · p. 2
• Combining arbitrary, overlapping knowledge sources remains challenging: architecture changes are limited, and objective add-ons cover only simple constraints.This motivates a framework for integrating arbitrary prior knowledge sources into NMT.Source: §2.1 · p. 2

2.2 Posterior Regularization

• Posterior regularization uses a desired distribution incorporating prior knowledge and a KL divergence from the model posteriors, with valid distributions specified through expected constraint features and bounds.The method is optimized with a sentence-level minorization-maximization algorithm akin to EM.Source: §2.2 · pp. 2–3
• Applying posterior regularization to NMT makes feature-expectation bounds difficult to set for real-valued features, and its extra posterior-inference step increases training time.Coverage-penalty values vary across sentences, complicating the choice of a bound for all training examples.Source: §2.2 · p. 3

3.1 Modeling

• The method adapts posterior regularization by using a log-linear desired distribution instead of the constrained posterior set in the original formulation.The features can incorporate signals shown effective in SMT, and the resulting training objective is differentiable.Source: §3.1 · p. 3
• Log-linear features provide a general framework for combining arbitrary prior knowledge sources in NMT, with transparency to neural model architectures.The approach associates discrete prior representations in Q with continuous representations learned from data using KL divergence.Source: §3.1 · p. 3
• Jointly training θ and λ avoids manually specifying feature weights γ and is reported to yield significant improvements in practice.The desired distribution Q guides learning of P; the text points to Table 1 for the improvement.Source: §3.1 · p. 3
• Standard stochastic gradient descent algorithms can optimize the model parameters.The passage refers to the training objective in Eq. (7).Source: §3.1 · p. 4

3.2 Feature Design

• SMT features over latent structures are inaccessible to NMT decoding, but linguistically meaningful internal information such as attention can still be used.Source: §3.2 · p. 4

3.2.1 Bilingual Dictionary

• A bilingual dictionary feature is defined for each dictionary entry, while entries containing out-of-vocabulary words must be discarded.Source: §3.2.1 · p. 4

3.2.2 Phrase Table

• Phrase-table features encode phrase-level correspondences, with one feature for each phrase-table entry and the number of features depending on the neural model vocabulary.Source: §3.2.2 · p. 4

3.2.3 Coverage Penalty

• The coverage penalty feature penalizes source words with low summed attention, and its value varies substantially across sentence lengths.Source: §3.2.3 · p. 4

3.2.4 Length Ratio

• With β = 1.2, a 10-word Chinese source and 12-word English translation score 1, while target lengths of 100 and 6 score 0.12 and 0.5.The length ratio feature encourages translations to fall within a reasonable range and penalizes translations that are too long or too short.Source: §3.2.4 · p. 4

• The training objective maximizes posterior-regularized likelihood and is differentiable, allowing standard stochastic gradient descent instead of the original minorization-maximization algorithm.Source: §3.3 · p. 5
• KL-divergence gradients require summing over all candidate translations, whose set contains every possible translation for a source sentence.Source: §3.3 · p. 5
• The method approximates the full candidate space with a sampled subspace, approximates Q on that subspace, and controls the sharpness of the normalized neural distribution.Source: §3.3 · p. 5

• The search process factorizes at the word level over the target-language vocabulary.Source: §3.4 · p. 5
• The neural model generates a k-best candidate list, and the decision rule selects the most probable candidate using the learned parameters.Source: §3.4 · p. 5
• This coarse-to-fine approach keeps the architecture transparent because decoding does not directly incorporate prior knowledge, whose direct integration depends on model structure and feature locality.Source: §3.4 · p. 5

4.1 Setup

• The experiments use 1.25M Chinese–English sentence pairs and test on NIST 2003, 2004, 2005, 2006, and 2008, comparing against RNNSEARCH, CPR, and POSTREG.NIST 2002 is used for validation; BLEU is the evaluation metric.Source: §4.1 · pp. 5–6

4.2 Main Results

• POSTREG improves over RNNSEARCH, but cannot use the coverage-penalty feature because an appropriate feature bound is difficult to specify; combining features yields modest further gains.The authors report that a loose bound can make training unstable and prevent convergence.Source: Table 1 · p. 6, §4.2 · p. 7
• The proposed approach significantly outperforms RNNSEARCH and POSTREG; with a single coverage-penalty feature, it scores 30.76 versus CPR’s 29.72.The authors suggest incorporating prior knowledge during modeling might be more beneficial than in decoding.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Effect of Reranking

• Reranking the neural model’s k-best list with prior-knowledge features usually improves translation quality.Table 2 reports the effect of reranking on translation quality.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Training Speed

• The approach converges in about 100K iterations when initialized from a RNNSEARCH model trained for 300K iterations, but each iteration is 1.5 times slower.On one Tesla M40 GPU, RNNSEARCH training takes four days and the proposed model requires three additional days.Source: §4.4 · pp. 7–8

4.5 Example Translations

• Examples associate the bilingual dictionary with translating listed words, the phrase table with short idioms and context-sensitive edits, coverage penalty with fewer omissions, and length ratio with reasonable sentence length.The reported examples contrast these feature effects with errors in RNNSEARCH outputs.Source: §4.5 · p. 8

Posterior regularization

• The framework represents the desired distribution with a log-linear model rather than a constrained posterior set, enabling arbitrary real-valued knowledge features and differentiable joint training.The authors describe this as a difference from posterior regularization in Ganchev et al. (2010).Source: §5 · p. 9

Prior knowledge in NMT

• Compared with prior work on injecting knowledge into NMT (Arthur et al., 2016; Tu et al., 2016; Cohn et al., 2016; Tang et al., 2016; Wang et al., 2016), the framework targets arbitrary sources while leaving the neural translation model unchanged.Source: §5 · p. 9

Related approaches

• Unlike approaches that treat neural translation models as log-linear features, this work connects the models via KL divergence, preserving transparency to model architectures and enabling application to other neural NLP models.He et al. (2016) also combine neural representation learning and log-linear prior knowledge, but use the neural model as a feature.Source: §5 · p. 9

• Experiments show significant improvements over both standard NMT and posterior regularization using constrained posterior sets.The framework guides end-to-end NMT toward desired behavior using a log-linear model that encodes prior knowledge.Source: §6 · p. 9