Source-linked AI summary
Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, Graham Neubig
TL;DR
BLEU-based NMT training can penalize semantically correct lexical variation and offers limited optimization guidance. The paper introduces SIMILE, a continuous semantic-similarity reward, and finds that it improves automatic and human evaluations while making optimization easier. The authors conclude that semantic similarity is a promising alternative training reward, while noting that the study evaluates only one metric and has observed failure cases.
Problem
BLEU-based training lacks partial credit, can penalize semantically similar lexical alternatives, and often provides flat optimization signals.
Method
The paper introduces SIMILE, an embedding-based continuous semantic-similarity reward for minimum-risk NMT training.
Results
SIMILE improves automatic evaluations, correlates better with human judgments, eases optimization, and produces translations with richer semantically important words.
Takeaways & Limitations
Continuous semantic similarity is presented as a promising alternative to sentence-level BLEU for NMT optimization.
Takeaways & Limitations
The study experiments with a single semantic-similarity metric, and qualitative analysis reports failure cases where SIMILE repeats phrases or misses much of a translation.
Abstract
from arXiv · showhide
While most neural machine translation (NMT) systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can substantially improve final translation accuracy. However, training with BLEU has some limitations: it doesn't assign partial credit, it has a limited range of output values, and it can penalize semantically correct hypotheses if they differ lexically from the reference. In this paper, we introduce an alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity. We evaluate on four disparate languages translated to English, and find that training with our proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and also that the optimization procedure converges faster. Analysis suggests that this is because the proposed metric is more conducive to optimization, assigning partial credit and providing more diversity in scores than BLEU.
1 Introduction
BLEU-based discriminative training is difficult because lexical matching penalizes semantically similar translations and provides sparse, flat optimization signals. The paper proposes SIMILE, a continuous semantic-similarity reward, and reports improvements across automatic and human evaluations.
- Motivation: BLEU penalizes semantically similar translations when their n-grams differ from the reference.Because unmatched n-grams receive no credit, lexical variation can be treated as error even when meaning is preserved.
- Motivation: BLEU is difficult to optimize because it provides no partial credit and often assigns identical scores to diverse candidate translations.These sparse and flat rewards give learning poor guidance through intermediate hypotheses.
- SIMILE: SIMILE uses an embedding model trained on paraphrase data to assign continuous semantic-similarity rewards to generated translations.Its continuous scores distinguish candidates that matching-based metrics such as BLEU score similarly.
- SIMILE: SIMILE provides partial credit and reduces penalties for semantically correct but lexically different translations.The metric is intended to provide more informative optimization signals than discrete matching-based rewards.
- Evaluation: Experiments translating Czech, German, Russian, and Turkish into English found consistent BLEU improvements and better human-judged quality for all but one language.The experiments used the 2018 WMT test sets and also evaluated the proposed semantic-similarity metric.
2 SIMILE Reward Function
SIMILE combines a semantic similarity model with a length penalty to provide a continuous reward for discriminative NMT training. Its design targets robust sentence-level similarity while addressing length mismatch and offering practical advantages over BLEU-based training.
- Metric design: SIMILE combines semantic similarity with a length penalty to evaluate generated translations against references.The similarity component is computed from sentence representations, while the length penalty addresses differences between hypothesis and reference lengths.
- Robustness: SIM is designed to be domain agnostic, and its performance remains improved over a BLEU baseline despite training primarily on subtitles and evaluating on news data.The authors present this domain-switch result as evidence of robustness.
- Similarity model: The similarity model encodes sentences by averaging 300-dimensional subword embeddings and calculating cosine similarity.The encoder is trained on paraphrase pairs, including 16.77 million filtered pairs from ParaNMT.
- Length penalty: The length penalty penalizes generated sentences whenever their length differs from the reference, after experiments with alternative penalties favored short sentences.The final formulation reduces the influence of this penalty by fixing α to 0.25.
- Scope and evaluation: The authors report practical advantages for training toward metrics other than BLEU, while limiting empirical comparison to one semantic similarity metric because of resource constraints.They state that semantic similarity is not an exact replacement for machine-translation quality measurement, which also captures fluency and other factors.
3 Machine Translation Preliminaries
The models are first trained with maximum likelihood and label smoothing, then fine-tuned with a weighted combination of token-level loss and minimum risk training. Minimum risk uses candidate hypotheses, references, and either BLEU- or SIMILE-based costs.
- Objective Functions: Models are initially trained with maximum likelihood using label smoothing before fine-tuning.The label-smoothing confidence penalty is set to 0.1.
- Objective Functions: Fine-tuning combines minimum risk training with the label-smoothed token loss through a weighted objective.The weighting coefficient γ is tuned from {0.2, 0.3, 0.4}.
- Objective Functions: Minimum risk training evaluates candidate hypotheses against a reference using costs based on either 1−BLEU or 1−SIMILE.Candidates are generated from 8-hypothesis n-best lists, without including the reference.
- Optimization: The optimization uses Nesterov’s accelerated gradient method with learning rate 0.25, momentum 0.99, and gradient renormalization to norm 0.1.The token-loss objective is trained for 200 epochs and the combined objective for 10 before learning-rate annealing.
4 Experiments
Experiments compare MLE, BLEU, SIMILE, and a half-BLEU/half-SIMILE objective on four translation directions into English. SIMILE performs best on automatic metrics across all languages and receives the highest human scores except for Turkish, where repetition is a problem.
- 4.1 Data: Experiments translate Czech, German, Russian, and Turkish into English using WMT data and official WMT 2018 test sets.Training and validation data differ by language pair, with SETIMES2 used for Turkish training.
- 4.2 Automatic Evaluation: The comparison includes MLE, BLEU-cost, SIMILE-cost, and a half-BLEU/half-SIMILE minimum-risk objective.The mixed objective uses cost 1 − 1/2(BLEU + SIMILE).
- 4.2 Automatic Evaluation: SIMILE performs best on BLEU and SIM evaluation metrics for all four languages.Using SIMILE as the cost produces larger BLEU improvements than using BLEU alone, while BLEU training also improves SIM.
- 4.3 Human Evaluation: SIMILE receives the highest human-evaluation score across all language pairs except Turkish.Turkish shows significant repetition in SIMILE outputs, and the authors hypothesize that SIMILE needs a minimum starting translation quality.
5 Quantitative Analysis
Quantitative analyses show that SIMILE provides finer-grained candidate distinctions, accelerates optimization, and improves performance across relevant evaluation dimensions, with stronger gains for low-frequency words and informative parts of speech.
- 5.1 Partial Credit: SIMILE distinguished candidate scores more finely than BLEU, with nonnegative score differences in 99.0% versus 85.1% of n-best comparisons.The average difference was 4.8 for SIMILE and 4.3 for BLEU.
- 5.2 Validation Loss: SIMILE training reduced both validation risk objectives faster than BLEU training and reached lower validation loss.This pattern held for expected BLEU and expected SIMILE costs.
- 5.2 Validation Loss: After one epoch, BLEU-trained models scored 86.71/27.63 on SIM/BLEU, compared with 87.14/28.10 for SIMILE-trained models.The same faster-improvement trend appeared across other language pairs.
- 5.3 Effect of n-best List Size: Increasing n-best list size improved BLEU and SIM more substantially for SIMILE-trained models than for BLEU-trained models.The authors suggest small lists may upper-bound performance, while SIMILE benefits when enough candidates are available to learn.
- Word-Level Analysis: SIMILE training produced more accurate low-frequency words, with the largest F1 differences concentrated in nouns, proper nouns, and numbers.These parts of speech are described as highly discriminative for sentence meaning.
6 Qualitative Analysis
Qualitative examples show SIMILE preserving important semantic content and improving fluency on longer sentences, while also exhibiting repetition and incomplete-translation failures.
- Translation Examples: SIMILE outputs sometimes added crucial words omitted by BLEU and MLE systems, preserving translation semantics.The examples include verbs, prepositions, adverbs, and nouns.
- Translation Examples: SIMILE also produced more fluent outputs and handled longer sentences effectively.These observations come from the fourth and fifth qualitative examples.
- Failure Cases: SIMILE failed in some cases by repeating phrases or including a semantically important noun phrase while missing the rest of the translation.BLEU avoided the repetition in one example, while SIMILE’s noun-phrase inclusion did not ensure a complete translation.
7 Metric Comparison
The metric comparison illustrates complementary behavior: SIMILE better distinguishes semantic differences that BLEU can miss, while remaining stable when lexical variation preserves meaning.
- Large SIM Difference: Large SIM differences with small BLEU differences identified cases where translations differed in meaning, including omissions and a negation despite higher BLEU.These examples show less accurate translations receiving higher BLEU scores.
- Large BLEU Difference: Large BLEU differences with small SIM differences captured cases where semantics remained similar despite synonyms, punctuation changes, or other lexical deviations.Such deviations had less impact on SIM scores than on BLEU scores.
8 Related Work
Prior work established extensive metric-optimization methods and highlighted that optimization stability can differ from evaluation quality; this paper positions SIMILE as a more stable NMT training metric with semantic-similarity foundations.
- Metric Optimization: Earlier machine-translation research optimized metrics through minimum error rate, minimum risk, maximum margin, and ranking methods.Och’s MERT work introduced metric optimization in statistical machine translation.
- Metric Stability: SMT studies found that the most effective and stable training metrics need not be the best automatic evaluation metrics, while BLEU remained a strong overall choice.The paper contrasts this history with its claim that SIMILE stabilizes NMT training more than BLEU.
- Semantic Metrics: Other SMT studies improved human judgments by optimizing semantic-role or softly matching metrics rather than BLEU.The paper presents SIMILE as achieving similar gains in NMT.
- NMT Training: NMT studies found sentence-level BLEU effective and robust for minimum-risk or structured-loss training, motivating the risk-based procedure used here.These NMT findings differ from the usual SMT pattern in which optimizing a metric improves that same metric most directly.
9 Conclusion
The paper proposes SIMILE as an alternative reward to BLEU for minimum risk training, reporting stronger automatic and human evaluation performance. The authors also find that SIMILE eases optimization and yields translations with richer semantically important content.
- SIMILE is proposed as an alternative to BLEU for use as a reward in minimum risk training.
- SIMILE outperforms BLEU on automatic evaluations and correlates better with human judgments.
- SIMILE eases optimization, while its optimized translations tend to contain more correct, semantically important words.
- The authors describe this work as the first proposal and evaluation of a continuous semantic-similarity metric for NMT optimization against sentence-level BLEU.
A.1 Annotation Instructions
The annotation instructions rate translation quality from 0 to 5 according to semantic equivalence, preservation of key information, and English well-formedness. Higher scores correspond to increasingly similar meanings and better-formed English.
- Scores range from 0 for completely different or meaningless output to 5 for essentially equal meaning in well-formed English.
- Score 1 indicates the same topic but different meaning, while score 2 indicates that some key information differs.
- Score 3 means key information is preserved although details differ.
- Score 4 denotes essentially equal meaning with some unnatural expressions.