PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation

Eleftheria Briakou, Marine Carpuat

arXiv:2105.15087v1cs.CL

TL;DR

The impact of mostly equivalent parallel samples containing a few semantically divergent tokens has remained unclear. This paper analyzes fine-grained divergences and introduces a factor-based NMT framework. On EN↔FR tasks, the framework mitigates their negative impact while improving confidence and calibration and reducing degenerated output.

  • Problem

    Prior work treats mismatches as noise or equivalent translations, leaving the impact of mostly equivalent pairs with a few divergent tokens unclear.

    Source: §1 · p. 1

  • Method

    The framework uses source-side divergence tags as feature factors and target-side tags as an additional output sequence in multi-task training.

    Source: §1 · p. 2

  • Results

    On EN↔FR tasks, fine-grained divergences hurt quality when they overwhelm training data, while the factor-based framework mitigates their impact and improves confidence and calibration.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Fine-grained divergent samples can still provide a useful training signal when modeled via factors.

    Source: §6 · p. 9

    The background training assumption treats each sentence pair as a translation of the other.

    Source: §2 · p. 2

Abstract

from arXiv · show

While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and target as noise. As a result, it remains unclear how samples that are mostly equivalent but contain a small number of semantically divergent tokens impact NMT training. To close this gap, we analyze the impact of different types of fine-grained semantic divergences on Transformer models. We show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions. Based on these findings, we introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences, improving both translation quality and model calibration on EN-FR tasks.

1 Introduction

Prior work treats parallel-sample mismatches as either noise or gold-standard translations, leaving the effect of mostly equivalent pairs with a few divergent tokens unclear. The paper analyzes these divergences and introduces a factor-based mitigation framework.

1 Introduction

  • Fine-grained divergences remain understudied because prior work treats coarse mismatches as noise and other parallel samples as gold-standard translations.The paper distinguishes mostly equivalent pairs with a few semantically divergent tokens from semantically unrelated noisy references.Source: §1 · p. 1
  • Synthetic fine-grained divergences hurt translation quality once they overwhelm equivalent data, increase degenerated outputs, and raise prediction uncertainty.The analysis controls divergence type and the ratio of equivalent to divergent samples.Source: §1 · p. 1
  • The divergent-aware framework marks tokens indicative of semantic divergence and improves translation quality, confidence, and calibration on EN↔FR tasks.It uses source-side tags as feature factors and predicts target-side tags as an additional output sequence.Source: §1 · p. 2

2 Background & Motivation

Fine-grained divergences differ from coarse noise because they affect only parts of mostly equivalent pairs, so their training signal may be imperfect yet useful. The paper examines how that signal interacts with NMT training and confidence.

2 Background & Motivation

  • Unlike coarse mismatches treated as noise, fine-grained divergences affect a small number of tokens within mostly equivalent pairs and also occur in high-quality parallel corpora.Prior work on noisy web-crawled data motivated filtering such samples (Koehn et al., 2018).Source: §2 · p. 2
  • NMT training assumes each sentence pair contains translations of one another and maximizes token-level cross-entropy under that assumption.This assumption frames how a divergent target can affect the training signal.Source: §2 · p. 2
  • With fine-grained divergences, partially corrupted prefixes can condition predictions or encourage wrong tokens, making the signal noisy yet potentially useful depending on the time step.The paper also examines possible effects on model confidence.Source: §2 · p. 2

3 Analyzing the Impact of Divergences

The paper simulates three types of fine-grained divergence, trains on mixtures of equivalent and divergent WikiMatrix data, and evaluates translation quality. Synthetic corruption reduces quality, with its impact varying by divergence type.

3.1 Method

  • The study injects increasing amounts of synthetic divergences into equivalent pairs, testing lexical substitution, phrase replacement, and subtree deletion one type at a time.The corruptions respectively use WordNet hypernyms or hyponyms, phrases with matching POS tags, and deleted dependency-tree subtrees.Source: §3.1 · pp. 2–3
  • Synthetic data is generated by corrupting equivalent pairs identified with a Divergent mBERT classifier that scores F1 84 on manually annotated WikiMatrix data.The classifier was trained on synthetic data.Source: §3.1 · p. 3

3.2 Experimental Set-Up

  • Training uses WikiMatrix French-English pairs, with official TED development and test splits; results are averaged across three random seeds.The TED splits contain 4,320 development and 4,866 test translation pairs.Source: §3.2 · p. 3

3.3 Findings

  • Corrupting equivalent bitext with fine-grained divergences hurts BLEU and METEOR across the board, usually in proportion to the share of corrupted training samples.The degradation is relatively smaller for METEOR than BLEU, which the authors attribute to METEOR matching synonyms.Source: §3.3 · p. 3
  • Lexical substitution causes the largest degradation for BLEU and METEOR, while phrase-replacement degradation becomes significant only after corrupting ≥70% of equivalents, measured by BLEU.Source: §3.3 · p. 3

SUBSTITUTION

Fine-grained divergences can degrade translation quality and increase degenerated outputs and model uncertainty. Their effects depend on divergence type: lexical substitution most harms translation quality, while phrasal mismatches produce more repetitive outputs.

SUBSTITUTION

  • When divergences overwhelm training data, BLEU and METEOR degrade, with Transformers particularly sensitive to LEXICAL SUBSTITUTION.Table 2 reports FR→EN results on the TED test set and compares corruption levels with EQUIVALENTS.Source: Table 2 · p. 4
  • PHRASE REPLACEMENT hurts model confidence most at training and inference, while SUBTREE DELETION affects confidence less than other divergence types.Figure 2 measures token probabilities conditioned on gold-reference prefixes and beam-search prefixes.Source: SUBSTITUTION · p. 4, Figure 2 · p. 4
  • With 50% or more divergent training samples, hypotheses are longer than references; analysis at 100% corruption suggests degenerations partly explain this length effect.Degenerations are defined as bland, incoherent, or repetitive-loop output.Source: SUBSTITUTION · p. 4
  • Training on fine-grained divergences increases degenerated outputs across beam sizes; phrasal mismatches produce more repetitions than lexical substitutions.At beam sizes ≥5, lexical substitutions nearly match EQUIVALENTS in repetition frequency.Source: Figure 3 · p. 4, SUBSTITUTION · p. 5
  • The impact on BLEU appears smaller than that of noise (Khayrallah and Koehn, 2018), suggesting noise-filtering techniques are suboptimal for fine-grained divergences.LEXICAL SUBSTITUTION causes the largest translation-quality degradation, while SUBTREE DELETION and PHRASE REPLACEMENT increase degenerated beam hypotheses.Source: SUBSTITUTION · p. 5

4 Mitigating the Impact of Fine-grained Divergences

The section introduces token-level divergence factors to help NMT distinguish equivalent from divergent content, then evaluates their effects on translation quality, degenerated outputs, confidence, and calibration. Factorized models mitigate degradation from naturally occurring divergences across the reported settings.

4.1 Factorizing Divergences for NMT

  • Source-side divergence tags are concatenated with token embeddings, while target-side tags are predicted as an additional sequence alongside target tokens.The target-factor prediction is time-shifted so the model predicts a reference token before its corresponding EQ/DIV label; at inference, input tokens receive EQ tags.Source: §4.1 · pp. 5–6

4.2 Experimental Set-Up

  • The experiments compare factorized models with LASER-filtered, equivalent-only, divergence-agnostic, and source-tagged models, including an out-of-domain medical test set.The study varies the amount and granularity of divergent WikiMatrix data and evaluates BLEU, METEOR, InfECE, and token-level accuracy.Source: §4.2 · p. 6

4.3 Results

  • On TED test sets, adding divergent data without modeling it hurts translation quality, with significantly larger drops when divergences overwhelm equivalent translations.EQUIVALENTS performs better or statistically comparably to all models and outperforms LASER across metrics and directions despite using about half the training data.Source: §4.3 · p. 6
  • DIV-FACTORIZED recovers translation quality across data setups, reaching scores comparable to EQUIVALENTS while mitigating degradation from divergent samples.Table 3 reports EN↔FR TED test results; the prose also reports approximately +1 BLEU over DIV-AGNOSTIC in both directions.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • At 55% real divergences, degenerated loops are almost twice as frequent across beam sizes, while DIV-FACTORIZED consistently mitigates them.At some lower divergence shares (20% and 33%), DIV-FACTORIZED reduces degenerated text by half relative to EQUIVALENTS.Source: §4.3 · p. 7, Table 5 · p. 7
  • DIV-AGNOSTIC models assign lower probabilities to gold references and predicted tokens, especially early in the sequence; factorization mitigates this confidence drop.Figure 4 summarizes average token probability over time on the TED test set.Source: §4.3 · p. 8, Figure 4 · p. 8
  • DIV-FACTORIZED models are more confident and accurate than DIV-AGNOSTIC models, while DIV-AGNOSTIC has smaller calibration errors than EQUIVALENTS and LASER.Table 6 reports that DIV-FACTORIZED yields the smallest calibration errors.Source: §4.3 · p. 8, Table 6 · p. 8

5 Related Work

Related work distinguishes fine-grained semantic divergences from coarse noise filtering and connects this paper to prior work on factored translation models and calibration. The paper focuses on semantic properties of training data and uses both source- and target-side factors.

Related work

  • Prior factored translation work injects linguistic features or predicts multiple target sequences; this paper combines source and target factors to encode semantics.Relevant approaches include source-side factors in MT (Haddow and Koehn, 2012; Sennrich et al., 2016b; Hoang et al., 2016) and target-side factors (García-Martínez et al., 2017).Source: §5 · p. 8
  • Calibration research addresses miscalibration through recalibration, label smoothing, and linguistic data properties; this paper instead examines semantic properties of training data.Prior work links miscalibration to EOS behavior, attention uncertainty, noisy references, and token properties (Kumar and Sarawagi, 2019; Ott et al., 2018; Müller et al., 2019; Wang et al., 2020).Source: §5 · p. 9

6 Conclusion

On EN↔FR tasks, fine-grained divergences can harm translation quality and confidence, while divergent-aware factorization mitigates their effects. The conclusion also identifies other text properties and language pairs, especially low-resource conditions, for future analysis.

6 Conclusion

  • When fine-grained divergences overwhelm training data, translation quality suffers; exposed models are less confident, harming beam search and increasing repetitive-loop outputs.Source: §6 · p. 9
  • On EN↔FR tasks, factor-based divergent-aware NMT mitigates divergent references’ quality impact, improves prediction confidence and calibration, and reduces degenerated text.Source: §6 · p. 9
  • The authors plan to extend the analysis to other parallel-text properties and language pairs, focusing on low-resource conditions where divergences are expected to be more prevalent.Source: §6 · p. 9

A WikiMatrix Fine-grained Divergences

  • Figure 5 shows bicleaner-score distributions for REFRESD and English-French WikiMatrix divergences.Source: Figure 5 · p. 13

B Sockeye2 configuration details

C Measuring Degenerated Hypotheses

  • The degeneration-checking algorithm identifies odd repetitions unsupported by the reference, excluding punctuation and conjunctions when counting repeated n-grams.An n-gram is considered repeated when it occurs more than once in the hypothesis or reference.Source: C Measuring Degenerated Hypotheses · p. 13

D Synthetic Divergences Statistics

E METEOR Results (addition)

METEOR comparisons show smaller improvements for DIV-FACTORIZED than the BLEU differences, and the authors caution that synonym matching may mislead medical-translation evaluation.

E METEOR Results (addition)

F Degenerated Hypotheses (addition)

For EN→FR, DIV-FACTORIZED reduces degenerated outputs across the beam settings reported in Table 12.

F Degenerated Hypotheses (addition)

  • DIV-FACTORIZED decreases the percentage of degenerated outputs caused by divergent data.Source: F Degenerated Hypotheses (addition) · p. 14
  • Table 12 compares degenerated-output percentages across beams for EN→FR and reports fewer repetitions with DIV-FACTORIZED.Source: Table 12

Related papers

Loading 2105.15087v1…