Source-linked AI summary
Tagged Back-Translation
Isaac Caswell, Ciprian Chelba, David Grangier
TL;DR
Noised-beam back-translation had been linked to source diversity, but this paper argues that its key function is signaling synthetic sources. It introduces Tagged Back-Translation, which matches noised back-translation on English-German and outperforms it on English-Romanian.
Problem
The paper questions whether noised-beam back-translation improves translation through source diversity or instead signals that training sources are synthetic.
Method
Tagged Back-Translation replaces heuristic source noising with a reserved token marking back-translated source sentences.
Results
Tagged Back-Translation matches or slightly exceeds noised variants on English-German and improves over standard and noised back-translation on English-Romanian, reaching 33.4 BLEU with iterative tagging.
Takeaways & Limitations
Noising in back-translation can function primarily as a synthetic-source indicator, allowing models to treat synthetic data as a different domain and separate helpful from harmful signals.
Takeaways & Limitations
English-German iterative Tagged Back-Translation showed no BLEU gains or losses, and the limited effect size and experiment count preclude strong conclusions about iterative tagging.
Abstract
from arXiv · showhide
Recent work in Neural Machine Translation (NMT) has shown significant quality gains from noised-beam decoding during back-translation, a method to generate synthetic parallel data. We show that the main role of such synthetic noise is not to diversify the source side, as previously suggested, but simply to indicate to the model that the given source is synthetic. We propose a simpler alternative to noising techniques, consisting of tagging back-translated source sentences with an extra token. Our results on WMT outperform noised back-translation in English-Romanian and match performance on English-German, re-defining state-of-the-art in the former.
1 Introduction
Back-translation uses abundant target-side monolingual data to create synthetic parallel pairs, and this work argues that noising mainly signals their synthetic origin rather than diversifying sources. Tagged Back-Translation replaces noising with an explicit source tag and performs comparably or better.
- Back-translation trains a target-to-source model on bitext, then translates target-side monolingual data to create synthetic pairs for source-to-target training.The resulting training mixture combines genuine parallel data with back-translated synthetic pairs.
- Prior large-scale studies found sampled and noised-beam back-translation outperform standard back-translation, attributing gains to increased source-side diversity.
- The paper instead hypothesizes that noise signals back-translated sources, allowing models to separate helpful target-language and weak cross-lingual signals from harmful translation bias.
- Noise did not improve or significantly degrade NMT accuracy in isolation, supporting its role as an indicator with limited loss of source-target mutual information.
- Tagged Back-Translation adds a reserved token to synthetic sources; it matches or slightly exceeds noised variants on English-German and improves over both standard and noised back-translation on English-Romanian.It also enables effective iterative back-translation for English-Romanian, where standard back-translation previously lost quality.
2 Related Work
Related work uses monolingual data through language models, back-translation, iterative procedures, and other training strategies, while source tags control model behavior across languages, styles, genders, and domains.
- Target-side language-model methods have improved NMT but generally underperform back-translation, which can yield substantially larger gains on the same English-German test set.One cited comparison reports under +0.5 BLEU for a language-model method versus over +4.0 BLEU for back-translation.
- Back-translation became prominent in NMT after its introduction to phrase-based systems, and later work explored iterative forward- and back-translation using monolingual data.
- Recent studies investigate diversity and complexity in synthetic training data, including selecting difficult words and sampling multiple sources per target sentence.
- Other monolingual-data approaches include forward translation and source copying.
- NMT source tags have been used to indicate target language, control formality or gender, and mark source-sentence domains.
3 Experimental Setup
The experiments evaluate noising and tagging strategies across WMT English-German, English-Romanian, and English-French data using filtered bitext, large monolingual corpora, Transformer models, and SacreBLEU.
- Experiments use WMT18 English-German, WMT16 English-Romanian, and WMT15 English-French bitext, with Newscrawl monolingual data for each language setting.The resulting bitext sizes include about 5.0M English-German pairs and 0.6M English-Romanian pairs; English-French bitext contains 41M unfiltered sentences.
- Monolingual data are deduplicated and filtered by length before and after back-translation, yielding 216.5M English-German and 2.2M English-Romanian sentences among the reported corpora.
- Evaluation relies on SacreBLEU, using the reference implementation intended to standardize BLEU computation because preprocessing differences can substantially change scores.
- The study uses Transformer-base for bitext noising and English-Romanian experiments, Transformer-big for English-German back-translation tasks, and 32k subword vocabularies.
- 3.4 Noising: NoisedBT applies 10% word dropout, 10% word blanking, and a 3-constrained permutation; P3BT uses only the constrained permutation.
- 3.4 Noising: Table 1 presents examples of the five noising settings examined in the paper.
4 Results
Experiments show that tagging synthetic sources can replace noising as the useful back-translation signal. TaggedBT matches or exceeds NoisedBT across English-German, English-Romanian, and English-French settings, while iterative gains remain task-dependent.
- 4.1 Noising Parallel Bitext: Noising 100% of EnDe bitext sources reduced SacreBLEU by only 2.5, indicating that source noise itself causes limited degradation.This motivates testing whether noise can identify synthetic data without substantially harming learned translation signals.
- 4.2 Tagged Back-Translation for EnDe: On 24M EnDe back-translated sentences, all source-marking methods performed about equally, with TaggedBT achieving the highest average BLEU by a small margin.Combining tagging and noising did not improve over either method alone, supporting a shared signaling function.
- 4.2 Tagged Back-Translation for EnDe: At full EnDe scale, TaggedBT matched or slightly outperformed NoisedBT on seven test sets and lost on one, with a largest advantage of +1.4 BLEU.Against Edunov et al.’s best model, it outperformed on four test sets and underperformed on two.
- 4.2 Tagged Back-Translation for EnDe: Training on NoisedBT alone produced about 1 BLEU lower performance than training on unaltered BT alone, challenging source-diversity as noise’s sole benefit.This comparison isolates the effect of noising when synthetic data is used without bitext.
- 4.3 Tagged Back-Translation for EnRo: On lower-resource EnRo, NoisedBT lagged standard BT by -0.6 BLEU, while TaggedBT exceeded standard BT by +0.4 BLEU and NoisedBT by +1.0 BLEU.The EnRo table compares different BT variants using SacreBLEU; prior-work scores may use a different metric protocol.
- 4.5 Iterative Tagged Back-Translation: Iterative BT improved dev+test average by +1.0 BLEU for NoisedBT and +0.7 BLEU for TaggedBT, but standard BT changed by -0.1 BLEU.An identical EnDe iteration-3 TaggedBT experiment showed neither gains nor losses, and the authors caution that the evidence is limited.
5 Analysis
The analysis examines how TaggedBT and NoisedBT alter attention and decoding when models distinguish synthetic from natural source text. TaggedBT concentrates attention on its tag while preserving diffuse attention and reasonable decoding quality without explicit source noising.
- Attention Sink Ratio: Attention Sink Ratio measures decoder attention on a source-token index relative to uniform attention, using top-layer attention to the first source token.The analysis applies this measure to sentences treated as back-translated through tagging or noising.
- Attention entropy: BT-trained models show unusually low attention entropy concentrated near the diagonal, whereas NoisedBT and TaggedBT produce higher, more diffuse entropy on natural text.The paper interprets low entropy as possible word-by-word translation bias and reports that TaggedBT achieves similarly high entropy without manually disrupting the input.
- Attention maps: Attention maps compare BT, NoisedBT, and TaggedBT across EnDe and EnRo, with TaggedBT strongly attending to the tag and NoisedBT learning diffuse attention.The comparison uses first-layer encoder attention for a random training example.
- Attention entropy: TaggedNoisedBT has lower entropy than TaggedBT because it assigns disproportionately high attention to the sentence-end token, reaching 1.4x TaggedBT’s ASR|x|.This is presented as the authors’ best explanation for the unexpected entropy result.
- Decoding with and without a tag: As-if-BT decoding tags or noises true source sentences, enabling comparison with standard decoding using the same models.Table 7 reports BLEU for these decoding conditions, while Table 8 provides example outputs.
- Decoding with and without a tag: Tagged decoding loses 2.9 BLEU on average versus standard decoding, but remains reasonably effective and produces about twice as many English outputs.The English-output rates are 2.7% for tagged decoding versus 1.2% for standard decoding over newstest2010-newstest2017.
6 Negative Results
Several tagging variants failed to improve translation consistently. Forward-translation tagging provided little benefit, while year-specific back-translation tags matched ordinary TaggedBT and did not improve standard decoding.
- Tagged Forward-Translation: Tagged forward-translation made no substantial difference and often lagged behind untagged forward-translation by approximately 0.2 BLEU.For EnDe, both forward-translation variants underperformed the bitext baseline; for EnRo, they performed about the same as back-translation.
- Combined Translation Data: Combining back-translation and forward-translation produced 33.9 SacreBLEU on test, slightly above iteration-3 TaggedBT, but tagging itself did not help this setup.
- Year-Specific Tags: Year-specific tags produced a model identical in performance to the normal TaggedBT model.
- Year-Specific Tags: Standard decoding achieved the highest BLEU in the year-tagged “as-if-bt” experiments despite only a slight correlation between year tags and their datasets.
7 Conclusion
The paper introduces TaggedBT as a simpler and more robust alternative to Noised Back-Translation. TaggedBT performs well across task settings, improves English-Romanian results, and supports the conclusion that noising primarily signals synthetic data to the model.
- TaggedBT improves over Noised Back-Translation while being simpler and more robust.
- TaggedBT performs well across low-resource, midresource, and iterative back-translation tasks, whereas standard and noised back-translation vary by task.
- 33.4 BLEU from Iterative TaggedBT gains 3.5 BLEU over the highest previously published WMT16 English-Romanian result known to the authors.
- TaggedBT improves WMT16 English-Romanian over vanilla back-translation by 0.4 BLEU and matches or exceeds the highest known back-translation-only English-German results on five of seven test sets.
- The authors conclude that back-translation noising mainly indicates that the source is synthetic, allowing the model to treat it as a separate domain.
8 Future Work
The paper proposes extending tagging beyond a single synthetic-data token. Future directions include finer-grained quality and domain tags, alternatives to data selection or curriculum learning, and comparisons with richer domain representations.
- Future work could apply finer-grained quality, domain, and synthetic-data tags to natural and synthetic data in both back-translation and forward-translation.
- Tagging could be investigated as an alternative to data selection and curriculum-learning approaches such as fine-tuning on in-domain data.
- The token-tagging method should be compared with more sophisticated approaches that concatenate a trainable domain embedding with every token embedding.