PaperlayerView exact revision on arXiv

Source-linked AI summary

Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

Marcin Junczys-Dowmunt

arXiv:1907.06170v1cs.CL

TL;DR

The paper addresses how to improve English-German translation by incorporating document-level context beyond strong sentence-level systems. It develops deep-transformer document-level systems using document-bounded sequences and related training and decoding methods. Preliminary human evaluations favor the document-level systems over comparable sentence-level systems, with source-based assessment also appearing to rank them above human references.

  • Problem

    The paper investigates whether document-level neural machine translation can improve English-German translation beyond strong sentence-level systems affected by translationese input and uncertain test-set interpretation.

  • Method

    The authors train deep transformer models on document-bounded sequences of up to 1000 subword segments and explore augmentation, BERT-based multi-task training, fine-tuning, ensembling, and two-pass decoding.

  • Results

    Preliminary human evaluations strongly prefer the document-level systems over comparable sentence-level systems, while source-based direct assessment appears to score them above human references.

  • Takeaways & Limitations

    Document-level systems can translate whole documents with improved quality over comparable sentence-level systems, and deeper models seem to benefit more from added context.

  • Takeaways & Limitations

    The authors caution that the preliminary human-evaluation ranking is difficult to interpret because raw scores and normalized z-scores differ substantially.

Abstract

from arXiv · show

This paper describes the Microsoft Translator submissions to the WMT19 news translation shared task for English-German. Our main focus is document-level neural machine translation with deep transformer models. We start with strong sentence-level baselines, trained on large-scale data created via data-filtering and noisy back-translation and find that back-translation seems to mainly help with translationese input. We explore fine-tuning techniques, deeper models and different ensembling strategies to counter these effects. Using document boundaries present in the authentic and synthetic parallel data, we create sequences of up to 1000 subword segments and train transformer translation models. We experiment with data augmentation techniques for the smaller authentic data with document-boundaries and for larger authentic data without boundaries. We further explore multi-task training for the incorporation of document-level source language monolingual data via the BERT-objective on the encoder and two-pass decoding for combinations of sentence-level and document-level systems. Based on preliminary human evaluation results, evaluators strongly prefer the document-level systems over our comparable sentence-level system. The document-level systems also seem to score higher than the human references in source-based direct assessment.

1 Introduction

The paper develops document-level English-German translation with deep transformer models for WMT19, building on strong sentence-level systems and large-scale filtered and back-translated data. Preliminary human evaluations favor the document-level systems over comparable sentence-level systems.

  • The submission focuses on document-level neural machine translation with deep transformer models for the WMT19 English-German news translation task.
  • The authors begin with sentence-level systems trained using data filtering and noisy back-translation, finding that back-translation mainly helps with translationese input.
  • Document-level models use authentic and synthetic parallel data with document boundaries to create sequences of up to 1000 subword segments.
  • The study explores fine-tuning, deeper models, ensembling, data augmentation, BERT-based multi-task training, and two-pass decoding for document-level translation.
  • Preliminary human evaluations strongly prefer the document-level systems over comparable sentence-level systems, and source-based assessment appears to place them above human references.

2 Sentence-Level Baselines

The sentence-level baselines combine filtered parallel data, noisy back-translation, deeper transformers, fine-tuning, and ensembling while accounting for source-language splits. Back-translation improves the originally-German side but can reduce quality on originally-English input, motivating fine-tuning and weighted heterogeneous ensembles.

  • Evaluation: The evaluation uses SacreBLEU on dev sets split by original source language, alongside combined and original test-set results.The split interpretation is uncertain: originally-English scores are treated as adequacy indicators and originally-German scores as fluency indicators.
  • Noisy Back-Translation: 550M noisy back-translated sentences improve the originally-German side but substantially reduce quality on the originally-English side.The unsplit WMT test results hide this directional trade-off.
  • Noisy Back-Translation: Fine-tuning the back-translation model on filtered parallel data restores and improves originally-English quality while retaining most originally-German gains.The model continues training on filtered data until the originally-English development score stops improving.
  • Deeper Models: 12-layer models mostly improve the originally-German side without reducing originally-English quality, whereas 18- and 24-layer trials show no further gains.The authors retain 12-layer models for subsequent experiments but note possible hyper-parameter problems in the deeper trials.
  • Ensembling: Heterogeneous ensembles strengthen originally-English results but lose quality on originally-German data, while down-weighting parallel-only models recovers part of that loss.The final sentence-level model uses 0.3 weight for four parallel-only models and 1.0 for four back-translation-plus-fine-tuned models.

3 Document-Level Systems

The paper extends document-level MT by consuming full documents within 1000 subword units, using deeper transformer models and BERT-style encoder training. It builds on long-sequence modeling while omitting sentence embeddings.

  • 3 Document-Level Systems: Document-level MT consumes full documents up to 1000 subword units, potentially spanning 20 or more sentences.This removes the two-sentence context limit used in earlier work.
  • 3 Document-Level Systems: The approach increases training-sequence length and model depth, while also using the BERT masked-LM criterion for encoder co-training.The paper does not use sentence embeddings.
  • 3 Document-Level Systems: A validation document is marked with explicit beginning, ending, and sentence-separator symbols for document-level processing.The example illustrates the markup used to represent document structure.

3.1 Data and Data Preparation

The data pipeline reconstructs or creates document boundaries across authentic, synthetic, and monolingual data, then segments documents consistently across languages. Augmentation compensates for limited authentic documents and missing boundaries.

  • 3.1 Data and Data Preparation: Document boundaries are restored for subsets of parallel data, while monolingual news-crawl data retains boundaries for all content.The datasets are assembled into real and fake documents with varying augmentation.
  • 3.1 Data and Data Preparation: Parallel sentences are concatenated into sequences with matching sentence counts on both language sides.Outputs are split at predicted separators and sentence-aligned with sentence-level translations as a fail-safe.
  • 3.1 Data and Data Preparation: Sequences use <BEG>, <END>, and <SEP> markers, with <BRK> and <CNT> for documents exceeding the 1000-subword limit.Long documents are broken consistently across languages, and all training and validation data receives the same markup.
  • 3.1 Data and Data Preparation: Authentic documents are augmented with up to 10 randomly selected continuous sub-documents per full document.This expands the relatively small corpus of approximately 200K original documents without allowing longer documents to dominate.
  • 3.1 Data and Data Preparation: Boundary-free authentic sentences are shuffled into fake documents with consistent internal sentence boundaries, repeatedly until reaching roughly half the back-translated corpus size.Back-translated news-crawl data uses its existing document boundaries without additional document-level augmentation.

3.2 Experiments

Experiments compare long-sequence document models, multi-task BERT-style encoder training, and second-pass decoding. Long document decoding preserves performance, while BERT effects are uncertain and second-pass decoding lowers BLEU.

  • 3.2 Experiments: Document-level models decoding sequences up to 1000 tokens do not appear to degrade translation performance relative to comparable sentence-level systems.BLEU trends broadly follow sentence-level results, including gains from deeper models.
  • 3.2 Experiments: Multi-task training adds a BERT-style masked-LM objective over monolingual English documents to the shared translation encoder.The masked-LM cost is trained alongside the translation cross-entropy objective.
  • 3.2 Experiments: Large-scale back-translated document training shifts quality toward the originally-English side, but the authors state that the improvement is generally unclear.They nevertheless select the multitask models based on a preference for that side's improvements.
  • 3.2 Experiments: Second-pass decoding combines sentence-level inputs with document-level post-editing, but both tested applications show BLEU degradation.The second-pass systems nonetheless rank among the better single document-level models when applied over the strong sentence-level ensemble.

3.3 Stacking and Ensembling

Ensembling combines diverse document-level models and can trade performance between original-language halves of the evaluation data. A four-model fine-tuned document ensemble is identified as promising.

  • 3.3 Stacking and Ensembling: A pure document-level ensemble of four fine-tuned 12-layer models is described as a promising candidate.Ensembling all eight diverse models raises quality on the originally-English side while losing comparable quality on the originally-German half.

4 Submissions

Microsoft submitted four English-German systems, including sentence-level, document-level, and hybrid systems, and compared them using BLEU and preliminary human evaluation. Document-level systems ranked strongly and received substantial human-evaluation preference, though the preliminary ranking warrants caution.

  • Submitted systems: Microsoft submitted four systems: an original WMT18 system, a sentence-level ensemble, a document-level ensemble, and a hybrid ensemble with second-pass decoding.The hybrid combined sentence-level and document-level systems through second-pass decoding.
  • Automatic evaluation: The document-level systems ranked second behind MSRA’s highest submission in BLEU on the WMT19 test set.Table 6 also reports BLEU scores for the split development set.
  • Human evaluation: Human evaluators strongly preferred the document-level systems over the comparable sentence-level system, despite this gain being less apparent in BLEU scores.The systems were comparable in raw data, model size, and training setting.
  • Human evaluation: The document-level systems appeared to outperform human references in quality, although their normalized z-score advantage was non-significant.The evaluation results were preliminary and organizer-reported.
  • Evaluation caveat: The authors questioned the preliminary Table 7 ranking because average raw scores and normalized z-scores differed substantially.They called for the final ranking and organizer commentary before drawing firm conclusions.
Loading 1907.06170v1…