Filtering and Mining Parallel Data in a Joint Multilingual Space

1 Introduction

• Noisy parallel data matters because neural machine translation systems are more sensitive to noise than statistical machine translation systems.Potential translations in comparable corpora, such as Wikipedia or news collections, also need to be identified and aligned.Source: §1 · p. 1
• A joint multilingual sentence embedding supports both filtering noisy bitexts and mining for additional bitexts in monolingual texts.A distance threshold filters source-target pairs or identifies candidate pairs from pairwise distances across source and target sentences.Source: §1 · p. 1
• The approach requires no additional features or classifiers.Source: §1 · p. 1

2 Related work

• Domain adaptation selects bitexts relevant to a test domain, but it does not necessarily remove pairs that are not mutual translations.The passage notes that domain adaptation was reported as less successful for neural machine translation than for phrase-based systems.Source: §2 · p. 1
• Some prior mining work uses web-page names or links, while other approaches use cross-lingual information retrieval or joint embeddings.Grégoire and Langlais (2017) use separate sentence spaces followed by a classifier trained on labeled data.Source: §2 · p. 1
• Unlike earlier cited embedding approaches that learn embeddings for two languages, this paper learns one joint embedding for up to nine languages.Source: §2 · p. 2

3 Multilingual sentence embeddings

• The intended joint space places entire sentences from different languages so that distance reflects semantic difference independently of language.Source: §3 · p. 2
• A shared encoder handles all input languages, with no target-language token and no language information in the encoded sentence representation.The architecture uses multiple sequence encoders and decoders trained on N-way aligned corpora; this paper replaces per-language encoders with a shared encoder.Source: §3 · p. 2
• The architecture was trained on nine Europarl languages, with about 2M sentences per language and one shared 20k BPE vocabulary.The nine languages are English, French, Spanish, Italian, Portuguese, German, Danish, Dutch, and Finnish.Source: §3 · p. 2
• A three-layer BLSTM produces 1024-dimensional sentence embeddings by max-pooling its outputs, and the trained encoder is used to embed sentences.The word embeddings are size 384, and the BLSTM hidden layer is 512-dimensional.Source: §3 · p. 2
• A related multilingual NMT approach uses a target-language token and sums encoder output states for sentence representations.The cited approach is España-Bonet et al. (2017).Source: §3 · p. 2

4 Experimental evaluation: BUCC shared task on mining bitexts

• The paper applies its approach to all BUCC language pairs, with task-specific adaptation limited to optimizing the distance threshold.For Russian and Chinese, it uses embeddings trained on the UN corpus.Source: §4 · pp. 2–3
• The BUCC evaluation data are summarized by sentence counts and the sizes of gold alignments.The text reports that roughly one fortieth of the sentences are aligned.Source: Table 1 · p. 2, §4 · p. 2
• BUCC’s official results are reported as F-scores for the 2017 and 2018 shared tasks.The cited description identifies VIC as the best-performing system and describes its STACC basis and task-specific adaptations.Source: §4 · p. 2, Table 2 · p. 3
• The system is on par with H2 on en-fr and outperforms all other approaches by a large margin, but does not match heavily tuned VIC.The approach uses no additional features or classifiers and is applied in the same way across language pairs; its performance varies little across languages.Source: §4 · p. 3, Table 3 · p. 3
• The paper’s BUCC results table reports precision, recall, F-measure, and the optimal distance threshold for its approach.Source: Table 3 · p. 3
• Some mined alignments incorrectly pair sentences that mainly enumerate named entities, and the authors suggest comparing entity counts as possible post-processing.Source: §4 · p. 3

5.1 Baseline NMT systems

• The WMT’14 English-to-German evaluation uses newstest-2014 with case-sensitive BLEU, comparing human-translated training data with all 4.5M human and crawled sentence pairs.The human-translated portion contains 2.1M sentences; the full training set adds 2.4M Common Crawl pairs.Source: §5.1 · p. 3

5.2 Filtering Common Crawl

• Pre-filtering removes sentences with more than three commas, limits length to under 50 words, and applies language identification; these steps discard 19% of the data, including almost 6% with an apparently wrong source or target language.Source: §5.2 · p. 4
• 0.3 BLEU improvement (24.75→25.06) at threshold 1.0 uses 3.4M rather than 4.5M sentence pairs.The filtered system reaches 25.06 BLEU; Figure 1 plots corpus size and BLEU against the distance threshold.Source: §5.2 · p. 4, Figure 1 · p. 4

5.3 Mining Parallel Data in WMT News

• 1M mined sentences yield 4.18 BLEU, rising to 7.77 BLEU with 4.3M sentences, still below a system trained on real parallel data.As the distance threshold grows, the mined pairs become less likely to be translations.Source: §5.3 · p. 5
• WMT training sentences average 24 words, compared with 8 words for mined texts; longer crawled-news sentences are less likely to have perfect translations, and short-sentence effects on NMT remain open.The authors say further research is needed on these sentence-length questions.Source: §5.3 · p. 5
• 0.45 BLEU improvement (21.87→22.32) comes from adding mined data to the 2.1M-sentence Europarl and News Commentary corpora, but not to the best filtered-Common-Crawl system.The authors suggest some mined data may overlap with Common Crawl.Source: §5.3 · p. 5, Table 6 · p. 5

• A joint multilingual sentence-embedding space and cosine distance support both noisy-bitext filtering and bitext mining in large news collections.Source: §6 · p. 5
• Filtering improves the WMT’14 English-to-German baseline by 0.3 BLEU while removing 25% of the training data.Source: §6 · p. 5
• The authors plan to scale the approach to larger corpora and identify MT confidence estimation and back-translation filtering as possible applications.They specifically mention applying it to data mined by the European ParaCrawl project.Source: §6 · p. 5

• The supplement shows mined BUCC sentence pairs for language pairs outside the official task, including French–German and Russian–Chinese.Source: A Supplemental Material · p. 8
• All but the last French–German example pair are perfect mutual translations; the last mismatches sentences that both enumerate country names.Source: A Supplemental Material · p. 8
• Comparing the number of named entities is suggested as a possible post-processing step to exclude some erroneous alignments.Source: A Supplemental Material · p. 8
• The Russian–Chinese example alignments seem perfect according to their English translations from a machine-translation system.Source: A Supplemental Material · p. 8