Cross-Lingual Natural Language Generation via Pre-Training

1 Introduction

• Most available annotated NLG datasets are single-language, typically English, limiting application deployment in other languages and motivating transfer to unseen languages.The stated aim is to boost performance in low-resource settings.Source: §1 · p. 1
• Cross-lingual NLG requires multilingual input understanding and multilingual target generation, making pre-training both encoder and decoder important.Its many-to-many language-pair setting grows with the square of the number of languages, and decoder knowledge transfer matters because the prediction space exceeds that of classification.Source: §1 · p. 1
• Earlier MT-based approaches either translate inputs and outputs around an English NLG model or create pseudo-training data, but multiple MT systems can propagate errors.The introduction also notes that pipeline methods lack explicit shared parameter space for direct transfer of task-specific supervision.Source: §1 · p. 1
• XNLG shares a sequence-to-sequence model across languages and pre-trains it with monolingual and cross-lingual objectives for transfer of monolingual NLG supervision.A monolingual NLG dataset is used for fine-tuning, followed by evaluation with other source and target languages.Source: §1 · pp. 1–2, Figure 1 · p. 1
• XNLG achieves competitive performance against machine-translation-based pipeline models in zero-shot cross-lingual NLG experiments.The introduced evaluation datasets cover question generation and abstractive summarization in English, Chinese, and French.Source: §1 · p. 2

2 Related Work

• Cross-lingual summarization studies use translated documents or summaries as pseudo-training data [Shen et al. (2018); Duan et al. (2019)] or combine monolingual summarization with machine translation [Junnan et al. (2019)].These systems test generation into a language different from the input, rather than transferring supervision across all language pairs.Source: §2 · p. 2
• Multilingual sequence-to-sequence training for question generation has used annotations in multiple languages [Kumar et al. (2019)], whereas this method also targets zero-shot transfer.Source: §2 · p. 2
• Monolingual pre-training work includes language modeling, auto-encoding, and machine translation objectives, as well as generative pre-trained models [Joshi et al. 2019; Yang et al. 2019; Liang et al. 2019; Dong et al. 2019; Song et al. 2019].This paper focuses on pre-training for cross-lingual NLG tasks.Source: §2 · p. 2
• Multilingual BERT and cross-lingual masked-language pre-training show cross-lingual representations and gains in classification or unsupervised translation [Wu and Dredze 2019; Lample and Conneau 2019].XNLG instead pre-trains both encoder and decoder for cross-lingual generation.Source: §2 · p. 2
• Directly fine-tuning a pre-trained translation model on downstream NLG tasks makes target-language control difficult, according to the cited experiments.Source: §2 · p. 2

3 Methods

• Language tags let XNLG encode multilingual inputs and condition generation on the desired target language.The model is based on Transformer and supports multiple languages in both its encoder and decoder.Source: §3 · p. 2

3.1 Pre-Training Tasks

• Monolingual MLM trains the encoder to predict randomly masked words from context, masking 15% of tokens with three replacement choices.A masked token is replaced by [M], a random token, or left unchanged with probabilities 0.8, 0.1, and 0.1.Source: §3.1 · p. 2
• XMLM applies masked-token prediction to concatenated parallel sentences, using their alignment to map cross-lingual texts into a shared space.It uses the monolingual masking strategy and language tags to indicate each token’s language.Source: §3.1 · pp. 2–3
• XAE uses machine translation on parallel sentence pairs to recover sentences from encoded representations, reducing dependence between target sentences and source-language tags.This addresses the tendency of DAE-only decoder pretraining to ignore the target-language tag and generate in the input language.Source: §3.1 · p. 3

3.2 Pre-Training Protocol

• The first pretraining stage uses MLM and XMLM to encode multilingual sentences into a shared embedding space.It combines monolingual and parallel corpora.Source: §3.2 · p. 3
• The second stage pretrains decoding with DAE and XAE while freezing the encoder, addressing untrained encoder-decoder attention and the mismatch with autoregressive decoding.Source: §3.2 · p. 3

3.3 Fine-Tuning on Downstream NLG Tasks

• For Any-to-Others NLG, fine-tuning only the encoder preserves target-language controllability that can be lost when all parameters are updated.The decoder and word embeddings remain frozen.Source: §3.3 · p. 4
• For Any-to-English NLG, the encoder can be frozen while updating the decoder, or all parameters can be fine-tuned for better English results with a slight performance drop.Source: §3.3 · p. 4

4 Experiments

• Across QG and abstractive summarization, experiments compare XNLG with cross-lingual pretrained models and machine-translation pipelines.The summarization setup uses English, French, and Chinese Gigaword2, sampling 500k/5k/5k training, validation, and test examples per language.Source: §4 · p. 4, §4.3 · p. 6

4.1 Training Details

• XNLG starts from 15-language XLM, then uses Wikipedia for DAE and MultiUN parallel data for XAE; cross-language downstream tuning updates encoder layers.The implementation also trains a two-language English/Chinese model and a three-language English/French/Chinese model.Source: §4.1 · p. 4

4.2 Question Generation

• In zero-shot QG, XNLG beats baselines automatically on Zh-Zh; human ratings improve in relatedness and correctness for Zh-Zh and En-Zh, and all three metrics for Zh-En.The Zh-Zh and En-Zh tables mark significant improvements at p < 0.05; the Zh-En results report significant gains across all three metrics.Source: Table 3 · p. 4, Table 4 · p. 4, Table 5 · p. 5

4.3 Abstractive Summarization

• XNLG outperforms all baselines on zero-shot French and Chinese summarization, with a larger baseline gap for Chinese.The authors interpret the larger Chinese gap as indicating more serious error propagation on distant language pairs.Source: Table 7 · p. 5, Table 8 · p. 5, §4.3 · p. 6

4.4 Ablation Studies

• For zero-shot Chinese QG, DAE benefits performance, while combining DAE and XAE alleviates spurious correlation and improves cross-lingual NLG.The ablation evaluates pre-training objectives on Zh-Zh-QG.Source: Table 9 · p. 6, §4.4 · p. 6
• Encoder fine-tuning supports English and Chinese QG; tuning all parameters gives the best English score but lowers Chinese performance, while decoder-only tuning underfits both.The authors report that decoder fine-tuning hurts cross-lingual decoding and leads the model to decode only English words.Source: Table 10 · p. 6, §4.4 · p. 6
• Cross-lingual supervision improves few-shot French and Chinese summarization, and its advantage over target-language-only training narrows as target training data grows.The comparison first fine-tunes on English summarization data, then on French or Chinese data.Source: Figure 3 · p. 6, §4.4 · p. 7

4.5 Case Studies

• In generated-question examples, XNLG matches references better than baselines; XLM produces random output on unseen languages, while the pipeline can propagate translation errors.For Zh-Zh-QG, the pipeline translates keywords twice, with “atomic bomb” rendered as “nuclear bomb” in one example.Source: Figure 4 · p. 7, §4.5 · p. 7

• The proposed pre-training method transfers monolingual NLG supervision signals to all pre-trained languages.Source: §5 · p. 7
• Fine-tuning once enables zero-shot cross-lingual NLG on several languages.Source: §5 · p. 7
• The model outperforms machine-translation-based pipeline models on several cross-lingual NLG tasks.Source: §5 · p. 7
• Future work aims to improve the pre-training method toward the fully unsupervised setting.Source: §5 · p. 7