Source-linked AI summary
XDLM: Cross-lingual Diffusion Language Model for Machine Translation
Linyao Chen, Aosong Feng, Boming Yang, Zihui Li
TL;DR
Cross-lingual diffusion pretraining remains understudied, despite prior diffusion work in NLP. XDLM introduces cross-lingual pretraining with a language-mapping objective, and its translation results are superior to certain continuous diffusion models, with an exception, and comparable to a discrete model on WMT14.
Problem
Cross-lingual applications of diffusion models, particularly pretraining, have received little research attention.
Source: §1 · p. 1Method
XDLM uses a cross-lingual denoising pretraining task and objective to model relationships between languages.
Source: §1 · pp. 1–2Results
XDLM performs better than certain continuous diffusion models except Difformer, and comparably to CMLM on WMT14.
Source: §3.3 · p. 4Takeaways & Limitations
XDLM integrates cross-lingual pretraining into diffusion-based text generation, with machine translation as its evaluated application.
Source: §5 · p. 5The RDM comparison used less extensive training due to machine limitations; the expanded BPE vocabulary adds complexity and out-of-vocabulary words, which could contribute to the performance gap.
Source: §4 · pp. 4–5
Abstract
from arXiv · showhide
Recently, diffusion models have excelled in image generation tasks and have also been applied to neural language processing (NLP) for controllable text generation. However, the application of diffusion models in a cross-lingual setting is less unexplored. Additionally, while pretraining with diffusion models has been studied within a single language, the potential of cross-lingual pretraining remains understudied. To address these gaps, we propose XDLM, a novel Cross-lingual diffusion model for machine translation, consisting of pretraining and fine-tuning stages. In the pretraining stage, we propose TLDM, a new training objective for mastering the mapping between different languages; in the fine-tuning stage, we build up the translation system based on the pretrained model. We evaluate the result on several machine translation benchmarks and outperformed both diffusion and Transformer baselines.
1 Introduction
Diffusion models have reached NLP, but cross-lingual diffusion pretraining remains understudied. XDLM addresses this gap with a cross-lingual pretraining architecture and a task designed to model relationships across languages.
1 Introduction
- Prior diffusion NLP work explored text generation, but cross-lingual applications—especially pretraining—remain under-investigated.GENIE uses large English corpora for pretraining before downstream fine-tuning.Source: §1 · p. 1
- Earlier text diffusion approaches include discrete denoising, continuous latent representations, and Transformer-based diffusion pretraining.The cited examples include discrete diffusion (Austin et al., 2021; Hoogeboom et al., 2021), continuous diffusion, and GENIE (Lin et al., 2022).Source: §1 · p. 1
- XDLM introduces an architecture for cross-lingual pretraining in diffusion-based text generation.The authors describe it as the first such architecture to their knowledge.Source: §1 · pp. 1–2
- Translation Diffusion Language Modeling (TDLM) pairs a pretraining task with a loss function to capture contextual correlations across language domains.The paper also discusses potential issues with the task.Source: §1 · p. 2
2 Cross-Lingual Diffusion Language Model
XDLM combines cross-lingual diffusion pretraining with translation fine-tuning. Its TDLM objective uses parallel-language context, while translation is generated non-autoregressively with diffusion.
Non-AutoRegressive (NAR) Machine Translation
- NAR translation predicts target tokens concurrently from the source, using an auxiliary length-prediction loss weighted with translation loss.Length prediction addresses the challenge of determining output sequence length.Source: §2 · p. 2
Diffusion Models
- The diffusion process adds scheduled noise in the forward direction and incrementally reconstructs the target sequence from noise in reverse.The work applies discrete diffusion and a routing mechanism to text generation and cross-lingual translation.Source: §2 · p. 2
Translation Diffusion Language Modeling (TDLM)
- TDLM concatenates source and target sentences with language and position embeddings, then masks 15% of tokens for denoising from cross-lingual context.The model predicts the noise and surrounding text.Source: §2 · p. 3
Translation Model Setting
- Fine-tuning uses the pretrained encoder-decoder for source-to-target translation with language and position embeddings aligned to each domain.Tokens from languages used in pretraining share embeddings with the translation task.Source: §2 · pp. 2–3
3 Experiments
XDLM is evaluated on three translation benchmarks against autoregressive and diffusion baselines. It outperforms certain continuous diffusion models, with Difformer an exception, and is comparable to CMLM on WMT14; BPE evaluation also yields an approximately 5% improvement over word-level evaluation across two datasets.
3.1 Baselines and Datasets
- The evaluation includes Opus-ENDE1, IWSLT14 DE-EN, and WMT14 EN-DE, with dataset statistics reported in Table 1.Source: §3.1 · p. 3, Table 1 · p. 3
- Baselines comprise autoregressive Transformers, continuous diffusion models, and discrete diffusion models.The listed groups include SeqDiffuSeq, DiffuSeq, Difformer, CMLM, and RDM.Source: §3.1 · pp. 3–4
3.2 Experimental Settings
- XDLM uses a six-layer-per-side Transformer encoder-decoder with hidden size 512 and eight attention heads; pretraining and fine-tuning use different learning-rate settings.Pretraining uses weight decay 0.0005, dropout 0.2, 4k-token batches, and 30k warm-up steps; fine-tuning uses a learning rate of 5e-5.Source: §3.2 · p. 4
3.3 Main result
- XDLM performs better than certain continuous diffusion models, except Difformer, and is comparable to CMLM on WMT14.The comparison concerns fine-tuned XDLM across several machine translation tasks.Source: §3.3 · p. 4
- Approximately 5% improvement is reported for BPE-level over word-level BLEU across two datasets.The authors attribute the difference to tokenization levels mitigating problem complexity.Source: §3.3 · p. 4
4 Ablation Study
XDLM’s BLEU scores stabilize after 10 reverse diffusion iterations. The discussion identifies training-scale and vocabulary constraints as possible contributors to performance differences.
Study on the Denoising Capacity at Intermediate Steps
- BLEU scores stabilize after 10 reverse diffusion iterations across the decoding settings discussed.The passage describes Figure 2 as showing BLEU changes with increasing reverse steps.Source: §4 · p. 4
Discussion
- The discussion links the RDM performance gap to unmatched training scale and says expanded-vocabulary BPE adds complexity and out-of-vocabulary words.The authors say machine limitations prevented experiments at the same level and suggest the vocabulary difference could account for some performance differential.Source: §4 · pp. 4–5
Ablation Study
- Table 2 is labeled as model performance, while Table 3 is labeled as BLEU results under different tokenization levels.The supplied passages identify the tables’ scopes but do not provide their cell values.Source: Table 2 · p. 5, Table 3 · p. 5
5 Conclusion and Future Work
The paper presents an architecture integrating cross-lingual pretraining into diffusion-based text generation and compares it with prior work using automated evaluation. It plans to add languages toward broader multilingual translation.
Conclusion
- The proposed architecture integrates cross-lingual pretraining into diffusion-based text generation.The conclusion describes this as being achieved through a carefully designed pretraining task.Source: §5 · p. 5
- The study compares its model with previous works using automated evaluation.Source: §5 · p. 5
Future Work
- The authors plan to add languages to build a robust multilingual model for more extensive cross-lingual translation tasks.Source: §5 · p. 5