Source-linked AI summary
Word Alignment by Fine-tuning Embeddings on Parallel Corpora
Zi-Yi Dou, Graham Neubig
TL;DR
Word alignment is broadly useful, yet prior statistical and neural approaches either rely on older word-based models, treat languages asymmetrically, or cannot easily use multilingual contextualized embeddings. The paper fine-tunes pretrained multilingual language models on parallel text with alignment-oriented objectives and extracts alignments through probability thresholding or optimal transport. The resulting aligner achieves state-of-the-art performance across five language pairs and robust zero-shot performance.
Problem
Word alignment needs methods that combine the strengths of parallel-corpus training with large-scale multilingual contextualized embeddings.
Method
The paper fine-tunes pretrained multilingual language models on parallel text with self-training and parallel-sentence identification objectives, then extracts alignments using probability thresholding or optimal transport.
Results
The aligner achieves state-of-the-art performance on five diverse language pairs and robust performance in zero-shot settings.
Takeaways & Limitations
Multilingual word aligners can provide robust out-of-the-box performance across different language pairs, including zero-shot settings.
Takeaways & Limitations
Evaluation mainly uses AER, which may not correlate well with statistical machine-translation performance and may not suit every task or condition.
Abstract
from arXiv · showhide
Word alignment over parallel corpora has a wide variety of applications, including learning translation lexicons, cross-lingual transfer of language processing tools, and automatic evaluation or analysis of translation outputs. The great majority of past work on word alignment has worked by performing unsupervised learning on parallel texts. Recently, however, other work has demonstrated that pre-trained contextualized word embeddings derived from multilingually trained language models (LMs) prove an attractive alternative, achieving competitive results on the word alignment task even in the absence of explicit training on parallel data. In this paper, we examine methods to marry the two approaches: leveraging pre-trained LMs but fine-tuning them on parallel text with objectives designed to improve alignment quality, and proposing methods to effectively extract alignments from these fine-tuned models. We perform experiments on five language pairs and demonstrate that our model can consistently outperform previous state-of-the-art models of all varieties. In addition, we demonstrate that we are able to train multilingual word aligners that can obtain robust performance on different language pairs. Our aligner, AWESOME (Aligning Word Embedding Spaces of Multilingual Encoders), with pre-trained models is available at https://github.com/neulab/awesome-align
1 Introduction
Word alignment supports many NLP and machine-translation applications, but established statistical and neural approaches have important limitations. This paper combines multilingual pretrained language models with parallel-text fine-tuning and alignment extraction methods, achieving strong performance across language pairs and robust zero-shot alignment.
- Word alignment supports translation lexicon learning, cross-lingual transfer, semantic parsing, speech recognition, and machine-translation methods.
- Traditional aligners based on IBM word-based translation models remain popular despite advances in neural NLP.
- Neural machine-translation aligners are directional and cannot easily exploit large-scale contextualized multilingual embeddings.
- The paper fine-tunes pretrained multilingual language models on parallel text using language-model, self-training, and parallel-sentence identification objectives.The self-training objective brings aligned words’ contextualized representations closer, while sentence identification brings parallel sentence representations closer.
- The proposed aligner extracts alignments using probability thresholding or optimal transport and achieves state-of-the-art performance on five language pairs.The authors also report more aligned contextualized representations after fine-tuning and robust zero-shot performance from multilingual aligners.
2 Methods
The method extracts bidirectional word alignments from multilingual contextualized embeddings, then fine-tunes the encoder on parallel text with objectives that improve alignment consistency and quality.
- Extracting Alignments from Embeddings: Contextualized embeddings are extracted from selected hidden states for each parallel sentence pair before alignment scores are computed.The model produces source and target vectors for words in context.
- Extracting Alignments from Embeddings: Probability thresholding normalizes embedding similarities into probability-simplex values and selects word pairs exceeding a threshold.Softmax and sparse α-entmax are possible normalizers; larger α produces sparser alignments.
- Extracting Alignments from Embeddings: Optimal transport treats parallel sentences as uniformly weighted point sets and finds a low-cost transition matrix from pairwise embedding distances.The resulting transition matrix is self-normalized and sparse, making it suitable for extracting alignments.
- Extracting Alignments from Embeddings: Final alignments intersect source-to-target and target-to-source matrices, marking a pair aligned only when both directional scores exceed the threshold.Growing heuristics such as grow-diag-final can also be applied.
- Extracting Alignments from Embeddings: Subword alignments become word alignments when any subword of one word aligns with any subword of the other.This conversion follows earlier work and supports word-level extraction from language-model embeddings.
- Fine-tuning Contextualized Embeddings: Fine-tuning combines masked and translation language modeling with self-training, parallel sentence identification, and consistency optimization over parallel data.Self-training pulls initially aligned contextualized representations closer, while consistency optimization explicitly encourages agreement between directional matrices.
3 Experiments
Experiments across five language pairs evaluate bilingual and multilingual aligners, extraction methods, language models, training objectives, supervision, and transfer settings. The proposed models generally improve alignment performance, while some objectives and heuristics have setting-dependent effects.
- Experimental Setup: Five language pairs—De-En, Fr-En, Ro-En, Ja-En, and Zh-En—are used to evaluate bilingual and multilingual word aligners.The study also includes zero-shot, supervised-signal, annotation-projection, and sentence-level transfer analyses.
- Bilingual Model Performance: Fine-tuned softmax models consistently improve over baseline aligners, while vanilla mBERT already outperforms GIZA++ in 4 of 5 settings.The vanilla mBERT comparison is especially strong for Zh-En, where parallel-data size is small.
- Multilingual Model Performance: Multilingual training further improves performance and consistently beats bilingual models and all baselines despite using fewer training examples per language pair.The multilingual aligners are reported to provide state-of-the-art and robust performance across language pairs.
- Multilingual Model Performance: Consistency optimization significantly helps Ro-En but hurts Ja-En and Zh-En because it raises recall while sacrificing precision.The datasets favor different recall–precision trade-offs.
- Extraction Methods: Probability thresholding consistently outperforms optimal transport on all five language pairs and is much faster; softmax is marginally better than α-entmax.α-entmax avoids manually setting the threshold, so both methods are used for the main results.
- Training Objectives: The self-training objective produces the largest improvement, whereas TLM and parallel sentence identification help marginally and MLM can sometimes reduce performance.The ablation concerns multilingual settings.
4 Related Work
Prior work spans statistical, neural, contextualized, supervised, and unsupervised approaches to word alignment, but each has important limitations or accessibility constraints.
- Statistical word aligners based on IBM translation models include widely used tools such as GIZA++ and fast align.
- NMT-based aligners can outperform statistical tools, but their directional architectures conflict with word alignment’s non-directional nature.
- MT-based aligners also cannot easily exploit large-scale contextualized embeddings from multilingual language models.
- Multilingual contextualized embeddings have enabled effective alignment extraction without explicit parallel-data training.
- Supervised neural aligners use alignment labels, but limited access to supervised data restricts their applicability.
5 Discussion and Conclusion
The paper presents a neural word aligner that fine-tunes multilingual embeddings and achieves strong performance across language pairs, including zero-shot settings.
- AWESOME combines multilingual embedding fine-tuning, alignment-specific objectives, and two alignment extraction methods.
- State-of-the-art performance is achieved on five diverse language pairs.
- Robust zero-shot performance supports using multilingual word aligners out-of-the-box across varied language pairs.
- The aligner can incorporate supervised signals in semi-supervised settings.
- The primary evaluation relies on AER, which may not correlate well with statistical machine translation performance or suit every task.
A Implementation Details
The implementation uses AdamW with fixed training and extraction settings, trains for one epoch, and evaluates alignment quality using AER.
- The models use AdamW with a learning rate of 2e-5 and batch size 8.
- The α-entmax parameter α is set to 1.5.
- The threshold c is 0 for α-entmax and 0.001 for softmax and optimal transport.
- Models are trained for one epoch on a single 2080 Ti, with convergence taking 3 to 24 hours depending on dataset size.
- Performance is evaluated using Alignment Error Rate (AER).
B Analysis
The analysis examines model behavior across settings, extraction methods, objectives, thresholds, language pairs, and qualitative alignment examples.
- Monolingual Alignment: mBERT can outperform previous monolingual alignment methods in recall and F1 without fine-tuning, while fine-tuning improves recall and supervised signals improve results further.
- Sensitivity Analysis: AER changes by at most 0.5% when the softmax threshold varies after fine-tuning.
- Comparisons with IterMax: Compared with IterMax, the model produces more accurate alignments, whereas IterMax trades precision for a small recall improvement.
- Ablation Studies on Training Objectives: The self-training objective is most effective, followed by translation language modeling and parallel sentence identification; masked language modeling can hurt performance.
- Experiments on More Language Pairs: The appendix reports experiments on additional language pairs without fine-tuning.
- More Qualitative Examples: Qualitative examples show more aligned contextualized representations and more accurate extracted alignments after fine-tuning.
- The English-Persian dataset used in related analysis was unavailable when the paper was written.