PaperlayerView exact revision on arXiv

Source-linked AI summary

SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings

Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze

arXiv:2004.08728v4cs.CL

TL;DR

Existing word alignment methods commonly require parallel training data, motivating approaches that work from monolingual resources. SimAlign extracts alignments from multilingual static and contextualized embeddings trained without parallel data or dictionaries. The resulting alignments outperform statistical aligners for four language pairs and match them for two, including a 5-point English-German F1 advantage over eflomal trained on 100k parallel sentences.

  • Problem

    Most word alignment approaches require parallel training data, which is scarce in low-resource and domain-specific settings and difficult to replace reliably with mined data.

  • Method

    SimAlign extracts alignments from similarity matrices built from multilingual static or contextualized embeddings trained on monolingual data without parallel supervision or dictionaries.

  • Results

    Alignments from embeddings are superior for four and comparable for two language pairs versus strong statistical aligners; contextualized embeddings score 5 percentage points higher F1 than eflomal for English-German with 100k parallel sentences.

  • Takeaways & Limitations

    Embedding-based alignment can provide strong word alignments without parallel training data, while static embeddings remain a viable alternative when only few parallel sentences are available.

  • Takeaways & Limitations

    Romanian performance may be affected by many-to-one links, absent determiners, and gold-standard annotation guidelines.

Abstract

from arXiv · show

Word alignments are useful for tasks like statistical and neural machine translation (NMT) and cross-lingual annotation projection. Statistical word aligners perform well, as do methods that extract alignments jointly with translations in NMT. However, most approaches require parallel training data, and quality decreases as less training data is available. We propose word alignment methods that require no parallel data. The key idea is to leverage multilingual word embeddings, both static and contextualized, for word alignment. Our multilingual embeddings are created from monolingual data only without relying on any parallel data or dictionaries. We find that alignments created from embeddings are superior for four and comparable for two language pairs compared to those produced by traditional statistical aligners, even with abundant parallel data; e.g., contextualized embeddings achieve a word alignment F1 for English-German that is 5 percentage points higher than eflomal, a high-quality statistical aligner, trained on 100k parallel sentences.

1 Introduction

Word alignments support translation, annotation projection, and related multilingual tasks, but common alignment methods rely on parallel data. SimAlign instead extracts alignments from multilingual embeddings trained on monolingual data, targeting settings where parallel data is unavailable or limited.

  • Word alignments support statistical and neural machine translation, annotation projection, typological analysis, and multilingual embedding creation.
  • Most established statistical and neural alignment approaches rely on parallel training data, whose scarcity limits applicability in low-resource and domain-specific settings.
  • SimAlign extracts word alignments from similarity matrices induced by multilingual static and contextualized embeddings without parallel data.
  • Mining parallel sentences assumes comparable monolingual corpora contain parallel sentences, and large amounts of mined data do not necessarily improve alignment quality.
  • The paper introduces three similarity-matrix alignment methods, extensions for null words and positional information, subword processing for rare words, and the SimAlign tool.

2 Methods

The methods construct a similarity matrix from sentence embeddings and extract binary alignments using Argmax, Itermax, or Match. Extensions incorporate positional locality and null-word detection to adjust alignment behavior.

  • Alignments from Similarity Matrices: The similarity matrix stores normalized embedding similarities between words in two sentences, and alignment extraction produces a binary matrix.
  • Alignments from Similarity Matrices: Argmax aligns each source word with its most similar target word, while Itermax repeatedly applies Argmax to emphasize previously unaligned word pairs.
  • Alignments from Similarity Matrices: Match formulates alignment as a maximum-weight maximal matching in the bipartite graph induced by the similarity matrix.
  • Alignments from Similarity Matrices: Match alignments are inherently bidirectional, so the method does not require postprocessing through symmetrization.
  • Distortion and Null Extensions: Distortion correction multiplies similarities by a positional penalty, favoring words with similar relative positions and using κ = 0.5.
  • Distortion and Null Extensions: The Null extension removes alignment edges when normalized similarity-distribution entropy exceeds a threshold, modeling untranslated words.

3 Experiments

The experiments compare static and contextualized multilingual embeddings with statistical alignment baselines, evaluating word- and subword-level alignments across six English-centered language pairs. Subword predictions are converted to word alignments for evaluation, while baseline systems use parallel training data.

  • Embedding Learning: Static alignments use fastText embeddings trained separately on Wikipedia and mapped into a shared multilingual space without cross-lingual supervision.The same procedure is applied at word and subword levels.
  • Embedding Learning: Contextualized alignments use pretrained mBERT and XLM-R representations, with word embeddings formed by averaging subword vectors.mBERT representations are evaluated across layers and concatenated layers; XLM-R representations are taken from individual layers.
  • Word and Subword Alignments: The study evaluates both word-level and subword-level alignment edges, converting subword alignments to word alignments because gold standards are word-level.The conversion heuristic aligns two words when any of their subwords are aligned, allowing one word to align with multiple others.
  • Baselines: The comparison includes fastalign/IBM2, eflomal, and Giza++/IBM4, all of which require parallel training data.The statistical baselines use established alignment models, with Giza++ trained using standard iterations for HMM and IBM Models 1, 3, and 4.
  • Evaluation Measures: Alignment quality is measured using precision and recall over predicted edges against sure and possible gold-standard edges.The evaluation follows the standard formulation based on predicted alignment set A and gold sets S and P, where S is a subset of P.
  • Experimental Setup: Experiments cover six language pairs involving English: Czech, German, Persian, French, Hindi, and Romanian.The reported language selection spans diverse settings, including morphologically rich and low-resource languages.

4 Results

Embedding-based alignments perform strongly across language pairs and remain competitive with statistical baselines even with substantial parallel training data. Contextualized models, subword processing, and method-specific extensions reveal consistent performance patterns and trade-offs.

  • Embedding Layer: Layer 8 gives the best alignment performance for mBERT and XLM-R, following a parabolic trend across layers.Early layers provide too little contextualization, whereas later layers specialize more strongly in masked language modeling.
  • Comparison with Prior Work: mBERT and XLM-R with Argmax consistently perform well, outperforming eflomal, fast-align, and Giza++ except for Romanian.The methods outperform prior work except on French, where they match prior performance, and Romanian.
  • Comparison with Prior Work: Even with 1.9M parallel sentences, mBERT outperforms fast-align and eflomal for ENG-DEU.Adding up to 37M ParaCrawl sentences slightly improves fast-align but decreases eflomal performance; mBERT-based alignments remain superior in the high-resource setting.
  • Comparison with Prior Work: fastText is competitive with eflomal below 1000 parallel sentences and outperforms fast-align with 10k sentences.For very small parallel corpora, under 10k sentences, fastText embeddings are presented as an alternative to fast-align.
  • Additional Methods and Extensions: Itermax slightly outperforms Argmax overall, while Match has the lowest F1 but generally higher recall.Itermax records 6 wins, 4 losses, and 2 ties, helping more for distant languages but harming French.
  • Additional Methods and Extensions: Null extensions usually increase precision at the cost of recall, while distortion priors help static embeddings but can harm Hindi and have little effect on mBERT.The reported ENG-CES Argmax F1 for static embeddings rises from .70 to .77 with distortion.
  • Words and Subwords: Subword processing slightly outperforms word-level processing for most methods and avoids the severe rare-word performance drop seen at the word level.For rare words, both eflomal and mBERT show strongly reduced word-level F1, but not subword-level F1.
  • Words and Subwords: For auxiliary alignments, eflomal averages an absolute positional distance of 2.72 versus 3.22 for mBERT.The example contrasts eflomal’s incorrect “time”–“hat” alignment with mBERT’s correct “has”–“hat” alignment.

5 Related Work

Prior alignment methods largely rely on parallel data, including statistical aligners, embedding-enhanced models, and neural approaches. Related work also explores attention and representation-based alignments, but these methods generally retain parallel-data requirements.

  • Statistical aligners such as IBM models, Giza++, fast-align, and eflomal are widely used for word alignment.
  • The paper contrasts these parallel-data-dependent approaches with alignments induced from embeddings trained on monolingual data only.
  • Most prior methods that incorporate embeddings, neural networks, or recurrent models require parallel training data.
  • NMT attention provides soft alignment information but often differs from conventional word alignments because it lacks access to the target word.
  • Methods extracting alignments from NMT models generally do not outperform fast-align, while some representation-based approaches require supervised alignment annotations.

6 Conclusion

The paper presents contextualized-embedding aligners that outperform state-of-the-art aligners for four language pairs and match them for two. Static embeddings can provide an alternative when little parallel training data is available, while the methods remain parallel-data-free.

  • Contextualized embeddings outperform state-of-the-art aligners for four language pairs and match their performance for two.
  • Static embeddings are a viable alternative to statistical aligners when few parallel training data are available.
  • The proposed methods require no parallel data for training and support trading precision against recall through Match, Itermax, and Null extensions.
  • Future work includes explicitly modeling fertility and investigating how to incorporate parallel data into the proposed methods.

A Additional Non-central Results

Additional results examine tabled method comparisons, symmetrization, and lower-resource reference settings. GDFA generally yields higher F1 and recall, whereas intersection favors precision.

  • Supplementary Figure 8 repeats Figure 6 with 100k/1000k rather than 1920k parallel sentences and reports similar main takeaways.
  • GDFA performs better than intersection in F1 11 times, while intersection wins four times and three comparisons are tied.
  • Intersection yields higher precision, whereas GDFA yields higher recall; intersection is therefore preferable for annotation projection, while GDFA is typically used in statistical machine translation.
  • Table 7 compares word- and subword-level alignment results, reporting the best overall result per column in bold.
  • Table 8 compares similarity-matrix alignment methods using subword-level results and identifies the best embedding-specific result across columns.

A.4 Alignment Examples for Different Methods

The supplementary material documents alignment examples, frequency-bin analyses, hyperparameter settings, and comparisons across methods and symmetrization choices. It also examines how κ and τ affect alignment performance.

  • Figures 10–13 provide examples illustrating how the alignment methods affect results.
  • Figure 8 groups alignment edges by the minimum frequency of their involved words and compares eflomal results using 100k and 1000k parallel sentences.
  • Table 10 lists customized hyperparameters and records whether they came from defaults, conventional choices, or grid-search selection.
  • Table 9 compares intersection and GDFA symmetrization at the word level, with the best result across rows highlighted for each method.
  • Figure 9 evaluates fastText across κ values and mBERT Match across τ percentiles, with τ offering a precision–recall tradeoff while F1 remains stable.

B.2 Null and Distortion Extensions

The null-word and distortion extensions improve or tune alignment behavior, with κ and τ selected based on observed performance trade-offs. Distortion helps when κ > 0, while lowering τ trades recall for precision without changing F1 substantially.

  • Distortion extension: κ > 0 improves performance, but its exact value is not decisive; the method uses κ = 0.5.Small similarity adjustments can suffice because larger changes may leave the argmax or optimal matching point unchanged.
  • Null-word extension: Lowering τ increases precision and decreases recall, while F1 remains stable.At the 100th percentile, no edges are removed and performance matches the configuration without null-word extension.
  • Null-word extension: The selected null-word threshold is τ at the 95th percentile of the aligned-edge similarity distribution.Threshold values depend on the similarity distribution of all aligned edges.

C Reproducibility Information

The reproducibility setup reports the computational hardware, runtime measurement protocol, and parameter counts for the proposed and pretrained components. Runtime comparisons cover aligning 500 English-German parallel sentences across five runs.

  • Hardware: Computations used up to 48 cores of an Intel Xeon CPU, 1TB memory, and one GeForce GTX 1080 GPU with 8GB memory.mBERT and XLM-R computations were performed on the GPU.
  • Runtime: Runtime was measured for aligning 500 parallel English-German sentences, with averages reported across five runs.The runtime results are provided in Table 12.
  • Runtime: Traditional baselines may require substantially more parallel data for better performance, increasing their runtime.The comparison names fast-align, GIZA++, and eflomal.
  • Parameters: All proposed methods are parameter-free, while pretrained fastText, mBERT, and XLM-R contain approximately 1 billion, 172 million, and 270 million parameters, respectively.The fastText estimate assumes up to 500k words per language, seven languages, and embedding dimension 300.

C.2 Data

The reproducibility materials document the datasets, non-default hyperparameters, and qualitative alignment examples. Figures compare alignment methods using gold-standard sure and possible edges and distinct visual markers for each method.

  • Hyperparameters: Table 10 lists only hyperparameters that differ from default values and reports the values used unless otherwise indicated.This defines the scope of the reproducibility hyperparameter overview.
  • Data: Table 11 provides an overview of the datasets and identifies languages using ISO 639-3 codes.Download links for all used data are provided separately.
  • Alignment examples: Figure 10 marks gold-standard sure and possible edges in dark and light green, respectively, while circles and boxes denote the first and second methods in each subfigure title.The accompanying examples include Argmax versus Itermax and Match versus Match+Null.
  • Alignment examples: Figures 11–13 provide additional alignment examples.Their captions identify them as further examples without specifying additional encodings or outcomes.
Loading 2004.08728v4…