PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment

Mengpeng Yang, Jingxu Yang, Chao Chen, Tian Xia, Yabo Sun, Qiang Liu

arXiv:2608.18474v1cs.CL

TL;DR

Existing aligners often specialize in one granularity and struggle with multilingual, long-text alignment. OmniAlign unifies word- and sentence-level alignment in one lightweight multilingual model, achieving competitive results across both tasks, stable long-text performance, and zero-shot generalization to unseen language pairs.

  • Problem

    Existing aligners generally specialize in either word- or sentence-level alignment, with limited multilingual and long-context capability for unified use.

  • Method

    OmniAlign uses one encoder-only model with contextual token similarity, sentence embeddings, anchor-constrained dynamic programming, and four-stage training for both alignment levels.

  • Results

    OmniAlign achieves competitive word- and sentence-alignment performance, stable long-text word alignment, and strong zero-shot generalization to unseen language pairs.

  • Takeaways & Limitations

    A single shared model can provide reliable multilingual alignment across word and sentence granularities, including long texts and unseen language pairs.

  • Takeaways & Limitations

    The approach assumes encoder-only contextual modeling is advantageous for word alignment and requires robust multilingual capability across languages.

Abstract

from arXiv · show

Cross-lingual sequence alignment is fundamental for building and exploiting parallel corpora, spanning mappings from documents and sentences down to words and subwords. Existing tools, however, typically specialize in a single granularity, so practitioners often need separate systems for word- and sentence-level alignment---especially in multilingual and long-text settings. We present OmniAlign, a unified multilingual aligner that supports both word-level and sentence-level alignment with a single lightweight model. Built on an encoder-only backbone with strong long-context modeling, OmniAlign induces word alignments from contextualized token similarity matrices, and obtains document-level $m$--$n$ sentence alignments via sentence embeddings combined with dynamic programming. To balance fine-grained alignment accuracy and sentence-representation quality, we use a four-stage training pipeline: alignment-oriented continued pre-training, self-supervised learning, supervised fine-tuning on human annotations, and sentence-embedding distillation from a strong multilingual teacher. Experiments show that OmniAlign achieves highly competitive performance on both word- and sentence-alignment benchmarks and generalizes well to unseen language pairs. Surprisingly, later-stage supervised fine-tuning on short texts further improves alignment quality while retaining the long-context understanding acquired in earlier training, keeping the model robust on long-text word alignment. \normalsize {\color{blue}\textbf{Code}: https://github.com/MilkDargon/OmniAlign}\par {\color{blue}\textbf{Model}: https://huggingface.co/WPS-Qingqiu/OmniAlign}

1 Introduction

Cross-lingual sequence alignment spans document-, sentence-, word-, and subword-level correspondences, but existing tools generally specialize in one granularity and lack a practical multilingual, long-context aligner covering both word and sentence alignment. OmniAlign addresses this gap with one lightweight multilingual model that produces both alignment types through shared encoder representations and task-specific procedures.

  • Motivation: Cross-lingual sequence alignment establishes correspondences across languages at document, sentence, word, and subword granularities.These alignment processes support the construction and exploitation of parallel corpora.
  • Limitations of Existing Methods: Existing methods generally target a single alignment granularity, leaving a gap for multilingual, long-context systems that jointly support word- and sentence-level alignment.Sentence-embedding approaches determine sentence alignments but do not provide fine-grained word-level information.
  • OmniAlign: OmniAlign is a unified multilingual aligner that supports word- and sentence-level alignment with one model and shared encoder representations.Task-specific alignment procedures produce the two output types, avoiding the need to maintain separate systems.
  • Contributions: 0.3B parameters: OmniAlign is an open-source model supporting bidirectional word- and sentence-level alignment across eleven major languages with efficient inference.The contribution emphasizes broad multilingual coverage while keeping the model lightweight.
  • Contributions: OmniAlign uses multi-stage training that begins with continued pre-training and combines self-supervised and supervised signals to improve token-level alignment representations.The stated recipe first enhances token-level semantic representations through continued pre-training before subsequent alignment optimization.

2 OmniAlign

OmniAlign uses the multilingual, long-context mGTE encoder to unify word- and sentence-level cross-lingual alignment. Its methods combine bidirectional token-similarity alignment, two-stage sentence dynamic programming, and a four-stage training strategy.

  • Model Backbone: OmniAlign adopts mGTE because encoder-only modeling supports fine-grained token dependencies, multilingual alignment, and longer inputs than traditional 512-token multilingual models.mBERT and the XLM family are described as limited to 512 tokens, whereas mGTE supports significantly longer sequences.
  • Word Alignment: Word alignments are induced by intersecting source-to-target and target-to-source probability matrices computed from contextualized token-embedding similarities.Token-level alignments are converted into word alignments when any token pair belonging to the corresponding words is aligned.
  • Sentence Alignment: Sentence alignment first retrieves top-k candidates, then uses dynamic programming within a diagonal search window to produce document-wide one-to-one anchors.The permitted transitions represent target insertion, source deletion, and one-to-one sentence alignment.
  • Sentence Alignment: Guided by one-to-one anchors, a second dynamic-programming stage searches local regions for flexible m–n alignments using span similarity and a token-length-ratio penalty.The token-level ratio is intended to be more robust than sentence-count constraints for structurally different language pairs such as Chinese and English.
  • Training Strategy: OmniAlign trains through four stages, beginning with continued pre-training and self-supervised optimization to improve alignment quality and stability.Continued pre-training uses parallel sentence variants with masked-token recovery, while self-supervision uses online alignment predictions and emphasizes long-text examples.

3 Experiments · 3.1 Datasets

The experiments use human-annotated word-alignment data across nine language pairs and sentence-alignment test sets across seven language directions. The datasets combine established corpora and evaluation sources, including KFTT and aligner-eval.

  • 3.1 Datasets: Word-alignment experiments cover nine human-annotated language pairs, all involving English.The pairs are de–en, fr–en, ro–en, ja–en, zh–en, es–en, pt–en, ru–en, and it–en.
  • 3.1 Datasets: The Japanese–English word-alignment dataset comes from the Kyoto Free Translation Task word-alignment corpus.Its data splits follow the stated protocol, though the supplied passage is truncated before the protocol details.
  • 3.1 Datasets: Word-alignment dataset statistics report average English-source test-set token counts alongside other dataset statistics.Table 3 defines “Avg. Tokens” as statistics for the English source text in the test set.
  • 3.1 Datasets: Sentence-alignment evaluation spans seven language directions, including Chinese–English, Spanish–English, Italian–English, German–English, French–English, Russian–English, and German–French.These directions broaden evaluation beyond the nine English-centered word-alignment pairs.
  • 3.1 Datasets: The Chinese–English sentence-alignment test set is drawn from the political domain of the aligner-eval dataset.The passage identifies this subset as the political (pol) domain.
  • 3.1 Datasets: Spanish–English, Italian–English, German–English, and French–English sentence-alignment test sets are obtained from Molfese et al. (2024).The supplied passage attributes these four test sets to that source.

3.2 Implementation

OmniAlign is initialized from mGTE6 and trained through continued pre-training, self-supervised learning, supervised word-alignment fine-tuning, and sentence-embedding distillation. The implementation uses staged hyperparameter changes, seventh-layer alignment induction, and partial freezing during distillation.

  • Continued pre-training: OmniAlign starts from mGTE6, a 12-layer Transformer with hidden size 768, and undergoes one epoch of continued pre-training with sequence length 8192.The batch size is 512, the learning rate is 2e−5, and model selection uses 104 held-out validation samples.
  • Self-supervised learning: Self-supervised learning uses the continued-pre-training hyperparameters for one epoch, with 500 validation samples and seventh-layer representations for alignment induction.The alignment threshold is 0.001, following Dou and Neubig (2021).
  • Supervised fine-tuning: Supervised word-alignment fine-tuning runs for five epochs with batch size 64 and learning rate 1e−4, retaining seventh-layer induction and threshold 0.001.The same alignment induction strategy is applied during this stage.
  • Sentence-embedding distillation: Sentence Embedding Knowledge Distillation fine-tunes only the last five Transformer layers for five epochs while freezing the embedding layer and first seven layers.LaBSE7 is the teacher model; the distillation batch size is 256, the validation set contains 1,000 samples, and the learning rate is 2e−5.

3.3 Baselines

OmniAlign is evaluated on both word- and sentence-alignment tasks against established statistical, pretrained-language-model, and sentence-alignment baselines. All sentence-alignment methods use their default configurations.

  • OmniAlign is evaluated on both word alignment and sentence alignment tasks.
  • Word Alignment Baselines: Word-alignment comparisons include statistical FastAlign and GIZA++ alongside pretrained-language-model methods such as SimAlign, Awesome-align, AccAlign, WSPAlign, and BinaryAlign.The statistical systems are implemented under the IBM Model framework.
  • Sentence Alignment Baselines: Sentence-alignment comparisons include Gale–Church, VecAlign, BleuAlign, BertAlign, SentAlign, and CrocoAlign.BleuAlign uses translations generated by WPS iCiba8, and all methods are evaluated under their default configurations.

3.4 Main Results

OmniAlign delivers strong results at both word- and sentence-alignment levels using one shared multilingual model. It ranks highly across word-alignment datasets, while sentence-level performance depends on language pair and alignment method.

  • Word Alignment Results: OmniAlign ranks first on four word-alignment datasets and second on three others across nine language pairs.The single shared checkpoint consistently outperforms comparable methods including SimAlign, Awesome-align, AccAlign, and WSPAlign (Multilingual).
  • Sentence Alignment Results: For sentence alignment, baseline sentence encoders uniformly use LaBSE, which also provides OmniAlign’s sentence representations through knowledge distillation.The comparison is designed to use the same underlying sentence representation model across encoder-based baselines.
  • Sentence Alignment Results: BertAlign achieves the best scores on en–it, en–ru, and de–fr when paired with the LaBSE encoder.Its advanced alignment algorithm gives it the strongest results on these three language pairs.
  • Sentence Alignment Results: BertAlign performs comparatively worse on en–es, en–de, and en–fr because its modified cosine similarity favors many-to-many alignments.The language-pair differences show that alignment behavior affects sentence-level results beyond the shared LaBSE representation.

3.5 Ablation Study

The ablations show that successive training stages substantially improve word alignment, while sentence alignment remains comparable to LaBSE. OmniAlign also preserves long-context robustness and generalizes across unseen language pairs through its shared multilingual encoder.

  • Subtractive Training Stages: Word-alignment AER decreases from 43.0% to 19.6%, 14.3%, and 8.5% as successive training stages are added.Each stage contributes positively to word-level alignment, although removing S.1 or S.2 does not significantly degrade the final evaluation metric.
  • Subtractive Training Stages: Sentence alignment is comparable to LaBSE and surpasses it on the EN–ES test set when only the last five layers are fine-tuned with fixed dynamic programming.The setup aligns multilingual sentence representations into LaBSE’s English embedding space.
  • Embedding Extraction Across Layers: mGTE-MLM-Base underperforms mBERT for word alignment, but supports a native maximum sequence length of 8192 tokens versus mBERT’s 512.This long-context advantage motivates task-oriented continued pre-training to improve word alignment while preserving long-context modeling.
  • Long-Text Word Alignment: OmniAlign’s zh–en AER rises only from 8.5 on zh–en–1 to 12.6 on zh–en–50 as input length increases.Previous methods degrade substantially and are typically limited to 512-token inputs; zh–en–50 reaches 1,850 tokens.
  • Zero-Shot Generalization: On unseen language pairs, OmniAlign matches or outperforms baselines on most pairs without additional training on the target languages.The authors attribute this generalization to a single shared multilingual encoder trained with multilingual parallel data and subsequent fine-tuning.

3.6 Case Study

The case studies show OmniAlign handling challenging word- and sentence-alignment conditions more effectively and interpretably than comparison systems. It recovers difficult correspondences, remains usable on technical multi-token text, and produces compact document-level segmentation.

  • Scenario 1: Word Alignment: OmniAlign better handles difficult word correspondences, including mapping “failure” to the Chinese negated expression “没有成功.”Models without continued pre-training often align only the positive lexical item and miss the negated expression.
  • Scenario 1: Word Alignment: On long technical paragraphs containing many multi-token named entities and domain phrases, BinaryAlign fails to produce usable outputs while OmniAlign recovers highlighted correspondences.The case study contrasts unusable BinaryAlign output with OmniAlign’s recovered alignments under this challenging setting.
  • Scenario 2: Sentence Alignment: For document-level sentence alignment, OmniAlign produces more compact and interpretable m–n units than BertAlign’s frequent broad merges of neighboring sentences.OmniAlign uses anchor-constrained dynamic programming with a token-length–based penalty, whereas BertAlign often merges sentences even when finer or constrained links are preferable.
  • Scenario 2: Sentence Alignment: In another example, OmniAlign uses overlapping many-to-many links, whereas BertAlign merges both English sentences into one three-sentence Chinese block.The reported alignments are OmniAlign [(1)]:[(1), (2)]; [(2)]:[(2), (3)] and BertAlign [(1), (2)]:[(1), (2), (3)].

4 Related work

Cross-lingual sequence alignment supports parallel-corpus construction and multilingual applications, but existing methods are typically specialized by granularity. Word alignment relies on statistical and neural approaches, while sentence/document alignment uses length-based, translation-assisted, and embedding-based methods.

  • Cross-lingual sequence alignment provides essential signals for parallel-corpus construction and downstream multilingual applications, yet prior work often targets only one granularity.
  • Word alignment: Classical word-alignment baselines include GIZA++ and FastAlign, alongside recent approaches.
  • Sentence/document alignment: Sentence/document alignment includes length-based Gale–Church, translation-assisted BleuAlign, and embedding-based systems such as VecAlign, BertAlign, SentAlign, and CrocoAlign.

5 Conclusion

OmniAlign is a lightweight unified multilingual aligner for word- and sentence-level alignment, combining contextualized representations with anchor-constrained dynamic programming. Its four-stage training pipeline balances fine-grained alignment accuracy and sentence-representation quality, yielding highly competitive performance at both alignment levels.

  • Unified aligner: OmniAlign unifies word- and sentence-level alignment in a single lightweight multilingual model.The model supports both alignment granularities within one system.
  • Model architecture: Its encoder-only architecture uses strong long-context modeling and induces token- and word-level alignments from contextualized representations.This design supports alignment from contextualized model outputs.
  • Sentence alignment: For document-level sentence alignment, OmniAlign integrates a two-stage, anchor-constrained dynamic programming algorithm.The algorithm is specifically used for document-level sentence alignment.
  • Training pipeline: OmniAlign uses a four-stage training pipeline combining continued pre-training, self-supervised learning, supervised fine-tuning, and sentence-embedding knowledge distillation.Supervised fine-tuning uses human-annotated data, while distillation uses a strong multilingual teacher model.
  • Results: Extensive experiments show highly competitive performance at both word- and sentence-alignment levels.The reported evaluation covers both alignment granularities.
Loading 2608.18474v1…