PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Unsupervised Word Alignment to Documents via Structural Constraints

Michelle Wastl, Jannis Vamvas, Rico Sennrich

arXiv:2608.21023v1cs.CL

TL;DR

Document-level word alignment is needed beyond sentence pairs, but directly applying sentence-level methods degrades as similarity matrices become larger and noisier. The paper introduces training-free MDPAlign and CTFAlign, which constrain alignment search using positional or adaptive coarse-to-fine structure. Across language pairs and models, CTFAlign provides the most robust performance and transfers gains to downstream evaluations.

  • Problem

    Existing word alignment methods largely operate on sentence pairs despite cross-lingual tasks increasingly requiring correspondences across full documents.

  • Method

    The paper introduces MDPAlign’s global diagonal prior and CTFAlign’s recursive restriction of alignment search to semantically plausible regions.

  • Results

    CTFAlign+Argmax reduces average AER from 0.341 to 0.248 with LaBSE across six language pairs and reaches 0.294 versus 0.295 for Qwen document- versus sentence-level alignment.

  • Takeaways & Limitations

    Both methods reduce sentence-to-document degradation, while CTFAlign is most robust across diverse document structures and language pairs and improves downstream evaluations.

  • Takeaways & Limitations

    The word-alignment gold data was tailored to sentences and had to be reconstructed for documents; insufficiently distinctive representations can also limit structural constraints and allow CTFAlign to propagate earlier errors.

Abstract

from arXiv · show

Word alignment has traditionally been studied between sentences, but many cross-lingual tasks increasingly require correspondences across full documents. While recent multilingual embedding models can encode long inputs, we show that applying algorithms designed for sentences directly to documents leads to performance degradation. To address this, we introduce CTFAlign, a lightweight, training-free approach for document-level word alignment. CTFAlign applies a coarse-to-fine refinement strategy that restricts the alignment search space to semantically similar regions. Additionally, we introduce MDPAlign, a simpler alternative that constrains alignments by position with a main diagonal prior. Both approaches operate directly on full documents without relying on sentence segmentation or sentence alignment. We evaluate these methods across six language pairs varying in typological distance, resourcedness, and document length. Averaged over three models, CTFAlign reduces word alignment error rate from 0.412 to 0.326. These gains transfer downstream, leading to improvements in document-level translation coverage evaluation and recognition of semantic differences. We release CTFAlign as a Python package and make the code and data to reproduce our experiments publicly available.

1 Introduction

Document-level word alignment is needed for increasingly document-scale cross-lingual tasks, but directly applying sentence-level methods degrades performance. The paper introduces lightweight approaches that constrain document-level search and improve downstream evaluations.

  • CTFAlign is closest to the gold alignment in an English–Japanese Wikipedia example, whereas MDPAlign misses a late-document bend and SimAlign produces implausible false positives.
  • Document-scale cross-lingual tasks increasingly require word correspondences across full documents, while existing alignment methods largely operate on sentence pairs.
  • Directly applying sentence-level alignment to documents degrades performance because larger similarity matrices become noisier.
  • The paper introduces MDPAlign and CTFAlign as two lightweight, model-agnostic methods for document-level word alignment.
  • The approaches reduce degradation from scaling sentence-level alignment to documents, with CTFAlign matching sentence-level performance in several settings.
  • Document-level alignment improvements transfer to translation coverage evaluation and token-level recognition of semantic differences.

2 Background & Related Work

Prior work centers on sentence-level alignment and often requires segmentation or sentence alignment for longer texts. MDPAlign extends positional priors to documents, while CTFAlign applies token-level coarse-to-fine refinement.

  • Most neural and embedding-based word alignment methods focus on sentence pairs, including unsupervised, supervised, sequence-labeling, and fine-tuned approaches.
  • Document-level word alignment remains underexplored, and longer-text pipelines commonly segment and align sentences or chunks before word alignment.
  • Such segmentation and sentence-alignment dependencies can be brittle and may discard document-level information or introduce cascading errors.
  • MDPAlign extends positional alignment assumptions to documents by imposing a global diagonal prior.
  • MDPAlign’s variants constrain search toward the main diagonal, with one strict variant using bandwidth k = 50.
  • Unlike sentence-level VecAlign, CTFAlign performs coarse-to-fine refinement directly on token-level similarity regions.

3 Methods

The methods constrain noisy document-level similarity search before applying embedding-based alignment. MDPAlign uses positional masking, while CTFAlign recursively retains semantically plausible regions and refines them to token resolution.

  • Document-level Word Alignment: Document-level alignment links semantically corresponding tokens across two documents, but long documents create noisier, less locally constrained search spaces.
  • Embedding-based Alignment: Embedding-based alignment computes a token-level cosine-similarity matrix and predicts links with algorithms such as Argmax or Itermax.
  • Structural Constraints: The central design constrains the similarity matrix before alignment to reduce the effects of an unconstrained document-level search space.
  • Main Diagonal Prior: MDPAlign maps tokens to relative document positions and applies diagonal-band masking, assuming corresponding content occupies approximately similar global positions.
  • Coarse-to-Fine Alignment: CTFAlign adapts to curved or shifted alignment structures by recursively restricting search to likely regions rather than enforcing a fixed diagonal.
  • Main Diagonal Prior: MDPAlign’s strict variant removes entries outside a diagonal band, while its fuzzy variant preserves all cells but applies exponential off-diagonal suppression.
  • Coarse-to-Fine Alignment: CTFAlign starts with a 2 × 2 average-pooled grid, retains aligned coarse blocks, masks other regions, and repeatedly halves block size until token resolution.
  • Coarse-to-Fine Alignment: Neighbor buffers and recovery of isolated row–column intersections improve robustness against errors made at coarse resolutions.

4 Experimental Setup

The experiments reconstruct document-level evaluation data, encode document pairs with long-context multilingual models, and compare constrained and unconstrained alignment baselines using AER.

  • Data: Evaluation reconstructs documents from sentence-level word-alignment datasets containing document membership and sentence order.The dataset covers English–French, English–Romanian, English–Japanese, English–Chinese, and English–Czech pairs in the supplied passage.
  • Models: Qwen3-Embedding-4B and mmBERT-base encode entire document pairs in a single forward pass.Qwen3-Embedding-4B supports 40,960 tokens, while mmBERT-base supports 8,192 tokens.
  • Baselines: SimAlign with Argmax is the baseline at both sentence and document levels.The unconstrained document-level setting measures document-scale alignment effects, while sentence-level performance provides an approximate upper bound.
  • Baselines: An LLM baseline autoregressively generates word-alignment labels from a simple one-shot prompt without fine-tuning.The marker-mode prompt was excluded because it would require as many passes as the documents contain words, making it infeasible for long documents.
  • Implementation Details: Development-set tuning selects MDPAlign fuzzy with k = 150 and CTFAlign with w = 8 for test-set evaluation.Both strict and fuzzy MDPAlign variants are evaluated, and the parameters k and w are searched on development data.
  • Evaluation: The evaluation reports test-set alignment error rate for Qwen3-Embedding-4B, mmBERT-base, and LaBSE.Additional metrics include F1, Precision, Recall, and corresponding macro averages.

5 Results & Discussion

Direct document-level alignment degrades sharply, especially for long documents, but structural constraints recover much of the lost quality. CTFAlign is generally robust to varied alignment structures, while MDPAlign depends on diagonal alignment and encoder characteristics.

  • Unconstrained document-level alignment: Directly applying SimAlign to full documents degrades alignment quality across models and language pairs, with especially severe effects on long documents.For mmBERT, document-level AER exceeds 0.8 on en–ja and la–gr; spurious links often involve repeated named entities, function words, and punctuation.
  • Unconstrained document-level alignment: GPT-5.4-mini’s alignment AER collapses to near-1.0 across all six language pairs when given full documents.The model remains competitive on some sentence-level pairs, but autoregressive label generation does not scale without document-level alignment fine-tuning.
  • Structural constraints: Structural constraints recover most of the document-level performance loss, narrowing the sentence–document gap to a few thousandths AER in several settings.Examples include en–ro with LaBSE+MDPAlign+Itermax, at 0.222 document-level versus 0.220 sentence-level, and en–zh with mmBERT+MDPAlign+Argmax, at 0.200 versus 0.192.
  • MDPAlign: MDPAlign improves alignment when paths remain near the global diagonal but can underperform the unconstrained baseline on curved paths such as en–ja.On en–ja, Qwen+MDPAlign yields 0.752/0.728 versus 0.602 for SimAlign.
  • CTFAlign: CTFAlign adapts to curved or shifted alignment regions and achieves strong results where a fixed diagonal prior is insufficient.It reaches 0.481 AER on en–ja with LaBSE, 0.293 versus 0.292 on la–gr with Qwen, and 0.068 on en–fr with LaBSE.
  • CTFAlign: CTFAlign+Argmax is the most robust configuration across six language pairs, reducing average AER from 0.341 to 0.248 with LaBSE and closing Qwen’s sentence–document gap.With Qwen, average document- and sentence-level AER are 0.294 and 0.295, respectively.
  • Encoder interactions: Structural constraints interact with encoder characteristics: MDPAlign outperforms CTFAlign on average with mmBERT, while CTFAlign performs best with LaBSE and Qwen.For Argmax variants with mmBERT, average AER is 0.437 for CTFAlign versus 0.387 for MDPAlign; recursive coarse-level errors can propagate in CTFAlign.

6 Downstream Task Evaluation

The authors test structural constraints in translation coverage evaluation and semantic-difference recognition. CTFAlign generally transfers the strongest alignment gains to both downstream tasks.

  • Translation Coverage Evaluation: DiffAlignCTF+Argmax is strongest on both language pairs and both error types at the document level.Gains are +0.055 for omissions and +0.008 for additions on ja–zh, and +0.036 for omissions and +0.02 for additions on en–de.
  • Translation Coverage Evaluation: The document-level advantage of CTFAlign increases as structural complexity increases, whereas MDPAlign remains effective for shorter, more rigid segments.The segment-to-document degradation is smaller here than in intrinsic alignment evaluation because segments span multiple sentences.
  • Recognition of Semantic Differences: Search-space restriction improves semantic-difference recognition over DiffAlign across en–de, en–fr, and en–it.Improvements are +0.044, +0.033, and +0.035 respectively, measured by Spearman correlations with gold token-level scores.
  • Recognition of Semantic Differences: DiffAlignCTF+Argmax yields the highest SwissGov-RSD correlations and establishes new state-of-the-art results, while MDPAlign offers little improvement.MDPAlign underperforms the baseline on en–it.
  • Cross-Task Findings: CTFAlign+Argmax remains the best-performing configuration across both downstream tasks.The authors interpret this as evidence that its gains transfer to realistic alignment-based applications.

7 Conclusion

The paper introduces lightweight, training-free constraints for document-level word alignment. Across language pairs and document structures, CTFAlign is the most robust and its improvements transfer to downstream document comparison tasks.

  • Contributions: The paper introduces MDPAlign and CTFAlign as lightweight, training-free approaches for document-level word alignment.MDPAlign imposes a global positional prior, whereas CTFAlign recursively restricts alignment to semantically plausible regions.
  • Contributions: Across six language pairs, both methods reduce the degradation caused by moving from sentence-level to full-document alignment.CTFAlign provides the most robust performance across diverse document structures and language pairs.
  • Downstream Transfer: CTFAlign improves translation coverage evaluation and semantic-difference recognition beyond intrinsic alignment evaluation.These downstream gains require neither sentence segmentation nor additional supervision or model retraining.

Limitations

The evaluation is limited by reconstructed sentence-level alignment data and dependence on encoder representations. These constraints leave document-level phenomena and representation failures only partially addressed.

  • Dataset Limitations: The word-alignment gold data was reconstructed from sentence-level annotations rather than collected directly for documents.This can leave document structures largely monotonic and omit long-range reordering or cross-sentence phenomena.
  • Dataset Limitations: Sentence-level annotations may miss long-range reordering, cross-sentence dependencies, discourse-driven shifts, and multi-sentence omissions or additions.These phenomena are not fully represented by sentence-level supervision.
  • Dataset Limitations: Concatenation produces few evaluation document pairs, including only one en–fr pair and one la–gr pair.This restricts the breadth of evaluation for those language pairs.
  • Dataset Limitations: Downstream evaluations partly mitigate the reconstructed-data limitation because SwissGov-RSD is annotated directly at the document level.WMT annotators also received broader contextual information, although its data was reconstructed from segment-level annotations.
  • Representation Dependence: The methods depend on the underlying multilingual encoder producing sufficiently distinctive cross-lingual token similarities.Structural constraints cannot fully compensate for weak representations, and CTFAlign may propagate earlier errors through iterative refinement.

Ethical Considerations

The supplied material reports no specific ethical risks and describes the artifacts as generally openly licensed. It also includes experimental and implementation details from the appendices.

  • Ethical Considerations: No specific ethical risks were identified for this work.The authors state that the datasets are unlikely to contain personally identifiable information or offensive content.
  • Ethical Considerations: The artifacts used in the work, except for the proprietary LLM, have open licenses.The statement concerns the artifacts’ licensing and the reported origin of the datasets.
  • Experimental Materials: The experiments include reconstructed document-level alignment datasets spanning varied document lengths and split sizes.Table 4 covers short texts such as en–zh and very long documents such as en–fr and la–gr.
  • Implementation Details: The appendix reports that document-level layer selection can differ substantially from sentence-level selection.The gap between the best and last layer exceeds 25 F1 points in the reported layer search.
  • Implementation Details: Applying unconstrained SimAlign directly to documents causes degradation, especially in recall.MDPAlign and CTFAlign improve recall by restricting the search space and suppressing spurious long-range matches.

F SwissGov-RSD Development Set Results

Structural constraints consistently improve document-level semantic difference detection over unconstrained DiffAlign on SwissGov-RSD. DiffAlignCTF+Argmax is the strongest configuration, with gains especially pronounced at layer 20.

  • Structural constraints consistently improve over unconstrained DiffAlign across all SwissGov-RSD language pairs.This pattern indicates that restricting the search space benefits document-level semantic difference detection.
  • DiffAlignCTF+Argmax achieves the strongest overall performance across the evaluated language pairs.
  • The gains are especially pronounced at layer 20, the average best layer for document-level word alignment across languages.
  • The final layer performs substantially worse than layer 20 across methods, consistent with multilingual embedding layers often being suboptimal for cross-lingual similarity.

G Main Results with Precision, Recall and F1

The main results compare alignment strategies using precision, recall, and F1 alongside AER across language pairs, models, and structural-constraint settings. Argmax generally favors precision and F1, while Itermax often recovers recall at a precision cost, but can produce spurious regions on noisy similarity matrices.

  • Figure 7 reports F1 across layers, language pairs, granularities, and three embedding models using Argmax on the development set.
  • Argmax-based variants favor precision and more often achieve the strongest F1 scores across MDPAlign and CTFAlign.
  • Itermax generally improves recall by recovering additional alignments, at the cost of precision, across both alignment approaches.
  • Figure 8 compares AER and F1 across k/w values for Qwen3-Embedding-4B, mmBERT-base, and LaBSE-concatenated-sentences.
  • CTFAlign+Itermax can create spurious alignment regions when applied to noisy similarity matrices.The failure is attributed to falsely aligned coarse similarity blocks at lower resolution.
  • Figure 9 contrasts CTFAlign+Itermax failure cases with a partially recovered CTFAlign+Argmax alternative on en–fr using mmBERT.
Loading 2608.21023v1…