Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

• Sentence encoders optimized for global semantic similarity may produce inconsistent or noisy token representations when repurposed for fine-grained alignment.Source: §1 · p. 1
• Word-level alignment can be too restrictive because equivalent meaning is often distributed across multiple tokens or phrases.Source: §1 · p. 1
• SALT injects span-aligned contrastive and translation objectives into existing encoders, alongside interpolation loss to preserve sentence embeddings.Source: §1 · p. 1, Figure 1 · p. 2
• SALT achieves the best overall performance among several fine-tuning strategies on four out of five token-level benchmarks.Source: §1 · pp. 1–2
• SALT improves token representations while preserving or sometimes improving sentence-level embedding quality.Source: §1 · p. 2
• SALT yields better-structured cross-lingual embeddings by improving hubness while preserving isotropy and reducing language-specific clustering.Source: §1 · p. 2

• Cross-lingual sentence encoders learn shared semantic spaces that map equivalent sentences across languages to nearby embeddings.Source: §2 · p. 2
• Earlier alignment research used word and phrase correspondences as translation units, whereas SALT uses span correspondences as a training signal.Source: §2 · p. 2
• Prior auxiliary word- or token-level objectives typically enhanced sentence embeddings, while SALT directly targets token- and span-level representation quality.Source: §2 · p. 3

Method

• SALT is a lightweight post-training method that aligns semantically equivalent spans without modifying the encoder architecture.It extracts span pairs from parallel sentences and refines token and span embeddings directly.Source: SALT: Span-Aligned Learning for… · p. 3, §3.1 · p. 3

Span extraction

• Aligned spans are contiguous token subsequences with similar contextual meaning, represented by pooling their contextual token embeddings.The method applies pooling separately to the source and target subsequences.Source: §3.1 · p. 3
• SALT obtains supervision from either LLaMA-based extraction or the CASE heuristic, which derives aligned spans from token alignments and constituent structure.CASE filters candidate pairs using bidirectional containment and a coverage threshold of c = 0.7.Source: §3.1.1 · p. 3
• Over 90% of human annotations for both LLaMA-based extraction and CASE fall in the top two semantic-equivalence categories.LLaMA performs better on Russian and French, while CASE matches or exceeds it in other languages.Source: §3.1.1 · pp. 3–4, Figure 2 · p. 4

Training objective

• SALT combines span-level contrastive and translation objectives with sentence-level interpolation to improve fine-grained representations while preserving the original sentence space.The translation objective predicts target-span tokens, while interpolation constrains the updated encoder toward the frozen encoder’s sentence representations.Source: §3.1.2 · p. 4

Data and training

• SALT is trained on NLLB Primary parallel data covering 129 language pairs, with 29 held-out test languages available for generalization evaluation.Each retained language pair is downsampled to 40,000 sentence pairs.Source: §4 · p. 5

Baselines

• The study compares XLM-R, XLM-Align, LaBSE, MEXMA, and SONAR fine-tuned with alternative objectives against SALT.Alternative SONAR variants include longer original-loss training and token-level objectives.Source: §4 · p. 5

Sentence evaluation

• Sentence mining uses FLORES-200 devtest across 80 languages, while classification uses English-only MTEB tasks.The reported sentence-mining metrics are xsim and xsim++.Source: §4 · p. 5

Token evaluation

• Token representations are evaluated with word alignment and zero-shot sequence labeling, avoiding reliance on sequence-level tasks that can mask token weaknesses.Sequence labeling trains a linear head on English and evaluates it in other languages.Source: §4.1 · pp. 5–6
• The sequence-labeling suite includes Massive slot filling, PAN-X named-entity recognition, UDPOS part-of-speech tagging, and WiC word-sense disambiguation.WiC uses concatenated contextual embeddings for the target word.Source: §4.1 · p. 6

Sentence-level evaluation

• SALT improves sentence-level quality rather than sacrificing it, consistently outperforming SONAR on cross-lingual sentence mining and improving STS17 while matching SONAR on XNLI and MIntent.Source: §5 · p. 6, Table 3 · p. 7

Span representations

• SALT span representations show higher cosine similarity for aligned spans than SONAR across layers and support high-quality translation with a frozen SONAR decoder.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

Token-level evaluation

• SALT achieves the best or joint-best performance on four of five cross-lingual token-level benchmarks.It outperforms alternative fine-tuning objectives on word alignment, PAN-X, Massive, and WiC.Source: §5 · p. 6, Table 2 · p. 7

Intermediate representations

• SALT’s deeper layers become more cross-lingually aligned for spans and words, shifting the usual pattern in which intermediate layers align best.Source: §5.1 · p. 7

Embedding geometry

• SALT achieves the highest word-retrieval recall and produces embeddings that are harder to classify by language than SONAR and MEXMA.Source: §5.1 · p. 8, Table 4 · p. 8
• SALT avoids the contrasting baseline failures of MEXMA’s anisotropy and SONAR’s hub collapse, yielding a better-distributed cross-lingual token space.SONAR exhibits severe hubness at 11.5 skewness, whereas SALT preserves isotropy while reducing language-specific clustering.Source: Figure 3 · p. 7, §5.1 · p. 8

Ablations

• The full SALT objective achieves the best sequence-tagging score and a highly competitive AER of 0.171 because its three loss terms provide complementary signals.No subset consistently dominates across both word alignment and sequence-tagging metrics.Source: §5.2 · p. 8

Span extraction

• Comparable performance can be achieved with the self-supervised CASE span extractor, indicating that SALT’s training regime is primarily responsible for its success.CASE is marginally better on Massive, while LLaMA-70B supervision is better on PAN-X, UDPOS, and WiC.Source: §5.2 · p. 8, Table 5 · p. 8

• SALT improves cross-lingual token representations through span-aligned supervision.Source: §6 · p. 9
• SALT consistently outperforms alternative fine-tuning strategies and competitive encoders across multilingual token-level benchmarks.Source: §6 · p. 9
• SALT preserves and in several cases improves performance on sentence mining and classification benchmarks.Source: §6 · p. 9
• Span-level supervision is a simple yet powerful inductive bias for multilingual representation learning.Source: §6 · p. 9

• SALT relies on parallel corpora to extract aligned spans, which may limit applicability to truly low-resource languages.Source: Limitations · p. 9
• CASE assumes that the underlying encoder already provides reasonably good cross-lingual token representations.Source: Limitations · p. 9
• Average pooling may fail to capture finer internal structure within longer or compositionally complex spans.Source: Limitations · p. 9
• The study evaluates SALT only as a post-training intervention, leaving span-level supervision during large-scale pre-training unexplored.Source: Limitations · p. 9

• Multilingual representation models may be used in sensitive applications such as surveillance or profiling, motivating responsible deployment and evaluation.Source: Ethics statement · p. 9
• Training data come from publicly available NLLB Primary parallel corpora that may contain societal or cultural biases.Source: Ethics statement · p. 9
• Human span-quality evaluation used voluntary participation, informed consent about withdrawal, anonymised annotator IDs, and institutional ethics approval.Source: Ethics statement · p. 9
• For non-whitespace scripts, tokeniser tokens are treated as words, with word embeddings formed by averaging their composing token embeddings.Source: A A note on segmentation · p. 13
• LLM-extracted spans are mapped back to SONAR tokens, assigning split tokens to the span with greatest string overlap.Source: §B.1 · p. 13
• Experiments use multilingual benchmark splits, frozen-encoder linear probes, bounded examples, sequence-length filtering, and dynamically constructed token-budgeted mini-batches.Source: §C.1 · p. 13, §C.2 · p. 13, §C.3 · p. 13

Training settings

• Sequence tagging is evaluated with English-only, all-language, and target-language-specific classifier training settings.Source: §D.2 · p. 14

Results

• SALT consistently outperforms SONAR across all three sequence tagging training settings.Source: §D.2 · p. 14
• UDPOS shows limited cross-lingual transfer, with single-language training substantially outperforming multilingual training.Source: §D.2 · p. 14

Held-out languages

• On held-out test languages, SALT outperforms the other losses except on UDPOS.Source: §D.3 · p. 14
• Held-out languages benefit from SALT particularly at earlier training stages, while trained languages continue improving over a longer training horizon.Source: §D.3 · p. 14, Figure 6 · p. 15

Figures

• Figure 5 compares SALT and SONAR under the three classifier-training modes, while Figures 6 and 7 show training dynamics and word/span similarity.Source: Figure 5 · p. 15, Figure 6 · p. 15, Figure 7 · p. 16

Cross-lingual retrieval

• SALT more frequently retrieves non-Latin-script tokens and places true translations closer in absolute cosine similarity.This occurs despite SALT having a lower average cosine similarity across the embedding space than the other encoders.Source: §E.3 · p. 16

Contextual retrieval

• SALT reduces retrieval of the same word from different languages in unrelated sentential contexts.The passage identifies this as a common failure mode in cross-lingual retrieval.Source: §E.3 · p. 16

Implications

• The results support using extracted spans as a training signal for cross-lingual representation learning.This conclusion follows from the high rate of acceptable human ratings across languages.Source: §G.1 · p. 16

System comparison

• SONAR with CASE produces performance closely resembling the LLM-based annotation results.The comparison is reported across the human-evaluation score distributions.Source: §G.1 · p. 16