PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

Guillem Ramírez

arXiv:2609.09953v1cs.CLcs.AI

TL;DR

Sentence encoders trained for cross-lingual sentence alignment are increasingly used for token-level tasks, exposing a mismatch between their training objective and downstream use. SALT injects span-level supervision through lightweight post-training of existing encoders. It achieves the strongest overall results on four of five token-level benchmarks while preserving or improving sentence-level performance.

  • Problem

    Sentence encoders optimized for sentence alignment are increasingly used for token-level applications, where their token representations can be inconsistent or noisy.

  • Method

    SALT applies span-aligned contrastive and translation objectives to existing encoders, with interpolation loss preserving the original sentence embedding space.

  • Results

    SALT achieves the strongest overall performance on four of five token-level benchmarks while preserving and sometimes improving sentence-level quality.

  • Takeaways & Limitations

    Span-level supervision is a simple inductive bias that improves cross-lingual token representations without sacrificing sentence-level quality.

  • Takeaways & Limitations

    SALT relies on parallel corpora and a heuristic assuming reasonably good underlying cross-lingual token representations, limiting adaptation to truly low-resource or severely underrepresented languages.

Abstract

from arXiv · show

Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment, they are increasingly also applied to token-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage. We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders. Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine-tuning strategies and competitive encoders. It also improves sentence-level performance on cross-lingual retrieval and classification tasks. These results demonstrate that span-level supervision is an effective signal for improving both token and sentence representations.

1 Introduction

Cross-lingual sentence encoders are increasingly used for token-level tasks despite being optimized for sentence alignment, creating noisy token representations. SALT addresses this mismatch with span-level supervision while preserving sentence-level embeddings.

  • Sentence encoders optimized for global semantic similarity may produce inconsistent or noisy token representations when repurposed for fine-grained alignment.
  • Word-level alignment can be too restrictive because equivalent meaning is often distributed across multiple tokens or phrases.
  • SALT injects span-aligned contrastive and translation objectives into existing encoders, alongside interpolation loss to preserve sentence embeddings.
  • SALT achieves the best overall performance among several fine-tuning strategies on four out of five token-level benchmarks.
  • SALT improves token representations while preserving or sometimes improving sentence-level embedding quality.
  • SALT yields better-structured cross-lingual embeddings by improving hubness while preserving isotropy and reducing language-specific clustering.

2 Related work

Prior work established cross-lingual sentence and phrase alignment, while auxiliary token objectives sought to improve sentence encoders. SALT instead directly targets token- and span-level representations during post-training and evaluation.

  • Cross-lingual sentence encoders learn shared semantic spaces that map equivalent sentences across languages to nearby embeddings.
  • Earlier alignment research used word and phrase correspondences as translation units, whereas SALT uses span correspondences as a training signal.
  • Prior auxiliary word- or token-level objectives typically enhanced sentence embeddings, while SALT directly targets token- and span-level representation quality.

SALT: Span-Aligned Learning for Cross-Lingual Tokens

SALT improves token and span representations by adding span-level supervision to pretrained cross-lingual sentence encoders while preserving sentence-level structure. It uses extracted aligned spans, multiple span objectives, and interpolation to retain the original embedding space.

  • Method: SALT is a lightweight post-training method that aligns semantically equivalent spans without modifying the encoder architecture.It extracts span pairs from parallel sentences and refines token and span embeddings directly.
  • Span extraction: Aligned spans are contiguous token subsequences with similar contextual meaning, represented by pooling their contextual token embeddings.The method applies pooling separately to the source and target subsequences.
  • Span extraction: SALT obtains supervision from either LLaMA-based extraction or the CASE heuristic, which derives aligned spans from token alignments and constituent structure.CASE filters candidate pairs using bidirectional containment and a coverage threshold of c = 0.7.
  • Span extraction: Over 90% of human annotations for both LLaMA-based extraction and CASE fall in the top two semantic-equivalence categories.LLaMA performs better on Russian and French, while CASE matches or exceeds it in other languages.
  • Training objective: SALT combines span-level contrastive and translation objectives with sentence-level interpolation to improve fine-grained representations while preserving the original sentence space.The translation objective predicts target-span tokens, while interpolation constrains the updated encoder toward the frozen encoder’s sentence representations.

4 Experiment details

The experiments compare SALT with multilingual encoders and alternative fine-tuning objectives across sentence- and token-level evaluations. Token quality is assessed through word alignment and zero-shot sequence-labeling tasks, while sentence performance uses retrieval and classification benchmarks.

  • Data and training: SALT is trained on NLLB Primary parallel data covering 129 language pairs, with 29 held-out test languages available for generalization evaluation.Each retained language pair is downsampled to 40,000 sentence pairs.
  • Baselines: The study compares XLM-R, XLM-Align, LaBSE, MEXMA, and SONAR fine-tuned with alternative objectives against SALT.Alternative SONAR variants include longer original-loss training and token-level objectives.
  • Sentence evaluation: Sentence mining uses FLORES-200 devtest across 80 languages, while classification uses English-only MTEB tasks.The reported sentence-mining metrics are xsim and xsim++.
  • Token evaluation: Token representations are evaluated with word alignment and zero-shot sequence labeling, avoiding reliance on sequence-level tasks that can mask token weaknesses.Sequence labeling trains a linear head on English and evaluates it in other languages.
  • Token evaluation: The sequence-labeling suite includes Massive slot filling, PAN-X named-entity recognition, UDPOS part-of-speech tagging, and WiC word-sense disambiguation.WiC uses concatenated contextual embeddings for the target word.

5 Results

SALT delivers the strongest overall token-level results among the considered strategies while preserving or improving sentence-level quality. Its gains coincide with better cross-lingual span alignment and a more language-invariant, less hub-prone embedding geometry.

  • Sentence-level evaluation: SALT improves sentence-level quality rather than sacrificing it, consistently outperforming SONAR on cross-lingual sentence mining and improving STS17 while matching SONAR on XNLI and MIntent.
  • Span representations: SALT span representations show higher cosine similarity for aligned spans than SONAR across layers and support high-quality translation with a frozen SONAR decoder.
  • Token-level evaluation: SALT achieves the best or joint-best performance on four of five cross-lingual token-level benchmarks.It outperforms alternative fine-tuning objectives on word alignment, PAN-X, Massive, and WiC.
  • Intermediate representations: SALT’s deeper layers become more cross-lingually aligned for spans and words, shifting the usual pattern in which intermediate layers align best.
  • Embedding geometry: SALT achieves the highest word-retrieval recall and produces embeddings that are harder to classify by language than SONAR and MEXMA.
  • Embedding geometry: SALT avoids the contrasting baseline failures of MEXMA’s anisotropy and SONAR’s hub collapse, yielding a better-distributed cross-lingual token space.SONAR exhibits severe hubness at 11.5 skewness, whereas SALT preserves isotropy while reducing language-specific clustering.
  • Ablations: The full SALT objective achieves the best sequence-tagging score and a highly competitive AER of 0.171 because its three loss terms provide complementary signals.No subset consistently dominates across both word alignment and sequence-tagging metrics.
  • Span extraction: Comparable performance can be achieved with the self-supervised CASE span extractor, indicating that SALT’s training regime is primarily responsible for its success.CASE is marginally better on Massive, while LLaMA-70B supervision is better on PAN-X, UDPOS, and WiC.

6 Conclusions

SALT improves cross-lingual token representations through span-aligned supervision while preserving or improving sentence-level quality. The results support span-level supervision as a powerful inductive bias for multilingual representation learning.

  • SALT improves cross-lingual token representations through span-aligned supervision.
  • SALT consistently outperforms alternative fine-tuning strategies and competitive encoders across multilingual token-level benchmarks.
  • SALT preserves and in several cases improves performance on sentence mining and classification benchmarks.
  • Span-level supervision is a simple yet powerful inductive bias for multilingual representation learning.

Limitations

The paper identifies scope, representation, and training-stage limitations for SALT. These include reliance on parallel data and existing token quality, average pooling for spans, and evaluation restricted to post-training intervention.

  • SALT relies on parallel corpora to extract aligned spans, which may limit applicability to truly low-resource languages.
  • CASE assumes that the underlying encoder already provides reasonably good cross-lingual token representations.
  • Average pooling may fail to capture finer internal structure within longer or compositionally complex spans.
  • The study evaluates SALT only as a post-training intervention, leaving span-level supervision during large-scale pre-training unexplored.

Ethics statement

The paper describes responsible-use considerations, training-data provenance, and experimental data and implementation practices. It also documents segmentation, span extraction, evaluation, and optimization procedures.

  • Multilingual representation models may be used in sensitive applications such as surveillance or profiling, motivating responsible deployment and evaluation.
  • Training data come from publicly available NLLB Primary parallel corpora that may contain societal or cultural biases.
  • Human span-quality evaluation used voluntary participation, informed consent about withdrawal, anonymised annotator IDs, and institutional ethics approval.
  • For non-whitespace scripts, tokeniser tokens are treated as words, with word embeddings formed by averaging their composing token embeddings.
  • LLM-extracted spans are mapped back to SONAR tokens, assigning split tokens to the span with greatest string overlap.
  • Experiments use multilingual benchmark splits, frozen-encoder linear probes, bounded examples, sequence-length filtering, and dynamically constructed token-budgeted mini-batches.

D.2 Sequence tagging results

SALT consistently improves sequence tagging over SONAR across English-only, multilingual, and language-specific classifier settings. It also generalizes to held-out languages, although UDPOS shows limited cross-lingual transfer.

  • Training settings: Sequence tagging is evaluated with English-only, all-language, and target-language-specific classifier training settings.
  • Results: SALT consistently outperforms SONAR across all three sequence tagging training settings.
  • Results: UDPOS shows limited cross-lingual transfer, with single-language training substantially outperforming multilingual training.
  • Held-out languages: On held-out test languages, SALT outperforms the other losses except on UDPOS.
  • Held-out languages: Held-out languages benefit from SALT particularly at earlier training stages, while trained languages continue improving over a longer training horizon.
  • Figures: Figure 5 compares SALT and SONAR under the three classifier-training modes, while Figures 6 and 7 show training dynamics and word/span similarity.

E.3 Qualitative analysis

SALT improves qualitative cross-lingual word retrieval by bringing true translations closer and reducing same-language, context-insensitive matches. These effects suggest better preservation of contextual meaning despite lower average cosine similarity across the embedding space.

  • Cross-lingual retrieval: SALT more frequently retrieves non-Latin-script tokens and places true translations closer in absolute cosine similarity.This occurs despite SALT having a lower average cosine similarity across the embedding space than the other encoders.
  • Contextual retrieval: SALT reduces retrieval of the same word from different languages in unrelated sentential contexts.The passage identifies this as a common failure mode in cross-lingual retrieval.

G.1 Results on human evaluation of extracted aligned spans

The human evaluation examines automatically extracted aligned spans judged for semantic equivalence by native speakers across five languages. Acceptable spans exceed 90% in almost all languages, supporting spans as a training signal and showing that SONAR with CASE closely matches LLM annotation.

  • Implications: The results support using extracted spans as a training signal for cross-lingual representation learning.This conclusion follows from the high rate of acceptable human ratings across languages.
  • System comparison: SONAR with CASE produces performance closely resembling the LLM-based annotation results.The comparison is reported across the human-evaluation score distributions.
Loading 2609.09953v1…