PaperlayerView exact revision on arXiv

Source-linked AI summary

SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges

Seongtae Hong, Youngjoon Jang, Jia-Heui Ju, Hyeonseok Moon, Heuiseok Lim

arXiv:2605.26002v1cs.IR

TL;DR

English-centric sparse-encoder vocabularies limit non-English language transfer because target-language terms lack representation dimensions. SemBridge uses multilingual dense embeddings and sparse semantic weighting to initialize target tokens, and experiments across five languages and four architectures show superior zero-shot and fine-tuned retrieval with faster convergence.

  • Problem

    English-centric sparse-encoder vocabularies contain too few non-English tokens, limiting representation of target-language semantics even with fine-tuning.

  • Method

    SemBridge aligns source and target vocabularies through multilingual dense embeddings and initializes each target token from a sparse weighted combination of semantically related source tokens.

  • Results

    Across five languages and four sparse architectures, SemBridge achieves superior zero-shot retrieval and improves fine-tuned performance while accelerating convergence.

  • Takeaways & Limitations

    SemBridge provides a practical solution for building high-performance sparse retrieval models in target-language environments.

  • Takeaways & Limitations

    Evaluation covers only Arabic, Chinese, Hindi, Korean, and Russian, while behavior in languages with different resources, scripts, or morphology remains underexplored.

Abstract

from arXiv · show

Sparse encoders offer high-precision retrieval by representing term importance within a vocabulary space, yet their English-centric structures pose a critical impediment to language transfer for non-English languages. To overcome this structural limitation, we propose SemBridge, a novel embedding initialization method designed for cross-lingual adaptation in sparse encoders by leveraging multilingual bridge models. SemBridge establishes semantic alignments between source and target vocabularies using multilingual dense embeddings as a bridge. Rather than directly relying on all source tokens, SemBridge selects a small set of semantically related source-language tokens and uses them to initialize each target-language token, effectively filtering out semantic noise and reconstructing target tokens as precise linear combinations of core synonyms. This accelerates convergence during fine-tuning and improves training efficiency. Extensive experiments across five languages and four sparse architectures demonstrate that SemBridge achieves superior zero-shot retrieval performance and consistently improves retrieval performance after fine-tuning compared to existing baselines. These results validate SemBridge as a practical solution for deploying high-performance sparse retrieval systems in diverse linguistic environments.

1 Introduction

Sparse encoders combine semantic understanding, keyword precision, efficiency, and interpretable term weights, but English-centric vocabularies limit non-English transfer. SemBridge addresses this structural bottleneck through semantically aligned target-token initialization and demonstrates strong transfer across languages and architectures.

  • Motivation: Sparse encoders represent contextual term importance in vocabulary-space vectors, combining semantic understanding with keyword precision and interpretable retrieval evidence.Their compatibility with inverted indexes also supports efficient large-scale retrieval.
  • Motivation: Most evaluated sparse-encoder vocabularies contain negligible proportions of non-English tokens, leaving too few dimensions for representing target-language terms.granite-30m-sparse contains only two Korean tokens, while splade-v3 shows significant English bias.
  • Motivation: Target-language fine-tuning remains limited when the sparse encoder lacks target-language vocabulary dimensions.The scarcity of target-language tokens constrains optimal performance even after fine-tuning.
  • SemBridge: SemBridge uses multilingual dense embeddings to align source and target vocabularies and initializes each target token from semantically related source tokens.Sparse semantic weighting replaces random or surface-overlap initialization while preserving source retrieval capabilities.
  • Results: Experiments across four sparse models and five languages show effective zero-shot transfer, superior fine-tuned performance, and faster convergence than baselines.Qualitative analysis indicates alignment with core source-language synonyms while filtering unnecessary semantic noise.

2 Related Work

Language transfer adapts resource-rich-language models to target languages under limited data and computation. Existing strategies include continued pretraining, fine-tuning, vocabulary changes, and generic embedding initialization, but semantic alignment remains central.

  • Sparse Encoders: Sparse encoders advance retrieval by predicting token importance in high-dimensional vocabulary-space vectors through several term-weighting and interaction strategies.These include estimating importance for all terms, re-estimating existing-term weights, latent-term expansion, and token-level interaction maximization.
  • Language Transfer: Language transfer adapts models pretrained in resource-rich languages to target environments with limited data and computational resources.Common approaches include continued pretraining and fine-tuning.
  • Language Transfer: Vocabulary expansion adds target-language tokens, while tokenizer replacement addresses vocabulary mismatch but requires initializing target embeddings without disrupting the source representation space.These strategies differ in whether they extend or replace the source vocabulary.
  • Language Transfer: Random and source-statistics-based initialization preserve generic distributional properties but fail to align source and target token semantics.The related-work discussion motivates semantic embedding initialization as a more targeted alternative.

3 SemBridge

SemBridge initializes target-token embeddings by preserving shared tokens and mapping non-overlapping tokens through multilingual semantic similarity. Entmax sparsifies source-token mixtures so each target token is reconstructed from a few relevant semantic correspondences.

  • Overview: SemBridge uses a source sparse encoder and target tokenizer to initialize embeddings tailored to the target language.The method is designed to transfer the source model’s representation space into the target vocabulary.
  • Overlapping Token Embedding Transfer: Overlapping target and source tokens are identified through exact or normalized matches and initialized by directly copying source embeddings.Shared tokens include language-agnostic items such as numbers, symbols, and proper nouns.
  • Cross-lingual Semantic Bridge: A multilingual dense embedding model projects source and target tokens into a shared semantic space to align tokens with different surface forms.Non-overlapping target tokens form R and are compared with the source vocabulary through dense representations.
  • Cross-lingual Semantic Bridge: For each uninitialized target token, SemBridge computes similarities against every source-vocabulary token and forms a similarity matrix quantifying semantic relevance.The matrix supplies the information used to derive source-token initialization weights.
  • Similarity-Based Sparse Weighting: Entmax transforms each similarity vector into sparse weights that remove irrelevant source tokens by assigning exact zeros to the distribution tail.The sparsity hyperparameter α controls how many source tokens contribute to each target embedding.
  • Similarity-Based Sparse Weighting: Each target embedding becomes a linear combination of a few core synonyms while preserving dimension and compatibility with the source model architecture.The resulting initialization inherits the source model’s sparse encoding capability without architectural modification.

4 Experimental Setup

The experiments transfer four sparse encoders to five target languages and evaluate retrieval with WebFAQ and MIRACL. They compare SemBridge with generic and statistical initialization methods using nDCG@10 and FLOPS.

  • Models and Languages: The study evaluates splade-v3, Splade_PP_en_v1, opensearch-neural-sparse-encoding-v13, and granite-embedding-30m-sparse.Target tokenizers cover Arabic, Chinese, Hindi, Korean, and Russian.
  • Models and Languages: Experiments transfer each source model independently to Arabic, Chinese, Hindi, Korean, and Russian target-language settings.The target tokenizers are ARBERT, bart-base-chinese, hindi-bert-v2, kobigbird-bert-base, and rubert-base-cased, respectively.
  • Evaluation: Retrieval is evaluated on WebFAQ and MIRACL, using nDCG@10 as the primary metric and FLOPS to assess sparsity and efficiency.The evaluation covers five target languages.
  • Baselines: SemBridge is compared with random, mean, univariate Gaussian, and other generic or statistical strategies for initializing non-overlapping target-token embeddings.Overlapping tokens are directly copied for all methods.

5 Experimental Results

SemBridge delivers stronger zero-shot transfer and fine-tuned retrieval than competing initialization methods, while also improving training convergence and efficiency across languages and sparse encoder architectures.

  • 5.1 Zero-shot Language Transfer: SemBridge consistently demonstrates superior initialization performance across all four sparse encoder models.Baseline, Random, and Mean yield near-zero or marginal performance across most language pairs, while Univar. and Multivar. show limited, high-variance improvements.
  • 5.1 Zero-shot Language Transfer: 0.422 and 0.522 are SemBridge’s average zero-shot nDCG@10 scores for Splade-v3 and Splade-PP, respectively, on WebFAQ.These scores indicate effective cross-lingual semantic correspondence in the representation space.
  • 5.2 Impact of Initialization on Fine-tuning: SemBridge outperforms baselines across five target languages, four models, and two datasets after fine-tuning.With Granite-30M-Sparse, SemBridge is the only method surpassing every baseline on both datasets.
  • 5.2 Impact of Initialization on Fine-tuning: Initialization quality continues to affect retrieval performance after identical fine-tuning, with noisy token correspondences leaving competing methods behind.SemBridge instead provides a foundation for preserving and leveraging the source model’s retrieval capabilities in the target language.
  • 5.3 Loss Trajectory: SemBridge generally begins with significantly lower loss and declines steeply during early training across Chinese, Korean, and Russian.For Russian, the initial loss is slightly higher but convergence is immediate; SemBridge maintains the lowest loss throughout training.
  • 5.3 Loss Trajectory: Figure 4 compares zero-shot nDCG@10 on WebFAQ and MIRACL while varying SemBridge’s sparsity level α from 1 to 4 against FOCUS and OFA.The figure evaluates how sparsity choices affect transfer performance.

6 Ablation and Analysis

Ablations show that sparse, semantically focused initialization improves transfer, while SemBridge balances retrieval quality with computational efficiency and produces more consistent cross-lingual token mappings.

  • Sparse Weighting: Softmax weighting all source tokens yields the lowest scores across most settings, indicating that irrelevant tokens introduce initialization noise.Entmax configurations with α ≥2 perform consistently better.
  • Sparse Weighting: Entmax with α = 3 or 4 outperforms FOCUS and OFA, although excessive sparsity may exclude meaningful semantic clues.The appropriate α may depend on the target language.
  • Efficiency Analysis: SemBridge achieves the highest nDCG@10 across models while maintaining significantly lower FLOPS than FOCUS after fine-tuning.FOCUS has excessively high FLOPS, whereas Gaussian exhibits very low FLOPS but unstable performance.
  • Qualitative Analysis: FOCUS and OFA often assign irrelevant source tokens, whereas SemBridge links target-language words for “home” to core synonyms such as “home,” “house,” “casa,” and “maison”.SemBridge’s mapping sparsity can be controlled with Entmax α; α = 4 suppresses noise by selecting core synonyms.

7 Conclusion

SemBridge transfers English-centric sparse encoders to target languages by aligning vocabularies through multilingual dense embeddings and sparsely weighting relevant source tokens. Across five languages and four sparse architectures, it outperforms existing initialization methods in zero-shot and fine-tuned settings while accelerating convergence.

  • Conclusion: SemBridge transfers English-centric sparse encoders to target languages through multilingual semantic bridging and sparse weighting of relevant source tokens.The method is designed to filter noise during target embedding initialization.
  • Conclusion: SemBridge outperforms existing initialization methods in both zero-shot and fine-tuned settings across five languages and four sparse architectures.The evaluated languages are Arabic, Chinese, Hindi, Korean, and Russian.
  • Conclusion: SemBridge significantly accelerates convergence during fine-tuning, supporting accurate token-level alignment for transferring sparse retrieval capabilities.The conclusion characterizes the method as a robust and practical solution for target-language sparse encoders.

Limitations

The study’s evidence is limited to five target languages, and SemBridge’s sparsity setting may require language- and tokenizer-specific selection.

  • Scope: Evaluation covers Arabic, Chinese, Hindi, Korean, and Russian, leaving languages with different resources, scripts, or morphological structures underexplored.The behavior of SemBridge in those settings remains unknown.
  • Sparsity Selection: SemBridge depends on Entmax α, whose optimal value may vary by target language and tokenizer.Excessive sparsity can exclude meaningful clues, while insufficient sparsity can introduce irrelevant source-token noise.

Ethical Considerations

The paper describes sparse-encoder adaptation experiments using public data and no human-subject interaction, while noting that multilingual bridge-model bias and uneven language coverage require validation in new settings.

  • Ethical Scope: The experiments use publicly available datasets and do not collect personal information or involve human subjects.The work focuses on adapting sparse retrieval models to target-language environments.
  • Model Risks: SemBridge relies on multilingual dense embedding models as semantic bridges, so their biases or uneven language coverage may affect the initialized sparse encoder.The paper recommends careful validation for real-world use in new languages or domains.
  • Vocabulary Analysis: The study analyzes vocabulary composition by decoding token IDs before language detection because tokenizer prefixes and whitespace markers can interfere with identifying token language.The analysis covers four source sparse encoders and five target languages.
  • Baselines: For non-overlapping target tokens, the evaluated baselines include random, mean, Gaussian, FOCUS, and OFA initialization strategies.Overlapping-token embeddings are directly copied from the source embeddings.
  • Experimental Resources: The target-language tokenizer summary reports each model, vocabulary size, and Hugging Face repository used in the experiments.

C Experiments Details

The experiments replace each source sparse encoder’s tokenizer with a pretrained target-language tokenizer and train under standardized fine-tuning settings.

  • Target-language vocabularies are built from pretrained tokenizers, whose tokenizers replace those of the source sparse encoders.The corresponding tokenizer vocabulary defines the target vocabulary Vt for target token embedding initialization.
  • Fine-tuning uses four NVIDIA A100 GPUs, one epoch, batch size 64, maximum sequence length 512, and bf16 precision.
  • Training uses AdamW with a 2 × 10−5 learning rate and a linear warm-up covering 5% of total steps.Document and query FLOPs regularization weights are 1 × 10−4 and 3 × 10−4, respectively.

D.1 Robustness to Bridge Models

The study evaluates SemBridge with multiple multilingual dense bridge models on zero-shot retrieval, showing that bridge-model choice affects transfer quality while SemBridge remains robust against baselines.

  • Three multilingual embedding models—MiniLM, mGTE, and Qwen3—are compared as auxiliary bridge models against FOCUS and OFA on WebFAQ and MIRACL.The comparison uses zero-shot retrieval performance measured by nDCG@10.
  • Bridge-model choice influences sparse-model language-transfer outcomes.
  • 0.422 average performance is achieved by bge-m3 for Splade-v3 on WebFAQ, ahead of mGTE at 0.378, Qwen3 at 0.306, and MiniLM at 0.303.
  • SemBridge maintains stable and superior performance over FOCUS and OFA across the evaluated bridge models.The results associate stronger dense-model capacity with more effective transfer and identify bridge-model similarity quality as critical for alignment accuracy.
Loading 2605.26002v1…