PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation

M. Ali Bayram, Banu Diri, Savaş Yıldırım

arXiv:2605.29992v1cs.CL

TL;DR

Multilingual embedding models must balance capacity across languages, leaving room for Turkish-specific tokenizer adaptation. The paper introduces an efficient tokenizer, cloning, and offline-distillation pipeline, producing a 200M-parameter model that surpasses its teacher on STSbTR and remains competitive across TR-MTEB.

  • Problem

    Multilingual embedding models allocate capacity across languages and large vocabularies, motivating Turkish-specific adaptation for morphologically rich language processing.

  • Method

    The paper adapts a multilingual teacher through hybrid tokenizer construction, weight-preserving cloning with embedding remapping, and offline distillation from precomputed embeddings.

  • Results

    77.55%/77.45% Pearson/Spearman on STSbTR versus the teacher’s 73.84%/72.92%, while TR-MTEB averages 63.9% across 26 tasks.

  • Takeaways & Limitations

    The approach offers a competitive cost–quality point with 33% fewer parameters and training costs of $5–$20.

  • Takeaways & Limitations

    Performance on lower-resource non-target languages may degrade, and very long-document performance is not extensively evaluated.

Abstract

from arXiv · show

Sentence embeddings are a foundational component for semantic search, clustering, classification, and retrieval-augmented generation. This paper presents embeddingmagibu-200m, a Turkish-focused sentence embedding model that produces 768-dimensional L2-normalized vectors and supports an 8,192-token context window, far exceeding the 512-token limit of earlier BERT-based Turkish encoders. Instead of full pretraining, an efficient three-stage adaptation pipeline is introduced: (1) construct a Turkish-optimized multilingual tokenizer with a 131,072 vocabulary by pruning redundant tokens from the teacher's vocabulary and incorporating multilingual tokens via frequency analysis on a 40-language corpus, (2) clone a teacher embedding model while preserving transformer backbone weights and initializing a compatible embedding table for the new vocabulary via mean-composition token mapping, and (3) perform offline embedding distillation from precomputed teacher vectors using a cosine similarity objective over a balanced 40-language Wikipedia corpus. The resulting student model contains approximately 200M parameters and trains in roughly four hours on a single GPU by avoiding online teacher inference during training, at a total cost of $5-$20. Empirically, Pearson/Spearman correlations of 77.55%/77.45% are obtained on STSbTR, surpassing the 300M-parameter teacher model (73.84%/72.92%). On TR-MTEB (26 tasks), a mean score of 63.9% is achieved (7th out of 26 models), providing a competitive cost-quality trade-off with 33% fewer parameters than the teacher. To facilitate reproducibility and downstream use, all artifacts are released including model weights, tokenizer files, precomputed embedding datasets, and open-source cloning and distillation tooling.

1 Introduction

The paper introduces embeddingmagibu-200m, a Turkish-focused sentence embedding model designed for efficient, long-context representation. It adapts a multilingual teacher through tokenizer surgery, embedding-table reconstruction, and offline distillation rather than training from scratch.

  • Model and motivation: Turkish agglutination causes general multilingual tokenizers to fragment morphologically complex words into semantically meaningless subwords, degrading representations and increasing token counts.The example evlerimizden (“from our houses”) illustrates arbitrary, non-morphic fragmentation.
  • Model and motivation: embeddingmagibu-200m outputs 768-dimensional L2-normalized sentence embeddings with an 8,192-token maximum sequence length and approximately 200M parameters.The model is provided in SentenceTransformers format and targets document-level retrieval with a long context window.
  • Tokenizer adaptation: A Turkish-optimized multilingual tokenizer combines 64K frequent Turkish tokens with frequency-selected multilingual tokens, producing a 131,072-token vocabulary.Redundant or alternative teacher tokens are pruned, while Wikipedia-40-langs frequency analysis supports multilingual coverage.
  • Tokenizer adaptation: The teacher’s transformer backbone is preserved while new token embeddings are initialized by mean-composition token mapping, retaining the teacher’s semantic space despite changed token identities.Each new token is mapped to one or more teacher tokens to compose compatible embeddings for the new vocabulary.
  • Offline distillation: Offline distillation matches precomputed teacher vectors for approximately 580K examples from a balanced 40-language Wikipedia corpus using a cosine similarity objective.Precomputing teacher embeddings avoids online teacher inference during training.

2 Related Work

Prior Turkish sentence-embedding work relies mainly on BERT-based encoders whose 512-token context limits document-level retrieval. Related research addresses tokenizer adaptation, vocabulary transfer, and offline distillation to improve efficiency and compatibility.

  • Turkish Sentence Embeddings and Benchmarks: 512-token context windows constrain BERT-based Turkish encoders’ utility for document-level retrieval.Turkish monolingual representation learning has traditionally used encoder-only architectures, particularly BERT variants such as BERTurk.
  • Tokenizer Adaptation for Morphologically Rich Languages: Agglutinative Turkish words are often fragmented into arbitrary, non-morphic subwords by general multilingual tokenizers.Suffix-rich forms such as yap-abili-yor-uz-dur create substantial challenges for BPE- and WordPiece-based tokenization.
  • Embedding Distillation: Offline distillation precomputes and stores teacher embeddings to reduce the computational overhead of running the teacher during training.Knowledge distillation transfers semantic capabilities from a larger teacher to a smaller student, including through multilingual sentence-level representation matching.

3 Method

The method adapts a multilingual embedding teacher to Turkish through hybrid tokenizer construction, backbone-preserving embedding remapping, and offline cosine distillation. Precomputed teacher embeddings enable efficient single-GPU training while retaining multilingual coverage and extending the context window.

  • Pipeline: The end-to-end pipeline combines tokenizer training, model cloning with embedding remapping, teacher-embedding precomputation, and embedding distillation.The design aims to retain the teacher’s semantic space while reducing parameters through a Turkish-optimized vocabulary.
  • Tokenizer construction: 131,072 tokens form a hybrid vocabulary combining 65,536 frequent Turkish tokens with frequency-selected multilingual tokens from Wikipedia-40-langs.Redundant or alternative subwords in the teacher’s original 256K tokenizer are pruned before multilingual tokens are added.
  • Model cloning: 8,192 tokens is the student’s maximum sequence length, with 768-dimensional unit-length embeddings produced by mean pooling, two 768 → 3072 → 768 projections, and final ℓ2 normalization.The student uses a Gemma3TextModel backbone initialized from the teacher and prompt-aware encoding with include_prompt=True.
  • Model cloning: The new embedding table is initialized by mean-composing mapped teacher token embeddings while preserving attention, feedforward, and layer-normalization weights exactly.This avoids random token embeddings and reduces the embedding-table parameter count when moving from the teacher’s 256K vocabulary to the student’s 128K vocabulary.
  • Offline distillation: Approximately four hours on one NVIDIA A100 80GB GPU is required for offline distillation, which precomputes teacher embeddings and trains the student with cosine similarity.The balanced 40-language corpus contains approximately 580K rows, with Turkish and English capped at 100K examples each and the other 38 languages at 10K each.
  • Offline distillation: 0.05 is the approximate final training loss after rapid early optimization from 0.09 to 0.07 within the first 200 steps.The logs associate this convergence with mean-composition initialization aligning vocabulary spaces before training.

4 Experiments

The experiments evaluate embeddingmagibu-200m using a balanced multilingual distillation corpus and two Turkish benchmarks: STSbTR and the 26-task TR-MTEB suite.

  • Datasets: ≈580K precomputed teacher-embedding rows from a balanced 40-language Wikipedia corpus support distillation, alongside the Cosmos Turkish Corpus for tokenizer training.Evaluation uses STSbTR and TR-MTEB.
  • STSbTR: STSbTR contains 5,749 training pairs and 1,379 test pairs rated from 0.0 to 5.0, with cosine similarity evaluated using Pearson and Spearman correlations.Both correlation coefficients are reported on training and test splits.
  • TR-MTEB: TR-MTEB spans 26 tasks across seven categories, reporting macro-averaged overall and category-level scores.Its categories are Retrieval, Classification, Clustering, STS, NLI, Bitext Mining, and Reranking.
  • Baselines: Baselines include the 300M-parameter teacher EmbeddingGemma-300M, embeddingmagibu-152m, Multilingual E5 variants, turkish-e5-large, and TabiBERT.The comparison set combines multilingual and Turkish-focused embedding models.

5 Results and Discussion

embeddingmagibu-200m outperforms its teacher on STSbTR and selected TR-MTEB categories while using fewer parameters, benefiting from Turkish-optimized tokenization and a long context window. Its offline-distillation pipeline achieves this performance at low training cost.

  • STSbTR Results: 77.55% test Pearson correlation ranks embeddingmagibu-200m 6th among 21 models, exceeding EmbeddingGemma-300M’s 73.84% by 3.71 percentage points.It also reaches 82.35% Pearson correlation on the training set, ranking 3rd overall.
  • TR-MTEB Results: 63.9% TR-MTEB average ranks embeddingmagibu-200m 7th out of 26 models, versus 65.2% for EmbeddingGemma-300M with 33% fewer parameters.The student exceeds its teacher on STS (77.5% vs. 72.9%), NLI (67.9% vs. 60.6%), and Bitext Mining (97.0% vs. 96.8%), but trails on Retrieval (72.2% vs. 75.9%) and Classification (68.5% vs. 71.8%).
  • Morphological Alignment and Learning Dynamics: +4.6% and +7.3% absolute gains over the teacher occur on STS and NLI, respectively, where fine-grained semantic compositionality benefits from morphologically aligned subwords.The results attribute these gains to resolving token fragmentation with a larger, Turkish-optimized vocabulary.
  • Long Context and RAG Suitability: 8K-token context enables embeddingmagibu-200m to encode whole documents without chunking, unlike most monolingual Turkish models limited to 512 tokens.The target tokenizer’s reduced token footprint further extends the effective context window for enterprise RAG applications.
  • Cost–Quality Frontier: $5–$20 and approximately four GPU hours yield 98.0% of the teacher’s TR-MTEB average with 33% fewer parameters.The model also outperforms its teacher on STS, NLI, and Bitext Mining, supporting targeted tokenizer adaptation and distillation as a cost-efficient strategy.

6 Ablations and Analysis

The ablations show that expanding the vocabulary improves semantic performance while increasing parameters, whereas tokenizer adaptation enables the 200M student to approach or exceed the larger teacher on target-language benchmarks with a smaller footprint.

  • Vocabulary Size Ablation: Doubling the vocabulary from 64K to 128K increases embedding parameters from 49.5M to 100.6M (+51M) and improves overall performance by +3.7%.Gains are largest in NLI (+12.3% absolute), STS (+5.7%), and Bitext Mining (+6.9%), while clustering changes by −0.2%.
  • Scale–Performance Frontier: 63.9% TR-MTEB average is achieved by embeddingmagibu-200m, versus 60.2% for embeddingmagibu-152m and 65.2% for the 300M teacher.The models use approximately 205M, 154M, and 300M parameters, respectively.
  • Parameter Footprint: 48.8% fewer embedding-layer parameters result from trimming the vocabulary from 256K to 128K, reducing the embedding layer from 196.6M to 100.6M parameters.The passage links this reduction to a smaller disk footprint, lower GPU VRAM consumption, and faster inference throughput.
  • Tokenizer-Controlled Experiments: Controlled tokenizer experiments compare MFT, Tabi, Cosmos, and Mursit with the same backbone, training corpus, random initialization, and cosine-distillation teacher embeddings.The experiments isolate tokenization design from pretraining history and data-mix confounders.

7 Limitations

The adaptation inherits the teacher’s semantic errors and biases, while vocabulary reduction and mean-composition initialization constrain representational capacity and sense disambiguation. Long-document performance and the absence of supervised Turkish NLI/IR training remain insufficiently evaluated or potentially limiting.

  • Teacher dependence: The student inherits representation errors and biases from EmbeddingGemma-300M because its semantic space is bounded by the teacher.Teacher-based distillation transfers limitations present in the teacher model.
  • Tokenizer capacity: Reducing the vocabulary from 256K to 128K reduces the model’s capacity to represent non-target languages.The limitation arises from shrinking the teacher vocabulary during tokenizer adaptation.
  • Initialization: Mean-composition initialization ignores polysemy and context, so averaging teacher subword embeddings cannot identify the relevant sense of a surface-form token.A surface-form token may map to subwords whose embeddings encode multiple senses.
  • Long-context evaluation: Performance on very long documents is not extensively evaluated despite support for 8,192-token inputs.Very long content may require chunking and aggregation strategies beyond simple mean pooling.
  • Training scope: The training recipe matches teacher embeddings for single texts without supervised contrastive training on Turkish NLI/IR data, potentially limiting ceiling performance.The omitted supervised training is identified as a possible source of lower ultimate performance.

8 Reproducibility

The paper supports reproducibility by releasing model, tokenizer, configuration, and distillation artifacts alongside cloning and distillation tooling. It also specifies the implementation pipeline, hardware, runtime, storage needs, and training hyperparameters.

  • Artifacts: Released artifacts include Hugging Face model weights, Ollama deployment files, the precomputed distillation dataset, tokenizer files, configurations, module weights, and model documentation.Tokenizer files include tokenizer.model, tokenizer.json, and tokenizer_config.json; configurations include config.json and modules.json.
  • Pipeline: The four-stage pipeline performs hybrid tokenizer construction, weight-preserving teacher cloning with embedding remapping, teacher embedding generation, and offline student distillation.The transformer-cloner and distil-trainer packages support cloning and distillation, while conceptual Python code and training scripts appear in Appendix A.
  • Compute requirements: Approximately 4 hours of distillation runs on a single NVIDIA A100 80GB GPU or equivalent hardware with ≥40GB VRAM, while the embedding dataset requires approximately 5–10GB of storage.These requirements define the reported hardware, runtime, and storage conditions for reproduction.
  • Training configuration: Training uses a 0.01 warmup ratio, 0.01 weight decay, a 1.0 maximum gradient norm, bf16 precision, enabled gradient checkpointing, enabled torch.compile, and cosine loss on final embeddings.These hyperparameters are specified in the Hugging Face model card.

9 Conclusion

embeddingmagibu-200m adapts multilingual embeddings to Turkish through tokenizer construction, weight-preserving cloning, and offline distillation, producing 768-dimensional embeddings with an 8,192-token context window. It surpasses its teacher on Turkish semantic similarity while reducing model size and training cost, and releases infrastructure for adoption and benchmarking.

  • Contributions: embeddingmagibu-200m provides 768-dimensional outputs, an 8,192-token context window, and a 128K multilingual vocabulary through a three-stage adaptation pipeline.The pipeline combines hybrid tokenizer construction, weight-preserving model cloning with embedding remapping, and offline distillation from precomputed teacher embeddings.
  • Empirical results: 77.55%/77.45% Pearson/Spearman on STSbTR surpasses EmbeddingGemma’s 73.84%/72.92%.The result indicates that Turkish-optimized tokenization combined with distillation improves Turkish semantic similarity over the multilingual teacher.
  • Empirical results: 63.9% average on TR-MTEB across 26 tasks places the model 7th out of 26 models.The model achieves this result with approximately 200M parameters, a 33% reduction from the teacher.
  • Efficiency: $5–$20 total training cost and approximately four hours of GPU time demonstrate the approach’s accessibility.Training uses approximately 200M parameters and avoids online teacher inference through offline distillation.
  • Adoption and future work: Released artifacts include open-source cloning and distillation tools, precomputed embedding datasets, and a Hugging Face Space for exploration and benchmarking.Future work covers alternative tokenizer methods, other morphologically rich languages, non-linear projections, and long-context Turkish retrieval-augmented generation.

A Implementation Code

The appendix provides Python implementation snippets covering the paper’s tokenizer construction, model cloning, teacher embedding generation, and student training pipeline.

  • A Implementation Code: Python snippets implement custom tokenizer construction, model cloning, teacher embedding generation, and student training.The appendix focuses on implementation coverage across these four components.

A.1 Conceptual Custom Tokenizer Construction … A.4 Offline Embedding Distillation

The adaptation pipeline builds a Turkish-optimized multilingual tokenizer, remaps a cloned teacher model’s embeddings, precomputes teacher vectors, and distills them offline using cosine distance. The consecutive stages prioritize frequent Turkish and multilingual tokens while preserving transformer weights and enabling efficient student training.

  • A.1 Conceptual Custom Tokenizer Construction: Frequency-filtered multilingual tokens from Wikipedia-40-langs are combined with Turkish selections to construct the final 128K (131,072)-tokenizer vocabulary.The multilingual analysis considers token lengths 1, 2, 3, and 4 before merging the selections.
  • A.1 Conceptual Custom Tokenizer Construction: 64K (65,536) frequent Turkish tokens are selected, then used to prune redundant teacher-tokenizer representations.A temporary Turkish tokenizer analyzes subword frequencies before selecting the most frequent tokens and removing teacher tokens resolvable by them.
  • A.2 Weight-Preserving Model Cloning: The student is initialized by cloning embeddinggemma-300m weights and remapping its embedding table with the transformer-cloner package.This stage preserves the teacher model’s weights while adapting the embedding table to the new tokenizer.
  • A.3 Teacher Embedding Generation: Before training, teacher embeddings are precomputed over the multilingual Wikipedia corpus for efficient offline distillation.Precomputation enables the training process to use stored teacher vectors rather than generating them during each training step.
  • A.4 Offline Embedding Distillation: The student minimizes cosine distance to the precomputed teacher embeddings using the distillation trainer.This defines the offline embedding-distillation objective for student training.
  • A.4 Offline Embedding Distillation: 256 batch size, 5e-5 learning rate, 1 training epoch, and bf16 precision configure the distillation run.The supplied training configuration specifies batch_size=256, learning_rate=5e-5, num_epochs=1, and precision='bf16'.
Loading 2605.29992v1…