PaperlayerView exact revision on arXiv

Source-linked AI summary

Yalda-Embedding: Open Development of a Competitive Persian Retrieval Model

Yalda Taheri, Mohammad Hassan Heydari, Armon Rasooli

arXiv:2610.32933v1cs.CLcs.AI

TL;DR

Persian dense retrieval remains constrained by limited language-specific training resources. YALDA-EMBEDDING adapts Qwen3-Embedding-0.6B with more than one million Persian query–document pairs and relational distillation. It leads the evaluated 0.6B models across eight FaMTEB tasks, and its released teacher representations support further student experiments without repeating offline teacher inference.

  • Problem

    Persian retrieval has limited language-specific training and evaluation resources, and gathering high-quality Persian data is challenging.

    Source: §I · p. 1

  • Method

    The paper adapts Qwen3-Embedding-0.6B using Persian query–document pairs and adds relational knowledge distillation from F2LLM-v2-8B.

    Source: §I · p. 1

  • Results

    0.5488: YALDA-EMBEDDING has the highest mean among evaluated 0.6B models across eight FaMTEB tasks and beats both same-scale decoder-based baselines on every task.

    Source: §VII · p. 6

  • Takeaways & Limitations

    Released weights, code, pairwise data, and teacher representations make the model and development choices available for inspection and extension without repeating offline 8B teacher inference.

    Source: §VII · p. 6

    Training uses one epoch and in-batch negatives only, without explicitly mined or supplied hard negatives.

    Source: §I · p. 1, B. InfoNCE Objective · p. 4

Abstract

from arXiv · show

Despite advances in multilingual text representations, effective dense retrieval for Persian remains constrained by the limited availability of language-specific training resources and compact embedding models. We present Yalda Embedding, a 595.8-million-parameter Persian retrieval model adapted from Qwen3-Embedding-0.6B using more than one million query-document pairs. On eight Persian retrieval tasks from FaMTEB, the model achieves the highest macro-average among the evaluated 0.6B models, scoring 0.5488 and outperforming Qwen3-Embedding-0.6B and F2LLM-v2-0.6B on every task. It also approaches Qwen3-Embedding-4B, winning on four tasks, tying on one, and trailing by 0.0075 in macro-average. An incremental ablation study examines contrastive batch size, optimization stabilization, and relational knowledge distillation from F2LLM-v2-8B. Under the one-epoch training setup, contrastive fine-tuning alone yields a modest mean improvement over initialization and regressions on two tasks. The final distilled configuration improves all eight tasks over the non-distilled model, raising the mean from 0.4813 to 0.5488 without increasing inference-time model size. These gains include hard-negative evaluation tasks despite training without explicitly mined or supplied hard negatives. We release model weights, training and evaluation code, the pairwise dataset, and its teacher-embedded counterpart, enabling inspection of data composition and training procedures, independent evaluation, and reuse without repeating expensive teacher inference.

I. INTRODUCTION

Persian retrieval resources remain limited, motivating adaptation of a multilingual foundation model for local retrieval. YALDA-EMBEDDING combines that approach with a large pairwise collection, incremental distillation experiments, and an open release.

I. INTRODUCTION

  • The model adapts Qwen3-Embedding-0.6B, using targeted Persian contrastive training to reshape its representation space for local retrieval tasks.The introduction frames this adaptation as attractive given limited Persian data and language-specific resources.Source: §I · p. 1
  • YALDA-EMBEDDING has the highest mean among evaluated 0.6B models and beats both same-scale baselines on all eight FaMTEB tasks, while winning four and tying one against Qwen3-Embedding-4B.Figure 1 presents the per-task comparison.Source: §I · p. 1, Figure 1 · p. 2
  • The training collection contains 1,035,981 Persian query–document pairs, and the incremental ablation reports a mean increase from 0.4813 without distillation to 0.5488 with the final configuration.The supplied passages identify the collection size and the ablation’s mean values.Source: §I · p. 1
  • The release includes model weights, training and evaluation code, the original pairwise dataset, and its teacher-embedded counterpart.Readers can inspect data and procedures, evaluate the released weights, and reuse teacher representations without repeating offline 8B inference.Source: §I · p. 1

II. RELATED WORK

Related work situates YALDA-EMBEDDING among contrastive text-embedding methods and Persian evaluation resources, then reviews decoder-only models adapted as embedding encoders.

A. Sentence and Text Embeddings

  • Sentence-BERT, SimCSE, E5, and multilingual E5 develop bi-encoder or contrastive approaches, while FaMTEB extends embedding evaluation to Persian and Tooka-SBERT supplies a Persian baseline.The cited approaches include,, [8], [4],,, and.Source: A. Sentence and Text Embeddings · p. 2
  • Embedding systems increasingly combine heterogeneous tasks, instructions, and staged contrastive learning, including GTE’s diverse training mixtures and synthetic-instruction fine-tuning [9],.Source: A. Sentence and Text Embeddings · p. 2

B. Decoder-Only Models as Embedding Encoders

  • Decoder-only models have been adapted as text encoders through pooling or bidirectional attention and contrastive learning; Qwen3 Embedding uses large-scale pretraining, supervised fine-tuning, synthetic multilingual data, and model merging [5], [12]–[14].The student retains the Qwen3-Embedding decoder architecture and Transformer components [15]–[19].Source: B. Decoder-Only Models as Embedding… · p. 2

C. Contrastive Learning and Knowledge Distillation

The section reviews contrastive learning and knowledge distillation, then describes a Persian pairwise training collection and its empirical source mixture. It also identifies the in-batch-negative setup and its potential false-negative trade-off.

C. Contrastive Learning and Knowledge Distillation

  • Larger contrastive batches provide more in-batch negatives, while retrieval-specific distillation transfers relevance behavior to smaller students; Fig. 2 distinguishes combined, InfoNCE, and KD losses.The displayed objective combines hard-label InfoNCE and knowledge-distillation losses using weight α.Source: C. Contrastive Learning and Knowledge… · p. 2, Figure 2 · p. 3

A. Persian Pairwise Collection

  • The Persian collection contains 1,035,981 query–positive-document pairs from 13 resources, normalized to a common schema and globally shuffled; Table I reports its composition.Source: A. Persian Pairwise Collection · pp. 2–3, Table I · p. 3
  • The source mixture is not resampled, so large web collections dominate update counts while smaller scientific and financial collections add domain diversity.Using the other 511 documents in each large batch as negatives is efficient but can introduce false negatives for semantically related queries.Source: A. Persian Pairwise Collection · p. 3

B. Teacher-Embedded Release

The teacher-embedded release provides precomputed F2LLM-v2-8B representations for the pairwise data. The paper also reports the embedding preparation cost and separates source training and evaluation partitions.

B. Teacher-Embedded Release

  • Each query and document has a precomputed F2LLM-v2-8B representation, retaining and renormalizing the teacher embedding’s first 1,024 of 4,096 dimensions before similarity scoring.Source: B. Teacher-Embedded Release · p. 3
  • Table II summarizes the student architecture implemented for training.Source: Table II · p. 3
  • Embedding the collection took approximately 250 hours on one NVIDIA P100 GPU and produced roughly 17.5 GB of teacher-enriched data; doing inference offline avoids loading the 8B teacher during TPU training.Source: B. Teacher-Embedded Release · p. 3
  • Training uses constituent datasets’ training splits, while evaluation uses their test splits, keeping the source-provided partitions separate.Source: B. Teacher-Embedded Release · p. 3

IV. METHODOLOGY

YALDA-EMBEDDING uses a shared Qwen3-initialized encoder to represent Persian queries and documents, then trains retrieval with a query-to-document InfoNCE objective using in-batch negatives.

A. Student Architecture

  • The shared student encoder starts from Qwen3-Embedding-0.6B and represents each text with the final hidden state at its last valid token.Queries use a fixed retrieval instruction; documents are encoded without one, and inputs are left-padded and truncated to 512 tokens.Source: A. Student Architecture · pp. 3–4

B. InfoNCE Objective

  • For each minibatch, embeddings are unit-normalized and query–document logits are cosine similarities scaled by τs = 0.05.The hard-label InfoNCE objective treats the matched document at the diagonal position as positive and all other batch documents as negatives.Source: B. InfoNCE Objective · p. 4
  • Training uses only the query-to-document direction and in-batch negatives, without explicitly mined or supplied hard negatives.Here, “hard-label” refers to diagonal positive labels, not hard-negative sampling.Source: B. InfoNCE Objective · p. 4

C. Relational Knowledge Distillation

Relational distillation trains the student to match the teacher’s distribution over documents in each batch. The final objective combines that distillation loss with hard-label InfoNCE.

C. Relational Knowledge Distillation

  • Teacher query and document embeddings are truncated to their first 1,024 Matryoshka dimensions, independently L2-normalized, and scored using τt = 0.07.For each query, teacher and student each define a distribution over all documents in the batch.Source: C. Relational Knowledge Distillation · p. 4
  • The distillation loss minimizes forward Kullback–Leibler divergence from the teacher’s batchwise document distribution to the student’s.This relational formulation transfers within-batch ranking geometry rather than regressing teacher coordinates, so teacher and student native embedding dimensions may differ.Source: C. Relational Knowledge Distillation · p. 4
  • The final objective sets α = 0.75, weighting relational distillation against hard-label InfoNCE.The earlier ablations use α = 0.25 and α = 0.50; retaining the 1,024-dimensional teacher prefix primarily reduces data-transfer and storage costs and aligns the deployed embedding width.Source: C. Relational Knowledge Distillation · p. 4

A. Optimization

The one-epoch training run uses a large global batch with warmup, cosine decay, and gradient clipping. Mixed precision and sharded TPU execution manage memory and computation.

A. Optimization

  • The complete pretrained student is fine-tuned for one epoch with global batch size 512, a 2 × 10^-5 peak learning rate, 10% warmup, cosine decay, and gradient clipping at 1.0.Of 1,035,981 examples, 1,035,776 are consumed in 2,023 complete optimization steps; the final incomplete batch’s 205 examples are omitted, and the final-run permutation seed is not fixed.Source: A. Optimization · p. 4

B. Mixed Precision

  • Memory-intensive forward activations and matrix multiplications use BF16, while parameters, gradients, optimizer moments, normalization, and loss calculations use FP32.The mixed-precision policy reduces activation memory and uses TPU BF16 throughput while retaining FP32 for state and operations more vulnerable to numerical instability.Source: B. Mixed Precision · p. 4

C. Distributed Execution

  • Training is sharded across eight TPU v5e devices, with 64 query–document pairs per device and parameters, optimizer state, and updates distributed across devices.All ablations and training runs required approximately 60 hours across the eight devices, including—not in addition to—the final model run.Source: C. Distributed Execution · p. 5

D. Checkpoint Export and Availability

YALDA-EMBEDDING leads the evaluated same-scale models across eight FaMTEB retrieval tasks and performs competitively against a larger Qwen reference. The released checkpoint can be evaluated alongside the reported comparisons.

D. Checkpoint Export and Availability

  • The exported checkpoint is saved as FP32 Safetensors, with tokenizer, configuration, and supporting files inherited from Qwen3-Embedding-0.6B.The model is available at the Yaldat/Yalda-Embedding repository.Source: D. Checkpoint Export and Availability · p. 5

A. Evaluation Setup

  • Evaluation uses test splits from eight FaMTEB retrieval tasks and reports their unweighted macro-average, not a score over the full FaMTEB suite.Models retain their configured prompting and pooling behavior; comparisons use retrieval scores at available precision and do not establish statistical significance.Source: A. Evaluation Setup · p. 5
  • YALDA-EMBEDDING has the highest mean among evaluated 0.6B models and outperforms the same-scale baselines on all eight tasks.It improves on the Qwen3-Embedding-0.6B initialization by 0.0775 absolute and exceeds F2LLM-v2-0.6B by 0.0425 in macro-average; Table IV summarizes macro-averages.Source: A. Evaluation Setup · p. 5, Table IV · p. 5
  • Against Diba-Embed’s 0.4788 macro-average, YALDA-EMBEDDING improves the mean by 0.0700 and scores higher on all eight tasks.The largest per-task gains are +0.11 on HotpotQA-FaHardNegatives, +0.09 on NFCorpus-Fa, and +0.09 on PersianWebDocumentRetrieval.Source: A. Evaluation Setup · p. 5
  • Against Qwen3-Embedding-4B, YALDA-EMBEDDING wins four tasks, ties one, trails on three, and is 0.0075 below in macro-average.The smaller model exceeds the reference on NFCorpus-Fa, SynPerQARetrieval, PersianWebDocumentRetrieval, and HotpotQA-FaHardNegatives, and ties it on FiQA2018-Fa.Source: A. Evaluation Setup · p. 5
  • Without explicit hard-negative training, YALDA-EMBEDDING scores 0.58 on HotpotQA-FaHardNegatives, 0.65 on MSMARCO-FaHardNegatives, and 0.43 on NQ-FaHardNegatives.It outperforms both evaluated 0.6B decoder-based baselines on all three tasks, using in-batch negatives and relational distillation during student training.Source: A. Evaluation Setup · pp. 5–6

B. Ablation Study

The ablation builds from contrastive training through optimization stabilization to relational distillation. Under the one-epoch setup, the final distilled model improves all eight tasks over the non-distilled recipe.

B. Ablation Study

  • The six cumulative configurations first develop the contrastive recipe, then test relational distillation weights of α = 0.25, 0.50, and 0.75.Ablation 2 changes batch size; Ablation 3 adds clipping, warmup, and cosine decay; Ablations 4–6 vary the distillation weight.Source: B. Ablation Study · p. 6
  • 0.5488 is the final distilled model’s mean, up 0.0675 from Ablation 3’s 0.4813, with gains on all eight tasks over both Ablation 3 and initialization.The final distillation step also reverses the NQ and SciFact regressions seen with contrastive fine-tuning alone.Source: B. Ablation Study · p. 6
  • Increasing batch size from 64 to 512 yields the largest individual average gain, +0.0388; the stabilization bundle adds +0.0075.Clipping, warmup, and cosine decay were introduced together, so their separate effects are not identified.Source: B. Ablation Study · p. 6
  • α = 0.25 relational distillation adds +0.0313 to macro-average and improves every task relative to Ablation 3.Increasing α to 0.50 adds +0.0188, improving seven of eight tasks, while α = 0.75 adds +0.0175 and improves seven tasks, leaving PersianWebDocumentRetrieval unchanged.Source: B. Ablation Study · p. 6

C. Training Dynamics

YALDA-EMBEDDING approaches Qwen3-Embedding-4B while leading the evaluated 0.6B models. Training dynamics show a trade-off between matching teacher relevance distributions and fitting hard-label positives, and the paper releases resources for inspection and reuse.

C. Training Dynamics

  • Higher KD weight yields lower teacher–student divergence but higher final hard-label InfoNCE loss across the three runs.Combined-loss magnitudes should not be compared directly because component weights differ.Source: C. Training Dynamics · p. 6
  • In the recorded α = 0.25 run, combined, InfoNCE, and KD losses shift from 1.015, 1.165, and 0.567 early to 0.559, 0.557, and 0.564 late.These numerical summaries apply only to Ablation 4; the teacher divergence remains close to its early value, maintaining a soft ranking constraint.Source: C. Training Dynamics · p. 6

VII. CONCLUSION

  • 0.5488 is the highest mean among evaluated 0.6B models; YALDA-EMBEDDING wins on four tasks and ties on one against Qwen3-Embedding-4B, with a 0.0075 macro-average gap.The model was adapted from Qwen3-Embedding-0.6B on more than one million query–document pairs.Source: §VII · p. 6
  • The release includes weights, training and evaluation code, training pairs, and reusable teacher representations for independent inspection and extension.The two-stage ablation documents the progression from contrastive training to relational distillation.Source: §VII · p. 6
Loading 2610.32933v1…