PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Retrievers for Polish and European Languages

Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec, Michał Perełkiewicz

arXiv:2609.12913v2cs.CL

TL;DR

Large retrievers based on multi-billion-parameter models make indexing, updates, and low-latency serving costly. The paper trains compact Polish and multilingual retrievers with a three-stage, teacher-supervised pipeline. PolDense and EuroDense perform strongly across their evaluated retrieval benchmarks, including against larger models.

  • Problem

    Multi-billion-parameter retrievers impose computational and memory costs that make indexing, frequent corpus updates, and low-latency serving costly.

    Source: §1 · p. 1

  • Method

    The authors train Polish and multilingual retrievers using cross-lingual alignment, relational distillation, and contrastive fine-tuning with teacher-generated supervision instead of original relevance labels.

    Source: §1 · p. 1, §3.1 · p. 2, §3.3 · p. 3

  • Results

    Across 41 Polish tasks, PolDense-1B establishes a new state of the art on PIRB; among evaluated multilingual models below 1B parameters, EuroDense ranks first on both aggregate measures and leads in seven of nine languages.

    Source: §5 · p. 6, §4 · p. 6

  • Takeaways & Limitations

    The results show that compact retrievers can be competitive with substantially larger models while offering strong quality-size trade-offs.

    Source: §5 · p. 6

    The models target retrieval rather than broader embedding tasks, and comparisons across Polish model sizes also reflect differing distillation-data amounts.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.

1 Introduction

Large language models have advanced retrieval, but their computational and memory demands make deployment costly. The paper introduces compact Polish and multilingual retrievers trained without original ground-truth relevance labels.

Motivation

  • Retrieval-augmented generation commonly combines a language model with retrieval, but multi-billion-parameter retrievers raise the costs of indexing, corpus updates, and low-latency serving.These demands limit practicality for collections containing millions of documents.Source: §1 · p. 1

Contributions

  • The paper introduces PolDense, six Polish retrievers from 17M to 1B parameters, and EuroDense, a 435M-parameter model for nine European languages.All models support inputs up to 8,192 tokens.Source: §1 · p. 1
  • The models use two knowledge-distillation stages followed by contrastive fine-tuning, with supervision from embedding models and a reranker rather than original relevance labels.The paper reports PolDense state-of-the-art performance and EuroDense best aggregate performance among evaluated multilingual models below 1B parameters.Source: §1 · p. 1

2 Related Work

Prior embedding research spans general-purpose multilingual models and Polish-specific retrieval models. The paper positions PolDense and EuroDense as efficient, language-focused alternatives.

Multilingual and LLM-based embeddings

  • Billions-parameter LLM-based embeddings can perform strongly on benchmarks such as MTEB, but their latency and infrastructure costs limit real-time retrieval use.Multilingual models may also face the curse of multilinguality: supporting many languages within a fixed parameter budget can worsen individual-language performance.Source: §2 · p. 2

Polish-language models

  • Polish retrieval and semantic-similarity work includes Polish SBERT, Silver-Retriever-Base-v1, MMLW, and Stella-PL-Retrieval.The passage also describes a broader progression from Sentence-BERT and SimCSE to E5 and BGE.Source: §2 · pp. 1–2

Positioning

  • The paper introduces dedicated Polish models for competitive quality and low latency, alongside a nine-language model intended to reduce parameter degradation and outperform solutions in its size class.The authors contrast language-specific efficiency with the challenges of supporting many languages within a fixed parameter budget.Source: §2 · p. 2

3 Methodology

The methodology trains compact retrievers through cross-lingual alignment, relational distillation, and retrieval-oriented contrastive fine-tuning. It applies the shared pipeline using separate Polish and multilingual data and model configurations.

3 Methodology

  • The three-stage pipeline uses two knowledge-distillation stages on parallel corpora followed by retrieval-oriented fine-tuning on question-document datasets.The same overall pipeline is used for all models.Source: §3 · p. 2

3.1 Cross-lingual alignment

  • Cross-lingual alignment trains a student to match a teacher’s source-sentence embedding and produce similar embeddings for its translations.The weighted objective combines MSE, which aligns dimensions, with cosine distance, which encourages agreement between vector directions.Source: §3.1 · p. 2

3.2 Relational distillation

  • Relational distillation combines embedding alignment, within-batch similarity matching, and a margin-based loss that reproduces the teacher’s pairwise-similarity ordering.The approach uses gradual unfreezing, removes the teacher-dimensional projection after distillation, and applies similarity losses to native and projected embeddings; multilingual training samples translations across languages.Source: §3.2 · pp. 2–3

3.3 Contrastive fine-tuning

  • Contrastive fine-tuning uses InfoNCE triplets, with reranker-selected positives scoring above 26 and hard negatives scoring between 12 and 20.For each query, candidates come from the top 16 retrieved by each of a dense retriever, SPLADE, and BM25; models trained on these relabeled examples consistently outperformed those using original relevance labels.Source: §3.3 · p. 3

3.4 Polish models (PolDense)

  • PolDense uses Ettin students, BGE-Multilingual-Gemma2 teachers, English-Polish parallel data, and Polish retrieval datasets for its six model sizes.The first stage uses about 20 million parallel pairs; the second uses about 70 million texts for 17M–150M models and 33 million for 400M–1B models, while contrastive fine-tuning draws on 13 retrieval datasets.Source: §3.4 · pp. 3–4

3.5 Multilingual model (EuroDense)

  • EuroDense starts from Stella-400M and uses translated data spanning nine languages, with Stella-400M for first-stage self-distillation and Pplx-Embed-v1-4B as the second-stage teacher.Its final stage uses 11 machine-translated retrieval datasets, randomly sampling available translations for each triplet component.Source: §3.5 · p. 4

4 Evaluation

Evaluation spans 41 Polish retrieval tasks and 150 multilingual datasets, showing strong quality-size trade-offs for PolDense and leading aggregate performance for EuroDense among smaller multilingual models.

4 Evaluation

  • The evaluation covers 41 Polish retrieval tasks and 150 multilingual datasets spanning all nine supported languages.The Polish tasks are from PIRB; the multilingual collection draws on established suites and standalone tasks.Source: §4 · pp. 4–5
  • Comparisons include multilingual embedding families and Polish-specific Stella-PL-Retrieval and Silver-Retriever-Base-v1 models.The multilingual set includes Multilingual-E5, Qwen3-Embedding, Pplx-Embed, Jina-Embeddings, Snowflake-Arctic-Embed, and BGE.Source: §4 · p. 5
  • Contrastive fine-tuning adds 1.43–3.14 NDCG@10 points for PolDense and 0.45 points for EuroDense on the multilingual suite.Each training stage improves retrieval quality, with overall gains tending to be larger for smaller models.Source: §4 · p. 5
  • PolDense-1B obtains the highest NDCG@10 among larger models on PIRB, outperforming evaluated models with up to 9B parameters.These include BGE-Multilingual-Gemma2 (9B), INF-Retriever-v1 (7B), and Qwen3-Embedding-8B.Source: Table 1 · p. 5, §4 · p. 6
  • PolDense variants form the Pareto frontier across evaluated sizes, with quality increasing as capacity grows and smaller models offering resource-constrained deployment options.Figure 2 plots mean NDCG@10 against model size; points farther upper left indicate a more favorable trade-off.Source: Figure 2 · p. 5, §4 · p. 6
  • Among small multilingual models, EuroDense-435M ranks first on both aggregate measures and leads in seven of nine languages, but trails larger models on both aggregates.It remains competitive with larger models particularly for Polish and German.Source: §4 · p. 6, Table 2 · p. 6

5 Conclusions

The paper presents compact Polish and multilingual retrievers whose results show favorable quality-efficiency trade-offs, including strong performance against larger models.

5 Conclusions

  • Across 41 Polish tasks, PolDense-1B establishes a new state of the art on PIRB, while EuroDense-435M ranks first among evaluated multilingual models below 1B parameters and leads in seven of nine languages.The authors conclude that these results show compact retrievers can be competitive with substantially larger models.Source: §5 · p. 6

Limitations

The reported evidence is limited to retrieval quality and parameter efficiency, and computational constraints narrow the training-data and multilingual-scaling comparisons.

Limitations

  • PolDense and EuroDense were designed and evaluated for document retrieval, not semantic similarity, clustering, or few-shot classification.They may underperform more general-purpose models on those tasks, so results do not establish overall embedding versatility.Source: Limitations · p. 7
  • The 400M and 1B PolDense models used 13 million distillation documents versus 50 million for smaller variants, confounding size comparisons with data volume.Only one 435M EuroDense variant was trained, covering nine selected European languages; multilingual scaling and generalization to other languages were not examined.Source: Limitations · p. 7

Ethical considerations

AI tools were used only for language editing, grammatical correction, and minor stylistic improvements, not for core scientific content.

Ethical considerations

  • AI tools assisted only with language editing, grammatical correction, and minor stylistic improvements, not core scientific content.Source: Ethical considerations · p. 7

A Training data

The first two training stages draw on question-like and passage-like corpora, with most datasets shared across both stages and model families. Three internally prepared corpora supplement external sources, while FineTranslations is used only for PolDense’s second stage.

A Training data

  • Most listed datasets serve both distillation stages and both model families; FineTranslations contributes only its Polish subset to PolDense’s second-stage corpus.The training-data summary distinguishes question-like from passage-like texts and reports each dataset’s source and record count.Source: A Training data · p. 13
  • Three internally prepared corpora are generated ELI5 answers, scientific abstracts, and a Polish–English Wikipedia parallel corpus.ELI5 answers and the abstracts and Wikipedia paragraphs are machine-translated into the other supported languages for EuroDense.Source: A Training data · p. 13
  • The training-data overview separately identifies query–passage datasets used in fine-tuning and marks their use for PolDense or EuroDense.Table 5 reports each dataset’s query and passage counts, name, and source.Source: A Training data · p. 13

B Teacher selection

In an English-only relational-distillation experiment, teacher choice did not track teacher retrieval score: Pplx-Embed-v1-4B produced the best student despite having the lowest teacher score. The comparison used the English NanoBEIR benchmark.

B Teacher selection

  • The controlled teacher comparison used an English-only corpus and a student initialized from mmBERT-base.The student was trained for five epochs with batch size 128 and a peak learning rate of 1×10−4.Source: B Teacher selection · p. 13
  • The experiment compared four teachers by mean NDCG@10 across NanoBEIR tasks, with the table reporting teacher–student score differences.The teachers were Qwen3-Embedding-8B, INF-Retriever-v1, QZhou-Embedding, and Pplx-Embed-v1-4B.Source: Table 3 · p. 13, B Teacher selection · p. 13
  • Pplx-Embed-v1-4B had the lowest teacher score but yielded the best student result and smallest teacher–student gap in the NanoBEIR comparison.Qwen3-Embedding-8B produced a slightly weaker student, while INF-Retriever-v1 and QZhou-Embedding yielded substantially lower scores.Source: B Teacher selection · p. 13, Table 3 · p. 13

C Evaluation data

The evaluation combines language-specific and multilingual retrieval datasets, covering Polish and nine European languages. Across 150 multilingual tasks, EuroDense-435M ranks among the top three models on 97 tasks and places first on 62.

C Evaluation data

  • Tables 4 and 5 summarize training data rather than evaluation results: the former covers the first two distillation stages and the latter fine-tuning.Their PD and ED columns identify whether datasets were used for PolDense or EuroDense.Source: Table 4 · p. 14, Table 5 · p. 14
  • English evaluation included 22 BEIR and RTEB tasks, using only publicly available RTEB tasks and excluding code retrieval.BEIR covers varied retrieval settings and domains, including argument retrieval, fact checking, and domain-specific question answering.Source: C Evaluation data · pp. 14–15
  • The evaluation also includes language-specific sets: German covers six tasks, French nine, and Spanish uses multilingual datasets plus three Spanish-specific datasets.German tasks include legal decisions and long-document retrieval; French includes MTEB-French retrieval tasks and the English–French CLIRudit dataset.Source: C Evaluation data · p. 15
  • Dutch evaluation uses 25 tasks, while Italian includes seven tasks and Portuguese combines multilingual datasets with three Portuguese-specific datasets.Russian evaluation uses rusBEIR and four tasks from multilingual datasets.Source: C Evaluation data · pp. 15–16
  • The evaluation draws primarily on monolingual benchmarks and multilingual datasets, adapting publicly available tasks for language-specific coverage.Polish evaluation used PIRB’s 41 multidomain tasks, including web-service questions and answers, while RTEB tasks ran through the MTEB framework.Source: C Evaluation data · pp. 13, 14, 16

D.1 Multilingual evaluation results for individual tasks

  • EuroDense-435M had the greatest average-score advantage over the second model in Polish, while its task wins were distributed across languages and competitors.It won nearly half of the Polish tasks; results on the remaining tasks were spread among several models.Source: §D.1 · p. 16
  • Language-level results were mixed: EuroDense led Dutch by 1.23 percentage points and narrowly led Russian, but ranked second in English.It achieved the best result on 15 of 25 Dutch tasks; in English it had the most task wins, five, yet ranked behind Pplx-Embed-v1-0.6B overall.Source: §D.1 · p. 16
  • Across 150 multilingual tasks, EuroDense-435M ranked among the top three models on 97 tasks (64.67%) and placed first on 62 (41.33%).Tables 6 and 7 report detailed results for ten multilingual models across the evaluated languages.Source: §D.1 · p. 16, Table 6 · p. 17, Table 7 · p. 18

D.2 Evaluation of monolingual models

PolDense-1B led the Polish comparison on 26 tasks, while EuroDense-435M ranked strongly across the language-specific evaluations, with its task-level wins varying by language.

Polish

  • 26 tasks (63.41%) went to PolDense-1B in Polish, while PolDense-400M ranked second 23 times and often outperformed the larger Stella-PL-Retrieval-8K.PolDense-150M ranked third 12 times and outperformed larger models in 10 of those cases.Source: §D.2 · p. 19, Table 8 · p. 21

German

  • EuroDense-435M led four German tasks and placed second on two others, behind Jina-Embeddings-v2-Base-DE and German-English-BGE-M3.It significantly outperformed the second-place model on LegalQuAD-de and the German part of MLDR.Source: §D.2 · p. 19, Table 9 · p. 21

French and Spanish

  • EuroDense-435M won seven French tasks (77.78%) and ranked among the top three on every task; in Spanish, it led nearly all tasks, placing second only on MLDR.French-BGE-M3 beat it once, while Spanish-GTE-Multilingual-Base beat it on the Spanish MLDR subset.Source: §D.2 · p. 19, Table 10 · p. 22, Table 11 · p. 22

Italian

  • EuroDense-435M outperformed Italian competitors on nearly all tasks, with WikipediaQA-ita the exception, where Multilingual-E5-Large-ita did better.Source: §D.2 · p. 19, Table 12 · p. 22

Portuguese

  • EuroDense-435M won every Portuguese task except JurisTCU-por, where Serafim-900m-Portuguese-pt-Sentence-Encoder-ir performed better.EuroDense showed particularly strong advantages on the Portuguese MLDR subset and Pira-por.Source: §D.2 · p. 19, Table 13 · p. 23

Dutch

  • 24 of 25 Dutch tasks went to EuroDense-435M; it lost only LegalQA-nl and beat second-place E5-Large-trm-nl by nearly 10 percentage points on average.Source: §D.2 · pp. 19–20, Table 14 · p. 23

Russian and English

Loading 2609.12913v2…