The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding

1 Introduction

• MTEB offers limited non-English support, translated datasets, and little domain coverage, while multilingual retrieval support is absent.These gaps are especially pronounced for non-English tasks.Source: §1 · pp. 1–2
• Scandinavian institutions rely on English-benchmark or bitext-mining proxies, whose performance may not transfer to their applications.The authors intend SEB to support informed model choices and standardized comparisons.Source: §1 · p. 2

1.1 Contributions

• SEB covers Danish, Swedish, Norwegian Bokmål and Nynorsk, and Bornholmsk, with tasks, domains, a model registry, and MTEB integration.The registry supports adding models and documenting evaluated implementations.Source: §1.1 · p. 2

2.1 Benchmarks

• Although MTEB includes more than 112 languages, Scandinavian representation is limited to two translated intent and scenario classification datasets, with no naturally occurring text.Its language coverage is concentrated in a handful of tasks, including bitext mining and translated datasets.Source: §2.1 · p. 2
• Existing Scandinavian benchmarks assess language understanding, but none focus on text-embedding tasks such as retrieval or clustering.Examples include Swedish Superlim, Norwegian NorBench, and cross-lingual ScandEval.Source: §2.1 · p. 3

2.2 Text Embeddings

• Multitask sentence-level multilingual embeddings are emerging, but comprehensive evaluation for multilingual tasks remains lacking.The authors state that this evaluation gap hinders assessment of model improvements.Source: §2.2 · p. 3

3.1 Design and Curation Rationale

• SEB comprises 24 datasets across at least 12 domains and nine tasks, with broad coverage for each language.The design prioritizes varied domains, usages, and embedding tasks.Source: §3.1 · p. 3
• SEB avoids adding translated English datasets to represent Scandinavian contexts and reduce possible inflation of multilingual-model performance.The authors retain existing translated MTEB datasets for compatibility, while noting the inflation hypothesis remains unproven.Source: §3.1 · pp. 3–4
• SEB relies on cross-lingual transfer among Scandinavian languages to estimate performance where language-specific datasets are limited.The rationale invokes linguistic similarities and shared cultural contexts.Source: §3.1 · p. 4
• A model registry documents evaluated implementations, while dataset limits and a public cache are intended to improve reproducibility and accessibility.Most datasets are capped at 2,048 examples, allowing most models to be run on a consumer-grade laptop.Source: §3.1 · p. 4

3.2 Datasets

• SEB datasets occupy distinct clusters; SNL Retrieval and NorQuad are highly similar, while SweFAQ represents public-sector FAQs.The authors attribute the SNL Retrieval–NorQuad similarity to their encyclopedic sources.Source: §3.2 · p. 4, Figure 2 · p. 5

4 Methodology

• SEB follows an MTEB-compatible approach to computing benchmark scores.Dataset construction and task formalization are described elsewhere in the paper.Source: §4 · p. 5

4.1 Models

• The model selection covers architectures, sizes, commercial APIs, and models claiming state-of-the-art results, grouped as self-supervised or supervised.The online dashboard includes additional models beyond those benchmarked in the paper.Source: §4.1 · p. 5
• Self-supervised candidates include BERT-family and XLM-R encoders, SimCSE dfm-encoder-large, and four Danish, Swedish, and Norwegian fastText models.The SimCSE model is trained on sentences from Danish Gigaword.Source: §4.1 · p. 5
• Supervised candidates include LaBSE, multilingual MiniLM and MPNet, SONAR, Swedish sentence transformers, and e5-mistral.The authors include SONAR because it claims improved performance over LaBSE.Source: §4.1 · pp. 5–6
• The comparison includes Cohere and OpenAI APIs and a translate-and-embed baseline using M2M100 with English e5 and multilingual e5 variants.For multilingual tasks requiring multiple languages, Danish is assumed as the origin language; language is assumed known for translation.Source: §4.1 · pp. 6–7

5 Results

• Commercial APIs lead overall, followed by public multilingual e5; the commercial–public gap exceeds eight points in retrieval, while open models outperform commercial solutions only in bitext mining.Source: §5 · p. 7
• Domain rankings are relatively stable, but e5-mistral-7b-instruct outperforms commercial solutions in legal, spoken-language, and fiction domains.Source: §5 · p. 7
• Pretrained encoders perform well on language acceptability and language detection, plausibly because natural-language pretraining preserves syntactic cues emphasized less by semantic-task models.Source: §5 · p. 7
• Contrastively trained or task-finetuned models typically outperform pretrained models, especially in retrieval, while LaBSE and SONAR excel at bitext mining.Source: §5 · p. 7
• Multilingual e5 models outperform translate-then-embed in almost all comparisons, including across model-size categories.The paper notes potential improvement with stronger translation and embedding models, but cites added computational costs and complexity as reasons for few benefits.Source: §5 · p. 7

5.1 Efficiency

• FastText has the highest throughput with average performance above multilingual XLM-R; e5 models offer peak performance or a strong performance–throughput–size balance.Figure 3 excludes commercial APIs; speed was measured on Dell PowerEdge C6420 CPUs.Source: §5.1 · p. 7, Figure 3 · p. 8

5.2 Limitations and Future Perspectives

• SEB’s coverage remains partial or absent for legal, governmental, and medical domains, and it lacks pair classification and document reranking.Source: §5.2 · p. 8
• The authors plan to continue expanding SEB and invite researchers to contribute evaluations across a broad range of languages.Source: §5.2 · p. 8

• SEB addresses the mainland Scandinavian text-embedding evaluation gap with 24 tasks, ten subtasks, and four task categories.Source: §6 · p. 8
• Evaluation of more than 50 models finds notable public–commercial and monolingual–multilingual performance gaps, especially in retrieval.The authors open-source SEB and integrate it with MTEB to encourage further development and community contributions.Source: §6 · p. 8

Checklist

• The checklist reports that code and a dashboard are available, and that dataset construction and task data splits are described.It also says compute details are specified in the acknowledgements.Source: Checklist · p. 14
• The authors report confidence intervals for rank, but not for individual tasks.Source: Checklist · p. 14
• The checklist says datasets are publicly available and that copies are retained in case they are removed from Hugging Face.Source: Checklist · p. 14
• The checklist states that DKHate contains offensive content and that applicable personally identifiable information was anonymized by dataset distributors.Source: Checklist · p. 14

A.1 Models

• Table 3 maps shortened model names from the main paper to their references and implementations.Source: §A.1 · p. 15

A.2 Domains Generalization

• Across domains, the e5 models and commercial APIs are generally the most competitive models.Source: §A.2 · p. 15

A.3 Dataset Embeddings

• Distinct dataset clusters in the embedding space suggest that omitting some datasets could bias evaluation; translated MASSIVE Intent and Scenario cover only a small subsection.Figure 4 shows 100 sampled documents per task embedded with embed-multilingual-v3.0 and projected using UMAP.Source: §A.3 · p. 15, Figure 4 · p. 17

A.4 Evaluation and Metrics

• SEB follows MTEB-like evaluation while allowing models to embed tasks differently, and reports accuracy, F1, V-measure, or nDCG@10 according to task.Classification uses repeated logistic-classifier training; bitext mining matches sentence pairs by cosine similarity, clustering uses mini-batch k-means, and retrieval ranks corpus documents by similarity.Source: §A.4 · pp. 16–17

A.5 Datasets Construction

• The benchmark reuses existing classification and bitext datasets, constructs clustering data from labeled corpora, and derives retrieval tasks from question-answer or news-summary datasets.For SNL and VG clustering, the authors selected a label hierarchy level and checked that it produced meaningful performance separation.Source: §A.5 · pp. 17–18

A.6 Datasets Statistics

• Table 5 provides dataset descriptions, descriptive statistics, task formalizations, and domain labels.Source: §A.6 · p. 18, Table 5 · p. 21

A.7 Long-term Availability and Stability

• Datasets are publicly hosted on Hugging Face at pinned revisions, with copies retained so they can be restored if removed.The authors state that dataset licenses make retaining copies viable.Source: §A.7 · p. 22

A.8 Results per Task

• The appendix directs readers to the online dashboard for an up-to-date overview of selected models’ results on each task.Source: §A.8 · p. 22