PaperlayerView exact revision on arXiv

Source-linked AI summary

sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak

Marek Šuppa, Ivan Vykopal, Andrej Ridzik, Kristián Sopkovič, Natália Kňažeková, Jaroslav Kopčan, Miroslav Blšták, Viktória Ondrejová, Daniel Hládek, Michal Gregor, Martin Tamajka, Marián Šimko

arXiv:2610.09152v1cs.CLcs.AI

TL;DR

Slovak is under-served in unified generative-LLM evaluation and in multilingual benchmarks that omit it or rely mostly on machine translation. sk-bench evaluates 55 open- and closed-weights models with a native-first suite, finding that provenance, adaptation, and reasoning relate differently to Slovak model performance.

  • Problem

    Slovak remains under-served by unified generative-LLM evaluation and multilingual benchmarks that omit it, offer minimal Slovak signal, or rely mostly on machine translation.

    Source: §1 · p. 1

  • Method

    sk-bench combines native Slovak data with selected translated proxies and evaluates 55 open- and closed-weights models under one harness.

    Source: §1 · pp. 1–2

  • Results

    Native and translated data preserve closed-form rankings (ρ ≥0.98), but translation separates top models less well, while human-authored and generated QA rankings differ (ρ = 0.72).

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results support using native data where translation fails, planning instruction repair after language adaptation, and enabling test-time reasoning before scaling up.

    Source: §1 · p. 2

    The benchmark covers textual single-turn evaluation in standard written Slovak, not dialogue, multi-turn agentic tool use, multimodality, code generation, or dialectal and colloquial variants.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33 scored task variants) across ten skill categories. Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK with Slovak-adapted instruction checkers and native Chiby/SKJ1 resources for Slovak grammar and morphology. We evaluate 55 open- and closed-weights models under one harness. The best open model trails proprietary APIs by 12.6 points. Model rankings are similar for native and translated closed-form data ($\rho\geq0.98$), though translation separates the strongest models less well. By contrast, human-authored and LLM-generated QA questions rank models differently ($\rho=0.72$). For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points. A small instruction set restores three quarters of that loss. Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above. Together, these findings suggest four design lessons for other under-resourced languages: use native data where translation fails, plan instruction repair after language adaptation, enable test-time reasoning before scaling up, and avoid overinvesting in target-language prompts. We release the data and code at https://github.com/slovak-nlp/sk-bench

1 Introduction

Slovak generative-LLM evaluation and Slovak-specific signal in multilingual benchmarks remain limited, despite substantial Slovak NLP resources. sk-bench combines native and selected translated data in one harness and uses it to study data provenance, language adaptation, and test-time reasoning.

1 Introduction

  • Slovak generative-LLM evaluation under a unified harness is lacking, while multilingual suites often omit Slovak or provide minimal signal and rely mostly on machine translation.The paper notes that machine translation inherits source-language and cultural biases and can leave exploitable artefacts.Source: §1 · p. 1
  • The benchmark studies whether data provenance changes model rankings, what Slovak adaptation costs and instruction tuning repairs, and whether test-time reasoning transfers.The three research questions separate translation from question generation, compare continued pretraining and instruction tuning across bases, and evaluate nine models in standard and reasoning modes.Source: §1 · pp. 1–2
  • sk-bench evaluates 55 open- and closed-weights models across 30 tasks and ten skill categories under one harness, with a unified Slovak leaderboard.Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK and native Chiby/SKJ1 resources that could not be built by translation.Source: §1 · p. 2
  • The harness uses one generate-and-match pipeline for open-weights and closed API models, and the paper releases prompts, run scripts, and per-task breakdowns.The authors present four design lessons for under-resourced-language benchmarks, grounded in findings on provenance, adaptation, and reasoning.Source: §1 · p. 2

2 Related Work

Prior Slovak and multilingual benchmarks cover subsets of NLP tasks, while sk-bench extends Slovak evaluation to a broader generative-LLM suite. Its paired provenance design also enables within-skill comparisons of native, translated, and generated data.

2 Related Work

  • skLEP is the closest Slovak precedent, bundling nine encoder-style NLU tasks; sk-bench reuses some subtasks and adds generative-LLM task types and Slovak exams.Other Slovak resources cover extractive QA, NER, summarisation, sentiment, STEM competition problems, and embeddings rather than a broad generative evaluation suite.Source: §2 · pp. 2–3
  • Single-language native benchmarks include C-Eval, IndoMMLU, ArabicMMLU, and BenCzechMark; sk-bench adds a per-skill contrast across native, translated, and LLM-generated data.That contrast tests whether a translated proxy ranks models reliably for a skill; no native-first generative-LLM suite of this breadth previously existed for Slovak.Source: §2 · p. 3
  • Slovak appears in multilingual suites such as Belebele, Okapi, and MultiWikiQA, while Multi-LoKo contrasts locally sourced and translated data across languages.sk-bench makes that provenance contrast per skill within one language; Chiby and SKJ1 pair naturalistic Slovak error data with expert-authored items across nine error types.Source: §2 · p. 3
  • sk-bench builds on established evaluation-harness and benchmark lineages, and adapts IFEval to Slovak through multilingual M-IFEval.The cited foundations include LightEval, HELM, lm-evaluation-harness, GLUE/SuperGLUE, BIG-bench, and XTREME.Source: §2 · p. 3

3 The sk-bench Task Suite

sk-bench combines native Slovak, Slovak domain, and translated or generated proxy data, making provenance explicit for comparisons of model rankings. The suite prioritises native data and includes tasks spanning language-specific skills and domains where native resources are limited.

3 The sk-bench Task Suite

  • The suite combines native Slovak data, Slovak domain data with professional or educational labels, and translated or generated proxies where no native benchmark exists.The benchmark keeps provenance explicit because it tests whether Slovak model rankings remain stable across data sources.Source: §3 · p. 3

3.1 Design Principles

  • Where possible, each skill is measured with differently sourced datasets, holding the skill fixed while comparing model rankings across provenance.New native datasets fill gaps where no native resource existed, enabling the provenance analysis.Source: §3.1 · p. 3

3.2 Inclusion Criteria

  • Dataset selection prioritises native Slovak data, redistribution-friendly public licences, test splits of at least 100 examples, and automatic reproducible scoring.Translated datasets are admitted only when no native equivalent exists; datasets failing the size or scoring criteria are excluded from main results.Source: §3.2 · p. 4

3.3 The Skill Categories

  • Native tasks cover classification, human-authored QA, exams, grammar and morphology, summarisation, instruction following, and long-context retrieval.Chiby and SKJ1 test grammaticality, Slohry tests inflectional morphology, and IFEval-SK uses checkers adapted to Slovak word boundaries, casing, and punctuation.Source: §3.3 · p. 4
  • Translated proxies cover subject knowledge, reasoning, and commonsense, while MultiWikiQA-SK supplies generated QA questions based on Slovak Wikipedia.The authors check for generator self-preference because Gemini 1.5 Pro generated the questions and belongs to a model family they evaluate; they report finding none.Source: §3.3 · p. 4

4 Evaluation Harness

The evaluation harness applies a shared generation-and-matching pipeline to open and closed models, with common prompts, few-shot policy, and scoring logic. It aggregates task metrics into reporting groups while accounting for output parsing and format variation.

4 Evaluation Harness

  • The harness uses the same prompts, few-shot policy, and scoring logic for open and closed models, generating text and matching it against gold answers or allowed labels.Open models run through vLLM and closed models through LiteLLM; task token budgets vary by task, and hand-written Slovak templates have parallel English versions.Source: §4 · p. 4
  • A fixed-order parser strips chat-template artefacts, and refusals or unresolvable outputs count as incorrect rather than being dropped.Small models in reasoning mode can exhaust the token budget before producing a parseable answer, while this rarely matters for standard models.Source: §4 · p. 4
  • The main score is an unweighted mean of reporting-group averages, with task macro-averaging within each group to prevent large datasets from dominating.Metrics include accuracy for classification and multiple choice, quasi-exact-match and quasi-F1 for QA, summarisation metrics, and strict/loose accuracy for IFEval-SK.Source: §4 · pp. 4–5

5 Experimental Setup

sk-bench evaluates 55 open-weight and proprietary models, including Slovak-adapted instruction-tuned models, under shared Slovak prompts and scoring. Its zero-shot results are paired with few-shot and English-prompt runs, and the release records materials needed for reproducibility.

5.1 Models Evaluated

  • 55 models span four size tiers from 350M to 1.1T parameters, including 49 open-weight and six proprietary models.Three Slovak base models were also instruction-tuned on the Slovak TaCo Alpaca split.Source: §5.1 · p. 5

5.2 Inference, Decoding, and Prompt Language

  • The default evaluation is zero-shot with greedy decoding, matched per-task token budgets, and Slovak instruction templates; reasoning mode uses an 8,192-token budget.Parallel English templates are available for each task.Source: §5.2 · p. 5

5.3 Few-Shot Policy

  • Three-shot runs use random examples from each task’s training or validation split, with both Slovak and English prompts.The primary results remain zero-shot.Source: §5.3 · p. 5

5.4 Reproducibility

  • The release includes prompts, scripts, configurations, settings, and raw outputs; API calls record model snapshots and dates, while open-weight generation is greedy.Closed APIs were queried once per model-task pair; prompt variants provide a spread that bounds prompt-induced variation.Source: §5.4 · p. 5
  • Table 1 reports zero-shot Slovak group means and an overall average, with prompt-template standard deviations shown as ±.The displayed models are the top two open-weight models per size tier and the top three proprietary APIs; records are determined across all 55 models.Source: Table 1 · p. 6

6 Results and Analysis

Across the benchmark, proprietary APIs lead the Slovak leaderboard, while rankings vary by data provenance and the effects of adaptation and reasoning depend on task and model capacity. Translation largely preserves closed-form rankings, but QA question provenance, language adaptation, and reasoning reveal important differences; prompt language, few-shot examples, and context length have more limited effects.

6 Results and Analysis

  • Gemini 3.1 Pro scores 72.6 overall, while the strongest open-weight model, gpt-oss 120B, scores 60.0.The open–closed gap is widest on instruction following; the reported summarisation ranking changes with the metric used.Source: §6 · p. 6

6.1 RQ1: Where Does Data Provenance Reorder Rankings?

  • NLI rankings are nearly identical for native and translated data (ρ = 0.99; identical top-10), while multiple-choice rankings correlate at ρ = 0.98 but share only 7 of 10 top models.The frontier size bands are too small to estimate divergence reliably.Source: §6.1 · p. 7, Table 2 · p. 7
  • Human-authored and LLM-generated QA rankings agree only at ρ = 0.72, with top-10 overlap of 6/10; disagreement is greatest below the frontier.The contrasts use differently sourced datasets rather than matched items, so content differences and provenance are mixed; an item-matched control is left to future work.Source: §6.1 · p. 7

6.2 RQ2: What Does Slovak Adaptation Cost, and Can It Be Repaired?

  • Qwen3-14B’s continued Slovak pretraining lowers its overall score from 47.4 to 33.5, improving grammar by 6.8 while reducing classification, reading, and knowledge scores.This is one adapted/base pair and supports a case study, not a population-level claim.Source: §6.2 · p. 8
  • Qwen3-SK instruction tuning gains +10.3 overall and restores about three quarters of its adaptation loss, but the tuning effect varies substantially by base model.Instruction following improves for two bases and is unchanged for the third; the small Alpaca set reliably teaches instruction format but does not add knowledge or reasoning.Source: §6.2 · p. 8

6.3 RQ3: Does Test-Time Reasoning Transfer to Slovak?

  • Models of 9B or more gain +8.5 to +12.5 overall from test-time reasoning, concentrated in instruction following, while the two smallest Qwen3.5 models regress by −16.8 and −9.7.Small-model regressions occur when the reasoning trace exhausts the token budget before a parseable answer is emitted.Source: §6.3 · p. 8

6.4 Robustness: Prompt Language, Few-Shot, Context Length

  • English prompts shift overall scores by only +0.4, while three-shot prompting yields +2.0 with Slovak and +1.6 with English across 52 models.On matched haystacks, Slovak retrieval declines from 58.1 at 4k to 54.4 at 16k and tracks English throughout, with no Slovak-specific length cliff up to 16k.Source: §6.4 · pp. 8–9

7 Conclusion

sk-bench evaluates Slovak across ten skills and 30 datasets, and the best open model trails proprietary APIs by 12.6 points. The authors draw four lessons for under-resourced-language benchmarks, including native data where translation fails, instruction repair after adaptation, and test-time reasoning before scaling.

7 Conclusion

  • The best open model trails proprietary APIs by 12.6 points in sk-bench’s evaluation of 55 open- and closed-weights models.The benchmark covers ten skills and 30 datasets; eleven resources are introduced or first packaged for generative-LLM evaluation.Source: §7 · p. 9
  • Translation preserves closed-form rankings (ρ ≥0.98) but separates top models less well; native data matters for grammar and morphology and for QA, where human- and LLM-generated questions rank models differently (ρ = 0.72).Grammar and morphology cannot be supplied by translation.Source: §7 · p. 9
  • Continued Slovak pretraining raises Qwen3-14B’s grammar score by 6.8 points but lowers its overall score from 47.4 to 33.5; a small instruction set recovers three quarters of the loss only when the base retains capability.The authors characterize the overall-score drop as catastrophic forgetting (Luo et al., 2023).Source: §7 · p. 9

L3: Enable test-time reasoning before scaling

  • Test-time reasoning adds 8.5 to 12.5 points for models of 9B and above, mostly in instruction following, but Qwen3.5 0.8B and 2B lose 16.8 and 9.7 points.The authors report that the smallest models exhaust their token budget before answering.Source: L3: Enable test-time reasoning before… · p. 9
  • Switching every template to English changes the overall score by only +0.4 points, while grammar shifts by −2.6.The authors advise against over-investing in target-language prompts.Source: L3: Enable test-time reasoning before… · p. 9
  • The authors recommend that under-resourced-language benchmarks include a native paired contrast per skill and re-implement instruction checkers natively rather than translating them.These recommendations follow from the benchmark’s findings on provenance, grammar, and morphology.Source: L3: Enable test-time reasoning before… · p. 9

Limitations

The limitations concern the benchmark’s coverage, translated and single-reference tasks, possible training-data leakage, score aggregation, and reproducibility of closed API results.

Coverage

  • The benchmark covers textual, single-turn evaluation in standard written Slovak, not dialogue, agentic tool use, multimodality, code generation, or dialectal and colloquial variants.These are stated as areas the benchmark does not currently cover.Source: Limitations · p. 9

Translated tasks

  • Translated tasks inherit cultural and topical bias from English-speaking contexts, so the authors caution against treating them as measurements of Slovak knowledge per se.They retain these tasks because no native Slovak counterpart of equivalent breadth exists.Source: Limitations · p. 9

Single-reference summarisation

  • Single-reference SlovakSum and SME-Sum scores can under-penalize correct paraphrases and over-penalize stylistic divergence, so the authors treat them as relative rankings rather than absolute quality measures.Each dataset provides one human reference per article; ROUGE and chrF are the metrics identified in this caveat.Source: Limitations · p. 9

Test-set contamination

  • The authors have not measured or certified that test items are absent from model training; public answer keys and source documents leave exposure possible even when the exact benchmark format is new.The passage distinguishes benchmark-format, answer-key, and source-document leakage; scores on previously released datasets, particularly for closed models with undisclosed training data, should not be read as contamination-free.Source: Limitations · p. 10

Aggregation

  • Equal weighting of reporting groups with task macroaveraging avoids large translated tasks dominating a group, but example-weighted aggregation would yield different rankings.Per-task and per-group breakdowns are released so readers can re-aggregate with their own weighting.Source: Limitations · p. 10

Reproducibility caveats

  • Closed-weights APIs can change without notice, preventing guaranteed bit-identical reproduction later; open-weights results require the released harness and checkpoints and the same vLLM version.The reproducibility caveat distinguishes closed API results from open-weights results.Source: Limitations · p. 10

Ethical Considerations

The authors limit redistribution according to source licences, flag potentially offensive content, and describe the use and verification of AI assistants.

Ethical Considerations

  • Redistribution follows each dataset’s licence, and components derived from non-commercial or research-only sources must not be used commercially.Tasks whose licences forbid redistribution are run from their original sources.Source: Ethical Considerations · p. 10
  • Hate-speech and toxicity tasks may contain offensive material; the authors mark them in metadata and recommend IRB-style review for follow-up annotation.They report adding no personally identifying information beyond what was already public and performing no re-identification of public social-media comments.Source: Ethical Considerations · p. 10
  • AI assistants helped generate result tables and figures, script parts of the harness and analysis, and improve manuscript language; the authors say they verified the work and own the design, experiments, analyses, and conclusions.LLM use in benchmark data generation and annotation is marked by provenance labels and documented in dataset appendices.Source: Ethical Considerations · p. 10

A Lessons for Under-Resourced-Language Benchmark Builders

The section turns Slovak benchmark findings into guidance: use native contrasts where translation fails, plan instruction repair after language adaptation, and enable reasoning with attention to model capacity. It also reports small overall effects from changing prompt language, limits on tested context lengths, and evaluation costs.

A Lessons for Under-Resourced-Language Benchmark Builders

  • Native and translated data preserve broad closed-form rankings but separate the frontier less reliably; human-authored and generated QA rank models differently (ρ = 0.72).Native and translated NLI agree on the entire top 10, while multiple choice agrees on 7 of 10; grammar and morphology require native evaluation.Source: A Lessons for Under-Resourced-Language… · p. 16, Figure 4 · p. 17
  • Qwen3-14B adaptation trades broad capability for a grammar gain; subsequent instruction tuning restores knowledge, reading, and instruction following but reverses that grammar gain.Across three Slovak bases, instruction-tuning effects are negative, flat, then positive as base capability rises, while instruction following improves or holds.Source: Table 3 · p. 16, Figure 5 · p. 17
  • Changing to English prompts moves the overall score by only +0.4; prompt language matters mainly for grammar, and test items remain Slovak.The prompt-language comparison excludes Slovak-only IFEval-SK and Slohry.Source: A Lessons for Under-Resourced-Language… · p. 17, Table 5 · p. 18
  • Reasoning mode adds +8.5 to +12.5 overall for models of at least 9B, but the two smallest models exhaust the token cap before emitting parseable answers.Their answer-extraction groups collapse even as instruction and grammar scores improve; a budget-matched control remains future work.Source: A Lessons for Under-Resourced-Language… · pp. 16–17, Table 4 · p. 17
  • Benchmark builders should ship native paired contrasts per skill and reimplement instruction and verifier checkers natively, since Slovak word boundaries, casing, and diacritics break the original IFEval checker.Source: A Lessons for Under-Resourced-Language… · p. 17
  • 58.1→55.6→54.4 retrieval from 4k to 16k; the Slovak–English gap narrows and slightly reverses, but 16k is the longest tested context.The study cannot locate a breaking point for capable models without evaluating longer haystacks.Source: A Lessons for Under-Resourced-Language… · p. 18, Table 7 · p. 19
  • Few-shot raises four-group overall by only +2.0 (Slovak) and +1.6 (English), helping classification and reading/QA but slightly hurting knowledge and reasoning.The pattern is similar for Slovak and English prompts; examples mainly help with output formats and label sets.Source: A Lessons for Under-Resourced-Language… · pp. 18–19, Table 8 · p. 19

B Costs

  • Evaluation consumed 5,354 GPU hours and 15,500 CPU hours across 764,079 open-model prompts; proprietary API costs ranged from $102.40 to $3,833.67.Source: B Costs · p. 19, Table 9 · p. 20

C Full Dataset Catalogue

This appendix catalogs benchmark resources, evaluated tasks and scoring procedures, and metadata for evaluated open-weights models. Only nine catalogued checkpoints report knowledge or pretraining cutoffs, limiting cutoff-versus-dataset-date contamination reasoning; Appendix G documents dataset availability.

C Full Dataset Catalogue

  • The catalogue records each dataset’s evaluation size, provenance, skill, licence, and description, and marks resources introduced or first packaged with sk-bench.The marked resources span items built from scratch, newly curated from public institutional sources, and expert resources first packaged for generative-LLM evaluation.Source: Table 10 · p. 21

D.1 Knowledge & Reasoning

  • The knowledge-and-reasoning tasks combine translated multiple-choice benchmarks with native school exams, professional exams, grammar questions, and Slovak trivia.Examples include the 57-subject Okapi-MMLU-SK, Financial Exams, and SKJ1 grammar-correction items.Source: §D.1 · p. 24

D.2 Reading & QA

  • Reading and QA span translated passage comprehension, native grammaticality and school-exam tasks, long-context retrieval, and human-authored versus LLM-generated free-text QA.MultiWikiQA-SK questions are generated over Wikipedia passages and rephrased to reduce surface overlap; skLEP-QA supplies the human-authored counterpart.Source: §D.2 · pp. 24–25

D.3 Classification

  • Classification covers entailment, sentiment, toxicity, and hate speech, with native and translated entailment sets and distinct toxicity and hate-speech tasks.SentiSK is evaluated in three-class and binary forms, while ToxicSK and skLEP-hate distinguish general toxicity from explicit hate.Source: §D.3 · pp. 25–26

D.4 Generation; D.5 Grammar & Morphology

  • Generation evaluates abstractive Slovak news summarisation, while grammar and morphology tasks test lexical-semantic analogies and production of inflected forms.Summaries are scored with ROUGE-L; Word Analogies and Slohry use exact match, with Slohry covering four morphology subtasks.Source: §D.4 · p. 26, §D.5 · p. 26

D.6 Instruction Following

  • IFEval-SK checks machine-verifiable constraints including length, casing, keywords, format, and response language with Slovak-adapted programmatic checkers.Scoring uses prompt-level strict accuracy, and the Slovak templates and checker registry were created for this adaptation.Source: §D.6 · p. 26

E Scoring and Parsing Details

  • About 1% versus 42%: small models’ multiple-choice accuracy falls to about 1% when reasoning-mode generations exhaust the budget, compared with 42% in standard mode.The harness counts refusals and unresolved outputs as incorrect; free-text QA normalization preserves Slovak diacritics and does not stem.Source: E Scoring and Parsing Details · pp. 26–27

F Model Catalogue

  • The open-weights model catalogue records parameter count, licence, published knowledge or pretraining cutoff, language coverage, pinned Hugging Face commit, and family report.Slovak-adapted and fine-tuned models are described separately in §5.Source: F Model Catalogue · p. 27
  • Only nine catalogued checkpoints report a knowledge or pretraining cutoff, limiting cutoff-versus-dataset-date contamination reasoning; Appendix G documents dataset availability instead.Entries without a published cutoff are marked “—”.Source: F Model Catalogue · p. 27

G Data Availability and Contamination

The paper distinguishes three forms of possible data leakage but performs no contamination measurement. It documents exposure opportunities and cautions that previously released native and translated data warrant particular care.

G Data Availability and Contamination

  • Leakage can involve benchmark-format prompts and answers, underlying answer keys, or source documents.The latter two correspond to annotation and raw-text contamination; benchmark-format leakage is specific to newly packaged suites.Source: G Data Availability and Contamination · p. 27
  • No contamination measurement was performed, so the reported numbers should not be read as contamination-free.Source: G Data Availability and Contamination · p. 27
  • Previously released native data and translated proxies warrant the most caution, especially for closed models with undisclosed training data and unpublished cutoffs.The newly packaged tasks are the only ones whose exact split was not previously circulating, though their source material may still have appeared in training.Source: G Data Availability and Contamination · p. 27

H Per-Task Detailed Results

The appendix supplies the full leaderboard and per-task results, then documents prompt and few-shot evaluation variants, a check of generated-QA self-preference, and curation details for new datasets.

H Per-Task Detailed Results

  • Table 14 reports all 55 models by reporting group and size tier, while Tables 15–16 provide per-task scores for standard and thinking-mode models.Task metrics include accuracy, quasi-exact-match, ROUGE-L, and IFEval strict accuracy; the leaderboard uses unweighted averages.Source: H Per-Task Detailed Results · p. 27, Table 14 · p. 30
  • 60.0 is the open-model score of gpt-oss 120B, which no larger frontier model exceeds; the best closed-model line marks Gemini 3.1 Pro at 72.6.The open frontier rises steeply through about 9B parameters and then flattens; 675B–1.1T MoEs sit below gpt-oss 120B.Source: Figure 7 · p. 29

H.1 English-prompt results

  • English prompts were run across tasks with English counterparts; IFEval-SK and Slohry are Slovak-only by construction.Table 17 contains the English per-task scores, with the Slovak-versus-English comparison discussed in the main text.Source: §H.1 · p. 32

H.2 Few-shot (3-shot) results

  • For 52 models, the appendix reports Slovak and English 3-shot per-task scores alongside the 0-shot results.Summarisation and instruction following were run zero-shot only.Source: §H.2 · p. 32

H.3 Generator self-preference check

  • Gemini models ranked lower on Gemini-generated MultiWikiQA-SK than on human-authored skLEP-QA, contrary to the tested self-preference pattern.Gemini 3.1 Pro ranked 1st on skLEP-QA and 4th on the generated set; Gemini 3.1 Flash-Lite ranked 30th and 38th, respectively. The authors attribute low QA ranking agreement (ρ = 0.72) to answerability rather than generator identity.Source: §H.3 · p. 32

I Collection and Curation of New Datasets

  • The curation appendix records how datasets were obtained, who curated them, and whether human annotators were involved.Chiby has a separate curation card, and IFEval-SK’s instruction registry is described separately.Source: I Collection and Curation of New… · p. 32

I.1 Scio

  • Scio was parsed and cleaned by the authors from admission-test papers, with answer-key alignment and no external annotators.The source artefact declares no licence.Source: §I.1 · p. 32

I.2 Autoškola

  • Autoškola uses official Slovak driving-licence questions across 35 topic categories, with the official answer key as gold; an LLM assisted dataset construction.Source: §I.2 · p. 32

J Summarisation Datasets Downsampling

SlovakSum and SME-Sum were downsampled by document length to reduce evaluation costs while retaining the datasets’ length distribution and summary statistics.

J Summarisation Datasets Downsampling

K Chiby-classification: Dataset Curation

This section documents the curation and annotation of Slovak sentence-correction classification data, a Slovak adaptation of IFEval, and prompt templates for benchmark tasks. The templates cover toxicity classification, summarisation, completion, question answering, multiple choice, grammar, causal reasoning, and word analogies, often with Slovak and English versions or multiple formulations.

K Chiby-classification: Dataset Curation

  • The Chiby v0.4 revisions were filtered to spelling, diacritics, and capitalisation pairs, with sentence-quality checks and cosine similarity above 0.99 used to retain matched Slovak sentences.Pairs with multiple error types, unsuitable lengths, URLs, or parsing problems were excluded.Source: K Chiby-classification: Dataset Curation · pp. 33–34
  • After preprocessing, error types comprised 67% spelling, 23% diacritics, and 10% capitalisation; 2,400 pairs were sampled to match these proportions.The sample contained 1,608 spelling, 552 diacritics, and 240 capitalisation pairs.Source: K Chiby-classification: Dataset Curation · p. 34
  • Three native Slovak annotators reviewed pairs individually, labeling the original, revision, or a corrected version; pairs too difficult to correct were discarded, leaving 1,716 pairs and 3,432 labeled rows.Because each pair had one annotator, inter-annotator agreement was not computed.Source: K Chiby-classification: Dataset Curation · p. 34, §K.1 · p. 34

L IFEval-SK: Slovak Instruction Registry

M Prompt Library

  • The prompt library provides three Slovak and English variants for most tasks, while IFEval-SK and Slohry use Slovak-only prompts.Templates include fields filled at run time.Source: M Prompt Library · p. 38

M.1 NLI — binary

  • Binary NLI prompts ask whether a hypothesis follows from or contradicts a premise, using either entailment labels or yes/no answers.Source: §M.1 · p. 38

M.2 NLI — 3-class

  • Three-class NLI prompts add a neutral relation alongside entailment and contradiction.Source: §M.2 · pp. 38–39

M.3 Sentiment — binary; M.4 Sentiment — 3-class; M.5 Hate speech — binary; M.6 Toxicity — binary

M.6 Toxicity — binary; M.7 Summarisation; M.8 Completion

  • The Slovak templates ask whether text is toxic (answer “áno” or “nie”), request a summary in at most 50 words, and prompt completion by choosing an option and giving its letter.Source: §M.6 · p. 40, §M.7 · p. 41, §M.8 · p. 41

M.9 Multiple choice — single answer; M.10 Multiple choice with context — single answer

  • Single-answer multiple-choice prompts require one option letter, while context-based versions explicitly restrict answers to the supplied context.The context templates state that each question has one correct answer.Source: §M.9 · p. 42, §M.10 · p. 43

M.11 Multiple choice — multiple answers

  • Multiple-answer questions allow one or more correct choices and request the corresponding letters separated by commas.Other variants request a list of the correct answer letters.Source: §M.11 · p. 44

M.12 Question answering — with context

  • Contextual question-answering prompts require Slovak answers based only on the supplied context, with some variants asking for direct answers without explanation.The templates place both the context and question in the prompt.Source: §M.12 · p. 45

M.13 Question answering — without context

  • Without-context question-answering prompts ask for direct answers in Slovak, with some variants explicitly instructing models not to explain.Source: §M.13 · p. 45

M.14 Chiby — grammaticality

  • Chiby grammaticality prompts judge whether a sentence is correct, treating spelling, morphology, and lexical errors as relevant while ignoring extra spaces.The instructions also distinguish dash and hyphen use and specify yes/no answers.Source: §M.14 · p. 46

M.15 SKJ1 — grammar correction

  • SKJ1 grammar-correction prompts ask models to select the grammatically correct form from five sentence options and respond with its letter.Source: §M.15 · p. 47

M.16 COPA — causal reasoning

  • COPA prompts ask models to select a plausible cause or consequence of a premise, including a direct consequence-identification formulation.The prompts require answers as “1” or “2” and provide English versions alongside the Slovak formulations.Source: §M.16 · pp. 47–48, §M.17 · p. 48

M.17 Word analogies

  • Word-analogy prompts ask models to infer the relationship between a word pair and supply the missing word, using several Slovak instruction formulations.Each formulation requests only the missing word; English counterparts are also provided.Source: §M.17 · p. 48
Loading 2610.09152v1…