PaperlayerView exact revision on arXiv

Source-linked AI summary

FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models

Joona Kytöniemi, Jousia Piha, Akseli Reunamo, Fedor Vitiugin, Farrokh Mehryary, Sampo Pyysalo

arXiv:2512.13330v1cs.CLcs.AI

TL;DR

Finnish model evaluation is constrained by English-dominated resources and shortcomings in data quality and prompt formulations. FIN-bench-v2 unifies Finnish benchmarks and screens tasks for reliable evaluation; its retained tasks show varied performance across tasks, domains, and prompts.

  • Problem

    Most evaluation resources are in English, and existing Finnish resources have data-quality and prompt-formulation shortcomings.

    Source: §1 · p. 1

  • Method

    The authors unify Finnish benchmarks, review translated data, create varied prompts, and retain tasks using learning-curve robustness criteria.

    Source: §1 · p. 1, §4.2.1 · p. 3, §4.2.4 · p. 5

  • Results

    Stable-signal screening excluded several widely used Finnish benchmarks, while larger instruction-tuned models showed performance variation across tasks, domains, and prompt formulations.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The publicly available suite provides Finnish cloze and multiple-choice evaluations across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, and alignment.

    Source: §6 · p. 9

    Because evaluated models were trained on partly undisclosed internet data, high scores may partly reflect memorization rather than reasoning.

    Source: Data Contamination and Memorization · p. 10

Abstract

from arXiv · show

We introduce FIN-bench-v2, a unified benchmark suite for evaluating large language models in Finnish. FIN-bench-v2 consolidates Finnish versions of widely used benchmarks together with an updated and expanded version of the original FIN-bench into a single, consistently formatted collection, covering multiple-choice and generative tasks across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, and alignment. All datasets are converted to HuggingFace Datasets, which include both cloze and multiple-choice prompt formulations with five variants per task, and we incorporate human annotation or review for machine-translated resources such as GoldenSwag and XED. To select robust tasks, we pretrain a set of 2.15B-parameter decoder-only models and use their learning curves to compute monotonicity, signal-to-noise, non-random performance, and model ordering consistency, retaining only tasks that satisfy all criteria. We further evaluate a set of larger instruction-tuned models to characterize performance across tasks and prompt formulations. All datasets, prompts, and evaluation configurations are publicly available via our fork of the Language Model Evaluation Harness at https://github.com/LumiOpen/lm-evaluation-harness. Supplementary resources are released in a separate repository at https://github.com/TurkuNLP/FIN-bench-v2.

1 Introduction

English-dominated evaluation resources hinder Finnish model development, while existing Finnish benchmarks have data-quality and prompt-formulation gaps. FIN-bench-v2 addresses these issues with reviewed data, varied prompts, and a unified evaluation format.

1 Introduction

  • Most evaluation resources are in English, hindering model development for Finnish, and existing resources have data-quality and prompt-formulation drawbacks.The original FIN-bench was an initial medium-scale effort for generative model evaluation in Finnish.Source: §1 · p. 1
  • Benchmark data quality is often unassessed across model sizes, and some machine-translated samples lack human review.The authors note that these issues may delimit tasks suitable for benchmarking different-sized models.Source: §1 · p. 1
  • Existing task formulations may overlook prompt sensitivity and be poorly suited to evaluating non-instruction-tuned models.The passage cites Voronov et al. (2024) on prompt sensitivity and Gu et al. (2025) on compatibility.Source: §1 · p. 1
  • FIN-bench-v2 combines systematic dataset-quality evaluation, diverse human-written prompts, and manual refinement of machine-translated GoldenSwag and XED.Multiple human annotators contributed to prompt creation.Source: §1 · p. 1
  • The suite is compatible with the widely used Language Model Evaluation Harness.Source: §1 · p. 2

2 Objectives for FIN-bench-v2

FIN-bench-v2 modernizes the original benchmark for maintainability and expands its task coverage. Its cloze and multiple-choice formulations are designed to accommodate differing behaviors of base and fine-tuned models.

2 Objectives for FIN-bench-v2

  • The project aims to make FIN-bench easier to maintain and use while expanding reliable evaluation across models of different sizes.Source: §2 · p. 2
  • The original FIN-bench relied on evaluation libraries that had become deprecated and difficult to use in 2025, motivating migration to the LM Evaluation Harness and HuggingFace Datasets.The modernization then developed into a broader effort to diversify task coverage.Source: §2 · p. 2
  • Cloze and multiple-choice formulations target different model behaviors: instruction-tuned models benefit from answer choices, while base models typically perform better with cloze completions.Source: §2 · p. 2

3 Tasks and Candidate Datasets for FIN-bench-v2

The candidate-dataset process combined reassessment of original FIN-bench tasks with investigation of new Finnish datasets across domains. The authors converted resources and created multiple prompt variants as part of building the suite.

3 Tasks and Candidate Datasets for FIN-bench-v2

  • Every original FIN-bench task and dataset was reassessed for retention, modification, or exclusion under the renewed evaluation framework.Source: §3 · p. 2
  • The candidate pool combined original FIN-bench tasks with datasets spanning domains including mathematics, geography, and medicine.The investigated candidates included ARC Challenge, Belebele, GoldenSwag, ScandiSent, SIB-200, SQuAD v2, TruthfulQA, ScaLA, XL-Sum, GSM8K, and MMLU.Source: §3 · p. 2
  • The authors used human-translated ARC Challenge, machine-translated resources, and manually corrected or expanded GoldenSwag and XED data.The passage identifies the ARC translation as Bijl de Vroe et al. (2025), GoldenSwag machine translation from LumiOpen (2025), XED from Öhman et al. (2020), and machine-translated XL-Sum and SQuAD from Nuutinen et al. (2025).Source: §3 · p. 2
  • Datasets were migrated into HuggingFace-compatible formats, and both cloze and multiple-choice prompts received five wording variants written by three people.The variants retain roughly the same meaning, supporting analysis of prompt sensitivity.Source: §3 · pp. 2–3

4 Methods and Evaluation Framework

FIN-bench-v2 candidate tasks are assessed using learning curves from pretrained models and four metrics designed to distinguish stable learning signals from noise. Tasks are retained only when they meet all four selection thresholds.

4 Methods and Evaluation Framework

  • The methods section describes how evaluation metrics determine whether candidate datasets are included in or excluded from FIN-bench-v2.Source: §4 · p. 3

4.1 Pre-training LLMs and other ANNs for Evaluation

  • Four 2.15B-parameter decoder-only models trained on 100B tokens from Finnish-language datasets, plus an English-data model, supply learning curves for task selection.Source: §4.1 · p. 3

4.2 Evaluation Metrics

  • Four metrics—monotonicity, signal-to-noise ratio, non-random performance, and model ordering consistency—assess whether tasks show stable progressive learning rather than volatile behavior.Source: §4.2 · p. 3

4.2.1 Monotonicity Index

  • Monotonicity uses Spearman’s ρ to measure whether task performance improves or remains stable across checkpoints; reliable tasks average ρ ≥0.5 across runs.Source: §4.2.1 · p. 3

4.2.2 Signal-to-Noise Ratio

  • SNRAgg aggregates prompt-specific signal-to-noise differences across five prompt variants using their median, limiting the influence of extreme or unstable variants.Source: §4.2.2 · p. 4

4.2.3 Model Ordering Consistency

  • τ-Consistency averages Kendall rank correlations between consecutive checkpoints to assess whether competing models’ relative rankings remain stable during training.Source: §4.2.3 · p. 4

4.2.4 Non-Random Performance Coefficient

  • NRC measures the gap between a task’s empirical maximum and its random baseline, assigning zero when performance does not exceed that baseline.Source: §4.2.4 · pp. 4–5

4.2.5 Final Metric for Task Selection

  • Tasks are retained only if they meet all four criteria: ρ ≥0.5, SNRAgg > 0, NRC ≥0, and τ-Consistency ≥0.7.Source: §4.2.5 · p. 5

5 Results

Results compare purpose-trained models used for task selection with larger instruction-tuned models evaluated across multiple-choice and generative tasks. Performance varies by model, task, and prompt formulation, and several candidate tasks were excluded after failing stability criteria.

5.1 Evaluation Strategy

  • The evaluation covered purpose-trained decoder-only models and larger instruction-tuned models, using normalized accuracy for multiple-choice tasks and selected reference-answer metrics for generative tasks.Purpose-trained-model criteria runs used k-shot multiple-choice settings with k ∈ {0, 1, 5}, zero-shot generative runs, and one selected metric per task.Source: §5.1 · p. 5, §5.2 · p. 5

5.2 Criteria Assessment Using Purpose-trained Models

  • Several candidate tasks failed the stability criteria; increasing context from zero to five examples yielded only approximately 5% gains, so those tasks were excluded.The retained-task strategy included all formulations if any one formulation satisfied the criteria; ARC Challenge passed all four criteria in both CF and MCF, unlike empirical judgments.Source: §5.2 · pp. 5–6, Figure 1 · p. 6, Figure 2 · p. 6

5.3.1 Detailed Evaluation of Multiple-Choice Tasks

  • Gemma 3 27B was the most robust multiple-choice performer, while Poro 70B often favored cloze prompts and Scout often favored visible answer options.Task-specific exceptions included ScandiSent scores above 0.92 for all models and formats, and near-random GoldenSwag MCF scores despite strong CF results.Source: §5.3.1 · pp. 6–7, Figure 3 · p. 7

5.3.2 Detailed Evaluation of Generative Tasks

  • Llama Poro 2 70B led zero-shot SQuAD FI F1 at 0.31, while Gemma 3 27B led TruthfulQA ROUGE-1 Max at 20.3 versus Scout’s 14.0.In one-shot SQuAD, Gemma and Scout roughly doubled F1 to 0.59, while Poro 70B fell to 0.16; one-shot TruthfulQA scores were near zero for all models.Source: §5.3.2 · p. 7, Figure 4 · p. 7

5.3.3 Effect of Prompt Variants

  • Belebele MCF prompt averages ranged from approximately 0.37 to 0.57 across models, showing that wording variants can substantially affect measured scores.Many other task and model combinations had closely clustered prompt-variant scores.Source: §5.3.3 · p. 7, Figure 5 · p. 8
  • MCF often scored slightly higher than CF, including on ARC Challenge, FIN-bench Emotions, and FIN-bench General Knowledge, but the pattern varied by model and task.Poro-34B-chat often showed similar or lower MCF scores; GoldenSwag was an exception, with zero-shot CF above 0.60 but MCF near random, while one-shot MCF recovered for all but Poro 34B Chat.Source: §5.3.3 · pp. 7–8, Figure 6 · p. 8

6 Conclusion and Future Work

FIN-bench-v2 modernizes and expands Finnish LLM evaluation in a unified framework, selecting tasks with stable learning-curve signals and evaluating larger instruction-tuned models. The authors release the suite and identify future work in broader domain coverage, dataset refinement, and contamination analysis.

Conclusion and Future Work

  • FIN-bench-v2 unifies Finnish multiple-choice and generative tasks across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, and alignment.Datasets were converted to HuggingFace Datasets and migrated to the LM Evaluation Harness.Source: §6 · p. 9
  • Datasets, prompts, and evaluation configurations are publicly released, with supplementary resources in a separate repository.The suite is available through the authors’ LM Evaluation Harness fork; the separate repository includes supplementary materials.Source: §6 · pp. 9–10
  • Five 2.15B-parameter decoder-only models supplied learning dynamics used to assess monotonicity, signal-to-noise, non-random performance, and model-ordering consistency.Tasks failing to provide stable, informative evaluation signals were excluded; selected tasks were also evaluated with larger instruction-tuned models across domains and prompt formulations.Source: §6 · p. 10
  • Future work targets additional domains, more challenging generative evaluation, dataset reannotation, and stronger contamination analysis as Finnish pretraining corpora emerge.The proposed domains include medicine, law, and safety-critical reasoning; dataset refinement targets translation quality, grammatical correctness, and label consistency.Source: §6 · p. 10

Limitations

The authors identify contamination, prompt sensitivity, and cultural bias as limitations affecting interpretation of benchmark performance. These concerns are especially relevant to open-weight models with undisclosed training data, prompt-based estimates, and translated tasks rooted in Anglocentric contexts.

Data Contamination and Memorization

  • Undisclosed pretraining data for larger open-weight models means contamination cannot be ruled out, so high scores may partly reflect memorization rather than reasoning.The authors contrast this concern with purpose-trained models whose training-data provenance is strictly controlled.Source: Data Contamination and Memorization · p. 10

Prompt Sensitivity and Brittleness

  • Averaging results across five reworded prompts gives a more stable capability estimate but does not fully remove prompt-engineering confounding.Semantically equivalent instructions can yield divergent performance, and phrasing may affect task outcomes.Source: Prompt Sensitivity and Brittleness · p. 10

Cultural and Linguistic Bias

  • Translated ARC and GoldenSwag retain Anglocentric assumptions, potentially penalizing models aligned with Finnish culture but lacking source-context knowledge.The authors note that linguistic translation does not necessarily adapt a dataset’s cultural context or common-sense assumptions.Source: Cultural and Linguistic Bias · p. 10

Resource Usage

The experimental phase used substantial computing resources across training and evaluation, with the final estimated computational load reported separately.

Resource Usage

  • 23 000 GPU hours were used overall, and the final computational load was estimated at 10.4 MWh.Training accounted for approximately 15 000 GPUh on LUMI; evaluation required 8 000 GPUh across LUMI and Mahti. The estimate excludes overhead from aborted runs.Source: Resource Usage · p. 10

A.1.1 ARC-Challenge-FI Prompt Templates

ARC-Challenge-FI adapts the ARC-Challenge science benchmark into Finnish and evaluates it with two prompt formulations. Five prompt variants per formulation are used to measure prompt sensitivity.

A.1.1 ARC-Challenge-FI Prompt Templates

  • ARC-Challenge-FI is a human-translated Finnish adaptation of ARC-Challenge, comprising 1 172 multiple-choice science questions without official splits.Cloze Formulation presents the question alone; Multiple-choice Formulation presents it with answer options. Each uses five prompt variants, p0–p4.Source: §A.1.1 · p. 14

Cloze formulation (CF) prompts arc_challenge_fi_cf_fbv2_p0

This section documents Finnish benchmark tasks and their cloze and multiple-choice prompt formulations, including prompt variants across question answering, reasoning, classification, knowledge, and alignment. It also describes k-shot assessment criteria and annotation review procedures for selected resources.

Cloze formulation (CF) prompts arc_challenge_fi_cf_fbv2_p0

A.1.2 Belebele-FI Prompt Templates

A.1.3 GoldenSwag-FI Prompt Templates

A.1.4 ScandiSent-FI Prompt Templates

A.1.5 SIB-200-FI Prompt Templates

  • SIB-200-FI prompts classify Finnish text into one of seven topics, with cloze inference or multiple-choice prompts that provide all topic labels.The archived subset has 1,004 examples, and each formulation has five prompt variants (p0–p4).Source: §A.1.5 · p. 19, Cloze formulation (CF) prompts… · p. 20

Cloze formulation (CF) prompts sib200_fi_cf_fbv2_p0

  • News-classification prompts ask models to assign text to one of seven topics, including politics, entertainment, science and technology, sports, travel, health, and geography.The supplied examples include both a cloze article prompt and multiple-choice formulations that present the topic list.Source: sib200_fi_cf_fbv2_p3 · p. 20, Multiple-choice formulation (MCF)… · p. 20

A.1.6 FIN-Bench Prompt Templates

FIN-Bench paraphrase

  • The paraphrase subset has 200 sentence pairs, asking models to judge whether texts have the same or nearly the same meaning in CF or choose “kyllä”/“ei” in MCF.The task probes semantic comparison beyond superficial lexical overlap.Source: finbench_hhh_alignment_mcf_fbv2_p4 · p. 27

FIN-Bench similarities/abstraction

  • The similarities/abstraction subset has 76 word pairs, requiring a shared-property or abstract-relation explanation in CF or selection of the best explanation in MCF.This task probes recognition of shared properties or abstract relations between words.Source: finbench_paraphrase_mcf_fbv2_p4 · p. 28

A.1.7 TruthfulQA-FI Prompt Templates

  • TruthfulQA-FI uses the Finnish openGPT-X/truthfulqax subset in a single default split, with mc1, mc2, and generative variants.The mc1 and mc2 variants differ in how answer options are treated.Source: §A.1.7 · p. 29
  • Five prompt variants (p0–p4) are defined for each TruthfulQA part to measure prompt sensitivity.SQuAD-FI likewise uses five generative prompt variants (p0–p4) for prompt-sensitivity measurement.Source: §A.1.7 · p. 29, §A.1.8 · p. 31

A.1.8 SQuAD-FI Prompt Templates

  • SQuAD-FI is a machine-translated version of SQuAD v2 with unanswerable questions removed and Finnish-translated titles; it has 84 688 training and 5 844 validation examples.It is a generative reading-comprehension task with no separate test split; each example provides a title, context, and question for generating a short answer span.Source: §A.1.8 · p. 31

A.6 K-shot Finetask Assessments

  • ARC-Challenge CF and MCF each pass monotonicity, low-noise, non-randomness, and ordering-consistency checks in the k-shot assessment.The table defines these checks as M, L, N, and O, respectively.Source: §A.6.1 · p. 43
  • Many cloze-formulation tasks pass all listed assessment checks, while the tables show failures on FIN-bench empirical judgments and TruthfulQA MC1.The checks are monotonicity, low noise, non-randomness, and ordering consistency; the table also lists a Fine column.Source: §A.6.2 · p. 44, §A.6.3 · p. 45

A.7 Multiprompt Writing Guidelines

  • The prompt-writing guidelines describe a task to create diverse prompts for Finnish datasets using dataset descriptions, task classes, prompt types, fields, and examples.The guidelines are presented in abridged form, with the original document available in the project repository.Source: Annotation Guidelines · p. 45, §A.7 · p. 45

Dataset description; Task class; Prompt type

1. Informed formulation; 2. Uninformed formulation; Dataset fields

  • Informed formulations put multiple-choice options in the prompt, while cloze and text-classification formulations omit choices or target labels from context; dataset fields indicate text placed in brackets.Natural-language generation requires no output formulation, but uses text related to dataset fields.Source: §1 · p. 46, §2 · p. 46, Dataset fields · p. 46

Guideline

  • The annotation instructions ask annotators to follow task class and prompt type, translate or write prompts from metadata, format target labels for informed formulations, and provide one prompt per task.The excerpt also says tasks are separated by header and page, and annotators mark their initials on prompt numbers.Source: Guideline · p. 46

A.8 GoldenSwag Annotation Guidelines

  • GoldenSwag annotation asks reviewers to judge whether an English-to-Finnish translation is adequate, correcting the Finnish text when it is not.The example pairs an English rope-skipping item with a Finnish translation.Source: An example annotation task · p. 46

A.9 XED (emotions_1k) Annotation Guidelines

  • XED review asks annotators to accept or reject assigned sentence labels to remove bad examples, judging whether each label is the most appropriate among eight emotion labels.The labels are anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.Source: §A.9 · p. 46

Related papers

Loading 2512.13330v1…