PaperlayerView exact revision on arXiv

Source-linked AI summary

FinGPT: Large Generative Models for a Small Language

Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo

arXiv:2311.05640v1cs.CL

TL;DR

Most open LLMs focus on languages with abundant data, leaving Finnish underrepresented despite limited resources for training large models. The study compiles Finnish data, trains monolingual models up to 13 billion parameters, continues pretraining 176-billion-parameter BLOOM as BLUUMI, and introduces FIN-bench. The models substantially outperform previously released Finnish models, while continued pretraining improves BLUUMI’s Finnish performance without significantly changing its English performance.

  • Problem

    Most open LLMs focus on languages with abundant data, leaving Finnish underrepresented despite limited resources for training large models.

    Source: §1 · p. 1

  • Method

    The study compiles Finnish data, trains monolingual models up to 13 billion parameters, continues pretraining 176-billion-parameter BLOOM as BLUUMI, and introduces FIN-bench.

    Source: §1 · p. 1, §2 · p. 2, §5.1 · p. 6

  • Results

    The models substantially outperform previously released Finnish models, while continued pretraining improves BLUUMI’s Finnish performance without significantly changing its English performance.

    Source: §5.2 · pp. 7–8, §6 · p. 9

  • Takeaways & Limitations

    The models and tools provide research resources and potential foundation models for Finnish and a template for developing models for other small languages.

    Source: §1 · p. 2, §6 · p. 9

    Limited Finnish text required repeated data epochs, which may reduce capability, especially for the largest monolingual model, and the models should not be used as-is for user-facing applications.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.

1 Introduction

This study addresses the limited data and model coverage available for Finnish by compiling Finnish resources and training generative models up to 13 billion parameters.

  • Finnish has limited resources for large-model training: fewer than 1% of Wikipedia and Common Crawl texts are Finnish, and models above one billion parameters were previously missing.Finnish is spoken by fewer than 6 million people, while most LLM work assumes hundreds of billions of readily available tokens.Source: §1 · p. 1
  • The study compiles broad Finnish data and trains monolingual models up to 13 billion parameters for 300 billion tokens.The dataset and training approach are designed for a small language with constrained data availability.Source: §1 · p. 1
  • It also continues pretraining the 176-billion-parameter BLOOM model with Finnish data to extend its Finnish coverage.This produces the model later called BLUUMI and combines Finnish with BLOOM’s original training data.Source: §1 · pp. 1–2
  • The work introduces FIN-bench and assesses multiple model aspects, including capability, alignment, bias, and toxicity.FIN-bench is a Finnish evaluation dataset based on BIG-bench and augmented with new tasks.Source: §1 · p. 2, Table 1 · p. 2

2 Models

The models use GPT-style architectures and BLOOM’s pretraining approach, with Finnish models trained from scratch and BLUUMI created by continuing BLOOM pretraining.

  • The study trains monolingual Finnish GPT-style models up to 13 billion parameters from scratch and continues pretraining 176-billion-parameter BLOOM to create BLUUMI.The Finnish models follow GPT-3 architecture settings, while BLUUMI mixes BLOOM’s original corpus with Finnish data.Source: §2 · p. 2
  • The models adopt BLOOM design choices including Alibi position embeddings, layer normalization, and the BLOOM software implementation.Continued pretraining is used because the 176-billion-parameter BLOOM model was substantially undertrained for its parameter count.Source: §2 · p. 2

3 Data

The study builds a broad Finnish pretraining corpus from web, reference, literary, news, social-media, and library sources, then applies filtering, deduplication, tokenization, and sampling procedures.

3.1 Data sources

  • The corpus spans diverse Finnish genres, registers, authors, and time periods through web crawls, Wikipedia, literature, news, books, social media, and other archives.Sources include Parsebank, mC4, CC-Fi, Fiwiki, Lönnrot, Yle, STT, ePub, Lehdet, Suomi24, Reddit-Fi, and ROOTS.Source: §3.1 · pp. 2–3

3.2 Preprocessing

  • Preprocessing combines dataset-specific extraction with approximate N-gram deduplication, heuristic Finnish-prose filtering, and N-gram perplexity filtering.The N-gram filter removes lines with perplexity above 100 000 and removes 15–20% of web-crawl text and 2–5% of social-media text.Source: §3.2 · pp. 3–4
  • Toxicity filtering removes 1–5% of most sources and up to 23% of CC-Fi text, while personal-data masking affects approximately 0.2% of characters.The toxicity classifier targets obscenities and identity attacks; it is not applied to news, Lönnrot books, or Wikipedia.Source: §3.2 · p. 4
  • The monolingual tokenizer uses byte-level BPE without Unicode normalization and a 131,072-token vocabulary to handle Finnish morphology.The larger vocabulary is motivated by Finnish’s agglutinative structure and high number of word forms.Source: §3.2 · p. 4

3.3 Data statistics

  • After preprocessing and oversampling open, high-quality resources, the final pretraining dataset contains 38 billion tokens.Register analysis finds broadly similar distributions across the three web datasets, with narrative text most frequent.Source: §3.3 · p. 4, §3.4 · p. 5

3.4 Register analysis

  • Register analysis covers mC4, CC-Fi, and Parsebank using categories such as narrative, interactive discussion, lyrical, how-to, and spoken text.Narrative is clearly most frequent in all three datasets, while how-to, spoken, and lyrical categories are small fractions.Source: Table 4 · p. 4, §3.4 · p. 5

4 Pretraining

Pretraining used the LUMI supercomputer and an adapted Megatron-DeepSpeed framework for AMD MI250X hardware, with parameters specified for both model-training approaches. Validation losses were stable, and larger models achieved lower loss.

4 Pretraining

  • Training used up to 192 nodes, corresponding to 1536 GPUs across AMD MI250X accelerator modules.Each node contained four MI250X GPUs, with each multi-chip GPU treated as two graphics compute dies.Source: §4 · p. 5
  • The adapted Megatron-DeepSpeed framework supported pipeline, tensor, and data parallelization across AMD GPUs and compute nodes.The adaptation used PyTorch ROCm, rocBLAS, MIOpen, RCCL, and Slingshot-fabric communication support.Source: §4 · p. 5
  • Training followed Brown et al. (2020) parameters for Finnish models trained from scratch, while BLUUMI retained BLOOM’s original parameters.The pretraining parameter values are reported in Table 5.Source: §4 · p. 5
  • Validation losses were stable across models trained from scratch, with larger models achieving lower loss.The curves use held-out validation data and 5-point moving-average smoothing.Source: §4 · p. 5, Figure 1 · p. 5

5 Evaluation

The evaluation introduces FIN-bench for Finnish few-shot assessment and examines model capability alongside alignment, bias, and toxicity. Monolingual models generally outperform prior Finnish models, while BLUUMI improves Finnish performance without significantly changing English performance, but alignment and residual toxicity remain concerns.

5.1 FIN-bench dataset

  • FIN-bench contains 3919 Finnish examples adapted from BIG-bench and augmented with tasks such as paraphrase, analogy, and emotions.The evaluation uses zero- to three-shot settings and summarizes performance with mean accuracy across tasks.Source: §5.1 · p. 6, §5.2 · p. 7

5.2 Few-shot results

  • Our best monolingual model exceeds the previous Finnish state-of-the-art result of 38.5% by over 10 percentage points, while BLUUMI exceeds it by over 20 points.Monolingual models at least match and usually outperform previously released Finnish models of comparable size.Source: §5.2 · p. 7
  • Performance generally rises with model size and the number of in-context examples, but the 13B models consistently underperform the 8B models, possibly because of overfitting.The authors estimate that approximately 10B parameters may be a practical limit for Finnish resources comparable to those available here.Source: §5.2 · p. 7
  • BLUUMI outperforms BLOOM on FIN-bench by 12–18 percentage points across few-shot tests, with no significant difference on English evaluations.The comparison indicates improved Finnish capabilities without compromising BLOOM’s existing English capabilities.Source: Figure 3 · p. 7, §5.2 · p. 8, Figure 4 · p. 8

5.3 Alignment

  • All models perform poorly on HHH alignment, exceed random performance only for the other category, and score particularly low for helpfulness.Observed size correlations are not statistically significant because differences remain within one standard deviation.Source: §5.3 · p. 8, Figure 5 · p. 8

6 Discussion and conclusions

The study introduces eight Finnish language models, FIN-bench, and evaluations of capabilities, alignment, bias, and toxicity. The models improve Finnish capabilities, while retaining limitations in alignment, bias, and toxic generation.

  • Eight models comprise seven monolingual Finnish models from 185 million to 13 billion parameters and the multilingual 176-billion-parameter BLUUMI.Source: §6 · p. 9
  • The study introduces FIN-bench and evaluates the models in few-shot settings, including alignment, bias, and toxicity.Source: §6 · p. 9
  • The models substantially outperform prior Finnish models, while continued pretraining improves BLUUMI’s Finnish capability without compromising its existing English capabilities.Source: §6 · p. 9
  • The models retain limitations in alignment, bias, and toxic content generation that the authors identify for future work.Source: §6 · p. 9, Figure 7 · p. 9

Limitations

The models’ internet-derived training data and limited Finnish resources constrain their safe use and reproducibility. The authors position them primarily as research resources rather than ready-to-deploy user-facing systems.

  • Internet-derived training data leave the models with established limitations, including alignment, bias, and toxicity problems despite filtering efforts.Source: Limitations · p. 10
  • The authors caution against using the models as-is in user-facing or high-impact applications, such as hiring decisions.Source: Limitations · p. 10
  • Limited Finnish text required repeated training epochs, which may reduce capability, especially for the largest monolingual model.Source: Limitations · p. 10
  • Use of National Library of Finland texts prevents complete replication without the library’s involvement.Source: Limitations · p. 10

A Timespan covered by Finnish datasets

The Finnish datasets span different periods, while web-based resources are dated by retrieval because their publication dates are difficult to assign. The Lönnrot dataset is an exception, covering mostly pre-1950 books.

B Comparison of mC4-Fi and CC-Fi datasets

mC4-Fi and CC-Fi use different Common Crawl subsets and processing pipelines, producing substantial URL differences. Their matching documents are textually similar but not identical, so URL overlap understates dataset distinctness.

C Full gender bias results on 13B model

The supplied passage lists toxicity dimensions rather than reporting gender-bias results for the 13B model.

C Full gender bias results on 13B model

  • The passage enumerates Identity attack, Insult, Obscene, Severe toxicity, Threat, and Toxicity as model-assessment dimensions.Source: D Toxicity scores · p. 15
  • No gender-bias comparison or numerical result is supplied in the passage.Source: D Toxicity scores · p. 15

F Full FIN-bench evaluation results

The supplied materials identify a figure with 3-shot results for each FIN-bench task plus HHH and a table presenting FIN-BENCH task examples.

F Full FIN-bench evaluation results

  • Figure 8 presents 3-shot results for each FIN-bench task together with HHH.Source: Figure 8 · p. 16
  • The supplied passages identify the evaluation visual and task-example table but provide no numerical comparisons or task-level outcomes.Source: Figure 8 · p. 16, Table 7 · p. 17

G FIN-bench examples

Related papers

Loading 2311.05640v1…