PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

Hafsteinn Einarsson

arXiv:2610.36230v1cs.CL

TL;DR

Icelandic encoder research had not yet established how far open-data ModernBERT training could go or how much target-language text controlled warm starts save. The paper trains and evaluates Icelandic encoders across model sizes and compares matched-shape English and multilingual initializations. Its 1B scratch model significantly beats IceBERT across eleven tasks, while warm starts with the Icelandic vocabulary match scratch training at equal text and can use substantially less.

  • Problem

    The paper revisits whether open-data Icelandic encoders can outperform IceBERT and whether warm-start benefits persist in controlled comparisons across source models, sizes, and target-language budgets.

    Source: §1 · p. 1

  • Method

    The authors train open-data ModernBERT encoders and evaluate 30 models under one protocol, comparing scratch training with same-shape English and multilingual warm starts across sizes and token budgets.

    Source: §1 · pp. 1–2, §5 · p. 4

  • Results

    The 1B scratch model significantly beats IceBERT on all eleven tasks, and the Ettin warm start ranks best on every ranked task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With the Icelandic vocabulary transplanted, warm starts were never significantly worse than scratch training on the same text, and from base size up were not significantly worse on four tasks using four times less text.

    Source: §1 · p. 2, §8 · p. 8

    The study covers one language and fine-tuning tasks; languages with less text or no same-shape source models may behave differently, and sentence embeddings, retrieval, and long-input task performance are untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

IceBERT, the reference encoder for Icelandic, found that a cleaned web crawl is enough to train a strong model and that starting from a multilingual model can help on some tasks. We revisit both findings with the ModernBERT architecture. We train Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, and rerun IceBERT's benchmarks and five newer tasks for 30 encoders under one protocol with confidence intervals. Our 1B model trained from scratch, to our knowledge the first monolingual Icelandic encoder of that size, is significantly better than IceBERT on all eleven tasks; warm-started from the English Ettin encoder of the same shape, it has the best score of all encoders on every task we rank. Because our model shapes match the English Ettin encoders and the multilingual mmBERT, we can also test the warm start under controlled conditions, at six model sizes and four token budgets. With our vocabulary transplanted, a warm start was never significantly worse than scratch training on the same Icelandic text, and from base size up it was not significantly worse on four tasks than scratch training on 4 times the text (8 times at base and 1B). As models grow, most of the transfer comes from the source model's transformer layers, so even an English source helps. Keeping the source vocabulary instead cost 5 F1 on question answering and nearly three times the compute.

1 Introduction

The paper revisits IceBERT’s claims about training strong Icelandic encoders on cleaned web data and transferring from multilingual models, using ModernBERT and controlled warm-start comparisons. It trains open-data encoders and evaluates them alongside prior models under a shared protocol.

1 Introduction

  • IceBERT found that a cleaned web crawl could train a strong Icelandic model and that multilingual initialization could help on some tasks.Its XLM-R-base warm start was best on NER and used about 8% of IceBERT’s GPU hours.Source: §1 · p. 1
  • The available open Icelandic sources provide 9B tokens after deduplication, compared with IceBERT’s 2.7B.ModernBERT, Ettin’s open English-and-code encoder ladder, and multilingual mmBERT motivate revisiting the earlier findings.Source: §1 · p. 1
  • IceBERT’s warm-start comparison varied vocabulary, data, model size, and training length, so it could not isolate the effect of starting weights.The authors’ same-shape English sources, and a same-shape multilingual source at base size, make controlled comparisons possible.Source: §1 · p. 1
  • The study trains Icelandic ModernBERT encoders from 12M to 1B parameters on open data and reruns IceBERT’s benchmarks plus five newer tasks for 30 encoders.The evaluation uses one protocol and bootstrap confidence intervals.Source: §1 · p. 1
  • The warm-start study compares English and multilingual sources of matching shape at six sizes and four token budgets, including an arm that separates transformer-layer transfer from embedding transfer.The models are released together with the scratch-trained models.Source: §1 · pp. 1–2
  • The 1B scratch model significantly beats IceBERT on all eleven tasks, while the Ettin warm start ranks best on every ranked task.With the paper’s vocabulary transplanted, warm starts were never significantly worse than scratch training on the same Icelandic text.Source: §1 · p. 2

2 Related work

Prior work established Icelandic and Nordic encoder baselines, introduced Icelandic evaluation tasks, and explored vocabulary transfer and continued pretraining. The paper identifies a gap in controlled encoder studies measuring warm-start data savings across model sizes and target-language budgets.

Icelandic and Nordic encoders

  • IceBERT trained Icelandic RoBERTa models, while Icelandic Gigaword studies found ELECTRA and ConvBERT beat a similar-sized multilingual model; ScandiBERT also included Icelandic and Faroese.Related Nordic work includes Norwegian encoder ladders and Finnish ModernBERTs, which led on long inputs but trailed XLM-R on most short-context tasks.Source: §2 · p. 2

Icelandic evaluation

  • ScandEval introduced Icelandic acceptability and QA datasets and found limited transfer between mainland Scandinavian languages and Icelandic and Faroese.Icelandic QA datasets include native-written NQiI and crowd-sourced RUQuAD; an audit found machine-translated test sets severely flawed, while Icelandic WinoGrande had no severely flawed items.Source: §2 · p. 2

Warm starts and vocabulary transfer

  • Vocabulary-transfer methods adapt source embeddings to a target tokenizer, while prior work suggests transformer layers can be partly language-independent.Studies of decoder warm starts report varied data savings, and XLM-R continued pretraining on Croatian and related languages reached scratch-trained model levels.Source: §2 · p. 2
  • The authors identify no prior encoder study that measures warm-start savings across several model sizes and target-language budgets or compares same-shape English and multilingual sources.Earlier IceBERT comparisons could not isolate warm-start effects because vocabulary, data, size, and training length differed.Source: §2 · p. 2

3 Data

The study builds its Icelandic training corpus from publicly available sources, filtering and deduplicating them before removing evaluation overlaps. The resulting corpus contains 9.02B tokens, with an additional CulturaX tranche used for models trained on twice the tokens.

3 Data

  • The corpus draws on nine open sources, including Icelandic Gigaword’s open portion, Wikipedia, Dynaword, MaCoCu, FineWeb-2, FinePDFs, HPLT 3.0, MADLAD-400, and IC3-v2.Restricted Gigaword news and books are excluded.Source: §3 · p. 2
  • A shared pipeline filters short, non-Icelandic, repetitive, or boilerplate documents, deduplicates across sources, and decontaminates against evaluation text using word 13-grams or short-sentence matches.Documents losing more than a fifth of their words to decontamination are dropped; 0.1% of each source is held out for validation.Source: §3 · pp. 2–3
  • 9.02B tokens remain after corpus processing; the doubled-token models add 0.34B surviving CulturaX tokens, a 3.7% increase.77% of CulturaX’s Icelandic documents already occur in the other sources.Source: §3 · p. 3
  • FineWeb-2 has near-duplicates for 84% of its documents, and about two thirds of HPLT, MADLAD, and IC3 documents do as well.Gigaword and FinePDFs contribute most of the text not found in another source; decontamination dropped 64,401 documents and shortened 111,092.Source: §3 · p. 3

4 Models

The paper uses ModernBERT architecture and Ettin-shaped model sizes, with an Icelandic-trained byte-level BPE tokenizer. Models are trained under 80- and 160-token-per-parameter schedules, except the 1B model, which uses only the 80-token schedule.

4 Models

  • The models follow ModernBERT and the Ettin size ladder, giving each model a same-shape English Ettin counterpart; the 1B model was added last and lacks a 160-token-per-parameter run.ModernBERT uses pre-normalization, GeGLU feed-forward layers, rotary position embeddings, and global attention every third layer, with 128-token sliding windows in the others.Source: §4 · p. 3
  • The byte-level BPE tokenizer has 32,768 tokens and splits held-out text into 1.54 tokens per word, versus 1.51 for IceBERT’s 50k vocabulary.A 50k vocabulary trained on this corpus would be 3.9% more compact, but its embedding matrix would be three quarters of the smallest model.Source: §4 · p. 3
  • At 80 tokens per parameter, Table 1 reports training-token counts and estimated node-hours by model size; models trained at 160 tokens per parameter see twice the tokens.The 1B model used 135 measured node-hours on 16 nodes and was not trained at 160 tokens per parameter.Source: Table 1 · p. 3
  • Training uses masked language modeling at a 30% masking rate, reduced to 15% during final decay, with schedules at 80 or 160 tokens per parameter.The first 90% of tokens use 1,024-token sequences and the final 10% use 8,192-token sequences; documents are packed without padding and kept separate by an attention mask.Source: §4 · p. 3

5 A warm start, revisited

The study controls warm-start comparisons by matching source encoders to scratch models in shape and keeping the Icelandic training and evaluation setup consistent. It varies the source, token budget, and pretraining seeds.

5 A warm start, revisited

  • Same-size Ettin sources match the study’s model shapes, and at base size mmBERT matches the shape; warm starts therefore change only starting weights.Architecture, data, tokenizer, schedule, and evaluation remain those of the scratch models.Source: §5 · pp. 3–4
  • Five arms compare random initialization, same-size English Ettin starts, Ettin layers with fresh embeddings, and two base-size mmBERT vocabulary treatments.The mmBERT arms either transplant the Icelandic vocabulary or retain mmBERT’s 256k Gemma tokenizer.Source: §5 · p. 4
  • Fast Vocabulary Transfer assigns each Icelandic token the mean embeddings and output biases of its Ettin-tokenizer pieces.Because FVT uses only the two tokenizers, the authors use it for both English and multilingual sources; they describe its estimate of embedding transfer as a lower estimate.Source: §5 · p. 4
  • Each arm is trained at 10, 20, 40, and 80 tokens per parameter, with shorter-budget schedules branched from the 80-token run.Each branch completes its own decay schedule, while sharing the main run’s warmup.Source: §5 · p. 4
  • Pretraining seeds vary by size and arm; the large scratch curve is a second run of its recipe and seed because the ladder’s large model kept no branch states.At 1B, scratch branches start from kept states of the 80-token run; the large scratch curve does not.Source: §5 · p. 4

6 Evaluation

Evaluation reruns IceBERT’s benchmarks and adds five newer Icelandic tasks. All encoders use a shared evaluation harness, tuned learning rates, multiple fine-tuning seeds, and hierarchical-bootstrap confidence intervals.

6 Evaluation

  • The evaluation adds five tasks to IceBERT’s six benchmarks: pooled extractive QA, linguistic acceptability, sentence-level error detection, and dependency parsing, alongside the benchmark reruns.QA uses pooled NQiI and RUQuAD test questions; acceptability is built ScaLA-style from two treebanks.Source: §6 · p. 4
  • Every encoder uses the same harness, with learning rates tuned on development data until the selected rate is interior to the search grid.At the selected rate, the authors run five seeds, with specified fold-based seed reuse for PoS tagging and WinoGrande.Source: §6 · pp. 4–5
  • Confidence intervals use hierarchical bootstrap resampling over pretraining seeds, fine-tuning seeds, and test items; model differences are paired over the same items.IcePaHC parsing is reported on the 58% of test sentences without a match; 45% of matchable IcePaHC test sentences occur in the corpus.Source: §6 · p. 5

7 Results

The 1B scratch model significantly outperforms IceBERT on all eleven tasks, while the Ettin warm start has the best mean on every ranked task. Results across newer tasks and controlled comparisons also show that warm starts can reduce Icelandic data needs, with benefits depending on task, size, and vocabulary.

7.1 The IceBERT benchmarks

  • The 1B scratch model significantly beats IceBERT on all eleven tasks, and the Ettin warm start has the best mean on every ranked task.The large Ettin warm start is also best or tied best below 1B on several tasks, while the scratch large model’s gains over IceBERT are more limited.Source: §7.1 · p. 5, Table 2 · p. 6

7.2 Further tasks

  • On ScaLA, the 1B scratch and warm-start models score 78.6 and 80.9 MCC, while mmBERT-base scores 50.1.On sentence-level error detection, the 1B models reach 53.5 MCC from scratch and 55.0 warm-started; dependency parsing separates most encoders little.Source: §7.2 · pp. 5–6, Table 2 · p. 6

7.3 Warm start against scratch

  • An English Ettin warm start reaches scratch’s 80-token-per-parameter MLM loss with 40 or fewer tokens at every size.With fresh embeddings, Ettin’s transformer layers recover 62–90% of the loss gain; their share of downstream gain grows with model size, though those estimates use one seed and noisy-difference ratios.Source: §7.3 · pp. 6–7, Figure 1 · p. 7
  • At base size, 20 tokens per parameter from Ettin are not significantly worse than scratch at 160 on any of four tasks; at 1B, 10 tokens suffice.At large size, 40 tokens per parameter from Ettin are not significantly worse than scratch at 80 or 160 on any of the four tasks; acceptability savings are less firm at some budgets.Source: §7.3 · pp. 6–7
  • Keeping mmBERT’s source vocabulary leaves QA 4.8–5.2 F1 behind mmBERT-transplant at every budget and costs 2.8 times the compute.The Gemma tokenizer splits Icelandic into 1.6 times as many tokens, and the keep-vocabulary model uses 307M rather than 136M parameters.Source: §7.3 · p. 8

8 Discussion

Open-data ModernBERT matches or beats IceBERT on its benchmarks, and controlled tests find that vocabulary-transplanted warm starts remain competitive with scratch training. Benchmark ceilings and task exposure qualify what the scores measure.

What changed since IceBERT

  • 2.5 macro F1 on binary and 4.4 on five-category token-level error detection, plus 5.5 MCC on sentence-level detection, are the largest gains over IceBERT.Against IceBERT-large, the corresponding gains are 1.2, 1.7 and 4.1.Source: §8 · p. 8
  • 60B tokens for our large model versus about 215B for IceBERT accompanies the gains, but architecture cannot be isolated from the other model differences.The comparison also changes corpus, tokenizer, size and training length.Source: §8 · p. 8

A warm start is the safe default

  • A 4.6 F1 QA deficit persists on questions fitting one window, and a larger window does not close it.Keeping mmBERT’s vocabulary lost 5 F1 on QA and cost 2.8 times the compute.Source: §8 · p. 8
  • A vocabulary-transplanted warm start was never significantly worse than scratch on the same text, and from base size up matched scratch trained on four times the text on four tasks.The comparison extends to eight times the text at base and 1B; acceptability is the weakest case, with wide intervals and roughly 4 MCC lower at base at 160 tokens per parameter.Source: §8 · p. 8

What the benchmarks no longer measure

  • Icelandic WinoGrande stays at chance through 378M parameters; only the 1B scratch model is clearly above chance without English WinoGrande data.Fine-grained PoS tagging and dependency parsing also separate models by at most 0.8 points and 2.3 LAS, respectively; the replaced-token objective contributes to Gigaword models’ edge on acceptability and error detection.Source: §8 · p. 8

Limitations

The study’s limitations concern seed coverage, comparison settings, possible evaluation-text contamination, and the scope of the data and tasks. The reported evidence is restricted to one language and does not evaluate retrieval or long-input task performance.

A Confidence intervals

  • At large size, a second scratch run differed by 3.1 MCC on acceptability and 1.5 points on sentence-level error detection, suggesting single-run differences of that size may be pretraining noise outside the intervals.Warm-start curves at base and each large and 1B model use one pretraining seed per arm; the base intervals cover only fine-tuning seeds and test items.Source: Limitations · pp. 8–9

Limitations

  • At shorter token budgets, fine-tuning reuses the 80-token learning rate; a base-size check suggests this may overstate warm-start gains by one or two points at 10 tokens per parameter.Warm-start pretraining uses the scratch peak learning rate; higher rates were not tried.Source: Limitations · p. 9
  • Source-model and IceBERT training data could contain evaluation text, which would favor warm starts and baselines over scratch models.The study reports IcePaHC parsing only on clean sentences because 45% of matchable test sentences occur in the corpus.Source: Limitations · p. 9
  • One language, 9B tokens of open text, and source models of matching shape define the study’s scope; retrieval and long-input task performance were not tested.The models are released, but not the corpus, because not every source licence permits redistribution; models fine-tuned on MIM-GOLD datasets are also not released.Source: Limitations · p. 9

B Pretraining hyperparameters

  • All models use AdamW, tied input/output embeddings, and local attention in two of every three layers, with global attention in the third.Masking is 30% for most tokens and 15% during decay; sequence length changes from 1,024 to 8,192 tokens for the final 10%.Source: B Pretraining hyperparameters · p. 13

C Data pipeline

The data pipeline filters Icelandic web text for language, length, and quality, deduplicates across sources, and removes evaluation overlaps.

C Data pipeline

  • Documents are filtered for length, Icelandic-language signals, alphabetic content, repetition, and boilerplate before deduplication.The rules also reject short lines and documents dominated by lines without punctuation.Source: C Data pipeline · p. 13
  • Exact copies are removed first, then MinHash over word 5-grams clusters near-duplicates, retaining the longest copy from the highest-priority source.Candidate pairs use estimated Jaccard similarity of at least 0.75; clusters are formed by union-find.Source: C Data pipeline · p. 13
  • Decontamination matches lower-cased word sequences against evaluation text.The passage specifies that matching is described in Section 3.Source: C Data pipeline · p. 13

D Fine-tuning and evaluation details

The evaluation applies a shared fine-tuning harness and reports results for all 30 encoders with confidence intervals. It also documents learning-rate selection, contamination checks, and task-specific training settings.

D Fine-tuning and evaluation details

  • Every encoder uses a linear head or parser on its final layer, with task-dependent pooling, batch size 32, and the epoch selected by development score.QA uses 384-token windows with stride 128.Source: D Fine-tuning and evaluation details · p. 13
  • Tables 3 and 4 report means and 95% confidence intervals for all 30 encoders across IceBERT’s tasks and the added tasks.Table 4 distinguishes LAS-H on the clean 58% of IcePaHC test sentences from LAS-H on the full test set.Source: Table 3 · p. 14, Table 4 · p. 15
  • When a model-size-dependent three-value grid selected an edge, two additional rates (twice/four times or half/quarter of the edge) were tested until the choice was interior; initial grids selected an edge in 132 of 161 new-task cases.WinoGrande was exempt because most models were near chance.Source: D Fine-tuning and evaluation details · p. 14
  • Fine-tuning was mostly stable at selected learning rates, but two binary error-detection runs collapsed, including one that predicted no errors.Collapsed runs during grid search were rejected using development data.Source: D Fine-tuning and evaluation details · p. 14
  • Confidence intervals use a hierarchical bootstrap over pretraining seeds, fine-tuning seeds, and test items, with model differences paired on the same item draws.IceBERT scores differ from its reported results on PoS tagging and NER, so comparisons are made within this harness.Source: D Fine-tuning and evaluation details · p. 14
  • Pretraining text for the five initially selected tasks was removed; later-task overlaps were counted, and IcePaHC parsing was reported on 2,968 test sentences without matches.Those 2,968 sentences are 58% of the 5,157-sentence test set; the full test set is reported in the appendix.Source: D Fine-tuning and evaluation details · pp. 14–15
  • Fine-tuning runs for 3, 5, or 10 epochs depending on task, with inputs truncated at 512 tokens for token classification and 256 for sentence and WinoGrande inputs.Runs use linear warmup and decay, weight decay 0.01, gradient clipping at 1.0, and bf16 autocast; non-finite bf16 runs are repeated in fp32.Source: D Fine-tuning and evaluation details · p. 15

E Additional checks

Additional checks test learning-rate choices, long-context behavior, tokenizer effects, and the data budgets needed by warm starts. The results qualify where the observed savings and comparisons hold.

E Additional checks

  • Retuning base-size models at 10 tokens per parameter reduces the warm-start gain to 4.2 F1 on QA and 6.5 MCC on acceptability, but both gains remain significant.Reusing the 80-token model’s rate overstated the gains by one or two points, but did not create them.Source: E Additional checks · pp. 15–16
  • At base size and 10 tokens per parameter, halving Ettin’s pretraining rate raises masked-token loss by 0.17 nats and is no better on any task; halving mmBERT-keep’s rate changes no task significantly.For Ettin, acceptability and sentence-level error-detection gains over scratch shrink and are no longer significant, so the observed gains on those tasks depend on the rate.Source: E Additional checks · p. 16, Table 11 · p. 20
  • For all 25 models with the paper’s vocabulary, using the full 8,192-token context lowers masked-token loss by 0.04 to 0.09 nats; most of the gain appears by 4,000 tokens.The check measures held-out documents and does not test long-input tasks.Source: E Additional checks · p. 16, Table 10 · p. 19
  • Wider QA windows raise mmBERT-keep to 52.7 F1 from 52.1 and mmBERT-transplant to 57.5 from 56.9, with neither change significant.Even with the wider window, mmBERT-keep remains 4.2 F1 [3.3, 5.1] behind mmBERT-transplant using the usual window; the tokenizer needs 1.6 times as many tokens for the same text.Source: E Additional checks · p. 16
  • Table 5 gives paired warm-start-minus-scratch differences at matched budgets and compares 80-token warm starts with 160-token scratch; Table 6 compares each warm-start budget with scratch at 160.Both tables report paired 95% confidence intervals; bold intervals exclude zero.Source: Table 5 · p. 17, Table 6 · p. 18
  • The supplementary tables document per-size pretraining hyperparameters and the sources’ token counts after filtering, deduplication, and decontamination.CulturaX was added only for the 160-token-per-parameter models.Source: Table 7 · p. 19, Table 8 · p. 19

Related papers

Loading 2610.36230v1…