PaperlayerView exact revision on arXiv

Source-linked AI summary

Polish ModernBERT: The Long and Short of Polish Language Understanding

Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec

arXiv:2609.01379v1cs.CL

TL;DR

Polish encoders have lacked a modern family spanning model scales and standard and long contexts. This paper introduces Polish ModernBERT and a long-context benchmark through staged pretraining adaptation. Across 30 tasks, the models achieve the strongest overall performance among evaluated Polish encoders, with especially large long-context gains and favorable efficiency.

  • Problem

    Polish encoders still largely rely on BERT/RoBERTa-style architectures, despite encoder-only Transformers remaining effective for discriminative and representation-learning tasks.

  • Method

    The paper introduces four Polish ModernBERT encoders across Base and Large scales and 512-token and 8K contexts, adapting pretraining through staged selection and adding LongContext.

  • Results

    Across 30 downstream tasks, Polish ModernBERT achieves the strongest overall performance among evaluated Polish encoders, with particularly large gains on long-document understanding.

  • Takeaways & Limitations

    Polish ModernBERT-8K-Base combines a 9.68-point LongContext gain over Polish RoBERTa-8K with 22% fewer parameters and favorable inference efficiency.

  • Takeaways & Limitations

    Most LongContext tasks derive from translated English datasets, and some source documents or labels may have appeared in evaluated models’ pretraining data.

Abstract

from arXiv · show

Encoder-only Transformers remain effective for discriminative and representation-learning tasks, yet Polish encoders still largely rely on BERT/RoBERTa-style architectures. We introduce \textbf{Polish ModernBERT}, a family of four Polish encoders available at Base and Large scales, each with 512-token and 8K context variants. We adapt the ModernBERT pretraining recipe through staged selection experiments and release a long-context benchmark covering legal topic classification, ideological decision-direction prediction, factual-consistency assessment over literary plot summaries, and human-rights violation assessment. Across 30 tasks, Polish ModernBERT achieves the best overall performance among the evaluated Polish encoders, reaching 83.99 and 85.11 for the Base-8K and Large-8K models, respectively. On long-context tasks, the 8K variants improve over matched Polish RoBERTa-8K baselines from 67.47 to 77.15 and from 75.88 to 78.49 at the Base and Large scales, respectively. The Base-8K model achieves this gain with 22\% fewer parameters (149M vs.\ 190M). Efficiency measurements in representative inference setups show lower peak memory usage and latency than matched Polish RoBERTa baselines in both 512-token and 8K settings. Polish ModernBERT-8K-Base additionally achieves the best result on a Polish retrieval benchmark among the evaluated encoders below 300M parameters.

1 Introduction

Polish ModernBERT addresses the lack of a modern Polish encoder family spanning model scales and context lengths by introducing four models, a staged pretraining adaptation, and a long-context benchmark. Across 30 tasks, it outperforms prior Polish encoders, especially on long-document understanding, while improving efficiency.

  • Motivation: Polish encoders still largely use earlier BERT and RoBERTa architectures, despite the competitiveness of encoder-only Transformers for discriminative and representation-learning tasks.This gap is particularly relevant for morphologically rich Polish, where language-specific tokenization and monolingual pretraining can improve lexical coverage and downstream representations.
  • Motivation: No previous Polish encoder family combined a ModernBERT-based architecture with Base and Large scales and both 512-token and 8K context lengths.Existing reference models include HerBERT, Polish RoBERTa-v2, and Polish RoBERTa-8K.
  • Contributions: Polish ModernBERT introduces four Polish encoders spanning Base and Large scales with maximum context lengths of 512 and 8,192 tokens.The models adapt ModernBERT through staged recipe selection, corpus refinement, and long-context continuation.
  • Results: Across 30 downstream tasks, Polish ModernBERT achieves stronger aggregate performance than prior Polish encoders, with particularly strong gains on long-document understanding.The models also reduce inference latency and peak memory usage relative to matched RoBERTa baselines and perform strongly on Polish retrieval.
  • Contributions: LongContext is a five-task Polish benchmark for long-document understanding, covering legal prediction, human-rights violation assessment, and factual consistency over literary plot summaries.The benchmark extends evaluation beyond the standard 512-token context window.

2 Related Work

Related work has improved BERT-style encoders through optimized training, architecture, efficiency, and long-context mechanisms, while multilingual and language-focused efforts extend these advances. Polish has strong encoder baselines but still lacks a modern family covering multiple scales and standard- and long-context settings.

  • Encoder architectures: BERT established encoder-only Transformers as a central discriminative NLP architecture, followed by improvements in pretraining objectives, data scale, attention, and representation design.RoBERTa optimized pretraining, ELECTRA introduced replaced-token detection, and DeBERTa proposed disentangled attention.
  • Long-context encoders: Longformer and BigBird extended BERT-style context lengths through sparse attention, while ModernBERT and NeoBERT combine newer architectural and implementation advances with native long contexts.MosaicBERT focuses on efficient pretraining within this broader modernization of encoder design.
  • Multilingual encoders: Multilingual encoders such as mBERT and XLM-R support cross-lingual transfer, with EuroBERT and mmBERT applying modern architectures and training to multilingual settings.These approaches share model capacity across multiple languages.
  • Language-focused encoders: Language-focused modern encoders have been developed for Ukrainian, Hebrew, Finnish, Turkish, and Brazilian Portuguese.These examples complement multilingual modernization with language-specific encoder efforts.
  • Polish encoders: Polish encoder work includes HerBERT, Polish RoBERTa, and Polish RoBERTa-8K, but Polish still lacks a family spanning multiple scales and both standard- and long-context settings.Polish RoBERTa-8K extends RoBERTa to long documents through continued pretraining.

3 Polish ModernBERT

Polish ModernBERT varies model capacity and context length while retaining ModernBERT architecture, uses scale-specific tokenizers and a 44.5B-token Polish corpus, and applies staged recipe selection followed by long-context continuation.

  • Model Family and Architecture: All variants retain ModernBERT components including rotary positional embeddings, GeGLU layers, pre-normalization, and alternating global and local sliding-window attention.These architectural choices are shared across the Polish model family.
  • Model Family and Architecture: The family spans Base and Large models with 512- and 8,192-token maximum contexts; 8K variants continue pretraining from corresponding 512-token checkpoints.During extension, global RoPE θ increases from 10,000 to 160,000 while local RoPE remains unchanged.
  • Model Family and Architecture: Base uses a 50K SentencePiece Unigram tokenizer, while Large uses a new 128,256-entry tokenizer with byte-level fallback for rare and morphologically complex Polish forms.The Base vocabulary is expanded to 50,008 entries for hardware-friendly embedding dimensions.
  • Pretraining Corpus: The main pretraining corpus contains 44.5B Polish tokens from a curated corpus, Polish Common Crawl, and filtered FineTranslations data.The curated component covers encyclopedic, scientific, educational, legal, parliamentary, literary, review, news, discussion-forum, and general web content.
  • Pretraining Corpus: FineTranslations retains documents with edu_score_raw > 1.24 or length above 2,000 tokens, with length-based admissions contributing approximately 3.3B of 12.4B tokens.The length criterion increases long-form representation while retained documents still undergo shared cleaning and quality filtering.
  • Training Procedure: Training uses stage-specific mixtures, shared cleaning and deduplication, and a compute-aware recipe search that propagates the best Base checkpoint across stages before transfer to Large.The final schedule has four 512-token stages followed by 8K continuation, with dynamic masking and cosine-decay schedules reinitialized at stage transitions.
  • Training Procedure: Along selected trajectories, Base and Large process approximately 271.4B and 344.1B non-padding tokens during 512-token pretraining, followed by 11.0B and 12.6B during context extension.Nominal token budgets differ from non-padding counts because non-padding tokens account for approximately 56% and 16% of capacity in the respective settings.

4 Evaluation

The evaluation covers 30 Polish NLU tasks, including the dedicated five-task LongContext benchmark for document understanding beyond 512 tokens. Models are compared across context lengths and scales using established protocols, with results reported through macro-averaged task performance and five-seed means.

  • Evaluation suite: 30 Polish NLU tasks are organized into KLEJ, FinBench, Other Tasks, and the five-task LongContext benchmark.The suite spans sentiment, entity, semantic-relation, financial, thematic, harmful-content, manipulation, and sequence-labeling tasks.
  • Evaluation suite: NKJP-NER* preserves sequence labeling while evaluating 512-token models on sentences and 8K models on full documents.Both variants are treated as one task using micro-averaged entity-level F1, so the setup is not a controlled context-length comparison.
  • LongContext Benchmark: LongContext targets document understanding beyond the standard 512-token context window and distinguishes standard- from extended-context encoders.Its five tasks are SCOTUS-DOM, SCOTUS-DEC, ECTHR-PL-AVA, ECTHR-PL-VA, and BOOKSUMMARY.
  • LongContext Benchmark: 74.5% of SCOTUS test examples exceed 4K tokens under the Polish ModernBERT Base tokenizer.SCOTUS-DOM predicts one of 11 legal issue areas, while SCOTUS-DEC predicts liberal or conservative ideological direction.
  • LongContext Benchmark: ECtHR-PL concatenates case facts into documents and defines multi-label tasks for alleged and court-found Convention violations.Cases exceeding 32,000 characters are removed before translation into Polish.
  • Fine-Tuning and Evaluation Protocol: Five random seeds are used for every model–task configuration, with mean test performance reported according to each task’s primary metric.Aggregate means and sample standard deviations are reported for principal matched Polish-model comparisons.

5 Results

Polish ModernBERT delivers stronger aggregate and long-context performance than Polish encoder baselines across scales, while also improving inference efficiency. Its retrieval evaluation follows the PIRB protocol across 41 datasets.

  • Encoder Performance: 83.99 is the highest overall Base-scale score, achieved by Polish ModernBERT-8K.The Base 512-token model reaches a short-context average of 84.96, while the 8K model reaches 85.36.
  • Encoder Performance: 35 of 60 scale-specific task comparisons favor Polish ModernBERT, versus 19 for the Polish RoBERTa family.The gains span KLEJ, FinBench, and Other Tasks, and Polish ModernBERT also outperforms EuroBERT and mmBERT in aggregate.
  • Long-Context Performance: Polish ModernBERT-8K ranks first in 9 of 10 LongContext task–scale comparisons.At Base, it ranks first on all five tasks; at Large, it ranks first on four of five, with BOOKSUMMARY favoring Polish RoBERTa-8K.
  • Long-Context Performance: 77.15 is the Base-8K LongContext average, 9.68 points above Polish RoBERTa-8K and achieved with 22% fewer parameters.The parameter counts are 149M for Polish ModernBERT-8K-Base and 190M for Polish RoBERTa-8K.
  • Long-Context Performance: 78.49 is the Large Polish ModernBERT-8K average, exceeding Polish RoBERTa-8K-Large by 2.61 points.Context extension raises the Polish ModernBERT Large average from 73.58 to 78.49.
  • Long-Context Performance: LongContext gains exceed those on existing datasets with long examples: 5.56 versus 0.86 points at Base and 4.91 versus 0.82 at Large.This contrast distinguishes benchmarks designed to stress long-document processing from datasets that merely contain long inputs.
  • Retrieval Performance: PIRB reports mean NDCG@10 across 41 retrieval datasets, with encoders compared within two parameter-size groups.The evaluation uses a shared contrastive-learning protocol and the original PIRB pooling, normalization, similarity, and evaluation settings.

6 Conclusion

The paper introduces four Polish ModernBERT encoders spanning two scales and two context lengths, alongside staged pretraining and a five-task long-context benchmark. Across 30 tasks, the models lead evaluated Polish encoders, with strong long-context, efficiency, and retrieval results.

  • Conclusion: Polish ModernBERT comprises Base and Large encoders with 512-token and 8K context lengths, plus staged pretraining and a five-task LongContext benchmark.The benchmark targets long-document understanding.
  • Conclusion: Across 30 downstream tasks, Polish ModernBERT achieves the strongest overall performance among the evaluated Polish encoders.The largest gains occur on long-document understanding.
  • Conclusion: 9.68 points and 22% fewer parameters characterize Polish ModernBERT-8K-Base's advantage over matched Polish RoBERTa-8K on LongContext.The models use 149M versus 190M parameters.
  • Conclusion: Polish ModernBERT-8K-Base achieves the best PIRB result among evaluated encoders below 300M parameters.The models also provide favorable inference efficiency.

Limitations

The study’s evidence is constrained by Polish-only experiments, classification-heavy evaluation, and limitations in benchmark data, domain coverage, and model-scale comparability.

  • Language scope: The procedure’s broader generality remains unestablished because experiments focus exclusively on Polish.Effectiveness may not transfer directly to languages with different linguistic properties, data availability, or tokenizer requirements.
  • Evaluation coverage: The evaluation suite is dominated by classification tasks and does not fully represent encoder applications such as reranking, span extraction, semantic search, or structured prediction.Broader evaluation across these task types would provide a more complete assessment of model capabilities.
  • Translated and generated evaluation data: Most LongContext tasks use machine-translated English data, while BookSummary also uses LLM-generated claims checked by a single annotator.Translation artifacts, model-specific generation patterns, or occasional label inconsistencies may remain.
  • Translated and generated evaluation data: Four LongContext tasks derive from public English datasets whose documents and labels may have appeared in evaluated models’ pretraining data.Translation reduces direct lexical overlap but does not eliminate possible cross-lingual contamination.
  • LongContext domain coverage: LongContext’s concentration on legal documents may limit the generalizability of benchmark conclusions.Suggested extensions include finance, science, administration, and news.
  • Model-scale comparisons: Base and Large variants use different tokenizers, so scale differences reflect both model capacity and tokenization rather than a controlled scaling study.Consequently, comparisons between model scales should not be interpreted as controlled scaling evidence.

Ethical Considerations

The released resources are intended for Polish language-understanding research, not individual identification or case-specific decisions, and model outputs require independent verification in high-stakes use.

  • Research scope: The benchmark is not intended for identifying individuals or making decisions about specific cases.Its legal and human-rights resources follow the structure of corresponding public source datasets.
  • High-stakes use: Legal and human-rights classification performance is not evidence that the models are suitable for autonomous legal decision-making.Model outputs may reflect biases in the pretraining and evaluation data.
  • Privacy and profiling: Public legal source documents may contain identifying details, so the benchmark should not be used to identify individuals or profile specific cases.This boundary follows from the contents of the public source datasets.

A.1 Domain Composition

The corpus is characterized with approximate automated domain estimates and processed through shared cleaning, language, quality, perplexity, and deduplication filters before a Polish ModernBERT baseline is trained.

  • Domain estimation: Domain composition is estimated by a multinomial Naive Bayes classifier using TF–IDF-weighted bag-of-words features on lemmatized tokens.The classifier was trained on approximately 15 GB of Polish text and achieved 78% validation performance.
  • Domain estimation: Fine-grained domain labels are aggregated into eight broad categories to produce more robust and interpretable corpus-level estimates.Category shares are computed by token count, with 19 predicted domains merged into broader groups.
  • Domain estimation: The resulting domain statistics are approximate corpus-level estimates rather than manual document-level annotations.No single domain dominates the token-weighted corpus; source mixtures differ in their domain proportions.
  • Corpus cleaning: Documents undergo sentence segmentation, normalization, URL removal, length filtering, and sentence-level Polish language identification.The language-identification threshold is kept low to avoid discarding valid but difficult-to-classify Polish sentences.
  • Corpus cleaning: A binary random-forest quality filter removes low-quality documents using 23 character-, word-, and sentence-level features.It was trained on 2,520 manually labeled Polish documents and achieved 96% validation accuracy.
  • Corpus cleaning: KenLM perplexity filtering removes documents with abnormal language-model scores, including noisy extraction artifacts and poorly encoded texts.Exact duplicates are removed with SHA-256 hashes and near duplicates with 128-permutation MinHash locality-sensitive hashing.
  • Baseline: The direct-transfer baseline achieved 83.76 on KLEJ, 83.69 on FinBench, and 83.73 on the combined 15-task development suite.Staged recipe search subsequently identified configurations with higher validation performance.

B.2 Pretraining Recipe Selection

The pretraining recipe is selected through staged, compute-aware experiments on Base, with checkpoints evaluated by macro-averaged KLEJ performance before transfer to Large and long-context continuation.

  • Selection procedure: Recipe selection varies one small set of factors per stage while keeping the remaining configuration fixed and propagating the best checkpoint forward.The selected recipe is then transferred to the Large variants.
  • Stage I: optimization schedule: Cosine decay achieved the best Stage I average KLEJ score at 83.42 among the tested learning-rate schedules.The comparison included warmup–stable–decay schedules with different decay phases and cosine decay with 6% linear warm-up.
  • Stage II: masking objective and ratio: Whole-word masking with probability 0.25 achieved the best Stage II average KLEJ score at 84.48.It outperformed the compared token-level MLM and alternative whole-word-masking configurations.
  • Stage III: corpus and masking refinement: Replacing the full corpus with the curated corpus increased average KLEJ from 84.72 to 85.39, while reducing masking probability to 0.15 reached 85.62.The latter configuration was selected for Stage IV.
  • Stage IV: annealing: The annealing configuration increased average KLEJ to 86.62 from 85.62 for the selected Stage III checkpoint.Stage IV reduced whole-word-masking probability from 0.15 to 0.08 and lowered the peak learning rate from 1 × 10^-4 to 5 × 10^-5.
  • Stage IV: annealing: Zero attention dropout achieved average KLEJ 87.00 versus 86.62 with dropout 0.1 under otherwise fixed conditions.Zero attention dropout was selected for the final 512-token models.
  • Context extension: Context extension initializes 8K variants from selected 512-token checkpoints and increases maximum sequence length from 512 to 8,192.The global RoPE base changes from 10,000 to 160,000 while the local RoPE base remains 10,000.

C.3 Performance by Input Length

Performance by input length varies substantially across tasks: Polish ModernBERT is consistently strong at Base scale, but longer inputs do not uniformly determine task difficulty.

  • Length-dependent trends: 18 of 20 task–bucket comparisons favor Polish ModernBERT at Base scale, versus 9 of 20 at Large scale.Large-scale advantages are more task-dependent because Polish RoBERTa and EuroBERT baselines are already strong.
  • SCOTUS tasks: SCOTUS scores vary non-monotonically with length, with Polish ModernBERT-8K-Base leading all four SCOTUS-Dec buckets.On SCOTUS-Dom, the Base-8K model leads three of four ranges, while the Large-8K model is strongest beyond 6K tokens.
  • BookSummary and ECtHR: BookSummary and ECtHR show clearer degradation at longer lengths than SCOTUS.Polish RoBERTa-8K-Base leads the longest BookSummary bucket, while Polish ModernBERT-8K-Base leads both ECtHR tasks.
  • Interpretation: Input length alone does not fully explain task difficulty, and the longest BookSummary bucket contains only 101 test examples.Bucket sizes should therefore be considered when interpreting length-dependent results.

D Efficiency Analysis

Polish ModernBERT improves the quality–efficiency trade-off over matched Polish RoBERTa models in representative 512-token and long-context inference settings.

  • Measurement setup: Measurements use BF16 inference on one NVIDIA H100 GPU, with matched examples and repeated runs.The 512-token and long-context settings use POLEMO2.0-IN and SCOTUS-Dom, respectively; tokenization is excluded from timed inference.
  • Short-context models: 83.77 to 84.96 is the average-score improvement of pl-ModernBERT-base over pl-RoBERTa-v2-base.Latency falls from 0.65 to 0.48 ms per sample, while peak memory falls from 2,344 to 1,084 MB.
  • Short-context models: 86.46 is the matched average score for pl-ModernBERT-large and pl-RoBERTa-v2-large.ModernBERT reduces latency from 1.29 to 0.69 ms per sample and peak memory from 3,604 to 2,946 MB while preserving average task performance.
  • Long-context models: 77.15 is the long-context average score of pl-ModernBERT-8K-base, 9.68 points above pl-RoBERTa-8K-base.The Base-8K model also uses less memory and slightly lower latency.
  • Long-context models: 78.49 is the highest long-context average score achieved by pl-ModernBERT-8K-large.Against pl-RoBERTa-8K-large, latency decreases from 28.97 to 21.69 ms per sample alongside a 2.61-point score improvement.

E Retrieval experiments

The retrieval experiments fine-tune Polish and multilingual encoders under a shared contrastive-learning protocol and evaluate them on the 41-dataset PIRB benchmark.

  • Benchmark and protocol: 8K-supporting models are compared because PIRB includes datasets containing long documents.All models first undergo contrastive fine-tuning on a corpus composed of Polish retrieval datasets.
  • Benchmark and protocol: 41 datasets comprise PIRB, a comprehensive Polish retrieval benchmark extended with 10 new datasets.The benchmark consolidates BEIR-PL, MaupQA, and the PolEval-2022 retrieval challenge.
  • Results: 55.36 vs. 54.10 NDCG@10 is the score comparison favoring Polish ModernBERT-8K-Base over Polish RoBERTa-8K-Base among smaller models.The mean NDCG@10 score is computed across all 41 retrieval datasets.
  • Results: 58.42 vs. 58.16 NDCG@10 is the larger-model comparison, with Polish RoBERTa-8K-Large slightly ahead of Polish ModernBERT-8K-Large.The difference is 0.26 points under the shared fine-tuning protocol.
  • Results: Both larger Polish encoders outperform all included dedicated multilingual embedding baselines except Snowflake-Arctic.Dedicated embedding models may use more elaborate training pipelines and additional data, so they are external reference points rather than directly comparable baselines.
Loading 2609.01379v1…