Source-linked AI summary
Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series
Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej
TL;DR
Bielik v3 PL addresses inefficient universal tokenization for Polish, where excessive subword fragmentation raises fertility and limits efficiency. The paper replaces the tokenizer with a Polish-optimized vocabulary, initializes embeddings with FOCUS, and applies continued pretraining plus post-training alignment. Across reported benchmarks, the PL models closely preserve original-tokenizer performance, surpassing counterparts on CPTUB and Polish EQ-Bench while retaining largely intact English capabilities.
Problem
Universal tokenizers can over-segment Polish, increasing fertility ratio, inference cost, and reducing information density in the context window.
Method
The models replace the Mistral-derived tokenizer with a Polish-optimized vocabulary, use FOCUS initialization, and apply staged continued pretraining and the original post-training pipeline.
Results
Across nine Polish and multilingual benchmarks, the PL models closely preserve original-tokenizer performance, surpassing counterparts on CPTUB and Polish EQ-Bench while keeping English capabilities largely intact.
Takeaways & Limitations
Polish-optimized tokenization reduces fertility from 3.22 to 1.62 tokens per word while maintaining comparable model performance.
Takeaways & Limitations
The models can produce factually incorrect, lewd, false, biased, or otherwise offensive outputs despite efforts to clean training data.
Abstract
from arXiv · showhide
The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.
1 Introduction
The paper introduces 7B and 11B Bielik v3 PL models with Polish-optimized tokenizers. It combines FOCUS-based tokenizer replacement with a multi-stage training pipeline and releases both models under Apache 2.0.
- Bielik v3 PL is introduced in 7B and 11B configurations with a tokenizer optimized specifically for Polish.
- FOCUS is used to replace the tokenizer while mitigating catastrophic forgetting.
- The training pipeline combines multi-stage pretraining and post-training while preserving performance comparable to models using the original tokenizer.
- Both model weights are released under the Apache 2.0 license.
2 Model Architecture
The Bielik v3 PL variants retain their base models’ architecture while using Polish-adapted tokenization. The design combines inference-efficient attention with long-context positional embeddings and different scaling strategies for the 11B and 7B models.
- Grouped-Query Attention reduces inference memory bandwidth and computational overhead by sharing key-value projections across query heads.
- Rotary Positional Embeddings support a native context window of up to 32,768 tokens while preserving sensitivity to token order over long sequences.
- The 11B model was expanded from a 32-layer Mistral-based backbone to 50 layers using Depth Up-Scaling.
- The 7B Minitron model was compressed from the 11B variant through structured pruning and knowledge distillation.
- The PL variants retain the same architectural design as their base models and differ in tokenizer and vocabulary.
3 Tokenizers
The tokenizer section motivates language-specific tokenization for Polish and evaluates it using efficiency metrics on a constitutional preamble. The APT4 tokenizer keeps vocabulary size near 32k while targeting lower fragmentation.
- Polish’s rich morphology, inflection, diacritics, and lexical variation make tokenization a critical efficiency factor.
- General-purpose tokenizers can over-segment Polish, increasing fertility ratio, inference cost, and reducing effective context information density.
- Very large vocabularies can reduce fragmentation but increase model size, memory consumption, and inference overhead.
- The dedicated Polish tokenizer uses approximately 32,000 tokens to reduce Polish fertility while retaining reasonable English and European-language coverage.
- APT4 extends the earlier APT3 design and refines segmentation of digits, punctuation, and special characters.
- The Constitution’s Polish preamble and official English translation provide a controlled cross-linguistic tokenizer benchmark.
4 Vocabulary Adaptation
Vocabulary adaptation uses FOCUS initialization followed by selective and then full continued pretraining, with the original post-training pipeline retained for comparison. The procedure is designed to stabilize the new token space while preserving internal representations.
- 4 Vocabulary Adaptation: Replacing a pretrained model’s tokenizer risks catastrophic forgetting because its learned knowledge may degrade during transition to a new embedding space.
- 4 Vocabulary Adaptation: FOCUS represents each target token as a sparse combination of semantically similar tokens from the original vocabulary.
- 4 Vocabulary Adaptation: FOCUS previously produced the lowest training loss after 4B continued-pretraining tokens and leading Open Polish LLM Leaderboard results on Bielik 1.5B v3.
- 4.1 Multi-Stage Continued Pretraining Pipeline: The first continued-pretraining stage updates the input embeddings, language-modeling head, and four boundary transformer layers over 4B tokens.
- 4.1 Multi-Stage Continued Pretraining Pipeline: After stabilization, all parameters are unfrozen for an additional 16B tokens of continued pretraining.
- 4.2 Post-Training: The adapted models use the same post-training pipeline as the original Bielik v3 models to support fair comparison.
5 Evaluation
The evaluation covers Polish-specific and multilingual benchmarks to assess Bielik v3 PL models across language understanding, generation, and reasoning. Open PL LLM Leaderboard testing uses multiple-choice and free-form generation methods in 0-shot and 5-shot settings.
- Evaluation scope: The evaluation spans Polish and multilingual benchmarks covering diverse language understanding, generation, and reasoning capabilities.The listed benchmarks include Open PL LLM, Polish EQ-Bench, CPTUB, Polish Medical, PLCC, Open LLM, INCLUDE-base-44, Belebele, and Flores.
- Open PL LLM Leaderboard: Open PL LLM Leaderboard tasks include sentiment classification, named entity recognition, topic categorization, reading comprehension, and question answering.
- Evaluation methodology: The benchmark uses both loglikelihood selection and free-form generation, with loglikelihood particularly suited to base models without instruction tuning.
- Evaluation methodology: Each task is evaluated in 0-shot and 5-shot configurations, with scores normalized against the random-choice baseline for the available answer options.
- Open PL LLM results: 65.93 is the 5-shot Open PL LLM average for Bielik-11B-v3.0-Instruct, while the Polish-tokenizer variants score 64.11 and 61.66.Bielik-Minitron-7B-v3.0-Instruct with the original tokenizer scores 62.46 for comparison.
5.2 Polish EQ-Bench
Polish EQ-Bench evaluates emotional intelligence, while CPTUB probes deeper comprehension through implicatures and adversarial questions. Bielik-11B-v3.0-Instruct remains competitive across these challenging Polish-language evaluations.
- Polish EQ-Bench: Polish EQ-Bench assesses emotion recognition, emotional implications, and sensitivity to nuanced affective states in conversational scenarios.
- Polish EQ-Bench: 71.20 is Bielik-11B-v3.0-Instruct’s Polish EQ-Bench score, compared with 73.8 for Bielik-11B-v2.6-Instruct.The model remains competitive with Llama-3.3-70B-Instruct at 70.73 and Qwen2.5-32B-Instruct at 71.15.
- CPTUB: CPTUB evaluates implicature understanding and tricky questions involving paradoxes, contradictions, absurd premises, and humorous misdirection.
- CPTUB: 3.73 is Bielik-11B-v3.0-Instruct’s overall CPTUB average, including 3.92 for implicatures and 3.19 for tricky questions.The model also scores 3.91 in language understanding, 3.88 in sentiment analysis, and 3.96 in phraseology.
5.4 Polish Medical Leaderboard
The Polish Medical Leaderboard tests medical knowledge and clinical reasoning using authentic Polish State Specialization Examination questions. Bielik-11B-v3.0-Instruct performs better than its base model, while Polish-tokenizer variants also retain substantial performance across Polish and English evaluations.
- Polish Medical Leaderboard: The Polish Medical Leaderboard uses authentic 2018–2022 Polish State Specialization Examination questions to assess medical knowledge and clinical reasoning.
- Polish Medical Leaderboard: 50.21% is Bielik-11B-v3.0-Instruct’s Polish Medical Leaderboard score, versus 45.86% for Bielik-11B-v3-Base-20250730.
- Polish Medical Leaderboard: 44.36% is the Polish Medical Leaderboard score for Bielik-Minitron-7B-v3.0-Instruct.The results support evaluation of domain-specific knowledge in Polish.
- Open LLM Leaderboard: 72.45 is Bielik-11B-v3.0-Instruct’s English Open LLM Leaderboard average, including 85.60 on GSM8K and 64.59 on ARC challenge.
- Open LLM Leaderboard: The Polish-tokenizer variants score 71.49 and 67.63 on the English Open LLM Leaderboard aggregate.The reported Polish-tokenizer results are 71.49 for Bielik-PL-11B-v3.0-Instruct and 67.63 for Bielik-PL-Minitron-7B-v3.0-Instruct.
5.6 INCLUDE-base-44
Evaluation covers INCLUDE-base-44, Belebele, FLORES, and additional Polish and English leaderboards, showing strong multilingual and Polish performance across several tasks.
- INCLUDE-base-44: 64.8 average across European languages and 69.0 on Polish-specific tasks place Bielik-11B-v3.0-Instruct highest among models listed on INCLUDE-base-44.Its Polish-specific score exceeds its multilingual average.
- Belebele: 82.98 average across European languages places Bielik-11B-v3.0-Instruct second on Belebele, improving over Bielik-11B-v2.6-Instruct at 68.67.The Polish tokenizer variants retain strong Polish accuracy but score lower on the European-language average.
- FLORES: 19.22 average BLEU across European language pairs ranks Bielik-11B-v3.0-Instruct second on FLORES, behind EuroLLM-9B-Instruct at 20.61.Bielik scores 18.54 BLEU to Polish and 19.91 from Polish.
- Additional evaluations: The PL checkpoints reach 64.11 and 61.66 on the 5-shot Open PL LLM averages for the 11B and 7B variants, respectively.Their Polish EQ-Bench scores are 71.15 and 66.89, while English Open LLM Leaderboard averages are 71.49 and 67.63.
- Additional evaluations: The PL checkpoints score 3.80 and 3.55 on CPTUB overall averages for the 11B and 7B Minitron variants, respectively.Their FLORES and INCLUDE results are also reported for both PL checkpoints.
6 Limitations and Biases
The Bielik v3 PL models retain standard LLM limitations, including possible factual errors and harmful or biased outputs.
- Limitations and Biases: The models can produce factually incorrect, lewd, false, biased, or otherwise offensive outputs and should not be relied upon for factual accuracy.The report notes that training-data cleaning efforts do not eliminate this possibility.
7 Conclusion
The Bielik v3 PL series replaces Mistral-derived tokenization with APT4 and combines staged adaptation with post-training alignment to improve Polish tokenization efficiency while preserving performance.
- Conclusion: The 11B and 7B models replace the Mistral-derived tokenizer with Polish-optimized APT4 while retaining a comparable vocabulary of approximately 32,000 tokens.The fertility ratio falls from 3.22 to 1.62 tokens per word on representative Polish text.
- Conclusion: FOCUS-based embedding initialization and staged continued pretraining use 4B tokens with partial freezing followed by 16B tokens of full adaptation.The same SFT, DPO-P, and GRPO post-training alignment is applied as in the original Bielik v3 models.
- Conclusion: Across nine Polish and multilingual benchmarks, the PL models closely preserve original-tokenizer performance and surpass those counterparts on CPTUB and Polish EQ-Bench.English-language capabilities remain largely intact.
- Conclusion: Both models are released under the Apache 2.0 license, and the methodology is presented as a reproducible blueprint for language-specific tokenizer adaptation.The blueprint combines vocabulary transfer, progressive unfreezing, and consistent post-training alignment.
A The preamble of the Constitution of the Republic of Poland
The Polish Constitution’s preamble establishes foundational values and principles for the Republic of Poland, with an English rendering included alongside the Polish text.
- Polish and English preamble: It connects the Polish nation with compatriots abroad, cooperation among countries, and responsibility toward the human family.These themes appear in the Polish and English passages.
- Polish and English preamble: The text recalls historical experiences, independence, cultural heritage, and the duty to preserve valuable achievements for future generations.The English passages provide corresponding statements about ancestors, heritage, and future generations.
- Polish and English preamble: The preamble invokes responsibility before God or conscience and establishes the Constitution as the basic law of the Republic of Poland.It frames the state around freedom, justice, cooperation between public powers, social dialogue, and subsidiarity.
- Polish and English preamble: The closing appeal emphasizes human dignity, freedom, solidarity, and respect for these principles as foundations of the Republic.The corresponding English passage explicitly presents these values as an unshakeable foundation.