Source-linked AI summary
Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation
Krzysztof Wróbel, Jan Maria Kowalski, Jerzy Surma, Igor Ciuciura, Maciej Szymański
TL;DR
Polish-language applications need efficient and accurate safety classifiers as multilingual safety resources remain scarce. Bielik Guard addresses this gap with compact Polish classifiers trained on community annotations, achieving strong Polish-content performance and high precision with low false-positive rates, while cross-taxonomy comparisons remain limited.
Problem
Multilingual safety resources remain scarce, creating a need for efficient and accurate classifiers for Polish-language applications.
Method
Bielik Guard comprises compact Polish safety classifiers trained on 6,885 community-annotated Polish texts using a five-category safety taxonomy.
Results
77.65% precision with 0.63% false positive rate was achieved by Bielik Guard 0.1B v1.1 on real user prompts, exceeding HerBERT-PL-Guard’s 31.55% precision at identical model size.
Takeaways & Limitations
The results support using authentic Polish data and community-driven annotation for compact safety classification, with precision prioritized for deployment considerations.
Takeaways & Limitations
Performance on other Slavic languages and specialized medical or legal domains is untested, and sophisticated adversarial attacks or prompt injection techniques were not evaluated.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) become increasingly deployed in Polish language applications, the need for efficient and accurate content safety classifiers has become paramount. We present Bielik Guard, a family of compact Polish language safety classifiers comprising two model variants: a 0.1B parameter model based on MMLW-RoBERTa-base and a 0.5B parameter model based on PKOBP/polish-roberta-8k. Fine-tuned on a community-annotated dataset of 6,885 Polish texts, these models classify content across five safety categories: Hate/Aggression, Vulgarities, Sexual Content, Crime, and Self-Harm. Our evaluation demonstrates that both models achieve strong performance on multiple benchmarks. The 0.5B variant offers the best overall discrimination capability with F1 scores of 0.791 (micro) and 0.785 (macro) on the test set, while the 0.1B variant demonstrates exceptional efficiency. Notably, Bielik Guard 0.1B v1.1 achieves superior precision (77.65%) and very low false positive rate (0.63%) on real user prompts, outperforming HerBERT-PL-Guard (31.55% precision, 4.70% FPR) despite identical model size. The models are publicly available and designed to provide appropriate responses rather than simple content blocking, particularly for sensitive categories like self-harm.
1. Introduction
Bielik Guard addresses the limited availability of efficient, Polish-specific safety classifiers for LLM applications. It introduces compact models, community-based data and a five-category taxonomy for Polish content moderation.
- Polish LLM applications lack dedicated safety tools that adequately reflect local linguistic and cultural nuances.
- Bielik Guard is a family of efficient safety classifiers specifically designed for Polish-language content.
- The family includes 0.1B and 0.5B variants optimized for deployment efficiency while maintaining high accuracy.
- Its community-driven annotation process produced 6,885 annotated Polish texts with more than 60,000 individual ratings.
- The taxonomy covers Hate/Aggression, Vulgarities, Sexual Content, Crime, and Self-Harm.
- The models are publicly available and deployed in production, with ongoing community feedback used to improve the system.
2. Related Work
Prior work includes multilingual, Polish-specific and taxonomy-based safety classifiers, while Polish resources and community annotation motivate a more locally grounded approach. Bielik Guard builds on these developments by emphasizing Polish data and context-sensitive safety judgments.
- Llama Guard pioneered instruction-tuned language models for taxonomy-based input-output safety classification.
- Llama Guard 3 extended safety classification to multilingual coverage with an 8B-parameter model and 14 MLCommons hazard categories.
- Qwen3Guard supports 119 languages with three-tiered severity classification and model sizes ranging from 0.6B to 8B parameters.
- HerBERT-PL-Guard classifies Polish content into 15 categories using manually annotated Polish data and translated PolyGuard and WildGuard datasets.
- Existing Polish solutions may not fully reflect real-world Polish traffic because translated datasets can miss local distributions and idiomatic patterns.
- The annotation approach treats safety judgments as subjective and context-dependent, targeting community consensus rather than expert-only ground truth.
3. Bielik Guard: Model Architecture and Training
Bielik Guard uses a focused Polish safety taxonomy covering five categories and deliberately excludes risks requiring changing factual knowledge or broader context. This scope supports more consistent annotation and deployment guidance.
- The taxonomy contains five categories: Hate/Aggression, Vulgarities, Sexual Content, Crime, and Self-Harm.
- Hate/Aggression covers attacks or discrimination against groups based on race, religion, gender, sexual orientation, or nationality.
- Vulgarities, Sexual Content, Crime, and Self-Harm target profane language, graphic sexual material, criminal assistance, and encouragement of self-injury.
- Disinformation, jailbreaking, and copyright violations are excluded because they require changing factual knowledge, broader context, or different annotation priorities.
- The focused scope is intended to enable more consistent annotation and clearer deployment guidelines.
3.2. Model Architecture
The two Bielik Guard variants explore different points in the efficiency–performance trade-off using Polish RoBERTa encoders of different capacities. Both add a multi-label head for independent prediction across five safety categories.
- The two variants use different base models to explore the efficiency–performance trade-off space.
- Bielik Guard 0.1B uses MMLW-RoBERTa-base, a 124M-parameter encoder with a 50,001-token vocabulary and 768-dimensional representations.
- Bielik Guard 0.5B uses PKOBP/polish-roberta-8k, a 443M-parameter encoder with a 128,064-token vocabulary and greater modeling capacity.
- Both variants use dropout with p=0.1, a linear projection to 5 output logits, and sigmoid activation for independent binary category classification.
3.3. Training Data and Methodology
The study builds Bielik Guard from a community-annotated Polish dataset using soft labels that preserve disagreement, while evaluating two training configurations and a 60% agreement threshold for binary metrics.
- Dataset: 6,885 unique Polish texts were collected through large-scale community engagement for training the safety classifiers.The corpus includes anonymized Polish LLM prompts and selected social-media content.
- Dataset: Over 1,500 volunteers provided more than 60,000 annotations, with each text receiving an average of 7–8 independent ratings.Approximately 55% of samples were safe and 45% harmful or potentially unsafe.
- Labeling: Annotation percentages were used as soft regression labels, preserving agreement information and representing controversial cases without premature binarization.A text labeled HATE by 66% of annotators is distinguished from one labeled unanimously.
- Training configurations: The models were trained under a 2:1 split and a near-complete configuration to balance robust evaluation with production performance.The 2:1 configuration uses 2,295 training and 4,590 test texts, whereas near-complete training uses 6,285 training and 600 test texts.
- Evaluation: Evaluation binarizes ground-truth labels at 60% annotator agreement while model predictions use a 0.5 threshold.Multi-label classification allows a text to belong to multiple categories simultaneously.
- Quality and consensus: Quality assurance combined deduplication, clustering analysis, and expert validation, treating disagreement as informative rather than requiring a single correct label.The methodology frames safety judgments as socially consensual and context-dependent.
3.4. Training Procedure
The models were fine-tuned with a standard transformer-classification procedure using BCE with soft labels, AdamW optimization, and shared augmentation for robustness testing.
- Optimization: Binary Cross-Entropy with soft labels outperformed the tested Mean Squared Error loss, without class weighting.The soft labels derive from percentage-based annotations.
- Optimization: AdamW used weight decay 0.01, a 2e-5 learning rate, 500 warmup steps, and linear decay.Training used batch size 32 for three epochs on an A100 GPU cluster.
- Label handling: Soft labels preserve annotator consensus during training, while evaluation uses 60% ground-truth binarization and 0.5 prediction binarization.Users can adjust the prediction threshold for their precision-recall requirements.
- Robustness: Both model variants used the same training procedure and augmentation strategy, including 15 text transformations for robustness evaluation on the initial test set.Transformations included diacritic manipulation, capitalization changes, character swaps, and spacing modifications.
3.5. Model Versions
Two versions of each model variant were developed, with v1.1 correcting a crime-category threshold calibration problem and becoming the production-focused version.
- v1.0: v1.0 models overreacted to crime-related content because of a classification threshold calibration issue.The issue affected the models' handling of the crime category.
- v1.1: 77.65% precision versus 67.27% for the 0.1B variant followed the v1.1 threshold correction on user prompts.Both versions used identical procedures and data splits; the difference was the threshold calibration fix.
- v1.1: 0.63% false positive rate versus 1.20% for the 0.1B variant followed the v1.1 correction on user prompts.The corrected models improved the precision-recall trade-off.
- Versioning: The authors distinguish configuration-specific models as v1.0a/v1.1a for the 2:1 split and v1.0/v1.1 for near-complete training.This versioning separates evaluation-focused and production-focused training configurations.
- Versioning: Subsequent analyses prioritize v1.1 models as production-ready variants with optimal precision-recall trade-offs.
4. Evaluation
Bielik Guard performs strongly across standard, perturbed, crime-saturated, and real-user evaluations, with the larger model generally offering better discrimination and the smaller model delivering highly precise, low-intrusion moderation. Results also expose precision–recall and cross-taxonomy comparability trade-offs relevant to deployment.
- Sojka Test Set: 0.791 micro-F1 and 0.785 macro-F1 make the 0.5B v1.1a model the strongest overall performer on the Sojka test set.Both v1.1a models maintain specificity above 0.96.
- Per-Category Analysis: F1 exceeds 0.87 for SELF-HARM and SEX, while HATE is most difficult at 0.628 and 0.667 across the two variants.CRIME is also challenging, with F1 values of 0.707 and 0.716; all categories have ROC AUC above 0.91.
- Gadzi Jezyk Benchmark: On the 97.1% crime-related Gadzi Jezyk benchmark, v1.1 raises precision and specificity but substantially lowers recall relative to v1.0.For 0.1B, precision is 98.5% versus 97.7% and recall is 55.7% versus 70.2%; for 0.5B, recall is 71.4% versus 76.2%.
- Comparison with State-of-the-Art Models: 77.65% precision and 0.63% FPR on Polish user prompts make Bielik Guard 0.1B v1.1 substantially less intrusive than compared alternatives.The 0.5B v1.1 variant also reaches 75.28% precision and 0.73% FPR.
- Discussion: Cross-model precision and FPR comparisons are limited because classifiers use different taxonomies, and recall cannot be reported for the user-prompt evaluation.Only texts flagged by at least one classifier were annotated, leaving direct recall comparison for future work.
- Discussion: The 0.5B v1.1 model consistently outperforms the 0.1B model, while the 0.1B variant matches HerBERT-PL-Guard’s 124M size and achieves 2.5× better precision.The largest gains occur on augmented data, where micro-F1 is 0.694 versus 0.638.
- Discussion: Bielik Guard prioritizes precision over recall because excessive false positives can erode user trust and lead users to disable safety features.This design choice supports lower user friction in production but does not maximize recall on crime-saturated benchmarks.
5. Deployment and Practical Considerations
Bielik Guard is deployed through standard Transformers pipelines and a production service with community feedback. Its response-oriented design supports intervention strategies, while documented limitations constrain language, taxonomy, domain, and adversarial coverage.
- Production deployment at guard.bielik.ai lets users test Bielik Guard interactively and provide feedback that informs dataset and model improvements.
- For self-harm content, the system is designed to provide support resources such as Polish crisis-helpline information rather than only blocking or flagging.
- Standard text-classification pipelines return probability scores for all five categories, enabling application-specific thresholds and response strategies.
- Limitations: The models are optimized for Polish, exclude disinformation, jailbreaking, and copyright violations, and may degrade on specialized domains or sophisticated attacks.
- Future Work: Future work includes integrating generative explanations, learning from production feedback, ablations, and further crowdsourcing development.
6. Conclusion
Bielik Guard combines compact Polish-focused classifiers with community-driven annotation and achieves strong Polish safety-classification performance. The models are publicly available and actively deployed, with community engagement supporting ongoing improvement.
- Bielik Guard achieves state-of-the-art Polish content-safety performance while maintaining compact model sizes.
- 77.65% precision and 0.63% false positive rate were achieved by Bielik Guard 0.1B v1.1 on real user prompts.
- The models are publicly available and actively deployed, with ongoing community engagement driving continuous improvement.
Affiliations
The supplied publication-status passage lists receipt, revision, and acceptance fields without dates.
- The manuscript records a received status, but no date is supplied.
- The manuscript records a revised status, but no date is supplied.
- The manuscript records an accepted status, but no date is supplied.