PaperlayerView exact revision on arXiv

Source-linked AI summary

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

Adam Górski, Mateusz Jąkalak, Rafał Jakubowski

arXiv:2609.29266v1cs.AI

TL;DR

Small Polish safety classifiers can underperform out of domain, raising the question of whether the gap reflects discrimination, calibration, or data distribution. This work combines a HerBERT-based, Focal + R-Drop classifier with style-matched synthetic data and calibration analysis; under matched tuning, it reports a significant micro-F1 lead over Sójka but no surviving macro-F1 lead. The calibration recovery depends on safe examples in the calibration set, and in-distribution improvements do not necessarily transfer out of domain.

  • Problem

    The study asks whether a different encoder and richer training can match Sójka’s OOD robustness, and whether small-model OOD underperformance reflects discrimination, calibration, or data distribution.

    Source: §1 · p. 3

  • Method

    The paper trains a five-category HerBERT classifier with Focal + R-Drop and style-matched synthetic prompts, then evaluates it against Sójka with matched threshold tuning and calibration analyses.

    Source: §1 · p. 3, §3.3 · p. 7

  • Results

    With both systems tuned on the same split, Baszta’s micro-F1 lead is significant, while its macro-F1 lead is an operating-point artifact.

    Source: §1 · p. 3

  • Takeaways & Limitations

    In-distribution macro-F1 gains from cost-sensitive weighting and mean pooling coincide with lower OOD macro F1 and calibration, so robustness must be selected for directly.

    Source: §5 · p. 28

    Temperature scaling transfers only when calibration includes negatives; Gadzi Język has almost none, and the balanced validation source has 4.7% training overlap.

    Source: §4.5.1 · pp. 12–13

Abstract

from arXiv · show

We develop a multi-label Polish content-safety classifier by fine-tuning allegro/herbert-base-cased (124M) across five categories (hate, vulgarity, sexual content, crime, self-harm) using a Focal + R-Drop objective, and evaluate the resulting model against Bielik Guard (S\'ojka) on the shared out-of-distribution Gadzi J\k{e}zyk benchmark. Both systems are given per-category threshold tuning on the same calibration split. Under that matched protocol our model holds a small but statistically significant lead in micro F1, while an apparent macro-F1 lead does not survive: it was an artifact of comparing a tuned model against an untuned one. We also report what that micro figure is worth. Because Gadzi J\k{e}zyk is 97% crime-positive, a classifier that flags crime on every input and nothing else already scores 0.910 micro F1 on the same test split, so micro separates neither system from a degenerate strategy and macro is the column that does. Per-category and per-protocol figures are reported in Section 4. The residual out-of-distribution gap is one of calibration rather than discrimination. Ranking quality stays high while positive probabilities collapse, and per-category temperature scaling recovers the loss where Platt scaling and isotonic regression do not. That recovery turns out to be conditional on the calibration set containing safe text. Gadzi J\k{e}zyk contains almost none, so thresholds fitted on it flag crime on every safe input, and a balanced refit buys a deployable operating point at the cost of adversarial recall. We report both operating points rather than only the flattering one. Two changes that are standard practice, per-class cost-sensitive weighting and mean pooling, each raise in-distribution macro F1 while lowering the out-of-distribution figure, which indicates that robustness has to be selected for directly rather than inherited from in-distribution accuracy.

1. Introduction

The paper asks whether a different backbone and richer training can match Sójka’s out-of-distribution robustness, and whether small-model failures reflect discrimination, calibration, or data distribution. Under matched threshold tuning, the reported micro-F1 lead is significant, while the macro-F1 lead is an operating-point artifact.

1. Introduction

  • The study compares HerBERT and Focal + R-Drop with Sójka on the shared five-category Gadzi Język benchmark, using a data-centric analysis of the out-of-distribution gap.The training pool has 26,248 samples versus Sójka’s ~6.9K crowd-labeled texts, without new crowd annotation.Source: §1 · p. 3
  • The completed sweep and ablations produced a model reported to exceed Sójka on OOD micro F1 and adversarial crime recall, with a directional macro lead and non-oracle calibration.The sweep comprised 62 Optuna trials, five full trainings of top configurations, and ten ablations.Source: §1 · p. 3
  • Per-category temperature scaling and style-matched synthetic data recover the OOD gap when calibration includes safe examples, unlike Platt scaling or isotonic regression.The introduction cautions that fitting on an all-positive benchmark is not deployable.Source: §1 · p. 3
  • With both systems threshold-tuned on the same split, Baszta’s micro-F1 lead over Sójka is significant, while its macro-F1 lead is an operating-point artifact.The introduction reports a paired significance test with p = 0.011.Source: §1 · p. 3
  • Cost-sensitive weighting and mean pooling both raise in-distribution macro F1 but lower OOD macro F1 and calibration.The introduction presents these controlled experiments as evidence of tension between in-distribution accuracy and OOD robustness.Source: §1 · p. 3
  • Across four held-out public benchmarks, the comparison splits two-two: Baszta leads on KLEJ CBD and PolyGuard-PL, while Sójka leads on BAN-PL and nominally HateCheck-PL.The audit also found PL-Guard had leaked into training, so it was excluded as a held-out benchmark.Source: §1 · p. 4

2. Background and Related Work

The work builds on Sójka/Bielik Guard as a community-annotated Polish safety reference and on existing Polish toxicity datasets. Its training and calibration choices draw on methods for imbalance, prediction consistency, probability calibration, and distribution matching.

Bielik Guard / Sójka

  • Sójka/Bielik Guard provides a 0.1B and 0.5B reference trained on ~6.9K crowd-labeled texts and evaluated on adversarial OOD benchmarks.The comparison uses the 0.1B variant because it is size-matched; the paper makes no claim against the stronger 0.5B model.Source: §2 · p. 4

Polish toxicity resources

  • PolEval cyberbullying, Hate-Speech-PL, BAN-PL, and translated Reddit self-harm corpora enlarge the training pool, though most datasets carry hate-only labels.The resulting label-support imbalance is addressed with focal weighting and synthetic balancing.Source: §2 · p. 4

Methodological context

  • Focal loss down-weights easy examples, R-Drop enforces prediction consistency under dropout, and post-hoc calibration methods address miscalibrated probabilities.The paper’s synthetic-data approach instead shifts training data toward the OOD prompt style.Source: §2 · p. 4

3. Materials and Methods

The methods pair a HerBERT backbone and Focal + R-Drop training with a multi-source Polish dataset and synthetic prompts styled after adversarial instructions. A staged sweep, full trainings, ablations, and final thresholded evaluation guide model selection.

3.1 Architecture and Objective

  • HerBERT led the next-best encoder by 32 pp of OOD macro F1 and outperformed Sójka’s MMLW-RoBERTa backbone, despite that encoder receiving its own sweep.No augmented MMLW counterpart was run, so the comparison leaves that cell empty rather than using an unmatched number.Source: §3.1 · p. 5, Table 2 · p. 5
  • The team dropped the SupCon contrastive term after preliminary runs showed no OOD gain, without a controlled ablation isolating its effect.The paper identifies this as a design decision based on preliminary evidence, not a measured result.Source: §3.1 · p. 5

3.2 Data Composition

  • The 26,248-sample pool combines public Polish safety sources and synthetic data; Gadzi Język’s 520 samples were held out and checked for exact-text overlap.The v2 validation split has 4,522 rows, including 1,741 safe examples, and is used for deployable calibration.Source: §3.2 · p. 6
  • The study adopts Sójka’s five-category taxonomy, maps source labels onto its heads, and reports no new annotation or inter-annotator agreement.Labels can overlap, so the paper reports macro and micro F1 rather than a single accuracy.Source: §3.2 · pp. 6–7

3.3 Synthetic Style-Matched Generation

  • Synthetic prompts target Gadzi Język’s imperative jailbreak style, with generated sets for crime, sex, and self-harm.The sets contain 340 crime samples, 272 sex samples, and 200 self-harm samples; the latter used a template-combination generator after API filtering blocked ~90% of requests.Source: §3.3 · p. 7
  • The synthetic examples are harmful requests phrased as directives, not narrow prompt-injection examples that override system prompts or smuggle retrieved instructions.Only three rows matched the narrower instruction-override pattern, and none appeared in sojka_val; the paper makes no instruction-override claim.Source: §3.3 · p. 7

3.4 Sweep Pipeline (Design and Execution)

  • The four-phase pipeline used an Optuna sweep, full training of top configurations, ten ablations, and final Gadzi Język evaluation with per-category optimal thresholds.The sweep completed 62 trials and five top-configuration full trainings.Source: §3.4 · p. 8

4. Experiments and Results

The experiments show that temperature scaling can recover OOD F1 under non-leaking evaluation, but its operating point depends on safe examples. With both systems tuned on the same split, only a small significant micro-F1 lead remains; the apparent macro lead disappears.

4.1 In-Distribution Validation (t = 0.5)

  • The carried-forward Baszta 1.0 checkpoint records 0.9111 validation macro F1; 0.9131 belongs to a sweep trial, not a trained checkpoint.Table 6 compares clean, online-augmentation, and aggressive-augmentation variants at fixed t = 0.5.Source: §4.1 · p. 8, Table 6 · p. 8

4.2 OOD Gadzi Język (oracle-thresholded - optimistic upper bound)

  • At oracle thresholds, Baszta 1.0 reaches 0.707 macro and 0.927 micro F1 versus Sójka’s 0.619 and 0.582 at default 0.5; test-set tuning makes this optimistic.Gadzi Język is 97.1% crime-positive, and the model was selected on this benchmark.Source: §4.2 · p. 9, Table 7 · p. 9

4.3 Diagnosis: Calibration, Not Discrimination

  • High AUC alongside a fourfold collapse in positive-probability scale indicates an OOD calibration shift, with ranking quality largely intact.The authors attribute the shift to high-γ Focal loss suppressing uncertain OOD predictions.Source: §4.3 · p. 10, Table 8 · p. 10

4.4 Negative Result: Classical Post-Hoc Calibration Does Not Transfer

  • Calibrators fitted on validation data yield only about 0.22 macro on OOD, while per-category thresholds fitted on the full test set reach 0.603.The comparison indicates that output recalibration alone does not transfer across this domain shift.Source: §4.4 · p. 10, Table 9 · p. 10

4.5 Positive Result: Temperature Scaling Transfers

  • 0.699 test-split macro F1 follows temperature scaling, up from 0.603, but the wide [0.493, 0.797] interval does not establish a margin over Sójka.The 172/348 calibration/test split uses thresholds and temperatures fitted only on the calibration split.Source: §4.5 · p. 11, Table 10 · p. 11

4.5.1 Deployability Caveat: The Split Still Has No Safe Text

  • Balanced calibration yields 0.952 macro F1 at 5.5% safe-text false-positive rate, but Gadzi macro falls to 0.496; temperature scaling transfers only when calibration includes negatives.The balanced calibration source is in-distribution and has 4.7% exact-text overlap with training.Source: §4.5.1 · pp. 12–13

4.5.2 Paired Significance Test vs. Sójka (both-tuned comparison)

  • p = 0.011 for the small matched-threshold micro-F1 lead; the apparent macro-F1 lead disappears, with our model better in only 12.5% of resamples.The paired comparison tunes both systems on the same calibration split and evaluates them on a disjoint test split.Source: §4.5.2 · pp. 13–14, Table 13 · p. 13

4.6 Ablations and the BCE Negative Result

The ablations show that synthetic data and the Focal + R-Drop objective are important to OOD performance in this imbalanced multi-label setting.

4.6 Ablations and the BCE Negative Result

  • 0.472 OOD macro followed removal of all synthetic data, a −16.2 pp regression from the 0.633 top-1 configuration.This was the largest single-factor ablation drop.Source: §4.6 · p. 14
  • 0.505 OOD macro followed replacing Focal with BCE, with vulgar F1 falling to 0.000.The BCE result matches the corresponding loss ablation; the authors conclude Focal + R-Drop is essential here.Source: §4.6 · p. 14
  • Disabling R-Drop cost 2.0 pp, freezing every encoder layer cost 3.8 pp, and removing layer freezing cost 5.2 pp.Source: §4.6 · p. 14

4.7 Synthetic Data Closes Category Gaps

Synthetic, style-matched prompts improve category-specific OOD results, while held-out benchmarks show that performance depends on task definition and operating point. The cross-taxonomy comparison likewise finds no universal winner.

4.7 Synthetic Data Closes Category Gaps

  • 0.985 crime F1 on Baszta 0.4 followed adding 340 style-matched synthetic crime prompts, up from 0.675 on Baszta 0.1.Withholding only synthetic crime data from the top-1 configuration cost 4.6 pp OOD macro; sex (+272) and self-harm (+200) prompts were added for Baszta 0.5.Source: §4.7 · p. 14
  • 0.570 plain versus 0.551 adversarial framing gave our model a 1.9 pp F1 drop whose 95% CI included zero; Sójka’s drop was 8.7 pp.The comparison is limited to one benchmark split and harmful requests framed as role-play or fiction, not instruction-override attacks.Source: §4.7 · pp. 14–15

4.8 External Public Benchmarks (Held-Out, Contamination-Audited)

  • Exact-text auditing established disjointness, not independence; PL-Guard was excluded after 899/900 test and 765/900 adversarial rows matched training data.KLEJ CBD and BAN-PL were evaluated only after overlapping rows were dropped; several other benchmarks were deferred because their task shapes did not align cleanly.Source: §4.8 · pp. 15, 16, 19
  • 0.767 HateCheck-PL F1 for Sójka came with a 66.1% false-positive rate on non-hateful cases, against our 0.644 F1 and 33.4% rate.On profanity without hate, the hate head abstained with 93% accuracy while the vulgar head fired on 96%.Source: §4.8 · p. 17, Table 16 · p. 17
  • 0.671 against 0.438 on KLEJ CBD and 0.562 against 0.475 on PolyGuard-PL gave us two benchmark leads; Sójka led on BAN-PL and HateCheck-PL.The split follows task objectives: KLEJ CBD and PolyGuard-PL assess harmful-language presence, while BAN-PL assesses moderation decisions.Source: §4.8 · p. 17, Figure 1 · p. 18, Table 17 · p. 18

4.8.1 Cross-Taxonomy Comparison Against Third-Party Guards

  • 0.754 F1 at 23.0% false positives gave Sójka a BAN-PL lead over our 0.667 at 85.5%, matching its moderation-decision training objective.In the broader comparison, taxonomy breadth can count policy-relevant flags as false positives, and only Baszta and Sójka were contamination-audited.Source: §4.8 · pp. 16, 18, §4.8.1 · p. 21

4.8 External Public Benchmarks (Held-Out, Contamination-Audited)

  • 0.102 to 0.855 was our false-positive-rate range across external benchmark negatives, versus 5.5% on genuinely safe validation text.KLEJ CBD was the only external set whose negatives behaved like safe text; BAN-PL negatives include unmoderated but profane posts.Source: Table 15 · p. 16, Figure 1 · p. 18, §4.8 · p. 19

4.8.1 Cross-Taxonomy Comparison Against Third-Party Guards

  • 0.816 F1 at a 7.0% false-positive rate made HerBERT-PL-Guard the strongest system on PolyGuard-PL, versus our 0.562 F1 at 35.7%.Its Gadzi Język detection rate was 0.992 against ours at 0.906.Source: §4.8.1 · p. 20, Figure 2 · p. 20, Table 18 · p. 20

4.9 Calibration Metrics and Reliability Diagrams

Calibration metrics confirm that the OOD probability-scale problem is not a discrimination failure, while showing that temperature scaling improves some calibration measures but worsens others.

4.9 Calibration Metrics and Reliability Diagrams

  • 0.092 versus 0.136 ECE and 0.075 versus 0.114 Brier show better OOD calibration for our model than Sójka.Source: §4.9 · p. 22, Table 19 · p. 22
  • 0.075 → 0.068 Brier after temperature scaling came with slightly higher ECE and MCE, so it improved Brier but not every calibration metric.The passage attributes the trade-off to sharpening under-confident crime logits for a better decision-time operating point.Source: §4.9 · p. 22, Table 19 · p. 22
  • Points above the diagonal in Figure 3 mark bins whose empirical positive frequency exceeds their mean predicted probability, indicating under-confidence.The diagrams show per-category detection reliability on Gadzi Język.Source: Figure 3 · p. 22

4.10 Inference-Time Deobfuscation

An inference-time canonicalizer targets character obfuscation without retraining. It recovers much of the attack-related macro-F1 loss, with the authors cautioning that some attacks mirror training augmentation.

4.10 Inference-Time Deobfuscation

  • The canonicalizer folds Cyrillic and Greek homoglyphs, repairs character spacing, collapses repeated characters, and reverses leetspeak before inference.URLs, emails, and numbers are protected; diacritics stripping is intentionally not reversed.Source: §4.10 · pp. 22–23, Table 20 · p. 23
  • +6.9 pp macro F1 recovery and an 8.7% → 3.0% prediction-flip reduction followed canonicalization across six attacks.It fully neutralized homoglyph attacks and recovered most character-repetition and space-insertion damage.Source: §4.10 · p. 23, Table 20 · p. 23
  • Three of six attacks used transforms also present in training augmentation, making this a partly closed-loop test rather than evidence against an adaptive attacker.Source: §4.10 · p. 23

4.11 Classical Bag-of-Words Baselines

The book-grounded training variants improve validation macro F1 but weaken out-of-distribution performance, while a classical lexical baseline captures much of both in-distribution and OOD performance. The experiments show that validation gains do not reliably transfer to OOD robustness.

4.11 Classical Bag-of-Words Baselines

  • TF-IDF logistic regression reaches 0.868 validation macro, about 4.3 pp below fine-tuned HerBERT, and 0.462 Gadzi Język macro versus HerBERT’s 0.496 balanced-point macro.Both trail Sójka’s 0.619 Gadzi Język macro; the comparison uses validation-tuned thresholds, with HerBERT at its deployable balanced operating point.Source: Table 21 · p. 23, §4.11 · p. 24

4.12 Book-Grounded Training Experiments: Cost-Sensitive Loss and Pooling

  • Per-class focal alpha raises validation macro by 0.25 pp (0.9111 → 0.9136) but lowers OOD macro (0.699 → 0.656) and worsens ECE (0.092 → 0.133) and Brier (0.075 → 0.093).Only KLEJ CBD improves among four external benchmarks; HateCheck-PL and PolyGuard-PL decline, while BAN-PL is unchanged within noise.Source: Table 22 · p. 24, §4.12 · pp. 24–25
  • The mean-pooling variant reaches the best validation macro (0.9138), but Gadzi Język macro falls to 0.606 and its paired micro-F1 lead over Sójka is not significant (p = 0.218).Its OOD ECE rises to 0.109; the authors attribute the transfer difference to [CLS] better matching short imperative prompts than mean pooling over mostly declarative training text.Source: §4.12 · p. 25
  • Validation macro ranks the three recipes in exactly reverse order of OOD macro; both book-motivated changes lower OOD macro and calibration, and mean pooling removes the significant micro-F1 lead.Mean pooling has the highest validation macro (0.9138) but Gadzi Język macro falls to 0.606; its paired micro-F1 difference is +0.009 (p = 0.218).Source: Table 23 · p. 25, §4.12 · p. 25

4.13 Deep Ensemble: Recovering Value from the Weaker Variants

Ensembling the weaker variants improves Baszta’s paired micro-F1 significance and OOD probability quality, but not macro F1. That gain is specific to Gadzi Język and does not transfer to the held-out external benchmarks.

4.13 Deep Ensemble: Recovering Value from the Weaker Variants

  • The Baszta 1.0 + Baszta 1.0-α ensemble raises micro-F1 over Sójka from 0.929 (p = 0.011) to 0.933 (p = 0.004) and achieves the best OOD Brier score (0.070), but does not improve macro F1.The weaker ensemble members drag down macro F1, leaving single-model Baszta 1.0 as the macro-optimal choice.Source: Table 24 · p. 26, §4.13 · p. 27
  • On held-out external benchmarks, the ensemble matches Baszta 1.0 on three of four sets and is slightly worse on PolyGuard-PL.Its sojka_val-fit operating point collapses onto Baszta 1.0’s on the balanced benchmarks.Source: Table 25 · p. 27, §4.13 · p. 27
  • None of the three new configurations robustly beats Baszta 1.0 on independent external data; the study’s generalising improvement is training-free inference-time deobfuscation.Baszta 1.0-α improves one of four external benchmarks, mean pooling is worse throughout, and the ensemble’s gain does not transfer.Source: §4.13 · pp. 27–28

5. Discussion

The discussion finds a tension between in-distribution accuracy and OOD robustness, with data distribution and calibration central to the results. Calibration gains depend on representative safe examples, while benchmark prevalence and synthetic-data constraints shape how the findings should be applied.

5. Discussion

  • Per-class cost-sensitive weighting and mean pooling raise validation macro but lower OOD macro and worsen calibration; selecting on in-distribution performance would choose the worst OOD model.For mean pooling, the significant micro-F1 lead over Sójka also disappears.Source: §5 · p. 28
  • Across Baszta 0.1–0.3, a full LR, freeze-depth, and γ sweep changed OOD macro F1 by only 0.4 pp; a RefusEU multi-label loading fix added 4.6 pp, while style-matched synthetic crime data added ~40 pp on its target category.Source: §5 · p. 28
  • Low-γ focal training with per-category temperature scaling recovers OOD probability scale only when calibration data includes safe examples; an all-positive fit flags 100% of safe text.The discussion characterizes this benefit as real but conditional on representative calibration data.Source: §5 · p. 28
  • Augmentation more than doubles the number of OOD crime samples scored above threshold, indicating that input-space perturbation partially simulates domain shift.This comparison holds hyperparameters fixed between Baszta 0.2 and its no-augmentation ablation.Source: §5 · p. 28
  • Guardrail rankings depend on traffic prevalence: the reference comparison uses a roughly 97% benign production stream, while the study’s sets are 44% and 100% harmful.The discussion argues that a guardrail should be re-measured at the prevalence it will encounter.Source: §5 · p. 28
  • Azure’s content filter blocked most self-harm synthetic requests, complicating defensive dataset construction; template-based generation is presented as a controllable fallback.Source: §5 · p. 28

6. Conclusion and Future Work

Under matched tuning, Baszta’s micro-F1 advantage over Sójka is significant but narrow, while its apparent macro-F1 advantage is an operating-point artifact. The conclusion also bounds the evidence with benchmark, contamination, and evaluation limitations, and identifies follow-up experiments.

6. Conclusion and Future Work

  • Under matched tuning, Baszta’s micro-F1 lead is significant (0.929 vs. 0.903, diff +0.026, p = 0.011), but its apparent macro lead is an operating-point artifact; an always-crime baseline scores 0.910 micro F1.Giving Sójka the same threshold tuning raises its five-way macro to 0.782, and Baszta is better in only 12.5% of resamples.Source: §6 · p. 29
  • The four held-out public benchmarks split two-two, while PL-Guard was unusable as a held-out benchmark because 899 of 900 rows had entered training.The exact-text audit found no overlap for the headline Gadzi Język set (0/520).Source: §6 · p. 29
  • Gadzi Język claims are limited by imperative-style synthetic prompts, exact-match-only contamination auditing, a 520-point test with rare categories, and OOD-based model selection.The macro lead is not significant in either direction; only the micro-F1 gap is significant (p = 0.011).Source: §6 · pp. 29–30
  • The Gadzi Język F1-optimal operating point flags 100% of safe text; a balanced point reaches macro 0.952 at 5.5% safe-text FPR but gives back adversarial macro.No single operating point optimizes both objectives.Source: §6 · p. 30
  • Under matched tuning, self-harm reaches parity with Sójka (0.820 vs. 0.815), while sex remains a clear gap (0.476 vs. 0.727) and a concrete modeling task.Source: §6 · p. 30
  • Proposed follow-up work includes cross-domain calibration transfer, a data-scaling curve, a synthetic-quality ablation, and a controlled Sójka comparison on a fresh OOD set.The conclusion calls for a paired, uncontaminated, safe-containing Polish benchmark as the decisive experiment.Source: §6 · p. 30

Appendix A. Comparison Summary vs. Sójka

Only rows with per-category tuning for both systems on the same calibration split support a head-to-head comparison. Under that protocol, the per-category gaps differ substantially from the untuned comparison, while deployable balanced results use a separate operating point.

Appendix A. Comparison Summary vs. Sójka

  • Under matched tuning, crime F1 is 0.983 vs. 0.946 (+3.7 pp) and hate F1 is 0.490 vs. 0.478 (+1.2 pp).Sex F1 is 0.476 vs. 0.727 (−25.1 pp); vulgarity F1 is 0.800 vs. 1.000, but the estimate is statistically unreliable with only 3 test positives.Source: Table 26 · p. 31, Appendix A · p. 31
  • Only the both-tuned rows compare systems: they use the same 187-sample calibration split and score the disjoint 333-sample test set.Rows using the 172/348 split leave Sójka at its default 0.5 threshold, and the oracle row fits thresholds on the test set.Source: Table 26 · p. 31
  • Deployable balanced rows use the §4.5.1 operating point fitted on calibration data that includes safe text, whereas Gadzi-calibrated rows characterize adversarial recall.The balanced rows are relevant to production; the Gadzi calibration split contains almost no safe text.Source: Appendix A · p. 32
  • The validation macro-F1 row is an internal sanity check, not a head-to-head comparison, because it uses Baszta’s validation split rather than Sójka’s test split.Its reported values are 0.9111 and 0.770.Source: Appendix A · p. 32

Appendix B. Reproducibility

The appendix documents the training, calibration, evaluation, and compute setup used to reproduce the reported runs. It also records which artifacts are withheld and why.

Appendix B. Reproducibility

  • The best model uses low-γ (1.5) Focal + R-Drop training, then fits per-category temperatures and thresholds on a held-out calibration split.The backbone is allegro/herbert-base-cased (124M), fine-tuned with PyTorch Lightning and FP16 mixed precision.Source: Appendix B · p. 32
  • Synthetic v2 data uses style-matched imperative Polish prompts for crime, sex, and self-harm, with LLM paraphrase for crime and deterministic template combination for sex and self-harm.Generation is fixed-seed and deduplicated.Source: Appendix B · p. 32
  • OOD evaluation uses a disjoint 172/348 Gadzi Język calibration/test split, with calibration-only per-category tuning and 95% bootstrap confidence intervals from 2,000 test resamples.The paired comparison uses a 2,000-resample bootstrap of macro/micro-F1 differences against Sójka’s recovered per-sample predictions.Source: Appendix B · p. 32
  • Training, evaluation, and calibration code and experiment schedules are available on request, while checkpoints and the derived corpus are not redistributed because training includes gated and sensitive sources.The appendix also describes classical baselines, controlled pooling and focal-alpha variants, a deep ensemble, and external benchmark evaluations without retuning.Source: Appendix B · pp. 32–33

Related papers

Loading 2609.29266v1…