Source-linked AI summary
ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring
Sait Furkan Teke
TL;DR
Fixed-answer decisions need probabilities for each allowed answer so callers can threshold actions and refer the rest to a person, making calibration consequential. The open Turkish model scores each option independently at shared positions and applies temperature scaling, with an expected-error value as a “not sure” signal. ufakzeka-karar ranks 7th; scored on the other four tracks only, its composite is 0.678, ranking 6th of 16.
Problem
Fixed-answer decisions need probabilities for each allowed answer so callers can threshold actions and refer the rest to a person, making calibration consequential.
Source: §1 · p. 1Method
Results
ufakzeka-karar ranks 7th; scored on the other four tracks only, its composite is 0.678, ranking 6th of 16.
Source: §6 · p. 6Takeaways & Limitations
The order-invariant head is reported as as accurate as the sequential head while never depending on option order.
Source: §7 · p. 7
Abstract
from arXiv · showhide
ufakzeka-karar is an open Turkish decision model with 182,494,466 parameters. Given a Turkish text and questions of a fixed answer type (a choice, a level on an ordered scale, or yes or no), it returns a temperature-scaled probability for every option and an expected error that serves as a "not sure" signal, without generating text and in one CPU forward pass for up to ten options. Built on the lab's ufakzeka-1-base, its head scores each option blind to the others at shared positions, so the answer does not depend on option order. A sequential head trained with shuffled options was about as accurate but changed 2.3 to 2.8 percent of its answers when only the option order changed; REINFORCE lost 10.2 points (0.102) of macro F1 to cross-entropy. On the open set of HakemBench v1.0 (4,275 questions, 7 tracks) the released model ranks 7th of 16 rows with a composite of 0.660 (95% interval 0.642 to 0.677). Temperature scaling lowers calibration error (smooth ECE) on the development set but raises it on held-out support questions, from 0.027 to 0.045 for the first scored run, which never trained on them; the released model later trained on them, so its 0.036 to 0.064 is not an unseen-question test. The released model is the last of three runs scored on HakemBench, and its numbers are not blind. The second run's new training data was aimed at the first run's errors on the full test set in guardrails, moderation and customer support, and the released run was trained after the second run's guardrail results on the full test set were read, under a protocol fixed in writing before any of its data, code or runs. All its numbers come after these readings; its guardrail, moderation and customer support numbers carry the flag "shaped by reading the test results". With every model scored on the other four tracks only, its composite is 0.678, 6th of 16. Weights and code are under Apache-2.0.
1 Introduction
Typed decisions require probabilities over allowed answers that support threshold-based action and referral to a person. The paper introduces an open Turkish model and evaluates order invariance, calibration, and performance on HakemBench, disclosing that earlier test results shaped its final run.
1 Introduction
- The model returns probabilities for allowed answers so callers can act above a threshold and send the rest to a person; this workflow depends on calibration.Running a CPU model locally also means the text need not leave the caller’s machine.Source: §1 · p. 1
- The paper contributes an open Turkish typed-decision model, order-invariant scoring, calibration tests on support questions held out of Run 1 training, and a disclosed HakemBench comparison.The reported median latency is 102.5 ms on one CPU thread; the model ranks 7th of 16 rows, and its final run was shaped by earlier full-test results.Source: §1 · p. 1
- Selection and acceptance rules were written down before their results were seen.Square brackets denote 95 percent bootstrap intervals throughout.Source: §1 · p. 2
- Jev’s public contract allows up to 255 options for choice questions and 2–10 ordered score levels; on HakemBench’s permutation probe, measured order sensitivity ranges from 0.039 (Jev 1.13) to 0.590 (Laya) [2].Set-LLM [9] proves invariance under a mask and position encoding that prevents set elements from seeing each other or differing in position, reporting no accuracy cost.Source: §1 · p. 2
2 Model
The model scores each option independently at shared positions, then normalizes scores across options. Its training objective combines proper scoring losses for answer distributions and ordered levels.
2 Model
- The released model has 182,494,466 parameters, nearly all in the ufakzeka-1-base backbone [1].The backbone has 24 layers, hidden size 768, a 40,960-entry Turkish tokenizer, and 13.5B pretraining tokens.Source: §2 · p. 2
- Each option attends to the text prefix and itself, not other options, and all options start at the same position before their pooled states are scored.Options are sorted by encoded text and returned in caller order; the real-weight permutation test requires bit-for-bit equality, while batches over ten options are split into passes of ten.Source: §2 · p. 2
- Cross-entropy trains against each row’s target distribution, with ranked probability score added for ordered-level questions and no reinforcement learning.The ranked probability score penalizes probability mass more when it is two levels away than when it is one; both losses are strictly proper.Source: §2 · p. 2
3 Data
The training mix combines licensed or project-written Turkish datasets with generated decision tasks, while the authors describe filtering, label sources, and limits on rights and evaluation generalization.
3 Data
- The released run uses licensed or project-written sources, including Turkish classification, court-decision, fact-checking, FAQ, and prompt-injection data.The mix includes 3,000 rows each from six converted labelled sets and 1,791 rows from three prompt-injection sets; it also adds 530 synthetic, model-written and curated conversational prompt-injection texts.Source: §3 · p. 2
- A collection’s license may not cover rights to the individual texts, which can remain with their authors or publishing sites.Source: §3 · p. 3
- For support, eight LLM-written question-template families were applied to FAQ and community-page pairs, yielding 12,957 training rows.After a later filter, 5,489 of the 6,000 source pairs remained.Source: §3 · p. 3
- Support labels came from two judge models, while the 255-question development panel matched the author’s human answers on 75.3 percent of items.The author supplied all human answers, so figures relying on them are described as indicative; four HakemBench support questions were held out of Run 1 training to test generalization to unseen questions.Source: §3 · p. 3
- Generated tracks were LLM-written and LLM-labelled because the authors found no licensed human-labelled Turkish set for them, and all files were checked against evaluation and benchmark texts for near-duplicates.The check used shared 8-grams and MinHash in both directions, and flagged rows were removed.Source: §3 · p. 3
4 Training and selection
Training and selection proceeded through three runs, with later data and choices responding to earlier full-test results. The released run’s guardrail, moderation, and support results are explicitly flagged as shaped by those readings.
4 Training and selection
- The released recipe used two epochs of AdamW training and selected seeds and checkpoints on a separate 255-question support development set.Training took 427.8 seconds on one NVIDIA H200, with 692.9 seconds total and 13.5 GB peak memory.Source: §4 · p. 3
- Run 2 added support questions, harmless guardrail messages, and model-labelled moderation comments in response to Run 1’s full-test errors.The additions included 721 harmless guardrail messages for training and 1,499 comments, 899 offensive and 600 clean.Source: §4 · p. 3
- Run 2 scored 0.991 macro F1 on its 224-row acceptance check but caught only 106 of 217 HakemBench attacks and passed 195 of 201 harmless look-alikes.Run 1’s acceptance-check score was 0.982; Run 2’s attack and look-alike results followed training on attacks drawn from templates or short translations.Source: §4 · p. 3
- Run 2’s support data made the held-out support set no longer a generalization test, and its moderation comments resembled model-written benchmark items.The held-out file shared 887 of 937 texts with the validation pool, and its labels came from the AI-model family that settled benchmark support gold; the authors warn that Runs 2 and 3 moderation results may read too high because of style likeness.Source: §4 · pp. 3–4
- Run 3’s written selection rule chose a run without the added look-alike rows, which caught 42 of 50 development attacks (recall 0.84) and passed 161 of 170 harmless messages.Four of six runs qualified, and the chosen run reached development macro F1 of 0.688.Source: §4 · p. 4
- All Run 3 numbers followed readings of earlier full-test results, so its guardrail, moderation, and customer-support results carry the flag “shaped by reading the test results”.The written protocol was fixed before Run 3 data, code, or runs; Run 2’s guardrail results had revealed a likely register shortcut.Source: §4 · pp. 3–4
5 Calibration
The paper evaluates temperature scaling and an expected-error “not sure” signal, including on support questions held out from Run 1. Temperature scaling improves calibration on the support development set but worsens it on held-out support questions.
Method
- The released run uses global T = 1.2614; its “not sure” value maps raw maximum probability to expected error using isotonic regression.Temperature form is selected by validation cross-entropy, preferring a simpler form within 0.001; calibration is assessed with Brier score and smooth ECE.Source: §5 · p. 4
Results
- 0.027 to 0.045: Run 1’s temperature scaling raises smooth ECE on held-out support questions it never trained on.The released model’s smooth ECE rises from 0.036 to 0.064 on that set, but it later trained on those questions, so this is not an unseen-question test.Source: §5 · p. 4
- 0.050 to 0.038: temperature scaling lowers the released model’s calibration error on the support development set.The held-out-question changes have no computed interval.Source: §5 · p. 4
- A likely explanation is that a temperature fitted to mildly overconfident validation data overcorrects on questions at a different confidence level.The precommitted selection rule means T = 1.2614 ships; T = 1 would have kept held-out calibration error at 0.036, while the benchmark’s post-hoc fit is 1.1733.Source: §5 · p. 5
- At confidence 0.9, 0.292 of open-set questions pass the threshold and 0.062 of those answers are wrong; at 0.99, error is 0.085 on 213 answers.At 0.99, 148 answers are guardrail answers and 18 are wrong (0.122; Wilson interval 0.078 to 0.184); the signal cannot be relied on to catch guardrail errors.Source: §5 · p. 5
6 Evaluation
On HakemBench’s open set, ufakzeka-karar ranks seventh, with results qualified by test-informed training on three tracks. Performance varies by track, and the evaluation also reports support, robustness, speed, and external-test results.
Evaluation caveat
- Run 3’s results are not blind: every score follows readings of the first two runs’ full-test results, and three tracks are flagged as shaped by those readings.The flagged tracks are guardrails, moderation, and customer support.Source: §6 · p. 5
Board results
- 7th on HakemBench’s public board; scoring only the other four tracks gives a composite of 0.678, 6th of 16, without an interval.The open set has 4,275 questions across seven tracks; its composite equally weights decision quality, calibration, and selective automation averaged over tracks.Source: §6 · pp. 5–6
By track
- Accuracy ranges from 0.569 [0.534, 0.605] on customer support to 0.952 on moderation, with 0.640 on fact-check triage.The weakest results include support choice and score accuracy of 0.490 and 0.481, and triage priority accuracy of 0.500 (macro F1 0.268).Source: §6 · p. 6
- The guardrail track has smooth ECE 0.158 versus 0.025 to 0.084 on other tracks; the model catches 194 of 217 attacks and passes 128 of 201 harmless look-alikes.The 181 court items and 28 public-set guardrail items are test splits of datasets represented in training, so the model is supervised in-distribution on them while other models answer zero-shot.Source: §6 · p. 6
Support
- On 84 human-check support questions, accuracy is 0.476 [0.369, 0.583], below the surface-cue baseline’s 0.619 [0.512, 0.714].The baseline cues were fitted on part of the open set, making that broader comparison partly in-sample.Source: §6 · p. 6
Probes and speed
- 0.000 order sensitivity by construction; paraphrase agreement is 0.883 [0.833, 0.932] versus Jev’s 0.926 [0.889, 0.963], with overlapping intervals.On an Apple M1 with one PyTorch thread and 13 sample questions, median latency was 102.5 ms per question (69.8 to 194.5 ms).Source: §6 · p. 6
External Turkish test sets
- External-set intervals overlap across every set, with no paired test; MMLU-Pro-TR accuracy is 0.098 [0.092, 0.103], slightly below chance.MASSIVE, MiDe22, and OffensEval are supervised evaluations; MMLU-Pro-TR, XCOPA, and X-FACT have no training-text overlap, though X-FACT’s Turkish claims all come from a FACTurk source.Source: §6 · p. 6
7 Ablations
The ablations compare the shipped head and objective alternatives under prespecified evaluation choices. The order-invariant head matched sequential accuracy without option-order dependence, while REINFORCE and int8 had the reported shortcomings.
7 Ablations
- Ablation choices, measures, and decision rules were fixed in writing before runs; training arms used three seeds and paired bootstrap evaluation.Table 4 reports the compared choice minus the shipped choice in macro-F1 points unless stated otherwise.Source: §7 · pp. 6–7, Table 4 · p. 7
- The order-invariant head was as accurate as the sequential head and never depended on option order.The four-point sequential-head loss in the second development round did not reproduce in Run 1, so the authors limit their claim to this smaller result.Source: §7 · p. 7, Table 4 · p. 7
- REINFORCE’s expected reward is linear in the probabilities, favoring all mass on one answer, whereas cross-entropy is minimized at the target.The REINFORCE arm used eight samples per question and the target’s mass on the sampled answer as reward.Source: §7 · p. 7
- No int8 file met the required 0.5-point macro-F1 tolerance against fp32, even with 24 feed-forward down-projection matrices kept in fp32.The reported loss came from near-tie answers, so the release uses fp32.Source: §7 · p. 7, Table 4 · p. 7
8 Limitations
The reported limitations include weak support routing, calibration and guardrail shortcomings, and uncertainty around labels and benchmark intervals.
8 Limitations
- On 84 human-checked support questions, routing fell below the surface-cue baseline; all human answers came from one person, with no inter-annotator agreement.Source: §8 · pp. 7–8
- Calibration does not transfer everywhere, the not-sure signal misses guardrail errors, and only 128 of 201 harmless look-alikes pass.Source: §8 · p. 8
- Released-model numbers followed readings of earlier test results, and guardrail, moderation, and support results are flagged; moderation may be inflated by style likeness.Source: §8 · p. 8
- World knowledge is weak, the model reads Turkish only and cuts text beyond 448 tokens; most training labels come from LLMs or rules, with no human-checked prompt-injection or relevance labels.Source: §8 · p. 8
- Board intervals may be too narrow because items written in one generation request are correlated.Source: §8 · p. 8
9 Cost and release
The paper reports a combined project and benchmark cost, releases weights and code under Apache-2.0, and describes extensive AI-tool use with author oversight.
9 Cost and release
- The project cost $236.90 in GPU time and hosted-model API calls, excluding the server, development AI model, and ufakzeka-1 base-model cost.The figure covers the model and HakemBench together and was read from providers’ dashboards.Source: §9 · p. 8
- The Apache-2.0 release includes weights with calibrator, a small inference file, code, and public result files.Source: §9 · p. 8
- AI models were used for coding, evaluation, review, drafting, and much training-data creation and labeling; the author directed the work, reviewed it, and takes responsibility.The author’s own answers were used for the support development set and human check.Source: §9 · p. 8