PaperlayerView exact revision on arXiv

Source-linked AI summary

HakemBench: A Turkish Benchmark of Typed Decisions

Sait Furkan Teke

arXiv:2610.02293v1cs.CL

TL;DR

Turkish typed-decision evaluation lacked a benchmark measuring calibration and selective automation. HakemBench releases an open benchmark and scoring harness, with gold labels largely set through AI passes and adjudication. Its results show substantial cross-track differences, while the authors flag limits in the gold and contamination checks.

  • Problem

    The authors found no Turkish benchmark measuring typed-decision calibration and selective automation, despite existing English benchmarks and Turkish exams and other evaluation suites.

    Source: §1 · p. 1

  • Method

    HakemBench releases an open Turkish typed-decision set and scorer, using proposals, blind AI passes, adjudication, and reported probes for evaluation.

    Source: §1 · p. 1, §3.1 · p. 3, §6 · p. 6

  • Results

    Fact-check triage is the hardest track for each of the nine highest rows, with the leader at macro F1 0.768 there versus 0.818 to 0.994 on its other tracks.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The benchmark pairs decision-quality scores with calibration and selective-automation measures and reports probes alongside them.

    Source: §1 · p. 1, §6 · p. 6

    Most gold is AI-made and not human-verified, and single-author audits provide no inter-annotator agreement.

    Source: §7 · p. 7

Abstract

from arXiv · show

HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores decision quality (macro F1), calibration (from the normalised Brier score) and selective automation (from the normalised area under the generalised risk-coverage curve), combines them by a geometric mean and reports intervals from 2,000 bootstrap draws; probes for option order, paraphrase, English translation and substituted names are reported alongside. Most gold labels come from blind passes of one AI model family compared with the votes of a panel of large language models from other model families; they are not human-verified. On a board of 16 rows the leader scores a composite of 0.888 and the lab's own model is 7th at 0.660. Its numbers are not blind. Earlier runs' test results shaped its training data, so its guardrail, moderation and customer support numbers are flagged; with every model scored on the other four tracks only, its composite is 0.678, 6th of 16.

1 Introduction

HakemBench addresses the lack of a Turkish benchmark for typed decisions that measures calibration and selective automation. It releases an open test set, a scorer, and a leaderboard while documenting how its gold labels were set and where AI-made gold is weak.

1 Introduction

  • Typed decisions require probabilities over fixed answer options, score levels, or yes/no outcomes so uncertain cases can be routed to a person.Examples include support-ticket escalation, phishing classification, and rubric-based student-answer scoring.Source: §1 · p. 1
  • The benchmark adds calibration and selective automation to Turkish typed-decision evaluation, a combination the authors found absent from existing Turkish benchmarks.English typed-decision benchmarks existed, while Turkish benchmarks included exams, mixed suites, and classification sets.Source: §1 · p. 1
  • The release combines an open Turkish test set with label-setting records, a scorer for decision quality, calibration and selective automation, and a public 16-row leaderboard.The board lists only models the authors ran themselves; the paper also reports weaknesses of its AI-made gold.Source: §1 · p. 1

2 Related work

Prior work supplies English typed-decision evaluation and related Turkish benchmarks, but the authors found no Turkish benchmark combining calibration with selective automation. The paper also situates its scoring and probes alongside related evaluation approaches.

2 Related work

  • JevBench evaluates English typed decisions across four axes, while HakemBench adopts its geometric mean and adds selective automation.Safety-Flag [6] separately reports error direction, calibration, and abstention for content moderators; HakemBench guardrail and moderation tracks distinguish misses from false alarms.Source: §2 · p. 1
  • Table 1 identifies the tracks in the HakemBench v1.0 open set and distinguishes question counts from choice, yes/no, and score question types.Source: Table 1 · p. 2
  • Turkish benchmarks span exams (TurkishMMLU [7], INCLUDE [8]), mixed and generative suites (Cetvel [9], TurkBench [10]), and TrGLUE classification sets [11]; TrGLUE is closest to our labelling.TrGLUE uses LLM labels with blind human review of disagreements; Umutlu et al. [3] found LLM judges weaker than human annotators on cultural common sense and fluency.Source: §2 · p. 2
  • The paper follows BIG-bench [12] in using a canary and notes that option reordering [18] and letter labels [19] can expose position and label bias.Preference leakage [13] describes a judge favouring models trained on data from a related model or model family.Source: §2 · p. 2

3 Task and construction

HakemBench defines a common typed-decision format and combines public-source and newly written items across its tracks. Its gold is primarily AI-generated through proposals, blind model passes, and adjudication, with specified exceptions and indicative single-author audits.

3 Task and construction

  • Each item pairs a Turkish text with one or more questions; models return option or score-level distributions, or a yes probability.Customer-support option descriptions include the gold’s written rubric.Source: §3 · p. 2
  • The set combines public-source fact-check, legal-routing, and guardrail items with 1,832 items written for the benchmark by LLMs.The listed public sources include parliamentary proceedings, Constitutional Court summaries, court-routing scenes, and Turkish prompt-injection sets; the passage specifies licensing and attribution conditions.Source: §3 · p. 2
  • Names and other identifying details are masked, and an audit and AI reviews screened for specified risks before flagged or unjudgeable items were removed by hand.The authors did not seek ethics review; masking public-record parliamentary and court texts reduces stored information but does not anonymise them.Source: §3 · p. 3

3.1 How the gold was made

  • Gold labels generally follow agreement between a build-process proposal and blind passes by one AI model family, with that family adjudicating disagreements.The proposal may be a panel vote, source label, or intended guardrail label; one pass could search the web using topical keywords, but not the item text.Source: §3.1 · p. 3
  • For subjective fact-check and message-type scales, an AI model re-decided questions under a written procedure, while support gold used blind passes under a written rubric.Four subject-knowledge question types generally follow model passes, except Constitutional Court items use the court record; human answers determine gold in other eligible cases, with stated exceptions.Source: §3.1 · p. 3
  • The single-author audits found gold-human differences on 3 of 53 objective questions and 39 of 90 subjective questions.The re-decided subjective gold agreed with 54 of 62 re-audit answers, and support rubric gold agreed with 88 of 119 human answers; 84 support answers form a human check, not gold.Source: §3.1 · p. 3

4 The v1.0 release

The v1.0 release includes item provenance and probes, with fixed parts for calibration and evaluation. Its contamination checks do not protect reported scores when a model has been trained on the open benchmark files.

4 The v1.0 release

  • Version 1.0 was frozen on 26 September 2026 and released on 1 October 2026 with items, gold, provenance, and probes.Source: §4 · p. 3
  • The 4,275 open questions’ gold sources are recorded by track, with Table 2 grouping AI or panel, human, and source origins.Pair rows record a claim answer made consistent with the same item’s priority answer.Source: Table 2 · p. 4
  • The fixed parts contain 1,198 items in A and 1,148 in B, with sibling items kept together; temperature is fit on A and the A-minus-B macro-F1 gap is flagged above 10 points.Part A contains 2,166 questions for temperature fitting.Source: §4 · p. 4
  • The probes test option permutations, paraphrases, English translations, and substituted names, places, organisations, dates, and amounts.The passages specify 10,038 permutation items, 162 paraphrases, 131 English items, and 423 slot-substitution items on 215 part A items.Source: §4 · p. 4
  • Canary, exact-match, 8-gram, and MinHash checks were used, but they do not protect scores of models trained on the open files because there is no hidden test set.The authors state that protection rests on the canary and slot signal.Source: §4 · p. 4

5 Scoring

The scorer combines decision quality, calibration, and selective automation into an equal-weight geometric-mean composite across seven tracks. It also reports per-track metrics and bootstrap intervals, while speed is reported separately.

Scoring

  • Per model, track, and question type, the scorer reports accuracy, macro F1, Brier score, smooth ECE, AUGRC, and median time per question.On guardrail and moderation tracks, it also counts positives caught and negatives passed.Source: §5 · p. 4
  • 95% intervals use 2,000 bootstrap resamples over items, drawing each item's questions together and reusing resamples across models.Table 3 presents these intervals in brackets; higher is better except for median milliseconds.Source: §5 · p. 4, Table 3 · p. 5
  • The composite is the equal-weight geometric mean of three 0-to-1 axes averaged over seven tracks, and it penalizes weakness on any axis.Decision quality is macro F1; calibration and selective automation are normalized against uniform-guess and always-wrong baselines, respectively, with scores floored at 0.Source: §5 · pp. 4–5
  • Speed is reported but excluded from the composite because hardware differs by model; post-hoc temperature is fitted on part A using cross-entropy.A fitted temperature above 1 indicates overconfidence on HakemBench.Source: §5 · p. 5

6 Results

The board’s four leading hosted chat models and Jev 1.13 score 0.825–0.888, while several middle ranks overlap ufakzeka-karar’s interval. Results also expose differences across tracks, gold sources and robustness probes, with the lab model’s scores qualified by test-informed training.

6 Results

  • 0.825 to 0.888: the three hosted chat models and Jev 1.13 lead clearly, while rows ranked 5th, 6th, 8th and 9th overlap ufakzeka-karar’s interval.DeepSeek V4 Pro ranks 9th despite decision quality of 0.761 because its calibration axis is 0.386; its label log-probability elicitation route also affects calibration.Source: §6 · p. 5
  • Fact-check triage is hardest for each of the nine highest rows; customer support ranges from 0.896 for the leader to 0.479 for ufakzeka-karar.Open-jev and Laya score 0.247 to 0.333 on customer support.Source: §6 · pp. 5–6
  • On the human check, the four leaders score 0.738 to 0.786, while the surface-cue baseline scores 0.619 and ufakzeka-karar 0.476.The human-decided gold’s top nine rows are closer together, ranging from 0.642 to 0.704; such gaps can also reflect its mostly subjective, disputed items.Source: §6 · p. 6
  • 0.272: the surface-cue baseline’s composite exceeds three rows, and its part A minus part B gap is 6.45 points [3.83, 8.86].On support, it beats ufakzeka-karar on choice accuracy (0.516 against 0.490) and score accuracy (0.507 against 0.481), partly in-sample.Source: §6 · p. 6
  • 0.941: ufakzeka-karar’s robustness, against Jev 1.13’s 0.943, is partly by construction because its option-order sensitivity is zero by design.On the measured paraphrase-agreement component, ufakzeka-karar scores 0.883 [0.833, 0.932] against Jev’s 0.926 [0.889, 0.963], with overlapping intervals.Source: §6 · p. 6
  • 0.059 [0.020, 0.107]: Jev 1.13 loses more accuracy on name swaps for public-source than written items, a signal that can point to contamination but does not prove it.The strata also differ in genre; among eleven rows, one interval above zero can arise by chance, and ufakzeka-karar’s interval is 0.016 [0.000, 0.033].Source: §6 · pp. 6–7
  • The released ufakzeka-karar scores are not blind: test results shaped training, so its guardrail, moderation and customer support numbers carry a flag.Scoring all models on the other four tracks only gives it a composite of 0.678, 6th of 16, with no interval.Source: §6 · p. 7

7 Limitations •

The gold is mostly AI-made and not human-verified, while several track-specific and evaluation-design constraints limit what the results establish. The benchmark also has boundaries in contamination checks, hosted-model probes, and data coverage.

7 Limitations •

  • Most gold is AI-made and disputes are mostly settled by that same model family; one person supplied all human answers, so inter-annotator agreement is unavailable.Audits indicate weaker areas but do not make the gold human-verified.Source: §7 · p. 7
  • Every open moderation and support item was model-written, so models trained on similar text styles can look better on those tracks.Source: §7 · p. 7
  • The guardrail source stratum has only 28 items—22 attacks and 6 harmless messages—too few to estimate a rate separately.Source: §7 · p. 7
  • Writer-generated items are correlated, making written-track intervals likely too narrow; no bootstrap over writer requests is reported.Source: §7 · p. 7
  • Without a hidden test set, contamination checks rely on the canary and slot signal; hosted models vary in elicitation, and hosted chat models lack probe rows.Source: §7 · p. 7
  • Fact-check sentences cover 2011–2023, with checkable-claim judgments ranging from 0.39 to 0.59 across five parties; education uses new topic headings, not new curriculum units.Source: §7 · p. 7

8 Use, maintenance and availability

HakemBench v1.0 is frozen and intended for evaluation, not training or decisions about people. Its data, harness, scoring, board, and model materials are public, and the board can be recomputed.

8 Use, maintenance and availability

  • HakemBench is for evaluating Turkish typed-decision models, not training, and is not validated for decisions about people; v1.0 is frozen.Source: §8 · p. 7
  • A recomputation on another machine reproduced the ranking, composites, and human check; others can run the open set and publish results with version and harness commit.Source: §8 · p. 8
  • The project cost $236.90 in GPU time and hosted-model API calls for the benchmark and ufakzeka-karar together, excluding server, development-model, and backbone costs.Source: §8 · p. 8
  • The data and card are on Hugging Face, while the harness, scorer, and board are on GitHub.The model weights and write-up are also publicly listed.Source: §8 · p. 8
  • AI tools contributed to code, text generation, labeling, review, scoring, probes, privacy and legal reviews, and report drafting; the author directed and reviewed the work.Source: §8 · p. 8

Related papers

Loading 2610.02293v1…