Evaluating and Benchmarking the System One Model Jev

1 Introduction

• Generative models make small software decisions indirectly, requiring prompted answers, generated-text parsing and token payments while often lacking available or reliable answer probabilities.These limitations motivate evaluating a model designed to return structured decisions directly.Source: §1 · p. 1
• Jev returns option choices, ordered-rubric scores or statement probabilities rather than generating text, answering questions about one state in parallel.Its returned probabilities are intended to be calibrated. To the authors’ knowledge at writing, no systematic evaluation on public benchmarks existed.Source: §1 · pp. 1–2
• 37 public datasets evaluate Jev zero-shot across seven task families and over 200 language varieties, using frozen templates and full evaluation splits.The evaluation comprises 346,009 requests and reports task performance with bootstrap confidence intervals, calibration, selective prediction and cost.Source: §1 · p. 2
• Qwen3.8-27B and Gemma-4-E4B provide comparisons through exact option probabilities on identical requests, contextualizing Jev’s accuracy and calibration and controlling its MMLU result.The introduction describes the comparison setup, not its outcomes.Source: §1 · p. 2
• 346,009 raw responses and a reusable evaluation harness enable offline recomputation of every table and figure and identical-request evaluation of other models.Responses are stored in a content-addressed cache, and the harness includes an open-weight LLM backend.Source: §1 · p. 2
• Failure analysis examines Jev’s weaknesses against vendor documentation, including a steep language gradient and unexplained MMLU mathematics performance probed for memorization.The introduction identifies these investigation targets without reporting their detailed results.Source: §1 · p. 2

2.1 Zero-shot classification with pretrained models

• Jev uses typed questions for zero-shot evaluation, replacing the prompting and answer parsing used with generative LLMs [9].Natural-language-inference classification [8] and generative prompting [9] enable tasks without task-specific training. GLUE [10], MMLU [11], BIG-bench [12] and HELM [13] measure multitask ability, while evaluation harnesses [14] standardize multiple-choice presentation and scoring.Source: §2.1 · p. 2

2.2 Calibration and selective prediction

• Jev exposes probabilities and confidence scores, making calibration and selective classification through abstention on uncertain inputs [21] directly measurable.Neural networks are often miscalibrated [15]; pretrained language models can be reasonably calibrated [16], but instruction tuning and RLHF tend to degrade calibration [18]. ECE and the Brier score summarize calibration, while selective classification trades coverage for accuracy [21].Source: §2.2 · p. 2

2.3 LLMs as judges

• Jev’s Score primitive targets rubric-based judgment, a setting encompassing retrieval relevance [22], summary quality [24] [25], and response helpfulness.The summary-quality work includes evaluation on SummEval [24] [25].Source: §2.3 · p. 2

2.4 Contamination

• Jev’s nonpublic training data prevent ruling out benchmark contamination, motivating indirect checks.Public benchmarks can leak into training data and inflate reported performance [27].Source: §2.4 · p. 2

3.1 Jev and its question primitives

• Jev requests pair a shared state with independently answered Choice, Score, or Noul questions, with every evaluation request pinned to jev-1.13.0.Choice returns an option, probabilities, and confidence over up to 255 options; Score returns level probabilities, their weighted score, and confidence over 2–10 rubric levels; Noul returns P(yes). State may be a string, JSON object, or array, fields can be referenced by name, and the reported service accepts 32k tokens for state plus longest question at US$0.042 per million input tokens, with free outputs.Source: §3.1 · pp. 2–3

3.2 Benchmark suite

• The 37-dataset suite covers all three Jev primitives, multi-question requests, and marketed applications including routing, guardrails, grounding checks, and rubric scoring.Validation splits replace unavailable test labels; SMS Spam, OpenAI moderation, and PubMedQA’s 1,000 expert-labeled questions are evaluated in full, while one broken SIB-200 configuration is skipped. BIG-bench [12] is restricted to 93 multiple-choice tasks, excluding specified numerical and non-language categories in line with arithmetic guidance, and capped at 500 examples per task, yielding 13,228 examples.Source: §3.2 · p. 3

3.3 Request construction

• Each example becomes one JSON-state request using a dataset-specific template frozen before evaluation, without evaluation-set prompt tuning or in-context examples.Identical requests are sent only once, and each template is piloted on 20 training or validation examples. Classification and multiple-choice tasks use Choice with nonnumeric answer keys; binary and multilabel detection use Noul questions; ordinal tasks use described Score levels, with ToxiGen receiving both Noul and Score questions.Source: §3.3 · p. 3

3.4 Metrics

• Primary metrics follow dataset conventions; Choice uses the highest-probability option, while Noul defaults to a 0.5 threshold and additionally receives AUROC and AUPRC evaluation.Metrics include accuracy, positive-class F1, GoEmotions macro-F1 [38], UNFAIR-ToS micro-F1 following LexGLUE [59], mean category AUPRC for OpenAI moderation [55], mean per-source balanced accuracy for LLM-AggreFact [3], and Spearman correlation for Score tasks, averaged over source documents for SummEval as in G-Eval [25]. Four datasets additionally use per-question F1-maximizing thresholds selected on 1,000 training/validation examples with a 0.01 grid and applied unchanged to evaluation.Source: §3.4 · pp. 3–4
• Calibration uses ECE with 15 equal-width bins and Brier score, while selective prediction ranks examples by Jev’s reported confidence.ECE uses top-option probabilities for Choice and P(yes) for Noul; selective prediction reports accuracy on the most confident 80% and 50%, plus area under the risk–coverage curve [21]. All primary metrics receive 95% percentile bootstrap confidence intervals from 500 example-level resamples.Source: §3.4 · pp. 4–5

3.5 Execution

• 346,009 requests were answered in a 5 h 15 min evaluation; one additional BIG-bench request was rejected for exceeding the context limit.A single-machine asynchronous client used up to 1,100 requests per minute and 32 concurrent connections.Source: §3.5 · p. 5

3.6 Open-weight reference models

• Qwen3.8-27B and Gemma-4-E4B answer identical requests through exact next-token probabilities over valid single-token option codes, without generating text.Questions are answered independently with thinking disabled; code probabilities are renormalized to derive choices, probability-weighted scores, and confidence using Jev’s documented formula, (K pmax − 1)/(K − 1). Prompts are frozen after 20 training/validation examples per dataset; median valid-code probability mass is 0.997 for Qwen and 1.000 for Gemma, reaching at least 0.93 for 99% of questions.Source: §3.6 · p. 5

3.7 Publication of code & responses

• All three models’ responses are published on Zenodo, alongside repository code supporting reproducibility, identical-request comparisons, and Jev version tracking.The repository contains dataset definitions and frozen templates, a rate-limited asynchronous harness with an exact-request cache for resumable runs, and scripts regenerating every table and figure. Published jev-1.13.0 responses provide a fixed reference for reanalysis and future versions; the harness includes an open-weight-model backend.Source: §3.7 · p. 5

4.1 Overall results

• 96.5% on IMDB, 95.5% on HellaSwag, and 99.3% on ARC-Easy illustrate Jev’s strong zero-shot performance on short-text decisions in widely spoken languages.Other accuracies include SST-2 96.4%, Rotten Tomatoes 93.3%, language identification 99.6%, WinoGrande 91.4%, CommonsenseQA 88.2%, ARC-Challenge 97.7%, and BoolQ 91.3%. These results use no examples and one generic template per dataset.Source: §4.1 · p. 6

4.1.1 Routing

• 89.5% accuracy on CLINC150 and 79.7% on Banking77 demonstrate routing performance with 151 options and 77 fine-grained, partly overlapping intents, respectively.CLINC150 in-scope accuracy is 92.1%; its explicit out-of-scope option recovers 77.6% of out-of-scope requests at 94.1% precision.Source: §4.1.1 · p. 6

4.1.2 Inference and grounding

• 81.4%, 74.0%, and 67.7% ANLI accuracy across rounds reflect increasing adversarial difficulty; LLM-AggreFact achieves 78.6% mean balanced accuracy across eleven grounding sources.LLM-AggreFact balanced accuracy ranges from 59.8% on ExpertQA to 90.6% on Reveal, while PAWS paraphrase-detection accuracy is 85.0%.Source: §4.1.2 · p. 6

4.1.3 Moderation and legal text

• 0.982 AUROC on prompt-injection detection coexists with perfect precision but only 50% recall, illustrating how a fixed 0.5 threshold undermines strong ranking.Harmful-content AUROC is 0.949 on ToxiGen, 0.989 for ToxicChat toxicity, 0.994 for ToxicChat jailbreaking, and 0.972 averaged across eight OpenAI moderation categories. OpenAI per-category AUPRC ranges from 0.93 for sexual content and 0.91 for self-harm to 0.53 for harassment.Source: §4.1.3 · p. 6
• 0.204 F1 and 0.784 AUROC on AGB-DE mark Jev’s hardest dataset: German-law judgments about void consumer-contract clauses, with only 4.9% positives.Jev leads Qwen by 20.4 F1 points because Qwen never predicts a void clause at the 0.5 threshold. The task requires legal domain knowledge rather than a snap decision.Source: §4.1.3 · p. 6, §4.2 · p. 6

4.1.4 Weak spots

• 58.5% on Emotion and 57.9% on SST-5 expose difficulties with noisy or fine-grained labels; low-resource AfriXNLI reaches 64.0%.Financial PhraseBank accuracy is 73.0%. BIG-bench ranges from 100% on several tasks to 32.2% on real_or_fake_text and 35.0% on minute_mysteries_qa, which require close reading or multistep reasoning—a difficulty TypeSafe AI flags.Source: §4.1.4 · p. 6

4.2 Comparison with open-weight LLMs

• 27 datasets favor Jev over Qwen3.8-27B, with nine losses and one tie; Jev also beats Gemma-4-E4B on all 37 datasets.Non-overlapping 95% bootstrap intervals support 11 Jev leads over Qwen, none of Qwen’s leads, and 31 Jev leads over Gemma. Jev’s undisclosed size, architecture, and training data prevent attributing these differences to particular model properties.Source: §4.2 · p. 6
• +15.1 points on WinoGrande, +10.1 on MMLU, +8.9 on Belebele, and +6.4 on BIG-bench are among Jev’s largest margins over Qwen3.8-27B.Qwen’s leads are at most 1.2 points on six datasets, with larger leads on UNFAIR-ToS (+8.3), HelpSteer2 (+3.1), and the 116-example prompt-injection set (+2.6).Source: §4.2 · p. 6

Calibration and selective prediction

• 0.028 pooled Choice ECE makes Jev better calibrated than Qwen (0.063) and Gemma (0.208), although average calibration across Choice/Noul datasets is comparable to Qwen.Across 22 Choice datasets and 279,925 answers, median per-dataset ECE is also 0.028 (unweighted mean 0.061); mean confidence exceeds accuracy on 20 datasets, with worse calibration associated with lower accuracy (Spearman ρ = −0.83). Across 33 Choice/Noul datasets, average ECE is 0.074 for Jev, 0.075 for Qwen and 0.184 for Gemma.Source: Table 3 · p. 8, Figure 2 · p. 9, §4.3 · p. 9
• 0.168 ECE reflects Jev’s multi-label Noul over-prediction: mean P(yes) is 0.209 versus an observed positive rate of 0.041.Each label is answered independently, without awareness that few labels usually apply; ranking nevertheless remains strong, with mean per-label AUROC of 0.873 on GoEmotions and 0.994 on UNFAIR-ToS. All three models over-predict positives here, but Qwen is best calibrated (ECE 0.119), ahead of Gemma (0.161) and Jev.Source: Figure 2 · p. 9, §4.3 · p. 9
• 0.052 ECE accompanies slight under-confidence on eight single-Noul binary datasets: Jev’s mean P(yes) is 0.465 against an observed positive rate of 0.518.Positive examples frequently receive probabilities below 0.5, explaining recall deficits despite high AUROC. Qwen is more under-confident, with mean P(yes) of 0.405 and ECE 0.113; Gemma’s ECE is 0.123.Source: Figure 2 · p. 9, §4.3 · p. 9
• 0.499 to 0.748 micro-F1 on UNFAIR-ToS and 0.243 to 0.353 macro-F1 on GoEmotions result from training-tuned Jev thresholds.Thresholds were tuned on 1,000 training/validation examples and applied unchanged to evaluation; GoEmotions thresholds had median 0.86, while ToxicChat’s jailbreak threshold of 0.85 raised F1 from 0.717 to 0.833, with toxicity barely changing. UNFAIR-ToS micro-F1 also improved for Qwen (0.582 to 0.757) and Gemma (0.302 to 0.510).Source: §4.3 · p. 9, Table 4 · p. 10
• 0.743 to 0.726 is Qwen’s jailbreak F1 decline after threshold tuning, showing that thresholds learned from 1,000 training examples do not always transfer.Jev gains nothing on AGB-DE because 0.5 is already optimal on its training sample; AUROC 0.784 indicates weak separation of void and valid clauses rather than a misplaced threshold. Qwen’s AGB-DE F1 nevertheless rises from 0.000 at threshold 0.5 to 0.179 after tuning.Source: §4.3 · pp. 9–10, Table 4 · p. 10
• 50% coverage raises Jev accuracy from 79.7% to 96.3% on Banking77 and from 81.5% to 98.2% on SIB-200.At the same coverage, ANLI accuracy rises from 73.9% to 86.9% and Emotion from 58.5% to 73.3%. Accuracy increases steadily as less-confident answers are withheld, supporting automatic handling of confident answers and routing the rest to humans or larger models.Source: Figure 3 · p. 10, §4.3 · p. 10
• 94.5% Banking77 accuracy at 50% coverage shows Qwen’s confidence is informative too, close to Jev’s 96.3%; Gemma separates correct from incorrect answers less effectively.At 50% coverage, Qwen reaches 88.0% on ANLI, 97.6% on MMLU and 97.7% on Belebele, versus Jev’s 86.9%, 98.1% and 98.7%, respectively. Gemma’s most confident half reaches only 63.2% accuracy on ANLI.Source: §4.3 · p. 10

Rubric scoring

• 0.890, 0.851 and 0.841 are Jev’s Spearman correlations for STS-B similarity, SST-5 sentiment and ToxiGen offensiveness, respectively.SST-5 is scored considerably better as an ordinal scale than its 57.9% five-way accuracy suggests.Source: §4.4 · p. 10, Table 5 · p. 11
• 0.554 is Jev’s average summary-level Spearman correlation across SummEval’s four dimensions, highest for coherence (0.639) and lowest for fluency (0.495).Jev systematically rates fluency lower than experts, with a mean absolute error of 1.77 rubric levels.Source: §4.4 · p. 10, Table 5 · p. 11
• On HelpSteer2, Jev’s correlations with human ratings range from 0.275 for coherence to 0.558 for verbosity; helpfulness and correctness have correlations of 0.399 and 0.369, respectively.Source: §4.4 · pp. 10–11
• 9 of 12 rubric dimensions show higher correlations for Qwen than Jev, mostly by small margins.Jev leads on ToxiGen and SummEval consistency and fluency; Gemma trails both models on every dimension.Source: Table 5 · p. 11, §4.4 · p. 11

4.5 Multilinguality

• 97.2% English accuracy on Belebele contrasts with a 91.1% median across 122 language varieties and substantial weaknesses in several African languages.On Belebele, 71 varieties reach at least 90%, while 10 fall below 70%, including Nigerian Fulfulde at 37.4%. On AfriXNLI, English reaches 90.5% and French 82.7%, versus 33.8–70.3% for 16 African languages.Source: §4.5 · p. 11
• 0.69 is the Spearman correlation between Belebele and SIB-200 accuracy, supporting language-specific rather than task-specific degradation.SIB-200 accuracy ranges from 91–92% for the best languages to 19.6% for Santali in Ol Chiki script. Weak low-resource languages also use Latin script, so script alone cannot explain the gaps, although the lowest SIB-200 results involve other scripts.Source: §4.5 · p. 11, Figure 5 · p. 12

4.6 Documented weak spots and an unexplained MMLU result

• 59.6% accuracy in C-Eval high-school mathematics contrasts with 94.8% in MMLU high-school mathematics, despite TypeSafe AI listing arithmetic and numeric precision as weaknesses.Calculation-heavy C-Eval subjects have a lower median and wider spread, with 69.9% accuracy in advanced mathematics and 72.9% in probability and statistics. Jev generates no intermediate reasoning.Source: §4.6 · p. 12, Figure 6 · p. 13
• 94.3% accuracy persists after rotating calculation-heavy MMLU answer options, but withholding questions reduces Jev’s accuracy to 31.5%, ruling out memorized positions or option sets.Both open models show the same pattern; withholding questions tests whether options alone reveal answers. These probes cannot distinguish memorized question-answer pairs from genuine single-step solving.Source: §4.6 · p. 12, Table 6 · p. 13
• 94.3% accuracy on calculation-heavy MMLU subjects exceeds Jev’s 91.3% on other subjects, reversing the pattern observed for both open-weight models.Qwen scores 74.7% versus 83.0%, and Gemma 54.6% versus 69.8%, on calculation-heavy versus other MMLU subjects. Calculation-heavy C-Eval subjects are harder for all three models.Source: §4.6 · p. 12, Table 6 · p. 13
• Training exposure to MMLU questions is the authors’ most plausible explanation, but cannot be confirmed without Jev’s training data.Changing numbers in calculation questions is proposed as the decisive future test. The authors therefore treat MMLU results, and results on other long-established English benchmarks, with caution.Source: §4.6 · p. 13

Cost and throughput

• US$9.15 covered the main evaluation’s 217.9 million input tokens, averaging 630 tokens per request.The entire study cost US$9.40, including a US$0.01 development pilot and US$0.23 for additional analyses.Source: §4.7 · p. 13
• 0.36 seconds was the mean latency per request, while throughput was constrained by the rate limit rather than the service.Source: §4.7 · p. 13
• 765 input tokens were used on average for a 28-question GoEmotions request, versus 559 for a single-question text-classification request.Multi-question requests therefore added little input-token cost.Source: §4.7 · p. 13
• 6.4 A40 GPU-hours were required for Gemma-4-E4B, versus 17.4 A100 GPU-hours for Qwen3.8-27B, to score the evaluation splits.Gemma processed 126M prompt tokens and Qwen processed 135M. These hardware- and implementation-dependent figures are not directly comparable with API prices.Source: §4.7 · p. 13

5 Limitations

• Reference comparisons use only two open-weight models with Jev-oriented prompts and no reasoning, excluding proprietary generative models.Allowing reasoning would likely improve reference-model accuracy at higher cost. Comparisons with published results are complicated by differing prompts, splits, and in-context examples.Source: §5 · pp. 13–14
• Each dataset uses one untuned prompt template, so alternative wording could improve or worsen results, including weaker performance on AGB-DE and HelpSteer2.The authors identify wording as a possible partial explanation for these weaker results.Source: §5 · p. 13
• Binary decisions generally use a fixed 0.5 threshold; Table 4 instead tunes thresholds on only 1,000 training examples per dataset.The limited tuning sample constrains the threshold-tuned evaluation.Source: §5 · p. 14
• Memorization probes test only two shallow forms and cannot detect memorized question-answer pairs.The probes therefore do not rule out this form of memorization.Source: §5 · p. 14
• Requests were executed once, leaving run-to-run variance unmeasured, and reported results apply specifically to jev-1.13.0.Client-side latency includes network time. TypeSafe AI’s jev-latest alias will likely move to newer versions.Source: §5 · p. 14
• The suite omits relevance judgments, and several datasets use validation rather than test splits because public test labels are unavailable.Jev’s Score primitive is described as a natural fit for graded relevance labels, but that application remains unevaluated.Source: §5 · p. 14

6 Conclusion

• Jev outperforms Qwen3.8-27B on most datasets and Gemma-4-E4B on all datasets when scored on identical requests.It delivers high zero-shot accuracy on sentiment, topic, intent and commonsense tasks at very low cost. Its choice-probability calibration matches Qwen’s averaged per dataset, and confidence scores support selective prediction.Source: §6 · p. 14
• Performance drops sharply for low-resource languages, fine-grained or noisy label sets, legal judgments and rubric-based assessment of generated text.These limitations emerge from an evaluation spanning 37 datasets and 346,009 requests.Source: §6 · p. 14
• Binary probabilities require application-specific thresholds, especially when querying many labels; tuning thresholds on a small training sample recovers much of the resulting gap.Source: §6 · p. 14
• Jev’s unexpectedly strong MMLU mathematics results, absent in both open models, are not explained by memorized answer positions or option artifacts.Source: §6 · p. 14
• The released suite, harness and raw responses enable recomputation of every result and identical-request comparisons with other typed decision models and generative LLMs.Source: §6 · p. 14

• Choice datasets are evaluated using accuracy and ECE of the top-option probability.Source: Table 8 · p. 18
• 80% and 50% most-confident subsets are evaluated for accuracy, with examples ranked by Jev’s confidence.Source: Table 8 · p. 18
• Area under the risk–coverage curve is also reported for choice datasets.Source: Table 8 · p. 18