PaperlayerView exact revision on arXiv

Source-linked AI summary

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin

arXiv:2610.11978v1cs.CL

TL;DR

Jev’s broader capability profile remains unclear because prior evaluations focused on individual applications or compared it with few LLMs, leaving its performance across capability levels and task demands insufficiently understood. The study evaluates Jev against 19 general-purpose LLMs in three tiers on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks, but on MathQA trails the frontier-model median by 17.7 percentage points and scores below all 19 LLMs.

  • Problem

    Jev’s broader capability profile remains unclear because prior evaluations focused on individual applications or compared it with few LLMs, leaving its performance across capability levels and task demands insufficiently understood.

    Source: §1 · p. 1

  • Method

    The study evaluates Jev against 19 general-purpose LLMs in three tiers on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding.

    Source: §1 · p. 2

  • Results

    Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks, but on MathQA trails the frontier-model median by 17.7 percentage points and scores below all 19 LLMs.

    Source: §1 · p. 2, §4 · p. 6

  • Takeaways & Limitations

    A specialized decision model can handle many constrained decisions, while decisions requiring multi-step calculation still call for a general-purpose LLM.

    Source: §4 · p. 6

    Published-score differences may partly reflect different request templates, as Jev is sensitive to how options are presented (TypeSafe AI, 2026b; Sun and Xu, 2026).

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.

1 Introduction

The paper asks whether a specialized decision model can match general-purpose LLMs on finite-answer tasks and how performance varies with task demands. It compares Jev with LLMs across 13 benchmarks, finding strengths on knowledge and commonsense tasks but a marked weakness on mathematical word problems.

1 Introduction

  • Prior Jev evaluations focused on individual applications or limited comparison sets; even a 37-dataset study compared Jev with only two open-weight LLMs.This left its performance across LLM capability levels and different knowledge and reasoning demands unclear.Source: §1 · pp. 1–2
  • The evaluation compares Jev with 19 LLMs across three capability tiers on 13 benchmarks spanning knowledge, commonsense, scientific question answering, mathematics, and multilingual understanding.The study uses standard multiple-choice benchmarks and examines how Jev’s relative performance varies with task demands.Source: §1 · p. 2
  • Jev achieves the highest accuracy among evaluated models on MMLU-Redux and ARC-Challenge, but trails the frontier-model median by 17.7 percentage points on MathQA and ranks behind all 19 comparison models.The results suggest that fixed answer choices do not remove the need for substantial calculation and multi-step reasoning.Source: §1 · p. 2
  • The paper uses Jev as a representative case study to investigate the capability boundary between specialized decision models and general-purpose LLMs in finite-answer tasks.It aims to provide empirical evidence for choosing between these model types in future decisionmaking systems.Source: §1 · p. 2

2 Related Work

Related work covers multiple-choice benchmarks for knowledge, commonsense, and multilingual evaluation, as well as prior applications of Jev and specialized models. This study applies standard benchmarks to Jev and compares it with general-purpose LLMs across capability tiers.

2 Related Work

  • Multiple-choice benchmarks measure knowledge and commonsense, with harder or cleaner successors and multilingual extensions broadening the benchmark landscape.The study uses these benchmarks unchanged to evaluate a specialized decision model.Source: §2 · p. 3
  • Prior Jev studies examine applications including classification, routing, medicine, law, judging, scientific decisions, and social-science annotation, but generally use limited comparison sets.This work instead compares Jev with three tiers of general-purpose LLMs on standard benchmarks.Source: §2 · p. 3
  • Task-fine-tuned smaller models can outperform zero-shot LLMs on text classification, while routing and cascades choose among general-purpose models of different sizes.This paper studies a specialized decision model as a distinct component and asks which constrained decisions it can take over.Source: §2 · p. 3

3 Experiments

The experiments compare Jev with 19 LLMs across 13 multiple-choice benchmarks, then examine its performance across knowledge, reasoning, and multilingual tasks. Jev is strong on knowledge and commonsense but weaker on math word problems and some reasoning-intensive comparisons.

3.1 Experimental Setup

  • The evaluation covers 13 benchmarks: eight for knowledge, three for reasoning, and two multilingual extensions of MMLU.These include math word problems requiring multi-step calculation and translated questions in 29 and 14 languages.Source: §3.1 · p. 3
  • Jev is compared with 19 LLMs in frontier, representative, and small tiers, with small models defined as having fewer than 10B parameters.Source: §3.1 · p. 3
  • The protocol uses standard benchmark splits and accuracy; Jev receives each question and its options as a single Choice request.Published scores are used only for rough comparison because prompts, few-shot examples, and reasoning settings differ.Source: §3.1 · pp. 3–4

3.2 Results

  • Jev is competitive with frontier LLMs on knowledge and commonsense, leads all models on MMLU-Redux and ARC-Challenge, but trails all 19 on MathQA by 17.7 points versus the frontier median.Source: Table 2 · p. 4, §3.2 · pp. 4–5, Figure 2 · p. 5
  • Outside math, Jev leads representative LLMs in 68 of 76 comparisons and all small LLMs in 62 comparisons, mostly by wide margins.The largest representative-model margins occur on GPQA Diamond and MMLU-ProX.Source: §3.2 · pp. 4–5
  • Jev's frontier-median gap grows from 1.2 points on MMLU to 5.8 points on the more reasoning-focused MMLU-Pro.Source: §3.2 · p. 5
  • Jev transfers well to translated questions, but on C-Eval it is about eight points below the frontier median and behind all representative LLMs from Chinese developers.MMLU-ProX is about one point below the frontier median and above all representative and small LLMs.Source: §3.2 · p. 5
  • Jev scores 6.2 points lower on HellaSwag, a gap that may partly reflect differing option-presentation templates.Source: §3.2 · p. 6

4 Conclusion

The conclusion reports Jev's comparison with general-purpose LLMs across three tiers and 13 multiple-choice benchmarks. It identifies a difference between knowledge-oriented decisions and those requiring multi-step calculation, and outlines two directions for future work.

  • Jev is competitive with frontier LLMs on knowledge and commonsense, but falls behind on mathematical word problems and all 19 LLMs on MathQA.Source: §4 · p. 6
  • The authors conclude that Jev can handle many constrained decisions, while decisions requiring multi-step calculation still call for a general-purpose LLM.Source: §4 · p. 6
  • Future work will evaluate all model tiers under one protocol and test whether Jev's option probabilities can indicate when to pass questions to a reasoning model.Source: §4 · p. 6

A Evaluation Details

The appendix details benchmark sizes, frontier-median aggregation, a TruthfulQA metric difference, and uncertainty intervals for selected scores.

  • The listed evaluation sets contain 14,042 MMLU items, 817 TruthfulQA items, 198 GPQA Diamond items, 1,172 ARC-Challenge items, and 254 AQuA-RAT items.Source: A Evaluation Details · p. 7
  • The frontier median uses six models on 12 benchmarks but only three on GPQA Diamond, with medians rounded to one decimal place before differences are computed.Source: A Evaluation Details · p. 7
  • TruthfulQA's selected-option accuracy is distinct from published MC2 scores, which are excluded from comparison counts and mean differences.Source: A Evaluation Details · p. 7
  • The reported 95% Wilson intervals are [67.7, 79.8] for Jev on GPQA Diamond, [78.0, 87.2] on AQuA-RAT, and [96.6, 98.3] on ARC-Challenge.Claude Opus 5.5's GPQA Diamond interval is [69.3, 81.2].Source: A Evaluation Details · p. 7

B Additional Results

The appendix presents Table 4 as the numerical counterpart to Figure 2, comparing Jev with frontier-model scores and ranks.

  • Table 4 supplies the numbers behind Figure 2's benchmark-by-benchmark Jev-versus-frontier-median comparison.Source: B Additional Results · p. 7, Table 4 · p. 8
  • Table 4 reports the frontier median and range, Jev's difference from the median, and Jev's rank among evaluated models.Source: Table 4 · p. 8
  • For GPQA Diamond, the table's frontier median and range include only the three frontier LLMs evaluated in this work.Source: Table 4 · p. 8

C Sources of Published Scores

Tables 5–7 distinguish published scores from evaluations conducted in this work and identify the sources and protocols for published results.

C Sources of Published Scores

  • Table 7 lists sources and protocols for published scores; the reported sources were accessed on September 30, 2026.DeepSeek-V3 uses the V3-0324 checkpoint for published MMLU-Pro and MMLU-Redux scores, and the original V3 for its other published scores.Source: C Sources of Published Scores · p. 8, Table 7 · p. 10
  • Tables 5 and 6 mark published scores in italics, link their superscripts to Table 7, and identify other scores as evaluated in this work.Table 5 covers representative LLMs; Table 6 covers small LLMs.Source: Table 5 · p. 9, Table 6 · p. 9
  • Tables 5 and 6 cover Jev against representative and small LLMs, respectively, and bold the best score on each benchmark.Table 5 notes that its MC2 metric is excluded from the last two rows.Source: Table 5 · p. 9, Table 6 · p. 9
  • Table 7 keys n, o, and b also cover vendor-reported GPQA Diamond scores for GPT-5.6 Sol, Gemini 3.8 Flash, and DeepSeek-V4.1-Flash in Table 2.Source: Table 7 · p. 10

Related papers

Loading 2610.11978v1…