PaperlayerView exact revision on arXiv

Source-linked AI summary

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Jonathan Geuter, Andrew Cheng, David Alvarez-Melis

arXiv:2610.17205v1cs.AI

TL;DR

Fixed-option decisions can provide probabilities for downstream confidence, but sequential text generation adds inference cost and latency; this paper asks whether competitive systems require specialized architectures or training. It reads answer-letter logits from open language models and applies LoRA fine-tuning; Gemma 4 31B reaches 84.4 Capability on OpenDecisions, exceeding Jev by 4.3 ± 2.5 points.

  • Problem

    For fixed-option decisions, generating text adds inference cost and latency, raising the question of whether competitive decision models need specialized architectures or training.

    Source: §1 · p. 1

  • Method

    The paper reads answer-letter logits in a single forward pass for option probabilities, then fine-tunes with LoRA adapters and a learned decision-head correction.

    Source: §4 · p. 4

  • Results

    Open models match Jev without fine-tuning across OpenDecisions and seven public decision datasets, while fine-tuned Gemma 4 31B reaches 84.4 Capability and exceeds Jev by 4.3 ± 2.5 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Open-weight language models already provide competitive probabilistic classification, and simple LoRA fine-tuning can improve their decision capabilities without a specialized architecture.

    Source: §6 · p. 9

    OpenDecisions may not represent all real-world applications; it is 78% English, includes many language-model-generated questions, and supports at most 16 answer options.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance \textit{without any additional training}, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.

1. Introduction

The paper asks whether competitive decision models require specialized architectures or training, and finds that open language models can match Jev without fine-tuning and exceed it after short supervised fine-tuning. It introduces a letter-logit readout and OpenDecisions to evaluate these systems.

1. Introduction

  • Single-pass decisions avoid the latency and inference cost of sequential text generation while providing probabilities that downstream systems can use as uncertainty estimates.The motivating question is whether specialized architectures or training methods are necessary for competitive decision models.Source: §1 · p. 1
  • Without fine-tuning, Gemma 4 31B scores 79.6 and Qwen3.6-27B 79.0 Capability on OpenDecisions, compared with Jev at 80.1.Both open models also achieve similar average accuracy to Jev across seven public decision datasets.Source: §1 · p. 1
  • A single forward pass reads answer-letter logits as option probabilities; fine-tuning adds LoRA adapters and a learned correction to the decision head.The decision head starts from language-model head rows corresponding to answer letters.Source: §1 · p. 1, Figure 1 · p. 2
  • After 400 training steps, Gemma 4 31B reaches 84.4 Capability on OpenDecisions, exceeding Jev by 4.3 ± 2.5 points and setting a new state-of-the-art.The result is averaged over three runs; the procedure takes about five GPU-hours per run.Source: §1 · p. 2
  • OpenDecisions contributes 2,352 validation questions across 11 task families, three question types, four difficulty levels, and 23 languages.Its labels are checked programmatically or by a panel of three models, and its scores broadly reproduce JevBench system rankings.Source: §1 · p. 2

2. Background and related work

Jev-style systems return probabilities over options for a supplied state and question, while prior work already adapts pretrained language models for classification. Open implementations use varied readouts and training approaches, and calibration remains important when downstream systems rely on probabilities.

2. Background and related work

  • Jev-style decisions take a state, question, and options, and return probabilities that let downstream systems select an answer and associate confidence with it.The format includes choice, yes/no, and ordered-score questions; probabilities should be well-calibrated to be reliable.Source: §2 · p. 2
  • Pretrained language-model classification predates Jev, with supervised output heads and prompt-based or candidate-label approaches represented in prior work.Examples include ULMFiT, GPT, BERT, PET, entailment-based zero-shot classification, and GLiClass.Source: §2 · p. 2
  • Open decision systems use answer-letter logits, option-order averaging, separate option prompts, or fine-tuning, but few report evaluations beyond JevBench.The related-work survey describes both untrained and fine-tuned implementations.Source: §2 · pp. 2–3
  • Concurrent work uses numbered-option probabilities and listwise fine-tuning; the paper reports similar zero-shot Capability for number and letter readouts, while the letter readout needs one forward pass.For Gemma 4 31B, the reported scores are 80.5 for numbered options and 79.6 for answer letters.Source: §2 · p. 3
  • Temperature scaling can improve calibration without changing the preferred answer, while answer probabilities can depend on option order and labels.The paper reports that its models’ output probabilities yield better calibration than Jev and leaves more sophisticated calibration approaches for future work.Source: §2 · p. 3

3. OpenDecisions: a validation set for Jev-style models

OpenDecisions addresses the limited size and task coverage of public decision benchmarks with a 2,352-question validation set designed to reflect JevBench’s task composition. Its construction combines varied question sources and coverage, verification procedures, and a scoring scheme that measures accuracy and calibration.

3. OpenDecisions: a validation set for Jev-style models

  • OpenDecisions addresses the small size and limited task coverage of public decision datasets, including JevBench-Open’s 231 questions.Larger multiple-choice benchmarks primarily test general knowledge rather than decisions based on supplied information.Source: §3 · p. 3
  • The 2,352 questions span 11 families, three question types, four difficulty tiers, and 23 languages, following the published mix of JevBench v1.6.The set includes 1,178 choice, 586 yes/no, and 588 score questions; 78% are in English.Source: §3 · p. 3
  • Items combine code-written and labeled examples, LLM-authored questions checked by a three-model panel, and public user requests retained when all panel models agree.The three sources contribute 599, 1,443, and 310 items, respectively.Source: §3 · p. 3
  • Paired questions are removed together if either fails panel verification, and 43 samples are removed after overlap checks against training data and released JevBench questions..Source: §3 · pp. 3–4
  • OpenDecisions system scores nearly reproduce JevBench leaderboard rankings, with a Spearman correlation of 0.90 across 14 public decision models.A fit using ten systems predicts four held-out systems’ leaderboard scores with a 3.0-point error.Source: §3 · p. 4
  • Capability averages chance-corrected accuracy and calibration, with tier-weighted accuracy and type-specific calibration scores.The paper reports 95% intervals as value ± 1.96 SE, combining question sampling and, for trained models, training-seed variation.Source: §3 · p. 4

4. A simple recipe

This section describes a single-pass answer-letter-logit readout and a supervised fine-tuning recipe that adds a trainable correction and calibrates probabilities.

4. A simple recipe

  • A single forward pass yields option probabilities by applying softmax to the valid answer-letter logits; the readout supports up to 16 options.Yes/no questions use two options, while scoring questions use ordered answer levels.Source: §4 · p. 4
  • The decision head adds a trainable correction to the pretrained answer-letter readout rather than learning a new head from scratch.The correction starts at zero, and after training the LoRA adapter can be merged into the pretrained weights for one-pass inference.Source: §4 · p. 4
  • Temperature scaling adjusts probabilities without changing the preferred answer, using one temperature per question type fitted on held-out training-distribution questions.The authors note that calibration data differ from OpenDecisions, so fitted temperatures do not always transfer well.Source: §4 · p. 4
  • The 417,825-question training pool combines public datasets and language-model-labeled examples, with no OpenDecisions questions used for training.The remaining 50,137 questions have soft labels from two LLM teachers and include synthetic business documents and relabeled decider questions.Source: §4 · p. 5, Table 2 · p. 5
  • Supervised fine-tuning freezes the original language-model weights while jointly training rank-32 LoRA adapters and the decision head with cross-entropy.Options are shuffled during training; the 400-step run processes about 76,000 questions, and Qwen3.5-4B uses three additional empirically helpful changes.Source: §4 · p. 5
  • The two models of about 30B parameters reach Jev’s level zero-shot, and training moves Gemma 4 31B clearly beyond it.The comparison is OpenDecisions Capability.Source: Figure 2 · p. 5

5. Results

The results compare four open instruction-tuned models with Jev and public systems on OpenDecisions and additional decision and multiple-choice datasets. They find strong zero-shot performance and further gains from short LoRA fine-tuning, alongside mixed results across model sizes and benchmark types.

5.1. LLMs as zero-shot decision models

  • 79.6 Capability for zero-shot Gemma 4 31B and 79.0 for Qwen3.6-27B compare with 80.1 for Jev; temperature scaling raises Gemma’s score from 72.9 without changing its preferred answers.The smaller Gemma 4 12B and Qwen3.5-4B score 75.4 and 63.6 on OpenDecisions, respectively.Source: §5.1 · p. 6

5.2. A short training run goes beyond Jev

  • 84.4 Capability on OpenDecisions for trained Gemma 4 31B, a 4.8 ± 2.0 gain over zero-shot and 4.3 ± 2.5 points above Jev.It is the highest OpenDecisions score among evaluated systems; its Calibration is 92.3, the best reported.Source: §5.2 · p. 6, Table 3 · p. 7

5.3. Public evaluation sets

  • 89.0 average accuracy for trained 31B and 89.4 for trained 27B exceed Jev’s 88.0 across seven held-out public decision datasets.On 7,800 held-out decider items, the trained 31B and 27B models lead Jev by 2.2 ± 0.7 and 1.6 ± 0.7, respectively; the 31B model’s mean ECE is 0.047 versus Jev’s 0.066.Source: §5.3 · p. 6
  • 92.1% accuracy and 80.1 Capability on JevBench-Open for trained Gemma 4 31B exceed Jev’s 86.6% and 73.6; on official JevBench v1.7, it scores 81.5 Capability versus Jev’s 77.1.JevBench-Open has 231 items, making Capability estimates noisy; the official leaderboard uses separate draws and a special abstention rule for yes/no answers.Source: §5.3 · p. 6
  • 92.9 macro accuracy for Jev exceeds the trained 31B model’s 88.8 on six standard knowledge-heavy benchmarks, while training raises accuracy for all four open models and halves 31B ECE from 0.111 to 0.051.These tasks emphasize knowledge more than reading a supplied state; the trained model leads the strongest public systems on this benchmark suite.Source: §5.3 · p. 7

5.4. Ablations

  • 84.9 Capability from LoRA with the original answer-letter head is comparable to 84.4 from joint training, while training only a decision head on frozen representations scores 78.9.The frozen-head-only score does not improve on the calibrated zero-shot score of 79.6, indicating that the main benefit comes from adapting the language model.Source: §5.4 · p. 8
  • 1.9–2.8 points below the reference are the variants that first train a head on frozen representations; pretrained-only Gemma 4 31B reaches 70.3 after fine-tuning, and longer training offers no consistent benefit.All three 31B seeds score highest at 400 or 800 steps and lower later, mainly because Calibration declines; the tested smaller models show a similar pattern.Source: §5.4 · p. 8
  • Similar Capability across LoRA ranks 16–128 suggests lower ranks may reduce training cost; removing about 50,000 model-labeled questions lowers the score by 1.9 ± 2.2, while hard labels show no measurable change.Expected utility improves accuracy by 0.9 ± 0.9 percentage points over cross-entropy, but with LoRA its result remains within the recipe’s three-seed range.Source: §5.4 · pp. 8–9

6. Discussion and limitations

Open-weight language models show decision capabilities without specialized architectures or calibrated reinforcement learning, and LoRA fine-tuning can improve them further. The discussion also identifies dataset and evaluation boundaries and directions for future work.

6. Discussion and limitations

  • Temperature scaling improves calibration, while open-weight language models already perform comparably to Jev without additional training or specialized architectures.The authors interpret these results as evidence of latent probabilistic-classification capabilities.Source: §6 · p. 9
  • LoRA fine-tuning improves decision capabilities, with ablations suggesting most benefit comes from adapting the language model rather than training a new decision head.A fixed readout performs comparably to joint training, making it a baseline for evaluating specialized designs.Source: §6 · p. 9
  • OpenDecisions may not represent all real-world applications; it is 78% English, includes many language-model-generated questions, and the implementation supports at most 16 options.Qwen3.6-27B contributes training supervision, while related Qwen3.8-27B participates in evaluation labeling, potentially introducing bias.Source: §6 · p. 9
  • The findings do not establish that reinforcement learning or specialized decision architectures cannot improve performance further.The authors also identify high-quality decision-model training data as a possible future avenue.Source: §6 · p. 9

AI Statement

The authors report that AI agents assisted with code, experiments, and writing, with generated content verified. The request format supplies evidence, a criterion, and options; the model processes the formatted input and returns probabilities for those options.

AI Statement

  • AI agents assisted with code, experiments, and writing; generated content was verified, and the authors take responsibility for the final work.Source: AI Statement · p. 9

A. Request format

  • Each request contains a state and one or more questions, with each question answered in its own forward pass.Questions use evidence, a criterion, and named options; choice, yes/no, and score questions encode options differently.Source: A. Request format · p. 13
  • The chat template renders the system and user messages and ends at the model’s next-token position, where the answer is read.The example Gemma 4 input contains 145 tokens.Source: A. Request format · p. 13
  • The decision head maps the final-position hidden state to option logits, which are softmaxed and temperature-scaled into returned probabilities.The first K option slots are used.Source: A. Request format · p. 14

B. Training details

Training-length experiments include branched learning-rate decays, while Qwen3.5-4B uses a model-specific combination of changes. OpenDecisions closely tracks public leaderboard rankings, though system strengths vary across item families.

B. Training details

  • Longer runs branch cosine-decay schedules from saved training states, while smaller-model runs also use shorter decays and schedule-specific endpoints.A branch preserves the saved weights, optimizer, and data position; the 400-step recipe branches at step 300.Source: B. Training details · p. 15
  • 66.6 OpenDecisions Capability results from combining three Qwen3.5-4B changes; the 31B recipe alone does not improve that model.The three changes are a higher LoRA learning rate, distillation, and partial training in evaluation format; the combination was selected on OpenDecisions and is therefore optimistic.Source: B. Training details · p. 15
  • Figure 4 plots OpenDecisions Capability against training steps for Qwen3.6-27B, Gemma 4 12B, and Qwen3.5-4B, with zero-shot performance shown dashed.Source: Figure 4 · p. 15

C. OpenDecisions details

  • Across 14 systems, OpenDecisions’ scores correlate with official scores at Spearman 0.90; a line fitted on the 10 old systems predicts their official scores with 3.0-point RMSE.The comparison covers public systems with official Capability ≥55, each run as its authors serve it and scored with the leaderboard’s scorer.Source: C. OpenDecisions details · p. 16, Table 11 · p. 16
  • The trained 31B model is best or near-best on most item families, while other systems lead on specific families and all systems answer negation pairs correctly.Quyet-1.0-Large leads on fairness pairs and composite rules; Eikos-27B on use cases and perturbation pairs; Jev on counting traps and calibration items.Source: C. OpenDecisions details · p. 16

C.1. Training–evaluation overlap and distribution differences

The analysis finds no exact matches between OpenDecisions and the configured training pool, but the datasets differ in measured similarity and observable input text. The authors caution that this does not establish novel skills or an independent test set.

C.1. Training–evaluation overlap and distribution differences

  • No exact matches appear among 2,389 evaluation questions under checks of full inputs, question text, or supporting context, including standardized text.These checks rule out identical-input reuse but not paraphrased questions or shared underlying skills.Source: §C.1 · p. 17
  • OpenDecisions’ median nearest-neighbor TF–IDF cosine similarity to training questions is 0.156, versus 0.763 for held-out training questions.Excluding training questions with repeated contexts lowers their median similarity to 0.509; the difference persists within question types.Source: §C.1 · p. 17, Table 13 · p. 17
  • A TF–IDF classifier distinguishes training from evaluation questions with 98.82% balanced accuracy, though wording, formatting, and task composition may contribute.Source: §C.1 · p. 17
  • OpenDecisions was used to select the training recipe, so it is a development set rather than an independent test set.The analysis also cannot rule out paraphrased overlap or overlap with backbone pretraining data.Source: §C.1 · p. 17

D. Public evaluation sets

The evaluation combines public held-out decision datasets with seeded samples of knowledge-heavy multiple-choice benchmarks, while checking for overlap with training data.

D. Public evaluation sets

  • The held-out decision sets span public classification, inference, paired-variant, and reference-answer datasets, plus decider tasks; inputs reach 16,384 tokens.A 13-word n-gram check found no overlap for the first six sets and template phrases in 71 of 7,800 decider items.Source: D. Public evaluation sets · p. 18
  • The knowledge-heavy evaluation samples six benchmarks; some are seen tasks with unseen items, while MMLU-Pro and SciQ are not in the training pool.A 13-gram check drops 110 items, including 92 BoolQ passages recurring with different questions.Source: D. Public evaluation sets · p. 18
  • Table 15 reports accuracy and raw-probability ECE on seeded samples without overlapping items, with macro accuracy averaged across six tasks.Its caption also gives Li & Wagle (2026) reference macro accuracies on their own samples.Source: Table 15 · p. 18, D. Public evaluation sets · p. 18

E. Zero-shot accuracy and model size

Zero-shot letter readout accuracy increases with model size before leveling off, and the largest models approach Jev. Tests of order-invariant prompts and alternative calibration losses retain the letter format as the preferred approach.

E. Zero-shot accuracy and model size

F. Order-invariant encoding and calibration losses

  • Order-invariant variants score 0.6 to 2.4 points below letter prompts, while sorting options before letter prompting provides exact input-order invariance without a meaningful accuracy change.The encoding shares the state and question, then scores each option in a separate segment that cannot see the others.Source: F. Order-invariant encoding and… · p. 19
  • The expected-utility objective adds an abstain option, rewarding correct answers +1, wrong answers −r, and abstention 0; its loss is negative expected reward.It is averaged equally with cross-entropy, and the best policy answers only above probability r/(1 + r).Source: F. Order-invariant encoding and… · p. 20
  • No tested calibration loss lowers mean ECE by more than 0.01 versus cross-entropy at 0.092; focal loss reduces accuracy by 2.1 ± 1.7 points.Expected utility with r = 3 scores 79.4 against 78.5 for cross-entropy, but its raw ECE advantage disappears after temperature scaling.Source: F. Order-invariant encoding and… · p. 20

G. Calibration

Temperature scaling improves zero-shot Gemma Capability scores, while its gains are smaller after fine-tuning. Calibration transfers imperfectly across sets, so fitted temperatures below one are restricted.

G. Calibration

  • Temperature scaling improves zero-shot Gemma 4 31B Capability by 6.7 points and Gemma 4 12B by 10.0 points.Table 17 compares raw and scaled Capability; Table 18 gives fitted temperatures by question type.Source: G. Calibration · p. 20, Table 17 · p. 21, Table 18 · p. 21
  • For trained Qwen3.6-27B, fitted temperatures reduce Capability from 81.2 to 81.0; for Qwen3.5-4B, from 66.6 to 64.6.Because calibration temperatures from a separate set do not always transfer, fitted values below one are set to one, preventing sharpening.Source: G. Calibration · p. 20, Table 18 · p. 21
  • Among public systems, temperature scaling improves Capability for deck31b (+3.2), SemIf (+3.7), Winnow-12B (+3.8), and Torchcast-decision-12B (+2.8), but reduces Jev from 80.1 to 78.8.Table 19 reports each system's higher score from its original or scaled predictions; temperatures below one are restricted.Source: G. Calibration · p. 20, Table 19 · p. 22
  • Temperature scaling raises calibration for every question type, but lowers score-question competence when it pulls the predicted expected level toward the middle.For zero-shot choice questions, calibration rises from 58.6 to 84.0; for score questions, competence falls from 64.5 to 54.1.Source: G. Calibration · p. 20, Table 20 · p. 22

H. Readout checks

The readout check finds little probability mass outside the option letters for zero-shot models. A separate comparison shows that instruction tuning improves scores under the tested prompts.

H. Readout checks

  • Ignored probability mass is at most 0.7% on average for every zero-shot backbone and set; the most likely token is almost always an option letter.Exceptions include “Neither” on 9 OpenDecisions items and “NONE” or “None” on 7 validation questions whose options do not fit the state.Source: H. Readout checks · p. 23, Table 21 · p. 23
  • With the plain completion prompt, instruction tuning adds 14.9 points for Gemma 4 31B, 15.9 for Gemma 4 12B, and 4.1 for Qwen3.5-4B.The chat-marker prompt does not work for the pretrained backbones; the trained pretrained 31B model scores 70.3, below the instruction-tuned zero-shot model at 79.6.Source: H. Readout checks · p. 23, Table 22 · p. 23

Related papers

Loading 2610.17205v1…