PaperlayerView exact revision on arXiv

Source-linked AI summary

Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models

Yuchuan Weng, Chi Liu, Xin Chen, Srinivasan Manoharan

arXiv:2610.29052v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Fixed-option operational decisions are commonly automated by generating and parsing text, which spends tokens on a small choice set and can fail to return valid structured output. The paper builds a self-hosted typed judge for coding sessions and applies the Eikos recipe to LoRA-adapted models that read the frozen output layer without adding readout parameters. On 111 hard-tier items, the 35B is significantly ahead of five open systems, but its +6.3-point difference from the commercial reference establishes neither superiority nor equivalence.

  • Problem

    Fixed-option operational decisions are commonly automated by generating and parsing text, which spends tokens on a small choice set and can fail to return valid structured output.

    Source: §1 · p. 1

  • Method

    The paper builds a self-hosted typed judge for coding sessions and applies the Eikos recipe to LoRA-adapted models that read the frozen output layer without adding readout parameters.

    Source: §1 · p. 2, §4.1 · p. 4, §5.2 · p. 11

  • Results

    On 111 hard-tier items, the 35B is significantly ahead of five open systems, but its +6.3-point difference from the commercial reference establishes neither superiority nor equivalence.

    Source: §7.1 · p. 11, §11 · p. 16

  • Takeaways & Limitations

    The studied typed-decision pattern is viable for both in-house session triage and open models, while the reported small-evaluation results leave some differences unresolved.

    Source: §10 · p. 15, §11 · p. 16

    The 1B and 35B were developed with public items visible, and effects below 1.5 points remain unresolved in the one-seed comparisons.

    Source: §10 · p. 15

Abstract

from arXiv · show

A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text. The commercial system Jev popularised this System~1 interface, and open reproductions have followed. We report two uses of the pattern. \emph{Part I} describes an in-house System~1 judge that triages about 148K agentic coding session records for a training corpus. It replaces a frontier LLM judge that cost on the order of \$1,000 per pass and whose stored labels matched a fresh two-judge consensus on only 26.9\% of the sessions where the two judges agreed; a self-hosted frozen backbone with a rank-16 LoRA adapter and a pointer head answers three typed questions in one forward pass, judges 1,000 sessions in about three minutes of one GPU, about \$0.06 at a reference rate and roughly 1\% of the frontier judge's cost, and keeps every session inside the organisation. \emph{Part II} asks whether a readout that adds no parameters can be competitive and whether training one helps. Applying the public Eikos recipe and corpus of about 25K decisions to three further backbones (1B, 2B and 35B), we read the frozen output layer of a LoRA-adapted decoder. On the 111 public hard items of JevBench our single-seed 35B, developed with those items in view, scores 79.3\%, not significantly different from the commercial reference (73.0\%, unadjusted $p=0.12$) and ahead of five open systems after Holm--Bonferroni correction; our 2B is ahead of all four kev checkpoints on the leaderboard. A single-seed ablation finds no detectable benefit from training the 26 letter rows of the output layer, and a retrain on a later corpus release moves the 2B's hard-tier accuracy by 11.7 points, which shows how little small evaluations resolve.

1 Introduction

Typed decisions replace generated, parsed replies with probabilities over fixed options from one forward pass. The paper applies this pattern to an in-house coding-session judge and to open models using a parameter-free readout.

1 Introduction

  • A typed decision returns probabilities for every option from logits or a small hidden-state head, rather than generating and parsing text.The paper contrasts this with prompting a chat model for JSON, which can spend hundreds of generated tokens on a small choice set and return text-composed confidence.Source: §1 · p. 1
  • The paper reports two uses of typed decisions: triaging in-house coding sessions for training data and testing open typed-decision models.Both use a one-pass decision interface, but Part I addresses an internal workload while Part II evaluates open models.Source: §1 · p. 2
  • About 148,000 session records are triaged by a self-hosted judge that answers three typed questions in one pass, at about $0.06 per 1,000 sessions.The frontier judge cost on the order of $1,000 per pass; the self-hosted judge keeps sessions inside the organisation.Source: §1 · p. 2
  • On 111 public hard items, the 35B model scores 79.3%, significantly ahead of five open systems but not significantly different from the commercial reference.The 35B was developed with the public items in view; all JevBench numbers in Sections 6–8 use the public half.Source: §1 · p. 2

2 Background and related work

This section situates typed-decision systems by benchmark scope, implementation choices, and related evaluation and training techniques. Open reproductions differ chiefly in where and how they compute the decision.

Background and related work

  • JevBench version 1.4.2.2 publishes 231 items across easy, standard, and hard tiers, while the judge tier and other items are operator-held.The public tiers contain 48 easy, 72 standard, and 111 hard items.Source: §2 · p. 2

Open reproductions

  • Open reproductions compute decisions with an encoder, direct letter logits, a pointer head, a trained code readout, or Eikos’s parameter-free letter-logit readout.Table 1 summarizes systems from their published code and documentation; Part II follows Eikos’s task-training approach on further backbones.Source: §2 · p. 2, Table 1 · p. 3

Related techniques

3 Part I. An in-house System 1 judge: the task

The paper studies replacing an LLM judge on a growing, sensitive workload of agentic coding sessions, using the open kev architecture as its starting point. It motivates a locally hosted typed judge by the workload’s scale, label drift, data boundary, and need for reproducible control.

3 Part I. An in-house System 1 judge: the task

  • The task is to decide whether each growing session belongs in an in-house coding-model training corpus, using three typed questions about coding work, internal domain, and one of eight fine-grained domains.The pool held about 148,000 records, representing about 114,000 distinct sessions; the workload cannot leave the organisation.Source: §3.1 · p. 3

3.1 148,000 sessions and a judge that could not keep up

  • $1,000 and roughly 73 hours were the projected cost and judging time for one frontier-LLM pass over 148,000 records; fresh judges agreed on only 75.2% of 500 sessions, and their shared labels matched stored labels on 26.9%.The labels came from a two-stage pipeline using rules first and an LLM judge for sessions the rules could not resolve.Source: §3.1 · p. 4

3.2 Two ways to judge a session

  • A frozen model with a small trained decision head assigns probabilities to typed outputs in one pass, instead of reading a session, reasoning in prose, and generating a verdict.The measured LLM-judge path took 1.77 seconds per session and cost about $6.30 per 1,000 sessions; the two approaches are not interchangeable at this scale.Source: §3.2 · p. 4

3.3 Why not call a hosted decision API

  • Session transcripts contain source code, diffs, internal identifiers, and other sensitive material, so the team keeps them inside the organisation rather than sending redacted traces to a hosted API.Self-hosting also lets the team pin the model, weight hash, and serving version for reproducible scores.Source: §3.3 · p. 4

4 Part I. The judge: architecture, training and results

The in-house judge uses a trained pointer head over a frozen, LoRA-adapted backbone to answer three typed questions from rendered session states. On the fixed held-out set, both mixed-retrain judges outperform zero-shot, while 200 sessions cannot distinguish the two backbones.

4.1 Architecture

  • The frozen decoder carries a rank-16 LoRA adapter and a trained pointer head; together they add 34M trainable parameters on 4B and 119M on 27B.The head projects the decision-position and option-span hidden states into 256 dimensions, then uses scaled dot products and softmax.Source: §4.1 · p. 4, Figure 1 · p. 5

4.2 What the judge sees

  • A deterministic script renders each session into a short state of request text, activity counts, tool and file-type summaries, detected frameworks, and repository actions.Both the typed judge and the frontier labeler use this same rendered state; each session is one server request, with the three questions scored as separate causal rows.Source: §4.2 · p. 5

4.3 Data, labels and evaluation

  • The 1,000-session bank was stratified across coding and non-coding sessions, then split 800/200 with strata balanced; held-out accuracy uses single-judge labels.Because the same judge confirmed candidate strata, ambiguous sessions may be under-represented and accuracy may be optimistic for borderline traffic.Source: §4.3 · p. 5, Table 2 · p. 6
  • Six legacy categories are absent from the eight-domain vocabulary, forcing those sessions into an available domain; they account for 27.6% of stored labels and 36.4% of fresh-judge labels in the 500-session sample.The domain question has no “other” option.Source: §4.3 · pp. 5–6

4.4 From zero-shot to a mixed-retrain judge

  • Both mixed-retrain judges beat zero-shot on all three questions by 9.5 to 17.0 points (exact McNemar p ≤0.005 in all six paired comparisons).The comparison is on 200 held-out sessions against single-frontier-judge labels.Source: Table 2 · p. 6, §4.4 · p. 6
  • The 27B pointer head scored 14% on an eight-way exam without eight-way training, versus 59% for released kev-4b and 52% for Phase 2 4B.Chance was 12.5%; the result motivated retraining on the mixed corpus.Source: §4.4 · p. 6
  • At n = 200, neither backbone is significantly more accurate: paired differences are nonsignificant, and all bootstrap intervals span zero.The minimum detectable paired gap is 3.7 to 8.5 points for this comparison; the authors call the cheaper 4B a reasonable default, not equally accurate.Source: §4.4 · p. 7
  • Both judges recalled only 2 of 18 new-feature sessions (4B) and 3 of 18 (27B); Refactor cannot be assessed with three held-out sessions.The authors note that feature-versus-modification boundaries may not be visible from state counts and file types, and that new-feature sessions could be under-counted when balancing a corpus.Source: §4.4 · p. 7

4.5 Overlap between training and held-out sessions

The held-out results remain favorable after removing overlapping sessions; measured inference is inexpensive, while implementation changes support multimodal and sharded backbones. The paper also distinguishes the deployment judge from the open-benchmark experiments and notes limits on reuse claims.

4.5 Overlap between training and held-out sessions

  • On the 144 held-out sessions without overlap, both trained judges still beat zero-shot on all three questions, while no backbone difference is significant.The 56 overlapping sessions were easier for every system; trained-judge accuracies on the clean subset are 1.5 to 3.8 points lower than in Table 2.Source: §4.5 · p. 7

4.6 Speed and cost

  • 176 ms per session for the 4B and 596 ms for the 27B yield about 2.9 and 9.9 minutes per 1,000 sessions; at the reference rate, inference costs about $0.06 and $0.62.The frontier judge takes about half an hour and costs about $6.30 per 1,000 sessions; its timing is end-to-end, unlike the typed judges’ server-side latency.Source: §4.6 · pp. 7–8
  • Projected across 148,000 records, the 4B judge would take about seven A100-hours, making repeated passes feasible compared with a roughly $1,000, 73-hour frontier-judge pass.The A100 price is an assumption, and the inference-cost comparison remains favorable below about $129 per A100-hour for the 4B and $12.7 for the 27B.Source: §4.6 · p. 8

4.7 Changes to the open kev implementation

  • The kev fork extracts text backbones from multimodal wrappers and supports multi-GPU sharding, with adapters kept on their parent layers’ devices.Both recurrent-layer backbones cannot use kev’s option-isolation mask, so each question runs as a separate causal row from the shared session state.Source: §4.7 · p. 9

4.8 Reuse, and how the two parts connect

  • The paper presents Part I as a deployment result and Part II as a separate benchmark study; whether the letter readout matches the pointer head on this task remains untested.Part I’s 200-session held-out set is class-balanced, uses an eight-category vocabulary, and includes 56 sessions overlapping training records.Source: §4.8 · p. 9

5 Part II. Open typed-decision models: readout, recipe and data

Part II tests whether a decoder’s existing output layer can support typed decisions without adding readout parameters. It applies the Eikos recipe and public corpus to three additional backbones, training LoRA adapters while leaving the output layer frozen.

5 Part II. Open typed-decision models: readout, recipe and data

  • Part II adds three backbones, paired JevBench comparisons, and a single-seed test of training the readout, but does not compare letter and pointer heads under matched conditions.The recipe, corpus, and evaluation battery follow Eikos; the two paper parts therefore test related uses of the interface rather than a shared experiment.Source: §5 · p. 10

5.1 A readout with no new parameters

  • The letter readout takes option-letter probabilities from the decoder’s frozen output layer at the position where an answer letter would be generated, training only the LoRA adapter.The prompt contains the evidence, criterion, and lettered options; Boolean, choice, and ordinal questions share the readout, with larger option sets handled in tournaments of at most 26.Source: §5.1 · p. 10

5.2 Recipe and data

  • The study uses Eikos’s recipe and public decision corpus, adapting 1B, 2B, and 35B backbones with LoRA and one-pass inference.Training includes soft cross-entropy, an auxiliary rationale loss used only in training, and per-example option reshuffling; the 1B and 2B also use paired-view KL divergence.Source: §5.2 · pp. 10–11

6 Experimental setup

The experiments evaluate three LoRA-adapted backbones on public JevBench items and a broader seven-suite battery. Paired tests use multiplicity correction, while the authors emphasize that the 111-item hard tier has limited power.

Experimental setup

  • We adapt MiniCPM5-1B, MiniCPM5-2B, and Qwen3.6-35B-A3B, score 231 public JevBench items, and also evaluate on the Eikos repository’s seven-suite battery of 7,371 items.Source: §6 · p. 11
  • Paired comparisons use exact McNemar tests and Holm–Bonferroni correction, with significance defined as adjusted p below 0.05.The 1B comparison with the commercial reference is descriptive, and model selection was not preregistered.Source: §6 · p. 11
  • A 111-item tier has 80% power only for differences of about ten points between runs of one recipe, or twelve to twenty points between unrelated systems.The authors treat nonsignificant results as not evidence of equivalence.Source: §6 · p. 11

7 Results on JevBench

On JevBench’s 111 public hard items, the 35B is significantly ahead of five open systems and not shown to differ from the commercial reference; calibration and selective-decision results carry additional dataset and comparability cautions.

7.1 Accuracy and paired comparisons

  • 54.1, 61.3, and 79.3 are the hard-tier accuracies across the authors’ 1B, 2B, and 35B builds, but the pattern is suggestive given single seeds and different backbones.The 35B’s 79.3 is 13.5 points above SimpleJev and 6.3 points above openjev-sglang, but those systems lack per-item scores for paired tests.Source: §7.1 · pp. 11–12
  • The 35B is significantly ahead of five open systems; against the commercial reference it is +6.3 points (unadjusted p = 0.119), so neither superiority nor equivalence is established on 111 items.The 2B is significantly ahead of all four kev checkpoints and not significantly different from SemIf, jqv, or reflex-4B; the 35B is not shown equivalent to the commercial reference.Source: §7.1 · p. 11

7.2 Calibration and selective decisions

  • The 35B commits on 42.3% of the 7,371-item battery and is wrong on 1.9% of those confident decisions, while the commercial system’s reported error is not comparable.Up to 31% of the battery comprises template siblings of training data; the 2B’s confident-decision error is 7.7% on four external suites versus 4.2% on the full battery.Source: §7.2 · p. 12, Table 4 · p. 13

8 Does the readout need to be trained?

The single-seed ablation detects no accuracy benefit from training the 26 letter rows, while run-to-run differences and a later 2B retrain show that small hard-tier comparisons remain difficult to resolve. The readout’s transfer to an untrained eight-way question was not tested.

Design

  • The ablation updates 26 rows (53K parameters) on about 23K decisions, rather than testing imajev’s shipped 256-row configuration.The trained and frozen arms share data, schedule, and seed; they differ only in whether the copied letter rows receive gradients.Source: §8 · p. 13

Result

  • Training the 26 letter rows yielded no detectable accuracy benefit at either size; the hard-tier test could resolve effects only around ten points or larger.On the 7,371-item battery, both sizes differed by +0.41 points, with item-sampling interval upper ends of +1.24 and +1.21 points; these intervals omit between-run variation.Source: §8 · p. 13

Run-to-run floor

  • Near-identical 2B runs differed by −1.11 points on the battery and +6.3 points on the hard tier, limiting what one-seed comparisons can resolve.Two pairs cannot estimate run-to-run spread; the observed treatment difference is confounded with training-trajectory variance.Source: §8 · pp. 13–14

The hard-tier floor

  • A later 2B retrain scored 73.0% versus 61.3% for the released 2B on the hard tier, a +11.7-point difference that cannot be attributed to any single change.The runs differ in corpus release, code version, batch shape, context limit, and loss; the authors treat hard-tier differences below about ten points between 2B-class systems as unresolved.Source: §8 · p. 14

Relation to Part I

  • The letter readout’s transfer evidence is limited to held-out family, topic, and language; the untrained eight-way question was not tested with it.Part I’s pointer-head results on that question do not establish how the letter readout would perform.Source: §8 · p. 14

9 Provenance, contamination and selection

Part II’s results have provenance and evaluation boundaries: every model used rows later retracted from the public corpus, public-item exposure affected some development choices, and the 1B and 35B remain unretrained. The contamination check found no flagged rows but cannot detect several forms of overlap or teacher exposure.

Training data and a retraction

  • All Part II models trained on 1,994 FinEntity-derived rows later retracted; only the 2B was retrained without them, and the finance results probably depend on those rows.On the related 300-item financial-sentiment task, the 2B retrain fell from 71.7% to 48.7%; the 1B and 35B have not been retrained.Source: §9 · p. 14

Contamination

  • The lexical contamination check flagged no snapshot rows against 231 public items, but cannot detect paraphrase, translation, or regeneration, and teacher exposure is unmeasurable.The corpus deliberately covers the benchmark task taxonomy, and five public items are too short for the check.Source: §9 · p. 14

Selection

  • The 1B and 35B were trained or selected after public-item scores were seen, making their public-item numbers optimistic in the usual way.The 35B exact-recipe build was chosen after both builds’ public scores were known; the upstream recipe and corpus were also indirectly exposed to public items.Source: §9 · pp. 14–15

Reproducibility

  • Part II used public inputs, but only the merged 2B model is released; the 1B and 35B adapters and their per-item scores are not.Part I’s internal session records cannot be released.Source: §9 · p. 15

10 Limitations

The authors bound both parts’ claims by evaluation design, data provenance, and deployment scope. In particular, public-only testing, limited seeds, unmatched comparisons, and since-retracted training rows constrain interpretation of the reported results.

Part I evidence

  • Part I’s judge was evaluated against one frontier judge on 200 class-balanced sessions, 56 of which overlap training records, and its domain question covers only eight of fourteen legacy categories.The conclusions hold without the overlapping sessions; deployment costs are GPU-time estimates and full-pool figures are projections from measured latency.Source: §10 · p. 15

Borrowed recipe, data and evaluation

  • The evaluation battery and training recipe are inherited from Eikos, while trade and rulebook suites are template siblings covering up to 31% of the battery.The authors caution that battery-wide confident-error rates are lower than on external suites.Source: §10 · p. 15

No matched comparison and no untrained baselines of our own

  • No matched same-backbone comparison or untrained baseline measures what the adapters add, and Part I lacks a cheap non-LLM classifier comparison.The kev comparison is across checkpoints differing in base model, data, and training; the ablation tests a 26-row version, not imajev’s shipped configuration.Source: §10 · p. 15

Evaluation

  • Every JevBench result is from the public half, with no sealed-half measurement; the 35B also requires about 80 GB of memory.Systems score far lower on the sealed half than on public items.Source: §10 · p. 15

One seed and selection

  • The 35B and 1B were developed with public items visible, and effects below about 1.5 battery points or about ten hard-tier points remain unresolved.The observed run-to-run differences and one retrain that also changed other factors limit what the single-seed evidence can establish.Source: §10 · p. 15

Retracted training rows

  • All Part II models used since-retracted rows, and only the 2B was retrained without them in a run that changed more than the data.The retrain’s changes prevent attributing its result to the retraction alone.Source: §10 · p. 15

11 Conclusion and next steps

The authors conclude that typed decisions performed usefully in both their in-house and open-model settings, while small evaluations limit what the results can establish. They propose further validation and caution that benchmark confidence does not establish deployment performance for consequential decisions.

11 Conclusion and next steps

  • The in-house judge and open typed-decision models performed strongly in their studied settings, but small evaluations leave important differences unresolved.The 35B was not significantly different from the commercial reference and ahead of five open systems on the hard tier; the 2B was ahead of four kev checkpoints, while a 111-item tier resolves differences only around ten points.Source: §11 · p. 16
  • The authors call for sealed evaluation, multi-seed ablation replication, retraining without retracted rows, and direct comparison of the open readout with the pointer head.They identify the 2B, after retraining without retracted rows, as the candidate deployable open model; its single-question latency is 20.5 ms versus 83.2 ms for the 35B on one H100.Source: §11 · p. 16

Broader Impact Statement

  • For consequential deployments, the authors recommend recalibrating on local traffic and keeping humans in the loop because benchmark error rates are not deployment-distribution measurements.They describe the confidence threshold as a policy choice.Source: Broader Impact Statement · p. 16

F Part I: inputs, training configuration and overlap check

The judge receives a deterministic, compact session state and answers binary coding/domain questions plus an eight-category domain classification. The section documents its LoRA training setup and an overlap-excluded evaluation.

Inputs

  • The judge classifies coding work, organisational-domain relevance, and one of eight fine-grained domains; the illustrative state shows the field layout, with altered values that are not from a real session.The domain categories, model-facing criteria, and support in a 200-session held-out set are listed in Table 14.Source: F Part I: inputs, training… · p. 21, Table 14 · p. 21

Training configuration

  • Both backbones use kev’s trainer, LoRA on every linear projection, and a 256-dimensional pointer head, with a 384-token state limit and seed 0.The runs use AdamW, a one-cycle schedule with 10% warm-up, bf16 computation, and A100-40GB GPUs; the 4B retains fp32 master weights and the 27B uses bf16 weights.Source: F Part I: inputs, training… · p. 21, Table 15 · p. 21

Overlap check

  • The overlap check found 56 held-out state matches among 1,818 training records; Table 16 evaluates the 144 held-out sessions with no matching state.Held-out sessions were rendered using the same script as the training corpus before identical-state matching.Source: F Part I: inputs, training… · p. 22, Table 16 · p. 22
  • For the six Table 16 comparisons, exact McNemar p-values against zero-shot are at most 0.017; 4B-versus-27B p-values are 0.52, 0.45, and 1.00.The table reports accuracy, percentages, and exact 95% Clopper–Pearson intervals for the 144 no-match sessions.Source: Table 16 · p. 22

Related papers

Loading 2610.29052v1…