PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

Tardelli Ronan Coelho Stekel

arXiv:2610.30970v1cs.CLcs.LG

TL;DR

Evidence comparing joint and per-option reading for typed decisions was limited, and the paper examines that choice alongside encoder size. Its 32M comparison favors per-option scoring from a reranker but not from a raw encoder; latency and task scope remain practical constraints.

  • Problem

    Controlled comparisons of joint and per-option readers for typed decisions were lacking, despite their differing costs and design choices.

    Source: §1 · p. 1, §10 · p. 14

  • Method

    The paper compares joint and per-option readers at 32M from different initial checkpoints and evaluates compact joint encoders across external and private tests.

    Source: §10 · p. 14

  • Results

    At 32M and matched compute, per-option scoring from a reranker leads joint scoring by +0.028 public-suite skill, while the readers are as good from a raw encoder.

    Source: §10 · p. 14

  • Takeaways & Limitations

    With a reranker and short option lists, per-option scoring is more accurate and order-invariant by construction; joint scoring uses one pass per question and showed no loss from a raw encoder in these runs.

    Source: §8 · pp. 13–14

    The models are over-confident on new tasks, do not replace supervised in-task training or large models on knowledge-heavy and new-domain decisions, and were trained and evaluated only in English.

    Source: §8 · p. 14

Abstract

from arXiv · show

Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.

1 Introduction

Closed-set decisions can be handled by prompting an LLM, scoring each option, or fine-tuning a task-specific classifier; controlled evidence comparing joint and per-option reading for typed decisions was lacking. The paper trains compact joint encoders and compares reader designs, external performance, robustness, latency, and private-test results.

1 Introduction

  • Closed-set decisions include changing label sets, for which prompting an LLM is slow and order-sensitive, NLI scoring requires one pass per option, and task-specific classifiers do not transfer to new label sets.The typed-decision format takes a structured request as input and returns probabilities for allowed options without generated text.Source: §1 · p. 1

1.1 Contributions

  • Crivo comprises six joint encoders from 17M to 1B parameters, fine-tuned from Ettin rerankers with an [OPT] readout adapted from Dinah-0.The evaluation includes a controlled 32M reader grid, external benchmarks audited for training overlap, and the board’s private tests.Source: §1 · p. 1
  • +0.058 public-suite skill at matched optimizer steps and +0.028 at approximately matched compute favor per-option scoring over joint scoring from a reranker; from a raw encoder, the readers are indistinguishable.The comparison used a 32M model and three seeds per arm; external benchmarks had the same direction but no significant difference on unseen tasks.Source: §1.1 · pp. 1–2
  • On all 209 S1-Arena tasks, Crivo-68M and Crivo-150M exceed the 435M DeBERTa-v3-large zero-shot classifier by +0.049 and +0.059 task-mean accuracy, and are at least on par on 145 unseen tasks.Crivo-400M and Crivo-1B match or exceed published zero-shot rows of up to 8B parameters on four unseen BTZSC datasets.Source: §1.1 · p. 2, Figure 1 · p. 2
  • Reordering options changes 12–23% of Crivo’s answers, compared with 2% for DeBERTa and none for per-option scoring; Crivo-17M’s median request latency is 13.5 ms on 16 CPU threads versus 728 ms for DeBERTa.The models are over-confident on unseen tasks.Source: §1.1 · p. 2
  • On private leaderboard tests, Crivo-17M, -32M, and -68M lead their size ranges on mostly in-distribution skills but stay near the floor on new domains.Development panels drawn from the training-data pool overestimated every model’s private score; Prumo matches Crivo-17M on same-skill tests while staying near chance externally.Source: §1.1 · p. 2

2 Related Work

Related work covers alternative ways to score candidate options, typed-decision models, compact encoder construction, calibration, benchmark reuse, and parameter-free baselines. Crivo’s joint reader builds on prior typed-decision readouts while allowing options to attend to one another.

2.1 Option scoring: per-option, joint and listwise

  • Per-option readers encode context once per option; joint readers place all options in one sequence, while listwise retrieval models can make results invariant to option order.Crivo uses a joint reader whose options attend to one another, with a fresh head on a reranker backbone and training on typed decisions.Source: §2.1 · pp. 2–3

2.2 Typed decision models

  • Typed-decision work includes the System One request format, S1-Arena, encoder-based decision models, and structural requirements such as permutation invariance.Crivo’s [OPT] readout comes from Dinah-0.Source: §2.2 · p. 3

2.3 Cross-encoders, compact encoders and distillation

  • Crivo uses Ettin reranker fine-tunes based on ModernBERT-style encoders, and smaller family members are distilled from larger ones using soft targets and attention distillation.The passage contrasts jointly encoding context and candidates in a cross-encoder with siamese bi-encoders.Source: §2.3 · p. 3

2.4 Calibration and selective prediction

  • Prior work reports that pretrained transformers are less calibrated out of domain, with decision-model findings including both over-confidence and under-confidence.The paper reports expected calibration error, Brier score, and selective-prediction risk.Source: §2.4 · p. 3

2.5 Leaderboards, holdout reuse and contamination

  • Prior work identifies test-set reuse, distribution shifts in fresh test sets, and selective leaderboard reporting as risks; contamination audits are one response.The paper’s negative result concerns its own development panel as a failed proxy, and it reports quality against cost.Source: §2.5 · p. 3

2.6 Rule-based and parameter-free baselines

  • Parameter-free classifiers can provide strong diagnostic baselines; Prumo combines fixed classical components to probe what a decision benchmark rewards without learning.Its components include BM25, code execution, game-tree search, truth tables, and calendar arithmetic.Source: §2.6 · p. 3

3 Task and Evaluation Protocol

The protocol defines typed decisions and evaluates Crivo on external benchmarks and a public 37-benchmark suite, with explicit contamination, baseline, statistical, robustness, and latency procedures.

3.1 Typed decisions

  • A request combines a state with one or more instructed questions; each expects probabilities over options, with the answer given by the argmax.The format includes multiple-choice, binary-predicate, and ordinal questions; probabilities also support ranking and selective prediction.Source: §3.1 · pp. 3–4

3.2 External benchmarks

  • System1-Arena covers 209 tasks across 13 non-agentic domains; BTZSC covers 22 classification datasets, of which only four are unseen.System1-Arena uses per-task accuracy averaged across tasks; BTZSC uses macro-F1 averaged across datasets.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Contamination policy and audit

  • External evaluation separates unseen data from seen splits and overlap using source-corpus and item-string audits; only unseen results are treated as evidence of generalisation.The audit defines unseen as a different source dataset or corpus and less than 0.5% test-item overlap under its stated matching rules.Source: §3.3 · p. 4

3.4 Baselines

  • Baselines include zero-shot NLI classifiers, small LLMs using option-letter readouts, an untrained reranker, and uniform guessing.Unprocessable inputs count as wrong; several other compact decision models were not run on the external benchmarks and have no published results there for comparison.Source: §3.4 · pp. 4–5

3.5 The Decision Index public suite

  • The Decision Index combines 37 benchmarks across five areas using a weighted mean of chance-corrected benchmark scores.Its official ranking uses private tests run by leaderboard maintainers.Source: §3.5 · p. 5

3.6 Statistics

  • Benchmark intervals use paired item-level bootstraps, with wider task- or dataset-resampling intervals addressing performance on similar tasks.The three-seed ablation intervals assume normality and are treated as suggestive when they barely exclude zero.Source: §3.6 · p. 5

3.7 Robustness and latency measurement

  • Option-order robustness measures answer changes under randomized, reversed, and swapped options; latency is timed per request on a GPU and a 16-thread CPU.The robustness sample includes 3,841 multiple-choice questions and 258 binary requests; CPU latency uses the first 80 requests answerable by every system.Source: §3.7 · p. 5

4 Models

Crivo uses a joint [OPT] readout with models initialized from Ettin rerankers, trained on a staged mixture of decision data. The section also describes Prumo, a no-learned-parameter floor.

4.1 Reader

  • The joint reader places one [OPT] marker per option in a single sequence, scores marker states with a question-type embedding, and normalizes scores across options.The per-option reader instead encodes one context-option sequence per option using the same head and softmax.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Model family

  • All six released Crivo models start from same-size Ettin rerankers and share a training recipe.Two earlier-recipe models appear only as the Crivo-17M teacher and in a recipe comparison.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Training data

  • The training mix combines tasksource families, procedural and generated decisions, synthetic tool-use dialogues, and capped training splits from public-suite source datasets.It contains 638,306 questions in stage 1, 169,437 in stage 2, and 24,635 in stage 3; 40 tasksource families are held out entirely.Source: §4.3 · p. 6

4.4 Objective, stages and selection

  • Training uses cross-entropy plus a Brier-score term, with teacher distillation for some data, and proceeds through stages up to 8,192-token inputs.Checkpoints are selected using internal panels, while each model's complete public-suite run and leaderboard submission occur after the checkpoint is fixed.Source: §4.4 · p. 6

4.5 Prumo: a floor with no learned parameters

  • Prumo applies fixed-order capabilities such as code execution, retrieval, logic, and arithmetic, returning uniform probabilities when no capability claims a question.Its outputs are deterministic across machines, but its request-code execution sandbox should not be exposed to untrusted input without further isolation.Source: §4.5 · pp. 6–7

5 Reader Study: Joint versus Per-Option Scoring

In a three-seed 32M study, per-option scoring outperformed joint scoring on the held-out suite even against a compute-matched joint control, but the advantage depended on reranker initialization. That accuracy edge came with substantially more encoder passes, while other joint-reader objective changes had small effects.

5.1 Design

  • J-comp matched P’s optimizer steps and per-step token budget by training for P’s number of steps, but repeated each row about 3.25 times rather than adding fresh data.It was close to, but not exactly matched in, encoded tokens and GPU time.Source: §5.1 · p. 7

5.2 Per-option against joint reading

  • +0.028 pub15 skill [0.015, +0.042] for P over J-comp at approximately matched compute; P also led on S1-60 and BTZSC-500.P scored +0.031 on S1-60 and +0.070 on BTZSC-500; these unseen-task differences were not significant. J-comp repeated rows rather than adding fresh data.Source: §5.2 · pp. 7–8, Figure 4 · p. 7, Table 3 · p. 8
  • The reranker initialization helped P on all three evaluation sets but helped J-comp only on BTZSC-500; from the raw encoder, the readers were indistinguishable at matched compute.The authors interpret the pattern as a format match: rerankers train on query-document pairs, which resemble per-option inputs, but note that the experiment does not test this mechanism.Source: §5.2 · p. 8
  • The released Crivo-32M scored 0.305 versus P’s 0.245 on pub15, but the authors caution that this comparison used different training budgets.On the external sets, P was close to the released model; those unpaired comparisons do not establish which would win with the same full training recipe.Source: §5.2 · p. 9

5.3 Cost

  • Per-option scoring required 5.2–13.1 times more forward sequences per request, with costs concentrated in mean and tail latency.On CPU, median latency was 41.0 versus 21.9 ms, while mean latency was 3,728 versus 139 ms; long requests with many options carried much of the cost.Source: §5.3 · p. 9

5.4 Objective and initialisation of the joint reader

  • On pub15, removing the Brier term changed skill by +0.002 [−0.009, +0.012], whereas removing distillation cost −0.003 [−0.006, −0.000] skill.The study could not claim that the Brier term improves calibration.Source: §5.4 · p. 9

6 External Results

Across external evaluations, compact Crivo models performed strongly against zero-shot baselines, especially on all S1-Arena tasks, while unseen-task results were more qualified. The section also reports option-order sensitivity, over-confidence on unseen tasks, and latency advantages over DeBERTa-v3-large on the stated hardware.

6.1 S1-Arena: external typed decisions

  • Across all 209 S1-Arena tasks, every Crivo model outperformed every baseline with item intervals excluding zero; all pairings also excluded zero at task level except Crivo-17M versus DeBERTa.On 145 unseen tasks, Crivo-68M (+0.027) and Crivo-150M (+0.028) were ahead with both intervals excluding zero, but neither survived Holm adjustment. The authors read results for models up to 150M as parity with a 435M zero-shot classifier.Source: §6.1 · p. 9

6.2 BTZSC: zero-shot text classification

  • On unseen BTZSC datasets, Crivo-400M scored 0.636 and Crivo-1B scored 0.686, matching or exceeding published zero-shot rows up to 8B parameters.The comparison was unpaired across separate runs, and the four-dataset unseen set had a wide spread, so the authors treat BTZSC as secondary evidence.Source: §6.2 · pp. 9–10, Table 5 · p. 10

6.3 Robustness to option order

  • Reordering options changed 12–23% of Crivo answers, while the NLI cross-encoder changed 2% and the per-option reader changed none.Crivo’s change rates fell with model size; the joint reader sees option order, unlike per-option scoring, which reads each option alone.Source: §6.3 · pp. 10–11, Figure 6 · p. 10

6.4 Calibration

  • Crivo models were over-confident on unseen tasks: on unseen S1-Arena tasks, every Crivo had a Brier score no better than the uniform guess (0.619).On BTZSC, every Crivo had lower ECE than every baseline across all 22 datasets, but this no longer held across the four unseen datasets.Source: §6.4 · p. 11, Table 6 · p. 11

6.5 Efficiency

  • Crivo-17M’s median latency was 13.5 ms on 16 CPU threads, versus 728 ms for DeBERTa-v3-large zs-v2 on the same requests.The median hides a heavy tail of long requests; the section reports that CPU means and 95th percentiles are dominated by a few very long requests.Source: §6.5 · p. 11, Table 7 · p. 11

7 Supporting Evidence: Private Tests of the Decision Index

The private Decision Index tests place Crivo models among the leaders of their size range, but new-domain scores remain weak for the smaller models. The section also finds that development panels mispredicted private results and that a parameter-free rule engine can perform well on format-specific tests without transferring to external benchmarks.

7 Supporting Evidence: Private Tests of the Decision Index

  • The board’s same-skill tests use fresh items for tasks whose train splits Crivo used, so only its new-domain tests measure new domains; the authors treat the board as supporting evidence after external benchmarks.The board is also sparse at these model sizes: only 24 of 204 sized entries have at most 420M parameters, and five are Crivo models.Source: §7 · pp. 11–12

7.1 Official scores

  • Crivo-68M (17.69) is not outscored by any other author’s entry below 395.9M parameters, and Crivo-17M starts the size–Full Pareto frontier.Crivo-1B scores 28.96, the highest-scoring encoder on the board, but is not on the frontier; the field below 420M is sparse.Source: §7.1 · p. 12, Table 8 · p. 13
  • New-domain scores stay between 3.2 and 5.5 through 150M parameters, then rise to 9.09 at 400M and 11.39 at 1B.Same-skill scores generally rise with size, from 15.88 to 38.75, except at 150M.Source: §7.1 · p. 12

7.2 Public index and private tests

  • The public index correlates with private same-skill scores across 208 entries (Pearson r = 0.981 [0.974, 0.986]), including after controlling for log parameters (partial correlation 0.965 [0.951, 0.974]).The bootstrap intervals are too narrow because the bootstrap is not clustered.Source: §7.2 · p. 12

7.3 A negative result: development panels moved with the training data

  • Internal development panels overestimated private same-skill scores by 5.6–7.1 points for every released model and ranked two training recipes in the wrong order.Under the new recipe, both panels rose at both sizes, while private scores fell from 27.74 to 24.93 at 150M and from 18.84 to 18.44 at 32M.Source: §7.3 · pp. 12–13

7.4 What a rule engine reveals about the board

  • Prumo, a rule engine with no learned parameters, scores 12.21 Full, ties Crivo-17M on same-skill tests (15.78 against 15.88), and is near chance on external benchmarks.The authors read this as evidence that part of the board rewards format-specific solving; Prumo’s board ECE is 0.047, mostly because unclaimed questions receive uniform answers.Source: §7.4 · p. 13
  • Prumo’s public–private gap across 34 benchmarks tracks how hand-written capabilities transfer to fresh items: lexical tool retrieval rises from 0.39 to 0.51, while CRUXEval falls from 0.997 to 0.28 and BBH from 0.47 to 0.07.The authors suggest exact solvers may stop firing when a task is presented in a different format.Source: §7.4 · p. 13

8 Discussion and Limitations

The discussion frames reader choice as a trade-off shaped by initialization, option-list length, and latency needs. It also sets limits on what the reader study and external evaluations establish, and on where the models should be used.

8 Discussion and Limitations

  • With a reranker and short option lists, per-option scoring is more accurate and order-invariant by construction; joint scoring costs one pass and showed no loss from a raw encoder in these runs.The choice also depends on how heavy the request distribution’s latency tail is.Source: §8 · pp. 13–14
  • The reader study covers one 150k-row subset and one size, 32M, and neither reader was controlled with the full recipe or tested above 32M.Matching compute by repeating rows leaves a stricter control—about 3.25 times as many distinct rows for the joint arm—for future work.Source: §8 · p. 14
  • External evaluation uses a name- and string-based S1-Arena audit and only four unseen BTZSC datasets; comparisons mix backbone, readout, and training data, isolating only the readout.The comparison is against generic NLI classifiers and small LLMs, not the closest same-interface encoders.Source: §8 · p. 14
  • Private scores come from one run on tests the authors cannot inspect, while latency lacks batched-throughput or operation-count measurements and uses an 80-request CPU sample.The small sample makes mean latency sensitive to a few long requests.Source: §8 · p. 14
  • Crivo is over-confident on new tasks, so routing thresholds must be set on the target task; the models do not replace supervised in-task training or large models for knowledge-heavy and new-domain decisions.Training and evaluation are in English only.Source: §8 · p. 14

9 Data and Code Availability

The paper releases the Crivo models and reader-study checkpoints, along with Prumo and experiment materials. The training mixtures are not redistributed because some sources carry their own licenses.

9 Data and Code Availability

  • The six Crivo model weights and cards are publicly available on Hugging Face under CC BY-NC 4.0.The v1 models are earlier revisions of Crivo-32M and Crivo-150M.Source: §9 · p. 14
  • The three per-option 32M reader-study checkpoints and inference engine are released, and Prumo is available under Apache-2.0 with public-suite run outputs.Source: §9 · p. 14
  • Experiment code, table-and-figure scripts, the experiment log, and an archived board snapshot are public, but the training mixtures are not redistributed.Some source datasets carry their own licenses.Source: §9 · p. 14

10 Conclusion

The conclusion recommends choosing between readers according to starting checkpoint and latency budget, while noting that compact joint encoders perform competitively in the stated evaluations. It identifies broader reader-study controls and evaluation coverage as future work.

10 Conclusion

  • At matched compute, per-option scoring is more accurate from a pair-trained reranker (+0.028 public-suite skill), while the cheaper joint reader is as good from a raw encoder.Per-option scoring costs 5–13 times the encoder passes; the conclusion reports the same sign on two external benchmarks but no significant difference on unseen tasks.Source: §10 · p. 14
  • Crivo models from 68M up beat a 435M zero-shot NLI classifier on all overlap-audited S1-Arena tasks, and models up to 150M are at least on par on unseen tasks.The 17M model answers a median CPU request in about 14 ms; on private leaderboard tests, the smallest models lead their size range on in-distribution skills but stay near the floor on new domains.Source: §10 · p. 14
  • Future work should train per-option readers with the full recipe and above 32M, compare same-interface encoders under the same audit, and widen new-domain coverage.Source: §10 · p. 15

Related papers

Loading 2610.30970v1…