PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyJev Technical Report

Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu, Liang Wu

arXiv:2610.00831v1cs.LG

TL;DR

Agent systems need probability-bearing choices among fixed options, but prose responses require parsing and provide no thresholdable probability. AnyJev reads option probabilities from a frozen model and corrects its label and position biases; on both 20-option tasks, rotations reduced order flips across tested models.

  • Problem

    For agent tasks requiring fixed-option decisions, prose must be parsed and provides no probability to threshold.

    Source: §1 · p. 1

  • Method

    AnyJev restricts a frozen model’s next-token distribution to option labels, divides out a label prior, and averages log probabilities over cyclic option rotations without parameter changes.

    Source: §1 · p. 1, §4.1 · p. 4, §10 · p. 11

  • Results

    On both 20-option tasks, order-flip rates fell from 0.33 to 0.138 and 0.183, respectively, on every tested model.

    Source: §10 · p. 11

  • Takeaways & Limitations

    A stopping rule selected against the full-rotation decision reduces the number of rotations read.

    Source: §10 · p. 11

    Confidence calibration uses a temperature fitted on 100 to 500 labeled examples of the same question; no label-free detection procedure was measured.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. Systems that need typed decisions today use models trained for that purpose. This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. The readout restricts the next-token distribution at the answer position to the option tokens. It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. Reading every rotation requires K prefills. A stopping rule selected against the full-rotation decision on unlabelled states cuts that. Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source.

1 Introduction

AnyJev reads fixed-option decisions from a frozen language model’s next-token distribution, avoiding prose parsing and outputting probabilities. It targets label and position biases in that readout without changing model parameters.

1 Introduction

  • For agent tasks such as routing, injection detection, and scoring, prose must be parsed and supplies no probability to threshold.A typed decision instead returns a distribution over a fixed set of options.Source: §1 · pp. 1–2
  • A single prefill reads the next-token distribution at the answer position, restricted to option-label tokens, with no parameter changes or generated tokens.AnyJev returns a distribution over options rather than parsed prose.Source: §1 · pp. 1–2
  • Reversing the option list changes the answer on 33% of items, averaged over eleven models on a 20-option task.The report identifies state-independent label preferences and position preferences as readout defects.Source: §1 · p. 2
  • On the public typed-decision benchmark subset, eight models span 0.549 to 0.798 accuracy, and debiasing does not measurably change accuracy.The report separates this accuracy result from order-dependence reduction and confidence calibration.Source: §1 · p. 2

2 Related Work

Prior work spans typed-decision interfaces, frozen-model readouts, and learned decision systems. AnyJev focuses on label and position bias and confidence calibration in readouts over frozen backbones.

2.1 Typed decision models and open implementations

  • Jev popularizes probabilistic Choice, Score, and Noul interfaces, while open implementations include frozen-model readouts and learned decision systems.Other releases explore candidate scoring, shared-context execution, image decisions, and numerical outputs [52] [51].Source: §2.1 · p. 3

2.2 Probability readouts and calibration

  • Prior work studies label-probability biases and calibration; AnyJev focuses on label and position bias and calibrated confidence on frozen backbones [20] [56] [55] [58] [19].Related studies also identify option-name sensitivity and inconsistency across logically linked questions [43] [29].Source: §2.2 · p. 3

2.3 Decision components in agent systems

  • Typed decisions appear in agent applications including browser control, history pruning, and code retrieval.Proposed combinations across agent and other application settings require application-level evaluation [47] [28] [50] [16] [45] [57] [9].Source: §2.3 · p. 3

3 The readout

The readout turns the next-token distribution into a probability distribution over fixed options using single-token labels. It also exposes answer-mass checks and identifies label and position preferences as sources of unreliable decisions.

3 The readout

  • Each option receives a single-token label, and the prompt places the state before the question and labeled options to allow prefix reuse across layouts.The prompt ends with an instruction to answer using only a label.Source: §3 · p. 3
  • The readout restricts and renormalizes the next-token distribution to the K label tokens, then maps the probabilities from positions back to options.Scoring labels rather than option strings avoids surface-form competition from spelling length or rarity [20].Source: §3 · p. 3
  • Answer mass measures the model probability reaching option tokens and can flag a prompt ending at the wrong answer position.The report excludes rows with low answer mass; a gpt-oss template fault produced 0.000 despite plausible-looking answers.Source: §3 · p. 4
  • Every benchmark row has median answer mass of at least 0.995, but three injection-task rows below 0.99 are excluded from that task’s statistics.The excluded values are 0.408, 0.710, and 0.976 on SmolLM2-1.7B, OLMo-2-7B, and Granite-3.3-8B, respectively.Source: §3 · p. 4
  • Reversing the option list changes the argmax on 32.8% of banking20 items and 33.4% of newsgroups items, averaged over eleven models.The readout also favors some labels regardless of state and some positions in the printed list.Source: §3 · p. 4

4 Correcting the two defects

AnyJev estimates and divides out a label prior, then averages log probabilities across cyclic option rotations to address position bias. The method’s exact cancellation depends on an additive bias model, while confidence calibration uses labeled examples.

4.1 The label prior

  • The label prior is estimated as the running mean of readout distributions on unlabelled inputs, held per question and layout, and divided out with α controlling correction strength.The method uses α = 0.75; its prior-marginal assumption can fail and lower accuracy when input labels are extreme.Source: §4.1 · p. 4

4.2 Position bias

  • AnyJev averages pre-prior log probabilities over K cyclic rotations so every option occupies every position once.If option logits decompose into an option term plus a position term, this log-space average exactly recovers the option distribution; averaging probabilities lacks that property.Source: §4.2 · p. 4
  • After averaging, reversal still changes answers on 0.138 of banking20 items and 0.183 of newsgroups items; canonical sorting would make prompts invariant to caller order but was not measured.The prior is averaged before correction because a per-layout prior contributes a term that cancels only when it depends on position alone; the additive bias model is an approximation.Source: §4.2 · pp. 4–5

4.3 Confidence

  • Neither correction makes confidence match accuracy, and the report did not measure a label-free procedure for detecting that mismatch.The reported temperature correction is fitted on 100 to 500 labeled examples of the same question [19].Source: §4.3 · p. 5

5 Reading fewer rotations

AnyJev reduces the cost of rotation averaging by stopping once a running decision appears settled, while targeting agreement with the full-rotation decision rather than accuracy. The procedure selects a threshold on unlabelled states and can verify its disagreement bound on separate states.

Stopping statistic

  • The threshold rule uses the top-two log-probability gap of the running rotation average and stops at the first value reaching τ, after at least two rotations.The log-space gap avoids the probability gap's saturation at 1 and is unchanged by renormalisation.Source: §5 · p. 5

Unlabelled stopping rule

  • The stopping target is agreement with the decision from all rotations on at least 1 − ε of unlabelled inputs, not accuracy.The procedure compares each stopped decision with the full-rotation decision on unlabelled states.Source: §5 · p. 5
  • A candidate threshold is selected using a Clopper–Pearson upper bound on the number of stopped decisions that disagree with the full-rotation decision.Thresholds are checked on a grid; if none satisfies the bound, the question reads every rotation.Source: §5 · pp. 5–6

Verified stopping results

  • On two of four cells, disjoint-state selection and verification certify a 0.0079 disagreement bound while reading 10.61 of 18 and 9.01 of 20 rotations.Qwen2.5-7B meets the target on both tasks; Qwen3-8B misses it on newsgroups at 0.0105, and no threshold clears it on massive_route.Source: §5 · p. 6

Serving design

  • Rotation requests trade fewer forwards against more completion rounds, while shared-prefix caching already recovers most of the cost of evaluating all rotations.Rotations are read in a van der Corput order so short prefixes place options in well-separated positions; selection and verification also require disjoint states because a bound for each grid candidate does not cover the selected threshold.Source: §5 · pp. 5–6

6 Experimental setup

The experiments evaluate frozen instruction-tuned models across typed-decision tasks, a public benchmark subset, and a separate full-split evaluation. The setup distinguishes unlabelled corrections from quantities fitted using gold labels and reports serving measurements under specified hardware and software conditions.

Models

  • Thirteen released instruction-tuned models from eight organisations are tested without fine-tuning, quantisation, or modification.The 20-option sweep uses eleven models and the benchmark run uses eight.Source: §6 · p. 6

Tasks

  • The 20-option sweep covers three tasks with 300 held-out items per cell, while timing uses an 18-option routing question.The tasks are banking20, 20 Newsgroups, and a binary prompt-injection question; routing is built from MASSIVE.Source: §6 · p. 6

Benchmark

  • The public JevBench subset scores 213 of 231 items and is reported separately from leaderboard entries because its overall score averages three tiers partitioning a larger set.Eighteen items with expected values outside the listed options are skipped rather than mapped.Source: §6 · p. 6

Fitting protocol

  • Rotation averaging uses no labels; the prior strength is selected on a separate gold-label sweep, temperature is fitted out of fold, and the stopping threshold uses unlabelled states of the same question.Temperature folds are over pooled items rather than within difficulty tiers.Source: §6 · p. 6

Settings and hardware

  • Main tables use every rotation and log-space combination; the benchmark omits the label prior to isolate rotations, and the separate BEV evaluation uses its own protocol.Experiments used one H100 NVL; the serving comparison additionally used vLLM 0.7.0 on the same host, measured serially while otherwise idle.Source: §6 · p. 7

7 Results

Rotation averaging improves order consistency and accuracy on both 20-option tasks, while benchmark accuracy changes are not measurable and calibration responds mainly to temperature. The full BEV evaluation and serving results show how outcomes vary by evaluation setting and engine configuration.

7.1 Order dependence falls on every model tested

  • Order-flip rates fall from 0.328 to 0.138 on banking20 and from 0.334 to 0.183 on newsgroups, while accuracy rises on all 11 models on both tasks.Mean accuracy gains are 7.2 and 5.9 points; the exact sign-test value is p = 4.9 × 10−4 for each count.Source: §7.1 · p. 7, Table 1 · p. 8
  • On the binary injection task, rotations reduce order-flip rate from 0.180 to 0.000 but do not raise accuracy significantly: they win on 3 of 7 non-tied rows (p = 0.77).The task has only two option positions, leaving little position bias to average away.Source: §7.1 · p. 7
  • On banking20, rotations and the prior lower calibration error from 0.251 to 0.223, and temperature lowers it further to 0.088.The temperature, rather than rotations, is the reported calibration improvement.Source: §7.1 · p. 7

7.2 The benchmark

  • On the JevBench public subset, accuracy ranged from 0.549 (Qwen3-1.7B) to 0.798 (Qwen3-32B); debiasing changed accuracy by no measurable amount, with pooled difference +0.006 across 1,704 decisions.The subset scored 213 items per model; its items had 2 to 6 options, averaging 3.6.Source: §7.2 · p. 8, Table 2 · p. 9
  • The benchmark's estimated accuracy change is +0.0059 per additional option, with interval [−0.0012, +0.0132], so the sample does not resolve the trend.Temperature lowers calibration error on seven of eight models; four improvements have intervals excluding zero, while Granite-3.3-8B's increase from 0.270 to 0.301 has an interval containing zero.Source: Figure 3 · p. 8, §7.2 · p. 9

7.3 Full BEV Decision Mix evaluation

  • On the full BEV Decision Mix test split of 46,320 decisions, rotation averaging scores 68.01% on Qwen3.5-9B and 67.59% on Qwen3-32B, versus raw-readout scores of 67.85% and 67.03%.Adding the prior lowers accuracy to 65.01% and 65.55%, although it improves the 9B model's ECE from 0.0490 to 0.0418; the bespoke system's 70.57% is an observed system difference, not an isolated fine-tuning effect.Source: §7.3 · p. 9

7.4 Serving cost

  • Reading two rotations per engine call issues 7.3 and reaches 37.2 decisions per second, 2.22 times the all-rotations rate; four per call reaches 31.8.The table covers 300 decisions on an 18-option question using one H100.Source: §7.4 · p. 10, Table 3 · p. 10

8 What did not work

Several alternatives either fail to generalize across questions or reduce accuracy, while cache reuse and packing rotations preserve the computation. These results motivate reducing the number of rotations rather than their unit cost.

Question-agnostic head

  • A question-agnostic head trained on other questions scores below the raw readout on held-out questions across all tested variants.Its best held-out score is 0.580, below 0.635 for the debiased readout and 0.628 for the raw readout; the signal is present per question, but not in a shared linear rule that reaches an unseen question.Source: §8 · p. 10

Reusing the question block’s cache

  • Reusing a cached question block is exact to 7.0 × 10−5 in fp32 across all 28 layers and reproduces the previous state's decision in a new request.The question and options follow the state, so no prefix cache reaches them; on a short state they account for 159 fixed prompt tokens.Source: §8 · p. 10

Making the options a set

  • Making option spans exchangeable by masking their interactions reduces shuffle sensitivity but collapses accuracy to 0.042, below the 0.056 chance level.On Qwen2.5-7B routing, the rotation average scores 0.633; masking option interactions while retaining positions scores 0.092, indicating the loss comes from blinding the model to other options.Source: §8 · pp. 10–11

Packing the rotations into one forward

  • Packing all rotations into one forward reproduces independent prefills to 2.2 × 10−5 and runs 5.9 times as fast.It is 1.20 times as fast as the shared-prefix path on a 10-token state and 1.11 times on a 200-token state, motivating a focus on reading fewer rotations.Source: §8 · p. 11

9 Limitations •

The report’s limitations include the readout’s inability to reason, gaps in benchmark coverage, and constraints on the strength and scope of its measurements.

9 Limitations

  • One prefill cannot reason; on the JevBench subset’s hardest tier, the eight models range from 0.343 to 0.590.The report limits this readout to the base model’s decision capability.Source: §9 · p. 11
  • JevBench items average 3.6 options, and no public many-option benchmark tests rotations where the mechanism has most to remove.The report says such a benchmark would test the mechanism in the range where it operates.Source: §9 · p. 11
  • Eighteen of the 231 public JevBench items are unscored, including 12 ordinal items whose expected value is not a listed option.Source: §9 · p. 11
  • The 20-option runs report aggregates rather than per-item outcomes, so the strongest available evidence is a sign test over eleven models.Source: §9 · p. 11
  • The reported intervals cover resampled items, not variation across runs, seeds, hosts, or batch composition in bfloat16 logits.Source: §9 · p. 11
  • The serving run selected and certified its threshold on the same sample, so its bound does not cover the selected threshold.Its held-out rate was 0.013 on 300 decisions, with a 95% interval of [0.004, 0.034] that contains the 0.01 target.Source: §9 · p. 11
  • Efficiency and accuracy were measured on different tasks; JevBench uses K=2 to 6, leaving little rotation budget to save.Source: §9 · p. 11
  • The affine map and per-question head require hidden states unavailable through hosted APIs, while the Section 4 readout needs only next-token log-probabilities.Source: §9 · p. 11

10 Conclusion

AnyJev presents typed decisions as a readout of pretrained models, correcting label and position biases without changing model parameters. On both 20-option tasks, the readout lowered order-flip rates on every tested model, while the report describes its measurements as early and task-dependent.

10 Conclusion

  • On both 20-option tasks, AnyJev lowered the order-flip rate on every tested model, from 0.33 to 0.138 and 0.183.The readout divides out a label prior and averages away the position term, using one prefill per rotation and no parameter changes.Source: §10 · p. 11
  • This early report covers the readout, corrections, and serving cost, but does not evaluate the labelled head.Source: §10 · p. 11
  • The label prior’s benefit is task-dependent, and the report describes its measurements as early rather than a settled ranking of decision systems.Source: §10 · p. 12

A Per-model results on the 20-option tasks

Across four model-task cells, rotation averaging reduces dependence on option order without exceeding the best single rotation. The section also reports per-model gate results, benchmark analyses, stopping certificates, and tests of truncated-forward agreement with full-depth decisions.

A Per-model results on the 20-option tasks

B Benchmark detail

  • Benchmark detail reports paired bootstrap intervals, item-clustered pooled option-count tests, and difficulty-tier accuracies on a public subset distinct from the overall-score partition.Temperature is refitted within each bootstrap resample.Source: Table 7 · p. 17, Table 8 · p. 17, Table 9 · p. 18

C Rotations and the certificate

  • Across four cells, the rotation average is 0.3–2.3 points below the best single rotation and 3.5–9.3 points above the worst.Rotations remove the caller’s ability to select a poor ordering and reduce sensitivity to option order.Source: Figure 4 · p. 18, C Rotations and the certificate · p. 18

D Reading a truncated forward

  • The tuned lens maps block-b states into the output-head basis using full-depth states on unlabelled inputs; depth is selected on calibration data disjoint from evaluation sets.The logit lens reads the truncated state through the output head without that map.Source: D Reading a truncated forward · p. 18
  • Seven of eight models jump from at most 0.47 to 0.90 agreement in one measured step; at 0.98, only Mistral-7B (66%) and Qwen3-8B (89%) save blocks.Their map-only errors are 0.000 and 0.007, respectively; Llama-3.2-3B’s is 0.040, which is why it reaches no depth at the 0.98 target.Source: D Reading a truncated forward · pp. 18–19
  • Selecting a threshold on one unlabelled third and bounding it on the other two certifies two of four cells at a 1% target; massive_route/Qwen3-8B has no qualifying grid threshold.The verify bound does not carry selection multiplicity.Source: Table 10 · p. 19

E Full BEV Decision Mix results

On this full-split comparison, Nimble has higher overall accuracy than AnyJev’s best result, but AnyJev has lower ECE; the evaluated L2-mono variant trails rotation averaging.

E Full BEV Decision Mix results

  • The full comparison covers 46 320 decisions across choices, yes/no decisions, and ordinal scores; ECE is averaged within shards, while Wilson intervals ignore repeated states and questions.The majority reference is an oracle diagnostic, not a deployable baseline.Source: E Full BEV Decision Mix results · p. 21
  • Nimble reached 70.57% overall accuracy versus AnyJev’s best 68.01%, while AnyJev had lower ECE (0.0490 versus 0.0889).Table 14 presents the full-split results.Source: Table 14 · p. 21, E Full BEV Decision Mix results · p. 22
  • Every measured system had its lowest accuracy on ordinal scores, and each tested prior correction lowered overall accuracy versus rotation averaging alone, even where ECE improved.Source: E Full BEV Decision Mix results · p. 22
  • L2-mono reached 67.24% test accuracy, 0.61 points below the raw single forward and 0.77 points below rotation averaging.Its 9.4% reduction in block count is not a measured latency improvement, and its 95.6% calibration agreement is not a test-set guarantee.Source: E Full BEV Decision Mix results · p. 22
  • These measurements cover two AnyJev bases on one dataset configuration, with a 2 048-token AnyJev prompt cap versus Nimble’s 8 192-token context and no cross-system latency or memory comparison.Source: E Full BEV Decision Mix results · p. 22

Related papers

Loading 2610.00831v1…