PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?

Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou

arXiv:2610.35826v2cs.CL

TL;DR

Selection bias can make LLM choices change when only presentation order changes, and systematic multi-option comparisons of Jev versions, Jev reproductions, and generative LLMs have been limited. This study shuffles options while holding question and candidate content fixed, tracking answers across eight presentations and comparing them with fixed-order repetition. Jev’s accuracy remains nearly unchanged while its failure set expands, and the comparisons show no general Jev advantage in preserving correct choices.

  • Problem

    Systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs in multi-option tasks remain limited, despite evidence that presentation changes can alter choices without adding evidence.

    Source: §1 · p. 2

  • Method

    The study compares two Jev versions, twelve Jev reproductions, and six generative LLMs by shuffling options across eight presentations while holding question and candidate content fixed, alongside fixed-order repetition.

    Source: §1 · p. 3

  • Results

    Jev’s accuracy varies by less than two percentage points while its cumulative failure set expands by 27.3–28.5%, and the study finds no general Jev advantage in preserving correct choices.

    Source: §1 · p. 3, §10 · p. 26

  • Takeaways & Limitations

    Stable accuracy and repeatable choices do not establish that initially correct choices survive equivalent presentations.

    Source: §1 · p. 3, §10 · p. 26

    The findings cover fixed task collections and eight presentations; subgroup factors vary together, and observed first failures do not predict growth under further requests.

    Source: §8 · p. 24

Abstract

from arXiv · show

Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27.3--28.5\% relative to the original order, with first failures continuing into the final presentation. Reordering loses substantially more initial successes than fixed-order repetition, a gap that also appears in highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.

1 Introduction

Presentation changes can alter choices without adding evidence, and systematic comparisons of Jev versions, Jev reproductions, and generative LLMs on multi-option tasks remain limited. The study tests whether Jev-style decision structure preserves correct choices across reordered presentations and finds no general Jev advantage.

1 Introduction

  • Selection bias arises when presentation influences a decision without adding evidence about which candidate is correct.A question answered correctly in one order may fail in another.Source: §1 · p. 2
  • Prompting does not reliably remove selection bias in tested settings, while calibration and targeted training offer possible ways to reduce it.This motivates asking whether models designed for candidate selection better preserve correct choices across presentations.Source: §1 · p. 2
  • Jev-style System One models predict probabilities over supplied candidates, but reproductions share a behavioral goal rather than a common architecture or recovered Jev internals.Implementations range from adapted decision heads to candidate probabilities extracted from frozen LLMs.Source: §1 · p. 2
  • Existing evidence challenges the assumption that structured decisions resist bias, while systematic multi-option comparisons across Jev models and generative LLMs remain limited.A single swap cannot show how failures spread over further orders, and aggregate accuracy can hide losses offset by recoveries.Source: §1 · pp. 2–3
  • The test shuffles options while preserving question and candidate content, comparing two Jev versions, twelve reproductions, and six generative LLMs.It tracks answers through eight presentations and compares reordering with fixed-order repetition.Source: §1 · p. 3
  • 27.3–28.5%: Jev’s cumulative failure set expands across eight presentations despite accuracy varying by less than two percentage points.First failures continue into the final presentation.Source: §1 · p. 3
  • 91.9–100% fixed-order agreement in Jev reproductions coexists with losses of 23.2–64.2% of initial successes under reordering.Reordering loses substantially more initial successes than fixed-order repetition for both Jev versions.Source: §1 · p. 3
  • The study also examines task, difficulty, menu size, voting, unanimity, and findings on MMLU-Pro and GPQA Diamond.A supporting survey organizes differences in representation, learning, and computation that controlled comparisons could isolate.Source: §1 · p. 3

2 Evaluation Setup

The evaluation uses fixed question collections and compares model selections across reordered presentations and fixed-order repetition. It defines measures that separate average accuracy, persistent correctness, failure discovery, wrong-option diversity, and agreement-based aggregation.

2.1 Datasets

  • The study evaluates 1,700 primary questions—1,200 from SuperGPQA and 500 from LEXam—and separately examines 300 MMLU-Pro and all 198 GPQA Diamond questions.Pooled scores weight questions equally and reflect the sample composition, not an equal-weight average of benchmark scores.Source: §2.1 · pp. 3–4

2.2 Models

  • The comparisons include Jev 1.13 and 1.14, six generative LLM baselines, and twelve Jev reproductions; LLM training, inference, and interfaces are not matched to Jev.Reproductions select the candidate with the highest native probability without generating an answer or using deliberation.Source: §2.2 · p. 4

2.3 Eight Presentations of Each Question

  • Each question is presented in original order, reversed order, and six random permutations, with candidate identifiers moving alongside their content and the full menu retained.Requests are independent; later presentations do not include earlier model answers.Source: §2.3 · p. 4

2.4 Repeated Requests in the Original Order

  • Fixed-order repetition holds candidate content, identifiers, and order constant across eight independent requests; the study compares its outcomes with eight reordered presentations.Same-candidate agreement requires the same candidate in every request so far; loss of initial successes is the fraction of first-request successes that become incorrect at least once.Source: §2.4 · pp. 4–5

2.5 Separating Scores, Correctness, and Error Identity

  • Retention is the fraction of initially correct questions that remain correct in every presentation through t; its complement is loss of initial successes.A question that becomes wrong and later recovers is not retained.Source: §2.5 · p. 5
  • The measures distinguish newly failed questions from distinct wrong-option pairs, because accuracy does not track changes in the identity of an incorrect answer.Repeated selection of the same wrong option adds no pair, while another wrong option for that question adds one.Source: §2.5 · p. 5

2.6 Stratification and Uncertainty

  • Difficulty analyses use SuperGPQA labels and stratify by difficulty and discipline; option-count and other subgroup comparisons are descriptive because groups also differ in benchmark composition and content.Exploratory pointwise 95% intervals summarize sampling variation without simultaneous coverage or a causal interpretation of group differences.Source: §2.6 · p. 6

2.7 Voting and Agreement Across Presentations

  • Plurality voting selects the most frequent candidate, with tied outcomes scored by expected correctness under a uniform draw; unanimous coverage and accuracy within that subset are reported together.Together these measures show the trade-off between conditional accuracy and how many questions receive a unanimous answer.Source: §2.7 · p. 6

3 Does Jev Resist Changes in Option Order?

Stable accuracy obscures changes in individual choices: Jev loses initially correct answers under reordering, even as some answers recover. Fixed-order agreement likewise does not ensure correct or order-robust choices, and errors often extend to multiple wrong candidates per question.

3.1 What Stable Accuracy Conceals

  • 409 transitions: Jev 1.14’s reversal includes 113 correct-to-incorrect and 120 incorrect-to-correct changes, plus 176 wrong-to-wrong switches; AutoTrust-JEV-9B has 714 transitions.Those transitions yield net gains of seven and three correct answers, respectively.Source: Figure 3 · p. 7, §3.1 · p. 7, Figure 4 · p. 7

3.2 First Failures Continue Beyond the Initial Presentations

  • 27.3–28.5%: Jev’s cumulative failed-question set grows by this amount, with first failures still appearing in the eighth presentation.The two Jev versions add 231 and 239 failed questions, respectively.Source: §3.2 · p. 8

3.3 Persistence on Shared Initial Successes

  • 72.2–73.0%: Jev retains this share of initial successes, versus 75.7–88.9% for four LLMs; lower Qwen3.8-Flash-Next and GLM-5.3-Flash retention rules out a blanket family ranking.On 487 questions all models initially answered correctly, Jev retains 85.0% and 84.4%; GPT-6.1-sol retains 98.4%.Source: §3.3 · p. 8

3.4 Repeatability Under a Fixed Order

  • Across eight original-order requests, Jev gives the same wrong answer on 37.5–39.3% of all questions, and its accuracy among unanimous answers is only 53.7–55.1%.Jev reproductions agree on 91.9–100% of all questions across eight requests, yet 48.5–72.2% receive the same wrong answer throughout.Source: §3.4 · p. 9
  • 25.8 and 25.9 percentage points: Jev’s changed-choice rates are higher under reordering than fixed-order repetition, while reordered presentations also lose 27.0–27.8% of initial successes versus 8.8–9.2% under repetition.Jev reproductions lose 23.2–64.2% of initial successes under reordering, versus 0–1.3% under repetition.Source: §3.4 · pp. 9–10, Figure 7 · p. 10, Figure 8 · p. 10

3.5 Errors Extend Beyond a Single Distractor

  • 70.8–89.5%: Extra wrong candidates account for this share of wrong-pair growth across Jev reproductions, showing that observed errors can extend beyond one distractor per question.For Jev, distinct wrong-pair counts rise 91.8–99.5%, compared with a 27.3–28.5% rise in failed questions.Source: §3.5 · p. 11, Figure 10 · p. 12

4 How Does Jev’s Order Robustness Vary Across Tasks?

Jev’s order sensitivity varies across tasks, difficulty levels, and menu sizes, while stable accuracy or fewer changed choices does not necessarily mean correct answers are retained. The subgroup profiles locate these vulnerabilities but do not isolate their causes.

4.1 Comparisons Across Benchmarks

  • On LEXam, Jev changes 39.8–42.0% of choices, but its stability ranking varies by benchmark and comparator.AutoTrust-JEV-27B-v1.1 changes fewer choices than either Jev version on both primary benchmarks, yet loses more initial successes than Jev 1.14 on LEXam.Source: §4.1 · p. 12, Figure 11 · p. 13, Figure 12 · p. 13

4.2 Difficulty and the Persistence of Correct Answers

  • On hard SuperGPQA questions, Jev loses 53.9–54.6% of initially correct answers; its changed-choice rate rises from 27.5–31.0% on easy questions to 60.8–62.5% on hard questions.Across reproductions, hard-question loss ranges from 36.8% to 88.5%; even within ten-option questions, Jev’s changed-choice rate rises from 27.7–29.9% on easy questions to 65.1–67.0% on hard questions.Source: §4.2 · p. 13, Figure 13 · p. 14, Figure 14 · p. 14
  • Within ten-option SuperGPQA questions, Jev loses 15.7–20.1% of initial successes on easy questions and 50.0–53.4% on hard questions.The loss is 28.0–29.5% across all 731 ten-option questions, while accuracy varies by only 2.05 percentage points; AutoTrust-JEV-27B-v1.1 also loses 18.2% on easy and 44.9% on hard questions.Source: Figure 15 · p. 14, §4.2 · pp. 14–15, Figure 16 · p. 15
  • Engineering and science contain many hard questions and much of Jev’s instability, but subject profiles do not identify an independent subject effect.AutoTrust-JEV-27B-v1.1’s always-wrong switches account for 15.8% of engineering questions and 18.4% of science questions, a form of variation voting cannot repair.Source: §4.2 · p. 15, Figure 17 · p. 16, Figure 18 · p. 17

4.3 Locating Instability Within Legal Tasks

  • On LEXam, Jev’s changed-choice rate rises from 20.8–21.6% in four-option groups to 68.0–73.6% in 32-option groups, but the groups contain different questions.All eight compared models have lower accuracy and more changed choices in the 32-option group; Jev reproductions also change more choices at 16 than eight options, and more at eight than four.Source: §4.3 · p. 15, Figure 19 · p. 17, Figure 20 · p. 18

4.4 Option Count and Continued Failure Discovery

  • In the four-option group, 199 of 203 questions have eight distinct observed orders, and Jev’s failure trajectories are unchanged when counting only first occurrences of full permutations.Presentations five through eight still expose five and two first failures for Jev 1.13 and 1.14, respectively; these are sampled menu orders, not necessarily distinct correct-answer positions.Source: §4.4 · p. 19

5 Can Aggregating Reordered Choices Correct Jev’s Errors?

Voting recovers some correct choices from reordered answers, but Jev’s gains are modest and unanimity leaves substantial error. Agreement and aggregation therefore provide limited safeguards in the evaluated settings.

5.1 Voting Recovers Only Part of the Error

  • Jev’s eight-answer plurality gains are 0.96 percentage points for version 1.13 and 1.01 for version 1.14, while LLM gains range from 0.46 to 5.63 points.The reference is mean single-presentation accuracy; Jev reproductions’ voting gains range from 0.74 points to 2.49 points.Source: Figure 25 · p. 20, §5.1 · pp. 20–21, Figure 26 · p. 20

5.2 Agreement and Persistent Error

  • Jev’s unanimous answers cover 57.7–59.1% of questions and have 62.1–63.3% accuracy, leaving 36.7–37.9% of unanimous answers wrong.Persistent unanimous mistakes account for 21.2–22.4% of all questions; Jev reproduction unanimous subsets reach only 30.2–66.3% accuracy at 24.1–63.7% coverage.Source: Figure 25 · p. 20, Figure 26 · p. 20, §5.2 · p. 21

5.1 Voting Recovers Only Part of the Error

  • Jev 1.14 gains only 0.30 points on LEXam, where many changed choices remain wrong throughout and supply no correct vote.Across the eight-call comparison, reordering adds only 0.79 and 1.00 points of voting gain for Jev 1.13 and 1.14, respectively, relative to each condition’s mean single-request accuracy; direct plurality-accuracy differences have intervals spanning zero.Source: §5.1 · pp. 20–21

5.2 Agreement and Persistent Error

  • At differing coverage, unanimous LLM answers are 17.8–21.5% wrong, compared with 36.7–37.9% for Jev; this comparison does not establish a matched-coverage ranking.Jev reproductions also receive the same wrong answer across every presentation on 12.1–27.0% of questions.Source: §5.2 · p. 21

6 Does Higher Accuracy Protect Correct Choices?

On MMLU-Pro, Jev’s higher accuracy does not eliminate losses of initially correct answers under reordering. The gap between fixed-order repeatability and preservation of correct choices also appears across Jev reproductions.

6 Does Higher Accuracy Protect Correct Choices?

  • On MMLU-Pro, Jev’s cumulative failures grow by 23 and 27 questions, losing 8.7% and 10.2% of initial successes under reordering, versus 1.1% and 2.7% under original-order repetition.Mean reordered accuracy is 86.4% for Jev 1.13 and 86.0% for Jev 1.14.Source: §6 · pp. 21–22, Figure 27 · p. 22
  • Jev’s unanimous answers cover 88.3% and 86.0% of MMLU-Pro questions, yet 8.7% and 8.1% of retained answers are wrong.The reliability of agreement depends on the task collection.Source: Figure 27 · p. 22, §6 · p. 22
  • All twelve Jev reproductions repeat the same candidate across eight original-order requests yet lose 8.6–45.0% of initial successes under reordering.The comparison describes the evaluated systems and does not isolate an architectural cause.Source: Figure 28 · p. 22, §6 · p. 22

7 Does a Four-Option Menu Remove the Vulnerability?

On four-option GPQA Diamond questions, reordering still causes Jev and its reproductions to lose initially correct answers. The gap between repeatability and preservation of successful decisions does not require long menus.

7 Does a Four-Option Menu Remove the Vulnerability?

  • On four-option GPQA Diamond, Jev loses 16.4% and 22.1% of initial successes under reordering, versus 4.1% and 7.1% under original-order repetition.The twelve reproductions repeat original choices perfectly but lose 23.7–66.7% of initial successes after reordering.Source: Figure 29 · p. 23, §7 · p. 23, Figure 30 · p. 23
  • Across the LLMs, initial-success loss ranges from 3.3% for GPT-6.1-sol to 44.9% for GLM-5.3-Flash, so these results do not establish a uniform family ranking.Jev accumulates 95 and 112 distinct wrong question–candidate pairs across its 76 and 89 failed questions; comparisons across benchmarks do not isolate candidate count from content or difficulty.Source: §7 · pp. 23–24

8 Implications and Scope

The comparisons show that a shared decision interface does not ensure comparable robustness, and that repeatability does not predict which models preserve correct choices. The findings are bounded by unresolved mechanisms and the tested task collections and presentations.

What the Jev comparison establishes

  • Four LLMs retain more shared initial successes than Jev, but Qwen3.8-Flash-Next, GLM-5.3-Flash, and task-dependent comparisons rule out a universal ranking.AutoTrust-JEV-27B-v1.1 retains more initial successes than the Jev references, while the other eleven reproductions retain fewer; high fixed-order agreement does not predict this ordering.Source: §8 · p. 24

Reordering diagnoses failures that agreement may not repair

  • Voting yields only a small accuracy gain for Jev, while unanimous errors pass an agreement filter unchanged.Always-wrong trajectories contain no correct vote; these persistent errors limit agreement as a safeguard but do not themselves demonstrate order bias.Source: §8 · p. 24

Behavioral differences do not identify their mechanism

  • The experiment does not isolate a causal order effect or distinguish identifier preferences from position and context effects.Candidate identifiers move with their content while entry order changes, and the study does not establish how a proprietary interface represents them.Source: §8 · p. 24
  • Training, inference, and interface differences across models preclude causal claims about decision heads or generation.The findings concern observed selections, not a recovered account of Jev’s proprietary mechanism.Source: §8 · p. 24

The task mixture limits generalization

  • Results cover fixed task collections and eight presentations; domain, language, difficulty, and candidate count vary together.Subgroup intervals are exploratory and do not adjust for multiple comparisons; continued first failures establish persistence only within the observed sequence.Source: §8 · p. 24

9 Related Work

Prior work documents selection biases, mitigation strategies, and sensitivity in typed decision models, while distinguishing correctness from consistency. This paper situates its contribution in tracing failures across repeated menu permutations and testing whether aggregation repairs them.

Selection bias and its sources

  • Reordering tests selection bias with question and answer content held fixed, but identifier preferences and position effects remain distinct sources.Zheng et al. (2024) identify option-ID preferences as a major source; prior studies also document response-order effects in pairwise judgments.Source: §9 · p. 25

Mitigation through calibration, training, and reasoning

  • Calibration, targeted training, and reasoning-model comparisons offer distinct mitigation approaches, while this study evaluates deployed systems without additional debiasing.Prior approaches estimate answer priors, strengthen symbol binding, reward permutation consistency, or combine permutation metrics with fine-tuning and aggregation.Source: §9 · p. 25

Jev-style modeling and selection bias

  • Prior studies find that typed decision models can be repeatable yet sensitive to response swaps, and that serialization changes can alter public Jev-style behavior.Lukassen et al. (2026) study pairwise judgments; Azizi et al. (2026) trace option-label effects to serialization in public implementations.Source: §9 · p. 25

Consistency, correctness, and aggregation

  • Consistency does not guarantee correctness: aggregation can stabilize an incorrect answer, and this study follows returned selections rather than probability changes.GRP’s joint correctness criterion combines order agreement with per-trial correctness, while the probability metric of Guda et al. (2025) tracks candidate-probability changes.Source: §9 · p. 25

10 Conclusion

The paper concludes that stable accuracy and repeatability can conceal lost correct choices under reordering, while leaving mechanisms unresolved. It surveys how decision models are constructed, used, and evaluated, emphasizing that a shared candidate-distribution interface does not specify their internal design.

10 Conclusion

  • Across eight presentations, Jev failure sets expand by 27.3–28.5% while accuracy remains nearly unchanged, and reordering loses more initial successes than fixed-order repetition.The LLM comparison shows no general Jev advantage; vulnerability also appears at higher accuracy and with short menus, while voting offers limited correction and unanimous errors can remain undetected.Source: §10 · p. 26

A Survey Overview

  • A candidate distribution can be extracted from a frozen language model or learned from labels, contrastive pairs, reasoning teachers, and policy rewards.Direct responses may also arise from intermediate exits, recurrent latent processing, or conditional reasoning, so output format alone does not specify how decisions are learned or computed.Source: A Survey Overview · p. 31

B.1 The Decision Contract

  • The decision contract lets callers supply finite candidates, but constraining outputs to that space does not ensure semantic correctness, calibration, or consistency.Public documentation does not disclose a reproducible Jev architecture, training distribution, or exact objective; public alternatives do not establish recovery of its internal algorithm.Source: §B.1 · p. 31

B.2 The Object of Contribution

  • Works may contribute at multiple levels, including model recipes, downstream selectors, benchmarks, or decision policies.The paper builds on survey distinctions among model roles and focuses on how mechanisms connect to workflow operations.Source: §B.2 · p. 32, Table 1 · p. 32

C Constructing Decision Distributions

  • Readout, candidate information sharing, supervision, and gradient estimation shape how order can enter predictions and which interventions can change it.These construction choices are examined before inference.Source: C Constructing Decision Distributions · p. 32

C.1 Decision Readout Before Specialized Training

Decision distributions can be read from an existing model or built through candidate representations with different interaction and reuse properties. These choices impose distinct invariances and trade-offs rather than guaranteeing robustness.

C.1 Decision Readout Before Specialized Training

  • A frozen readout restricts and normalizes existing language-model predictions over candidate choices without updating model weights.Fitting its temperature still requires calibration data, making it a control for separating specialized training from a different probability-string inference procedure.Source: §C.1 · p. 32

C.2 Candidate Representations, Interaction, and Reuse

  • CLM scores candidates independently, so adding an alternative changes the softmax denominator but not existing pairwise odds when inputs and temperature stay fixed.MetaEncoder also uses reusable candidate features, but its request representation can depend on the set; a bi-encoder label alone therefore does not establish invariant pairwise odds.Source: §C.2 · p. 34
  • Jeva reuses request-independent candidate representations yet lets its final request state depend on the full option set, with symmetric aggregation designed for permutation-equivariant probabilities.Its open-set output separates competition among options from estimated coverage; empirical quality still depends on learning and evaluation.Source: §C.2 · p. 34
  • Candidate-independent block-causal attention excludes interactions among candidate blocks, but its candidate states depend on the request and cannot generally be cached across requests.The design aims to remove serialization-order effects, yet reported residual permutation flips distinguish that structural objective from exact numerical behavior.Source: §C.2 · p. 34
  • Prism-1 isolates candidate blocks and reuses their position, making scores independent of serialization order and preserving pairwise odds under fixed-prefix deterministic computation.That invariance restricts tasks where an option’s score should change with its rivals.Source: §C.2 · p. 34

C.3 Supervision and the Target Distribution

Decision models can share a direct-candidate interface while differing in what their supervision teaches the output distribution to represent. Targets may come from labels, teacher predictions, sampled answers, or workflow-specific feedback, so similar training losses need not fit the same quantity.

C.3 Supervision and the Target Distribution

  • Supervision source determines what a fitted decision distribution represents: labels encode outcomes, contrastive pairs distinguish matched inputs from negatives, and teacher probabilities reflect the teacher’s information and computation.The section distinguishes several target constructions rather than treating a shared loss form as evidence that they estimate the same quantity.Source: §C.3 · pp. 34, 35, 37
  • Decision-model recipes range from residual heads trained on mixed labels and teacher distributions to candidate-prefix adaptation and pointer or native-vocabulary readouts.The section also describes modality-specific heads and contrastive state–action alignment; readout design and training target vary across models.Source: §C.3 · pp. 35–36
  • Other teacher signals include reasoning-endpoint distributions, full-depth predictions, and black-box service outputs, each preserving different information or computation.Workflow-specific recipes additionally train value estimates or agent-state decisions, rather than only a general candidate distribution.Source: §C.3 · pp. 36–37
  • Sampled-answer targets are vote histograms, and LDM’s SPRD requires a top vote share of at least 0.75 across four samples; SPRD-C also requires agreement with the constructed label.The consistency filter does not establish independent correctness, and LDM applies NLL and teacher-to-student KL to different record types.Source: §C.3 · pp. 36–37, Figure 33 · p. 38
  • Under a shared rationale distribution and endpoint sampler, averaging conditional probabilities and counting sampled answers can estimate the same marginal distribution.Averaging removes final-answer sampling noise under those assumptions, but actual SII and LDM datasets are not thereby shown equivalent.Source: §C.3 · p. 37

C.4 Statistical Objectives and Gradient Credit

The paper separates the probability objective from the method used to send its learning signal to parameters. It also distinguishes direct decisions from adaptive execution paths, which can change computation and sometimes the returned prediction.

C.4 Statistical Objectives and Gradient Credit

  • A proper probability score rewards truthful reporting in expectation, whereas sampled-answer correctness generally favors a mode of the outcome distribution rather than reporting that distribution.Thus, calling a method “RL” alone identifies neither its statistical objective nor an advantage over direct proper-score fitting.Source: §C.4 · p. 38
  • Training signals may differentiate a reported distribution, perturb output logits, or add score-function credit through a rationale policy; these routes assign credit differently.OpenJev-RLCD retains generated rationales at inference, a property of that method rather than every rationale-based procedure.Source: §C.4 · pp. 38–40
  • Qwen-Audio-Jev’s sampled-action estimator yields the direct proper-score gradient in expectation under its construction, but establishes no empirical advantage in variance, efficiency, or calibration.The estimator uses importance weighting, an exploration mixture, a leave-one-out baseline, and detached weights.Source: Figure 34 · p. 39, §C.4 · p. 40

D Executing Decision Models

  • Direct outputs do not imply a fixed computation budget: students can omit teacher reasoning, intermediate heads can stop before later layers, and recurrent models can use hidden steps.Generated-token count alone cannot compare these computational demands.Source: §D.1 · p. 40

D.1 Direct Outputs and Adaptive Computation

  • LDM routes low-margin cases to an untuned base-model reasoning rollout, while StartLux conditionally combines initial and reasoning-conditioned predictions using the same weights.Their model called, trigger, and returned prediction differ; SanSi also varies hidden recurrent loops, which need not improve every task.Source: §D.1 · pp. 40–41, Figure 35 · p. 41
  • Execution comparisons must account for backbone depth, reader calls, generation, caching, and scheduling; returned-answer length alone is not a cost measure.Spatial prediction can require sequential inference over predicted crops even when teacher-forced stages are packed into one training pass.Source: §D.1 · p. 42

D.2 Serving and the Returned Prediction

A deployed decision depends on more than its model head: serving transformations and workflow placement affect the returned prediction, computational cost, and what downstream outcomes can be attributed to the decision component. The section surveys these distinctions across execution, evidence acquisition, memory, and agent learning.

D.2 Serving and the Returned Prediction

  • Serving can change prediction semantics as well as execution cost: postprocessing can move probability mass, scalar reduction can preserve different information, and batching can alter shared computation.Jeff’s binary commitment transformation moves intermediate probabilities beyond the benchmark abstention interval, so calibration evaluation must specify which distribution it measures.Source: §D.2 · pp. 42–43

E Decision Models in Downstream Workflows

  • Decision models intervene at different workflow stages, including controlling an ongoing trajectory, selecting completed outcomes, acquiring evidence, or assisting another learner.These roles carry different error and recovery costs, even when they use the same decision model.Source: E Decision Models in Downstream… · p. 44, §E.1 · p. 44

E.1 Controlling a Trajectory or Selecting Its Outcome

  • Completed-run selection scores trajectories after their generation cost has already been incurred, unlike online action selection that changes a run’s next state.A cheap verifier may reduce selection overhead without making the total procedure cheaper than generating one run.Source: §E.1 · pp. 44–45

E.2 Changing the Evidence Available to a Decision

  • Clarification acquires external answers, whereas memory control changes how past evidence is organized and retrieved for later decisions.Both can affect subsequent decisions without changing the decision model’s parameters.Source: §E.2 · p. 45

E.2.1 Acquiring Missing Information

  • LAVOIR ranks missing information by predicted value, asks the highest-valued eligible slot, and then re-encodes the state with the answer or decides whether to hand off.Its learned value is a proxy dependent on the frozen predictor and answer distribution; evidence is strongest on synthetic multi-turn dialogues, with limited real single-exchange tests and one training seed.Source: §E.2.1 · pp. 45–47, Figure 36 · p. 46

E.2.2 Controlling Memory Construction and Retrieval

  • Jev-Mem separates memory writing, which retains observations while selectively judging inferred relations, from retrieval, which expands evidence until sufficient, low-benefit, or resource-limited.The reported evaluation uses LoCoMo with GPT-4o-mini and does not evaluate the implementation’s Laya controller or LongMemEval configuration.Source: §E.2.2 · p. 47, Figure 37 · p. 48

E.3 Decision Models in Agent Learning

  • A fixed Jev service can alter another agent’s data, exploration, rewards, or updates, so reported learning outcomes concern those interventions rather than decision-probability quality alone.The cited MiniGrid and Atari evidence uses three seeds, and replay assistance does not uniformly help.Source: §E.3 · pp. 48–49

F From Local Probabilities to Workflow Reliability

  • The experiment measures selections, leaving calibration and downstream utility unresolved; accurate probabilities, good local actions, and complete task success are distinct evaluation levels.Attributing downstream gains requires separating decision components from the generator, task decomposition, and execution policy.Source: §E.1 · p. 45, F From Local Probabilities to Workflow… · p. 49

F.1 Local Probability Quality and Decision Utility

The survey separates local probability quality from decision utility and examines how sensitivity to request composition, model choices, and surrounding policies affects reliable decisions. It concludes that structural advantages require controlled comparisons and evidence that improvements survive equivalent presentations and benefit complete workflows.

F.1 Local Probability Quality and Decision Utility

  • Calibration, distribution fidelity, and decision utility measure distinct properties, and neither calibration nor fidelity alone specifies the action taken or its cost.On synthetic problems with known conditional targets, exact uncertainty helps assess probability estimation but does not establish true uncertainty for arbitrary real tasks.Source: §F.1 · pp. 49–50
  • SanSi’s 1.4B model’s expected calibration error rises from 0.082 ± 0.010 at loop three to 0.093 ± 0.012 at loop eight.The comparison uses different readout depths within each trained model and shows additional latent computation can worsen calibration within the trained loop range.Source: §F.1 · p. 50

F.2 Composition and Input Sensitivity

  • Sensitivity across requests can reflect prompt construction or numerical precision as well as architectural dependencies, so the intended invariance must be specified first.The reviewed permutations preserve question content and candidate identities, while adding candidates or revealing evidence can change the decision problem.Source: §F.2 · pp. 50–51
  • Serial correctness retention is not agreement with a native answer or fidelity to a plan, and candidate restrictions can change the search space and call count.Related robustness studies also connect valid but incorrect choices to modeled program consequences, not measured real-world losses; release postprocessing can preserve wrong actions as well as correct ones.Source: §F.2 · p. 51

F.3 Complete-Workflow Outcomes

  • Task-level gains from Jev interventions do not uniformly translate into better outcomes across the evaluated workflows.The ALFWorld comparisons separate interface effects from decision-component effects; AndroidWorld lacks the same three-way control, and one integration also differs in action granularity.Source: §F.3 · p. 51

G Open Questions

  • Open questions include which improvements preserve correct choices across equivalent presentations and related decisions, with the intended invariance specified before testing.The paper notes that independence of irrelevant alternatives is not universally appropriate because an added option may provide context or change the task.Source: G Open Questions · pp. 52–53
  • Future comparisons should hold objectives and perturbations fixed when testing gradient estimators, and distinguish training support from inference computation when studying reasoning transfer.The proposed workflow tests vary decision model and policy while controlling candidate construction, budgets, fallback rules, and answer generation.Source: G Open Questions · pp. 52–53

H Survey Synthesis

  • A shared candidate-distribution interface accommodates varied designs but does not ensure correct choices survive reordering; explaining differences requires controlled comparisons under common information and resource budgets.The survey also calls for testing whether local improvements benefit the workflow the model controls.Source: H Survey Synthesis · p. 53

I Extended Public Implementation Inventory

The inventory catalogs public Jev-style implementations and documented mechanisms, spanning different candidate representations, training approaches, and inference designs. The authors caution that public artifacts alone do not establish independent reproduction.

I Extended Public Implementation Inventory

  • Table 3 summarizes additional implementations and documented mechanisms, but public artifacts do not by themselves establish independent reproduction.The inventory distinguishes public-data scope, model variants, and whether recipes, weights, or inference sources are available.Source: Table 3 · p. 53, I Extended Public Implementation… · pp. 53–54
  • The inventory includes varied candidate representations and decision heads, including pointer heads, label-token readouts, and option-conditioned or permutation-equivariant scoring.Examples include LoRA or full adaptation, up to 16 choices per pass with tournaments for larger menus, and randomized assignment of reserved atomic label tokens.Source: I Extended Public Implementation… · pp. 54–55
  • Documented training recipes range from teacher-generated hard labels and public-label replay to self-distillation, external-teacher soft targets, and mixed supervision.Some entries describe model-card or serving evidence rather than a fully specified training objective.Source: I Extended Public Implementation… · p. 54
  • Inference designs include frozen-model order averaging, letter readouts with temperature calibration, and version-specific training or policy variants.Some releases omit an RL stage or do not disclose the training objective; a correct-answer-probability reward with a length penalty is not a proper score.Source: I Extended Public Implementation… · pp. 54–55
  • Additional designs include task-oriented extensions and readout heads, while some documented interfaces leave post-training objectives unspecified.The inventory also describes separate probability and confidence heads, option-shuffled training pairs, and learned or frozen encoders for candidate scoring.Source: I Extended Public Implementation… · p. 55

Related papers

Loading 2610.35826v2…