PaperlayerView exact revision on arXiv

Source-linked AI summary

From Probabilities to Decisions: Search and Multi-Teacher Distillation with Jev

Mohamad Yazan Sadoun, Sarah Sharif, Yaser Mike Banad

arXiv:2610.09188v1cs.AI

TL;DR

Probability-only judgments can score specified choices quickly, but decision systems under time pressure must compare alternatives and anticipate consequences. This paper connects Jev to search and stored chess knowledge, distills its judgments into a reusable evaluator, and tests an LLM partnership. The combined labels improve chess results over an equal-budget Qwen-only evaluator, while Jev alone performs usefully in passage reranking and Qwen adds little.

  • Problem

    Decision systems under time pressure must compare alternatives and anticipate consequences, while Jev supplies probabilities only for specified choices and repeated calls are too costly for search.

    Source: §1 · p. 1

  • Method

    The paper combines Jev with search and stored chess knowledge, then distills pairwise judgments from Jev or Jev and an LLM into a compact evaluator.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Across tasks, Jev–Qwen labels beat Qwen trained on twice the pairs by +9.6 Elo in chess, while passage reranking gains at most +0.004 from adding Qwen.

    Source: §1 · p. 1, §5.2 · p. 3

  • Takeaways & Limitations

    Search supplies lookahead and distillation makes judgment reusable, while an LLM partner improves chess results but adds at most +0.004 in passage reranking.

    Source: §1 · p. 1, §10 · p. 6

Abstract

from arXiv · show

Probability-only models, which TypeSafe calls System One models, return calibrated probabilities for fixed choices in milliseconds and generate no text. We study one such model, Jev, through two tasks that require decisions under tight constraints. In bullet chess, a bot that places Jev's judgment inside Stockfish search alongside an opening book and endgame tablebases climbs above a 2200 Lichess bullet rating against other bots. Live model calls are too slow for search, so we distill pairwise judgments into a compact evaluator that runs at every position. We then ask how best to spend a fixed labeling budget when an LLM, Qwen3-32B, is available as a second teacher. In chess, averaging both judges' labels beats spending the whole budget on Qwen alone by 9.6 Elo (95% interval 4.3 to 14.9), and the gain replicates on fresh openings; a second answer from the same judge is no substitute, and Jev is the strongest partner for Qwen among the models tested. In passage reranking, Jev's labels alone train a reranker that scores as high as Qwen's, from 21 minutes of API calls instead of 5.1 GPU-hours, and adding Qwen gains at most a few thousandths in ranking quality. Search supplies the lookahead, distillation makes the judgment cheap enough to use at every position, and an LLM partner pays off in chess.

1 Introduction

The introduction asks how probability-only judgments can support decisions under time pressure. It connects Jev to search, stored chess knowledge, and a learned evaluator, then tests the approach in chess and passage reranking.

Introduction

  • Jev supplies probabilities for specified choices, while search, stored knowledge, and a learned evaluator supply complementary decision-making functions.The paper studies this composition in bullet chess, where long-term consequences must be considered within a short clock.Source: §1 · p. 1
  • Because repeated model calls are too costly for search, the authors distill offline preferences into an evaluator and average Jev and LLM probabilities before fitting it.They test whether the second judge improves decisions for the same labeling budget.Source: §1 · p. 1
  • Chess experiments find a benefit from combining Jev with Qwen, while passage reranking finds Jev alone useful and at most +0.004 from adding Qwen.Together, the tasks show that an LLM partner's value depends on the task.Source: §1 · p. 1

2 Putting Jev’s Judgment to Work

A complete bullet-chess bot combines Jev-derived evaluation with Stockfish search and stored chess knowledge, and its recorded rating rises from 1753 to 2268. Because live Jev calls are too slow for search, the paper distills judgments into a reusable evaluator.

2.1 Chess as a proof of concept

  • The initial bot also uses a masters opening book for established continuations and Syzygy tablebases for exact information in covered endgames.The evaluator guides search outside those resources, while Jev also chooses among near-equal moves.Source: §2.1 · p. 1
  • 1753 to 2268: the complete bot’s recorded Lichess bullet rating rises with Stockfish search, a Jev-trained evaluator, opening book, tablebases, and live Jev choices.The trajectory covers a segment against other bots at one minute per side with no increment.Source: §2.1 · p. 1, Figure 1 · p. 2

2.2 Distilling judgment for fast search

  • A live Jev call takes 177 ms at the median, and querying it only to break exact search ties reduces strength by 28.7 Elo over 400 games.The reported interval is [−51.1, −6.6], with 3,875 calls.Source: §2.2 · p. 2
  • Offline distillation fits a small evaluator to move-comparison probabilities and stores its weights in the engine, letting search reuse its learned judgment at every evaluated position.The same training step can absorb LLM judgments without adding model calls during play.Source: §2.2 · p. 2

3 Distilling One or More Judges

The paper turns pairwise judgments from one or more teachers into a compact evaluator. It averages teacher probabilities for partnerships, fits the evaluator to those preferences, and folds its scores into Stockfish’s search evaluation.

Distilling One or More Judges

  • Each judge compares candidate moves in both presentation orders; Jev returns probabilities directly, while Qwen and the other tested models provide probabilities through their respective interfaces.The same board text and question are used for Qwen, with reasoning mode off.Source: §3 · p. 2
  • For a Jev–Qwen partnership, the training target averages their probabilities; single-judge evaluators use that judge’s probabilities, a form of multi-teacher distillation [9].The combination tests whether a second judge’s answers improve the evaluator.Source: §3 · p. 2
  • The 443-weight evaluator learns a weighted sum of position features from judge probabilities using binary cross-entropy in a Bradley–Terry preference model.Its weights cover 384 piece-square values and 59 positional features; piece-square weights are centered at zero to preserve fixed material values.Source: §3 · p. 2
  • Stockfish uses a stored integer-weight evaluator combining material, a hand-written positional baseline, and a scaled learned score, without requiring a judge during play.The learned and baseline terms are folded together before play.Source: §3 · p. 2

4 Controlled Evaluation

The controlled evaluation holds the search and evaluator form fixed while varying learned judgments, label budgets, and partner models. It specifies data construction, fitting and calibration procedures, and matched chess-game comparisons.

Controlled Evaluation

  • The controlled study varies learned judgments while holding search and evaluator form fixed, testing label-source value, partnership value, and robustness to changed labels or partners.Figure 2 depicts distillation, equal-budget comparisons, and substitute controls.Source: §4 · p. 2, Figure 2 · p. 3
  • Four judges label 63,135 candidate pairs from 13,000 positions in 2,042 games, with each pair presented in both orders; Jev+Qwen uses 252,540 responses.Equal-budget single-judge comparisons label a second block of 63,135 pairs on new positions.Source: §4 · p. 2
  • Evaluators share code, features, game-based splits, and candidate penalties; the stronger baseline Hs beats Hm by 55.2 Elo [29.6, 81.4].A further split of training data selects the regularization parameter by validation log loss.Source: §4 · p. 2
  • Learned-score scale is selected from four standard-deviation multiples by validation games against Hs, with the smaller scale preferred in a tie.Adding the learned score to Hs at full scale reduces playing strength; rescaling Hs alone costs −49.0 Elo [−72.2, −26.1].Source: §4 · p. 3
  • Chess matches use 60 seconds per side, both colours for every opening, and 95% bootstrap intervals clustered by opening or shared book prefix.Comparisons use the same openings for opponents, while new comparisons draw fresh sets.Source: §4 · p. 3
  • A search-generated reference evaluator learns Stockfish 17.1’s move preferences or centipawn position values using the same features and fitting procedure.It tests what those features and fitting steps capture from labels produced by a chess search.Source: §4 · p. 3

5 What the LLM Partnership Adds

A Jev–Qwen partnership improves chess play under equal labeling budgets, while controls implicate Jev’s position-dependent judgments and comparisons show that label-level accuracy does not predict performance after search. The value of another judge depends on the partner and task.

5.1 Learning a useful search evaluator

  • On the mobility baseline, the Jev-trained evaluator adds 38.9 Elo [29.9, 47.9] over 2,000 games; in direct 400-game matches, Jev+Qwen beats Jev alone by 77.7 Elo [52.5, 103.7].Source: §5.1 · p. 3
  • Against the stronger Hs baseline, Jev+Qwen adds +33.8 Elo [25.2, 42.4] and beats Qwen by 18.1 Elo [10.8, 25.5] over 2,000 games.Jev alone adds +2.8 [−5.5, 11.0], compared with Qwen’s +16.7 [9.0, 24.4].Source: §5.1 · p. 3

5.2 A partnership or more training pairs?

  • At equal response budget, Jev+Qwen beats Qwen trained on twice the pairs by +9.6 Elo [4.3, 14.9], replicated at +8.4 [2.9, 14.1] on fresh openings.Two answers from the same judge, extra pairs, and matching Qwen’s average distance from 0.5 do not recover the partnership’s benefit.Source: §5.2 · pp. 3–4, Table 1 · p. 4

5.3 The information in Jev’s labels

  • Uniformly softening Qwen, freely shuffling Jev’s probabilities, and recalibrating Qwen at T ≈18 with twice the pairs all trail the real combination.A stricter shuffle’s lead is +5.3 Elo [−0.5, 11.0]; the free shuffle indicates that Jev’s match between probabilities and positions carries the gain.Source: §5.3 · p. 4

5.4 Choosing a partner

  • Jev+Qwen leads OLMo+Qwen and DiffusionGemma+Qwen under both partner-selection rules; with per-subset settings, its edge over Qwen on twice the pairs is +2.2 Elo [−2.4, 6.8].The full-data advantage over Qwen on twice the pairs is +9.6 Elo [4.3, 14.9].Source: §5.4 · p. 4, Table 3 · p. 4

5.5 Why measure decisions after search?

  • Similar judge agreement does not predict playing strength: Jev’s evaluator has higher move accuracy than the combined evaluator on quiet positions, yet loses in direct play.On 1,285 quiet positions, accuracy is 62.4% versus 60.7%; full-search play tests how scores perform across evaluated continuations.Source: §5.5 · pp. 4–5

6 Beyond Chess: Passage Reranking

Jev’s labels train a passage reranker with quality similar to Qwen’s, while adding Qwen yields little and splitting judgments can sacrifice the benefit of broader Jev coverage. In this task, Jev alone provides useful labels at substantially lower labeling time.

6 Beyond Chess: Passage Reranking

  • Rerankers trained on both judges’ probabilities for 80,000 MS MARCO pairs rerank BM25’s top 100 passages for 97 human-judged TREC queries, then run without judge calls.The evaluation uses TREC Deep Learning 2019 and 2020 queries and five training seeds.Source: §6 · p. 5
  • Jev’s reranker scores 0.637 NDCG@10 versus Qwen’s 0.630, a +0.008 difference [−0.007, 0.022], and improves on BM25’s 0.495 by +0.142 [0.098, 0.186].A public MS MARCO cross-encoder of the same size scores 0.708.Source: §6 · p. 5
  • Collecting Jev’s 160,000 answers took about 21 minutes of API time without a local GPU, versus Qwen’s 5.1 GPU-hours in this labeling setup.Across chess and retrieval, the chess evaluator gains from partnership, whereas the reranker trains as well from Jev alone.Source: §6 · p. 5
  • Adding Qwen to Jev gains at most +0.004 [−0.003, 0.011] in tested combinations; splitting presentation orders between judges gives similar quality at the same answer budget.When choosing between two judges on half the pairs and Jev on all pairs, the combination loses 0.032 NDCG@10 [0.018, 0.045].Source: §6 · p. 5

7 What the Proof of Concept Shows

The proof of concept treats probability-only judgment as positional guidance, while search supplies lookahead and stored chess knowledge handles familiar positions. Its results show that evaluator usefulness and the value of an additional judge depend on the task.

7 What the Proof of Concept Shows

  • Search explicitly explores continuations, the learned evaluator guides positions, and opening and endgame tools provide stored chess knowledge.Distillation makes positional guidance cheap enough to reuse throughout the search tree.Source: §7 · p. 5
  • Jev-derived evaluation improves chess and Breakthrough play but weakens Othello and Connect-4, where tested judges rarely prefer corners and the evaluator gives them negative weight.The results indicate that labels need knowledge suited to the task.Source: §7 · p. 5
  • Jev and Qwen together produce a stronger chess player, while Jev alone scores as high as Qwen in passage reranking and adding Qwen gains at most +0.004.The paper argues that another judge is worth its label budget only when it improves the deployed evaluator enough to justify the cost.Source: §7 · pp. 5–6
  • Table 4 covers the cost of obtaining 126,270 judge responses, corresponding to 63,135 pairs presented in both orders.The caption also notes that effective rates reflect concurrent requests.Source: Table 4 · p. 6

8 Labeling Cost and Reproduction

The paper reports the cost of collecting chess labels and describes the shared evaluator form and reproduction details. API wall-clock time for Jev is reported separately from GPU time for other judges.

8 Labeling Cost and Reproduction

  • Jev labeled 126,270 responses for $1.97 and about 16 minutes of API time; Qwen and OLMo each used over an hour of GPU time.Jev's row is API wall-clock time, while the other figures are GPU time on listed hardware; responses cover 63,135 pairs in both orders.Source: §8 · p. 6
  • DiffusionGemma, served through djev on one RTX 5090, completed the labels in 0.28 GPU-hours.TypeSafe's stated Jev rate was $0.042 per million input tokens, with output free and 372.3 billed tokens per call.Source: §8 · p. 6
  • All chess evaluators use the same 443-weight form during play, and appendices specify splits, features, prompts, and training procedures.The paper says the measured configurations are reported in Table 4.Source: §8 · p. 6

9 Related Work

The related work places the study alongside distillation, multi-judge aggregation, language-model judges, and game evaluation methods. Its focus is probability-only judgments distilled into a compact evaluator used within search.

9 Related Work

  • Knowledge distillation trains smaller networks to match averaged probabilities [9], while multiple-judge learning has also been studied.Motif uses collected language-model preferences to learn a reinforcement-learning reward [17].Source: §9 · p. 6
  • Prior methods model annotator errors, combine imperfect label sources, or average reward-model weights [20]; this study averages judges' probabilities.The comparisons hold model size, search, and response budget fixed.Source: §9 · p. 6
  • Related work evaluates language models as preference judges [21], diverse judge panels [22], correlated Jev and LLM failures [23], and confidence-gated Jev judgments [24].This paper transfers judgments into an evaluator that runs without either judge and tests offline uncertainty-based allocation.Source: §9 · p. 6
  • Game evaluators have been learned from search leaves [25], outcomes, pairwise comparisons [27], and commentary [28], while NNUE supports traditional search.Other work trains a transformer on Stockfish scores to play without search [30] and studies planning with language models [31].Source: §9 · p. 6

10 Conclusion

The conclusion presents probability-only judgments as components for time-constrained planning: search supplies lookahead, distillation makes judgments reusable, and an additional judge must earn its budget through its contribution. The accompanying sections detail fitting, scale selection, deployment, judge-combination tests, and feature swaps.

10 Conclusion

  • Combining Jev and Qwen improves the full-data chess evaluator, while Jev alone trains a useful reranker and adding Qwen gains at most +0.004 in NDCG@10.The conclusion assigns search the role of considering consequences and distillation the role of making judgment reusable.Source: §10 · p. 6

A Fitting Details

  • The chess fitting procedure holds out one fifth of source games for evaluation and one fifth of the remainder for selecting training settings.For the equal-response-budget comparison, each judge labels 126,270 pairs from two blocks; the second adds positions from mostly the same source games.Source: A Fitting Details · pp. 7–8

B Scale Selection

  • Table 5 compares Jev+Qwen with single-judge evaluators trained on twice the pairs across scales; Table 6 selects scale by validation Elo against published tables.Table 5 uses 400 games on 200 unused openings; Table 6 tests 200 games on the same 100 colour-paired openings.Source: Table 5 · p. 8, Table 6 · p. 8, B Scale Selection · p. 8

C The Recorded Lichess Deployment

  • The recorded Lichess bullet trajectory rises from 1753 to 2268 for a complete bot combining Stockfish search, a Jev evaluator, stored chess knowledge, and live Jev choices.The controlled evaluator tests disable the opening book, tablebases, and live queries during play.Source: C The Recorded Lichess Deployment · p. 8

D Testing All Judge Combinations

E Locating the Learned Contribution

  • −8.7 [−25.8, 8.4] for Jev+Qwen and +9.7 [−5.2, 24.7] for DiffusionGemma+Qwen are the changes after rescaled piece-square weights are swapped.The swaps hold total score variation fixed; the broader feature swaps and their scale effects are described in the experiment setup.Source: E Locating the Learned Contribution · p. 8

F Results at Other Time Controls

Jev+Qwen retains advantages over single-judge evaluators at longer controls, while Jev-trained scores transfer unevenly across games and fixed-position accuracy need not match direct-game strength.

G Transfer to Other Games

  • Transfer varied by game: Jev-trained scores add +38.9 Elo in chess and +219.9 Elo in Breakthrough, but yield −619.4 Elo in Othello and −157.7 Elo in Connect-4.In Othello, all three tested judges preferred non-corner moves in most pairs; Jev preferred the corner in 14.4%.Source: G Transfer to Other Games · p. 9

H Move Accuracy on Fixed Positions

  • On 1,285 quiet positions, Jev selects the better move 62.4% of the time and Jev+Qwen 60.7%, although Jev+Qwen wins their direct match by 27.0 Elo.Including material and mobility, move accuracy is 65.29% for Jev and 64.44% for Jev+Qwen.Source: H Move Accuracy on Fixed Positions · p. 9

I Choosing the Weight-Size Setting

  • With settings selected per subset on half the data, Jev+Qwen leads Qwen trained on twice the pairs by +2.2 [−2.4, 6.8]; the full-data comparison is +9.6 [4.3, 14.9].After recalibrating partners’ probabilities against Stockfish, the Elo changes are +3.8 [−2.1, 9.6] for OLMo+Qwen, +6.1 [−0.2, 12.4] for DiffusionGemma+Qwen, and −2.1 [−8.6, 4.4] for Jev+Qwen.Source: I Choosing the Weight-Size Setting · p. 9

F Results at Other Time Controls

  • At 180 seconds per side, Jev+Qwen leads Qwen trained on twice the pairs by 15.6 Elo [5.7, 25.8] and Jev trained on twice the pairs by 17.4 Elo [6.3, 28.7].At 10 seconds per side plus 0.1 seconds per move, the differences are +6.9 [−4.0, 18.1] against Qwen and +21.9 [9.2, 34.7] against Jev.Source: F Results at Other Time Controls · pp. 8–9, Figure 5 · p. 10

K Examining the Othello Failure

  • In Othello, adding the Jev-trained evaluator yields −603.9 Elo, while subtracting it yields +157.7 Elo in a separate 200-game check.The labels’ candidate moves were legal; Jev preferred corner moves in 14.4% of the tested pairs.Source: K Examining the Othello Failure · p. 10

J Results for Each Training Subset

L Reproduction Details

  • The 13,000-position chess set came from 2,042 games; each position retained up to four material-only Stockfish moves within 20 centipawns of the best line.Fresh-opening sets excluded positions and move sequences used earlier; chess-match intervals generally resampled opening clusters.Source: L Reproduction Details · pp. 10–11

M Prompt

  • Judges compared two candidate moves in both presentation orders and averaged probabilities; a second label set presented all candidate moves for each position.Source: M Prompt · p. 11

N Retrieval Details and Comparisons

In passage reranking, Jev labels perform near Qwen’s, while adding Qwen changes NDCG@10 little at fixed pair coverage; concentrating judgments trades broad coverage for weaker or uncertain gains.

  • At 5,000 steps and 80,000 answers, Jev scores 0.550 and Qwen 0.560; doubling to 10,000 steps and 160,000 answers adds +0.038 for Jev and +0.030 for Qwen.The study labels 80,000 passage pairs from 8,000 MS MARCO queries with each judge and trains MiniLM rerankers on Bradley–Terry soft labels.Source: N Retrieval Details and Comparisons · p. 11
  • On 500 human-graded pairs, Jev gives the same answer in both presentation orders on 94.8% of pairs and Qwen on 90.6%; on training pairs the rates are 86.7% and 72.0%.Source: N Retrieval Details and Comparisons · p. 11
  • Using both judges on every pair scores 0.641 NDCG@10, a +0.004 difference [−0.003, 0.011] from Jev alone; splitting presentation orders scores 0.636, a −0.001 difference [−0.011, 0.009].Source: N Retrieval Details and Comparisons · p. 11
  • At 5,000 steps, averaging both judges on 40,000 pairs loses 0.032 [0.018, 0.045] to Jev alone on 80,000 pairs; certainty-gated Qwen labels give −0.007 [−0.027, 0.012] at 160,000 answers.At 80,000 answers, dropping 20% of pairs costs 0.050, while Qwen labels on uncertain pairs recover 0.004.Source: N Retrieval Details and Comparisons · p. 11
Loading 2610.09188v1…