PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions

Xiaolu Ding, Kai Yang, Dongdong Liu

arXiv:2610.37732v1cs.AIcs.CLcs.LG

TL;DR

Direct probabilities are fast but often wrong on multi-step typed decisions, while thinking is costly. ThinkJev combines direct and thinking answers from one hybrid model and one processed prompt; at a 5% target, Qwen3.5-27B answers 24.5% directly with 3.9% realized error, while anytime stopping retains most thinking accuracy with fewer tokens.

  • Problem

    Direct one-pass answers often miss multi-step decisions, while thinking first improves them but costs seconds rather than milliseconds.

    Source: §1 · pp. 1–2

  • Method

    ThinkJev reads the same option-letter slots from one hybrid model directly or after a thought, using a dev-set test to gate direct answers and checkpoints to select thinking budgets.

    Source: §1 · p. 2, §3 · p. 5, §3.6 · p. 8

  • Results

    At a 5% target, Qwen3.5-27B answers 24.5% of decisions directly with 3.9% test error; anytime stopping reaches 81.6% accuracy against 82.7% for always thinking with 72% of thought tokens.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Direct-answer confidence alone is a weak guide to when thinking will fix an answer, so confidence-based gating saves little thinking overall.

    Source: §1 · p. 3

    The readout is limited to decisions with at most 16 options, each represented by one letter token; rotations cover cyclic shifts, not arbitrary permutations.

    Source: §6 · p. 18

Abstract

from arXiv · show

Many uses of language models end in a typed decision: a choice among a closed set of options, whose probabilities code can act on. Reading those probabilities from one forward pass is fast but often wrong on decisions that take several steps, such as arithmetic, counting, and date comparison. Letting the model think first fixes most of these errors, but costs seconds per decision instead of milliseconds. To pay for thinking only where it helps, ThinkJev builds a fast and a slow system from one open hybrid-thinking model that can answer directly or after a thought. System 1 reads the model's probabilities for the option letters when it answers directly; System 2 reads the same letters after the model has thought on the same cached prompt, so escalating repeats no work and both systems give probabilities over the same options. Because the letters can be read after any number of thought tokens, one sampled thought also gives the answer under every thinking budget. Without training, ThinkJev lets System 1 answer alone only where a statistical test on labeled dev data guarantees that its accepted answers stay under a target error rate. At a 5% target, Qwen3.5-27B answers 24.5% of 8359 test decisions this way with a realized error of 3.9% (Qwen3.5-4B and 9B: 6.5% and 0.0%). For the rest, stopping each thought as soon as its answer is confident reaches 81.6% accuracy, against 82.7% for always thinking, with 72% of the thought tokens. Choosing when to think from System 1's confidence alone saves less, because the decisions System 1 is least sure of are often not the ones thinking fixes. On the public items of JevBench, an independent benchmark of typed-decision services, Qwen3.5-27B thinks on 29.4% and is not distinguishable in accuracy from the two thinking APIs listed (exploratory). Code and data generators are available at https://github.com/ding-xiaolu/ThinkJev.

1 Introduction

Typed decisions require probabilities over fixed options that code can use, but direct one-pass answers often fail on multi-step questions and thinking is costly. ThinkJev combines direct and thinking answers from one processed prompt, with a dev-set error guarantee for direct answers and answers available at every thinking budget.

1 Introduction

  • 52.3% of multi-step decisions are answered correctly by a single forward pass with Qwen3.5-27B, versus 40.3% with Qwen3.5-4B.SemIf and AnyJev also stop at this single pass.Source: §1 · p. 2
  • After thinking, Qwen3.5-27B answers 87.5% of the same decisions correctly, but thinking takes a median of 30.3 s versus 101 ms to process the prompt and 53 ms to read a direct answer with Qwen3.5-4B.Thinking adds little accuracy on routine decisions.Source: §1 · p. 2
  • One hybrid-thinking model supplies both systems: the same option-letter answer slots are read directly or after a thought, and one thought yields answers at every token budget.The direct and thinking prompts share a prefix, and escalation continues from the already processed prompt.Source: §1 · p. 2
  • A dev-set Learn-then-Test threshold bounds the error rate of decisions answered directly; decisions below that confidence threshold are escalated to thinking, with calibrated probabilities over the same options.Decisions from one template count as one sample to avoid inflating the guarantee with near-duplicates.Source: §1 · p. 2
  • One sampled thought can be cut at token checkpoints to produce the answer under each thinking budget, allowing an anytime policy to stop at its first confident answer.System 1 is the zero-token budget.Source: §1 · p. 2
  • The paper also introduces a code-generated benchmark of multi-step decisions spanning eight families, four difficulty levels, English and Chinese, plus chess puzzles.Both decisions and labels are generated by code.Source: §1 · p. 3

2 Background and related work

Related work addresses typed-decision probabilities, adaptive thinking, and routing or risk control. ThinkJev differs by reusing one model and cached prompt, reading closed-set probabilities at different thinking budgets, and applying a statistical test to the direct-answer threshold.

2 Background and related work

  • SemIf and AnyJev provide direct-only option readouts; OpenJev accepts a preset thought budget, while ThinkJev reads all budgets from one thought and chooses per decision whether to think.AnyJev adds cyclic-shift debiasing, batch priors, target-risk coverage, and hidden-state heads.Source: §2 · p. 4
  • Typed-decision services return option probabilities and confidence, and prior open-model methods reproduce direct readouts or add debiasing and calibration techniques.ThinkJev’s gate follows confidence-based routing but sends the low-confidence branch to the same model’s thinking mode.Source: §2 · p. 4
  • Training-based adaptive-thinking methods change model weights, whereas ThinkJev trains nothing; Sonata predicts a budget before a thought, while ThinkJev reads budgets from one thought.DART is training-free but targets free-form answers, and DEER exits a reasoning trace when a trial answer is confident.Source: §2 · p. 4
  • FrugalGPT and RouteLLM route across models, whereas ThinkJev escalates within one model on a cached prefix and calibrates System 2 on the same scale as System 1.Learn-then-Test provides the finite-sample threshold guarantee applied to System 1.Source: §2 · p. 4

3 Method

ThinkJev builds direct and thought-based decision systems from the same option slots and shared prompt prefix. Its policies govern when to escalate, how long to think, and when System 1 may answer under an error guarantee.

3 Method

  • Both systems read probabilities over the same options, sharing prompt processing while allowing user-controlled thinking budgets.A decision comprises a state, question, and 2–16 options; the shared-prefix and checkpoint reads meet the method’s requirements.Source: §3 · p. 5

3.1 System 1: reading the answer slots

  • System 1 reads option-letter logits after one forward pass, with a dev-fitted temperature; option rotations debias confidence against letter-position effects.The rotated read averages option log-probabilities geometrically and removes position bias additive in logit space.Source: §3.1 · pp. 5–6

3.2 Shared prefix and fork

  • The systems share one prefilled prompt prefix, but System 1 reads a cache copy while System 2 continues on the original to avoid repeating prefill.The fork is located at the end of the prompts’ longest common token prefix because Qwen3.5’s recurrent state cannot be rolled back.Source: §3.2 · p. 6

3.3 System 2: the same slots after a thought

  • System 2 reads the same option-letter slots after a thought, producing calibrated probabilities over the same options without parsing generated text.Its probabilities use a separate temperature fitted on dev.Source: §3.3 · p. 7

3.4 Anytime readout

  • One sampled thought supplies answers at successive token checkpoints, so the answer under each budget can be read without sampling a separate thought.The method uses checkpoints at 0, 128, 256, 512, 1024, and 2048 tokens, plus the full thought of up to 4096 tokens; each checkpoint has its own dev-fitted temperature.Source: §3.4 · p. 7

3.5 Calibrated gating and the anytime policy

  • The gate escalates when System 1 confidence falls below a dev-selected threshold; the anytime policy stops at the first confident checkpoint or uses the full thought.The gate chooses the threshold to minimize thought tokens while keeping dev accuracy within the selected tolerance of always thinking.Source: §3.5 · pp. 7–8

3.6 Risk guarantee for System 1

  • Learn-then-Test selects a System 1 threshold on dev so accepted answers meet a target error bound with confidence at least 1 − δ.Thresholds are tested from strictest to loosest, stopping at the first failure; accepted test decisions are reported by coverage and realized risk.Source: §3.6 · p. 8

3.7 A server for typed decisions

  • thinkjev-serve maps each choice, yes/no, or score question to one decision and returns calibrated probabilities from whichever system answers.Temperatures and the threshold come from an evaluation report; the service uses TypeSafe’s public API request and response shapes.Source: §3.7 · p. 8

4 Experimental setup

The experiments compare four hybrid-thinking models across routine, public-reasoning, and generated multi-step decisions. They evaluate fixed savings and risk criteria alongside routing baselines, with fitting and operating-point selection confined to dev data.

Models

  • Four BF16 hybrid-thinking models are evaluated on the same decisions and settings, with Qwen3.5-27B used for the main text and figures.The models are Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, and Qwen3.8-27B; quantities are reported for all four.Source: §4 · p. 8

Data

  • The evaluation combines routine decisions, public reasoning benchmarks, and generated multi-step decisions plus chess puzzles; dev/test splits keep variants of one template together.Only the generated-and-chess block includes Chinese decisions, and each source is used by all four models.Source: §4 · pp. 8–9, Table 2 · p. 9

Criteria

  • The prespecified savings criterion requires gate accuracy within 1.5 points of always thinking while escalating at most half of decisions; the risk criterion checks System 1 against each target.The anytime curve is descriptive, and thresholds, temperatures, probes, and operating points are selected on dev rather than test.Source: §4 · p. 9

Policies and baselines

  • Policies are simulated offline from runs containing both system outputs, comparing the confidence gate and anytime policy with always-direct, always-thinking, text-based pre-routers, and an oracle.The pre-routers use hidden-state probes, a zero-shot glance question, or supervised hashed n-grams to decide whether to escalate.Source: §4 · pp. 9–10

Metrics

  • Evaluation reports accuracy, mean thought tokens, ECE, coverage and realized risk, plus normalized accuracy–coverage area and grouped bootstrap intervals for gate differences.Realized-risk intervals use an exact binomial calculation over accepted test decisions and treat decisions as independent.Source: §4 · p. 10

Compute and latency

  • Latency is measured for Qwen3.5-4B on one H200 at batch size 1 over a stratified subset of 600 test decisions.The single-process measurement runs with nothing else on the machine.Source: §4 · p. 10

5 Results

On Qwen3.5-27B, the anytime policy matches the gate’s 81.6% test accuracy while using fewer thought tokens than always thinking. Results show where thinking helps, how risk-controlled direct answers and budget-based stopping perform, and how the policies compare across benchmarks, routers, latency, and JevBench.

5.1 Main result

  • 81.6% accuracy with 1,344 thought tokens per decision: the anytime policy matches the gate’s accuracy while using 72% of always thinking’s tokens on 8,359 test decisions.Always thinking reaches 82.7% accuracy with 1,856 tokens per decision; the gate reaches 81.6% with 1,511 tokens.Source: §5.1 · p. 10, Table 3 · p. 11

5.2 Where the thinking goes

  • 77.1% of block C decisions are escalated, where System 1 is 52.3% accurate and System 2 is 87.5% accurate; escalation reaches 99.7% for date comparison and multi-step arithmetic.For CLINC150 intent routing, System 1 is 95.5% accurate and the gate escalates 9.8%. Chess and typed decisions also show that low System 1 confidence does not reliably identify cases where thinking helps.Source: §5.2 · p. 11, Table 4 · p. 11, Figure 3 · p. 12

5.3 Public benchmarks

  • On public benchmarks, thinking adds most on reasoning tasks, while the gate escalates more there and lets System 1 answer most classification decisions.For Qwen3.5-27B, BIG-Bench Hard rises from 70.6% with System 1 to 94.8% with System 2; BANKING77 rises from 85.0% to 86.4%. These scores are not comparable with published scores because the evaluation protocol differs.Source: §5.3 · pp. 12–13, Table 5 · p. 13

5.4 The anytime curve

  • 81.4% accuracy is reached by 2,048 thought tokens, close to the full-thought 82.7%; the anytime policy uses 72% of always thinking’s tokens at the same 81.6% accuracy as the gate.Each checkpoint has its own dev-fitted temperature to maintain calibration; the authors report the anytime curve descriptively and draw no directional conclusion from it.Source: §5.4 · p. 13

5.5 Risk guarantee / 5.6 Scale

  • At a 5% target risk, System 1 answers 24.5% of Qwen3.5-27B test decisions with 3.9% realized risk; the guarantee does not improve steadily with model scale.Qwen3.5-4B accepts 6.5% and Qwen3.5-9B 0.0% at that target; Qwen3.8-27B exceeds the 10% target with 10.5% realized risk. Across Qwen3.5 models, System 1 accuracy rises from 50.9% to 62.2%, while System 2 rises from 77.4% to 82.7%.Source: §5.5 · p. 14, §5.6 · p. 14, Table 6 · p. 14

5.7 Gating versus pre-routing

  • The zero-shot router ranks decisions better than the gate on Qwen3.5-9B, Qwen3.5-27B, and Qwen3.8-27B, while all policies remain far from the oracle.The best normalized area is 0.491 on Qwen3.5-27B; the n-gram router is worse everywhere, and the probe router is worse on Qwen3.5-4B and Qwen3.5-27B and ties on the other two models.Source: §5.7 · pp. 14–15, Table 8 · p. 16

5.8 Latency

  • 30.3 s is the median time for System 2 on Qwen3.5-4B, compared with 101 ms to process the shared prefix and 53 ms to read System 1’s answer slots.Latency was measured on Qwen3.5-4B only, across 600 stratified test decisions, all of them thinking.Source: §5.8 · p. 16

5.9 External check on JevBench

  • On JevBench, Qwen3.5-27B’s gate answers 224 of 231 items correctly while thinking on 29.4%; its accuracy is not distinguishable from either listed thinking API in this exploratory comparison.The test does not resolve small gaps at near-ceiling accuracy, so “cannot be told apart” does not mean a tie; the listed systems were measured by JevBench, not by the authors.Source: Table 9 · p. 17, §5.9 · p. 17

6 Analysis and limitations

The analysis identifies decisions where thinking changes correctness, then sets out limits on the evidence, calibration, readout, and cost of the approach.

6 Analysis and limitations

  • 1,971 Qwen3.5-27B test decisions changed from wrong to right after thinking, while 260 changed from right to wrong.Most fixes were in block C (1,212 fixed, 70 broken), followed by block B (576 fixed, 91 broken).Source: §6 · p. 18
  • At least 2,231 test decisions per model differed in correctness between System 1 and System 2, leaving decisions where policy choice matters.Qwen3.5-4B had 2,900 such decisions.Source: §6 · p. 18
  • One sampled thought per decision leaves noise in System 2 accuracy and the anytime curve; checkpoint equivalence applies to that sample, not the expected answer at a budget.Source: §6 · p. 18
  • The typed-decision labels come from one teacher model, Chinese decisions occur only in block C, and public benchmark items may overlap training data.The generated block C is not in the model's training data, and the two data sources are reported side by side rather than pooled.Source: §6 · p. 18
  • Calibration and thresholds may not transfer across domains or thinking budgets, and the risk guarantee assumes exchangeable dev and test groups.Deployments need labeled decisions from their own domain and refitting when the thinking budget changes.Source: §6 · p. 18
  • The readout covers decisions with at most 16 options represented by one letter token, while rotations cover cyclic shifts rather than arbitrary permutations.Source: §6 · p. 18
  • Thought tokens dominate escalated-decision cost, while the anytime cost accounting excludes short forward passes used to read earlier checkpoints.Source: §6 · p. 18
  • The study did not evaluate Jev on its test set or run DART or Sonata; its hidden-state router is a stand-in, and a combined gate was fit post hoc.The combined gate was fit only on System 1's correctness.Source: §6 · p. 18

7 Conclusion

ThinkJev uses one hybrid-thinking model for fast direct decisions and slower decisions after thought, with probabilities read from the same option letters. Its results show where direct-answer guarantees and stopping thoughts can reduce effort, while confidence alone is a weak guide for routing.

7 Conclusion

  • One open hybrid-thinking model supplies direct and post-thought systems using the same option letters on one processed prompt, without training.Reading partway through one sampled thought yields the answer at each thinking budget.Source: §7 · pp. 18–19
  • At a 5% target, Qwen3.5-27B answers 24.5% of decisions alone with a 3.9% test error rate; stopping thoughts reaches 81.6% accuracy versus 82.7% for always thinking with 72% of thought tokens.Smaller models answer fewer decisions alone: Qwen3.5-4B accepts 6.5%, and Qwen3.5-9B accepts 0.0%.Source: §7 · p. 19
  • System 1 confidence is a weak guide to when to think: the target of thinking on at most half of decisions failed on every model.The decisions with lowest System 1 confidence are often not the ones thinking fixes; on JevBench, the open 27B model was in the accuracy band of the listed thinking APIs.Source: §7 · p. 19
  • The proposed next step is a router that predicts where thinking helps, requiring System 2 labels on dev; multiple thoughts per decision could reduce anytime-curve noise.The paper also releases code, data generators, decision IDs and splits, evaluation reports, and a typed-decision server.Source: §7 · p. 19

A The anytime readout equals a thinking budget

The anytime readout is intended to match the answer from running the same sampled thought under a given thinking budget. The equivalence depends on consistent sampling and reads, and the implementation uses cache checkpoints.

A The anytime readout equals a thinking budget

  • For a fixed prompt, sampler, and seed, reading the answer slots after cutting a sampled thought at token t gives the same logits as a run with budget t.If the thought ends before the budget binds, the answer is the final System 2 read.Source: A The anytime readout equals a… · pp. 21–22
  • The equivalence requires autoregressive sampling whose position-wise draws do not depend on budget, appending the early-stop sentence only when needed, and numerically identical sampling and read logits.With batched GPU kernels, equivalence is up to bounded floating-point differences; the batch comparison uses a vLLM budget-t run and replay checks.Source: A The anytime readout equals a… · p. 22
  • At each checkpoint, the implementation reads answer slots from a cache copy with the early-stop sentence while the original cache continues the thought.The final chunked-cache read and checkpoint reads match their corresponding one-shot reads in the reported tests.Source: A The anytime readout equals a… · p. 22

B Generated decisions (block C)

Block C contains generated, code-labeled decisions across multiple reasoning families, languages, and difficulty levels, alongside a separate set of chess best-move questions.

B Generated decisions (block C)

  • The generated decisions cover eight families, including date arithmetic, conditional counting, chained rules, multi-step arithmetic, logic grids, graph tasks, and scheduling.Labels are computed by code, with logic-grid solutions checked for uniqueness and answers checked across possible cases where specified.Source: B Generated decisions (block C) · p. 22
  • Each family has four difficulty levels and English and Chinese versions, with answer positions, classes, and numeric-answer ranks balanced by family, language, and level.The default file contains 2,400 English and 1,600 Chinese decisions, and a test regenerates it byte for byte and independently re-derives labels.Source: B Generated decisions (block C) · p. 22
  • The chess decisions turn Lichess puzzles into four-option best-move questions using legal SAN options, with difficulty assigned by rating bucket.Source: B Generated decisions (block C) · p. 22

C Results per source

This section presents test accuracy and escalation by source, language, and difficulty, alongside a representative generated decision for each family.

C Results per source

  • Tables 11–14 report test accuracy and gate escalation by source, language, and difficulty for each model.Each line is weighted by the number of decisions; level “–” marks sources without difficulty levels.Source: C Results per source · p. 22, Table 11 · p. 23, Table 12 · p. 24
  • Table 10 shows one generated English decision at level 1 for each family, abridged from the released file.Source: Table 10 · p. 23

D Prompts

The prompts are fixed across sources and benchmarks. They implement direct answering, thinking-based answering, and a zero-shot router using the same decision content.

D Prompts

  • Every read uses the same prompts across sources, with nothing tuned per benchmark.Source: D Prompts · p. 26
  • System 1 renders each decision as JSON evidence, a criterion, and lettered options, then reads the next-token distribution over option letters.Option IDs are not shown to the model, and rotations move descriptions between letters without other changes.Source: D Prompts · p. 26
  • System 2 enables thinking on the same two messages, continuing from their shared token prefix before reading the answer slots.Source: D Prompts · p. 26
  • Thoughts cut at a checkpoint or budget are closed with an early-stop sentence, and answer slots are read at the same letters as System 1.Source: D Prompts · pp. 26–27
  • The zero-shot router asks whether a decision can be resolved reliably at a glance without step-by-step work, scoring the “Yes” option.Source: D Prompts · p. 27
  • Benchmark decisions preserve multiple-choice questions and options, while classification tasks use constrained label subsets and JevBench follows its server mapping.The BANKING77 and CLINC150 subsets differ from their usual full-label protocols.Source: D Prompts · p. 27

E Server interface

The server accepts typed decision requests and returns probabilities and confidence in corresponding answer formats. Its optional thinking setting controls routing and can expose checkpoint answers as a stream.

E Server interface

  • Requests provide state and typed questions, with choice options or score criteria represented in the public request format shared with TypeSafe.ThinkJev is independent of TypeSafe.Source: E Server interface · pp. 27–28
  • Choices retain their options, nouls map to true/false, and scores map to levels whose output is the probability-weighted level.Source: E Server interface · p. 27
  • The thinkjev setting selects automatic gating, System 1 only, or System 2, and responses include probabilities, confidence, and per-question routing traces.Source: E Server interface · p. 27
  • Questions run sequentially, are limited to 16 options or levels, and count thought tokens as output tokens.Source: E Server interface · p. 27
  • Streaming returns System 1 probabilities followed by the answer at each thought checkpoint.Source: E Server interface · pp. 27–28

F Reproduction

The reproduction materials pin model versions, prompts, splits, and evaluation outputs. The accompanying figures identify the reported test views and their plotted quantities.

F Reproduction

  • The evaluated Qwen models are pinned by Hugging Face revision, and the release instructions specify commands and reports for reproducing results.Source: F Reproduction · p. 28
  • The release split is checked byte for byte, while reports and outputs are checksummed and regenerated from specified cascade and router artifacts.The JevBench check uses its public items and a pinned commit.Source: F Reproduction · p. 28
  • Figures 4 and 5 show test accuracy and escalation by task group and difficulty, respectively, for Qwen3.5-27B.Source: Figure 4 · p. 29, Figure 5 · p. 30
  • Figure 6 plots System 2 accuracy and expected calibration error at thinking budgets, using one sampled thought per decision.Source: Figure 6 · p. 30
  • Figure 7 compares target risk with realized test risk for accepted System 1 answers at δ = 0.05, with exact binomial 95% intervals.A hollow marker denotes a target with no accepted decisions.Source: Figure 7 · p. 31

Related papers

Loading 2610.37732v1…