PaperlayerView exact revision on arXiv

Source-linked AI summary

LEV: Thinking Fast and Deep with Looped Decision Models

Dingjie Song, Zhiling Yan, Yuxuan Zhang, Lichao Sun

arXiv:2610.20951v1cs.AIcs.CLcs.LG

TL;DR

Decision models answer quickly but spend the same computation on easy and hard questions, while generated reasoning incurs token-decoding cost. LEV reruns a natively looped model and reads options after each step, allowing depth to vary by question. Accuracy rises with depth, and below 50 ms LEV-1.4B and LEV-3B outperform same-latency dense models and cascades by 5.4 to 8.8 points.

  • Problem

    Decision models use fixed computation for every question, although some questions do not need more computation and harder questions can benefit from it.

    Source: §1 · p. 1

  • Method

    LEV reads options after each step of a natively looped backbone and uses a stopper fitted on unlabelled requests to choose question depth.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    Below 50 ms, LEV-1.4B and LEV-3B are 5.4 to 8.8 points more accurate than dense models and cascades of the same latency.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    On-demand depth makes depth a serving-time setting, letting one decision model trade latency for accuracy per question rather than per deployment.

    Source: §8 · p. 11

    The study covers publicly available looped backbones of 1.4B to 3B parameters and reports JevBench results on 231 public items; loop-step savings are not wall-clock savings without batched serving.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Decision models answer typed questions about an input with one probability per option, computed in a single forward pass without generating text. Like the fast System 1 of dual-process theory, they answer at once, but their depth is fixed at training time, so they spend the same computation on easy and hard questions. Models that reason in generated text can think longer, but they pay for every extra step in decoded tokens. We present LEV, a decision model that thinks deeper by rerunning its layers in a loop instead of generating tokens. LEV builds on a language model pretrained as a loop, reads the options after every loop step and trains all steps jointly, so one checkpoint can answer at any depth. On three looped backbones, accuracy rises with depth on the same weights, mostly on hard questions, and a 2.6B LEV goes from 63.2% at depth one to 76.2% at depth four on JevBench. On-demand depth, set per question by a small stopper with a threshold fitted on unlabelled requests, saves 15% to 24% of the computation at a cost of under one accuracy point, level with or slightly above a fixed mix of depths. On a locked test set, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense (non-looped) models and cascades of the same latency below 50 ms.

1 Introduction

Decision models answer quickly but spend the same computation on every question, while generated reasoning adds decoded-token cost. LEV uses a natively looped backbone to provide per-question depth, with accuracy gains concentrated on hard questions and stronger same-latency results than dense models and cascades.

1 Introduction

  • Decision models spend the same computation on every question, while generated reasoning makes even easy questions pay for decoded tokens.On JevBench, Kev-4B answers 28 more items correctly than Kev-0.8B, with the gains coming from standard and hard tiers because both answer every easy item.Source: §1 · p. 1
  • Below 50 ms, LEV-1.4B and LEV-3B are 5.4 to 8.8 points more accurate than dense models and cascades of the same latency.Across three looped backbones, accuracy rises with depth on the same weights, mostly on hard questions.Source: Figure 1 · p. 1, §1 · p. 2
  • LEV is a System 1.5 decision model that thinks in loop steps rather than generated tokens, using a backbone pretrained as a loop.A shared readout scores options after each step, and all steps are trained jointly so one set of weights serves every depth.Source: §1 · p. 2
  • A lightweight stopper assigns per-question depth by predicting agreement with the deepest answer, with its threshold set on unlabelled requests.This target avoids labels and is level with or above a fixed mix of depths, unlike confidence thresholds set on labelled data.Source: §1 · p. 2
  • On-demand depth saves 15% to 24% of computation at a cost of under one accuracy point, while remaining within one point of the deepest readout on a locked test.The stopper spends more steps on harder questions without seeing their difficulty.Source: §1 · p. 2
  • LEV gains most from depth on hard questions and, below 50 ms, exceeds dense models and cascades of the same latency; LevBench was frozen before any model read it.LevBench tiers move step by step away from training data.Source: §1 · p. 2

2 Related Work

Related work spans decision models, generative answering, looped language models, and adaptive computation. LEV combines repeated loop steps with decision readouts and an unlabelled-data stopping rule.

Decision models

  • Decision models score options in one pass, whereas generative models name options and may add reasoning text at the cost of decoding time.Generative answers can be sensitive to option order and labels (Robinson et al., 2023; Zheng et al., 2024).Source: §2 · p. 2

Looped models and adaptive computation

3 Motivation

The motivation is that extra computation helps only a subset of questions, while common alternatives either incur broad costs or fail to use added loops. LEV instead allocates loop steps by question and can draw on a backbone pretrained to loop.

Observation 1: Extra computation pays off on few questions, and a cheap first read finds them

  • Kev-4B answers 28 more JevBench items correctly than Kev-0.8B, entirely from standard and hard tiers because both models answer every easy item.The models differ on only 42 of 231 items, and sending the small model's least confident half to Kev-4B nearly matches Kev-4B on all items.Source: §3 · p. 3

Observation 2: Existing ways to add computation cost too much or stay unused

  • Cascades reread uncertain questions and require a second model, while generated reasoning improves hard accuracy by 9 to 13 points but makes median hard items three to four times slower.Every question pays for reasoning text, including easy ones.Source: §3 · p. 3

LEV design

  • LEV repeats the same decoder stack and reads options after each step; its stopper predicts agreement with the deepest answer so questions can exit at different depths.Each packed request state runs as deep as its deepest question.Source: Figure 3 · p. 4

Observation 2: Existing ways to add computation cost too much or stay unused

  • Retrofitted loops change accuracy by less than half a point, and higher gate learning rate, a third pass, or a wider block does not help.The gates stay near zero; removing them lowers accuracy on new sources by 2.2 points, while a backbone pretrained as a loop is open from the start.Source: §3 · p. 4

4 Method

LEV combines a decision model with a natively looped language-model backbone, reading the options at successive depths and deciding when each question can stop. Its shared readouts are trained across depths, and its stopper is fitted using agreement with the deepest answer rather than correctness labels.

4.1 Preliminaries

  • Questions share an encoded state but attend only to their own tokens, and a pointer head scores options using the <decide> and option-closing states.The option probabilities are produced by a softmax with a temperature fitted after training.Source: §4.1 · p. 4
  • A looped transformer repeatedly applies the same decoder-layer stack, giving R steps an effective depth of L · R while retaining the parameters of L layers.Ouro applies language-modeling loss after every step, training each step to end in a readable state.Source: §4.1 · p. 4

4.2 Looped backbone with a readout at every step

  • LEV uses a natively looped backbone and a shared pointer head to produce an option readout at every step, with training loss averaged across steps.Serving to depth k computes only the first k steps, at about k/R of the full cost; each depth is trained as a decision model.Source: §4.2 · pp. 4–5

4.3 Deciding when to go deeper

  • At each depth, a stopper predicts whether the current answer agrees with the deepest answer, using readout features from the current and preceding depths.The stopper is a logistic regression per depth; the cited features include top probability, option margin, normalized entropy, answer changes, KL divergence, log K, and depth.Source: §4.3 · p. 5
  • The stopper and its threshold can be fitted without correct-answer labels by requiring at least 98% agreement with the deepest answers on fitting questions.The threshold is the smallest one meeting that agreement target; the fixed-depth frontier is the piecewise-linear accuracy-cost curve from mixing adjacent depths.Source: §4.3 · p. 5

5 Experimental Setup

The experiments compare three LEV models, trained on a shared dataset and recipe, against dense controls and reference systems. Evaluation uses frozen tuning and held-out panels, including LevBench, and reports accuracy, calibration, cost, and latency.

5 Experimental Setup

  • LEV uses three looped backbones: Ouro-1.4B and Ouro-2.6B run four steps with pretraining loss at every step, while Nanbeige4.2-3B runs two steps with loss only at the last.Dense controls are Qwen3-1.7B-Base, Qwen3.5-2B-Base, and Qwen3.5-4B-Base; references include Kev-0.8B, Kev-4B, and Jev.Source: §5 · pp. 5–6
  • Table 1 reports accuracy and median latency across six panels; its average is the unweighted mean of Kev-Breadth and LevBench.Bold and underline mark the best and second-best values among identically trained LEV models and dense controls; API measurements include network time and are not comparable to local measurements.Source: Table 1 · p. 6
  • All models are LoRA-trained for one epoch on the same 31,329 records, with no Jev labels used for training.The recipe and stopper design were fixed using three tuning panels before evaluation on two held-out panels.Source: §5 · p. 6, Evaluation panels · p. 6

Evaluation panels.

  • Evaluation includes frozen Kev panels, LevBench tiers increasingly distant from training data, Kev-Breadth, and 231 public JevBench items.Reported measures include panel accuracy, hard-tier JevBench ECE, on-demand cost, and median per-request latency on one H200 GPU in bf16.Source: Evaluation panels · p. 6

6 Main Results

Across backbones, additional loop steps improve accuracy on the same weights, with gains concentrated on harder and less familiar data and diminishing at greater depth. LEV also offers multiple operating points and on-demand computation, while comparisons with dense models show latency advantages without fewer operations per model match.

6 Main Results

  • 13.0 points is the JevBench gain for LEV-2.6B across depths; extra steps raise same-weight accuracy with diminishing returns, and gains are larger farther from training sources.The held-out average rises from 51.3% at depth one to 62.0% after the first extra step, compared with 65.7% at depth four; Kev-Seen gains are about 7 points versus 9.0 to 19.5 points on four more distant panels.Source: §6 · p. 7
  • 4.0 points is LEV-3B's held-out-average advantage over Qwen3-1.7B at 23 versus 22 ms; LEV-2.6B is 3.1 points above Qwen3.5-2B at 101 versus 105 ms.LEV-3B is level with Qwen3.5-4B on JevBench and LEV-2.6B with released Kev-4B, but each loop step runs the full backbone.Source: §6 · p. 7
  • On-demand depth saves computation at almost no accuracy cost and beats dense cascades at low latency.LEV matches larger models with fewer parameters, but not fewer operations, because every loop step runs the whole backbone.Source: §6 · p. 7
  • The stopper's on-demand operating point is reported as saving computation at almost no accuracy cost while outperforming dense cascades at low latency.The paper's broader conclusion is that one checkpoint can cover multiple operating points; the reported model comparisons include cases level with or above dense models at similar latency.Source: §6 · p. 7

7 Analysis

Across five research questions, the analysis examines where added loop depth helps, how to choose it on demand, how LEV compares with dense models at matched latency, and which training choices shape its depth curve.

7.1 RQ1: Where does extra depth help?

  • From depth one to four, LEV-2.6B gains a net 10 standard and 19 hard JevBench items, while the easy tier is nearly solved at every depth.On LevBench, its first extra step fixes 20.4% and breaks 8.6% of questions; 12.6% are right at a shallower depth but wrong at depth four, and Kev-Breadth gains are concentrated in language understanding rather than the two taste tasks.Source: §7.1 · pp. 7–8, Figure 5 · p. 7

7.2 RQ2: Does on-demand depth beat a fixed depth?

  • On held-out panels, the stopper stays within one point of the deepest readout while using 76% to 85% of its cost.Its advantage over the fixed-depth frontier is small: 0.2 points for LEV-2.6B on held-out panels, versus 1.4 points on tuning panels; locked tests again show at most a 1.0-point loss to the deepest readout.Source: §7.2 · p. 8, Figure 6 · p. 8
  • On JevBench, on-demand LEV-2.6B uses 1.2, 2.4, and 3.0 steps on easy, standard, and hard items, respectively.Across LevBench tiers it uses 66% of deepest-pass cost on the in-distribution tier and 75% to 87% on tiers further from training data; harder generated questions also receive more steps.Source: §7.2 · p. 8, Figure 7 · p. 9

7.3 RQ3: How should the depth be chosen?

  • Labelled confidence-based early-exit rules do not beat the fixed-depth frontier, and shallow confidence is overconfident on new data and weak at identifying errors that more steps fix.The agreement-target stopper sets its threshold using unlabelled requests; 500 per fold bring LEV-2.6B within 0.2 points of the full fit, while a labelled correctness target falls below the frontier for both Ouro models.Source: §7.3 · p. 9, Figure 8 · p. 9

7.4 RQ4: Does LEV beat dense cascades at the same latency?

  • At 39 to 46 ms, on-demand LEV-1.4B and LEV-3B are 7.0 to 8.8 points more accurate than the only cascade at that latency and 5.4 to 7.5 points above the best dense option.LEV-2.6B at about 150 ms is no better than Kev-4B alone; part of the low-latency advantage reflects that no dense option answers between 22 and 79 ms, while LEV models use at or below dense options’ FLOPs.Source: §7.4 · pp. 9–10, Figure 10 · p. 10

7.5 RQ5: How do training choices shape the depth curve?

  • Jointly trained LEV-2.6B is level with single-depth Ouro-2.6B models at depths two and four and at most 1.3 points behind at depth one; last-step-only training leaves shallow readouts unusable.On pooled held-out panels, increasing training data raises depth-one accuracy from 50.5% to 62.9% and deepest-readout accuracy from 65.3% to 73.0%, while cost falls from 0.76 to 0.63; on untouched Kev-Breadth, the deepest readout stays between 67.3% and 67.7%.Source: §7.5 · pp. 10–11, Figure 11 · p. 10
  • Full-weight fine-tuning lowers both Ouro models’ deepest-readout accuracy by 1.3 to 1.9 points and removes their gain over the frontier, while LEV-3B is unchanged.A Brier term, increasing step weights, and extra loop steps change deepest-readout accuracy by at most 0.9 points and raise frontier gain by at most 0.1 points.Source: §7.5 · p. 11

8 Conclusion

LEV uses loop steps instead of generated tokens, with one checkpoint serving multiple depths. Its accuracy rises with depth, mostly on hard questions, while unlabelled-request stopping reduces computation and small models outperform same-latency dense models and cascades below 50 ms.

8 Conclusion

  • One checkpoint serves several operating points, with accuracy rising as depth increases, mostly on hard questions.The stopper spends more steps on harder questions without seeing their difficulty, and predicting agreement with the deepest answer works where labelled confidence thresholds do not.Source: §8 · p. 11
  • Below 50 ms, the 1.4B and 3B LEV models are 5.4 to 8.8 points more accurate than dense models and cascades at the same latency.Setting the stopper threshold on unlabelled requests saves 15% to 24% of computation at a cost of under one accuracy point.Source: §8 · p. 11
  • The authors identify recovering shallow-step correct answers and testing larger looped backbones as directions for future work.A stopping target that recovers such answers could add accuracy beyond the deepest readout; larger backbones would test whether gains on hard questions grow with scale.Source: §8 · p. 11

Limitations

The limitations bound the study to currently available looped backbones, public JevBench items, and latency measurements without optimized serving. The accompanying appendices report routing, generated-reasoning, retrofitted-loop, training, evaluation-panel, and LevBench details.

Limitations

  • Latency was measured one request at a time without fused kernels, graph capture, or state caching, and optimized serving for looped and dense models remains future work.The study counts on-demand cost in loop steps; batching requests at different depths could convert the savings into wall-clock time.Source: Limitations · p. 11

A.1 Routing from Kev-0.8B to Kev-4B

  • Confidence routing exceeds random routing once more than one-seventh of items are sent to Kev-4B, but the curve is flat for the fifth where Kev-0.8B is most confident.Kev-0.8B’s median top probability is 0.80 on items both models answer correctly and 0.44 on items only Kev-4B answers correctly.Source: §A.1 · p. 14, Figure 12 · p. 15

A.2 Generated Reasoning

  • At an 8,192-token budget, djev’s thinking mode returned no valid answer for 72 of 534 benchmark decisions and got 2 of 48 easy items wrong.The comparison uses JevBench-reported hard accuracy and median latency for hard items; each server’s endpoint latency is comparable only between its own modes.Source: §A.2 · p. 15

A.3 Retrofitted Loops

  • No retrofitted-loop variant improved accuracy on any tuning panel with a confidence interval above zero; the default loop’s second pass contributed less than 1% of its residual update.The default loop’s gates had a mean magnitude of about 0.006 after training.Source: §A.3 · p. 15, Table 3 · p. 16

B.1 Training Settings

  • LEV and dense-control training settings are summarized in Table 4; the section also notes that special-token delimiters are protected from user text that spells them.The delimiters are special tokens in the backbone tokenizer, and matching user text is rewritten to prevent forged boundaries.Source: §B.1 · p. 16, Table 4 · p. 16

B.2 Evaluation Panels

  • The calibration partition fits readout temperatures and early-exit thresholds, while the locked test uses LevBench and Kev-Breadth test partitions plus the test partitions behind the tuning panels.The calibration partition is included in the data used to fit the stopper.Source: §B.2 · p. 17, Table 5 · p. 17

B.3 Training Sets

  • Train-31k contains 31,329 records after filtering; Train-60k and Train-120k add screened records in nested prefixes that preserve the pool’s part proportions.The added pool was screened against evaluation partitions, including public JevBench items, and duplicates or matching records were removed.Source: §B.3 · p. 17, Table 6 · p. 18

B.4 LevBench

  • LevBench’s tiers progress from training-source items through held-out templates and never-trained datasets; its 300 unknowable questions omit the deciding fact and test confidence.The suite was screened against training and evaluation material, then frozen before any model read it.Source: §B.4 · p. 18

B.5 Evaluation Protocol

The evaluation appendix specifies how models, latency, and calibration were measured, then reports paired comparisons, on-demand depth, and shallow-confidence diagnostics. The reported results include depth-wise accuracy differences and tests of calibration and stopping behavior.

B.5 Evaluation Protocol

  • The evaluator reproduces the official harness exactly on Kev-4B and matches Kev-0.8B overall accuracy, differing by under one point on its hard tier.Latency is the median over 231 public JevBench items, one request at a time after warm-up; the Intelligence score is a weighted, chance-corrected accuracy measure.Source: §B.5 · p. 18, Table 7 · p. 18, Table 8 · p. 19

C.2 Paired Comparisons

  • LEV-2.6B at depth four is 8.2 points ahead of LEV-1.4B (95% CI [+3.5, +13.0]) and 12.6 points ahead on the hard tier (95% CI [+4.5, +21.6]).Depth four versus depth one is significant by McNemar’s test for LEV-2.6B (p < 0.001) and LEV-1.4B (p = 0.002).Source: §C.2 · p. 19, Figure 13 · p. 20

C.3 Calibration

  • Temperature scaling lowers hard-tier ECE at depth four from 0.255 to 0.231 for LEV-1.4B and from 0.218 to 0.192 for LEV-2.6B.After scaling, LEV-1.4B’s ECE falls through depth three and rises slightly at depth four; LEV-2.6B has its lowest ECE at depth one.Source: §C.3 · p. 19, Table 9 · p. 20

D.1 Development Panels

  • On tuning panels, an oracle reaches 85.9% accuracy at cost 0.34, versus 77.4% for the deepest readout; using one temperature across depths changes on-demand accuracy by at most 0.2 points.The locked test was read once after the final recipe and stopper were fixed, and its pooled held-out rows are also reported in Table 2.Source: §D.1 · p. 20, §D.2 · p. 20, Table 10 · p. 21

D.3 Further Panels

  • On the two training-source panels, every model computes less than it does on MMLU-Pro; Table 13 reports LEV-2.6B’s on-demand depth by LevBench tier.The panels include consumer-complaint documents, developer-tooling decisions, and 1,000 MMLU-Pro questions excluded from training.Source: §D.3 · p. 22, Table 12 · p. 22, §D.4 · p. 22

D.4 Tiers and Difficulty

  • Figure 14 reports the share of 300 unknowable LevBench questions answered with top probability at least 0.9 by depth and on-demand.The accompanying tier analysis orders LevBench tiers by distance from training data and breaks questions with difficulty k down by k.Source: Figure 14 · p. 22, §D.4 · p. 22

E.1 Stopper and Top-Probability Rule

  • The top-probability rule stops at the first temperature-scaled depth whose top probability reaches a threshold chosen on calibration data to keep accuracy within one point of the deepest readout.The selected thresholds are 0.81 for LEV-2.6B, 0.76 for LEV-1.4B, and 0.69 for LEV-3B.Source: §E.1 · p. 23

E.2 Shallow Confidence

  • On Kev-Breadth, shallow top probability’s AUROC for identifying wrong depth-one answers fixed by deeper reads stays between 0.39 and 0.53 across backbones and training variants.Kev-Breadth ECE ranges from 0.07 to 0.12 across readouts, with neither more training data nor full-weight fine-tuning lowering it consistently.Source: §E.2 · p. 23, Table 15 · p. 24

E.3 Early-Exit Rules and Stopper Variants

The appendix compares stopping rules with fixed-depth frontiers, tests threshold fitting on unlabelled requests, and evaluates LEV against dense models and cascades. It also describes the training-data and dense-control comparisons.

E.3 Early-Exit Rules and Stopper Variants

  • On tuning panels, the LEV-2.6B stopper reaches 76.7% accuracy at cost 0.64, 1.4 points above the fixed-depth frontier (95% CI [+0.6, +2.1]).Against random escalation with the same stopping-depth distribution, its top-probability rule gains 3.8 points on Kev-Seen, 2.5 on Kev-Transfer, and 2.2 on Kev-Skills.Source: Table 16 · p. 24, §E.3 · p. 25

E.4 Unlabelled Requests

  • A stopper fitted on Kev-Seen falls 0.2 to 1.3 points below the frontier on every other panel, while one fitted on another panel is at most 0.7 points below it anywhere.With no requests from the other fold, the LEV-2.6B stopper falls 1.5 points below the deepest readout, compared with 0.7 points using all other-fold questions.Source: §E.4 · p. 25

F Dense Models and Cascades

  • Table 17 compares on-demand LEV with cascades or the best dense option at equal mean latency; LevBench test latencies are 46, 155, and 39 ms for LEV-1.4B, LEV-2.6B, and LEV-3B.Cascade requests pay for both models, whereas on-demand LEV pays for the deepest step its state runs; latency was estimated from one-request-at-a-time measurements.Source: Table 17 · p. 25, F Dense Models and Cascades · pp. 25–26

G.1 Training Data and Full Weights

  • The pooled held-out comparison reports training-data and full-weight variants, with part of LevBench in-distribution for Train-60k and Train-120k.The full-weight variant uses a backbone learning rate ten times lower than LoRA while keeping the data, schedule, and number of steps the same.Source: Table 19 · p. 26, §G.1 · p. 26

G.2 Dense Controls

  • Dense controls trained on Train-31k, Train-60k, and Train-120k are compared with the deepest readouts of looped models of about the same size.The reported table notes that part of LevBench is in-distribution for the two larger training sets.Source: §G.2 · p. 27, Table 20 · p. 27

G.3 Joint Training

Joint training makes shallow and deep readouts usable from one LEV-2.6B checkpoint, while recipe variants expose trade-offs between shallow accuracy and added depth. The reported gains from additional loop steps are limited beyond the fourth step on LevBench.

G.3 Joint Training

  • On JevBench, jointly trained LEV-2.6B scores 63.2%, 71.0%, and 76.2% at depths one, three, and four, versus 60.6%, 72.3%, and 77.9% for separately trained single-depth models.Table 21 reports accuracy and paired differences for joint versus single-depth training.Source: Table 21 · p. 27, §G.3 · p. 27
  • Training only at the final step leaves shallow readouts unusable: depth one scores 36.4% on LevBench versus 46.9% with joint training, and on-demand depth uses 84% rather than 76% of the deepest-pass cost.The final-step-only model’s stopper almost never stops at depth one.Source: §G.3 · p. 27

G.4 Losses and Loop Steps

  • A step-weighted loss costs LEV-2.6B 2.2 depth-one points on LevBench while leaving its deepest readout level; six steps add at most one point after step four on LevBench and none on Kev-Breadth.LEV-3B gains 0.5 points from its third step on pooled held-out panels; a Brier term raises LEV-2.6B’s deepest LevBench accuracy by 1.3 points but not its Kev-Breadth accuracy.Source: §G.4 · p. 27, Table 22 · p. 28

Related papers

Loading 2610.20951v1…