A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models

1 INTRODUCTION

• For loops that settle, compression error does not accumulate; the account holds for deterministic DDIM sampling, whose loop never settles.The study tracks more than 30 models from five families, and reports that 3-bit Huginn improves from 3% to 40% on GSM8K after sixteen precise final loops, versus 42% at full precision.Source: §1 · p. 2
• A fixed rounding error tilts the loop’s bowl once, shifting its settled state by a bounded amount rather than pushing it farther with each loop.An answer is lost only when this shift exceeds the readout’s tolerance.Source: §1 · p. 2
• A shared answer-preserving room and label-free sensitivity measurement predict which models collapse before deployment.A model fails when the compression push exceeds its room.Source: §1 · p. 2
• A collapsed model can recover because it still settles; a few final loops with 8-bit weights pull its state back.The paper says a law stated before the runs predicts how much recovery occurs.Source: §1 · p. 2
• Stopping at the halting head and finishing with 8-bit loops beats fixed-depth inference on Sudoku-Extreme and Maze-Hard by up to 15 points under a third of the weight traffic.The controller does not require a full-precision copy of the compressed weights.Source: §1 · p. 2

2.1 LOOPED MODELS AND SETTLING

• A looped model repeatedly applies one weight-tied map to a latent state, with the input entering each loop and a readout mapping the final state to an answer.A loop settles when its late steps shrink; learned loops need not be contractions, so settling is measured rather than assumed.Source: §2.1 · p. 3

2.2 PUSH, SENSITIVITY AND ROOM

• The settling shift is the movement of the settled state under compressed weights, driven by the push a compressed loop makes there.Sensitivity S is measured without labels using small relative weight noise on 256 unlabelled inputs; Sσ tracks measured push (Spearman 0.91).Source: §2.2 · p. 3

2.3 COMPRESSION, FIXED ERROR AND FRESH ERROR

• Weights are quantized after training by round-to-nearest at per-tensor, per-output-channel, or 32-input-weight-group scales; fixed and fresh Gaussian errors separate error size from fixedness.A third variant gives noise a consistent sign; pruning or distillation leaves TRM unable to solve a puzzle.Source: §2.3 · p. 3

2.4 MODELS AND MEASUREMENTS

• Models are compared with full precision on the same examples and loop counts; survival means retaining at least half of full-precision accuracy at default depth.A widening gap requires the paired bootstrap interval for its change to lie below zero; fidelity is the cosine between final states.Source: §2.4 · pp. 3–4

3.1 A COMPRESSED MODEL THAT SETTLES KEEPS ITS ACCURACY AS IT THINKS LONGER

• None of 22 outer-cycle series widened their gap, while deterministic DDIM FID rose from 40 at 10 steps to 304 at 100 because its loop never settles.Fast inner cycles were an exception, so the authors make no claim about them; 21 of 22 supervision-step series held or narrowed their gap.Source: §3.1 · p. 4

3.2 ERROR SIZE SETS THE CLIFF; FIXEDNESS DECIDES REPAIR

• Zero-mean fresh noise still drove Sudoku-MLP accuracy from 62.9% to 2.3% as noise rose from 0.2 to 0.5 tensor scales, so collapse needs no accumulating bias.Fresh error partly healed in all 30 smaller-noise draws, while fixed error healed in none of 15 MLP and 15 HRM draws; 5 of 15 attention fixed-error draws partly healed.Source: §3.2 · pp. 4–5

3.3 A COLLAPSED MODEL STILL SETTLES, AND A FEW PRECISE LOOPS BRING IT BACK

• In 88% of collapsed models, late steps still shrank nearly as quickly as in surviving models, showing collapse can leave the loop settling off target.The example’s late-step ratio was 0.93.Source: §3.3 · p. 5
• After 16 finishing steps, the median collapsed model retained 96% of full-precision accuracy with full-precision weights and 95% with 8-bit weights.The finishing law placed all 20 scorable models within its prespecified tolerance, with predicted and observed gains correlated at 0.997.Source: §3.3 · p. 5

3.4 A MODEL FAILS WHEN ITS PUSH EXCEEDS ITS ROOM

• A shared room and label-free sensitivity predicted tolerance for 18 models across three families with R2 = 0.72 and median leave-one-out error of a factor of 1.5.The room is fitted rather than measured, and the shared-room fit overestimates MLP-mixer tolerance while underestimating attention-mixer tolerance.Source: §3.4 · p. 6
• Quantization-aware training widened tolerance by cutting sensitivity about 1.7 times while leaving the room unchanged.A fidelity cut set using 50 labelled examples can place a new model’s boundary, but the cut is model-specific for language models.Source: §3.4 · p. 6, §3.5 · p. 7

3.5 LOOPED LANGUAGE MODELS

• Huginn rose from 3% to 40% GSM8K accuracy with precise final loops, against 42% at full precision; Recurrent-OLMo-2 rose from 2% to 45%, against 41%.Ouro is a boundary case: it loses accuracy with extra loops even at full precision, so the settling account gives no recovery guarantee.Source: §3.5 · p. 6, Table 2 · p. 7

4.1 STOP, THEN FINISH

• The controller stops at the first loop when the halting head fires, then applies a few 8-bit finishing steps and reads out from the 8-bit weights.The head’s firing rule is set on full-precision development rows and held fixed for compressed models; the finishing-step count and depth cap are selected on development rows.Source: §4.1 · pp. 7–8

4.2 RESULTS

• 14.7 points: stopping and 8-bit finishing score 75.8 versus fixed-depth inference’s 61.1 on Sudoku-Extreme, at 16.1 versus 55 cost units.The comparison averages 20 compressed models; the gain interval is [14.3, 15.2].Source: Table 3 · p. 8, §4.2 · p. 8
• 7.6 points: the controller raises Maze-Hard accuracy from 70.9 to 78.5 at the same cost as fixed-depth inference.Source: §4.2 · p. 8
• Stopping alone scores 63.0 at cost 23.4, while adding 8-bit finishing reaches 75.8 at cost 16.1 by allowing a lower depth cap.Source: §4.2 · p. 8, §4.1 · p. 8
• Sampling adds three points at nearly twice the cost, and with sampling every model ends above fixed-depth inference.Source: §4.2 · p. 9
• Uniform refinement scores 72.4 at three times the controller’s cost, while confidence-gated refinement scores 71.5 at cost 27.6.Source: Table 3 · p. 8, §4.2 · p. 9
• 8-bit finishing reaches 75.8 accuracy at cost 16.1, versus 75.7 at cost 25.8 with full-precision finishing, so deployment needs no full-precision copy.Source: §4.2 · p. 9
• The halting head outperforms a latent-update-size threshold, which falls below it in 22 of 46 models, including models that pass the answer before settling.Source: §4.2 · p. 9

• Weight-tied depth spans transformers, language models, equilibrium models, and recurrent puzzle solvers; here, settling also determines what compression costs.Source: §5 · p. 9
• Against accounts that attribute collapse to accumulating bias, the paper argues that collapse needs no bias: the loop still settles and lost answers can be recovered.It measures this behavior in reasoners without the convergence certificate available for monotone equilibrium networks.Source: §5 · p. 9
• The controller applies a fixed-point perturbation bound, margin-over-sensitivity radius, and iterative refinement to learned loops, pairing halting-head early exit with finishing steps.Unlike early exit alone, the controller adds finishing; the paper also relates fresh loop errors to stochastic rounding.Source: §5 · p. 9

• For a loop that settles, fixed compression error is paid once, and a label-free measurement of the settling shift can predict which formats survive.A few 8-bit finishing steps can recover accuracy lost to a coarse format; loops that do not settle, including deterministic samplers, remain outside this picture.Source: §6 · p. 10
• Attention mixers have several times the room of MLP mixers, while quantization-aware training lowers sensitivity without changing the room.The discussion proposes training for settling, choosing a mixer with wider room, and keeping a halting head; sensitivity penalties are a possible next step.Source: §6 · p. 10
• The largest compressed looped model has a few billion parameters, so the paper cannot say whether much larger models settle.Costs are modeled from weight traffic rather than timed on hardware; the depth claim covers supervision steps and outer cycles.Source: §6 · p. 10
• The conclusion presents a one-run test for a new looped model: run it longer and observe whether the gap grows.The paper reports that its controller turns collapsed low-bit reasoners into models that beat fixed-depth inference at a third of the weight traffic.Source: §6 · p. 10

• The authors state that they chose the research ideas and experiments, checked every number against written records, and take responsibility for the paper.Source: AI USE DISCLOSURE · p. 10

REPRODUCIBILITY STATEMENT

• Every compressed model is compared with its own full-precision reference on the same examples and platform, and the code, checkpoints, and result files are slated for release.The reported experiments use two GPU platforms and do not pool results across them.Source: REPRODUCIBILITY STATEMENT · p. 10, A MODELS · p. 15, §B.2 · p. 16

A MODELS

• The study covers 33 models from five families, with each model characterized by its repeated map, default depth, and evaluation metric.The set includes puzzle reasoners, implicit models, looped language models, and a diffusion model.Source: A MODELS · p. 15, Table 4 · p. 15, Appendix G · p. 15

Appendix G

• The families use distinct loop structures, including recurrent blocks, convolutional recall, equilibrium solvers, looped language-model blocks, and diffusion denoisers.The DEQ implementations track the lowest-residual iterate, while the diffusion comparisons use matching starting noise and, for DDPM, matching step noise.Source: Appendix G · p. 15

B.1 FORMATS AND INJECTED ERROR

• Weights are quantized after training with per-tensor, per-output-channel, or grouped scales; the experiments also specify activation quantization and fixed- versus fresh-error injections.Calibration rows are kept separate from evaluation rows, and per-call output error is measured along each model’s trajectory.Source: §B.1 · pp. 15–16

B.2 PAIRING AND VERDICTS

• Depth-wise gap changes are classified using paired bootstrap intervals, with a separate slope check to detect widening between endpoints.Comparisons use 95% paired bootstrap intervals over examples; the extended runs also classify catch-up, widening, tracking, and constant gaps.Source: §B.2 · p. 16

B.3 PRUNING AND DISTILLATION ERASE THE PUZZLE REASONERS

• Structured pruning at 25% sparsity and distillation into a one-layer student each leave puzzle reasoners unable to solve a whole puzzle, despite retaining much per-cell accuracy.The comparisons use exact puzzle and cell accuracy for dense models, pruned models, and distilled students.Source: §B.3 · p. 16, Table 5 · p. 16

B.4 WEIGHT-TRAFFIC COST MODEL

• At batch size one, the cost model counts weight bits read: one full-precision loop costs 32 bits per weight, while compressed and 8-bit finishing loops are costed by their respective bit widths.Activations and compute are excluded, and these costs are not measured inference times.Source: §B.4 · p. 16, Table 6 · p. 16

C THINKING LONGER

• Figure 1b normalizes each family’s loop axis by its default depth and plots the gap change from that depth, with paired bootstrap intervals.This normalization is intended to avoid mistaking a model still settling at shallow depth for an effect of looping longer.Source: C THINKING LONGER · p. 17

C.1 NO SURVIVING PUZZLE REASONER LOSES ACCURACY ALONG OUTER CYCLES

• No surviving puzzle-reasoner series widens along outer cycles, and only per-tensor 4-bit weights widen along supervision steps; fast inner cycles are not covered by that conclusion.Collapsed series widen because full precision improves while the collapsed model remains near zero.Source: §C.1 · p. 17, Table 7 · p. 17, Table 8 · p. 17

C.2 IMPLICIT MODELS AND RECURRENT CONVNETS LOSE NOTHING TO EXTRA LOOPS WHILE THEY SURVIVE

• After convergence, MDEQ-Small changes by at most 0.1 points; DeepThinking changes by at most 1.1 points when iterations double, while the collapsed DEQ-Transformer reaches a residual 481 times smaller than full precision’s.The DEQ-Transformer’s small residual can therefore coexist with collapse.Source: §C.2 · pp. 17–18

C.3 THE DETERMINISTIC SAMPLER COMPOUNDS ERROR WITH STEPS; THE ANCESTRAL SAMPLER DOES NOT

• At 4-bit grouped weights, DDIM’s FID distance from full-precision samples rises from 39.5 at 10 steps to 304 at 100, while DDPM’s distance is the same from 50 to 1000 steps.The comparison uses FID-5k against full-precision samples from the same sampler and starting and step noise.Source: §C.3 · p. 18, Table 11 · p. 18

C.4 CLIPPING DECIDES THE PER-TENSOR 4-BIT RESULT ON MAZE

• With per-tensor W4A4 on Maze, maximum-over-calls clipping yields 0.0% while 99.99th-percentile clipping yields 82.2%.Under percentile clipping, the gap closes early and then holds near −6 points.Source: §C.4 · p. 19, Table 12 · p. 19

D.1 MORE LOOPS REPAIR A FRESH ERROR AND LEAVE A FIXED ONE, EXCEPT ON SOME ATTENTION DRAWS

• At the smaller fresh-noise levels, all 30 draws close part of their gap, while matched fixed noise closes none on MLP or HRM and only some attention draws.Among 43 quantized models, the only models that catch up are three surviving 3-bit attention formats and three partial cases; no quantized MLP or HRM model catches up.Source: §D.1 · p. 19, Table 13 · p. 19

D.2 FRESH ZERO-MEAN NOISE ALONE PRODUCES THE CLIFF; A BIAS OR A FIXED DRAW ONLY MOVES IT

• Fresh zero-mean noise alone produces sharp accuracy cliffs; keeping noise fixed across loops or adding bias moves the cliff rather than explaining it by nonzero mean.Across Sudoku-MLP variants, collapse occurs within a three-times-wide band when measured by per-call output error ρ, versus an eight-times-wide band in σ.Source: §D.2 · p. 20, Table 14 · p. 20, Table 15 · p. 20

D.3 ATTENTION DOES NOT BECOME ONE-HOT, SO ONE-HOT MAPS DO NOT EXPLAIN THE EXCEPTION

• The proposed one-hot attention explanation fails: attention entropy and maximum probability stay similar, and attention-path error does not shrink selectively in models that catch up.The prediction and its pass criterion were specified before the records were examined.Source: §D.3 · p. 20, Table 16 · p. 20

E.1 A COLLAPSED MODEL STILL SETTLES

• In 88% of collapsed models, late steps still shrink at nearly the rate of surviving models, indicating that compression can shift the settled state without stopping settling.The table summarizes the pre-specified contraction-loss test and its outcome.Source: §E.1 · p. 21, Table 17 · p. 21

E.2 PREDICTIONS WRITTEN BEFORE THEIR RECORDS

• The return test’s proposed pooled predictor was not decisive: 29 of 64 collapsed models returned even from the test’s grid cap, and return was only one-sidedly linked to correctness.For P3, halting-logit saturation leaves few examples for the tc comparison in MLP and HRM, while the correct-draw result is unaffected.Source: §E.2 · p. 21, Table 18 · p. 22

E.3 RECOVERY CLIMBS WITH THE FINISHING STEPS, AND 8-BIT FINISHING RECOVERS AS MUCH AS FULL PRECISION

• Recovery rises with finishing steps, and after 16 steps 8-bit finishing retains 0.95 against 0.96 for full precision on both platforms.The table reports recovery across collapsed models and notes that surviving models already retain 0.97 and 1.00 at one step on L40S and A100, respectively.Source: §E.3 · p. 22, Table 19 · p. 23

E.4 THE FINISHING LAW PREDICTS THE GAIN ON EVERY MODEL IT CAN SCORE

• For 20 matched TRM and HRM models, the finishing law predicts observed gain within its stated tolerance, with mean absolute error 1.52 points and r=0.997.Where states do not return, the gain vanishes; the MDEQ-XL prediction is reported but not scored.Source: §E.4 · p. 23

F THE TOLERANCE LAW

• The tolerance law uses a shared room δ⋆ and label-free sensitivity S to predict tolerated weight error as σ1/2 = δ⋆/S.The same noise level is drawn once and kept across loops; the reported cross-platform agreement for S is within 2.7% on 18 models.Source: F THE TOLERANCE LAW · p. 23

F.1 THE IMPLIED ROOM SPANS 0.17 TO 0.89 ACROSS THE 19 MODEL–TASK PAIRS

• Among TRMs in Tables 20 and 21, MLP-mixer models have implied room 0.17–0.37, while attention models are higher at 0.51–0.55; implied room is S·σ1/2, the push at which accuracy halves.Table 20 reports an L40S implied-room range of 0.19–0.89; Table 21 reports 0.17–0.52 for its nine scored Sudoku variants on A100.Source: §F.1 · p. 23, Table 20 · p. 24, Table 21 · p. 24

F.2 ONE SHARED ROOM PREDICTS TOLERANCE TO A MEDIAN FACTOR OF 1.5

• Across 19 model–task pairs, the fitted room is 0.344 and the tolerance law has R2(log, slope −1) = 0.72, with median leave-one-model-out error ×1.52.Across families, the room differs: implicit models tolerate more than sensitivity predicts, while the DEQ-Transformer tolerates less.Source: §F.2 · p. 24

F.3 PARAMETER-FREE PREDICTIONS FOR NEW MIXER SEEDS LAND INSIDE THE ERROR BAND

• All six new mixer-seed predictions fall inside the stated error band, with attention tolerating about 4.1× the weight noise of the MLP mixer on L40S.The predictions were parameter-free but not blind, since five seed evaluation files already existed when predictions were written.Source: §F.3 · p. 24, Table 22 · p. 25

F.4 THE TOKEN MIXER’S SENSITIVITY SHRINKS THE MIXER BIAS BUT KEEPS ITS SIGN (POST HOC)

• Token-mixer-only sensitivity raises fit R2 from 0.84 to 0.91 and shrinks the mixer bias, but the choice of this law was post hoc.The token-mixer share of sensitivity differs by architecture.Source: §F.4 · p. 25, Table 23 · p. 25

F.5 LSQ TRAINING LOWERS THE SENSITIVITY AND LEAVES THE ROOM UNCHANGED

• Quantization-aware training raises full-precision tolerance from 0.135 to about 0.23 by reducing sensitivity 1.30×–1.71×, while the implied room does not move.The law over-predicts LSQ models and their full-precision comparators because the MLP family’s room is below the pooled fit.Source: §F.5 · p. 25, Table 24 · p. 25

F.6 DISTANCES MEASURED AT ONE NOISE LEVEL DO NOT PREDICT TOLERANCE

• A distance measured at one noise level does not predict tolerance: the radius law has R2(log) = −0.84, and the zero-parameter room misses new-seed tolerance by ×8.9.The reported explanation is that settling shift grows faster than linearly with noise.Source: §F.6 · p. 25

F.7 FIDELITY FLAGS COLLAPSE WITH A PER-MODEL CUT, AND THE HALTING HEAD IS SAFE ON SURVIVING MODELS

• Fidelity separates collapsed from surviving puzzle models with AUROC 0.98 on L40S, while the halting head answers wrongly in at most 0.7% of cases where it fires on surviving Sudoku models.The push at the settled state also separates collapse with AUROC 0.97 on L40S and 0.98 on A100, using a model-specific cut.Source: §F.7 · p. 26

G LOOPED LANGUAGE MODELS

• In looped language models, return does not reliably separate collapse: Huginn and Recurrent-OLMo-2 collapsed states remain recoverable, whereas Ouro does not settle in the same sense.Ouro loses accuracy beyond its trained loop count even in bf16, and precise final loops provide no held-out gain for surviving compressed Ouro models at the same weight traffic.Source: §G.1 · p. 26, §G.2 · p. 27, Table 25 · p. 27

G.3 FIDELITY ORDERS COLLAPSE WITHIN EACH LOOPED LANGUAGE MODEL, WITH A CUT PER MODEL

• Fidelity orders collapse within each looped language model, but its decision boundary must be set per model; on disjoint compressed models, the per-model cut reaches mean accuracy from 0.889 to 0.999.The pool combines three platforms within a model, and the calibration split is unseeded.Source: §G.3 · pp. 27–28

H.1 THE PROTOCOL OF TABLE 3, AND THE ROWS IT LEAVES OUT

• The controller evaluation chooses configurations on development rows and scores held-out rows; among stopping rules, only the halting head improves on the 16-step protocol depth.The protocol uses 20 quantized Sudoku-Extreme models at a budget of 64 cost units.Source: §H.1 · p. 28, Table 27 · p. 28

H.2 THE HALTING HEAD STOPS EARLY AND KEEPS ANSWERS THAT LATER LOOPS LOSE

• On surviving models, the halting head stops Sudoku 1.75–2.2× earlier at a loss of at most 0.9 points and HRM Sudoku 1.5× earlier without loss; stopping can also raise accuracy past the room.A fixed depth equal to the head’s mean steps loses 4–5 points on TRM and 2.4 on HRM.Source: §H.2 · pp. 28–29

H.3 FINISHING SCHEDULES AND THE COST OF A FINISHING STEP

• On TRM Sudoku-attention, 224 compressed steps followed by 32 8-bit finishing steps score 91.9, a +4.7 gain over compressed-only inference at matched weight traffic.An 8-bit finishing step costs two to four compressed steps, compared with eight to sixteen for full precision; activation traffic and switching weight sets are not charged.Source: §H.3 · p. 29, Table 28 · p. 29

H.4 THE CONTROLLER’S GAIN COMES FROM COLLAPSED MODELS

• For four Maze models at budget 32, the controller with 8-bit finishing scores 78.5 at cost 9.1 versus fixed depth’s 70.9 at cost 9.0, a gain of 7.6 points.Full-precision finishing reaches 78.3 at cost 20.0; across the Sudoku grid, the reported controller gains are concentrated on collapsed models.Source: §H.4 · pp. 29–30, Table 29 · p. 30

H.5 THE HALTING HEAD PICKS THE PUZZLES TO FINISH BETTER THAN CHANCE

• With full-precision finishing over 24 models, head-gated finishing scores 73.9 against 72.0 for random sets at identical cost; with 8-bit finishing on 20 Sudoku models, it scores 73.2 against 71.3.The passage reports the head ahead of the random mean in all 17 active full-precision models and above the random range in all 15 active 8-bit models.Source: §H.5 · p. 30, Table 30 · p. 31

H.6 REDRAWING THE ROUNDING BEATS LATENT SAMPLING, AND THE ROUNDING GRID ITSELF DOES NOT MATTER

• Stochastic-rounding best-of-16 with the draw trusted most by the head beats full-precision sampling with tuned latent noise by 5.7 to 7.1 points on held-out rows.A 0.8 threshold on the head’s acceptance rate classifies all 11 held-out models; the head is a stronger selector than agreement across draws.Source: §H.6 · p. 31, Table 31 · p. 31