PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Looped Transformers Better for (Almost) Free

Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, Ruixiang Zhang

arXiv:2610.02185v1cs.LG

TL;DR

Standard decoding discards decodable intermediate states in looped Transformers. LoopCD contrasts an earlier recurrent prediction with the final one, improving results across four model families and matching or exceeding unguided baselines in selected half-iteration settings with fewer FLOPs.

  • Problem

    Standard decoding uses only the final recurrent state, although earlier loop states can also be decoded into valid next-token distributions.

    Source: Introduction · p. 2

  • Method

    LoopCD is training-free contrastive decoding that guides final predictions away from an earlier recurrent state in hidden or logit space.

    Source: Introduction · p. 2, LoopCD: Contrastive Decoding from… · p. 3

  • Results

    Across four model families, LoopCD improves performance across reasoning, code, and multiple-choice tasks; with half the iterations, it matches or exceeds deeper unguided accuracy in six settings with 22.5% to 48.2% fewer FLOPs.

    Source: Introduction · p. 2, Conclusion · p. 12

  • Takeaways & Limitations

    The results show that recurrent states can provide internal weak-and-strong prediction guidance without auxiliary models or external training.

    Source: Introduction · p. 2, Conclusion · p. 12

    FLOP counts represent theoretical arithmetic workload rather than end-to-end wall-clock speedups, and reduced-depth experiments cover only the multiple-choice suite.

    Source: FLOP accounting · p. 20

Abstract

from arXiv · show

Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.

Introduction

LoopCD uses intermediate recurrent states as aligned weaker references for final predictions, enabling training-free contrastive decoding. Its logit and hidden variants improve benchmark performance and can preserve full-depth baseline accuracy at half depth with fewer forward FLOPs.

Introduction

  • Each recurrent pass yields a decodable state, and the trajectory provides an aligned weak-to-strong prediction pair for the same prefix without auxiliary models or modified prompts.LoopCD contrasts the first iteration with the final state, requiring no additional training or recurrent loops.Source: Introduction · p. 2, LoopCD: Contrastive Decoding from… · p. 3

LoopCD: Contrastive Decoding from Recurrent States

  • LoopCD-Logits contrasts predictions after the output layers with one extra output pass, while LoopCD-Hidden contrasts states before those layers with no extra output overhead.The method also offers fixed guidance strength and token-level adaptive guidance scaled by prediction uncertainty.Source: Introduction · p. 2, LoopCD: Contrastive Decoding from… · p. 4

Introduction

  • At half recurrent depth, LoopCD matches or exceeds unguided full-depth accuracy across evaluated model families while reducing forward FLOPs by 22.5% to 48.2%.The compute reduction follows from halving recurrent iterations while retaining baseline-level performance.Source: Introduction · p. 2
  • At full depth, LoopCD improves four model families; LoopCD-Logits raises Ouro-2.6B-Thinking AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden raises Huginn HumanEval pass@1 from 22.56% to 31.71%.The evaluated tasks span mathematical reasoning, code generation, and multiple-choice benchmarks.Source: Introduction · p. 2, Table 1 · p. 5

Experiments

Across four looped Transformer families, LoopCD improves full-depth predictions on reasoning, code, and multiple-choice tasks, and at half depth can match or exceed full-depth unguided performance while reducing forward FLOPs.

Gains on Mathematical Reasoning and Code Generation

  • 61.88% to 73.33%: adaptive LoopCD-Logits raises Ouro-2.6B-Thinking’s AIME 2024 pass@1, with gains also on AIME 2025 and OlympiadBench.It also improves pass@1 by 5.27 to 7.33 points and pass@10 by 2.53 to 4.12 points across evaluated benchmarks; adaptive guidance achieves the highest mean accuracy.Source: Gains on Mathematical Reasoning and… · pp. 5–6
  • 22.56% to 31.71%: LoopCD-Hidden raises Huginn HumanEval pass@1, while adaptive LoopCD-Logits raises it from 23.17% to 28.66% in another reported configuration.Combining hidden states before the output layers adds no extra output pass; the hidden method also improves every code column at both evaluated depths.Source: Gains on Mathematical Reasoning and… · p. 6, Consistent Improvements Across… · p. 7, Table 3 · p. 7

Consistent Improvements Across Architectures and Suites

  • Every evaluated model improves its seven-benchmark mean under LoopCD-Logits, with adaptive strength leading across nearly all configurations.Improvements hold across Ouro, Huginn, Parcae, and Looped-Qwen3 despite their different recurrence training methods, diverse prompting formats, and generative chain-of-thought evaluation on MMLU-Pro.Source: Consistent Improvements Across… · pp. 6–7, Table 2 · p. 6

Fewer Iterations, Matching or Superior Performance

  • At half depth, LoopCD adds 0.75 to 1.29 points to the six evaluated multiple-choice settings, matching or surpassing full-depth unguided baselines.Huginn-0125 at 16 of 32 iterations exceeds its full-depth baseline by 1.02 points with LoopCD-Logits and 0.51 with LoopCD-Hidden; Looped-Qwen3 matches its baseline under LoopCD-Hidden.Source: Fewer Iterations, Matching or Superior… · p. 7
  • 22.5% to 48.2% fewer forward FLOPs: reduced-depth LoopCD matches or exceeds full-depth unguided accuracy.Guided runs at half depth use 0.52 to 0.78 of full-depth unguided forward FLOPs, accounting for LoopCD-Logits’ extra post-loop readout pass.Source: Fewer Iterations, Matching or Superior… · p. 7

How and Why LoopCD Works

LoopCD gains are strongest when an early recurrent prediction disagrees with the final one, and its re-ranking mainly changes uncertain decisions. Guidance strength also behaves differently in scoring and generation, while margin-based adaptation helps stabilize decoding.

Why guidance works

  • Huginn’s first step disagrees with its final prediction on 51% of ARC-Challenge questions and yields 3.07 points, versus 7% disagreement and 0.17 points at its sixteenth step.The divergence from the final prediction also falls with depth: 3.0 nats to 0.04 for Huginn and 0.71 bits to 0.013 for Ouro-1.4B.Source: How and Why LoopCD Works · p. 9, Figure 6 · p. 9
  • LoopCD-Logits gains most from the first recurrent state, while Huginn’s LoopCD-Hidden reference peaks at the sixth step after its early noise degrades hidden contrasts.Parcae and Looped-Qwen3 start from deterministic representations and effectively use their first state in both guidance spaces.Source: How and Why LoopCD Works · pp. 9–10, Figure 7 · p. 9

5.2 Re-ranking Close Decisions Under Uncertainty

  • On HellaSwag, orthogonal re-ranking alone gains +1.72 points versus +0.76 for Ouro-1.4B and +2.27 versus +1.56 for Ouro-2.6B.The parallel contrast component acts as temperature scaling without changing token order; the orthogonal component changes relative token distances and drives top-token re-ranking.Source: How and Why LoopCD Works · p. 10, Figure 8 · p. 10
  • On the least-confident fifth of ARC-Challenge questions, guidance adds +6.4 to +13.3 points across four models, versus at most +0.4 on the most-confident fifth.Only 5.8% to 14.9% of answers flip, and the net gain of 2.1 to 3.5 points comes from resolving close decisions.Source: How and Why LoopCD Works · p. 10, Figure 8 · p. 10

5.3 Tuning Guidance Strength Across Scoring and Generation

  • Multiple-choice scoring tolerates a broad guidance-strength band, but autoregressive generation needs smaller values to prevent compounding errors; weaker references degrade accuracy in both task families.The scoring gains peak near an omitted strength value, while the passage does not provide the generation window’s numeric bounds.Source: How and Why LoopCD Works · pp. 10–11, Figure 9 · p. 11
  • Adaptive margin gating applies maximum guidance to contested tokens and withholds updates on settled decisions, preserving positive gains across caps where fixed strength overshoots.The rule scales guidance using the top-two probability margin.Source: How and Why LoopCD Works · p. 11, Figure 10 · p. 11

Conclusion

LoopCD uses intermediate recurrent states as weak predictions to guide the final prediction, with one extra output pass or none. It improves decoding quality and can reduce recurrent compute while matching or exceeding deeper unguided models.

Conclusion

  • Ouro’s mean pass@1 on mathematical reasoning rises by five to seven points at the same depth, while multiple-choice and generation suite means improve across four families.With half the iterations, LoopCD matches or exceeds the deeper unguided model in six settings while using 22.5% to 48.2% fewer FLOPs.Source: Conclusion · p. 12
  • LoopCD contrasts weak and strong predictions from states produced by the same layers for the same token, which ordinary decoding leaves unused.It continues the change from the earlier prediction through the final one at the cost of one extra output pass or none.Source: Conclusion · p. 12
  • Recurrence settles a prediction early, and the first iteration records the direction of its movement for re-ranking close decisions toward the option recurrence was raising.The passage suggests models such as Ouro, trained with their intermediate predictions in mind, may make the weak-and-strong prediction pair more useful.Source: Conclusion · p. 12

LOOPCD References

The supplied material lists references and notes appendix expansions to Sections 2 and 3 on the looped-transformer setup and LoopCD.

LOOPCD References

Looped Transformers and LoopCD

  • The appendix expands Sections 2–3 with state indexing and the Eq. (3) output interface, descriptions of four evaluated families, and LoopCD-Logits’ probability form.Source: Looped Transformers and LoopCD · p. 15

States and coda layers · Model architectures

LoopCD uses aligned recurrent states from complete iterations as guidance through the model’s existing output interface. Its logit and hidden-state variants accommodate different coda architectures and can also be applied to damped frozen-window recurrence.

States and coda layers

  • LoopCD-Logits sends earlier and final states through the same output interface and combines their logits; LoopCD-Hidden combines states first and applies that interface once.In both cases, modified logits feed the evaluation’s existing scoring or token-selection rule.Source: States and coda layers · p. 15

Model architectures

  • Ouro repeats its decoder stack and trains predictions at intermediate depths; reported Ouro experiments use logits from the earlier state after final normalization and the vocabulary head (Zhu et al., 2025a).Ouro has no prelude or coda Transformer layers; a hidden-state update could precede normalization and the head.Source: Model architectures · p. 15
  • Huginn initializes a separate recurrent state with Gaussian noise, conditions updates on its prelude representation, and combines recurrent outputs before the coda and head for LoopCD-Hidden (Geiping et al., 2025).Earlier logits instead require a pass through the coda and head.Source: Model architectures · p. 15
  • Parcae conditions its separately initialized recurrent state on a normalized prelude and projects recurrent outputs before the coda; its hidden-state form combines states before projection (Prairie et al., 2026).LoopCD-Logits applies the projection, coda, and head to the earlier state, while the hidden-state form uses the coda layers once.Source: Detailed Experimental Setup · p. 16

Detailed Experimental Setup

  • Training-Free Looped Transformers repeatedly apply a frozen window with damped residual updates; the evaluated window has four layers, with fifteen before it and seventeen after it (Chen et al., 2026).Earlier logits require the frozen tail and head; the reduced-depth result uses the first four of eight substeps, not a reconfigured four-substep wrapper.Source: Detailed Experimental Setup · p. 16
  • For LoopCD-Logits, normalizing the guided logits yields the probability form for the selected coefficient at each token position, including under the adaptive rule.This identity applies before evaluation temperature or truncation and does not directly extend to LoopCD-Hidden when the coda is nonlinear.Source: Detailed Experimental Setup · p. 16
  • LoopCD and unguided runs share checkpoints, prompts, generation and evaluation settings, with recurrent depths matched except in reduced-budget ablations.Reduced-budget comparisons use repeated standard-budget logit baselines where available and dedicated baselines for hidden-state runs; guided-score subscripts denote percentage-point changes from matched baselines.Source: Detailed Experimental Setup · p. 16
  • Evaluations cover likelihood-scored multiple-choice tasks, greedy generation and code benchmarks, and mathematical reasoning measured with pass@1 and pass@10.The math suite includes AIME 2024, AIME 2025, and OlympiadBench; AIME estimates use 16 sampled solutions per problem.Source: Detailed Experimental Setup · pp. 16–17

Prompts and sampling … Reasoning metrics: pass@1 and pass@10 across sampled trajectories

LoopCD-Logits usually improves MMLU-Pro but has mixed GSM8K results, while reasoning pass@1 and pass@10 can move in opposite directions. Evaluation prompts and guidance strengths vary across benchmark and model configurations.

Prompts and sampling

  • Prompting uses zero-shot code benchmarks, five-shot CoT MMLU-Pro, and zero-shot mathematical reasoning; GSM8K uses three shots except eight-shot GSM8K-CoT and hidden-state generation runs.Tables A2 and A3 document configurations for multiple-choice scoring and generation.Source: Prompts and sampling · p. 17, Table A2 · p. 17, Table A3 · p. 17

Strengths and reference states

  • Fixed and adaptive strengths are evaluated for regular-depth logit-space setups, while hidden-state and reduced-depth experiments use fixed strength alone.Hyperparameters are selected through screening sweeps, with standard-depth values recorded in Table A4.Source: Strengths and reference states · p. 17, Table A4 · p. 18

Extended Results in Full Recurrent Depth

  • The full-depth appendix adds LoopCD-Logits GSM8K and MMLU-Pro results and Looped-Qwen3 reasoning metrics; LoopCD-Hidden reference states and strengths are specified separately.The hidden-state settings are specified in Appendix B.4.Source: Extended Results in Full Recurrent Depth · p. 18

GSM8K and MMLU-Pro under LoopCD-Logits

Reasoning metrics: pass@1 and pass@10 across sampled trajectories

  • Reasoning gains are an order of magnitude larger than multiple-choice changes, plausibly because guidance can shift hundreds of sequential token decisions and pass@ rewards success among samples.For Ouro-Thinking models, both pass@1 and pass@10 rise across nearly every configuration; aggregate pass@ scores do not directly measure trajectory diversity.Source: Reasoning metrics: pass@1 and pass@10… · pp. 18–19
  • On Looped-Qwen3 AIME 2024, fixed-strength guidance lowers pass@1 from 64.79 to 61.88 but raises pass@10 from 82.91 to 86.17.This is the only mathematical-reasoning block where pass@1 declines while pass@10 rises.Source: Reasoning metrics: pass@1 and pass@10… · p. 18, Table A6 · p. 19

Computation and Cost Analysis

  • The computation appendix covers FLOP accounting for both guidance forms, reduced-depth performance recovery and compute tradeoffs, and representational differences between LoopCD-Logits and LoopCD-Hidden.The passage describes the appendix’s scope without reporting specific values.Source: Computation and Cost Analysis · p. 19

FLOP accounting · Reduced-depth evaluation: performance recovery and computational tradeoffs

Halving recurrent depth weakens unguided performance, but LoopCD recovers the deficit across evaluated configurations; its compute tradeoff depends on whether guidance adds post-loop readout work or only lightweight hidden-state operations.

FLOP accounting

  • LoopCD-Hidden’s lightweight vector operations and single standard coda pass add negligible theoretical FLOPs, whereas LoopCD-Logits adds architecture-dependent post-loop work.FLOPs are estimated for a 512-token prefill at two FLOPs per multiply-and-accumulate operation; they do not represent wall-clock speedups.Source: FLOP accounting · pp. 19–20

Reduced-depth evaluation: performance recovery and computational tradeoffs

  • LoopCD recovers the performance deficit caused by halving recurrent iterations across all evaluated configurations.These reduced-depth experiments use the multiple-choice suite.Source: Reduced-depth evaluation: performance… · p. 20, FLOP accounting · p. 20
  • Across six half-depth settings, guidance-inclusive compute is about one-half to three-quarters of the full-depth forward pass.Savings are bounded by the recurrent loop’s share of computation, while LoopCD-Logits overhead comes from post-loop layers; for Looped-Qwen3, that overhead exceeds loop savings, making LoopCD-Hidden necessary for effective FLOP reduction.Source: Reduced-depth evaluation: performance… · p. 21, Table A9 · p. 22
  • On Parcae-1.3B, LoopCD-Hidden retains roughly sixty percent of LoopCD-Logits’ gain (0.85 versus 1.38), while on Ouro it frequently outperforms logits.Hidden-state retention tracks coda depth: Huginn’s shallow two-layer coda yields nearly identical gains to fixed logits on the seven-benchmark mean.Source: Reduced-depth evaluation: performance… · p. 21
  • 5.13 versus 3.97 points on the four-column code-generation mean, LoopCD-Hidden exceeds fixed LoopCD-Logits at one Huginn depth; at the other, it scores 3.76 versus 3.04 points.The comparison covers both evaluated Huginn depths against their respective baselines.Source: Figure A2 · p. 22, Reduced-depth evaluation: performance… · p. 23
  • At identical guidance strengths, hidden-state guidance moves Ouro’s output distribution more than logits, but moves it less for Parcae and Huginn.Ouro’s optimal hidden-state strengths are lower than its logit strengths: 0.25 versus 0.5 for Ouro-1.4B and 0.5 versus 1 for Ouro-2.6B.Source: Reduced-depth evaluation: performance… · p. 23
  • Reference guidance gain tracks disagreement with the final prediction rather than standalone accuracy, making the first iteration the optimal logit reference across stable architectures.Parcae-1.3B’s competent sixth step yields only one point of gain after converging toward the final state.Source: Reduced-depth evaluation: performance… · p. 24
  • Huginn’s noisy first hidden-state reference is detrimental, although its first logits form an excellent reference because the coda and head project away the initial noise.Huginn’s hidden-state guidance instead peaks at the sixth state; the initial noise-removal step dominates the first-to-final hidden-state contrast.Source: Reduced-depth evaluation: performance… · pp. 24–25

Geometric convergence in representation space

Standardized cosine similarity tracks how intermediate representations approach the final state. Ouro shows sharp drops at recurrent-iteration boundaries, while Looped-Qwen3 moves toward the final representation more smoothly.

Geometric convergence in representation space

  • The analysis compares intermediate and final states using cosine similarity after standardizing each hidden dimension over scored positions and unrolled layers.Standardization reduces distortion from dimensions with excessively large variances.Source: Geometric convergence in… · p. 25
  • Ouro’s similarity to the final state drops sharply at pass boundaries and recovers within each pass, while Looped-Qwen3 climbs more smoothly.Figure A6 shows this contrast across ten configurations.Source: Geometric convergence in… · p. 25, Figure A5 · p. 26, Figure A6 · p. 26

Intra-iteration dynamics and output convergence

Within-pass predictions drift sharply, making internal layers poor guidance references, while completed recurrent iterations converge to distributions that retain ambiguity. The appendix characterizes LoopCD-Hidden as orthogonal re-ranking, with a parallel update component responsible for temperature sharpening.

Intra-iteration dynamics and output convergence

  • Ouro-1.4B pass-end JSDs are 0.71, 0.13, and 0.013 bits (Ouro-2.6B: 0.56, 0.06, and 0.008), while within-pass divergence averages 0.57–0.84 bits.The figure measures mean JSD between each unrolled layer's prediction and the final prediction on ARC-Challenge.Source: Figure A7 · p. 27, Intra-iteration dynamics and output… · p. 27
  • Huginn’s median next-token entropy settles at 2.7 nats by iteration 16 and stays flat through 32, yet 67% of positions remain above 2 nats at iteration 32.Halving Huginn’s iterations avoids accuracy penalties, and the converged distribution retains meaningful ambiguity—roughly an even choice among seven tokens.Source: Figure A7 · p. 27, Intra-iteration dynamics and output… · pp. 27–28
  • On ARC-Challenge, internal-layer references lower accuracy by up to 20.6 points for Ouro-1.4B and 24.9 for Ouro-2.6B, while completed-iteration references raise it by 2.1 and 2.6 points.The tested internal layers were 0, 8, 16, and 22/46 of the final pass; unlike dense networks, looped architectures lack trained exit points within a pass.Source: Intra-iteration dynamics and output… · p. 27

Re-ranking Close Decisions

  • The appendix decomposes LoopCD-Hidden’s logit update as orthogonal re-ranking, with the parallel component dictating temperature sharpening.It also reports that guidance gains concentrate on the model’s most uncertain decisions and presents a step-by-step case study.Source: Re-ranking Close Decisions · p. 28

Concentration of guidance gains on uncertain decisions · Geometric equivalence of LoopCD-Hidden and orthogonal re-ranking

Guidance gains concentrate on low-confidence decisions, while LoopCD-Hidden largely reproduces the orthogonal re-ranking behavior of LoopCD-Logits. The alignment varies by architecture and reference state, with initialization noise weakening it for Huginn’s first state.

Concentration of guidance gains on uncertain decisions

  • Guidance acts as a tie-breaker, changing a final choice only when its contrast exceeds the probability margin between the top two options.Confident predictions remain unperturbed, while undecided predictions can shift.Source: Concentration of guidance gains on… · p. 28
  • +3.7 points for Ouro-1.4B and +7.3 for Ouro-2.6B come from the least confident fifth; no other fifth contributes more than +0.5, and guidance flips 3.9%–14.9% of answers overall.The most confident fifth sees virtually zero impact; net gains reflect more fixed answers than broken ones on close decisions.Source: Table A12 · p. 28, Concentration of guidance gains on… · p. 28

Geometric equivalence of LoopCD-Hidden and orthogonal re-ranking

  • In architectures where the entire stack loops, normalization discards hidden-update magnitude before the vocabulary head, so LoopCD-Hidden’s directional shift acts like the orthogonal re-ranking component of LoopCD-Logits.A logit contrast decomposes into a component parallel to final logits, which rescales the distribution like temperature, and an orthogonal component that re-ranks tokens.Source: Geometric equivalence of LoopCD-Hidden… · pp. 28–29

Concentration of guidance gains on uncertain decisions

  • In one ARC-Challenge example, tornado’s score rose +0.90 versus earthquake’s +0.32 across early passes, but guidance overturned earthquake’s final lead of less than 0.01.The passage describes guidance as extending momentum already present in the recurrent states, not introducing external knowledge.Source: Concentration of guidance gains on… · p. 28, Table A13 · p. 29

Geometric equivalence of LoopCD-Hidden and orthogonal re-ranking

Guidance Strength Dynamics

  • For Huginn, LoopCD-Hidden’s update has 0.89 cosine similarity with re-ranking when using the optimal sixth state as reference, but 0.62 when forced to use the first state.The first hidden state is dominated by initialization noise, producing random rather than coherent directional shifts.Source: Guidance Strength Dynamics · p. 30

The double-edged effect of temperature sharpening

LoopCD-Logits’ temperature sharpening helps on some benchmarks but hurts on others, making its advantage over LoopCD-Hidden dataset-dependent. Guidance also has narrow useful bounds: stronger updates reach more decisions but can overshoot close predictions.

The double-edged effect of temperature sharpening

  • LoopCD-Hidden excels on HellaSwag, while LoopCD-Logits leads on ARC-Challenge, ARC-Easy, SciQ, and MMLU.On HellaSwag, the full logit update for Ouro-1.4B gains +0.76 points, versus +1.72 for its re-ranking component and +1.73 for LoopCD-Hidden; on ARC-Challenge, re-ranking peaks at +2.65 points and the full logit update at +2.47. The choice typically sways net accuracy by one or two percentage points.Source: The double-edged effect of temperature… · p. 30
  • Generation benchmarks show a narrow positive-guidance window, with catastrophic degradation beyond the beneficial range.On GSM8K, Ouro-1.4B loses 8 points and then 20 points at the reported respective strengths; HumanEval declines by 10 and 22 points at those same strengths.Source: The double-edged effect of temperature… · p. 30, Figure A9 · p. 31
  • Negative guidance degrades the seven-benchmark multiple-choice mean across all models at every tested value, converging to reference accuracy at zero.This backward interpolation along the contrast vector supports the interpretation that the vector encodes the direction of model improvement.Source: The double-edged effect of temperature… · p. 30
  • Larger guidance strengths change more answers, but moderate strengths best balance gains on uncertain questions against overshooting close predictions.Gains first concentrate in the least-confident fifth, then spread into more confident fifths; at higher strength, accuracy in the least-confident fifth recedes as corrected predictions flip back into errors. Aggregate scores peak between moderate strengths, motivating adaptive gating that withholds guidance on settled predictions.Source: The double-edged effect of temperature… · p. 31, Related Work · p. 32

Related Work

  • LoopCD differs from training-time recurrence modifications by using temporal contrasts between early and final passes of the same recurrent block at inference.This temporal reference is a valid, fully projected prediction sharing the final state’s vocabulary and feature space; spatial-layer contrast can fail in recurrent architectures.Source: Related Work · p. 32

Related papers

Loading 2610.02185v1…