PaperlayerView exact revision on arXiv

Source-linked AI summary

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu, Junyu Wang, Zhizheng Wu

arXiv:2609.03379v2cs.LG

TL;DR

Latent looping adds inference depth without extra tokens, but earlier loop states are inaccessible and fixed loop counts allocate depth uniformly. RecurTrace adds loop-time memory and oracle-distilled halting; it matches or exceeds fixed depth across six tasks and improves same-budget accuracy from 0.6B to 8B.

  • Problem

    Latent looping adds effective depth without extra tokens, but fixed loop counts allocate depth uniformly and iterations lack direct access to earlier loop states.

    Source: §1 · pp. 1–2

  • Method

    RecurTrace combines Loop Memory Attention over earlier same-position loop states with an oracle-distilled halting head trained to continue when deeper loops lower loss.

    Source: §1 · p. 2

  • Results

    RecurTrace matches or exceeds the best fixed depth across six tasks and improves accuracy over same-budget baselines at every scale from 0.6B to 8B.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The loop trajectory can support both memory and adaptive computation, with chain-of-thought decoding a natural extension.

    Source: §6 · p. 7

    The study uses frozen Qwen3 decoder-only models and direct-answer targets; halting is sequence-level and memory covers same-position states within a three-loop window.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.

1 Introduction

RecurTrace addresses latent loops’ limited access to earlier computations and fixed compute budgets with loop-time memory and oracle-distilled, input-adaptive halting. It reports stronger MathQA accuracy at roughly two loops and accuracy gains over same-budget baselines across model scales.

1 Introduction

  • A looped block carries only its latest hidden state forward, while fixed loop counts give easy and hard questions the same inference depth.On MathQA, accuracy peaks at two loops, stays near 53.6% at the eight-loop training clamp, and falls to 47.5% at sixteen loops.Source: §1 · p. 1
  • The halting head uses oracle supervision for whether a deeper loop still lowers loss, rather than a compute penalty or confidence rule.The oracle records when the answer stops improving, and the model halts per input.Source: §1 · p. 2
  • 56.9% accuracy at 2.0 loops improves on the best fixed loop budget by 2.2 points at comparable mean depth.The figure reports 56.92% at approximately 2 loops, 1.25 points above TaH-Mismatch.Source: Figure 1 · p. 1, §1 · p. 2
  • Across 0.6B, 1.7B, 4B, and 8B, RecurTrace raises generation accuracy over same-budget baselines, with the gain reaching 3.4 points at 8B.Teacher-forced loss also falls at all four scales; added modules hold at most about 2.2% of parameters, and base weights stay frozen.Source: §1 · p. 2
  • Loop Memory Attention lets each token attend to its own states from earlier loops, adding a loop-time memory without an external store.The looped block remains weight-tied.Source: §1 · p. 2

2 Related Work

Related work explores recurrent depth, external memory, and adaptive computation, while RecurTrace combines access to a loop’s history with sequence-level oracle-supervised halting. The figure summarizes its encode–think–decode architecture.

2 Related Work

  • Plain-looping methods pass only the previous output, whereas RecurTrace adds attention to intermediate states and grafts it onto a frozen dense model.Dreamer also attends along depth, but uses a recurrent sparse-expert architecture.Source: §2 · p. 2
  • External stores such as Frey et al. (2026) and MeSH (Yu et al. 2025) add memory outside the recurrence, while RecurTrace queries the trajectory already produced.The concurrent LoopUS uses a confidence rule and keeps no addressable history.Source: §2 · p. 2
  • Adaptive-computation approaches use priors, penalties, confidence, or token-level routing; RecurTrace instead halts per sequence using an oracle for whether deeper looping lowers loss.Think-at-Hard iterates tokens predicted wrong after one pass, unlike RecurTrace’s sequence-level depth choice.Source: §2 · p. 2
  • Latent looping scales per-token computation without emitting tokens, unlike test-time approaches that spend extra decoding compute.The related-work discussion frames RecurTrace as asking how many loops to use and when to stop.Source: §2 · p. 2
  • Figure 2 depicts a frozen encoder, a variable-depth weight-tied block with loop-time memory, and a frozen decoder after halting.The block re-injects its input after the first loop and uses a three-loop sliding window.Source: Figure 2 · p. 3

3 Method

RecurTrace repeats a compact middle-layer block between frozen encoder and decoder, using loop-time memory and variable-depth training to support adaptive inference. The selected block and oracle-distilled halting head determine where and when computation repeats.

3 Method

  • The model encodes through layers before a contiguous middle block, repeats that weight-tied block, then decodes through the remaining layers.The design follows an encode–think–decode view.Source: §3 · p. 3
  • Behavioral ablation selects layers 12–14 for the 0.6B and 1.7B bases and layers 15–17 for the 4B and 8B bases.The authors first locate a broad middle span whose representations move least between adjacent layers; probe-core and late-block controls are comparable or weaker.Source: §3 · p. 3
  • After the first loop, the block re-injects its input and adds each looped layer’s memory term before running the unchanged layer.Input injection starts as a no-op because its scalar α is initialized to zero.Source: §3 · p. 3
  • Loop Memory Attention reads each layer’s own same-position states from a sliding window of earlier loops; the memory is empty on the first loop.Attention runs along loop time without mixing token positions, and its gated path starts near closed; the window caps memory cost.Source: §3 · pp. 3–4
  • Sampling loop depth during training lets one checkpoint serve multiple test-time depths and extrapolate beyond training depths.The sampled depth has mean near 3.9, with about a fifth of passes running six or more loops.Source: §3 · p. 4
  • A halting head predicts whether to continue from pooled loop state, using oracle labels for when a deeper loop helps; inference stops at its threshold or the loop budget.The threshold is selected on held-out data as the smallest mean depth that does not lose likelihood, with minimum and maximum loop bounds.Source: §3 · p. 4

4 Experiments

Experiments test adaptive halting on MathQA, transfer across reasoning tasks, and scaling across four model sizes. RecurTrace beats the best fixed depth at comparable compute and improves over same-budget baselines across model sizes and benchmark families.

4.1 Experimental Setup

  • The 1.7B MathQA halting comparison evaluates 2,400 unique items across eight non-overlapping test seeds and three training seeds, with thresholds selected on held-out data.Across scales, the base weights are frozen and only loop, memory, injection, and halting parameters are trained.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Learned Halting on MathQA

  • 56.9% accuracy at 2.04 loops beats fixed T=2 by 2.2 points at matched compute and exceeds LoopUS-Conf (55.3%, 3.2 loops) and TaH-Mismatch (55.7%, 2.1 loops).The comparison is on 1.7B MathQA; Table 2 reports three-training-seed means.Source: §4.2 · pp. 4–5, Table 2 · p. 5
  • ACT and PonderNet stay at one loop, while CALM reaches 54.1% accuracy at 5.6 loops; RecurTrace’s two-loop floor prevents collapse and its learned head adds 53 correct answers.Without the floor, the head mostly stops after one loop; the floor guarantees depth while the head allocates it.Source: §4.2 · p. 5
  • At the illustrative τ=0.10 operating point, hard MathQA items average 3.10 loops versus 2.50 for easy items, with a heavier tail despite both groups peaking at two.The hard–easy mean gap remains positive at the main τ=0.50 operating point, where 97.3% of items stop at the floor.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Adaptive halting matches or exceeds each task’s best fixed depth using 1.1–2.8 loops on average, and cuts mean loops by 65–87%.Each task uses its own halting head and held-out threshold; the per-task heads run without a two-loop floor.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Scaling and Breadth

  • Generation accuracy gains over same-budget baselines rise from 0.6 points at 0.6B to 3.4 points at 8B, while teacher-forced NLL falls at all four scales.The NLL reductions are 0.15, 0.08, 0.02, and 0.01 nats from 0.6B through 8B.Source: §4.3 · p. 6, Figure 4 · p. 6
  • At 1.7B, accuracy rises by 0.8 points on train-covered tasks and by 1.9 points on classic tasks; NLL decreases for both benchmark families.The classic-task accuracy comparison is 64.5% to 66.4%.Source: §4.3 · p. 6, Table 4 · p. 6

5 Ablations and Analysis

The analysis separates gains from loop depth, memory, and halting, and examines when memory helps. It finds that earlier-state access makes additional iterations more reliable, while the experiments remain bounded to specific architectures, memory span, and output settings.

5.1 What Drives the Gain

  • At 1.7B, Loop Memory Attention improves accuracy over plain looping by 2.3 points on MathQA and 2.4 points on the classic suite; it also reverses NLL regressions at 1.7B, 4B, and 8B.Plain looping raises NLL by 0.07, 0.05, and 0.06 nats at those scales, while memory lowers it by 0.03, 0.03, and 0.02 nats.Source: §5.1 · p. 6, Table 5 · p. 6
  • Memory’s benefit depends on task structure: plain looping leads by 4.7 points on pointer chasing, while memory leads by 17.7 on symbolic state tracking and 16.7 on arithmetic.The token-wise gate suppresses memory for lookup and activates it when intermediate state must persist.Source: §5.1 · pp. 6–7, Table 5 · p. 6
  • A matched-size non-looped adapter raises accuracy from 44.8% to 45.4% and lowers 22-task NLL by 0.01 nats, versus RecurTrace’s 1.2-point gain and 0.08-nat reduction.The comparison indicates that added capacity alone does not account for the larger gains.Source: §5.1 · p. 7
  • At 8B, adding memory to the same plain-looping checkpoint lowers teacher-forced NLL by 0.03–0.04 nats across both seeds, separating the gain from additional optimization.The continuation uses an equal number of training steps with and without Loop Memory Attention.Source: §5.1 · p. 7
  • Removing QK-normalization causes the largest degradation; freezing loop-distance bias or initializing gates open also hurts, while W=3 performs best among tested windows.The ablations suggest learning how to weight earlier states and introducing memory gradually are useful design choices.Source: §5.1 · p. 7, Table 6 · p. 7

5.2 Halting Behavior

  • ACT and PonderNet collapse to one loop under penalty-based halting on the frozen backbone; LoopUS-Conf uses 3.2 loops for 55.3%, while TaH-Mismatch reaches 55.7% at 2.1 loops.The authors distinguish this post-hoc frozen-backbone result from the end-to-end setting for which ACT and PonderNet were designed.Source: §5.2 · p. 7
  • Sampling loop counts during training lets one checkpoint serve multiple inference depths; its loop-distance bias remains defined beyond the eight-loop training range.At sixteen loops, accuracy is 47.5%, below the two-loop peak of 54.7%.Source: §5.2 · p. 7

5.3 Limitations

  • The study uses sequence-level halting and same-position memory over a fixed three-loop window, on frozen Qwen3 decoder-only models with direct-answer targets.Other architectures and longform generation remain untested; latent looping and chain-of-thought are described as complementary, not direct substitutes.Source: §5.3 · p. 7

6 Conclusion

RecurTrace combines memory over earlier loop states with input-specific adaptive halting. It matches or exceeds the best fixed depth across six tasks and improves accuracy over same-budget baselines from 0.6B to 8B.

6 Conclusion

  • RecurTrace matches or exceeds the best fixed depth across six tasks and improves accuracy over same-budget baselines at every scale from 0.6B to 8B.The results suggest the loop trajectory can support both memory and adaptive computation, with chain-of-thought decoding named as a possible extension.Source: §6 · p. 7
  • Loop Memory Attention lets each iteration access earlier loop states, while an oracle-distilled halting head selects the loop count for each input.Together, they turn a pretrained LLM into a memory-augmented looped reasoner with adaptive halting.Source: §6 · p. 7

A Implementation Details

The appendix documents RecurTrace’s model configuration, two-stage training, compute accounting, and evaluation data, including synthetic tasks designed to test stateful reasoning and extrapolation.

A Implementation Details

  • RecurTrace uses four pretrained Qwen3 bases, loops three contiguous layers, and trains added modules while leaving base weights frozen.The 0.6B and 1.7B models loop layers 12–14; the 4B and 8B models loop layers 15–17.Source: A Implementation Details · p. 10
  • Stage 1 trains Loop Memory Attention and input-injection scalars under a matched same-budget protocol; Stage 2 freezes those modules and distills the halting head.The baseline fine-tunes the full base model, while RecurTrace trains only its added modules in Stage 1.Source: A Implementation Details · p. 10
  • At 8B, analytical training FLOPs are comparable and no larger for RecurTrace than the baseline, though the accounting is not a wall-clock measurement.The halting-head distillation is a RecurTrace-only Stage 2 cost outside the matched budget; the reported mismatch favors the baseline.Source: A Implementation Details · p. 10, Table S1 · p. 10
  • The reasoning mixture contains about 1.15 million direct-answer examples, and its overlap audit removes 327 near-duplicates, or 0.03%, with zero exact matches.The audit uses exact normalized matching and a 13-gram Jaccard threshold above 0.85; training data are fixed across methods.Source: A Implementation Details · p. 11
  • Three synthetic generators contribute about 150 thousand training items each, with held-out symbol pools for the symbol-based tasks and fresh test problems for arithmetic.Training difficulty samples step counts from 2 to 8; test buckets extend to 10–16 steps, with 2,000 examples in the extrapolation subset.Source: A Implementation Details · p. 11
  • Pointer chasing is a stateless lookup, whereas symbolic state tracking and arithmetic require carrying intermediate values across steps.The latter two tasks are the generators identified as rewarding access to earlier loop states.Source: A Implementation Details · p. 11

B Block Selection

Block selection first locates a reasoning-critical middle span with an angular-distance probe, then compares compact three-layer windows by downstream behavior.

B Block Selection

  • The angular-distance probe identifies a middle span where adjacent-layer distances flatten, using Kneedle boundaries and downstream comparisons to choose a compact loop window.It reads the final-token residual stream and averages adjacent-layer distances across about ten thousand inputs.Source: B Block Selection · p. 11
  • Table S2 records loop, memory, and halting defaults across scales, including ACT’s expected-depth cost and PonderNet’s geometric prior settings.The worked-example table distinguishes pointer chasing from symbolic state tracking and arithmetic, which require carrying intermediate values.Source: Table S2 · p. 12, Table S3 · p. 12
  • Both 28-layer Qwen3 bases use layers 12–14, selected as the compact block after the probe placed the encoder in the first eleven layers.The chosen layers sit 15 to 25 percent below their neighbors in angular distance, with no sufficiently separated fourth layer.Source: B Block Selection · p. 12
  • At 4B and 8B, layers 15–17 are selected over probe-core or late controls based on likelihood and generation comparisons.At 4B, the probe-core window ties the selected window; at 8B, layers 15–17 are strongest on both measures, while late layers 27–29 give almost no improvement.Source: B Block Selection · p. 12

C Loop Memory Attention Details

Loop Memory Attention gives each looped layer access to its own earlier states along the loop-time axis, with bounded memory and gated updates.

C Loop Memory Attention Details

  • Each looped layer attends at the same sequence position to its own past-loop states, while ordinary self-attention retains cross-token mixing.This design adds no path for one token to see another and cannot leak future tokens.Source: C Loop Memory Attention Details · p. 12
  • The memory query is the previous loop’s output of the same layer, and a three-loop sliding window keeps the memory span fixed as depth grows.Removing QK-normalization causes the largest degradation in Table 6, indicating that it stabilizes memory training.Source: C Loop Memory Attention Details · p. 12
  • The token-wise memory gate starts near 0.047 and the input-injection scalar starts at zero, keeping updates small or inactive at initialization.Because injection begins only after the first loop, every one-loop run exactly matches the frozen base computation.Source: C Loop Memory Attention Details · pp. 12–13
  • With fixed memory window W, memory-attention cost grows as O(seq×loops), and the added modules account for about 1.4–2.2% of parameters across scales.At 8B, the added modules are about 0.12B parameters; measured runtime is about 1.1× baseline at two loops, 1.3× at four, and 1.7× at eight.Source: C Loop Memory Attention Details · p. 13
  • At generation time, one halting decision from the prompt state sets the depth reused for all generated tokens; memory storage remains constant in loop count.The looped layers’ self-attention cache still grows with loop count; at 1.7B and two loops, cache additions are about 15% of the base key-value cache.Source: C Loop Memory Attention Details · p. 13

D Variable-Depth Training

Variable-depth training exposes RecurTrace to loop counts from shallow to deep, while full backpropagation through the loop and synchronized depth across workers support training.

D Variable-Depth Training

  • With a mean near 3.9 loops, the default sampler assigns about 10% of passes to one loop and 8% to eight, with about one-fifth running six or more.The clamped log-normal Poisson sampler uses a mean-four target, σ=0.5, and an eight-loop clamp; its heavy tail exposes the model to deep unrolls.Source: D Variable-Depth Training · p. 13
  • The three-layer block and three-state memory window keep the unrolled graph small enough for full backpropagation through the loop.Training does not use truncated backpropagation.Source: D Variable-Depth Training · p. 13
  • A shared seed lets all eight data-parallel workers sample the same loop count per step without extra communication.The trainer marks parameters unused when a one-loop step does not touch the memory modules, preserving correct gradient synchronization.Source: D Variable-Depth Training · p. 13

E Halting Head and Calibration

The halting head learns from loss-based oracle labels and stops using a held-out threshold. On MathQA, adaptive halting improves on fixed-depth compute, with most items stopping at the two-loop floor.

E Halting Head and Calibration

  • A pooled-state linear head learns continue labels when a deeper loop lowers per-example loss by more than the default 0.01 margin.The head is trained at probe depths {1, 2, 4, 6}; the other parameters are frozen during this second-stage training.Source: E Halting Head and Calibration · p. 13
  • With thresholds selected on held-out data, leave-one-seed-out checks retain a +2.0-point gain, ranging from +0.4 to +3.8 points.The main comparison uses τ=0.50 and spends 2.04 loops on average; no threshold is selected on the test set.Source: E Halting Head and Calibration · pp. 13–14
  • 2.2 points: RecurTrace answers 1366 of 2400 MathQA items versus 1313 for fixed T=2 at matched mean depth, with a 95% bootstrap interval of [+0.8, +3.6] points.The paired comparison favors adaptive halting with p < 0.001 on every training seed.Source: E Halting Head and Calibration · p. 14, Table S6 · p. 15
  • At inference depths 3, 5, and 7, which are not directly supervised, halting decisions stay within 1.2 points of the nearest supervised depth.The calibration table covers held-out MathQA items and reports balanced accuracy and AUROC against oracle continue labels.Source: E Halting Head and Calibration · p. 14, Table S4 · p. 14
  • 97.3% of items stop at the two-loop floor, while the remaining 65 continue and account for the entire accuracy gain; no item reaches depth 16.The mean depth is 2.04 loops.Source: E Halting Head and Calibration · p. 14, Table S5 · p. 14
  • 56 items flip from wrong to right versus 3 from right to wrong; all 56 gains come from items receiving more than two loops.The pattern holds across all three training seeds.Source: E Halting Head and Calibration · p. 14, Table S6 · p. 15
  • About 85% of deepened items are oracle-positive and 86% flip wrong-to-right; the loss oracle marks loss reductions, not greedy-answer correctness.The loss oracle has about 36% recall among oracle-positive items, while the answer-flip rate concerns a distinct outcome.Source: E Halting Head and Calibration · pp. 14–15
  • ACT and PonderNet collapse to 1.0 mean loops across six penalty settings; LoopUS-Conf and TaH-Mismatch test confidence and answer-mismatch supervision instead.The reported collapse is for penalty-based halting fitted post hoc to a frozen looped backbone.Source: E Halting Head and Calibration · p. 15

F Benchmark Breakdown and Diagnostics

Benchmark results compare RecurTrace with same-budget baselines across task suites and model scales, while held-out tasks remain diagnostic rather than part of the main breadth claim.

F Benchmark Breakdown and Diagnostics

  • The evaluation reports generation accuracy and teacher-forced NLL at loop counts {1, 2, 4, 6, 8}, using bucket-stratified sampling.The train-covered suite has fourteen tasks and the classic suite has eight multiple-choice benchmarks.Source: F Benchmark Breakdown and Diagnostics · p. 15
  • At 4B and 8B, same-budget generation accuracy gains are +2.2 and +3.4 points, respectively.At 0.6B and 1.7B, NLL falls and accuracy rises on train-covered tasks and the classic suite; the classic-suite gain at 1.7B is +1.9 points.Source: F Benchmark Breakdown and Diagnostics · p. 15
  • Held-out and BIG-Bench Hard results are diagnostic evidence, not part of the main breadth claim.At 1.7B, gains are +0.3 pp on BIG-Bench Hard subsets and +0.4 pp on remaining held-out tasks.Source: F Benchmark Breakdown and Diagnostics · p. 15

G Difficulty-Adaptation Details

The reported operating point spends most computation at the two-loop floor and directs additional loops disproportionately to hard items. Lower thresholds widen the hard–easy loop gap but can reduce accuracy when the floor is removed.

G Difficulty-Adaptation Details

  • The easy–hard split is defined by correctness at one loop, distinct from the fixed-two-loop reference used for paired transition counts.Items answered correctly at T=1 are easy; those answered incorrectly are hard.Source: G Difficulty-Adaptation Details · pp. 15–16
  • Table S8 measures NLL and accuracy changes at fixed two-loop depth against a same-budget baseline; held-out BIG-Bench Hard tasks remain diagnostic only.The table covers per-suite generation results at 0.6B and 1.7B.Source: Table S8 · p. 16, Table S7 · p. 16
  • At τ=0.50 with Tmin=2, about 50 of the 65 deepened items are hard, and 56 flip wrong-to-right versus 3 right-to-wrong.The hard-minus-easy loop gap is +0.06 loops, positive on all eight seeds.Source: G Difficulty-Adaptation Details · p. 16
  • At τ=0.10 without a floor, the hard–easy gap averages 0.59 loops, but the 2.80-loop operating point reaches only 53.6% accuracy.The table describes this as an adaptation illustration, not a competing accuracy result: some items stop too early while others are overcomputed.Source: G Difficulty-Adaptation Details · p. 16, Table S9 · p. 16
  • A lower threshold raises mean depth and widens the hard–easy gap; with τ=0.50 and no floor, mean depth falls to 1.06 loops.Adding Tmin=2 forces at least two loops and raises the mean to 2.04.Source: G Difficulty-Adaptation Details · p. 16

H Additional Ablations and Discussion

Across scales, generation-accuracy gains grow while teacher-forced NLL reductions shrink; additional controls test memory, capacity, task type, and adaptive loop allocation. The results include a fixed-depth 4B scale check and mixed outcomes across task types at 8B.

Memory Against Plain Looping

  • At 1.7B, memory improves generation accuracy by 2.3 points on MathQA and 2.4 on the classic suite over the memoryless baseline.Across 0.6B, 1.7B, 4B, and 8B, memory changes best-loop NLL from +0.07 to −0.03 nats at 1.7B and from +0.06 to −0.02 at 8B, relative to the same-budget baseline.Source: H Additional Ablations and Discussion · p. 17

Adaptive Loop Allocation

  • At τ=0.10, the MathQA threshold sweep reaches 53.6% accuracy at 2.80 loops; hard items receive more loops than easy ones, with a mean gap of +0.59.The sweep’s accuracy declines after its optimum as extra loops are wasted.Source: Figure S1 · p. 17, Table S10 · p. 17, Table S11 · p. 17

Matched Same-Start Memory Control at 8B

  • At 8B, matched continuation with memory lowers teacher-forced NLL by 0.040 and 0.028 nats at the best loop, and by 0.016 and 0.050 at four loops.Both continuations start from the same trained plain-looping checkpoint and receive the same extra training budget.Source: H Additional Ablations and Discussion · p. 17

Task Type Decides Plain Against Memory

  • Plain looping extrapolates slightly better on pure pointer-chasing chains, while memory performs better on stateful symbolic and arithmetic chains.The token-wise gate is designed to close memory on stateless patterns and open it on stateful ones.Source: H Additional Ablations and Discussion · p. 17

Parameter-Matched Non-Loop Control

  • At 1.7B, the matched-capacity non-loop control reaches 45.4% generation accuracy versus 44.8% for baseline, while RecurTrace reaches 46.0%.The control accounts for about half of RecurTrace’s +1.2-point gain; its NLL reduction is 0.01 nats versus 0.08 for RecurTrace.Source: H Additional Ablations and Discussion · pp. 17–18

Per-Scale Summary

The 8B Comparison

Related papers

Loading 2609.03379v2…