Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

1 INTRODUCTION

• LoopLMs reuse shared Transformer blocks across latent computation steps, increasing computational depth without increasing parameter count.The paper frames effective post-training of these models as challenging.Source: §1 · p. 1
• Existing methods leave intermediate computation weakly supervised, incur loop-scaled memory costs, or require stronger external teachers or privileged information.GRPO assigns reward credit only to the terminal loop; RLTT extends credit across loops but retains per-loop log-probabilities, while OPD and OPSD face teacher-availability or context-mismatch constraints.Source: §1 · pp. 1–2
• LoopOPD uses student-generated rollouts and same-prefix token distributions to distill a frozen terminal-loop policy into an intermediate-loop student.The teacher's advantage comes from additional recurrent computation, not extra parameters or privileged information.Source: §1 · p. 2
• D-LoopOPD refreshes the terminal-loop teacher at each optimization step, creating feedback through shared parameters; analysis gives sufficient conditions for simultaneous local descent at both loop depths.Table 1 compares methods' supervision signals and teacher requirements.Source: §1 · p. 2, Table 1 · p. 3
• Across mathematical reasoning, general reasoning, and code generation, LoopOPD and D-LoopOPD improve over vanilla models, with D-LoopOPD further improving mathematical performance over LoopOPD.Both methods outperform existing baselines on average for mathematical reasoning.Source: §1 · p. 2
• On Ouro-2.6B-Thinking, terminal-loop AIME 2024 Avg@16 rises from 62.08 to 71.67, with cross-loop gains and improvements in general reasoning and code generation.The models were trained only on mathematical data.Source: §1 · p. 3

2 RELATED WORK

• LoopLMs extend recurrent Transformer depth to language models, using shared blocks repeatedly before token generation and enabling reasoning and test-time-compute scaling.The paper builds on this structure by treating additional loop computation as supervision.Source: §2 · p. 3
• LoopOPD provides dense token-level supervision to an intermediate loop using the model's own terminal-loop distribution.Unlike related intermediate-supervision approaches, its teacher and student share the same context.Source: §2 · p. 3
• LoopOPD derives teacher advantage from additional recurrent computation while retaining on-policy compatibility; D-LoopOPD refreshes the terminal-loop teacher as parameters update.The contrast is with OPD's external teacher and OPSD's privileged-information context mismatch.Source: §2 · p. 3

3 PRELIMINARIES

• A LoopLM applies shared-weight Transformer blocks recurrently for L iterations before emitting each token, with each loop-depth hidden state defining a next-token distribution.The terminal-loop distribution is used for standard autoregressive generation; distributions at depths below L are intermediate-loop distributions.Source: §3 · pp. 3–4
• GRPO samples response groups from the old terminal-loop policy and uses scalar rewards, with gradients propagating through all loops to update shared parameters.The token-level importance ratio and clipped objective form its optimization setup.Source: §3 · p. 4
• RLTT extends reward credit from the terminal loop to all loops while retaining group-relative advantage, but its per-loop log-probabilities make memory usage scale linearly with loop count.This memory cost limits long-rollout training.Source: §3 · p. 4
• OPD uses token-level teacher distributions rather than scalar outcome rewards, evaluating teacher and student on the same prefixes sampled from the student policy.Its objective is reverse-KL distillation from a frozen teacher.Source: §3 · p. 4

4.1 LOOPOPD: CROSS-LOOP ON-POLICY DISTILLATION

• LoopOPD distills the terminal-loop distribution into an intermediate-loop student on shared student-generated prefixes using a reverse-KL objective.The described setup uses loop ℓ=2 as student and loop L=4 as teacher.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 DYNAMIC LOOPOPD: RECURRENT SELF-IMPROVEMENT

• D-LoopOPD uses the current terminal-loop policy as teacher, and shared-parameter updates make the updated teacher supervise the next student update.Repeating this process forms a closed feedback loop called recurrent self-improvement.Source: §4.2 · p. 5, §4.1 · p. 5

4.3 THEORETICAL ANALYSIS

• For an exact gradient step with a step-frozen teacher, a nonzero distillation gradient and step size 0 < η_t < 2/M_t guarantee lower batch-averaged KL divergence to that teacher.The guarantee assumes the empirical loss has an M_t-Lipschitz gradient near the update segment.Source: §4.3 · p. 6
• Positive alignment between the distillation gradient and task-surrogate gradients, under local smoothness, supplies sufficient conditions for local improvement at both loop depths.The analysis links the training update to evaluation-task gradients at both depths.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETUP

• Experiments use Ouro-1.4B-Thinking and Ouro-2.6B-Thinking, compare against GRPO, RLTT, and OPSD, and train post-training methods on DAPO-Math-17k.Evaluation covers AIME 2024–2026, OlympiadBench, BeyondAIME, MMLU, MBPP/MBPP+, and HumanEval/HumanEval+.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

• On Ouro-2.6B-Thinking, average AIME Avg@16 was 55.49 for vanilla, 56.60 for GRPO, 59.37 for LoopOPD, and 62.36 for D-LoopOPD.D-LoopOPD achieves the best Avg@16 on all three AIME benchmarks in the Ouro-2.6B group; its OlympiadBench score is 65.77, while its BeyondAIME score of 34.00 remains comparable to vanilla and GRPO.Source: Table 2 · p. 7, §5.2 · p. 7
• On Ouro-2.6B-Thinking, D-LoopOPD improves over vanilla on all five general-reasoning and code-generation benchmarks.The passage reports gains of 0.58 on MMLU, 1.06 on MBPP, 1.33 on MBPP+, 0.61 on HumanEval, and 0.61 on HumanEval+.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 TRAINING DYNAMICS AND CROSS-LOOP PERFORMANCE

• D-LoopOPD improves Avg@16 over vanilla at every loop depth, with a gain of 8.34 points at directly supervised loop 2.At loops 3 and 4, Avg@16 increases by 5.41 and 9.59 points, reaching 66.87 and 71.67, respectively.Source: Figure 3 · p. 8, §5.3 · p. 8
• During training, D-LoopOPD achieves higher rollout reward and student-loop validation Avg@16 than LoopOPD.Both quantities are measured at student loop ℓ=2.Source: Figure 3 · p. 8, §5.3 · p. 8
• D-LoopOPD maintains higher teacher and student entropy than LoopOPD through most of training, with the gap narrowing toward later steps.The passage suggests that dynamically refreshed teachers maintain less peaked distributions, supporting more diverse student rollouts and broader exploration.Source: Figure 4 · p. 9, §5.3 · p. 9

5.4 ABLATION STUDY

• Using loop 2 as the D-LoopOPD student yields the strongest terminal-loop result, improving loop 4 Avg@16 from 62.08 to 71.67.Loop 1's student Avg@16 remains 0.63 after training, while loop 3 starts near the terminal teacher at 61.46 versus 62.08.Source: §5.4 · p. 9

6 CONCLUSION

• Figure 5 compares AIME 2024 Avg@16 at the student and terminal loops for D-LoopOPD student-depth variants.The caption identifies Ouro-2.6B-Thinking as the evaluated model.Source: Figure 5 · p. 9
• LoopOPD uses a frozen terminal-loop policy to supervise an intermediate-loop student, while D-LoopOPD refreshes that teacher through shared parameter updates.The analysis gives sufficient conditions for a single gradient update to improve task surrogates locally at both loop depths.Source: §6 · p. 9
• Experiments show mathematical gains, improvement at deeper loops beyond the supervised student, and general-reasoning and code gains despite math-only training.The authors conclude that additional recurrent computation can serve as an effective source of supervision for LoopLM post-training.Source: §6 · p. 9

A PROOFS

• The derivation holds sampled tokens, response lengths, and teacher probabilities fixed, uses reverse KL, and normalizes each token occurrence by its response length.The teacher target is stopped during differentiation.Source: A PROOFS · p. 13
• When both loop-specific improvement coefficients are positive, the stated step-size condition makes one update locally improve both loop depths.The proof applies the update bound at the intermediate and terminal depths.Source: A PROOFS · p. 14
• In the aligned example, a gradient step improves correct-token probabilities at both depths; in the contrasting example, it improves the student while worsening the terminal probability.The contrasting case has γℓ,t > 0 and γL,t < 0, so a stronger terminal teacher alone does not ensure γL,t > 0.Source: A PROOFS · p. 14

B ALGORITHMS

• Each algorithm evaluates teacher and student distributions on student-generated prefixes; LoopOPD freezes the terminal teacher, whereas D-LoopOPD refreshes it after each update.The refreshed terminal policy supervises the next iteration.Source: B ALGORITHMS · pp. 14–15

C.1 ADDITIONAL TRAINING DYNAMICS

• Table 4 reports AIME 2024 validation Avg@16 during training for Ouro-2.6B-Thinking at student loop ℓ=2.The cited passage identifies the table's scope but gives no values.Source: §C.1 · p. 16, Table 4 · p. 16
• RLTT produces substantially shorter responses and higher truncation throughout training, while LoopOPD and D-LoopOPD generate longer responses with near-zero truncation.RLTT used a 2,048-token maximum rollout length, versus 31,744 tokens for LoopOPD and D-LoopOPD.Source: §C.1 · p. 16, Figure 6 · p. 16

C.2 CROSS-LOOP PERFORMANCE

• The cross-loop evaluation reports AIME 2024 Avg@16 / Pass@16 at each Ouro-2.6B-Thinking loop depth with a 31,744-token maximum response length.Table 5 contains the exact values corresponding to the cross-loop results in Figure 3(b).Source: §C.2 · p. 16

C.3 EFFECT OF DECODE BUDGET

• During training for Ouro-2.6B-Thinking, RLTT exhibits substantially higher rollout truncation than LoopOPD and D-LoopOPD throughout training, while D-LoopOPD continues to benefit from longer generation.Source: Figure 7 · p. 17, §C.3 · p. 17
• At 31K tokens, D-LoopOPD reaches 71.67 Avg@16 versus RLTT’s 56.67, and it overtakes RLTT at 16K tokens.D-LoopOPD exceeds the vanilla model across decode budgets and from 2K tokens onward; RLTT falls below vanilla at 16K and 31K.Source: §C.3 · p. 16, Figure 8 · p. 17

D.1 TRAINING CONFIGURATION

• All post-training methods use DAPO-Math-17k, and each training run uses four NVIDIA H200 GPUs.LoopOPD and D-LoopOPD use uniform recurrent-loop weights, student depth ℓ=2, and terminal depth L=4.Source: §D.1 · pp. 17–18
• Table 7 summarizes training and validation hyperparameters for the different methods.The supplied passage does not provide the table's hyperparameter values.Source: Table 7 · p. 18

Mathematical Reasoning

• Mathematical reasoning uses zero-shot generation with thinking and prompts models to reason step by step and put the final answer in a box.AIME 2024, 2025, and 2026 use 30 problems each and report Avg@16 and Pass@16 across 16 sampled completions per problem.Source: §D.2 · pp. 18–19
• OlympiadBench uses 675 text-based problems, while BeyondAIME uses its official 100-problem test split; both use temperature 1.0, top-p 0.7, and a 31,744-token maximum.Source: §D.2 · p. 19

MMLU

• MMLU uses five subject-specific development demonstrations and the subject description, then selects the answer choice with the highest log-likelihood.Evaluation covers the full test split of 14,042 questions across 57 subjects.Source: §D.2 · p. 19

Code Generation

• MBPP, MBPP+, HumanEval, and HumanEval+ are evaluated with EvalPlus using one greedily decoded program per problem, with a maximum of 768 new tokens.The “+” variants additionally include extended tests.Source: §D.2 · pp. 19–20

Question and Ground Truth

• The problem asks for the least base b with more than ten b-eautiful integers, and the supplied ground truth is b = 211.A b-eautiful integer has exactly two base-b digits whose sum equals its square root.Source: E CASE STUDY · p. 20

Response from Vanilla Model

• The vanilla response does not reach an answer, instead revising its count-based reasoning through several mistaken or incomplete approaches.It first reports only three solutions for b = 7, then considers b = 121 but concludes that its count is still below the required eleven; the excerpt says no final answer is provided.Source: E CASE STUDY · pp. 21, 24

Later response

• A later response expresses a two-digit number as x × b + y and begins reducing the condition to an equation using s = x + y.The excerpt ends before this response provides a count or answer, so its outcome is not shown.Source: E CASE STUDY · pp. 24–25

• Each s with 2≤s≤b−1 and (b−1)|s(s−1) corresponds to exactly one b-beautiful number, and every b-beautiful number arises from such an s.For b=2, there are no such values of s.Source: E CASE STUDY · p. 25
• Because consecutive integers are coprime, each prime power of d is assigned to either s or s−1, and the Chinese Remainder Theorem gives a unique residue for each coprime split.Source: E CASE STUDY · p. 26
• For d=b−1 with r distinct prime factors, k(k−1) ≡0 (mod d) has 2^r solutions in [1,d], yielding 2^r−1 valid candidates after excluding k=1.Source: E CASE STUDY · p. 26
• For b=211, there are 15 b-beautiful numbers, making 211 the least b ≥2 with more than ten.Source: E CASE STUDY · p. 28