PaperlayerView exact revision on arXiv

Source-linked AI summary

Loop the Loopies!

Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai

arXiv:2607.16051v2cs.CLcs.AI

TL;DR

Under fixed pre-training compute, vanilla parameter scaling can outperform looping, leaving recurrent depth without a clear path to competitive large language models. Loopie introduces compute-matched looped MoE models and reports consistent gains over matched vanilla Transformer baselines, alongside strong mathematical reasoning and coding abilities.

  • Problem

    Under fixed compute, prior evidence suggests looped Transformers trade non-recurrent capacity for recurrence, limiting clear evidence that recurrent depth can support flagship-scale performance.

  • Method

    Loopie combines layer-loop recurrence with a compute-matched scaling recipe to train two looped MoE models, followed by large-scale post-training.

  • Results

    Loopie consistently outperforms compute-matched vanilla Transformer baselines across model scales and develops strong mathematical reasoning and coding abilities.

  • Takeaways & Limitations

    The results suggest recurrent computation can serve as a practical scaling axis for large language models when jointly optimized with architecture and training efficiency.

  • Takeaways & Limitations

    Post-training experiments focus primarily on mathematical and code reasoning, without extensive exploration of scientific question answering, instruction following, alignment, or agentic task-solving.

Abstract

from arXiv · show

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

1 Introduction

The introduction asks whether looped Transformers can match or exceed vanilla Transformers under the same pre-training compute budget. It presents the Loopie Series, a compute-matched scaling recipe, scalable looped MoE models, and large-scale post-training for reasoning.

  • Looped Transformers repeatedly apply the same model across recurrent steps, offering an alternative to conventional depth scaling through parameter sharing.This approach has shown strong empirical performance across multiple domains.
  • Prior studies report that recurrent computation can outperform vanilla Transformers on in-context learning and data-fitting tasks while using fewer parameters for multi-step gradient descent.
  • The introduction highlights a compute-matching problem: Ouro-2.6B with 4 loops corresponds to 10.4B parameters, while Huginn-3.5B with 32 loops corresponds to a 112B-parameter baseline.These comparisons motivate evaluating looped and vanilla Transformers under equal pre-training compute.
  • The Loopie Series introduces Loopie-20B-A2B and Loopie-6B-A0.6B, each trained with two loop steps using the Loopie Recipe for fixed-compute scaling.The paper also reports extensive ablations, scalable looped MoE models, and large-scale post-training with a novel supervised pre-training stage that yields strong reasoning abilities.

2 The Loopie Series … 2.4 Loopie Recipe

The Loopie Series introduces layer-loop recurrence in MoE Transformers and argues that it scales better than model-loop under large pre-training budgets. Its Loopie Recipe converts reduced activation memory into larger microbatches and additional capacity, selecting models by measured optimizer-step time rather than theoretical FLOPs.

  • 2.1 Architecture: Loopie largely follows Qwen3-MoE as a decoder-only MoE Transformer, but applies recurrence within each layer before advancing to the next layer.This layer-loop design differs from model-loop recurrence, which repeatedly unrolls the entire Transformer stack.
  • 2.2 Why Layer-Loop?: After approximately 1.2 trillion training tokens, layer-loop overtakes model-loop and improves more rapidly thereafter in pre-training experiments.Layer-loop initially trails model-loop on downstream benchmarks, but the early advantage does not persist as training increases.
  • 2.2 Why Layer-Loop?: Layer-loop provides better execution locality because repeated applications of each layer are adjacent in forward and backward computation graphs.This shortens parameter and gradient reuse distance and simplifies activation checkpointing and gradient accumulation under sharding or offloading.
  • 2.3 Motivation: Looped Transformers face a fixed-compute challenge because looping a model N times approximately multiplies training compute by N while reducing non-recurrent capacity relative to standard baselines.Prior work therefore framed recurrence mainly as parameter-efficient rather than compute-efficient, leaving its path to flagship performance unclear.
  • 2.3 Motivation: The Loopie Recipe makes recurrence competitive by jointly choosing stored width, stored depth, and recurrent depth under a fixed pre-training compute budget.This treats loops as part of a recurrence-width-depth trade-off rather than as a direct substitute for additional parameters.
  • 2.4 Loopie Recipe: The recipe halves stored layers, executes each stored layer twice with layer-loop, doubles per-device microbatch size, and reinvests measured efficiency gains into capacity.The optimizer-step time is kept approximately matched to the non-recurrent reference while the token budget and optimizer updates remain aligned.
  • 2.4 Loopie Recipe: The final comparison is compute-matched by realized end-to-end training time, not analytical FLOP equality, because nominal proxies omit communication, optimizer, launch, and utilization effects.For Loopie-20B-A2B, matching requires changing the schedule from (b0, g0) to (2b0, g0/2), with measured efficiency attributed entirely to the doubled microbatch size.
  • 2.4 Loopie Recipe: Loopie-20B-A2B is selected by benchmarking candidates in Megatron-LM and choosing the configuration whose measured optimizer-step time most closely matches the non-recurrent reference.The recurrent seed preserves approximately the reference’s leading-order Transformer block work while substantially reducing stored-activation memory.

2.5 Results

Loopie-20B-A2B is compared with a vanilla 30B-A3B Qwen3-like MoE Transformer trained under the same compute budget. Although Loopie initially trails, it overtakes the baseline after roughly 600 billion tokens and maintains an advantage thereafter.

  • After roughly 600 billion tokens, Loopie-20B-A2B overtook the compute-matched vanilla baseline and maintained a consistent advantage.Loopie initially lagged behind during the early training phase.

2.6 Scalability · 2.7 Ablations · 2.8 Why Only Two Loop Steps?

Across a compute-matched scaling ladder, Loopie consistently outperforms vanilla counterparts as model size grows. Ablations show that the layer-loop schedule matters, while two loop steps offer the observed trade-off between recurrence and scalable pre-training compute.

  • 2.6 Scalability: Across all four scaling rungs, Loopie consistently outperforms its compute-matched vanilla counterpart, and the performance gap does not vanish with model size.The ladder compares four non-recurrent MoE baselines with four compute-matched Loopie models using two layer-loop steps.
  • 2.6 Scalability: As baseline parameter count grows from 0.15B to 1B, Loopie continues delivering stronger downstream accuracy under matched pre-training budgets.The scaling ladder is designed to match effective pre-training compute rather than stored parameter count.
  • 2.7 Ablations: The layer-loop pattern substantially improves average performance across eight downstream benchmarks relative to a matched ablation with recurrence removed.The ablation preserves the backbone, optimizer, data mixture, token budget, active-parameter count, and overall computation budget.
  • 2.7 Ablations: Because active parameters and total computation are unchanged, the ablation result indicates that Loopie’s gains depend on how recurrence is scheduled, not merely on adding computation.The findings align recurrence with the hierarchical structure of Transformer representations rather than treating naive looping as sufficient.
  • 2.8 Why Only Two Loop Steps?: In fixed-budget pre-training, loop count directly allocates FLOPs, so increasing recurrence must be weighed against using those FLOPs for width, depth, or additional tokens.For R loop steps, per-token training cost scales approximately linearly with R: C(D, L, R) ∝LD^2R.
  • 2.8 Why Only Two Loop Steps?: The loop-count sweep suggests that marginal returns from additional recurrence decay rapidly in the large-scale pre-training regime.The comparison uses N× layer-loop steps versus N× stored layers while holding other architectural details fixed and reports average scores across eight benchmarks after 250B tokens.
  • 2.8 Why Only Two Loop Steps?: Loopie therefore chooses R = 2, the smallest nontrivial recurrent setting, enabling one ordinary transformation followed by one local refinement step per layer.This keeps the compute multiplier small enough for scalable pre-training while changing the computation qualitatively.
  • 2.8 Why Only Two Loop Steps?: The choice R = 2 is a pre-training efficiency trade-off, not a claim that larger loop counts are ineffective in every setting.Larger R may be useful when inference-time computation is cheap, adaptive computation is available, or recurrent reasoning itself is the goal.

3 Pre-Training

Loopie models are pre-trained in two stages: large-scale training on 3T tokens followed by 1.26T-token high-quality annealing emphasizing synthetic, STEM, code, mathematical reasoning, and web data. Stage 1 repeats a high-quality 570B-token corpus for four epochs, while Stage 2 combines specialized reasoning, code, web, and mathematics sources.

  • Two-stage pre-training: 3T tokens are used for large-scale Stage 1 pre-training, followed by 1.26T tokens of high-quality Stage 2 annealing.Stage 2 emphasizes high-quality synthetic, STEM, code, mathematical reasoning, and web data.
  • Implementation: The models use the Qwen3 tokenizer family and Megatron-LM for pre-training.Pre-trained checkpoints are evaluated with the LM Evaluation Harness framework.
  • Stage 1: 570B unique tokens are repeated for four epochs in Stage 1, totaling approximately 2.28T training tokens.Both Loopie-20B-A2B and Loopie-6B-A0.6B use Nemotron-CC-v2-HQ, motivated by the effectiveness of repeated high-quality training.
  • Stage 2: 1263B tokens compose the Stage 2 annealing pool, led by 351B SFT-style tokens, 277B specialized tokens, and 262B code tokens.The mixture combines high-quality SFT-style, synthetic reasoning, code, synthetic web, and mathematics data.
  • Stage 2: 126B tokens from quality-scored mathematics data are included to strengthen mathematical reasoning and symbolic problem-solving.Only Nemotron-CC-Math-v1 documents with quality scores ≥4 are used.

4 Post-Training

Loopie’s post-training transforms Loopie Base into Loopie Thinking through supervised pre-training followed by reinforcement learning. Supervised pre-training combines supervised targets with language-model-scale optimization, improving reasoning and general capabilities simultaneously.

  • Training pipeline: The pipeline anneals a pretrained model into Loopie Base, then applies supervised pre-training and Math RL to produce Loopie Thinking.The overview identifies these sequential stages.
  • Supervised pre-training: Supervised pre-training continues for 2T tokens across instruction-following, reasoning, coding, mathematics, and tool-use data.This stage is intended to build broad task-following and problem-solving capabilities while preserving general capability.
  • Reinforcement learning: Reinforcement learning follows SPT to enhance reasoning, long-horizon problem solving, and reliable thinking traces, yielding the final Loopie Thinking model.The reported RL sequence trains on mathematical tasks first, then coding tasks after mathematical performance saturates.
  • Supervised pre-training: SPT uses an SFT-style target-only loss with pre-training-scale batch sizes, sequence lengths, and token budgets.Prompt, context, and padding tokens are excluded from the loss; only supervised target tokens contribute.
  • Supervised pre-training: SPT improves reasoning and pre-training metrics simultaneously, while conventional SFT improves reasoning but degrades pre-training metrics.SPT consistently improves downstream pre-training metrics such as ARC-Challenge and MMLU throughout approximately 10 epochs on 2T tokens.
  • Supervised pre-training: 128 million tokens per global batch make SPT approximately 1,000 times larger per batch than conventional SFT, reducing overfitting risk.The larger optimization scale produces distinct training dynamics compared with standard SFT.

5 Related Work

Related work traces looped models from recurrent and weight-tied architectures to modern systems that treat recurrence as a scaling axis. It also studies their recursive, algorithmic, adaptive-computation, optimization, mechanistic, and post-training properties.

  • Architectural lineage: Looped models reuse a learned internal operator within one forward computation, following Neural GPUs, ACT, Universal Transformers, and Deep Equilibrium Models.Related Transformer families include ALBERT, tied Transformers, depth-adaptive Transformers, feedback-memory models, recurrent-memory Transformers, and block-recurrent Transformers.
  • Recursive biases: Weight sharing biases looped models toward reusable update rules, computation decoupled from parameter count, iterative algorithms, knowledge re-access, compositionality, latent scratchpads, and adaptive computation.These biases support program emulation, optimization procedures, graph algorithms, length extrapolation, multi-hop composition, and hidden-state reasoning.
  • Theory and analysis: Formal and mechanistic studies characterize shared-depth capabilities, fixed-point trainability, loop-aware scaling, recurrent-step representations, and mixed evidence for literal latent chains of thought.Analyses cover instruction-level programs, latent chains of thought, timestep-modulated approximation, normalized-gradient updates, power iteration, stability, iso-depth scaling, fixed points, replayed inference stages, and functional specialization.
  • Scaling and dynamic depth: Recent looped language models make recurrence a third scaling axis alongside parameter and token counts, with variable test-time depth, retrofitted recurrence, dynamic routing, selective iteration, and layer skipping or repetition.Examples include Huginn, Ouro, LoopUS, Mixture-of-Recursions, CoTFormer, AdaPonderLM, Think-at-Hard, and Chain-of-Layers.
  • Efficiency and post-training: Parallel and memory-efficient unrolling methods reduce attention, cache, activation, or parameter costs, while trajectory-supervision methods train models across latent iterations rather than only on final answers.Examples include Parallel Loop Transformers, LT2, MELT, LASER, Hyperloop, CHERRY, LoopRPT, RLTT, denoising recursion, and Generative Recursive Reasoning.

6 Future Work

The study leaves several directions open because post-training focused mainly on mathematical and code reasoning, while inference-time computation and broader capabilities were not systematically explored. Future work should also examine supervised pre-training and Loopie’s interaction with newer architectural designs.

  • Capabilities and post-training: Post-training primarily covered mathematical and code reasoning, leaving scientific QA, instruction following, conversational alignment, and agentic task-solving largely unexplored.These omissions were attributed to computational constraints.
  • Pre-training ablations: A sufficiently comprehensive ablation study of supervised pre-training remains future work because of limited computational resources.
  • Inference-time computation: Systematic matching and optimization of inference-time computation remain important future directions, with Parallel Loop Transformer offered as a promising example.The study primarily matched compute budgets during pre-training rather than investigating inference-time computation.
  • Architectural extensions: Because experiments used the clean Qwen3-30B-A3B base architecture to avoid confounds, Loopie’s interaction with newer complementary architectural designs remains unstudied.

7 Conclusion

Loopie makes recurrent depth competitive under a matched pre-training compute budget by combining layer-loop recurrence with hardware-aware scaling. It consistently outperforms compute-matched vanilla Transformer baselines and gains mathematical reasoning and coding abilities through post-training.

  • 7 Conclusion: Loopie combines layer-loop recurrence with a hardware-aware scaling recipe to make recurrent depth competitive under matched pre-training compute.The model family is described as looped Mixture-of-Experts language models.
  • 7 Conclusion: Loopie consistently outperforms compute-matched vanilla Transformer baselines across model scales.A post-training pipeline based on Supervised Pre-training and reinforcement learning further equips Loopie with mathematical reasoning and coding abilities.

A Architecture Details

Table 4 presents the architecture details of two Loopie models: Loopie-20B-A2B and Loopie-6B-A0.6B.

  • A Architecture Details: Table 4 details the architecture of Loopie-20B-A2B.
  • A Architecture Details: Table 4 details the architecture of Loopie-6B-A0.6B.
  • A Architecture Details: The table covers both Loopie architectures.

B Pre-training Details

This section presents the pre-training details for Loopie-20B-A2B and Loopie-6B-A0.6B across two stages.

  • B Pre-training Details: The pre-training setup covers Stage 1 and Stage 2 for both Loopie models.The models are Loopie-20B-A2B and Loopie-6B-A0.6B.
  • B Pre-training Details: Stage 1 pre-training details are provided for Loopie-20B-A2B and Loopie-6B-A0.6B.
  • B Pre-training Details: Stage 2 pre-training details are provided for Loopie-20B-A2B and Loopie-6B-A0.6B.

C Scaling Ladder Details

Table 7 specifies the Loopie scaling ladder’s architectures and pairs each model with a vanilla MoE baseline under matched pre-training wall time. It defines the notation used for dimensions, layers, heads, recurrent steps, and Width/Depth.

  • Scaling ladder: Each Loopie model is paired with a non-recurrent vanilla MoE baseline under matched pre-training wall time.
  • Architecture notation: D denotes hidden dimension, DMoE denotes each expert’s hidden size, and L denotes the number of stored Transformer/MoE layers.
  • Architecture notation: Heads denotes attention-head count, Head Dim denotes each head’s dimension, and N denotes recurrent layer-loop steps.
  • Architecture notation: Width/Depth denotes D/(LN).

D Supervised Pre-training Details

This section presents the supervised pre-training hyperparameters for Loopie-20B-A2B and Loopie-6B-A0.6B.

  • Table 8 lists the supervised pre-training hyperparameters for Loopie-20B-A2B and Loopie-6B-A0.6B.
Loading 2607.16051v2…