PaperlayerView exact revision on arXiv

Source-linked AI summary

Dory1.0-4B: A Looped Normalized-GPT

Maxim Fishman, Simeng Sun, Shantanu Acharya, Faisal Ladhak, Brian Chmiel, Santiago Akle Serano, Ron Banner, Boris Ginsburg, Krishna C. Puvvada

arXiv:2611.03998v1cs.CL

TL;DR

Test-time compute increasingly relies on longer reasoning traces, while latency, KV-cache footprint, and serving cost grow with generated tokens. Dory1.0-4B uses a shared recurrent block to provide elastic inference depth in one checkpoint. Deeper loops improve its average performance across 12 benchmarks and reduce output tokens on agentic tasks.

  • Problem

    Test-time compute is often spent generating longer reasoning traces, while latency, KV-cache footprint, and serving cost grow with each generated token.

    Source: §1 · p. 1

  • Method

    Dory1.0-4B is a looped language model whose shared recurrent block can run at variable depths, with SFT and RLVR training that routes harder prompts to deeper loops.

    Source: §1 · pp. 1–2, §7 · p. 15

  • Results

    With two or more loops, Dory1.0-4B outperforms Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks; four loops generate 1.3–2.5× fewer tokens on agentic benchmarks than one loop.

    Source: §7 · p. 16

  • Takeaways & Limitations

    One checkpoint can trade inference compute for accuracy per request without retraining or swapping models.

    Source: §1 · p. 1

    Benchmark scores may differ from Artificial Analysis results, though the shared evaluation protocol makes scores comparable across models within this evaluation.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

We introduce Dory1.0-4B, a 4-billion-parameter looped (latent-recurrent) language model with elastic depth: a single checkpoint serves different inference compute budgets. Dory1.0-4B is built on a normalized Transformer (nGPT) backbone with interleaved sliding-window and full-attention layers, organized into an input block, a shared recurrent block, and an output block. At inference, the recurrent block can be applied a variable number of times. We pre-trained Dory1.0-4B on 6 trillion tokens. The first 4.5 trillion tokens were trained in NVFP4, and additional pre-training then extended the context length to 256K tokens. Next, we post-trained the model using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), with a difficulty-aware loop curriculum that routes harder prompts to deeper loops. Increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient on agentic benchmarks: with four loops, it generates 1.3-2.5x fewer tokens than with one loop, as the extra computation is spent in latent space rather than on longer outputs. To reduce memory during looping, Dory1.0-4B caches keys and values only from the last loop, keeping its KV-cache footprint equal to that of a non-looped model at any loop count while retaining most of the accuracy gains. We open-source the BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.

1. Introduction

Dory1.0-4B uses a shared recurrent block to let one checkpoint trade inference compute for accuracy, addressing the growing token-space costs of test-time compute. Deeper loops improve benchmark performance and token efficiency, alongside post-training, cache, and checkpoint choices for deployment.

1. Introduction

  • With two or more loops, Dory1.0-4B outperforms Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks.Its equal-weight average rises from 35.2 at one loop to 40.6 at four loops.Source: §1 · p. 1, Figure 1 · p. 2
  • Four loops reduce generated tokens per attempt by 1.3–2.5× across four agentic benchmarks, partially offsetting higher compute per token.The largest reductions are on GDPval-AA v2 and Terminal-Bench v2.1.Source: §1 · p. 1, Figure 1 · p. 2
  • A shared 40-layer recurrent block runs k times between 16-layer input and output blocks, with the input representation re-injected before each loop.The model uses a 3:1 interleaving of sliding-window and full-attention layers.Source: §1 · p. 2, Figure 2 · p. 3
  • Dory1.0-4B was pretrained on 6T tokens, with the first approximately 4.5T trained in NVFP4, then extended to a 256K-token context.The paper also reports open-sourced MXFP8, NVFP4, and Q5_K_M checkpoints.Source: §1 · p. 2
  • Post-training combines SFT with RLVR, whose difficulty-aware loop curriculum preferentially routes prompts with lower profiled pass rates to deeper loops.At four loops, the curriculum raises Terminal-Bench v2.1 accuracy from 14.0% to 21.4%.Source: §1 · p. 2
  • Last-loop KV caching overwrites prior loop cache entries, keeping attention KV memory at one-loop size for any loop count.At four loops, scores change by −2.1 to +0.4 points relative to per-loop caching, with nearly unchanged output lengths.Source: §1 · p. 2

2. Architecture

Dory1.0-4B adapts the normalized Transformer with normalized sublayer outputs, learned residual blending, and a gated attention design. Its 72-layer stack interleaves sliding-window and full-attention layers.

2. Architecture

  • The nGPT implementation L2-normalizes gated attention outputs and SwiGLU activations before their output projections.These normalizations are implemented with fused Triton kernels.Source: §2 · p. 4
  • Each residual update blends normalized sublayer output with normalized residual using learned per-channel weights, then normalizes the result.Residual-rate, query–key scale, and logit-scale parameters are initialized at 0.0177 while the effective residual rate starts at 0.05.Source: §2 · p. 4
  • The 72-layer stack interleaves 27 sliding-window and 9 full-attention layers, alongside 36 MLP-only layers.The attention layers follow a 3:1 sliding-window/full-attention pattern.Source: §2 · p. 4

3. Pretraining

Pretraining progresses from NVFP4 training through looped BF16 continuation to a long-context extension. The section also specifies training procedures and fixed-loop base-model evaluations.

3.1. NVFP4 Training

  • NVFP4 block-scaling uses E2M1 4-bit operands for supported linear-layer matrix multiplications while retaining BF16 weights and FP32 optimizer states.The recipe yields an approximately 20% end-to-end training speedup over BF16; the data curriculum shifts from diversity-favoring phase 1 to quality-favoring phase 2.Source: §3.1 · p. 4

3.2. Training with Loops

  • The recurrent block is shared across k loops, with the input representation re-added and L2 normalization applied at both loop boundaries.Loop counts are sampled from 1–4, with rotating one-loop overrides on approximately 9.9% of replicas.Source: §3.2 · pp. 4–5

3.3. Truncated Backpropagation

  • Depth-one truncated backpropagation disables gradient tracking on earlier recurrent invocations and trains the final invocation, input block, and output block.Next-token cross-entropy is applied only after the final sampled invocation, without intermediate-exit or early-loop auxiliary losses.Source: §3.3 · p. 5

3.4. Long-Context Extension

  • Long-context extension increases sequence length from 32,768 to 262,144 tokens using approximately 300B additional tokens.The full-attention RoPE base increases from 500,000 to 10,000,000 while the sliding-window size remains 1,024.Source: §3.4 · p. 5

3. Pretraining

  • The extended base model has seen approximately 6.3T training tokens, including an additional approximately 300B-token long-context phase.The section covers the pretraining recipe, data, and hyperparameters.Source: §3 · p. 4, §3.5 · p. 5, §3.4 · p. 5

3.5. Base Model Evaluations

  • Base-model evaluations use fixed loop counts k∈{1, 2, 4, 6, 8}, with six- and eight-loop runs measuring inference-depth extrapolation beyond training depths.Evaluations use raw prompts without a chat template or system instruction; Table 1 reports percentages with higher scores better.Source: §3.5 · pp. 5–6, Table 1 · p. 6

4. Post-Training

Dory1.0-4B post-training combines multi-loop supervised fine-tuning with task-reward RLVR, which preferentially routes harder prompts to deeper loops. Deeper execution improves results across several benchmarks, while reducing output tokens on agentic tasks.

4. Post-Training

  • The post-training pipeline combines SFT with 200-step RLVR across multiple loop counts, using task-specific rewards and routing lower-pass-rate prompts deeper during RLVR.SFT starts from the long-context extended checkpoint; RLVR starts from the nine-epoch SFT checkpoint.Source: §4 · p. 7

4.1. Supervised Fine-Tuning

  • Across one-loop evaluations from iterations 3,600 to 32,400, SciCode rises 13.11% to 18.14%, Terminal-Bench v2.1 4.2% to 13.5%, and GPQA Diamond 43.13% to 54.95%.Over the same interval, IFBench increases from 28.94% to 38.50%, MATH-500 from 60.2% to 75.2%, and AA-LCR from 29.5% to 37.3%. SFT trains on thinking and non-thinking responses, optimizing assistant tokens while masking system, user, and tool tokens.Source: §4.1 · p. 8

4.2. RLVR

  • With the same 4:1:1:1 prompt-loop distribution, the α=0.3 curriculum improves all six benchmarks at one and four loops, with the largest gains on agentic tasks at four loops.Lower profiled pass rates bias prompts toward deeper loops, while the routing retains substantial randomness.Source: §4.2 · p. 9

4.3. Evaluation

  • At four loops, Dory1.0-4B beats all evaluated baselines on τ3-Banking, IFBench, and AA-LCR, and scores 35.11% vs. Qwen3.5-9B’s 28.21% on SciCode.Between one and four loops, GDPval-AA v2 rises from 815 to 967 Elo, SWE-Bench Verified from 50.3% to 61.8%, and Terminal-Bench v2.1 from 12.5% to 22.2%; AA-Omniscience accuracy remains essentially unchanged.Source: §4.3 · p. 11, Table 2 · p. 11

4.4. Token Efficiency

  • Going from one to four loops reduces output tokens per attempt by 1.3–2.5× across agentic benchmarks, with savings coming from fewer turns, shorter turns, or both.At two loops, generation compute is 1.03–1.25× one-loop compute across four tasks, while recovering 42–86% of four-loop accuracy gains; the authors recommend two loops as a default and four when accuracy matters most.Source: §4.4 · p. 12, Table 3 · p. 12

5. Quantization

The section evaluates weight-only quantization for Dory1.0-4B, applying quantization-aware distillation (QAD) when post-training quantization falls short. MXFP8 remains broadly comparable to BF16, and quantization accelerates inference.

5.1. Overview

  • The quantization study targets MXFP8, NVFP4, and Q5_K_M (GGUF), keeping learned vectors and scalars in high precision.PTQ is followed by QAD when needed to recover accuracy relative to the BF16 checkpoint.Source: §5.1 · p. 12

5.2. Post-Training Quantization (PTQ)

  • PTQ is applied to weight matrices except token embeddings and the LM head; QAD is used if a quantized model falls short of target accuracy.MXFP8 uses standard maximum scaling; NVFP4 uses dynamic MSE scale selection, while the other formats use maximum scaling.Source: §5.1 · p. 12, §5.2 · p. 13

5.3. Quantization-Aware Distillation (QAD)

  • QAD trains the PTQ student against a frozen BF16 teacher on the same SFT sequence and loop count, minimizing full-vocabulary forward KL at temperature one.Only valid assistant next-token targets contribute to the objective; contextual tokens still affect assistant predictions.Source: §5.3 · p. 13

5.4. Results

  • MXFP8-PTQ remains broadly comparable to BF16, while NVFP4-QAD and Q5_K_M-QAD retain competitive performance with task-dependent accuracy gaps.Table 4 compares the variants at one, two, and four inference loops.Source: Table 4 · p. 13, §5.4 · p. 14
  • Up to 2.44× inference speedup over BF16 is measured for MXFP8 and NVFP4 quantization.The speedups are measured on NVIDIA DGX Spark.Source: §5.4 · p. 14, Table 5 · p. 14

6. Inference Optimization

The section tests last-loop KV caching and self-speculative decoding as inference optimizations for Dory1.0-4B. Last-loop caching limits KV memory growth, while self-speculation improves throughput and request duration under measured conditions.

6. Inference Optimization

  • The section evaluates two optimizations: last-loop caching reduces attention KV memory growth, and self-speculative decoding combines shallow drafting with deeper verification.Self-speculative decoding is evaluated separately using per-loop KV caches.Source: §6 · p. 14

6.1. Last-Loop KV Caching

  • Last-loop caching overwrites each recurrent layer’s cache at every loop, retaining final-loop keys and values and making attention KV memory independent of loop count.Earlier-loop queries consequently attend to final-loop entries for prior tokens rather than same-loop entries.Source: §6.1 · p. 14
  • At four loops, last-loop caching changes scores by −2.12 to +0.42 points, keeps KV memory at its one-loop size, and changes output lengths little.The output-token ratio ranges from 1.0% fewer tokens on GPQA Diamond to 3.4% more on Terminal-Bench v2.1.Source: §6.1 · pp. 14–15, Table 6 · p. 15

6.2. Self-Speculative Decoding

  • The same checkpoint drafts at one loop and verifies four proposed tokens at four loops; rejection sampling accepts proposals and corrects rejections to preserve the verifier’s sampling distribution in exact arithmetic.Draft activations are reused during verification to avoid redundant computation for already processed positions.Source: §6.2 · p. 15
  • With KV and activation reuse, self-speculation reaches 1.37–1.71× the throughput of autoregressive decoding at four loops across measured configurations.Gains generally decrease at larger batch sizes, but persist through batch size 256 for shorter prompts and batch size 64 for 16K-token prompts.Source: §6.2 · p. 15
  • Self-speculative decoding achieves 1.11–1.53× speedups in mean prefill-plus-decode request duration across six benchmarks.The measured gains depend on workload and serving conditions.Source: §6.2 · p. 15

7. Conclusion

Dory1.0-4B is a 4-billion-parameter looped model whose shared recurrent block can be applied a variable number of times. Across 12 benchmarks, deeper execution improves average results and token efficiency, while last-loop caching and quantized checkpoints target inference memory and deployment cost.

7. Conclusion

  • A single checkpoint serves different inference compute budgets by varying how often its shared recurrent block is applied.The model uses a normalized Transformer backbone with interleaved sliding-window and full-attention layers.Source: §7 · p. 15
  • With two or more loops, Dory1.0-4B outperforms Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks.The second loop gives the largest accuracy gain per unit of added compute.Source: §7 · p. 16
  • Four loops generate 1.3–2.5× fewer tokens than one loop on agentic benchmarks.Last-loop KV caching keeps the cache footprint equal to that of a non-looped model at any loop count.Source: §7 · p. 16
  • The released checkpoints include BF16 and quantized MXFP8, NVFP4, and GGUF versions.The paper presents these formats as ways to reduce deployment cost.Source: §7 · p. 16
Loading 2611.03998v1…