PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M

Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju

arXiv:2609.14715v1cs.AIcs.CL

TL;DR

The paper asks what controlled scaling buys in the sub-150M regime and tests a deeper, narrower 110M model against a 53M baseline. Scaling improves the reported metrics, and value residuals plus Muon further raise ARC-Easy; the authors retain that stack while reporting trade-offs for the alternatives.

  • Problem

    The paper asks what simply scaling the same recipe buys in the sub-150M regime.

    Source: §1 · p. 1

  • Method

    The study doubles model size with a deeper, narrower geometry while holding other recipe choices fixed, then tests successive ablation rungs against the 110M baseline.

    Source: §1 · p. 1, §5 · p. 3

  • Results

    Scaling 53M to 110M improves every reported metric, and value residuals plus Muon raise ARC-Easy by a cumulative +3.6 to 56.1 at near-flat BLiMP.

    Source: §4 · p. 2, §8 · p. 7

  • Takeaways & Limitations

    JugnuLM-110M-R2 is the strongest kept stack, while the data blend and logit-distillation settings are dropped.

    Source: §8 · p. 7

    A larger-token competitive run remains future work, with the authors proposing approximately 25–40B tokens and educational-data upweighting to target the ARC gap.

    Source: §8 · p. 7

Abstract

from arXiv · show

We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack.

1 Introduction

The section asks what controlled scaling buys in the sub-150M regime. It doubles JugnuLM’s parameter count while changing geometry and holding the rest of the recipe fixed.

1 Introduction

  • The central question is what simply scaling the same recipe buys in the sub-150M regime.The authors frame this as a cheap, controlled first question.Source: §1 · p. 1
  • JugnuLM scales from 53M to ∼110M parameters and changes only the geometry to a deeper, narrower design.The geometry follows MobileLLM’s depth-over-width finding [9].Source: §1 · p. 1
  • Holding other choices fixed isolates the effect of scale and depth from changes in method or data.Source: §1 · p. 1

2 The two models

The two models share the entire recipe; their geometry and token budget differ.

2 The two models

  • Only geometry and token budget differ between the two models.Source: §2 · p. 1, Table 1 · p. 2
  • The two models share the entire recipe.Source: Table 1 · p. 2
  • Table 1 presents the comparison as a shared-recipe setup with geometry and token budget as the differences.Source: Table 1 · p. 2

3 Training

The models use a shared FineWeb-Edu training setup, including the same batch scale, hardware, and schedule. R2 additionally uses Muon.

3 Training

  • Both models train on FineWeb-Edu (sample-10BT) using bf16, DDP, torch.compile, and four RTX PRO 4500 Blackwell GPUs.Source: §3 · p. 2
  • Training uses a cosine schedule from 1.5 × 10−3 to 1.5 × 10−4, an approximately 0.5M-token global batch, and z-loss.The z-loss is used for logit stability.Source: §3 · p. 2
  • R1 and R2 share this training setup, while R2 additionally swaps the optimizer to Muon.Source: §3 · p. 2
  • The 110M variants finish at approximately 2.85 cross-entropy versus approximately 3.07 for 53M, despite seeing fewer tokens.The three 110M rungs have similar training loss; their levers separate downstream rather than on the raw objective, with one exception noted in the passage.Source: §3 · p. 2

4 Results

Scaling to 110M improves the reported metrics, and the 110M baseline nearly matches GPT-X2-125M on BLiMP at fewer parameters. Adding the kept ablation rungs raises ARC-Easy further.

4 Results

  • 81.25 BLiMP for 110M (R0) essentially matches GPT-X2-125M’s 81.28 at ∼12% fewer parameters, while R2 reaches 56.10 ARC-Easy, within ≈1 point of GPT-X2-125M.Source: Table 2 · p. 2
  • +3.11 BLiMP, +1.05 ARC-Easy, and −0.09 byte-perplexity mark the 53M →110M (R0) jump.The two ablation rungs then add a further +3.6 ARC-Easy.Source: §4 · p. 2
  • On BLiMP-versus-size, the 110M rungs cluster with 125–143M models as BLiMP saturates; on ARC-Easy-versus-size, the ladder climbs from R0 through R2 toward GPT-X2-125M.Source: §4 · p. 2
  • All 110M variants finish below 53M in pretraining cross-entropy despite seeing fewer tokens, and their three curves nearly coincide.Source: Figure 1 · p. 3

5 The ablation ladder

The ablation ladder adds one change at a time to R0 and keeps it only when it beats the prior rung. Value residuals and Muon are kept; the data blend and both distillation settings are dropped.

5 The ablation ladder

  • ARC-Easy rises +3.6 (52.48 →56.10) across kept rungs R1–R2; later rungs are dropped, leaving R2 the champion stack.The ladder keeps a change only if it beats the previous rung on ARC-Easy.Source: Table 3 · p. 3, §5 · p. 3, Figure 2 · p. 4

R1 — value residuals

  • ARC-Easy improves +2.2 with value residuals at a BLiMP tie (SE ≈0.14) and slightly lower perplexity, so R1 is kept.Learned gates converge to non-trivial values from −2.5 to +3.2; the custom residual pathway must also be applied during evaluation.Source: §5 · pp. 3–4

R2 — the Muon optimizer

  • ARC-Easy gains a further +1.43 (54.67 →56.10) with Muon, while BLiMP dips −0.32 and perplexity falls; R2 is kept.Muon’s validation-perplexity lead over AdamW shrinks from −26% at step 1000 to ≈−1% by the end, but the downstream ARC gain persists.Source: §5 · p. 4

R3 — a data blend

  • R3’s data blend gives the best BLiMP (81.79) and byte-perplexity (1.909), but ARC-Easy falls −2.48 (56.10 →53.62), so it is dropped.The authors suggest FineWeb-Edu’s educational filtering aligns with ARC-Easy, while general-web and math additions do not replace that signal.Source: §5 · pp. 4–5

5 The ablation ladder

  • Across the 110M rungs, BLiMP saturates while ARC-Easy climbs from R0 through R2, then falls back at R3.R2’s ARC-Easy result is within ≈1 point of GPT-X2-125M.Source: Figure 3 · p. 5

R4 — logit distillation

  • Heavy distillation reaches ARC-Easy 56.99 but byte-perplexity rises to 2.178; light distillation restores byte-perplexity to 1.917 but ARC-Easy falls to 55.47.Neither setting beats R2 on the blended efficiency score; the distillation runs also used a 6B-token region, versus R2’s full ∼10B pool.Source: §5 · p. 5

6 Discussion

The 110M model scored higher on every metric despite seeing fewer tokens, supporting a capacity-and-depth gain rather than a data-volume explanation. The report treats this conventional scaling result as a control for the ablation ladder.

6 Discussion

  • With ≈8B versus ≈12B tokens, the 110M model scored higher on every metric, though the comparison is not token-matched.The authors say this caveat strengthens the direction of the result.Source: §6 · p. 5
  • On the fixed FineWeb-Edu probe, Muon’s validation-perplexity lead over AdamW shrinks from −26% at ≈0.5B tokens to ≈−1% by the end.R0 and R1 track each other, while the blend sits ≈4–7% higher because 45% of its training data is off the probe distribution; the authors caution this is not a quality verdict.Source: Figure 4 · p. 6
  • The deep-thin design spends the extra budget on depth, growing from 8 to 23 layers rather than increasing width, and reaches grammar competence comparable to wider, larger, more-trained models.The authors follow MobileLLM’s depth-over-width finding [9].Source: §6 · p. 6
  • The 110M model is a deliberately conventional control that establishes how far plain scaling goes before the ablations are measured against it.The report identifies it as rung R0 of the ablation ladder.Source: §6 · p. 6

7 Reproducibility

The authors provide public code, configurations, model weights, and model cards, and document how to evaluate R0 and the custom ablation checkpoints.

7 Reproducibility

  • Code, configurations, weights, and model cards for JugnuLM-53M, JugnuLM-110M R0, and ablation rungs R1–R4b are publicly available.The code and configurations are at the Jugnu repository, with weights and model cards on Hugging Face.Source: §7 · p. 6
  • R0 evaluation uses the EleutherAI harness with the Hugging Face model interface and the BLiMP, ARC-Easy, and WikiText tasks.The command specifies bfloat16, automatic batch sizing, and the JugnuLM-110M checkpoint.Source: §7 · p. 6
  • R1–R4b checkpoints require the rebuilt model with remote code enabled, because loading them as stock Qwen3 silently drops the custom value-residual pathway and understates results.The value-residual pathway must also be active during evaluation.Source: §7 · pp. 6–7

8 Conclusion

The deep-thin 110M model and the kept value-residual and Muon levers form the strongest kept stack, while data blending and distillation were dropped after mixed results. The conclusion identifies larger-budget training and improved distillation as future directions.

8 Conclusion

  • Value residuals and Muon together lift ARC-Easy by +3.6 to 56.1 at near-flat BLiMP, while the deep-thin 110M matches a 125M reference’s BLiMP with ≈12% fewer parameters and fewer training tokens.JugnuLM-110M-R2 is the strongest kept stack, competitive with the top of the sub-150M field on the authors’ efficiency estimate.Source: §8 · p. 7
  • The diverse blend improved aggregate language modelling but lost ARC-Easy, while distillation reached 56.99 ARC-Easy only at a perplexity cost that a gentler setting erased.The authors conclude that at this scale ARC-Easy tracks FineWeb-Edu’s educational filtering rather than raw diversity; offline logit distillation from a 1.7B teacher was not a clean win at this teacher–student gap and token budget.Source: §8 · p. 7
  • The authors propose a 25–40B-token run with a warmup-stable-decay schedule and educational-data upweighting, and further distillation tests using reverse-KL, a smaller teacher, or full-corpus logits.These are presented as directions most likely to close the ARC gap and improve distillation’s mechanism.Source: §8 · p. 7
Loading 2609.14715v1…