Source-linked AI summary
Squeezing More from Limited Data with Recursive Transformers
Serdar Gülbahar, Lukas Edman, Alexander Fraser
TL;DR
The paper asks how to scale language-model computation when data is limited but compute is relatively abundant, a setting where standard web-scale intuitions fail. It studies recursive Transformers with shared depth and factorized embeddings, finding stronger performance than standard Transformers at 10M and 100M words while remaining competitive with BabyLM Challenge 2025 models.
Problem
Under fixed data, model size affects overfitting and standard Transformers couple per-token computation to representational capacity, limiting scaling choices.
Method
The paper evaluates RecursiveGPT, which reuses a shared Transformer block across depth and uses factorized embeddings to reduce vocabulary-map parameters.
Results
RecursiveGPT outperforms standard Transformers at both 10M and 100M words and remains competitive with BabyLM Challenge 2025 winners.
Takeaways & Limitations
Limited-data pre-training should be treated as its own scaling regime, with architectures matched to fixed-data constraints rather than simply shrinking web-scale models.
Takeaways & Limitations
Hyperparameter tuning was not performed separately for every combination of corpus, budget, model size, and recurrent depth.
Abstract
from arXiv · showhide
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.
1 Introduction
Limited-data pre-training behaves differently from web-scale scaling: model size becomes a regularization choice, and useful compute must be scaled without proportionally increasing trainable capacity. The paper studies RecursiveGPT as an architectural response across data budgets and evaluations.
- Motivation: Under fixed data and abundant compute, increasing model size changes the bias–variance tradeoff because additional parameters cannot be paired with additional data.This makes model size itself a regularization choice.
- Architectural bottleneck: Standard Transformers couple per-token computation to representational capacity, partly because embeddings and the language-model head can dominate small models’ parameter budgets.
- Approach: RecursiveGPT combines factorized embeddings with recursive weight sharing to reduce vocabulary-map parameters and increase computational depth without proportional parameter growth.
- Study scope: The study spans 10M–100M-word budgets, two corpora, model sizes from 13M to 1.2B parameters, and multiple benchmarks.
- Findings: Optimal model size depends strongly on both the data budget and the downstream evaluation, while standard architectures scale down poorly.
- Results: RecursiveGPT outperforms standard Transformers at both 10M and 100M words while remaining competitive with BabyLM Challenge 2025 winners.
- Positioning: The released training pipeline positions RecursiveGPT as a compute-scaling pathway under fixed data, complementary to specialized data-efficient training methods.
2 Background
Limited-data language modeling creates a tension between reducing parameters to control overfitting and preserving enough computation for each token. The background motivates recursive Transformers and factorized embeddings as architectural ways to separate these concerns.
- Limited-data regime: When data is the binding constraint, larger models can overfit fixed corpora, making parameter count a form of regularization.
- Scaling bottleneck: With fixed vocabulary size, shrinking hidden width reduces Transformer-block parameters roughly quadratically but embedding parameters only linearly.
- Scaling bottleneck: Small language models can therefore allocate most parameters to token embeddings rather than to computation performed at each layer.
- BabyLM Challenge: The BabyLM Challenge evaluates language models under at most 10M- or 100M-word budgets for 10 epochs without restricting FLOPs.
- BabyLM Challenge: BabyLM 2025 systems use varied objectives and training recipes, including masked, combined causal-and-masked, and diffusion-style masked language modeling.
- Recursive Transformers: Recursive Transformers share a transition function across depth, so increasing recurrent depth adds computation without another full set of block parameters.
- Factorized embeddings: Factorized embeddings replace a V × H matrix with V × E and E × H matrices, separating embedding size E from Transformer hidden size H.
- Factorized embeddings: When E ≪ H, factorized vocabulary maps use O(V E + E H) parameters instead of O(V H) while leaving hidden size unchanged.
3 Optimal Scale for Standard Transformers Under Data Scarcity
Standard Transformers have no single optimal scale under data scarcity: the preferred parameter count varies with both the data budget and downstream evaluation. Larger models can continue fitting the training data while downstream generalization saturates or declines.
- Scaling Experiment Setup: The scaling experiment sweeps standard Transformers across 10M–100M-word budgets, two corpora, and evaluations including BLiMP, LAMBADA pass@5, and COMPS.The setup uses 12 layers and hidden sizes from 256 to 3072, with factorized embeddings for the three smallest models.
- Scaling Behavior by Evaluation: BLiMP saturates relatively early, whereas COMPS continues benefiting from larger models, especially at 50M and 100M words.LAMBADA becomes more scale-favoring as dataset size grows, while low-budget curves flatten or decline earlier.
- Scaling Behavior by Evaluation: Performance is non-monotonic in parameter count, and the preferred scale differs substantially across downstream evaluations.The same qualitative pattern remains broadly stable on ClimbMix despite shifts in absolute scores.
- Scaling Behavior by Evaluation: Training loss decreases steadily with parameter count even when downstream performance saturates or declines, consistent with overfitting at small data budgets.The losses are directly comparable because all data subsets use the same tokenizer and vocabulary.
- Scaling Behavior by Evaluation: Once model size exceeds the data-dependent optimum, adding parameters no longer reliably improves generalization and can amplify overfitting.The results motivate increasing per-token computation without proportionally increasing trainable parameters.
4 RecursiveGPT Architecture
RecursiveGPT separates computational depth from parameter count by repeatedly applying a shared Transformer block and reducing vocabulary-map parameters with factorized embeddings. Its depth sweep shows useful gains up to an intermediate recurrent depth, followed by a slight decline at larger depths.
- Architecture: RecursiveGPT applies a shared causal-decoder Transformer block recurrently across depth, using a fixed number of recurrent steps.The design is inspired by ALBERT and Universal Transformer, while keeping depth explicit to address optimization difficulties.
- Architecture: Increasing recurrent depth adds computation without introducing another full set of block parameters.Factorized embeddings reduce parameters spent on vocabulary maps, while most block weights are reused across steps.
- Architecture: Each recurrent step has separate normalization and bias parameters that provide lightweight depth conditioning.The learned biases function as a lightweight depth embedding, while attention and MLP weights remain shared.
- Implementation: An expansion factor of 16 is used in the feed-forward sublayer because MLP parameters scale linearly with hidden size while total parameters scale quadratically.This helps reach the desired parameter count without simply increasing hidden size.
5 Results
Across limited-data evaluations, recursive Transformers provide a compute-scaling alternative to standard models, with strongest gains at 10M words and competitive performance against BabyLM baselines.
- 46.16 average score is achieved by the strongest recursive setting, versus 45.07 for the best standard model and 44.93 for an unfactorized standard model.
- At 10M words, RecursiveGPT outperforms both standard models on all three evaluations.
- At 100M words, RecursiveGPT-Large reaches the best average score among the authors’ models while remaining smaller than the 1.22B standard baseline.
- At 10M words, RecursiveGPT leads BLiMP and EWoK, whereas AMLM leads COMPS and average score.
- At 100M words, both RecursiveGPT models outperform GPT-BERT on every reported benchmark and the overall average.
- 5.2 Ablations: Removing factorized embeddings substantially reduces performance, while shared normalization slightly improves BLiMP but lowers the other scores and average.
- Increasing standard-model compute through more training epochs degrades performance at both word budgets when FLOPs are matched.
6 Discussion
The discussion frames RecursiveGPT as an architectural route for scaling compute under fixed data, while emphasizing optimization and computation-control limitations.
- Factorized embeddings and recursive weight sharing address parameter allocation and compute-depth coupling, respectively.
- RecursiveGPT is intended as an architectural base that can remain orthogonal to specialized data-efficient training techniques.
- Recursive weight sharing makes optimization more delicate because recurrent transitions can amplify perturbations or add little useful computation.
- Performance improves through recurrent depth 16 before declining slightly at larger depths, reflecting the trade-off in repeated computation.
- Adaptive halting and depth scheduling could reduce future recursive-model costs, but the paper does not use these techniques.
7 Conclusion
The paper concludes that limited-data pre-training is a distinct scaling regime in which recursive computation can improve performance without proportionally increasing parameter count.
- Across 10M–100M word budgets, standard Transformers often show non-monotonic performance with parameter count.
- The best standard-model size depends on both the data budget and the downstream target.
- RecursiveGPT decouples recurrent depth from parameter count, creating a separate axis for scaling compute under fixed data.
- RecursiveGPT improves over causal GPT-BERT on all 100M-word evaluations and remains close to the best 10M-word models.
- Future work includes combining recursion with specialized objectives, adaptive halting, depth scheduling, partial sharing, and richer depth conditioning.
Limitations
The study’s conclusions are constrained by limited hyperparameter tuning and a narrow explored recursive-Transformer design space, with deeper recurrence increasing training and inference cost.
- Limitations: Hyperparameters were not retuned for every combination of corpus, data budget, model size, and recurrent depth.The authors state that a full search was computationally infeasible with their resources.
- Limitations: The study covers only a small portion of the recursive Transformer design space.The models use one shared block, fixed recurrent steps, and simple per-depth conditioning parameters.
- Limitations: At 100M words, LAMBADA pass@5 is 9.4 points lower on ClimbMix, although qualitative dependence on data budget and evaluation target remains similar across corpora.BLiMP stays within 1.2 points, while COMPS reaches a maximum gain of 1.6 points on ClimbMix.
B Architecture Details
The standard baseline is a 12-layer decoder-only causal Transformer with fused attention components, gated attention outputs, zero-initialized projections, and optional factorized embeddings.
- Architecture: The baseline uses 12 independent pre-norm Transformer blocks with causal self-attention and dense two-layer ReLU2 MLPs.The architecture has no recursive weight sharing or mixture-of-experts layers.
- Attention: Attention uses fused QKV projections, 64-dimensional heads, RoPE, query-key normalization, variable-length causal attention, and FlashAttention 2.A learned per-head sigmoid gate is applied to the scaled dot-product attention output.
- Initialization: Attention and MLP output projections are zero-initialized.
- Embeddings: Input embeddings are untied from the language-model head, and both use ALBERT-style factorized projections where applicable.The vocabulary contains 32,768 BPE tokens.
- Optimization: Transformer blocks use Muon with neuronwise adaptive learning rates, while embeddings, normalization, and auxiliary parameters use Adam.The two optimizer groups use learning rates 0.02 and 0.005, respectively.
- Prediction analysis: Figure 7 is a histogram of settle depths for RecursiveGPT-Large, separating all 65,536 analyzed token positions from final document tokens.
- Prediction analysis: Figure 8 compares final agreement with stable-from-depth agreement relative to the depth-24 prediction.
C Prediction Depth and Adaptive Computation
RecursiveGPT’s intermediate predictions often stabilize before the final recurrent step, suggesting a possible adaptive-halting opportunity, while compute accounting scales shared-block execution with recurrent depth.
- Prediction dynamics: Settle depth is the earliest recurrent depth whose top-1 prediction matches and remains unchanged from the final depth-24 prediction.The analysis uses 65,536 tokens from randomly selected documents in the 100M-word RecursiveGPT-Large model.
- Prediction dynamics: 19.1% of all tokens settle at depth 24, 13.5% at depth 14, and 10.8% at depth 23.The last-token positions show a similar pattern.
- Adaptive computation: 80.9% of tokens are stable by depth 23, while 29.9% require the last two depths.About half are stable by depth 16 and 70.1% by depth 22.
- Compute accounting: Effective parameter counting excludes vocabulary maps but includes dense embedding-to-hidden and hidden-to-embedding projections.This convention is used in the FLOP estimate.
- Compute accounting: Estimated training compute is 6NeffT, with 10 epochs corresponding to approximately 130M BPE tokens at 10M words and 1.3B at 100M words.
- Compute accounting: For recursive models, only the shared block is multiplied by recurrent depth when computing effective parameters.Step-specific normalization and bias parameters and fixed projections are counted once.
- Embedding analysis: The factorized-embedding sweep found E=192 highest on average, E=64 strongest on BLiMP, and E=320 strongest on LAMBADA pass@5, without a precise optimum.Nearby settings differed only slightly.
F Compute Matching with Additional Epochs
The study also compares recursive models with standard Transformers trained for many additional epochs to approximately match recursive-model compute.
- Compute matching: At 10M words, standard baselines were trained for 146 epochs, and at 100M words, for 50 epochs.These schedules approximately match the estimated training FLOPs of RecursiveGPT at the respective budgets.