PaperlayerView exact revision on arXiv

Source-linked AI summary

Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

Tingshuo Fan, Hongtao Mu, Tianyu Zhou, Hansen Liu, Tao Ji

arXiv:2609.09691v1cs.CLcs.AI

TL;DR

Low-resource language modeling may benefit from more computation without proportionally increasing parameters, but the effectiveness of shared recurrent depth is uncertain. The paper studies Looped GPT-BERT with mixed masked and causal objectives in BabyLM Strict-small, finding comparable selected-task performance with fewer parameters alongside task-specific limitations.

  • Problem

    The paper asks whether recurrent computation from shared parameters can compensate for reduced physical depth and parameter capacity in low-resource language-model pretraining.

  • Method

    The study combines GPT-BERT’s hybrid MNTP/CLM objective with depth-wise recurrent parameter sharing and compares objective ratios, architectures, and loop counts in BabyLM Strict-small.

  • Results

    Looped models preserve or improve selected syntactic abilities and achieve comparable selected linguistic and downstream performance to public baselines with fewer parameters.

  • Takeaways & Limitations

    Under constrained data and parameter budgets, recurrently reused layers can support syntactic understanding and generation, while benefits vary across tasks.

  • Takeaways & Limitations

    The looped design is weaker on Entity Tracking, and gains from additional loops saturate quickly and differ across tasks.

Abstract

from arXiv · show

When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final $4\times12$ model uses four physical layers for twelve recurrent traversals and contains 12.18M parameters. The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48. Compared with public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model's representational space.

1 Introduction

This paper asks whether repeatedly applying shared parameters can compensate for reduced physical depth and parameter capacity in low-resource language-model pretraining. It introduces Looped GPT-BERT and evaluates objective ratios, recurrent depth, and parameter efficiency in BabyLM Strict-small.

  • Motivation: Shared-parameter recurrence can create a deeper computational path without proportionally increasing learned weights.The central question is whether recurrent computation can compensate for reduced physical depth and parameter capacity under constrained data and parameters.
  • Setting: BabyLM Strict-small provides a controlled evaluation setting for linguistic, downstream, and human-like behavior using developmentally plausible pretraining data.The study uses a cleaned 7.48M-word English corpus in the BabyLM 2026 Strict-small track.
  • Approach: Looped GPT-BERT combines GPT-BERT’s masked next-token and causal language-modeling objectives with depth-wise recurrent parameter sharing.The design builds on recurrent-depth and cross-layer sharing approaches while adapting them to language-model pretraining.
  • Research questions: The study examines how the BERT:GPT ratio, physical-layer count, and loop count affect training and downstream abilities.It compares a 12-layer non-looped model with four physical layers applied 1, 3, 6, or 12 times.
  • Contribution: The final 12.18M-parameter model achieves comparable performance to public Strict-small baselines on linguistic and downstream tasks.The paper evaluates whether recurrent computation retains language ability with substantially fewer parameters.

2 Related Work

Prior work motivates parameter sharing and recurrent depth as ways to improve learning efficiency, but does not establish that shared loops suit all language capabilities. This paper positions its contribution as a direct evaluation across linguistic and state-tracking tasks.

  • BabyLM and data-efficient pretraining: BabyLM research evaluates learning efficiency under controlled data budgets, where architecture, objective, preprocessing, and curriculum can affect small-model performance.Prior findings report no single approach dominating every downstream task and show that data composition effects depend on model scale.
  • Hybrid objectives: GPT-BERT combines causal and masked next-token prediction through a shared decoder-style vocabulary projection and loss interface.MNTP shifts masked-token supervision to the preceding position so causal and masked examples use the same next-token head.
  • Recurrent depth and sharing: Universal Transformers and ALBERT demonstrate recurrent or cross-layer sharing, while Looped Transformers increase effective depth by iterating a shared backbone.Looped Transformers use a fixed input and prior loop state, with parameter count governed mainly by physical layers.
  • Open question: Recurrent-depth Transformers can improve systematic generalization and depth extrapolation, but excessive loops may reduce prediction quality through overthinking.Existing evidence does not establish uniform suitability across lexical, syntactic, world-knowledge, and discourse-state learning.

3 Method

The method builds a cleaned small-data corpus, trains GPT-BERT with mixed causal and masked objectives, and replaces independently parameterized depth with recurrently reused physical layers. Loop count increases computation while keeping the shared backbone’s parameter count fixed.

  • 3.1 Corpus Preparation and Tokenization: Six source corpora are rule-cleaned into the new_data1 corpus, yielding 7,482,189 whitespace-delimited words.Sources include BNC Spoken, CHILDES, Project Gutenberg, OpenSubtitles, Simple English Wikipedia, and Switchboard.
  • 3.1 Corpus Preparation and Tokenization: The main and loop experiments use an 8k BPE tokenizer chosen as a parameter-allocation trade-off for the small-data setting.The official 16k tokenizer is used only for early non-looped ratio experiments.
  • 3.2 Hybrid GPT-BERT Objective: GPT-mode rows use causal attention and supervise every non-padding next token, whereas BERT-mode rows use bidirectional attention with selected masked targets shifted left.The shift keeps masked supervision aligned with GPT-BERT’s decoder-style next-token head.
  • 3.2 Hybrid GPT-BERT Objective: Only shifted positions corresponding to selected masks contribute to the BERT loss, while a batch-level BERT:GPT ratio assigns complete rows to the two modes.The final setting uses a 1:3 ratio, with one quarter of rows using MNTP and three quarters using CLM.
  • 3.3 Looped Transformer Backbone: The looped backbone has four independently parameterized physical layers and applies the same stack repeatedly to the static representation plus the prior loop state.The recurrence is z_t+1 = F_θ(x + z_t), with the resulting state carried into the next traversal.
  • 3.3 Looped Transformer Backbone: A 4 × L model performs 4L layer applications while retaining four sets of Transformer parameters, so loop count increases effective depth without making the model equivalent to a non-shared Transformer.The 4 × 3, 4 × 6, and 4 × 12 variants perform 12, 24, and 48 layer applications respectively.
  • 3.4 Training Configuration: Training uses LAMB with cosine decay followed by a constant minimum learning rate, and all runs use random seed 42.The reported peak learning rate is 0.0141 and the decay lasts 2,600 steps.

4 Experiments

The experiments evaluate objective mixing, looped depth, task performance, and inference cost in BabyLM Strict-small. Results show that shared-layer recurrence can preserve selected abilities with fewer parameters, but benefits vary by task and computation increases with loop count.

  • BabyLM 2026 Leaderboard Results: The final 4×12 model has 12.18M parameters and reaches 71.19 on BLiMP and 62.55 on GLUE.These results are compared with public GPT-2 and GPT-BERT causal-focus references.
  • Effect of the BERT:GPT Ratio: Increasing GPT-heavy supervision to a 1:3 BERT:GPT ratio lowers mixed training loss and improves BLiMP, BLiMP Supplement, and COMPS.Under the 8k tokenizer, Entity Tracking decreases from 38.66 to 22.87, showing task-dependent effects.
  • Effect of Looped Depth: Increasing loop count lowers training loss and generally raises token accuracy, with the largest improvement occurring from 4×1 to 4×3.Further increases from 4×3 to 4×6 and 4×12 provide smaller gains.
  • Effect of Looped Depth: Under the 1:3 ratio, 4×3 raises BLiMP from 65.08 to 71.21, slightly above the 12-layer non-looped score of 70.54.Beyond 4×3, additional loop counts do not produce consistent gains across tasks.
  • Effect of Looped Depth: Entity Tracking remains weaker for most looped variants, and under 1:3 its score falls from 28.66 at 4×6 to 15.78 at 4×12.The authors associate this limitation with reduced representational space from reusing four attention/FFN parameter sets.
  • Inference Cost: 4×3 improves latency, token throughput, and peak memory relative to the 12-layer non-looped model, while 4×6 offers a performance–inference-speed compromise.Increasing from 4×3 to 4×6 and 4×12 roughly doubles latency at each step, while throughput decreases and memory use increases.

5 Optimization Ablations

The optimization ablations examine peak learning rate and decay schedules for stable, effective training. Earlier decay improves early and middle training, while the selected schedule is a practical single-seed compromise rather than uniformly superior.

  • 5.1 Selecting the Peak Learning Rate: 0.0141 achieves the lowest final loss (3.6847) and is selected as the peak learning rate.The sweep found that 0.02 degraded after approximately step 600 and produced gradient-norm spikes, while 0.01 had the highest final token accuracy.
  • 5.2 Learning-Rate Decay and Stability: 2,600-step decay lowers loss faster and raises token accuracy sooner than 4,080-step decay in early and middle training.The 4,080-step schedule gradually catches up later, but the 2,600-step curves are smoother overall.
  • 5.2 Learning-Rate Decay and Stability: The 2,600-step schedule is chosen as a practical compromise between stability and late-stage exploration under the current compute budget and single-seed setting.It is not uniformly superior at every point in training.
  • 5.2 Learning-Rate Decay and Stability: The decay comparison uses 4 × 6 and 4 × 12 models, with solid and dashed lines representing 2,600-step and 4,080-step decay.The 2,600-step schedule reaches 10% of the peak learning rate and then holds 0.00141, whereas the slower schedule maintains a higher late-stage rate.

6 Conclusion

The paper combines GPT-BERT’s hybrid objective with depth-wise recurrent parameter sharing to study parameter-efficient language modeling under BabyLM Strict-small constraints. A 12.18M-parameter looped model performs comparably to larger references on selected metrics, while task-specific weaknesses and rapidly saturating loop gains remain.

  • 6 Conclusion: Looped GPT-BERT combines GPT-BERT’s hybrid CLM/MNTP training with depth-wise recurrent parameter sharing.The design studies a “fewer parameters, more computation” approach under BabyLM Strict-small constraints.
  • 6 Conclusion: 12.18M parameters support four physical layers reused for twelve loops, compared with a 29.9M non-looped model with a similar training objective.The comparison concerns parameter efficiency rather than matched computation.
  • 6 Conclusion: 71.19 on BLiMP and 62.55 on GLUE compare with 71.66 and 65.13 for the official GPT-BERT causal-focus reference.These results are reported for the smaller final looped model.
  • 6 Conclusion: Looped models preserve or improve selected syntactic abilities, while Entity Tracking becomes weaker with parameter sharing.Additional loops saturate quickly and produce different effects across tasks.

Limitations

The study’s conclusions are bounded by its small-data, small-model, short-context, single-seed, and unmatched-compute evaluation design. The authors caution against directly extrapolating these findings to larger models or longer contexts.

  • Limitations: The experiments use one cleaned corpus, one 8k tokenizer, and a single random seed, limiting conclusions about preprocessing, tokenization, and training variance.Sub-percentage-point differences may fall within training variance.
  • Limitations: The 4 × 12 model performs substantially more layer applications and computation than the 12-layer non-looped baseline, so this is not a FLOP-matched comparison.The study evaluates parameter efficiency rather than equal computation.
  • Limitations: Changing the BERT:GPT ratio also changes supervision density and attention visibility, so mixed training losses are not perfectly homogeneous across ratios.This complicates direct interpretation of objective-ratio comparisons.
  • Limitations: The final model is weak on AoA and Entity Tracking, indicating limited cognitive-similarity and state-tracking ability.These weaknesses define a capability boundary for the reported model.
  • Limitations: The conclusions are limited to approximately 12M parameters, a 7.48M-word corpus, ten epochs, and a maximum sequence length of 128.They should not be directly extrapolated to larger models or longer contexts.

Ethics Statement

The model uses English text from BabyLM sources and introduces no newly collected personal data. Source content may retain biases or inappropriate material, while cleaning may remove or reshape language patterns.

  • Ethics Statement: The model uses English text supplied by or derived from BabyLM and introduces no newly collected personal data.The sources include books, subtitles, encyclopedic text, and dialogue.
  • Ethics Statement: Original source materials may retain social biases or inappropriate content because automated cleaning cannot guarantee their removal.This follows from the composition of the source data and the limits of the cleaning process.
  • Ethics Statement: Cleaning may remove dialectal, conversational, or minority-language patterns and introduce stylistic preferences through normalization.The paper states that normalization can shape the resulting data distribution.

Reproducibility Statement

The final model and intermediate checkpoints are available in a Hugging Face repository, alongside loading code and training resources.

  • The Hugging Face repository provides the final model, sixteen intermediate checkpoints, model configuration and implementation files, and tokenizer files.The accompanying code repository includes training configurations and scripts.
Loading 2609.09691v1…