PaperlayerView exact revision on arXiv

Source-linked AI summary

Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

Francois Meyer

arXiv:2609.01151v1cs.CL

TL;DR

Fixed tokenisation limits language models, while prior learnable-tokenisation evidence was narrow in linguistic scope. The paper develops SubSegGPT and SubSegDeBERTa for BabyLM pretraining and finds track-dependent gains, alongside higher computational cost.

  • Problem

    Fixed subword tokenisation can misalign with morphemes and may be suboptimal for language-model learning, while it remains unknown whether learnable tokenisation improves sample-efficiency beyond prior narrow evaluations.

  • Method

    The paper develops SubSegGPT, a decoder-only GPT-2-based model, and SubSegDeBERTa, an encoder-based DeBERTa-v2 model that learns tokenisation during autoregressive or masked-language pretraining.

  • Results

    SubSegDeBERTa is strongest in STRICT, improving average zero-shot performance by 3.16 points over GPT-2, while SubSegGPT outperforms baselines in STRICT-SMALL.

  • Takeaways & Limitations

    Learnable subword tokenisation is effective across BabyLM settings, with SubSegDeBERTa and SubSegGPT showing complementary strengths at different data scales.

  • Takeaways & Limitations

    Subword segmental modelling improves sample-efficiency at the cost of compute-efficiency; SubSegDeBERTa required approximately 5× the A100 GPU hours of DeBERTa.

Abstract

from arXiv · show

In the standard LM training pipeline, subword tokenisation is applied as a preprocessing step. Subword segmental language modelling is an alternative paradigm in which tokenisation is learned during training, allowing the model to discover subword units that optimise its training objective. In this paper, we present our submission to the 2026 BabyLM Challenge, for which we develop two new subword segmental LMs: SubSegGPT and SubSegDeBERTa. SubSegGPT is a decoder-only model that learns tokenisation during autoregressive pretraining. SubSegDeBERTa is an encoder-based model that jointly learns to generate and tokenise masked words. We train both for the Strict and Strict-small tracks. Our top submission to Strict is SubSegDeBERTa, which achieves notable gains in zero-shot evaluation. Our top submission to Strict-small is SubSegGPT, which outperforms tokenisation-based baselines. Our results show that learnable subword tokenisation can improve sample-efficiency for BabyLM pretraining. We analyse the subword learning dynamics of our models and find that tokenisation gradually converges on subword units that balance morphological alignment and fine-grained segmentation.

1 Introduction

The paper replaces fixed preprocessing tokenisation with subword segmental models that learn tokenisation during training, targeting developmental plausibility and sample-efficient BabyLM pretraining. Across the 2026 BabyLM tracks, SubSegDeBERTa is strongest in STRICT, while SubSegGPT is better suited to STRICT-SMALL.

  • Standard tokenisers may misalign with morpheme boundaries and are developmentally implausible because children learn to segment language incrementally.
  • Frequency-based tokeniser objectives provide no guarantee that subword units optimise language-model learnability, which may matter especially in small-data settings.
  • Subword segmental modelling learns tokenisation end-to-end by marginalising over all possible tokenisations and optimising the language-model objective.
  • SubSegGPT is a decoder-only GPT-2-based model, whereas SubSegDeBERTa is an encoder-based DeBERTa-v2 model that jointly generates and tokenises masked words.
  • SubSegDeBERTa improves average zero-shot performance by 3.16 points over GPT-2 in STRICT, while SubSegGPT outperforms baselines in STRICT-SMALL.
  • Both models show rapid initial tokenisation change followed by stabilised trajectories toward higher fertility and limited morpheme-boundary alignment.

2 Background

Subword segmental language modelling treats tokenisation as a latent component jointly learned with language modelling. It marginalises over bounded, word-internal segmentations using character-level context and dynamic programming, extending earlier segmentation work toward modern BabyLM architectures.

  • Earlier segmental models learned task-specific segmentation for handwriting, speech, translation, word discovery, or language modelling, but SSLM targets subword tokenisation.
  • Previous SSLM evaluations focused on low-resource, agglutinative Nguni languages, leaving broader pretraining sample-efficiency uncertain.
  • SSLM treats tokenisation as a latent variable inferred jointly with model parameters, allowing any LM architecture to learn tokenisation during training.
  • For a sequence S, SSLM computes probability by summing probabilities across all candidate tokenisations, rather than using one predetermined subword sequence.
  • Candidate subword segments cannot exceed maximum character length L, and word boundaries prevent segments from spanning whitespace.
  • Next-subword probabilities use the untokenised preceding character context, while dynamic programming makes marginalisation over candidate tokenisations tractable.

3 Models

The paper introduces SubSegGPT and SubSegDeBERTa, two subword segmental language models that learn tokenisation jointly with language modelling. SubSegGPT targets autoregressive decoding, while SubSegDeBERTa extends the framework to masked, bidirectional word generation.

  • SubSegGPT: SubSegGPT adapts GPT-2 into a decoder-only subword segmental language model for autoregressive training.Its character-level backbone conditions next-subword probabilities on the preceding character history.
  • SubSegGPT: SubSegGPT mixes a fixed frequent-subword lexicon with a character-level decoder that can generate arbitrary segments.A learned gate dynamically balances the lexicon and character-generation probabilities.
  • SubSegGPT: Dynamic programming marginalises over candidate tokenisations, enabling SubSegGPT to train end-to-end by negative log-likelihood.The model jointly optimises autoregressive language modelling and subword tokenisation.
  • SubSegDeBERTa: SubSegDeBERTa is an encoder-based masked subword segmental model and the first such extension to masked language modelling.It masks words and jointly learns to generate and tokenise them using bidirectional character-level context.
  • SubSegDeBERTa: SubSegDeBERTa computes masked-word probabilities by marginalising over candidate tokenisations and trains with masked-word negative log-likelihood.The method uses dynamic programming and limits segment length to a preset character maximum.
  • SubSegDeBERTa: A word context encoder combines bidirectional masked-word context with left-to-right within-word history for subword scoring.A one-layer character-level LSTM produces representations at every character position.

4 Experimental Setup

The experiments pretrain both models in the BabyLM STRICT and STRICT-SMALL tracks and compare them with matched fixed-tokenisation baselines. Evaluation covers zero-shot, finetuning, and human-likeness tasks using the official pipeline with model-specific zero-shot wrappers.

  • Baselines: Matched GPT-2 and DeBERTa-v2 baselines isolate the effect of learning tokenisation instead of applying fixed tokenisation.The baselines use corresponding backbone architectures and the same training data.
  • Model configuration: The study matches backbone size to the BabyLM GPT-2 baseline while adding subword-segment scoring subnetworks.Character-based embeddings contribute negligibly to total parameter count.
  • Evaluation: The official evaluation pipeline includes zero-shot linguistic-knowledge tasks, finetuning tasks on (Super)GLUE, and human-likeness tasks.Human-likeness results are reported in the appendix because both models and baselines perform poorly there.
  • Evaluation: Model-specific wrappers transform sentence or word marginals into the per-position quantities required by zero-shot evaluation.The official pipeline remains unchanged except for one line of code.

5 Results

Learnable tokenisation improves zero-shot performance, especially at the larger STRICT scale, while SubSegGPT is better suited to STRICT-SMALL. These gains diminish for finetuned language-understanding tasks in STRICT but persist for SubSegGPT in STRICT-SMALL.

  • Zero-shot evaluation: SubSegGPT outperforms both tokenisation-based baselines in STRICT zero-shot evaluation.
  • Zero-shot evaluation: SubSegGPT outperforms both baselines in STRICT-SMALL, whereas SubSegDeBERTa does not outperform DeBERTa and reaches similar average performance.
  • Zero-shot evaluation: SubSegGPT gains +3.57 for BLiMP and +5.52 for BLiMP Supplement over GPT-2 on the only STRICT-SMALL tasks where models reliably exceed chance.
  • Scale dependence: SubDeBERTa requires a larger data scale for its sample-efficiency gains, while SubSegGPT is better suited to the more data-constrained setting.
  • Finetuning evaluation: Finetuning diminishes subword-segmental benefits in STRICT, but SubSegGPT achieves the best overall STRICT-SMALL performance and transfers its pretraining sample-efficiency to downstream finetuning.

6 Analysing Subword Learning

The models’ tokenisations change rapidly at first, then stabilise into finer-grained segmentations with limited morphological alignment. Qualitative examples show both morpheme discovery and architecture- or assumption-related violations.

  • Quantitative dynamics: The analysis tracks boundary flip rate, fertility, and morpheme-boundary F1 across checkpoints to measure tokenisation change, granularity, and morphological alignment.
  • Quantitative dynamics: During STRICT-SMALL pretraining, both models rapidly change tokenisation initially, then converge on a settled scheme with increasing fertility.
  • Quantitative dynamics: The learned tokenisations are more aggressively segmented than the BabyLM 16k-vocabulary BPE baseline, whose fertility is 1.23.
  • Linguistic properties: The models balance partial morphological alignment against finer-grained segmentation, reflecting demands of sample-efficient language modelling.
  • Linguistic properties: Examples include suffixes such as “–ing” and “–s” and compound constituents such as “sand–box,” alongside morphologically unsound splits.
  • Limitations: Removing architectural and modelling constraints could permit unrestricted subword learning, but marginalising over unbounded segment lengths is computationally infeasible.

7 Conclusion

The paper introduces two modern subword segmental language models that learn tokenisation during pretraining: SubSegGPT for autoregressive modelling and SubSegDeBERTa for masked modelling. Their complementary performance across data scales supports end-to-end subword modelling as a direction for BabyLM research.

  • SubSegGPT and SubSegDeBERTa learn tokenisation during training to optimise autoregressive and masked-language-model pretraining objectives, respectively.
  • SubSegDeBERTa is the strongest submission in STRICT, while SubSegGPT performs strongly in STRICT-SMALL.
  • The results show that subword segmental language models are effective beyond the low-resource, agglutinative languages targeted by earlier work.
  • The findings suggest reconsidering fixed tokenisation conventions and identify end-to-end subword modelling as a future direction for BabyLM research.

8 Limitations

Subword segmental modelling improves sample-efficiency at the cost of compute-efficiency, and hyperparameter exploration was limited rather than exhaustive.

  • Computational cost: Subword segmental training requires more computation because it marginalises over several tokenisations instead of using a single fixed tokenisation.
  • Computational cost: Training SubSegDeBERTa required approximately 5× the A100 GPU hours of DeBERTa.
  • Computational cost: The sample-efficiency objective therefore comes with lower compute-efficiency, including more training FLOPs and longer training times.
  • Hyperparameter search: The authors did not perform a full grid search and varied one hyperparameter at a time from BabyLM baseline defaults.
  • Hyperparameter search: Most hyperparameter experiments used STRICT-SMALL for faster iterations, with more limited experimentation in STRICT.

B Evaluation Tasks

The evaluation covers zero-shot, finetuning, and human-likeness tasks across the STRICT and STRICT-SMALL tracks, using behavioural and acquisition-based measures for the latter.

  • The official evaluation pipeline contains zero-shot, finetuning, and human-likeness task types for the STRICT and STRICT-SMALL tracks.
  • Zero-shot: Zero-shot evaluation includes BLiMP, BLiMP Supplement, EWoK, COMPS, Entity Tracking, and English Global PIQA.
  • Finetuning: Finetuning evaluation includes BoolQ, MultiRC, RTE, WSC, MRPC, QQP, and MNLI.
  • Human likeness: Human-likeness evaluation uses reading-time and eye-tracking correlations plus age-of-acquisition comparisons with child vocabulary learning curves.

C Evaluation Wrappers

Because the models produce marginal probabilities rather than fixed-vocabulary per-position logits, the authors implement wrappers that convert their outputs for the official evaluation tasks.

  • The wrappers transform marginal probabilities into quantities required by evaluation pipelines because the models lack fixed-vocabulary per-position logits.
  • Zero-shot: SubSegGPT converts full-sentence log-probabilities into per-character estimates so summed positions recover the true sentence log-probability.
  • Zero-shot: The SubSegGPT wrapper disables sentence masking so minimal-pair evaluation compares full sentence log-probabilities.
  • Human likeness: For human-likeness tasks, word-level surprisal is computed as −log p(w_i | w_<i), normally by summing subword-token log-probabilities.
  • Zero-shot: SubSegDeBERTa supports pseudo-log-likelihood scoring by masking each word and summing its marginal probability across the sentence.

D Human-Likeness Results

The models show weak correspondence with human psycholinguistic data: they slightly improve reading-time prediction, but baseline surprisals perform better, and acquisition scores show no child-data correlation.

  • Neither model exhibits strong correlations with human psycholinguistic data.
  • Tokenisation dynamics: Learned tokenisations undergo rapid early change before stabilising at higher fertility and limited morpheme-boundary alignment.
  • Reading time: Model word surprisal slightly improves reading-time prediction, but baseline surprisals produce greater improvements than SubSegGPT and SubSegDeBERTa.
  • Age of acquisition: AoA scores are zero or negative for all tested models, indicating no correlation between model word acquisition patterns and child acquisition data.
Loading 2609.01151v1…